Page MenuHomePhabricator

[mlir] make MLIRContext available to getDependentDialects
AbandonedPublic

Authored by ftynse on Jun 24 2022, 7:08 AM.

Details

Summary

The dependent dialect mechanism allows the pass manager to load upfront the
dialects containing the entities that may be produced by the passes. Until now,
it only allowed the pass to provide a fixed list of dialects. Make a refrence
to the MLIRContext object available in getDependentDialects API to make this
more flexible. For example, an interface-based pass may query if interface
implementations are provided for certain ops and only require the dialects to
be loaded when they are, thus potentially reducing the number of dialects to
load. This is particularly useful for external, out-of-tree, interface
implementations that need to be plugged into passes from a different tree.
Since most of the passes are unlikely to use this functionality, the original
context-less getDependentDialects overload is preserved as is.

Diff Detail

Event Timeline

ftynse created this revision.Jun 24 2022, 7:08 AM
Herald added a project: Restricted Project. · View Herald Transcript
ftynse requested review of this revision.Jun 24 2022, 7:08 AM
mehdi_amini requested changes to this revision.Jun 24 2022, 7:10 AM

I don't quite get the use case, but more importantly this makes the getDependentDialects "contextual": the context can change between the time getDependentDialects is called and the moment the pass pipeline runs.

This revision now requires changes to proceed.Jun 24 2022, 7:10 AM

See the child patch for the use case.

ftynse added a comment.Jul 7 2022, 9:38 AM

Ping.

We've hit another case that is solved by this: bufferization. For example, bufferization of tensor.generate produces scf.for. The logic is provided by the external interface implementation. The pass itself doesn't know about the exact logic and only uses the interface, but would have to still list the scf dialect as dependent. With this, the pass would be able to query the known interface implementations for the list of dependent passes via the context. I expect the same to be true for many other interface-based passes: transform, tiling, canonicalizer to some extent (now we have to load the dialects canonicalization depends on in parent dialect), etc.

but more importantly this makes the getDependentDialects "contextual"

Yes, and I think this is actually the desired behavior for interface-driven passes.

I know there is some explicit expectation of passes being somehow agnostic to what is registered/loaded in the context. However, this is more of a "thou shalt not" recommendation, since practically one can easily write a pass conditioned on something being available in the context by simply querying it in the pass entry function.

the context can change between the time getDependentDialects is called and the moment the pass pipeline runs.

Technically, it can. Practically, this seems highly unlikely. There are two dozen lines between the call to getDependentDialects and the passes actually running https://github.com/llvm/llvm-project/blob/6c3990acfbb933a61e2c74332bb252121c06bd14/mlir/lib/Pass/Pass.cpp#L791-L817, and they don't return control back to the caller. So there seem to be two possibilities: (1) another thread modifies the context between these two calls, but modifying the context concurrently sounds like a terrible idea to start with; (2) one of the getDependentDialects modifies the context, which is solvable by sprinkling some const on it or providing some immutable wrapper.

Ping.

I stil have the same concerns, despite the use cases you're trying to enable. I haven't had time to try to collect all the invariants this can potentially break and I don't think I'll be able to get to it before I'm back from my current vacation (on 7/18): this is tricky.

ftynse added a comment.Jul 8 2022, 2:24 AM

Okay, I'll ping you again then.

Leaving this here so I don't forget: we have always had a pass with registration-dependent behavior -- canonicalizer. Depending on whether the op is registered (unregistered ops can be created via Operation::create), it will or will not canonicalize it. This works because canonicalization and folding are privileged and have opaque hooks that the context is aware of. I suspect that if we were to write the canonicalizer today using interfaces, we would have hit the same issue. What I'd like is to generalize this sort of passes, with appropriate safety mechanisms, in a way that doesn't require privileged hooks in the context.

I don’t quite get what is privileged about the canonicalizer here? At least I don’t believe it comes from hooks in the context, the way it is handled there is by saying that canonicalization patterns introduce static dialect dependencies: you necessarily load dialect B when loading dialect A if a canonicalization pattern from dialect A is emitting an entity from dialect B.
But this is more of a policy than anything and we could move canonicalization patterns to use an interface without any change to how we manage this.

ftynse added a comment.EditedJul 8 2022, 2:51 AM

There are several privileged things that may be a bit conflated. For what you are asking specifically, we have these static dialect dependencies because of canonicalization. We don't introduce them because of other passes AFAIK. This makes the canonicalization privileged in a sense. We could also add these static dependencies because of other passes (e.g., tensor bufferizes to scf so it should depend on scf in the case above), but it doesn't sound like it would scale well. I am looking for a mechanism to generalize and modularize this.

Side note: don't feel pressured to answer, we can look at code and specific options separately.

Depending on whether the op is registered

Something I wanted to point out is that "registered" here has a different meaning: an op is "registered" after a dialect is loaded, not just merely registered.
Part of my concerns here (which I need to think more about, it is a bit fuzzy for me right now) is about acting on "registered but not loaded" here.

(unregistered ops can be created via Operation::create),

Just a nit: but unless the dialect is loaded, this isn't possible by default (there is a flag in the context to allow it explicitly, but that's not really how systems are intended to be setup). If the dialect is loaded, then yeah you can do that, but dialects are supposed to register their operation on load so I think it is "work as intended".

ftynse abandoned this revision.Mon, Jul 25, 4:26 AM

We discussed offline and found a different solution that loads dialects together with the extension.