This is an alternative to D105078 which doesn't require exporting anything additional in our modulemaps.
I'm still not sure exactly what the compiler is doing in D105078, but the gist seems to be that if class C (in module M) inherits publicly from class D (in module N), and M doesn't re-export N, then it's as if that inheritance relationship doesn't exist. If a TU that imports M calls a function f(c) that would ordinarily find f(D&), it'll actually unambiguously find f(...) or whatever other less-good candidates exist. In this particular case, a call to &c doesn't find D::operator& but rather prefers the built-in candidate.
The solution adopted by me here is: don't inherit, then. "Inheritance is for sharing an interface"; here we don't want the interface to be shared; so, eliminate the inheritance and the modules problem goes with it.