All data structures and values associated with handling virtual functions / inheritance, as well as RTTI, are globals and thus can only reside in the global address space. This was not taken fully taken into account because for most targets, global & generic appear to coincide. However, on targets where global & generic ASes differ (e.g. AMDGPU), this was problematic, since it led to the generation of invalid bitcasts (which would trigger asserts in Debug) and less than optimal code. This patch does two things:
- ensures that vtables, vptrs, vtts, typeinfo are generated in the right AS, and populated accordingly;
- removes a bunch of bitcasts which look like left-overs from the typed ptr era.
This is a bit more noisy than I'd have liked, but functionality is somewhat spread out. There's one bit of less than ideal code, stemming from the fact that functions are in the generic AS, and thus it's necessary to insert a constexpr cast from generic to global when populating the vtable. Adjusting appears disruptive enough to prefer to do it separately (unless I missed something obvious).
I noticed that we have some old fixes downstream that conflicts with the changes you've made here. I thought that perhaps we could get rid of those now when you've fixed the code upstream.
Isn't the VTable holding function pointers when not using the relative layout, and then this should be a pointer to the ProgramAddressSpace and not a pointer to the DefaultGlobalsAddressSpace?
Downstream we've been using a special FnVoidPtrTy here. Defined as FnVoidPtrTy = Int8Ty->getPointerTo(DL.getProgramAddressSpace());.