What is the optimization (TL;DR version):
The transformation tries to convert a __dynamic_cast function call, into an address comparison and VFT-like lookup, when the following conditions are met:
- the destination type is a leaf type, i.e. is never derived from (similar to C++ final semantics) in the entire program.
- the static type of the expression being casted is a public base (potentially multi-base and never private) class of the destination type.
Given the C++ expression: NULL != dynamic_cast<A*>(ptr) // where B* ptr; which coming out of clang would look like so: NULL ! = __dynamic_cast(ptr, &_ZTI1B, // typeinfo of B, the static type of ptr. &_ZTI1A, // typeinfo of A, the destination type. hint) // a static hint about the location of the source subobject w.r.t the complete object.
If the above conditions can be proven to be true, then an equivalent expression is:
(destType == dynamicType) where: std::typeinfo *destType = &_ZTI1A; std::typeinfo *dynamicType = ((void**)ptr)[-1];
A C++ dynamic_cast<A*>(ptr) expression can either
- be folded by the frontend into a static_cast<A*>(ptr), or
- converted to a runtime call to __dynamic_cast if the frontend does not have enough information (which is the common case for dynamic_cast).
The crux of the transformation is trying to prove that a type is a leaf.
We utilize the !type metadata (https://llvm.org/docs/TypeMetadata.html) that is attached to the virtual function table (VFT) globals to answer this question.
For each VFT, the !type MD lists the other VFTs that are "compatible" with it. In general, the VFT of a class B is considered to be "compatible" with the VFT of a class A, iff A derives (publicly or privately) from B.
This means that the VFT of a leaf class type is never compatible with any other VFT, and we use this fact to decide which type is a leaf.
The second fact that we need to prove is the accessibility of the base type in the derived object.
Unfortunately we couldn't find a way to compute this information from the existing IR, and had to introduce a custom attribute that the Frontend would place on the __dynamic_cast call. The presence of the attribute implies that the static type (B in our example) is a public base class and never a private base class (in case there are multiple subobjects of the static_type inside the complete object) of the destination type (A in our example). Hence, if the attribute gets deleted by some pass, our transformation will simply do nothing for that __dynamic_cast call.