This patch attempts to fix the undefined behavior in __tree by changing the node pointer types used throughout. The pointer types are changed for raw pointers in the current ABI and for fancy pointers in ABI V2 (since the fancy pointer types may not be ABI compatible).
The UB in __tree arises because tree downcasts the embedded end node and then deferences that pointer. Currently there are 3 node types in __tree.
- __tree_end_node which contains the __left_ pointer. This node is embedded within the container.
- __tree_node_base which contains __right_, __parent_ and __is_black. This node is used throughout the tree rebalancing algorithms.
- __tree_node which contains __value_.
Currently __tree stores the start of the tree, __begin_node_, as a pointer to a __tree_node. Additionally the iterators store their position as a pointer to a __tree_node. In both of these cases the pointee can be the end node. This is fixed by changing them to store __tree_end_node pointers instead.
To make this change I introduced an __iter_pointer typedef which is defined to be a pointer to either __tree_end_node in the new ABI or __tree_node in the current one.
Both __tree::__begin_node_ and iterator pointers are now stored as __iter_pointers.
The other situation where __tree_end_node is stored as the wrong type is in __tree_node_base::__parent_. Currently __left_, __right_, and __parent_ are all __tree_node_base pointers. Since the end node will only be stored in __parent_ the fix is to change __parent_ to be a pointer to __tree_end_node.
To make this change I introduced a __parent_pointer typedef which is defined to be a pointer to either __tree_end_node in the new ABI or __tree_node_base in the current one.
Note that in the new ABI __iter_pointer and __parent_pointer are the same type (but not in the old one). The confusion between these two types is unfortunate but it was the best solution I could come up with that maintains the ABI.
The typedef changes force a ton of explicit type casts to correct pointer types and to make current code compatible with both the old and new pointer typedefs. This is the bulk of the change and it's really messy. Unfortunately I don't know how to avoid it.
Please let me know what you think.