pointer_traits<P>::to_pointer(r) is not required to return a deallocatable pointer; indeed generally it *cannot* determine a deallocatable representation without help from the allocator. Therefore, we must not rely on representations derived from to_pointer when deallocating; we must pass to deallocate the exact same pointer (or at least a pointer linearly derived from the pointer) that we originally received from allocate.
I have an example fancy-pointer allocator here which might be convertible into a test case:
https://github.com/Quuxplusone/from-scratch/blob/master/include/scratch/bits/containers/segmented-allocator.h