The change was reverted by r352722,
the original patch was reviewed in https://reviews.llvm.org/D56838
The quadratic behavior is avoided by moving the use referring the
inttoptr instruction to the beginning of the use list, making the
complexity of the inttoptr-look-ups amortized O(1).