This reverts commit 0ebc1fb29f278db0665423f15c53e6ee9601dddb.
The behaviour should have been the same as before unless specifying CMAKE_BUILD_RPATH,
which was previously broken.
However, this seems to have broken builds for some people that don't specify it.
Reverting until I can investigate.