If they're left in the cache then they can't be removed efficiently when the cache is notified to unlink a @llvm.assume call, and that can lead to values from different functions entirely remaining there. ValueTracking.cpp does not like this.
There aren't that many users of the assumption cache so we could modify them to cope with this situation, but I decided the cache was still messed up and most RAUW calls are fairly swiftly followed by removing the original so not much is lost by transferring any assumptions.
Unfortunately I couldn't write a test for this, the example I had was huge (from Clang's ExprConstant) and didn't reproduce under opt even with the whole file.