Load instructions go through the default case of SimplifyInstruction and use the heavy ConstantFoldInstruction which tries to recursively constant fold any constantexpr operands. I believe this is harder than we try for other instructions in InstSimplify. We usually use some entry point like ConstantFoldBinaryOpOperands or ConstantFoldCompareInstOperands which bypass this logic.
This patch calls directly to ConstantFoldLoadFromConstPtr if the load has a constant pointer and isn't volatile.
On one benchmark cpp file I ran through callgrind, this showed instructions executed in SimplifyInstruction reduce by almost half.