I was looking at an improvement to InstSimplify when I noticed that this add fold in InstCombine was preventing icmp simplifies.
There's a later fold that special-cases (add (xor with signbit), C), so I don't think we're losing anything by limiting this.
'test20' and 'xor_sign_bit' in add.ll verify that the later fold still works, so there are no regressions in existing tests.