Looks like function specialisation was running at all optimisation levels (if enabled on the command line, it is not on by default). I think that was an oversight and not something we want to do. Function specialisation duplicates functions when it triggers, so the backend is processing more functions/instructions resulting in compile-time increases, which seems more appropriate with -O3 and inline with GCC.
Please note that since function specialisation is not enabled by default, this didn't require updating any pass manager tests.
Do we need to consider SizeLevel here?