Actually no. I also think we can enable for earlier targets. (GCC documented it's enabled for all targets with VSX) Previously, I thought we did not tested targets <P8 well.
The backend implementation only depends on altivec. I think it is better to align with backend implementation. And check the cpu will also have problem if we have something like: -mcpu=pwr8 -mattr=-altivec
I understand that the back end only needs Altivec to support the operations, but enabling this on non-vsx subtargets is different from what GCC does. I don't see a good reason for us to depart from that.