X86 codegen uses function attribute min-legal-vector-width to select the proper ABI. The intention of the attribute is to reflect user's requirement when they passing or returning vector arguments. So Clang front-end will iterate the vector arguments and set min-legal-vector-width to the width of the maximum for both caller and callee.
It is assumed any middle end optimizations won't care of the attribute expect inlining and argument promotion.
- For inlining, we will propagate the attribute of inlined functions because the inlining functions become the newer caller.
- For argument promotion, we check the min-legal-vector-width of the caller and callee and refuse to promote when they don't match.
The problem comes from the optimizations' combination, as shown by https://godbolt.org/z/zo3hba8xW. The caller foo has two callees bar and baz. When doing argument promotion, both foo and bar has the same min-legal-vector-width. So the argument was promoted to vector. Then the inlining inlines baz to foo and updates min-legal-vector-width, which results in ABI mismatch between foo and bar.
This patch fixes the problem by expanding the concept of min-legal-vector-width to indicator of functions arguments. That says, any passes touch functions arguments have to set min-legal-vector-width to the value reflects the width of vector arguments. It makes sense to me because any arguments modifications are ABI related and should response for the ABI compatibility.