LegalizerHelper::insertParts uses extractGCDType on registers split into a desired type and a smaller leftover type. This is used to populate a list of registers. Each register in the list will have the same type as returned by extractGCDType.
If we have
- ResultTy = s792
- PartTy = s64
- LeftoverTy = s24
When we call extractGCDType, we'll end up with two different types appended to the list:
Part: gcd(792, 64, 24) => s8
Leftover: gcd(792, 24, 24) => s24
When this happens, we'll hit an assert while trying to build a G_MERGE_VALUES.
This patch changes the code for the leftover type so that we reuse the GCD from the desired type.
e.g.
Leftover: gcd(792, 8, 24) => s8
It also adds an assert that the types we get from both calls to extractGCDType are actually the same.
I was thinking of just writing out LLT GCDTy = getGCDType(getGCDType(..., ...), ...); here to avoid having to do the first reg separately.