Rather than assertion failing, we can fall back to the non-optimized version which works for any shadow scale.