In D131528 using Info.EvalMode == EvalInfo::EM_ConstantExpression is not strict enough to restrict the diagnostic to only constant expression contexts. It is sometimes set in cases where we are still determining if we are in a constant expression context.
Using InConstantContext will tighten the restriction.