In a statement like Map[A] = Map[B], first the right hand side is evaluated as a reference, then left hand side is evaluated. If the left hand side operator[] invocation grows the map, the previous reference may be invalidated.
GCC seems to dereference the right hand side reference only after evaluating the left hand side, while Clang dereferences it before. (If the value type is larger type, Clang also dereferences it after the left hand side operator[] call.)
With GCC, a cast to Value* isn't enough to make it dereference the right hand side reference before invoking (while that is enough to make Clang/LLVM do the right thing for larger types), but storing it in an intermediate variable in a separate statement works.