Currently we do layout as if non alloc sections had an actual address and then set it to zero. This produces a few odd results where a symbol has an address that is inconsistent with the section address.
The simplest way to fix it is probably to just set the address earlier.
The behavior of bfd seems to be similar, but it only sets the non alloc section address if the section is missing from the linker script or if the script has an explicit " : 0" setting the address of the output section (which the default script does).