Adjusting by 2 breaks DWARF output. With this fix, programs start to compile and produce valid DWARF output.
I'm not quite sure whether this is the correct fix. I feel like this patches over a bug somewhere else. Feel free to suggest a different way to do it.