This fixes file paths on Windows, as you can now write, for example, d:\foo\bar.txt, but does not break the case that this tokenization logic was originally designed for, which is to allow escaping of things like quotes and double quotes, so that all of the escapable characters can appear in the same string together.
I think this is a much simpler and more straightforward fix than adding a different tokenizer. Having only 1 tokenizer means we don't have to worry in the future about tests breaking because someone wrote their test on unix and tried to execute a command with a syntax that wouldn't work on Windows, etc. So hopefully we can reach an agreement about this fix.