Use gcc's rules for parsing gcc-style response files

[thakis]

Summary

In gcc, \ escapes every character. It is true that this makes it harder to mention Windows files in rsp files, but not doing this means clang disagrees with gcc, and also disagrees with the shell (on non-Windows) which rsp file quoting is supposed to match. clang isn't free to choose what to do here.

In general, the idea for response files is to take bits of your command line and write them to a file unchanged, and have things work the same way. Since the command line would've been interpreted by the shell, things in the rsp file need to be subject to the same shell quoting rules.

People who want to put Windows-style paths in their response files either need to escape their backslashes, or use clang-cl which uses cl.exe/cmd.exe quoting rules.

Fixes PR27464.

[rnk]

lgtm

I guess my change was a bad idea. This is changing three year old behavior, but I don't think it will cause too much breakage.

[probinson]

In principle it's hard to disagree but this might be a backward-compatibility problem for PS4 (Windows hosted but using gcc style driver). At this point I honestly can't remember what our story is for response files. :-) but might not have a chance to investigate until Monday. Would you mind holding off just a bit?

lib/Support/CommandLine.cpp

535

Typo

[thakis]

Sure, no rush. I'm also adding an rsp quoting override flag in http://reviews.llvm.org/D19425 , maybe you could use that if this change is otherwise a problem for you?

[probinson]

In D19417#409377, @thakis wrote:

Sure, no rush. I'm also adding an rsp quoting override flag in http://reviews.llvm.org/D19425 , maybe you could use that if this change is otherwise a problem for you?

Thanks for waiting. We force Windows parsing and can use the rsp-quoting flag to set our default accordingly. So, this change is no problem on our part.

[thakis]

Thanks for checking! Landed in r267556 and update clang's tests to match in r267557.

lib/Support/CommandLine.cpp

static bool EatsUnboundedNumberOfValues(const Option *O) {
   return O->getNumOccurrencesFlag() == cl::ZeroOrMore ||
          O->getNumOccurrencesFlag() == cl::OneOrMore;
 }
 
 static bool isWhitespace(char C) { return strchr(" \t\n\r\f\v", C); }
 
 static bool isQuote(char C) { return C == '\"' || C == '\''; }
 
-static bool isGNUSpecial(char C) { return strchr("\\\"\' ", C); }
-
 void cl::TokenizeGNUCommandLine(StringRef Src, StringSaver &Saver,
                                 SmallVectorImpl<const char *> &NewArgv,
                                 bool MarkEOLs) {
   SmallString<128> Token;
   for (size_t I = 0, E = Src.size(); I != E; ++I) {
     // Consume runs of whitespace.
     if (Token.empty()) {
       while (I != E && isWhitespace(Src[I])) {
         // Mark the end of lines in response files
         if (MarkEOLs && Src[I] == '\n')
           NewArgv.push_back(nullptr);
         ++I;
       }
       if (I == E)
         break;
     }
 
-    // Backslashes can escape backslashes, spaces, and other quotes.  Otherwise
-    // they are literal.  This makes it much easier to read Windows file paths.
-    if (I + 1 < E && Src[I] == '\\' && isGNUSpecial(Src[I + 1])) {
+    // Backslashe escapes the next character.
+    if (I + 1 < E && Src[I] == '\\') {
       ++I; // Skip the escape.
       Token.push_back(Src[I]);
       continue;
     }
 
     // Consume a quoted string.
     if (isQuote(Src[I])) {
       char Quote = Src[I++];
       while (I != E && Src[I] != Quote) {
-        // Backslashes are literal, unless they escape a special character.
-        if (Src[I] == '\\' && I + 1 != E && isGNUSpecial(Src[I + 1]))
+        // Backslashes escape the next character.
+        if (Src[I] == '\\' && I + 1 != E)
           ++I;
         Token.push_back(Src[I]);
         ++I;
       }
       if (I == E)
         break;
       continue;
     }

unittests/Support/CommandLineTest.cpp

void testCommandLineTokenizer(ParserFunction *parse, const char *Input,
   EXPECT_EQ(OutputSize, Actual.size());
   for (unsigned I = 0, E = Actual.size(); I != E; ++I) {
     if (I < OutputSize)
       EXPECT_STREQ(Output[I], Actual[I]);
   }
 }
 
 TEST(CommandLineTest, TokenizeGNUCommandLine) {
-  const char *Input = "foo\\ bar \"foo bar\" \'foo bar\' 'foo\\\\bar' "
-                      "foo\"bar\"baz C:\\src\\foo.cpp \"C:\\src\\foo.cpp\"";
-  const char *const Output[] = { "foo bar", "foo bar", "foo bar", "foo\\bar",
-                                 "foobarbaz", "C:\\src\\foo.cpp",
-                                 "C:\\src\\foo.cpp" };
+  const char *Input =
+      "foo\\ bar \"foo bar\" \'foo bar\' 'foo\\\\bar' -DFOO=bar\\(\\) "
+      "foo\"bar\"baz C:\\\\src\\\\foo.cpp \"C:\\\\src\\\\foo.cpp\"";
+  const char *const Output[] = {
+      "foo bar",     "foo bar",   "foo bar",          "foo\\bar",
+      "-DFOO=bar()", "foobarbaz", "C:\\src\\foo.cpp", "C:\\src\\foo.cpp"};
   testCommandLineTokenizer(cl::TokenizeGNUCommandLine, Input, Output,
                            array_lengthof(Output));
 }
 
 TEST(CommandLineTest, TokenizeWindowsCommandLine) {
   const char *Input = "a\\b c\\\\d e\\\\\"f g\" h\\\"i j\\\\\\\"k \"lmn\" o pqr "
                       "\"st \\\"u\" \\v";
   const char *const Output[] = { "a\\b", "c\\\\d", "e\\f g", "h\"i", "j\\\"k",