driver: Add a `--rsp-quoting` flag to choose which unquoting behavior to use in rsp files.

[thakis]

Summary

Currently, clang-cl always uses Windows style for unquoting, and clang always uses POSIX style for unquoting.

In general, response file quoting should match the shell the response file is used in. On Windows, it's possible to run clang-cl in a bash shell, or clang in cmd.exe, so a flag for overriding the default behavior is natural there. On non-Windows, Windows probably never makes sense (except maybe in Wine), but having clang-cl behave differently based on the host OS seems strange too -- people who want to use clang-cl on non-Windows with "native" response files will have to pass --rsp-quoting=posix.

This also includes the clang-side test changes needed for http://reviews.llvm.org/D19417

[rnk]

Looks good

tools/driver/driver.cpp

350

Should we call it "posix" or "gnu"? The reference implementation we are trying to match is part of libiberty, which is why I called the tokenizer TokenizeGNUCommandLine.

[thakis]

Thanks!

tools/driver/driver.cpp

350

I figured the reference implementation tries to implement http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html from posix, but I don't care much about the name. If you prefer gnu, I'm happy to use gnu. (In the same vein: Do you have an opinion on "windows" vs "cl" vs "msvc"?)

[rnk]

tools/driver/driver.cpp

350

I buy that reasoning, let's use "posix" then.

I also like "windows" for the other style. The algorithm documented at https://msdn.microsoft.com/en-us/library/17w5ykft(v=vs.85).aspx is pretty standard. New versions of mingw use the exact same algorithm.

[thakis]

Landed the new flag and its tests in r267474, thanks! (Didn't land the test updates needed after http://reviews.llvm.org/D19417 yet, since that's not in yet.)

include/clang/Driver/Options.td

def internal_debug_Group :
   OptionGroup<"<clang debug/development internal options>">,
   Group<internal_Group>, HelpText<"DEBUG/DEVELOPMENT OPTIONS">;
 
 class InternalDriverOpt : Group<internal_driver_Group>,
   Flags<[DriverOption, HelpHidden]>;
 def driver_mode : Joined<["--"], "driver-mode=">, Group<internal_driver_Group>,
   Flags<[CoreOption, DriverOption, HelpHidden]>,
   HelpText<"Set the driver mode to either 'gcc', 'g++', 'cpp', or 'cl'">;
+def rsp_quoting : Joined<["--"], "rsp-quoting=">, Group<internal_driver_Group>,
+  Flags<[CoreOption, DriverOption, HelpHidden]>,
+  HelpText<"Set the rsp quoting to either 'posix', or 'windows'">;
 def ccc_gcc_name : Separate<["-"], "ccc-gcc-name">, InternalDriverOpt,
   HelpText<"Name for native GCC compiler">,
   MetaVarName<"<gcc-path>">;
 def ccc_pch_is_pch : Flag<["-"], "ccc-pch-is-pch">, InternalDriverOpt,
   HelpText<"Use lazy PCH for precompiled headers">;
 def ccc_pch_is_pth : Flag<["-"], "ccc-pch-is-pth">, InternalDriverOpt,
   HelpText<"Use pretokenized headers for precompiled headers">;
 

lib/Driver/Driver.cpp

void Driver::BuildJobs(Compilation &C) const {
   // about any unused arguments.
   if (Diags.hasErrorOccurred() ||
       C.getArgs().hasArg(options::OPT_Qunused_arguments))
     return;
 
   // Claim -### here.
   (void)C.getArgs().hasArg(options::OPT__HASH_HASH_HASH);
 
-  // Claim --driver-mode, it was handled earlier.
+  // Claim --driver-mode, --rsp-quoting, it was handled earlier.
   (void)C.getArgs().hasArg(options::OPT_driver_mode);
+  (void)C.getArgs().hasArg(options::OPT_rsp_quoting);
 
   for (Arg *A : C.getArgs()) {
     // FIXME: It would be nice to be able to send the argument to the
     // DiagnosticsEngine, so that extra values, position, and so on could be
     // printed.
     if (!A->isClaimed()) {
       if (A->getOption().hasFlag(options::NoArgumentUnused))
         continue;

test/Driver/at_file.c

 // RUN: %clang -E %s @%s.args -o %t.log
 // RUN: FileCheck --input-file=%t.log %s
 // RUN: %clang -E %s @%s.args.utf16le -o %t.log
 // RUN: FileCheck --input-file=%t.log %s
 
 // CHECK: bar1
 // CHECK-NEXT: bar2 zed2
 // CHECK-NEXT: bar3 zed3
 // CHECK-NEXT: bar4 zed4
 // CHECK-NEXT: bar5 zed5
 // CHECK-NEXT: 'bar6 zed6'
 // CHECK-NEXT: "bar7 zed7"
 // CHECK-NEXT: foo8bar8zed8
 // CHECK-NEXT: foo9'bar9'zed9
 // CHECK-NEXT: foo10"bar10"zed10
 // CHECK: bar
-// CHECK: zed12
+// CHECK: zed1
 // CHECK: one\two
-// CHECK: c:\foo\bar.c
+// CHECK: c:foobar.c
 
 foo1
 foo2
 foo3
 foo4
 foo5
 foo6
 foo7

test/Driver/at_file.c.args

 -Dfoo1=bar1 -Dfoo2="bar2 zed2"
 -Dfoo3='bar3 zed3'
 "-Dfoo4=bar4 zed4"
 '-Dfoo5=bar5 zed5'
 -Dfoo6="'bar6 zed6'"
 -Dfoo7='"bar7 zed7"'
 -Dfoo8=foo8"bar8"zed8
 -Dfoo9=foo9\'bar9\'zed9
 -Dfoo10=foo10\"bar10\"zed10
 -D foo11
--Dfoo12=zed12\
+-Dfoo12=zed1\
+2
 -Dfoo13='one\\two'
 -Dfoo14='c:\foo\bar.c'

test/Driver/at_file_win.c

+// RUN: %clang --rsp-quoting=windows -E %s @%s.args -o %t.log
+// RUN: FileCheck --input-file=%t.log %s
+
+// CHECK: bar1
+// CHECK-NEXT: bar2 zed2
+// CHECK-NEXT: bar3 zed3
+// CHECK-NEXT: bar4 zed4
+// CHECK-NEXT: bar5 zed5
+// CHECK-NEXT: 'bar6 zed6'
+// CHECK-NEXT: 'bar7 zed7'
+// CHECK-NEXT: foo8bar8zed8
+// CHECK-NEXT: foo9\'bar9\'zed9
+// CHECK-NEXT: foo10"bar10"zed10
+// CHECK: bar
+// CHECK: zed12
+// CHECK: one\two
+// CHECK: c:\foo\bar.c
+
+foo1
+foo2
+foo3
+foo4
+foo5
+foo6
+foo7
+foo8
+foo9
+foo10
+#ifdef foo11
+bar
+#endif
+foo12
+foo13
+foo14

test/Driver/at_file_win.c.args

+-Dfoo1=bar1 -Dfoo2="bar2 zed2"
+-Dfoo3="bar3 zed3"
+"-Dfoo4=bar4 zed4"
+"-Dfoo5=bar5 zed5"
+-Dfoo6="'bar6 zed6'"
+-Dfoo7='"bar7 zed7"'
+-Dfoo8=foo8"bar8"zed8
+-Dfoo9=foo9\'bar9\'zed9
+-Dfoo10=foo10\"bar10\"zed10
+-D foo11
+-Dfoo12=zed12
+-Dfoo13=one\two
+-Dfoo14=c:\foo\bar.c

tools/driver/driver.cpp

int main(int argc_, const char **argv_) {
   // response files written by clang will tokenize the same way in either mode.
   bool ClangCLMode = false;
   if (TargetAndMode.second == "--driver-mode=cl" ||
       std::find_if(argv.begin(), argv.end(), [](const char *F) {
         return F && strcmp(F, "--driver-mode=cl") == 0;
       }) != argv.end()) {
     ClangCLMode = true;
   }
+  enum { Default, POSIX, Windows } RSPQuoting = Default;
+  for (const char *F : argv) {
+    if (strcmp(F, "--rsp-quoting=posix") == 0)
+      RSPQuoting = POSIX;
+    else if (strcmp(F, "--rsp-quoting=windows") == 0)
+      RSPQuoting = Windows;
+  }
 
   // Determines whether we want nullptr markers in argv to indicate response
   // files end-of-lines. We only use this for the /LINK driver argument with
   // clang-cl.exe on Windows.
-  bool MarkEOLs = false;
+  bool MarkEOLs = ClangCLMode;
 
-  llvm::cl::TokenizerCallback Tokenizer = &llvm::cl::TokenizeGNUCommandLine;
-  if (ClangCLMode) {
+  llvm::cl::TokenizerCallback Tokenizer;
+  if (RSPQuoting == Windows || (RSPQuoting == Default && ClangCLMode))
     Tokenizer = &llvm::cl::TokenizeWindowsCommandLine;
-    MarkEOLs = true;
-  }
+  else
+    Tokenizer = &llvm::cl::TokenizeGNUCommandLine;
 
   if (MarkEOLs && argv.size() > 1 && StringRef(argv[1]).startswith("-cc1"))
     MarkEOLs = false;
   llvm::cl::ExpandResponseFiles(Saver, Tokenizer, argv, MarkEOLs);
 
   // Handle -cc1 integrated tools, even if -cc1 was expanded from a response
   // file.
   auto FirstArg = std::find_if(argv.begin() + 1, argv.end(),