Teach Clang how to use response files when calling other tools

[rafaelauler]

Summary

This patch addresses PR15171 and teaches Clang how to call other tools with response files, when the command line exceeds system limits. This is a problem for Windows systems, whose maximum command-line length is 32kb.

I introduce the concept of "response file support" for each Tool object. A given Tool may have full support for response files (e.g. MSVC's link.exe) or only support file names inside response files, but no flags (e.g. Apple's ld64, as commented in PR15171), or no support at all (the default case). Therefore, if you implement a toolchain in the clang driver and you want clang to be able to use response files in your tools, you must override a method (getReponseFileSupport()) to tell so.

I designed it to support different kinds of tools and internationalisation needs:

• VS response files ( UTF-16 )
• GNU tools ( uses system's current code page, windows' legacy intl. support, with escaped backslashes. On unix, fallback to UTF-8 )
• Clang itself ( UTF-16 on windows, UTF-8 on unix )
• ld64 response files ( only a limited file list, UTF-8 on unix )

With this design, I was able to test input file names with spaces and international characters for Windows. When the linker input is large enough, it creates a response file with the correct encoding. On a Mac, to test ld64, I temporarily changed Clang's behavior to always use response files regardless of the command size limit (avoiding using huge command line inputs). I tested clang with the LLVM test suite (compiling benchmarks) and it did fine.

Note: It depends on http://reviews.llvm.org/D4896 landing in the LLVM tree.

[thakis]

(prior art: http://reviews.llvm.org/D4104)

[rafaelauler]

In this new patch, I now use writeFileWithEncoding by passing an enum member (conforming to my new patch in D4896). I also updated my changes in Tool, separating ResponseFileSupport in ResponseFileSupport, which will only tell whether the tool has full support for response files, file lists or no support, and ResponseFileEncoding, which will tell which encoding we should use when writing this response file.

I believe this makes the whole encoding issue much more clear and explicit. I also adopted Rafael's suggestion and added a comment explictly stating which tools need to use the current code page in their response file encoding.

[rafaelauler]

Rebase & update the patch according to rnk's and rafael's suggestions on D4896.

[silvas]

Lots of naked const char * type stuff here (that includes unique_ptr<char[]>). Better to use StringRef, SmallVectorImpl<char>&, and raw_ostream& where possible, maybe with judicious use of BumpPtrAllocator, memcpy, and strcpy.

There are a lot of places where you have to separate loops: one for computing a buffer size, and another for actually doing work. This is a maintenance problem (will the length computation stay in sync with the code?) and also causes two traversals over the memory. By directly doing push_back's onto a vector or writing into a raw_ostream, you avoid doing two traversals.

lib/Driver/Job.cpp

122 ↗

(On Diff #12873)

This would be a lot nicer with ArrayRef<const char *>; that way, you won't lose the length.

169 ↗

(On Diff #12873)

Check your naming here and in other places.

lib/Driver/Tools.h

546–550 ↗

(On Diff #12873)

Ew... let's not have a bunch of random #ifdef's all over the code outside libSupport.

[rafaelauler]

Hi Sean,

Thanks a lot for your careful review. I have answered your questions below, and I have few concerns. Please let me know if I am missing something:

This code is on the critical path for commands with very large lengths and this is a non-negligible time. For UNIX, for example, this would be triggered for commands spanning more than 2MB of arguments and Clang spends a good amount of time to parse even trivial arguments, because it is a huge number of arguments.

Another important point is that, currently, the Command class stores arguments as a list of char *.

This is the reason for this design with lots of naked char *. It avoids creating StringRefs, that would measure the length of char* in their constructor and cause more overhead. It also avoids using high-level classes such as streams and bump allocators because, since this is likely to be called only when the number of arguments is very large, would cause them to repeatedly perform heap allocations until they finally settle to support the final size of the string. Either that or we would need to work with very large slabs, wasting memory. In this design, the length is calculated once and a single allocation is made, similar to what is done in Execute() in Support/Windows/Program.inc.

Therefore, by calculating the size of the final command line beforehand, even though it looks like we are losing by iterating over all args twice (one for length calculation and other for parsing), we win by eliminating inefficiencies in the allocation mechanism in runtime or in memory consumption.

Regards,
Rafael

lib/Driver/Job.cpp

122 ↗

(On Diff #12873)

That's a good point. However, I am afraid that I would still need to recalculate the length by traversing each argument, calculating its length and assessing whether it needs quoting or not.

169 ↗

(On Diff #12873)

Thanks for pointing out, I fixed the naming. Will submit the patch soon.

lib/Driver/Tools.h

546–550 ↗

(On Diff #12873)

I agree that this is inadequate. My previous patch version delegated this reasoning to libSupport. However, it was more difficult to understand and I changed it for clarity. Now, it is obvious that the response file encoding for this tool depends on whether it is running on Windows or not. In my previous libSupport version, I had enum members that meant "this is UTF on UNIX but UTF16 on Windows", and so on. I think this is not a good design because if you need to support a new tool, you may potentially need to add a new enum member in libSupport to encode your tuple <MyUNIXEncoding, MyWindowsEncoding>.

Since the encoding depends on the tool and on the operating system, I would appreciate if you have any suggestion on how to code this. Meanwhile, I will think in something.

[rafaelauler]

Updating patch wrt to API changes in D4896. Fixing naming.

[rafaelauler]

Removed the ifdefs from several instances of getResponseFileEncoding() by using a new datatype called EncodingStrategy, introduced in the llvm side of the patch at D4896. This datatype stores the desired encoding on both UNIX and Windows systems, letting the writeFileWithEncoding() function in Support to select the appropriate encoding to use.

[rafaelauler]

Conforming with my last update on D4896, change the encodings to be exclusive for Windows platforms, making it clear that we always use UTF8 on Unix systems. Convert ResponseFile from const char * to StringRef.

[• rafael]

include/clang/Driver/Job.h

87 ↗

(On Diff #12960)

printArgsWithRespFile

100 ↗

(On Diff #12960)

The function should be lowercase, the variable Uppercase.

lib/Driver/Job.cpp

32 ↗

(On Diff #12960)

I wonder if we should say that the job needs a response file based only on llvm::sys::argumentsFitWithinSystemLimits and then emit an error if Creator.getResponseFilesSupport() says it is not supported.

279 ↗

(On Diff #12960)

The code in PrintArgsWithRespFile looks fairly duplicated with what just follows. Couldn't this just patch Arguments and use the existing code path?

[rafaelauler]

Hi Rafael,

I have answered your comments below, thanks. I will upload a patch fixing your concerns shortly after this message.

Sean, regarding the change to llvm::sys::argumentsFitWithinSystemLimits() to avoid code duplication, I am afraid it is not necessary after my new patch. I will send it now.

Regards,
Rafael

include/clang/Driver/Job.h

87 ↗

(On Diff #12960)

changed

100 ↗

(On Diff #12960)

changed

lib/Driver/Job.cpp

32 ↗

(On Diff #12960)

The problem is that llvm::sys::argumentsFitWithinSystemLimits() is not exact, see:

bool llvm::sys::argumentsFitWithinSystemLimits(ArrayRef<const char*> Args) {

static long ArgMax = sysconf(_SC_ARG_MAX);

// System says no practical limit.
if (ArgMax == -1)
  return true;

// Conservatively account for space required by environment variables.
ArgMax /= 2;

...
Thus, sometimes it may say that you need a response file but, when building the argv vector, you discover that it actually fits in system limits. Therefore, I think that our best action is to silently try to pass the arguments without a response file because it may actually work.

279 ↗

(On Diff #12960)

Done.

[rafaelauler]

This update refactors the new code in Job.cpp to use streams, no longer building and managing its own char buffers. Thanks to Sean’s suggestion, the code now is much more simple. However, I wanted to build a stream that could directly write to the response file with the correct encoding, but raw_fd_ostream lacks the capability to write files with different encodings. Therefore, I added this capability to raw_fd_ostream with a small modification and introduced a new constructor variant that creates buffers that, when flushed to a file, is written in a different encoding. If you think it is inadequate to have such a feature in raw_fd_ostream, I can work on creating my own stream class to do so.

I also refactored part of the write function in raw_fd_ostream out of this class, to avoid duplicating code in this implementation. Then, I changed by writeFileWithEncoding() function to write in a given file descriptor, which is assumed to be opened, rather than opening the file by my own and then closing it. This enabled me to make raw_fd_ostream work with different encodings with little extra code.

[silvas]

lib/Driver/Job.cpp

105 ↗

(On Diff #13116)

Tiny nit: Use nullptr instead of 0

133–139 ↗

(On Diff #13116)

In the future, consider using std::find_if instead of writing this function http://en.cppreference.com/w/cpp/algorithm/find

I'm not sure a linear search here is acceptable though, see my comment below.

144–150 ↗

(On Diff #13116)

This seems like an unusual thing to have a helper for. The real problem seems to be that this is part of a larger block of code which is duplicated. Just pulling out this part doesn't help things: get rid of this helper function and deduplicate a larger scope.

163 ↗

(On Diff #13116)

Returning a SmallVector by value is undesirable. in this case, I think NRVO will negate part of the issue (large sizeof), but you still have the problem that this hardcodes what SmallVector size the caller is using. I would recommend switching this to taking a non-const SmallVectorImpl<const char *> &Out instead of returning a SmallVector.

172 ↗

(On Diff #13116)

isInInputList seems to be doing linear work. If Arguments is mostly Inputs, then this will go quadratic. Is that a potential problem?

185–186 ↗

(On Diff #13116)

Use ArrayRef here.

187–190 ↗

(On Diff #13116)

Simpler:

for (auto Arg : Inputs)
  OS << Arg << '\n';

Also, for clarity, consider just saying const char * instead of auto.

205 ↗

(On Diff #13116)

I don't think this is correct. ResponseFile.data() isn't guaranteed to be 0 terminated (and you don't want to embed that hidden assumption). Maybe use .append(begin, end)?

216–232 ↗

(On Diff #13116)

This loop scares me. Could you clean this up in a separate patch? Or at least add comment here for now.

235 ↗

(On Diff #13116)

This call seems duplicated, along with a lot of stuff in this if. Pull this into a helper?

lib/Driver/Tools.h

309 ↗

(On Diff #13116)

Tiny nit: Don't use virtual with override.

531–537 ↗

(On Diff #13116)

This seems highly repetitive. Is there a way to reduce it?

Also, I'm not sure that it makes sense to have a method returning WindowsEncodingMethod for a netbsd tool (same goes for most tools that have no reason to run on windows).

[rafaelauler]

Sean,

Thanks for the fast reply. I answered some of your comments below. The remaining comments were fixed in the next patch (will upload it now).

Best regards,
Rafael Auler

lib/Driver/Job.cpp

205 ↗

(On Diff #13116)

Great observation. I'll change ResponseFile back to a const char * because, otherwise, I would need to copy it to a null-terminated string storage, since I may need to directly store it in the argv vector (which only contains null-terminated strings).

216–232 ↗

(On Diff #13116)

Sure, will work on it in a separate patch.

lib/Driver/Tools.h

531–537 ↗

(On Diff #13116)

WindowsEncodingMethod, when used for UNIX tools, covers the case where these tools run in MinGW environments (in a cross-compilation setup).

[rafaelauler]

I addressed Sean's comments regarding my last patch update:

• Job.cpp: Removed the quadratic time complexity from buildArgv (for file lists)
• Job.cpp: Removed/Refactored duplicated code
• Job.cpp/Tools.h: Implemented other minor changes
• Tools.h: Refactored/Created a superclass for all GNU tools that have the same behaviour when it comes to response files, avoiding repetitive member function implementations across many classes.

[silvas]

This is looking a lot better.

In particular, I feel like the testing needs a bit more work; I'm not seeing the connection between the added tests and the functional changes in this patch.

I'd like espindola to give a second opinion on the code, and he can probably suggest improvements to the testing.

include/clang/Driver/Job.h

103 ↗

(On Diff #13150)

Is there ever any situation where NeedsResponseFile == true but ResponseFile != nullptr? In general, what is the meaning of the 4 different possibilities for

(ResponseFile {=,!}= nullptr) x (NeedsResponseFile {=,!}= true)

[• rafael]

My main comment is on the llvm side. We really can't have something as core as raw_fd_ostream handling char conversion. Some options are

• build a buffer and write it at once.
• have another streamer implementation. Hopefully implement it in a way that avoids reasoning about partial multibyte chars. A simple option might be to build the entire buffer in memory inside the streamer and convert on close (and error on sync?). Even for large response files, I expect the work being done by the sub process to always dwarf the cost of creating it, so we probably don't need to get fancy.

I have one more nit on the code itself and one on the tests.

include/clang/Driver/Tool.h

81 ↗

(On Diff #13150)

All the overrides of these 3 methods always return a symbol constant, no? Maybe it would be better to make them non-virtual:

ResponseFileSupport getResponseFilesSupport() const {

return RespFileSupport;

}

and set RespFileSupport in the constructor of the base classes. We do things like that in, for example, MCAsmInfo.

test/Driver/response-file.c

17 ↗

(On Diff #13150)

Please don't introduce an use of awk :-)

Can you use clang -E to produce a large file on the fly? Is 2MB really too big for current linux systems?

[rafaelauler]

Hi Sean and Rafael,

Thanks for your feedback. I answered your comments below. I will also upload a patch now, both on the LLVM and on the Clang side, addressing Rafael's concerns regarding the streams and his virtual removal suggestion.

Regarding the testing, I had some difficulty in figuring out good tests for the response files mechanism. Personally, to ensure that the mechanism works fine, I change needsResponseFile to always return true and use clang in several regular scenarios to check that the response files work.

Without this hack, to test it, I need to truly generate a very big response file, which involves exercising the compilation of a large project (especially to test -filelist in ld64, which cannot be tested with our -DTEST -DTEST... approach). I guess this kind of test is more appropriate to the test suite (compiling big projects).

One thing I can think of now is to create a new flag "-force-response-file" for testing purposes. Is this reasonable?

Best regards,
Rafael Auler

include/clang/Driver/Job.h

103 ↗

(On Diff #13150)

In particular, we use this to protect against a user of this class that does not call setResponseFile() even if we return true in needsResponseFile(). If the user did not configure a response file, ResponseFile will be a nullptr and we need to disable resp file usage.
On the other hand, if the user *did* call setResponseFile() but we don't need it (NeedsResponseFile = false), we should also disable response file usage.

test/Driver/response-file.c

17 ↗

(On Diff #13150)

I'm no fan of awk, but I had a small problem with this test. Previously, I used a shell "for" loop to generate a large response file on the fly, but this made this test unavailable to windows machines without bash.

To make this work on Windows, I looked at what LLVM required from its Windows users to correctly run tests, and I saw that it requires that the user installs the gnuwin32 package with many tools, including awk. Awk was the only tool that I could think of where I could implement a loop to synthesise this large response file on the fly, making this test work both on unix and on windows machines.

Sorry but I'm not sure if I understood what you said about using clang -E to produce a large file on the fly. If you are asking whether clang -E uses response files, the idea here -- and it is actually based on a post that you wrote a longe time ago, giving ideas to test response files :-) -- is to pass many -D params to the driver, which, in turn, will be forced to use a response file when calling "clang -cc1 -E".

Yes, 2MB is enough to trigger response files in current linux systems. However, I can make the file larger, but the time of this test will also rise (a lot, depending on the size and if clang was compiled with -O0). The bottleneck is in Clang's parameter processing logic, which is stressed in this test.

[rafaelauler]

Hi,

This is the updated patch implementing:

• using a raw_string_ostream to buffer the response file contents before calling writeFileWithEncoding() to write all the contents at once.
• removing virtual from Tool's member functions that deal with response files, following Rafael's suggestion.

Best regards,
Rafael Auler

[• rafael]

Without awk this is fine by me :-)

The idea of adding -force-response-file as a testing option would work, but I am not sure whether it is worth it or not.

Please just wait for Sean to give his OK too.

test/Driver/response-file.c

17 ↗

(On Diff #13191)

My idea for using clang -E to generate the test is something like

clang -E response-gen.c | grep -v \# > %t2

with response-gen.c being

#define M -DTEST
#define M2 M M
#define M4 M2 M2
#define M8 M4 M4
#define M16 M8 M8
#define M32 M16 M16
#define M64 M32 M32
#define M128 M64 M64
#define M256 M128 M128
M256

with as many MXXX as you need.

[rafaelauler]

Hi Rafael,

Thanks for the suggestion on removing awk from the lit test, I integrated it into this new version of the patch.

[• rafael]

LGTM.

Sean, what do you think?

[silvas]

I'm still uneasy about NeedsResponseFile (see my comment below).

Also a few style things.

include/clang/Driver/Job.h

101–104 ↗

(On Diff #13239)

I still don't think this makes sense. You previously replied that this is done to "protect the user", but I think that two combinations are programming errors: We should assert if:
ResponseFile == nullptr and NeedsResponseFile == true
ResponseFile != nullptr and NeedsResponseFile == false
(if this is not the case, please provide an example where this could happen and clang would behave correctly)

This leaves only two valid options:
ResponseFile == nullptr and NeedsResponseFile == false
ResponseFile != nullptr and NeedsResponseFile == true

So NeedsResponseFile is redundant.

It seems like right now you are using NeedsResponseFile just as a flag that roughly means "remember to do some other work at some faraway place in the code". I really don't like having such a random coupling across disparate parts of the code. Can you centralize the logic of checking !llvm::sys::argumentsFitWithinSystemLimits(_Arguments) && Creator.getResponseFilesSupport() != Tool::RF_None with the logic of setting ResponseFile?

lib/Driver/Job.cpp

120–122 ↗

(On Diff #13239)

Can we simplify this by just always quoting?

124–129 ↗

(On Diff #13239)

Please make this a for loop so that it is easy to see what is being looped over. In particular, so that the Arg++ is easily visible.

for (; *Arg != '\0'; Arg++)

lib/Driver/Tools.cpp

5854–5865 ↗

(On Diff #13239)

Simpler:

for (const auto &II : Inputs) {
  if (!II.isFilename())
    break;
  InputFileList.push_back(II.getFilename());
}

[rafaelauler]

Hi Sean,

I agree that having a flag that triggers action in a far away code is bad, and, as such, I was studying the code to see if I could do a larger refactoring to put all response-file-related stuff in a single place, freeing us from any far away code that is important to response files. However, this turned out to be tougher than I think. Because we need both information from Tool objects and from the Driver (to determine whether the command will actually run, in which case we build a resopnse file), I couldn't centralize everything, but I did centralize the small logic that you requested. Unfortunately, we still have some far away code that is important (in Tools.cpp when we call setInputFiles(), crucial to enable support for file lists in ld64), but I believe that this is the best we can do. Therefore, I ended up with only minor modifications regarding the last patch, although I did address all your concerns.

After some assessment and because the Driver is the top level class that calls the other objects (Compilation, Job etc) to run a command, I put the small logic that determines whether to use response files in a Driver private member function. The reason for this is because we don't need to allocate a response file if we are not running the commands (and there are some code paths that instantiate Driver, Compilation and Job objects just to fetch the arguments, but will never run the commands). Therefore, in the Driver, we are able to accurately determine when the command will actually run, triggering the code that determines where response files are needed just before running the command. Also, the Driver seemed a natural place to put such control logic. On the other hand, I had to remove the constness of the Driver::ExecuteJobs -- notice that this function will now provision resources for response files (their name) and, therefore, can't be const. This is a small change that does not impact any code in the tree -- in fact, no user expected ExecuteJobs to be const.

I also addressed your other (minor comments). Yet, I took a look at the loop you mentioned that it would be important to be cleaned up (in Command::Print) and couldn't determine a good way to refactor this. I saw the logic there, and it seems that it is very particular of what we want when receiving crash reports (skip some arguments and quote others). When you mentioned that this needs to be cleaned up, did you mean to remove such idiosyncrasies and just send all arguments to the crash report, or just a smaller refactoring?

Thanks for putting in time to see this,
Rafael Auler

[silvas]

LGTM.

As for Command::Print, I was just hoping for an easy way to see what it is doing, but I guess from what you are telling me it is not so easy. It's not a huge deal.

[rafaelauler]

If it is okay for Rafael too, could you please commit this patch? I don't
have commit access.

Best regards,
Rafael Auler

[rnk]

Closed by commit rL217792 (authored by @rnk).

[rnk]

I saw this hadn't landed in a week, so I went ahead and did it in r217792.

[rafaelauler]

Thanks Reid.

cfe/trunk/include/clang/Driver/Driver.h

 
 namespace clang {
 namespace driver {
 
   class Action;
   class Command;
   class Compilation;
   class InputInfo;
+  class Job;
   class JobAction;
   class SanitizerArgs;
   class ToolChain;
 
 /// Driver - Encapsulate logic for constructing compilation processes
 /// from a set of gcc-driver-like command line arguments.
 class Driver {
   llvm::opt::OptTable *Opts;
private:
   llvm::opt::DerivedArgList *
   TranslateInputArgs(const llvm::opt::InputArgList &Args) const;
 
   // getFinalPhase - Determine which compilation mode we are in and record 
   // which option we used to determine the final phase.
   phases::ID getFinalPhase(const llvm::opt::DerivedArgList &DAL,
                            llvm::opt::Arg **FinalPhaseArg = nullptr) const;
 
+  // Before executing jobs, sets up response files for commands that need them.
+  void setUpResponseFiles(Compilation &C, Job &J);
+
 public:
   Driver(StringRef _ClangExecutable,
          StringRef _DefaultTargetTriple,
          DiagnosticsEngine &_Diags);
   ~Driver();
 
   /// @name Accessors
   /// @{
public:
   ///
   /// \param C - The compilation that is being built.
   void BuildJobs(Compilation &C) const;
 
   /// ExecuteCompilation - Execute the compilation according to the command line
   /// arguments and return an appropriate exit code.
   ///
   /// This routine handles additional processing that must be done in addition
-  /// to just running the subprocesses, for example reporting errors, removing
-  /// temporary files, etc.
-  int ExecuteCompilation(const Compilation &C,
-     SmallVectorImpl< std::pair<int, const Command *> > &FailingCommands) const;
+  /// to just running the subprocesses, for example reporting errors, setting
+  /// up response files, removing temporary files, etc.
+  int ExecuteCompilation(Compilation &C,
+     SmallVectorImpl< std::pair<int, const Command *> > &FailingCommands);
   
   /// generateCompilationDiagnostics - Generate diagnostics information 
   /// including preprocessed source file(s).
   /// 
   void generateCompilationDiagnostics(Compilation &C,
                                       const Command *FailingCommand);
 
   /// @}

cfe/trunk/include/clang/Driver/Job.h

class Command : public Job {
 
   /// The executable to run.
   const char *Executable;
 
   /// The list of program arguments (not including the implicit first
   /// argument, which will be the executable).
   llvm::opt::ArgStringList Arguments;
 
+  /// Response file name, if this command is set to use one, or nullptr
+  /// otherwise
+  const char *ResponseFile;
+
+  /// The input file list in case we need to emit a file list instead of a
+  /// proper response file
+  llvm::opt::ArgStringList InputFileList;
+
+  /// String storage if we need to create a new argument to specify a response
+  /// file
+  std::string ResponseFileFlag;
+
+  /// When a response file is needed, we try to put most arguments in an
+  /// exclusive file, while others remains as regular command line arguments.
+  /// This functions fills a vector with the regular command line arguments,
+  /// argv, excluding the ones passed in a response file.
+  void buildArgvForResponseFile(llvm::SmallVectorImpl<const char *> &Out) const;
+
+  /// Encodes an array of C strings into a single string separated by whitespace.
+  /// This function will also put in quotes arguments that have whitespaces and
+  /// will escape the regular backslashes (used in Windows paths) and quotes.
+  /// The results are the contents of a response file, written into a raw_ostream.
+  void writeResponseFile(raw_ostream &OS) const;
+
 public:
   Command(const Action &_Source, const Tool &_Creator, const char *_Executable,
           const llvm::opt::ArgStringList &_Arguments);
 
   void Print(llvm::raw_ostream &OS, const char *Terminator, bool Quote,
              bool CrashReport = false) const override;
 
   virtual int Execute(const StringRef **Redirects, std::string *ErrMsg,
                       bool *ExecutionFailed) const;
 
   /// getSource - Return the Action which caused the creation of this job.
   const Action &getSource() const { return Source; }
 
   /// getCreator - Return the Tool which caused the creation of this job.
   const Tool &getCreator() const { return Creator; }
 
+  /// Set to pass arguments via a response file when launching the command
+  void setResponseFile(const char *FileName);
+
+  /// Set an input file list, necessary if we need to use a response file but
+  /// the tool being called only supports input files lists.
+  void setInputFileList(llvm::opt::ArgStringList List) {
+    InputFileList = std::move(List);
+  }
+
   const char *getExecutable() const { return Executable; }
 
   const llvm::opt::ArgStringList &getArguments() const { return Arguments; }
 
   static bool classof(const Job *J) {
     return J->getKind() == CommandClass ||
            J->getKind() == FallbackCommandClass;
   }

cfe/trunk/include/clang/Driver/Tool.h

 //===--- Tool.h - Compilation Tools -----------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_DRIVER_TOOL_H
 #define LLVM_CLANG_DRIVER_TOOL_H
 
 #include "clang/Basic/LLVM.h"
+#include "llvm/Support/Program.h"
 
 namespace llvm {
 namespace opt {
   class ArgList;
 }
 }
 
 namespace clang {
 namespace driver {
 
   class Compilation;
   class InputInfo;
   class Job;
   class JobAction;
   class ToolChain;
 
   typedef SmallVector<InputInfo, 4> InputInfoList;
 
 /// Tool - Information on a specific compilation tool.
 class Tool {
+public:
+  // Documents the level of support for response files in this tool.
+  // Response files are necessary if the command line gets too large,
+  // requiring the arguments to be transfered to a file.
+  enum ResponseFileSupport {
+    // Provides full support for response files, which means we can transfer
+    // all tool input arguments to a file. E.g.: clang, gcc, binutils and MSVC
+    // tools.
+    RF_Full,
+    // Input file names can live in a file, but flags can't. E.g.: ld64 (Mac
+    // OS X linker).
+    RF_FileList,
+    // Does not support response files: all arguments must be passed via
+    // command line.
+    RF_None
+  };
+
+private:
   /// The tool name (for debugging).
   const char *Name;
 
   /// The human readable name for the tool, for use in diagnostics.
   const char *ShortName;
 
   /// The tool chain this tool is a part of.
   const ToolChain &TheToolChain;
 
+  /// The level of support for response files seen in this tool
+  const ResponseFileSupport ResponseSupport;
+
+  /// The encoding to use when writing response files for this tool on Windows
+  const llvm::sys::WindowsEncodingMethod ResponseEncoding;
+
+  /// The flag used to pass a response file via command line to this tool
+  const char *const ResponseFlag;
+
 public:
-  Tool(const char *Name, const char *ShortName,
-       const ToolChain &TC);
+  Tool(const char *Name, const char *ShortName, const ToolChain &TC,
+       ResponseFileSupport ResponseSupport = RF_None,
+       llvm::sys::WindowsEncodingMethod ResponseEncoding = llvm::sys::WEM_UTF8,
+       const char *ResponseFlag = "@");
 
 public:
   virtual ~Tool();
 
   const char *getName() const { return Name; }
 
   const char *getShortName() const { return ShortName; }
 
   const ToolChain &getToolChain() const { return TheToolChain; }
 
   virtual bool hasIntegratedAssembler() const { return false; }
   virtual bool hasIntegratedCPP() const = 0;
   virtual bool isLinkJob() const { return false; }
   virtual bool isDsymutilJob() const { return false; }
+  /// \brief Returns the level of support for response files of this tool,
+  /// whether it accepts arguments to be passed via a file on disk.
+  ResponseFileSupport getResponseFilesSupport() const {
+    return ResponseSupport;
+  }
+  /// \brief Returns which encoding the response file should use. This is only
+  /// relevant on Windows platforms where there are different encodings being
+  /// accepted for different tools. On UNIX, UTF8 is universal.
+  ///
+  /// Windows use cases: - GCC and Binutils on mingw only accept ANSI response
+  /// files encoded with the system current code page.
+  /// - MSVC's CL.exe and LINK.exe accept UTF16 on Windows.
+  /// - Clang accepts both UTF8 and UTF16.
+  ///
+  /// FIXME: When GNU tools learn how to parse UTF16 on Windows, we should
+  /// always use UTF16 for Windows, which is the Windows official encoding for
+  /// international characters.
+  llvm::sys::WindowsEncodingMethod getResponseFileEncoding() const {
+    return ResponseEncoding;
+  }
+  /// \brief Returns which prefix to use when passing the name of a response
+  /// file as a parameter to this tool.
+  const char *getResponseFileFlag() const { return ResponseFlag; }
 
   /// \brief Does this tool have "good" standardized diagnostics, or should the
   /// driver add an additional "command failed" diagnostic on failures.
   virtual bool hasGoodDiagnostics() const { return false; }
 
   /// ConstructJob - Construct jobs to perform the action \p JA,
   /// writing to \p Output and with \p Inputs, and add the jobs to
   /// \p C.

cfe/trunk/lib/Driver/Compilation.cpp

bool Compilation::CleanupFile(const char *File, bool IssueErrors) const {
   // intentionally not overwritten them.
   if (!llvm::sys::fs::can_write(File) || !llvm::sys::fs::is_regular_file(File))
     return true;
 
   if (std::error_code EC = llvm::sys::fs::remove(File)) {
     // Failure is only failure if the file exists and is "regular". We checked
     // for it being regular before, and llvm::sys::fs::remove ignores ENOENT,
     // so we don't need to check again.
-    
     if (IssueErrors)
       getDriver().Diag(clang::diag::err_drv_unable_to_remove_file)
         << EC.message();
     return false;
   }
   return true;
 }
 

cfe/trunk/lib/Driver/Driver.cpp

if (FailingCommands.empty()) {
     if (!C.getArgs().hasArg(options::OPT_save_temps))
       C.CleanupFileList(C.getTempFiles(), true);
 
     Diag(clang::diag::note_drv_command_failed_diag_msg)
       << "Error generating preprocessed source(s).";
   }
 }
 
-int Driver::ExecuteCompilation(const Compilation &C,
-    SmallVectorImpl< std::pair<int, const Command *> > &FailingCommands) const {
+void Driver::setUpResponseFiles(Compilation &C, Job &J) {
+  if (JobList *Jobs = dyn_cast<JobList>(&J)) {
+    for (JobList::iterator I = Jobs->begin(), E = Jobs->end(); I != E; ++I)
+      setUpResponseFiles(C, **I);
+    return;
+  }
+
+  Command *CurCommand = dyn_cast<Command>(&J);
+  if (!CurCommand)
+    return;
+
+  // Since argumentsFitWithinSystemLimits() may underestimate system's capacity
+  // if the tool does not support response files, there is a chance/ that things
+  // will just work without a response file, so we silently just skip it.
+  if (CurCommand->getCreator().getResponseFilesSupport() == Tool::RF_None ||
+      llvm::sys::argumentsFitWithinSystemLimits(CurCommand->getArguments()))
+    return;
+
+  std::string TmpName = GetTemporaryPath("response", "txt");
+  CurCommand->setResponseFile(C.addTempFile(C.getArgs().MakeArgString(
+      TmpName.c_str())));
+}
+
+int Driver::ExecuteCompilation(Compilation &C,
+    SmallVectorImpl< std::pair<int, const Command *> > &FailingCommands) {
   // Just print if -### was present.
   if (C.getArgs().hasArg(options::OPT__HASH_HASH_HASH)) {
     C.getJobs().Print(llvm::errs(), "\n", true);
     return 0;
   }
 
   // If there were errors building the compilation, quit now.
   if (Diags.hasErrorOccurred())
     return 1;
 
+  // Set up response file names for each command, if necessary
+  setUpResponseFiles(C, C.getJobs());
+
   C.ExecuteJob(C.getJobs(), FailingCommands);
 
   // Remove temp files.
   C.CleanupFileList(C.getTempFiles());
 
   // If the command succeeded, we are done.
   if (FailingCommands.empty())
     return 0;

cfe/trunk/lib/Driver/Job.cpp

 //===--- Job.cpp - Command to Execute -------------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Driver/Driver.h"
 #include "clang/Driver/DriverDiagnostic.h"
 #include "clang/Driver/Job.h"
 #include "clang/Driver/Tool.h"
 #include "clang/Driver/ToolChain.h"
+#include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/StringSet.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/Support/Program.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cassert>
 using namespace clang::driver;
 using llvm::raw_ostream;
 using llvm::StringRef;
+using llvm::ArrayRef;
 
 Job::~Job() {}
 
 Command::Command(const Action &_Source, const Tool &_Creator,
                  const char *_Executable,
                  const ArgStringList &_Arguments)
     : Job(CommandClass), Source(_Source), Creator(_Creator),
-      Executable(_Executable), Arguments(_Arguments) {}
+      Executable(_Executable), Arguments(_Arguments),
+      ResponseFile(nullptr) {}
 
 static int skipArgs(const char *Flag) {
   // These flags are all of the form -Flag <Arg> and are treated as two
   // arguments.  Therefore, we need to skip the flag and the next argument.
   bool Res = llvm::StringSwitch<bool>(Flag)
     .Cases("-I", "-MF", "-MT", "-MQ", true)
     .Cases("-o", "-coverage-file", "-dependency-file", true)
     .Cases("-fdebug-compilation-dir", "-idirafter", true)
static void PrintArg(raw_ostream &OS, const char *Arg, bool Quote) {
   while (const char c = *Arg++) {
     if (c == '"' || c == '\\' || c == '$')
       OS << '\\';
     OS << c;
   }
   OS << '"';
 }
 
+void Command::writeResponseFile(raw_ostream &OS) const {
+  // In a file list, we only write the set of inputs to the response file
+  if (Creator.getResponseFilesSupport() == Tool::RF_FileList) {
+    for (const char *Arg : InputFileList) {
+      OS << Arg << '\n';
+    }
+    return;
+  }
+
+  // In regular response files, we send all arguments to the response file
+  for (const char *Arg : Arguments) {
+    OS << '"';
+
+    for (; *Arg != '\0'; Arg++) {
+      if (*Arg == '\"' || *Arg == '\\') {
+        OS << '\\';
+      }
+      OS << *Arg;
+    }
+
+    OS << "\" ";
+  }
+}
+
+void Command::buildArgvForResponseFile(
+    llvm::SmallVectorImpl<const char *> &Out) const {
+  // When not a file list, all arguments are sent to the response file.
+  // This leaves us to set the argv to a single parameter, requesting the tool
+  // to read the response file.
+  if (Creator.getResponseFilesSupport() != Tool::RF_FileList) {
+    Out.push_back(Executable);
+    Out.push_back(ResponseFileFlag.c_str());
+    return;
+  }
+
+  llvm::StringSet<> Inputs;
+  for (const char *InputName : InputFileList)
+    Inputs.insert(InputName);
+  Out.push_back(Executable);
+  // In a file list, build args vector ignoring parameters that will go in the
+  // response file (elements of the InputFileList vector)
+  bool FirstInput = true;
+  for (const char *Arg : Arguments) {
+    if (Inputs.count(Arg) == 0) {
+      Out.push_back(Arg);
+    } else if (FirstInput) {
+      FirstInput = false;
+      Out.push_back(Creator.getResponseFileFlag());
+      Out.push_back(ResponseFile);
+    }
+  }
+}
+
 void Command::Print(raw_ostream &OS, const char *Terminator, bool Quote,
                     bool CrashReport) const {
   // Always quote the exe.
   OS << ' ';
   PrintArg(OS, Executable, /*Quote=*/true);
 
-  for (size_t i = 0, e = Arguments.size(); i < e; ++i) {
-    const char *const Arg = Arguments[i];
+  llvm::ArrayRef<const char *> Args = Arguments;
+  llvm::SmallVector<const char *, 128> ArgsRespFile;
+  if (ResponseFile != nullptr) {
+    buildArgvForResponseFile(ArgsRespFile);
+    Args = ArrayRef<const char *>(ArgsRespFile).slice(1); // no executable name
+  }
+
+  for (size_t i = 0, e = Args.size(); i < e; ++i) {
+    const char *const Arg = Args[i];
 
     if (CrashReport) {
       if (int Skip = skipArgs(Arg)) {
         i += Skip - 1;
         continue;
       }
     }
 
     OS << ' ';
     PrintArg(OS, Arg, Quote);
 
     if (CrashReport && quoteNextArg(Arg) && i + 1 < e) {
       OS << ' ';
-      PrintArg(OS, Arguments[++i], true);
+      PrintArg(OS, Args[++i], true);
     }
   }
+
+  if (ResponseFile != nullptr) {
+    OS << "\n Arguments passed via response file:\n";
+    writeResponseFile(OS);
+    // Avoiding duplicated newline terminator, since FileLists are
+    // newline-separated.
+    if (Creator.getResponseFilesSupport() != Tool::RF_FileList)
+      OS << "\n";
+    OS << " (end of response file)";
+  }
+
   OS << Terminator;
 }
 
+void Command::setResponseFile(const char *FileName) {
+  ResponseFile = FileName;
+  ResponseFileFlag = Creator.getResponseFileFlag();
+  ResponseFileFlag += FileName;
+}
+
 int Command::Execute(const StringRef **Redirects, std::string *ErrMsg,
                      bool *ExecutionFailed) const {
   SmallVector<const char*, 128> Argv;
+
+  if (ResponseFile == nullptr) {
     Argv.push_back(Executable);
     for (size_t i = 0, e = Arguments.size(); i != e; ++i)
       Argv.push_back(Arguments[i]);
     Argv.push_back(nullptr);
 
     return llvm::sys::ExecuteAndWait(Executable, Argv.data(), /*env*/ nullptr,
                                      Redirects, /*secondsToWait*/ 0,
+                                     /*memoryLimit*/ 0, ErrMsg,
+                                     ExecutionFailed);
+  }
+
+  // We need to put arguments in a response file (command is too large)
+  // Open stream to store the response file contents
+  std::string RespContents;
+  llvm::raw_string_ostream SS(RespContents);
+
+  // Write file contents and build the Argv vector
+  writeResponseFile(SS);
+  buildArgvForResponseFile(Argv);
+  Argv.push_back(nullptr);
+  SS.flush();
+
+  // Save the response file in the appropriate encoding
+  if (std::error_code EC = writeFileWithEncoding(
+          ResponseFile, RespContents, Creator.getResponseFileEncoding())) {
+    if (ErrMsg)
+      *ErrMsg = EC.message();
+    if (ExecutionFailed)
+      *ExecutionFailed = true;
+    return -1;
+  }
+
+  return llvm::sys::ExecuteAndWait(Executable, Argv.data(), /*env*/ nullptr,
+                                   Redirects, /*secondsToWait*/ 0,
                                    /*memoryLimit*/ 0, ErrMsg, ExecutionFailed);
 }
 
 FallbackCommand::FallbackCommand(const Action &Source_, const Tool &Creator_,
                                  const char *Executable_,
                                  const ArgStringList &Arguments_,
                                  std::unique_ptr<Command> Fallback_)
     : Command(Source_, Creator_, Executable_, Arguments_),

cfe/trunk/lib/Driver/Tool.cpp

 //===--- Tool.cpp - Compilation Tools -------------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Driver/Tool.h"
 
 using namespace clang::driver;
 
-Tool::Tool(const char *_Name, const char *_ShortName,
-           const ToolChain &TC) : Name(_Name), ShortName(_ShortName),
-                                  TheToolChain(TC)
-{
-}
+Tool::Tool(const char *_Name, const char *_ShortName, const ToolChain &TC,
+           ResponseFileSupport _ResponseSupport,
+           llvm::sys::WindowsEncodingMethod _ResponseEncoding,
+           const char *_ResponseFlag)
+    : Name(_Name), ShortName(_ShortName), TheToolChain(TC),
+      ResponseSupport(_ResponseSupport), ResponseEncoding(_ResponseEncoding),
+      ResponseFlag(_ResponseFlag) {}
 
 Tool::~Tool() {
 }

cfe/trunk/lib/Driver/Tools.h

private:
     void AddClangCLArgs(const llvm::opt::ArgList &Args,
                         llvm::opt::ArgStringList &CmdArgs) const;
 
     visualstudio::Compile *getCLFallback() const;
 
     mutable std::unique_ptr<visualstudio::Compile> CLFallback;
 
   public:
-    Clang(const ToolChain &TC) : Tool("clang", "clang frontend", TC) {}
+    Clang(const ToolChain &TC) : Tool("clang", "clang frontend", TC, RF_Full) {}
 
     bool hasGoodDiagnostics() const override { return true; }
     bool hasIntegratedAssembler() const override { return true; }
     bool hasIntegratedCPP() const override { return true; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 
   /// \brief Clang integrated assembler tool.
   class LLVM_LIBRARY_VISIBILITY ClangAs : public Tool {
   public:
     ClangAs(const ToolChain &TC) : Tool("clang::as",
-                                        "clang integrated assembler", TC) {}
+                                        "clang integrated assembler", TC,
+                                        RF_Full) {}
 
     bool hasGoodDiagnostics() const override { return true; }
     bool hasIntegratedAssembler() const override { return false; }
     bool hasIntegratedCPP() const override { return false; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 
+  /// \brief Base class for all GNU tools that provide the same behavior when
+  /// it comes to response files support
+  class GnuTool : public Tool {
+    virtual void anchor();
+
+  public:
+    GnuTool(const char *Name, const char *ShortName, const ToolChain &TC)
+        : Tool(Name, ShortName, TC, RF_Full, llvm::sys::WEM_CurrentCodePage) {}
+  };
+
   /// gcc - Generic GCC tool implementations.
 namespace gcc {
-  class LLVM_LIBRARY_VISIBILITY Common : public Tool {
+  class LLVM_LIBRARY_VISIBILITY Common : public GnuTool {
   public:
     Common(const char *Name, const char *ShortName,
-           const ToolChain &TC) : Tool(Name, ShortName, TC) {}
+           const ToolChain &TC) : GnuTool(Name, ShortName, TC) {}
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output,
                       const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
 
     /// RenderExtraToolArgs - Render any arguments necessary to force
public:
     void RenderExtraToolArgs(const JobAction &JA,
                              llvm::opt::ArgStringList &CmdArgs) const override;
   };
 } // end namespace gcc
 
 namespace hexagon {
   // For Hexagon, we do not need to instantiate tools for PreProcess, PreCompile and Compile.
   // We simply use "clang -cc1" for those actions.
-  class LLVM_LIBRARY_VISIBILITY Assemble : public Tool {
+  class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool {
   public:
-    Assemble(const ToolChain &TC) : Tool("hexagon::Assemble",
+    Assemble(const ToolChain &TC) : GnuTool("hexagon::Assemble",
       "hexagon-as", TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
 
     void RenderExtraToolArgs(const JobAction &JA,
                              llvm::opt::ArgStringList &CmdArgs) const;
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 
-  class LLVM_LIBRARY_VISIBILITY Link : public Tool {
+  class LLVM_LIBRARY_VISIBILITY Link : public GnuTool {
   public:
-    Link(const ToolChain &TC) : Tool("hexagon::Link",
+    Link(const ToolChain &TC) : GnuTool("hexagon::Link",
       "hexagon-ld", TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
     bool isLinkJob() const override { return true; }
 
     virtual void RenderExtraToolArgs(const JobAction &JA,
                                      llvm::opt::ArgStringList &CmdArgs) const;
     void ConstructJob(Compilation &C, const JobAction &JA,
protected:
     void AddMachOArch(const llvm::opt::ArgList &Args,
                        llvm::opt::ArgStringList &CmdArgs) const;
 
     const toolchains::MachO &getMachOToolChain() const {
       return reinterpret_cast<const toolchains::MachO&>(getToolChain());
     }
 
   public:
-    MachOTool(const char *Name, const char *ShortName,
-               const ToolChain &TC) : Tool(Name, ShortName, TC) {}
+  MachOTool(
+      const char *Name, const char *ShortName, const ToolChain &TC,
+      ResponseFileSupport ResponseSupport = RF_None,
+      llvm::sys::WindowsEncodingMethod ResponseEncoding = llvm::sys::WEM_UTF8,
+      const char *ResponseFlag = "@")
+      : Tool(Name, ShortName, TC, ResponseSupport, ResponseEncoding,
+             ResponseFlag) {}
   };
 
   class LLVM_LIBRARY_VISIBILITY Assemble : public MachOTool  {
   public:
     Assemble(const ToolChain &TC) : MachOTool("darwin::Assemble",
                                               "assembler", TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 
   class LLVM_LIBRARY_VISIBILITY Link : public MachOTool  {
     bool NeedsTempPath(const InputInfoList &Inputs) const;
     void AddLinkArgs(Compilation &C, const llvm::opt::ArgList &Args,
                      llvm::opt::ArgStringList &CmdArgs,
                      const InputInfoList &Inputs) const;
 
   public:
-    Link(const ToolChain &TC) : MachOTool("darwin::Link", "linker", TC) {}
+    Link(const ToolChain &TC) : MachOTool("darwin::Link", "linker", TC,
+                                          RF_FileList, llvm::sys::WEM_UTF8,
+                                          "-filelist") {}
 
     bool hasIntegratedCPP() const override { return false; }
     bool isLinkJob() const override { return true; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
void ConstructJob(Compilation &C, const JobAction &JA,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 
 }
 
   /// openbsd -- Directly call GNU Binutils assembler and linker
 namespace openbsd {
-  class LLVM_LIBRARY_VISIBILITY Assemble : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool  {
   public:
-    Assemble(const ToolChain &TC) : Tool("openbsd::Assemble", "assembler",
+    Assemble(const ToolChain &TC) : GnuTool("openbsd::Assemble", "assembler",
                                          TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output,
                       const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
-  class LLVM_LIBRARY_VISIBILITY Link : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Link : public GnuTool  {
   public:
-    Link(const ToolChain &TC) : Tool("openbsd::Link", "linker", TC) {}
+    Link(const ToolChain &TC) : GnuTool("openbsd::Link", "linker", TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
     bool isLinkJob() const override { return true; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 } // end namespace openbsd
 
   /// bitrig -- Directly call GNU Binutils assembler and linker
 namespace bitrig {
-  class LLVM_LIBRARY_VISIBILITY Assemble : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool  {
   public:
-    Assemble(const ToolChain &TC) : Tool("bitrig::Assemble", "assembler",
+    Assemble(const ToolChain &TC) : GnuTool("bitrig::Assemble", "assembler",
                                          TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
-  class LLVM_LIBRARY_VISIBILITY Link : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Link : public GnuTool  {
   public:
-    Link(const ToolChain &TC) : Tool("bitrig::Link", "linker", TC) {}
+    Link(const ToolChain &TC) : GnuTool("bitrig::Link", "linker", TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
     bool isLinkJob() const override { return true; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 } // end namespace bitrig
 
   /// freebsd -- Directly call GNU Binutils assembler and linker
 namespace freebsd {
-  class LLVM_LIBRARY_VISIBILITY Assemble : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool  {
   public:
-    Assemble(const ToolChain &TC) : Tool("freebsd::Assemble", "assembler",
+    Assemble(const ToolChain &TC) : GnuTool("freebsd::Assemble", "assembler",
                                          TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
-  class LLVM_LIBRARY_VISIBILITY Link : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Link : public GnuTool  {
   public:
-    Link(const ToolChain &TC) : Tool("freebsd::Link", "linker", TC) {}
+    Link(const ToolChain &TC) : GnuTool("freebsd::Link", "linker", TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
     bool isLinkJob() const override { return true; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 } // end namespace freebsd
 
   /// netbsd -- Directly call GNU Binutils assembler and linker
 namespace netbsd {
-  class LLVM_LIBRARY_VISIBILITY Assemble : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool  {
 
   public:
     Assemble(const ToolChain &TC)
-      : Tool("netbsd::Assemble", "assembler", TC) {}
+      : GnuTool("netbsd::Assemble", "assembler", TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
-  class LLVM_LIBRARY_VISIBILITY Link : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Link : public GnuTool  {
 
   public:
     Link(const ToolChain &TC)
-      : Tool("netbsd::Link", "linker", TC) {}
+      : GnuTool("netbsd::Link", "linker", TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
     bool isLinkJob() const override { return true; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 } // end namespace netbsd
 
   /// Directly call GNU Binutils' assembler and linker.
 namespace gnutools {
-  class LLVM_LIBRARY_VISIBILITY Assemble : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool  {
   public:
-    Assemble(const ToolChain &TC) : Tool("GNU::Assemble", "assembler", TC) {}
+    Assemble(const ToolChain &TC) : GnuTool("GNU::Assemble", "assembler", TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output,
                       const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
-  class LLVM_LIBRARY_VISIBILITY Link : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Link : public GnuTool  {
   public:
-    Link(const ToolChain &TC) : Tool("GNU::Link", "linker", TC) {}
+    Link(const ToolChain &TC) : GnuTool("GNU::Link", "linker", TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
     bool isLinkJob() const override { return true; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output,
                       const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 }
   /// minix -- Directly call GNU Binutils assembler and linker
 namespace minix {
-  class LLVM_LIBRARY_VISIBILITY Assemble : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool  {
   public:
-    Assemble(const ToolChain &TC) : Tool("minix::Assemble", "assembler",
+    Assemble(const ToolChain &TC) : GnuTool("minix::Assemble", "assembler",
                                          TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output,
                       const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
-  class LLVM_LIBRARY_VISIBILITY Link : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Link : public GnuTool  {
   public:
-    Link(const ToolChain &TC) : Tool("minix::Link", "linker", TC) {}
+    Link(const ToolChain &TC) : GnuTool("minix::Link", "linker", TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
     bool isLinkJob() const override { return true; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output,
                       const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 } // end namespace solaris
 
   /// dragonfly -- Directly call GNU Binutils assembler and linker
 namespace dragonfly {
-  class LLVM_LIBRARY_VISIBILITY Assemble : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool  {
   public:
-    Assemble(const ToolChain &TC) : Tool("dragonfly::Assemble", "assembler",
+    Assemble(const ToolChain &TC) : GnuTool("dragonfly::Assemble", "assembler",
                                          TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
-  class LLVM_LIBRARY_VISIBILITY Link : public Tool  {
+  class LLVM_LIBRARY_VISIBILITY Link : public GnuTool  {
   public:
-    Link(const ToolChain &TC) : Tool("dragonfly::Link", "linker", TC) {}
+    Link(const ToolChain &TC) : GnuTool("dragonfly::Link", "linker", TC) {}
 
     bool hasIntegratedCPP() const override { return false; }
     bool isLinkJob() const override { return true; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output,
                       const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 } // end namespace dragonfly
 
 /// Visual studio tools.
 namespace visualstudio {
   class LLVM_LIBRARY_VISIBILITY Link : public Tool {
   public:
-    Link(const ToolChain &TC) : Tool("visualstudio::Link", "linker", TC) {}
+    Link(const ToolChain &TC) : Tool("visualstudio::Link", "linker", TC,
+                                     RF_Full, llvm::sys::WEM_UTF16) {}
 
     bool hasIntegratedCPP() const override { return false; }
     bool isLinkJob() const override { return true; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,
                       const char *LinkingOutput) const override;
   };
 
   class LLVM_LIBRARY_VISIBILITY Compile : public Tool {
   public:
-    Compile(const ToolChain &TC) : Tool("visualstudio::Compile", "compiler", TC) {}
+    Compile(const ToolChain &TC) : Tool("visualstudio::Compile", "compiler", TC,
+                                        RF_Full, llvm::sys::WEM_UTF16) {}
 
     bool hasIntegratedAssembler() const override { return true; }
     bool hasIntegratedCPP() const override { return true; }
     bool isLinkJob() const override { return false; }
 
     void ConstructJob(Compilation &C, const JobAction &JA,
                       const InputInfo &Output, const InputInfoList &Inputs,
                       const llvm::opt::ArgList &TCArgs,

cfe/trunk/lib/Driver/Tools.cpp

if (HWDiv == "arm") {
     Features.push_back("+hwdiv-arm");
     Features.push_back("+hwdiv");
   } else if (HWDiv == "none") {
     Features.push_back("-hwdiv-arm");
     Features.push_back("-hwdiv");
   } else
     D.Diag(diag::err_drv_clang_unsupported) << A->getAsString(Args);
 }
- 
 // Handle -mfpu=.
 //
 // FIXME: Centralize feature selection, defaulting shouldn't be also in the
 // frontend target.
 static void getARMFPUFeatures(const Driver &D, const Arg *A,
                               const ArgList &Args,
                               std::vector<const char *> &Features) {
   StringRef FPU = A->getValue();
void ClangAs::ConstructJob(Compilation &C, const JobAction &JA,
   // creating an object.
   // TODO: Currently only works on linux with newer objcopy.
   if (Args.hasArg(options::OPT_gsplit_dwarf) &&
       getToolChain().getTriple().isOSLinux())
     SplitDebugInfo(getToolChain(), C, *this, JA, Args, Output,
                    SplitDebugName(Args, Inputs));
 }
 
+void GnuTool::anchor() {}
+
 void gcc::Common::ConstructJob(Compilation &C, const JobAction &JA,
                                const InputInfo &Output,
                                const InputInfoList &Inputs,
                                const ArgList &Args,
                                const char *LinkingOutput) const {
   const Driver &D = getToolChain().getDriver();
   ArgStringList CmdArgs;
 

 
 void darwin::Link::ConstructJob(Compilation &C, const JobAction &JA,
                                 const InputInfo &Output,
                                 const InputInfoList &Inputs,
                                 const ArgList &Args,
                                 const char *LinkingOutput) const {
   assert(Output.getType() == types::TY_Image && "Invalid linker output type.");
 
+  // If the number of arguments surpasses the system limits, we will encode the
+  // input files in a separate file, shortening the command line. To this end,
+  // build a list of input file names that can be passed via a file with the
+  // -filelist linker option.
+  llvm::opt::ArgStringList InputFileList;
+
   // The logic here is derived from gcc's behavior; most of which
   // comes from specs (starting with link_command). Consult gcc for
   // more information.
   ArgStringList CmdArgs;
 
   /// Hack(tm) to ignore linking errors when we are doing ARC migration.
   if (Args.hasArg(options::OPT_ccc_arcmt_check,
                   options::OPT_ccc_arcmt_migrate)) {
void darwin::Link::ConstructJob(Compilation &C, const JobAction &JA,
   case LibIOMP5:
     CmdArgs.push_back("-liomp5");
     break;
   case LibUnknown:
     break;
   }
 
   AddLinkerInputs(getToolChain(), Inputs, Args, CmdArgs);
-  
+  // Build the input file for -filelist (list of linker input files) in case we
+  // need it later
+  for (const auto &II : Inputs) {
+    if (!II.isFilename()) {
+      // This is a linker input argument.
+      // We cannot mix input arguments and file names in a -filelist input, thus
+      // we prematurely stop our list (remaining files shall be passed as
+      // arguments).
+      if (InputFileList.size() > 0)
+        break;
+
+      continue;
+    }
+
+    InputFileList.push_back(II.getFilename());
+  }
   if (isObjCRuntimeLinked(Args) &&
       !Args.hasArg(options::OPT_nostdlib) &&
       !Args.hasArg(options::OPT_nodefaultlibs)) {
     // We use arclite library for both ARC and subscripting support.
     getMachOToolChain().AddLinkARCArgs(Args, CmdArgs);
 
     CmdArgs.push_back("-framework");
     CmdArgs.push_back("Foundation");
if (!Args.hasArg(options::OPT_nostdlib) &&
     // endfile_spec is empty.
   }
 
   Args.AddAllArgs(CmdArgs, options::OPT_T_Group);
   Args.AddAllArgs(CmdArgs, options::OPT_F);
 
   const char *Exec =
     Args.MakeArgString(getToolChain().GetLinkerPath());
-  C.addCommand(llvm::make_unique<Command>(JA, *this, Exec, CmdArgs));
+  std::unique_ptr<Command> Cmd =
+    llvm::make_unique<Command>(JA, *this, Exec, CmdArgs);
+  Cmd->setInputFileList(std::move(InputFileList));
+  C.addCommand(std::move(Cmd));
 }
 
 void darwin::Lipo::ConstructJob(Compilation &C, const JobAction &JA,
                                 const InputInfo &Output,
                                 const InputInfoList &Inputs,
                                 const ArgList &Args,
                                 const char *LinkingOutput) const {
   ArgStringList CmdArgs;

cfe/trunk/test/Driver/Inputs/gen-response.c

+#define M -DTEST
+#define M1 M M M M M M M M M M
+#define M2 M1 M1 M1 M1 M1 M1 M1 M1 M1 M1
+#define M3 M2 M2 M2 M2 M2 M2 M2 M2 M2 M2
+#define M4 M3 M3 M3 M3 M3 M3 M3 M3 M3 M3
+#define M5 M4 M4 M4 M4 M4 M4 M4 M4 M4 M4
+#define TEXT M5 M5 M5
+TEXT

cfe/trunk/test/Driver/response-file.c

+// REQUIRES: long_tests
+
+// Check that clang is able to process short response files
+// Since this is a short response file, clang must not use a response file
+// to pass its parameters to other tools. This is only necessary for a large
+// number of parameters.
+// RUN: echo "-DTEST" >> %t.0.txt
+// RUN: %clang -E @%t.0.txt %s -v 2>&1 | FileCheck %s -check-prefix=SHORT
+// SHORT-NOT: Arguments passed via response file
+// SHORT: extern int it_works;
+
+// Check that clang is able to process long response files, routing a long
+// sequence of arguments to other tools by using response files as well.
+// We generate a 2MB response file to be big enough to surpass any system
+// limit.
+// RUN: %clang -E %S/Inputs/gen-response.c | grep DTEST > %t.1.txt
+// RUN: %clang -E @%t.1.txt %s -v 2>&1 | FileCheck %s -check-prefix=LONG
+// LONG: Arguments passed via response file
+// LONG: extern int it_works;
+
+#ifdef TEST
+extern int it_works;
+#endif