RFC: [clang] Multithreaded compilation support -- NOT FOR SUBMIT

[aganea]

Summary

This is very preliminary change which adds support for (clang-cl) /MP (Build with multiple processes). Doc for /MP is here.

Support for -j to the clang driver could also be added along the way.

I would simply like a general advice on this change. I don't plan to commit this as it is. If the general idea seems good, I'll cut it down in several smaller pieces.

The change about having explicit return codes (Program.h / enum ProgramReturnCode) should probably be discussed separately, although if you have an opinion about that, please do.

Some timings

Full rebuild of LLVM + Clang + LLD (at r341847), Release, optimized tablegen. VS2017 15.8.3, Ninja 1.8.2, CMake 3.12.2

Config 1 : Intel Xeon Haswell 6 cores / 12 HW threads, 3.5 GHz, 15M cache, 128 GB RAM, SSD 550 MB/s

MSBuild, MSVC /MP	(56min 43 sec)	2 parallel msbuild
MSBuild, Clang + LLD	(2hour 40min)	2 parallel msbuild
MSBuild, Clang /MP + LLD	(51min 8sec)	2 parallel msbuild
Ninja, MSVC	(37min)
Ninja, Clang [1]	(31min 52sec)

Config 2 : Intel Xeon Skylake 18 cores / 36 HW threads, x2 (Dual CPU), 72 HW threads total, 2.3 GHz, 24.75M cache, 128 GB RAM, NVMe 4.6 GB/s

MSBuild, MSVC /MP	(12min 8sec)	32 parallel msbuild
MSBuild, Clang + LLD	(29min 12sec)	32 parallel msbuild
MSBuild, Clang /MP + LLD	(9min 22sec)	32 parallel msbuild
Ninja, MSVC	(7min 35sec)
Ninja, Clang [1]	(11min)

[1] Clang compiled with Clang.

Some remarks:

• Ninja is better in regards to discovering cmake "features" (Looking for ... - found); whereas the Visual Studio generator goes though a MSBuild instance for each "feature" which makes it very slow.
• Ninja seems to have a better job allocation that MSBuild, which compiles monolithically each vcproj one by one.
• Ninja + Clang on the Skylake is significantly worse than Ninja + MSVC, probably because of the poor Windows 10 thread scheduler which is known to under-perform on high cores count, see here. The reason is that for each file compiled, clang-cl.exe invokes an additional -cc1 child process.
• Each new clang-cl.exe child process creation costs between 60-100ms (that is the time spent between the parent's CreateProcess() and the first file accessed by the child .exe). In contrast MSBuild + Clang /MP uses only one parent .exe per vcproject.

[zturner]

What about the timings of clang-cl without /MP?

[hans]

In D52193#1237468, @zturner wrote:

What about the timings of clang-cl without /MP?

And one using Ninja rather than msbuild.

I think the real question is whether we want clang and clang-cl to do this. I'm not sure we do as it adds complexity for solving a problem that's better solved at the build system level.

[zturner]

In an ideal world yes, but the reality is that many people still use
MSBuild, and in that world /MP presumably helps quite a bit. And given that
many people already depend on this functionality of cl, it’s a potential
showstopper for migrating if we don’t support it. That said, if the benefit
isn’t significant that’s a stronger argument for not supporting it imo

[aganea]

The goal of this change is frictionless compilation into VS2017 when using clang-cl as a compiler. We've realized that compiling Clang+LLD with Clang generates a faster executable that with MSVC (even latest one).
I currently can't see a good way of generating the Visual Studio solution with CMake, while using Ninja+clang-cl for compilation. They are two orthogonal configurations. Any suggestions?

[zturner]

In D52193#1238923, @aganea wrote:

The goal of this change is frictionless compilation into VS2017 when using clang-cl as a compiler. We've realized that compiling Clang+LLD with Clang generates a faster executable that with MSVC (even latest one).
I currently can't see a good way of generating the Visual Studio solution with CMake, while using Ninja+clang-cl for compilation. They are two orthogonal configurations. Any suggestions?

I don't think this is necessary. I think your updated Matrix is pretty good.

I'm surprised to see that Ninja + Clang is slower than Ninja + MSVC. Are you using lld here?

[zturner]

That said, the numbers are pretty convincing

These two rows alone:
MSBuild, Clang + LLD (29min 12sec) 32 parallel msbuild
MSBuild, Clang /MP + LLD (9min 22sec) 32 parallel msbuild

Are enough to make this patch worthy of serious consideration. (I haven't looked at the content / complexity yet, was waiting on the numbers first).

[aganea]

In D52193#1238944, @zturner wrote:

In D52193#1238923, @aganea wrote:

The goal of this change is frictionless compilation into VS2017 when using clang-cl as a compiler. We've realized that compiling Clang+LLD with Clang generates a faster executable that with MSVC (even latest one).
I currently can't see a good way of generating the Visual Studio solution with CMake, while using Ninja+clang-cl for compilation. They are two orthogonal configurations. Any suggestions?

I don't think this is necessary. I think your updated Matrix is pretty good.

I'm surprised to see that Ninja + Clang is slower than Ninja + MSVC. Are you using lld here?

Yes, all the ‘Clang’ rows use both clang-cl and lld-link.
Like stated above, I think this is caused by a lot more processes (clang-cl.exe) being invoked. In contrast, cl.exe does not invoke a child process. There are about 3200 files to compiles, which makes 6400 calls to clang-cl. At about ~60ms lead time per process, that adds up to an extra ~3min wall clock time. It is the simplest explanation I could find.

[zturner]

In D52193#1238978, @aganea wrote:

In D52193#1238944, @zturner wrote:

In D52193#1238923, @aganea wrote:

The goal of this change is frictionless compilation into VS2017 when using clang-cl as a compiler. We've realized that compiling Clang+LLD with Clang generates a faster executable that with MSVC (even latest one).
I currently can't see a good way of generating the Visual Studio solution with CMake, while using Ninja+clang-cl for compilation. They are two orthogonal configurations. Any suggestions?

I don't think this is necessary. I think your updated Matrix is pretty good.

I'm surprised to see that Ninja + Clang is slower than Ninja + MSVC. Are you using lld here?

Yes, all the ‘Clang’ rows use both clang-cl and lld-link.
Like stated above, I think this is caused by a lot more processes (clang-cl.exe) being invoked. In contrast, cl.exe does not invoke a child process. There are about 3200 files to compiles, which makes 6400 calls to clang-cl. At about ~60ms lead time per process, that adds up to an extra ~3min wall clock time. It is the simplest explanation I could find.

Would you mind syncing to r342002 and trying again? I don't doubt your results, but I'm interested to see how much of an improvement this patch provides.

commit 840717872a0e0f03b19040ef143bf45aa1a7f0a0
Author: Reid Kleckner <[e-mail]>
Date:   Tue Sep 11 22:25:13 2018 +0000

    [cmake] Speed up check-llvm 5x by delay loading shell32 and ole32

    Summary:
    Previously, check-llvm on my Windows 10 workstation took about 300s to
    run, and it would lock up my mouse. Now, it takes just under 60s.
    Previously running the tests only used about 15% of the available CPU
    time, now it uses up to 60%.

    Shell32.dll and ole32.dll have direct dependencies on user32.dll and
    gdi32.dll. These DLLs are mostly used to for Windows GUI functionality,
    which most LLVM tools don't need. It seems that these DLLs acquire and
    release some system resources on startup and exit, and that requires
    acquiring the same highly contended lock discussed in this post:
    https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and-i-cant-move-my-mouse/

    Most LLVM tools still have a transitive dependency on
    SHGetKnownFolderPathW, which is used to look up the user home directory
    and local AppData directory. We also use SHFileOperationW to recursively
    delete a directory, but only LLDB and clang-doc use this today. At some
    point, we might consider removing these last shell32.dll deps, but for
    now, I would like to take this free performance.

    Many thanks to Bruce Dawson for suggesting this fix. He looked at an ETW
    trace of an LLVM test run, noticed these DLLs, and suggested avoiding
    them.

    Reviewers: zturner, pcc, thakis

    Subscribers: mgorny, llvm-commits, hiraditya

    Differential Revision: https://reviews.llvm.org/D51952

Notes:
    git-svn-rev: 342002

[rnk]

This is pretty cool. The process launching APIs in LLVM were pretty basic, left a lot to be desired, returned ints, etc etc. This addresses a lot of that.

clang/trunk/lib/Driver/Driver.cpp

3030

Seems nice to save a syscall and not ask how many cores we have if we were given an explicit value first.

llvm/trunk/lib/Support/Windows/Program.inc

424

I guess no Posix implementation? It's kind of hard to know if we made the right abstractions without doing it for Windows and *nix.

[hans]

Thanks for adding the Ninja numbers. It confirms that Ninja is significantly faster than MSBuild + /MP.

Since that's the case, maybe we should be poking MS about making MSBuild faster instead of adding /MP support to Clang? Or making it easier to use Ninja in VS projects? Your patch says RFC after all :-)

[aganea]

In D52193#1239171, @hans wrote:

Thanks for adding the Ninja numbers. It confirms that Ninja is significantly faster than MSBuild + /MP.

It is true for the 6-cores Xeon, but not for the dual-18-cores. I think there are two issues there, somehow unrelated to /MP: 1. Invoking the child clang-cl -cc1 for each cpp file makes things much slower. I’ve clocked the invocation at about 60-100ms (which is caused mainly by loading dlls & registery accesses). Most likely Reid’s change about delay loading dlls should improve that. 2. The other issue is, I think, the process context switching at the OS-level. This link: https://www.phoronix.com/scan.php?page=article&item=2990wx-linux-windows&num=4 - shows that multi-threading is significantly faster on a Linux machine, as far as high cores count goes (when compared with the same test ran on the same machine under Windows 10).

Since that's the case, maybe we should be poking MS about making MSBuild faster instead of adding /MP support to Clang? Or making it easier to use Ninja in VS projects? Your patch says RFC after all :-)

Sad for Microsoft, but at this point it is a _design_ change MSBuild needs. And as a PM I would rather invest in bindings to better build systems (Ninja, Fastbuild). However I expect there are still users of MSBuild out there, and without /MP this means essentially that migrating to clang-cl requires also changing their build system and their VS2017 integration with the said build system.

[thakis]

Since it helps existing msbuild configs, adding this seems like a good thing to me.

clang-cl isn't supposed to do (explicit) registry accesses when you hold it right (pass in -fms-compatibility-version etc). Have you seen registry access costs, or is that speculation?

Re fastbuild: We use ninja with https://chromium.googlesource.com/infra/goma/client/ as distributed build system, which gives use the same build system on all platforms, -j1000 builds, and we can build 40K translation units in a few minutes. We're pretty happy with our setup :-)

[aganea]

clang-cl isn't supposed to do (explicit) registry accesses when you hold it right (pass in -fms-compatibility-version etc). Have you seen registry access costs, or is that speculation?

No speculation, I will share the FileMon logs shortly. It is indirectly dependent DLLs that access the registry, not clang-cl itself.

[aganea]

It seems Reid's change has done good: cmake is not as significant as before in the build process. See below the improved timings:

The tests consist in a full cleanup (delete build folder), cmake to regenerate, then a full rebuild of LLVM + Clang + LLD (at r342552), Release target, optimized tablegen.
VS2017 15.8.3, Ninja 1.8.2, CMake 3.12.2
For the clang-cl tests, I'm not using any official LLVM release, only the binaries I built myself. lld-link is used in that case.

(built with MSVC) means the LLVM toolchain used to perfom this test was compiled in a previous run with MSVC cl 15.8.3
(built with Clang) means the LLVM toolchain used to perform this test was compiled in a previous run with Clang at r342552

I took the best figures from several runs (ran in a random order).

---

Config 1 : Intel Xeon Haswell 6 cores / 12 HW threads, 3.5 GHz, 15M cache, 128 GB RAM, SSD 550 MB/s

MSBuild :

MSVC cl /MP	(50min 26sec)	2 parallel msbuild
MSVC cl /MP	(40min 23sec)	8 parallel msbuild
MSVC cl /MP	(40min 5sec)	16 parallel msbuild
clang-cl (built with MSVC)	(43min 36sec)	16 parallel msbuild
clang-cl (built with Clang)	(43min 42sec)	16 parallel msbuild
clang-cl /MP (built with MSVC)	not tested
clang-cl /MP (built with Clang)	(36min 13sec)	8 parallel msbuild
clang-cl /MP (built with Clang)	(34min 57sec)	16 parallel msbuild

Ninja:

MSVC cl	(33min 29sec)
clang-cl (built with MSVC)	(30min 2sec)
clang-cl (built with Clang)	(28min 29sec)

---

Config 2 : Intel Xeon Skylake 18 cores / 36 HW threads, x2 (Dual CPU), 72 HW threads total, 2.3 GHz, 24.75M cache, 128 GB RAM, NVMe 4.6 GB/s

MSBuild :

MSVC cl /MP	(10min 3sec)	32 parallel msbuild
clang-cl (built with MSVC)	(24min 15sec)	32 parallel msbuild
clang-cl (built with Clang)	(21min 21sec)	32 parallel msbuild
clang-cl /MP (built with MSVC)	(7min 52sec)	32 parallel msbuild
clang-cl /MP (built with Clang)	(7min 30sec)	32 parallel msbuild

Ninja:

MSVC cl	(7min 25sec)
clang-cl (built with MSVC)	(8min 23sec)
clang-cl (built with Clang)	(8min)

llvm/trunk/lib/Support/Windows/Program.inc

424

Yes, I currenly don't have an Unix box at hand. But I will implement it shortly as it needs to be atomically commited with the Windows part.

[zturner]

The process stuff looks cool and useful independently of /MP. Would it be possible to break that into a separate patch, and add a unit test for the behavior of the WaitAll flag?

[aganea]

@thakis > clang-cl isn't supposed to do (explicit) registry accesses when you hold it right (pass in -fms-compatibility-version etc). Have you seen registry access costs, or is that speculation?

Please see this log:

clang-cl-log.zip5 KBDownload

- the child clang-cl -cc1 takes about ~117ms until it gets into the global initializers. This is on my Haswell PC. On the Skylake, this takes "only" ~60ms.
This probably explains why Ninja is slower on the Skylake when using clang-cl as a compiler. There should be a shorter codepath maybe when only a single .cpp is being compiled, and avoid running the child process.

[aganea]

In D52193#1241056, @zturner wrote:

The process stuff looks cool and useful independently of /MP. Would it be possible to break that into a separate patch, and add a unit test for the behavior of the WaitAll flag?

Of course, will do.

[thakis]

In D52193#1241067, @aganea wrote:

@thakis > clang-cl isn't supposed to do (explicit) registry accesses when you hold it right (pass in -fms-compatibility-version etc). Have you seen registry access costs, or is that speculation?

Please see this log:

clang-cl-log.zip5 KBDownload

- the child clang-cl -cc1 takes about ~117ms until it gets into the global initializers. This is on my Haswell PC. On the Skylake, this takes "only" ~60ms.
This probably explains why Ninja is slower on the Skylake when using clang-cl as a compiler. There should be a shorter codepath maybe when only a single .cpp is being compiled, and avoid running the child process.

Huh, interesting! I had a local hack years ago where I had measured how much not spawning a subprocess for cc1 saves (it looked like https://reviews.llvm.org/D52411) and over here it didn't do anything. Can you check if patching that in helps you a lot? If so, we should reconsider doing something like that.

[thakis]

...and to reword this a bit: Clang taking a long time to start up in some configurations is a bug we should profile and fix :-)

[aganea]

In D52193#1243456, @thakis wrote:

...and to reword this a bit: Clang taking a long time to start up in some configurations is a bug we should profile and fix :-)

This is time spent in ntdll.dll loading various low-level libraries like kernel32.dll, KernelBase.dll, combase.dll and so on. So at least 20ms just getting to main(). Then, about ~10ms spent loading dbghelp.dll while installing the exception handler in llvm::RegisterHandler. So a lot of wasted time. The easiest would be to just not invoke a new child process!

---

New timings without the child clang-cl.exe being invoked (hacked from D52411). The improvement is significant. Tested at r343846.

Config 1: Intel Xeon Haswell 6 cores / 12 HW threads, 3.5 GHz, 15M cache, 128 GB RAM, SSD 550 MB/s

Ninja:

MSVC cl	(37min 19sec)
clang-cl (built with Clang)	(in progress-will update a bit later)
clang-cl no -cc1 process (built with Clang)	(in progress-will update a bit later)

Config 2: Intel Xeon Skylake 18 cores / 36 HW threads, x2 (Dual CPU), 72 HW threads total, 2.3 GHz, 24.75M cache, 128 GB RAM, NVMe SSD 4.6 GB/s

Ninja:

MSVC cl	(7min 33sec)
clang-cl (built with Clang)	(9min 2sec)
clang-cl no -cc1 process (built with Clang)	(7min 9sec)

This asks whether the improvement will be of the same order, if invoking just one clang-cl.exe for the whole compilation process. A sort of local compilation-as-a-service.
In that scenario, Ninja could invoke clang-cl.exe and pass it all the files to be compiled, and let clang iterate and multi-thread internally. That could be quite a significant change, however the improvements could be important.
However that would probably break the concept behind Goma I suppose. Unless you split the invocations to precisely one clang-cl per available (remote) agent.

[aganea]

Updated tests again. This time I've closed all applications, all unused services, and tried to have a "clean" machine.

New timings without the child clang-cl.exe being invoked (hacked from D52411). The test consists in building Clang+LLVM+LLD at r343846 using the compiler specified on each line of the table.

Config 1: Intel Xeon Haswell 6 cores / 12 HW threads, 3.5 GHz, 15 MB cache, 128 GB RAM, SSD 550 MB/s (Windows 10 Fall Creators update 1709)

Ninja:

MSVC 15.8.6 (cl + link)	29 min 4 sec
clang-cl + lld-link 7.0 official release	25 min 45 sec
clang-cl + lld-link 8.0 r343846 built with clang-cl 7.0 official (+D52411, no -cc1 process)	23 min 27 sec

Config 2: Intel Xeon Skylake 18 cores / 36 HW threads, x2 (Dual CPU), 72 HW threads total, 2.3 GHz, 24.75 MB cache each, 128 GB RAM, NVMe SSD 4.6 GB/s (Windows 10 Fall Creators update 1709)

Ninja:

MSVC 15.8.6 (cl + link)	6 min 40 sec
clang-cl + lld-link 7.0 official release	6 min 24 sec
clang-cl + lld-link 8.0 r343846 built with clang-cl 7.0 official (+D52411, no -cc1 process)	5 min 10 sec

I'm wondering if the improvement is of the same order on Linux. I'd be interested to see your build times, out of curiosity? Can any of you try it with and without D52411?

[zturner]

I can try to get some timings from my machine. How do we handle crash
recovery in the case where we don't spawn a child process? I thought the
whole reason for spawning the cc1 driver as a separate process was so that
we could collect and report crash information in a nice way. Not having
that seems like a heavy price to pay.

[aganea]

In D52193#1260862, @zturner wrote:

I can try to get some timings from my machine. How do we handle crash
recovery in the case where we don't spawn a child process? I thought the
whole reason for spawning the cc1 driver as a separate process was so that
we could collect and report crash information in a nice way. Not having
that seems like a heavy price to pay.

clang.exe has already a general exception handler (installed by InitLLVM::InitLLVM/sys::PrintStackTraceOnErrorSignal), which prints the callstack in case of a crash. So you wouldn't need the child process to do post-crash processing.

However, when running within a parent clang.exe, the parents calls Driver::generateCompilationDiagnostics() after a crash to create the repro (preprocessed source).

In a scenario where a single clang.exe is running (no child) we could modify Driver::generateCompilationDiagnostics() so it gets called from within the exception handler, and invoke a child process to collect the preprocessed source.

I wrote a fully fledged crash reporting system which does that, so I know that's possible. The tricky thing is to ensure Driver::generateCompilationDiagnostics() doesn't touch potentially invalid structures (if a crash occured) or allocate memory. The way around that is to pre-allocate and pre-copy the structures that you will be accessing in the case of a crash. But I don't see it as a hurdle in this case.

[rnk]

In D52193#1260933, @aganea wrote:

I wrote a fully fledged crash reporting system which does that, so I know that's possible. The tricky thing is to ensure Driver::generateCompilationDiagnostics() doesn't touch potentially invalid structures (if a crash occured) or allocate memory. The way around that is to pre-allocate and pre-copy the structures that you will be accessing in the case of a crash. But I don't see it as a hurdle in this case.

Our existing stack dumper breaks all kinds of rules in this area. It uses std::vectors to make the stack trace. As much as it pains me that it's not 110% correct, nobody has complained so far that they don't get stack traces from clang when it crashes asynchronously inside of malloc.

Nice, the numbers for avoiding -cc1 are pretty compelling. It would also make debugging easier. I'd love to never see that question again on cfe-dev.

To handle pre-processed source generation, the best idea I could come up with was to pre-allocate the "generate crash diagnostics" command line and then spawn off a child process in the signal/exception handler to the heavy lifting. We're not concerned about generating pre-processed source in cases of system low memory or instability, so the relative unreliability of creating a child process isn't a huge concern.

[angushewlett]

Having applied this patch to the clang latest (which took a bit of doing, and it's possible I messed up along the way) - all working in general, and a roughly 6x speed-up on my i7-7820X. However, PCH support does not seem to work correctly.

Specifically, it looks like when a file is set to build with /Yc (Create PCH), this is not executed serially with respect to the other jobs. Therefore it attempts to compile targets with /Yu (Use PCH) before the PCH has been generated, and fails.

When one or more files are passed with /Yc, PCH generation needs to execute to completion before the other jobs are started.

If I compile without using PCHs, it all works correctly (but clearly there's some performance hit).

Can anyone confirm if they've got /Yc & /Yu working correctly on their system?

Thanks,

Angus.

[aganea]

That is strange. As you can’t mix /Yc and /Yu on the command line, MSBuild should issue two different calls to clang-cl, one with /Yc and one /Yu /MP. Can you check with ProcMon that commands are issued in the right order?

[yurybura]

Dear @aganea,
Do you have any plans to make this PR compatible with trunk? Now MSVC with /MP builds much faster than clang-cl (at least 2x faster for our project)...
Thanks!

[angushewlett]

In D52193#1457612, @aganea wrote:

That is strange. As you can’t mix /Yc and /Yu on the command line, MSBuild should issue two different calls to clang-cl, one with /Yc and one /Yu /MP. Can you check with ProcMon that commands are issued in the right order?

They're issued in the right order, but the second doesn't wait for the first to complete.

[aganea]

In D52193#1589096, @yurybura wrote:

Do you have any plans to make this PR compatible with trunk? Now MSVC with /MP builds much faster than clang-cl (at least 2x faster for our project)...

I'll get back to this after the vacation (somewhere in August)

In D52193#1589150, @angushewlett wrote:

They're issued in the right order, but the second doesn't wait for the first to complete.

Ok, I thought the wait was handled by MSBuild. If I understand correctly what you're saying, MSBuild issues both commands at the same time, but the second cl.exe /Yu /MP somehow waits for the first cl.exe /Yc to complete? (maybe by checking if the PCH is still open for write?)

[angushewlett]

In D52193#1589657, @aganea wrote:

In D52193#1589150, @angushewlett wrote:

They're issued in the right order, but the second doesn't wait for the first to complete.

Ok, I thought the wait was handled by MSBuild. If I understand correctly what you're saying, MSBuild issues both commands at the same time, but the second cl.exe /Yu /MP somehow waits for the first cl.exe /Yc to complete? (maybe by checking if the PCH is still open for write?)

That appears to be the case, yes.

[tycho]

I rebased this myself on the release_90 branch and I was pleasantly surprised that I got the merge right on the first try (?!), and it actually works well without any significant changes (other than merge conflict resolutions).

I've run into two problems with it though:

• Running clang-cl.exe /MP without a number following the argument doesn't seem to automatically pick up the number of available available CPUs. I haven't debugged that yet, but it's probably a trivial fix.

• Precompiled headers aren't being built properly when using clang-cl.exe /MP under Visual Studio. If I do a clean build with e.g. /MP16, it errors immediately with this:

1>project_pch.cpp
1>CL : error : unable to read PCH file .\Release\.\/project_pch.pch: 'no such file or directory'
1>CL : fatal error : PCH file '.\Release\.\/project_pch.pch' not found: module file not found
1>Done building project "project.vcxproj" -- FAILED.

When running with -v on clang-cl.exe's command line, I can see that it starts doing a compile with -emit-pch and another with -emit-obj, apparently running simultaneously. It looks like the object file needs the PCH to be compiled first, so it fails when it doesn't find the .pch file the other process should be working on. I bumped msbuild.exe's verbosity up to see what it tried to do, and it only invoked clang-cl once, to compile one file (project_pch.cpp) with the "emit precompiled header" arguments on the command line. So this shouldn't be too crazy-tough to fix.

(I think my short-term/wrong workaround for this will be to ignore /MP when an input file is type c++-header. The more accurate solution would be to make the .obj depend on the .pch)

---

Has there been any further work on this change outside of what's recorded in this thread?

[tycho]

OK, those two problems were actually easy enough to fix.

diff --git a/lib/Driver/Driver.cpp b/lib/Driver/Driver.cpp
index 2f98cf9e04..d449735973 100644
--- a/lib/Driver/Driver.cpp
+++ b/lib/Driver/Driver.cpp
@@ -3241,7 +3241,8 @@ void Driver::BuildActions(Compilation &C, DerivedArgList &Args,
   }
   if (FinalPhase != phases::Preprocess) {
     if (Arg *A = Args.getLastArg(options::OPT__SLASH_MP)) {
-        C.CoresToUse = llvm::hardware_concurrency();
+      C.CoresToUse = llvm::hardware_concurrency();
+      if (A->getNumValues() == 1)
         StringRef(A->getValue()).getAsInteger(10, C.CoresToUse);
     }
   }

And:

diff --git a/lib/Driver/Driver.cpp b/lib/Driver/Driver.cpp
index d449735973..55b9ca511b 100644
--- a/lib/Driver/Driver.cpp
+++ b/lib/Driver/Driver.cpp
@@ -3239,7 +3239,7 @@ void Driver::BuildActions(Compilation &C, DerivedArgList &Args,
     Args.eraseArg(options::OPT__SLASH_Yu);
     YcArg = YuArg = nullptr;
   }
-  if (FinalPhase != phases::Preprocess) {
+  if (!YcArg && FinalPhase != phases::Preprocess) {
     if (Arg *A = Args.getLastArg(options::OPT__SLASH_MP)) {
       C.CoresToUse = llvm::hardware_concurrency();
       if (A->getNumValues() == 1)

The result is a thing of beauty:

[tycho]

I did some local work to make this build and pass almost all tests on Linux as well, not to make use of the multi-process features but just to avoid having a separate "for Windows only" branch, and to ensure tests pass across platforms. Unfortunately my change is not at all acceptable for inclusion because it's extremely Linux-specific, and Unix/Program.inc very explicitly says "only use generic UNIX code here". I use pidfds and a timerfd combined with epoll_wait to implement sys::WaitMany. It works well, but I'd be concerned about how to implement this as smoothly for any other *nix platform. pidfd/timerfd/epoll made it downright easy, but I started off trying to look at options that would work more universally on e.g. Linux *and* the BSDs, but I couldn't find anything that wouldn't require building a Rube-Goldberg machine juggling timers, alarms, signals, etc, etc.

One thing I noticed in this was one test in particular: Clang :: Driver/cl-fallback.c. It seems to expose a small design oversight. Since we don't ever use Command::Execute or the overrides for it, we don't ever fall back to the other command in FallbackCommand::Execute.

Another thing I see as a big problem for this patch is the change in return value for sys::ExecuteAndWait. I think the rationale of using bool would be a good one in a new project: separating execution failure from child process runtime failure makes sense. But changing it now does cause problems if you merge patches that expect the old return values, and the build silently continues (i.e. if (llvm::sys::ExecuteAndWait(...)) is a common construct). I noticed this in particular when I rebased on top of the LLVM master branch, as there were several new sys::ExecuteAndWait callers (e.g. llvm/tools/llvm-reduce). The difference in return type would probably be less dramatic if it didn't invert the logic (i.e. previously 0 == success, but now true means success and 0 != true, so the whole if(ExecuteAndWait thing breaks terribly). It'd actually be better if the function was renamed or took a different set of arguments, just to cause the build to explicitly break so the callers could be addressed.

Anyway, I'd love to see this patch or a successor to it move forward, as we'd like to start building things with LLVM on *all* platforms where I work, and this would greatly help with the issues for the Visual Studio users (which are the majority of the prospective users at my office).

[aganea]

Hi @tycho ! Sorry for not getting back earlier.
I implemented an alternate approach last year, which proved to be better (in terms of build times) than what I proposed in this demo patch. That is, using a thread pool instead of the process pool as implemented here. This makes compilation with /MP considerably faster, and allows for neat optimizations like file caching, re-using allocated memory pages, and possibly other things down the line. But before getting there, it requires somehow making cl::opt and the CommandLineParser thread-safe, thus revive this RFC thread. In essence, we want a mode where cl::opts have their data optionally stored in a stack context, so that we can safely call a CC1Command in each thread. I'll write a proposal to revive the RFC.

clang/trunk/include/clang/Driver/CLCompatOptions.td

def _SLASH_Gr : CLFlag<"Gr">,
   HelpText<"Set __fastcall as a default calling convention">;
 def _SLASH_Gz : CLFlag<"Gz">,
   HelpText<"Set __stdcall as a default calling convention">;
 def _SLASH_Gv : CLFlag<"Gv">,
   HelpText<"Set __vectorcall as a default calling convention">;
 def _SLASH_Gregcall : CLFlag<"Gregcall">,
   HelpText<"Set __regcall as a default calling convention">;
 
+def _SLASH_MP : CLCompileJoined<"MP">, HelpText<"Build with multiple processes">;
+
 // Ignored:
 
 def _SLASH_analyze_ : CLIgnoredFlag<"analyze-">;
 def _SLASH_bigobj : CLIgnoredFlag<"bigobj">;
 def _SLASH_cgthreads : CLIgnoredJoined<"cgthreads">;
 def _SLASH_d2FastFail : CLIgnoredFlag<"d2FastFail">;
 def _SLASH_d2Zi_PLUS : CLIgnoredFlag<"d2Zi+">;
 def _SLASH_errorReport : CLIgnoredJoined<"errorReport">;

 def _SLASH_GT : CLFlag<"GT">;
 def _SLASH_GZ : CLFlag<"GZ">;
 def _SLASH_H : CLFlag<"H">;
 def _SLASH_homeparams : CLFlag<"homeparams">;
 def _SLASH_hotpatch : CLFlag<"hotpatch">;
 def _SLASH_JMC : CLFlag<"JMC">;
 def _SLASH_kernel : CLFlag<"kernel">;
 def _SLASH_LN : CLFlag<"LN">;
-def _SLASH_MP : CLJoined<"MP">;
 def _SLASH_openmp : CLFlag<"openmp">;
 def _SLASH_Qfast_transcendentals : CLFlag<"Qfast_transcendentals">;
 def _SLASH_QIfist : CLFlag<"QIfist">;
 def _SLASH_Qimprecise_fwaits : CLFlag<"Qimprecise_fwaits">;
 def _SLASH_Qpar : CLFlag<"Qpar">;
 def _SLASH_Qvec_report : CLJoined<"Qvec-report">;
 def _SLASH_u : CLFlag<"u">;
 def _SLASH_V : CLFlag<"V">;

clang/trunk/include/clang/Driver/Compilation.h

class Compilation {
 
   /// Whether an error during the parsing of the input args.
   bool ContainsError;
 
   /// Whether to keep temporary files regardless of -save-temps.
   bool ForceKeepTempFiles = false;
 
 public:
+    /// Number of processors/cores to use for compiling the jobs
+  unsigned CoresToUse = 1;
+
   Compilation(const Driver &D, const ToolChain &DefaultToolChain,
               llvm::opt::InputArgList *Args,
               llvm::opt::DerivedArgList *TranslatedArgs, bool ContainsError);
   ~Compilation();
 
   const Driver &getDriver() const { return TheDriver; }
 
   const ToolChain &getDefaultToolChain() const { return DefaultToolChain; }
bool CleanupFileMap(const ArgStringMap &Files,
                       const JobAction *JA,
                       bool IssueErrors = false) const;
 
   /// ExecuteCommand - Execute an actual command.
   ///
   /// \param FailingCommand - For non-zero results, this will be set to the
   /// Command which failed, if any.
   /// \return The result code of the subprocess.
-  int ExecuteCommand(const Command &C, const Command *&FailingCommand) const;
+  //int ExecuteCommand(const Command &C, const Command *&FailingCommand) const;
+
+  JobExecutionState ExecuteCommandAsync(const Command &C) const;
 
   /// ExecuteJob - Execute a single job.
   ///
   /// \param FailingCommands - For non-zero results, this will be a vector of
   /// failing commands and their associated result code.
-  void ExecuteJobs(
+  void executeJobs(
       const JobList &Jobs,
-      SmallVectorImpl<std::pair<int, const Command *>> &FailingCommands) const;
+      SmallVectorImpl<JobExecutionState> &FailingCommands) const;
+
+  void waitOnJobs(SmallVectorImpl<JobExecutionState> &ActiveJobs) const;
+
+  void terminateAllJobs(SmallVectorImpl<JobExecutionState> &ActiveJobs) const;
 
   /// initCompilationForDiagnostics - Remove stale state and suppress output
   /// so compilation can be reexecuted to generate additional diagnostic
   /// information (e.g., preprocessed source(s)).
   void initCompilationForDiagnostics();
 
   /// Return true if we're compiling for diagnostics.
   bool isForDiagnostics() const { return ForDiagnostics; }

clang/trunk/include/clang/Driver/Driver.h

 //===--- Driver.h - Clang GCC Compatible Driver -----------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_DRIVER_DRIVER_H
 #define LLVM_CLANG_DRIVER_DRIVER_H
 
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/LLVM.h"
 #include "clang/Driver/Action.h"
+#include "clang/Driver/Job.h"
 #include "clang/Driver/Phases.h"
 #include "clang/Driver/ToolChain.h"
 #include "clang/Driver/Types.h"
 #include "clang/Driver/Util.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Option/ArgList.h"
 #include "llvm/Support/StringSaver.h"
private:
   TranslateInputArgs(const llvm::opt::InputArgList &Args) const;
 
   // getFinalPhase - Determine which compilation mode we are in and record
   // which option we used to determine the final phase.
   phases::ID getFinalPhase(const llvm::opt::DerivedArgList &DAL,
                            llvm::opt::Arg **FinalPhaseArg = nullptr) const;
 
   // Before executing jobs, sets up response files for commands that need them.
-  void setUpResponseFiles(Compilation &C, Command &Cmd);
+  void setUpResponseFiles(Compilation &C, Command &Cmd) const;
 
   void generatePrefixedToolNames(StringRef Tool, const ToolChain &TC,
                                  SmallVectorImpl<std::string> &Names) const;
 
   /// Find the appropriate .crash diagonostic file for the child crash
   /// under this driver and copy it out to a temporary destination with the
   /// other reproducer related files (.sh, .cache, etc). If not found, suggest a
   /// directory for the user to look at.
public:
 
   /// ExecuteCompilation - Execute the compilation according to the command line
   /// arguments and return an appropriate exit code.
   ///
   /// This routine handles additional processing that must be done in addition
   /// to just running the subprocesses, for example reporting errors, setting
   /// up response files, removing temporary files, etc.
   int ExecuteCompilation(Compilation &C,
-     SmallVectorImpl< std::pair<int, const Command *> > &FailingCommands);
+     SmallVectorImpl<JobExecutionState> &FailingCommands);
 
   /// Contains the files in the compilation diagnostic report generated by
   /// generateCompilationDiagnostics.
   struct CompilationDiagnosticReport {
     llvm::SmallVector<std::string, 4> TemporaryFiles;
   };
 
   /// generateCompilationDiagnostics - Generate diagnostics information

clang/trunk/include/clang/Driver/Job.h

 
 #include "clang/Basic/LLVM.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/iterator.h"
 #include "llvm/Option/Option.h"
+#include "llvm/Support/Program.h"
 #include <memory>
 #include <string>
 #include <utility>
 #include <vector>
 
 namespace clang {
 namespace driver {
 
 class Action;
+class Command;
 class InputInfo;
 class Tool;
 
 // Re-export this as clang::driver::ArgStringList.
 using llvm::opt::ArgStringList;
 
 struct CrashReportInfo {
   StringRef Filename;
   StringRef VFSPath;
 
   CrashReportInfo(StringRef Filename, StringRef VFSPath)
       : Filename(Filename), VFSPath(VFSPath) {}
 };
 
+struct JobExecutionState {
+    bool isFree() const { return !C; }
+
+    JobExecutionState() : C() {}
+    const Command *C;
+    llvm::sys::ProcessInfo P;
+};
+
 /// Command - An executable path/name and argument vector to
 /// execute.
 class Command {
   /// Source - The action which caused the creation of this job.
   const Action &Source;
 
   /// Tool - The tool which caused the creation of this job.
   const Tool &Creator;
public:
   // FIXME: This really shouldn't be copyable, but is currently copied in some
   // error handling in Driver::generateCompilationDiagnostics.
   Command(const Command &) = default;
   virtual ~Command() = default;
 
   virtual void Print(llvm::raw_ostream &OS, const char *Terminator, bool Quote,
                      CrashReportInfo *CrashInfo = nullptr) const;
 
-  virtual int Execute(ArrayRef<Optional<StringRef>> Redirects,
-                      std::string *ErrMsg, bool *ExecutionFailed) const;
+  virtual bool Execute(ArrayRef<llvm::Optional<StringRef>> Redirects,
+                       llvm::sys::ProcessInfo &PI) const;
+
+  JobExecutionState
+  ExecuteAsync(ArrayRef<llvm::Optional<StringRef>> Redirects) const;
 
   /// getSource - Return the Action which caused the creation of this job.
   const Action &getSource() const { return Source; }
 
   /// getCreator - Return the Tool which caused the creation of this job.
   const Tool &getCreator() const { return Creator; }
 
   /// Set to pass arguments via a response file when launching the command
public:
   FallbackCommand(const Action &Source_, const Tool &Creator_,
                   const char *Executable_, const ArgStringList &Arguments_,
                   ArrayRef<InputInfo> Inputs,
                   std::unique_ptr<Command> Fallback_);
 
   void Print(llvm::raw_ostream &OS, const char *Terminator, bool Quote,
              CrashReportInfo *CrashInfo = nullptr) const override;
 
-  int Execute(ArrayRef<Optional<StringRef>> Redirects, std::string *ErrMsg,
-              bool *ExecutionFailed) const override;
+  bool Execute(ArrayRef<llvm::Optional<StringRef>> Redirects,
+               llvm::sys::ProcessInfo &PI) const override;
 
 private:
   std::unique_ptr<Command> Fallback;
 };
 
 /// Like Command, but always pretends that the wrapped command succeeded.
 class ForceSuccessCommand : public Command {
 public:
   ForceSuccessCommand(const Action &Source_, const Tool &Creator_,
                       const char *Executable_, const ArgStringList &Arguments_,
                       ArrayRef<InputInfo> Inputs);
 
   void Print(llvm::raw_ostream &OS, const char *Terminator, bool Quote,
              CrashReportInfo *CrashInfo = nullptr) const override;
 
-  int Execute(ArrayRef<Optional<StringRef>> Redirects, std::string *ErrMsg,
-              bool *ExecutionFailed) const override;
+  bool Execute(ArrayRef<llvm::Optional<StringRef>> Redirects,
+               llvm::sys::ProcessInfo &PI) const override;
 };
 
 /// JobList - A sequence of jobs to perform.
 class JobList {
 public:
   using list_type = SmallVector<std::unique_ptr<Command>, 4>;
   using size_type = list_type::size_type;
   using iterator = llvm::pointee_iterator<list_type::iterator>;

clang/trunk/lib/Driver/Compilation.cpp

for (const auto &File : Files) {
     // Otherwise, delete all files in the map.
     if (JA && File.first != JA)
       continue;
     Success &= CleanupFile(File.second, IssueErrors);
   }
   return Success;
 }
 
-int Compilation::ExecuteCommand(const Command &C,
-                                const Command *&FailingCommand) const {
-  if ((getDriver().CCPrintOptions ||
-       getArgs().hasArg(options::OPT_v)) && !getDriver().CCGenDiagnostics) {
+JobExecutionState Compilation::ExecuteCommandAsync(const Command &C) const {
+  JobExecutionState S;
+  S.C = &C;
+
+  if ((getDriver().CCPrintOptions || getArgs().hasArg(options::OPT_v)) &&
+      !getDriver().CCGenDiagnostics) {
     raw_ostream *OS = &llvm::errs();
 
     // Follow gcc implementation of CC_PRINT_OPTIONS; we could also cache the
     // output stream.
     if (getDriver().CCPrintOptions && getDriver().CCPrintOptionsFilename) {
       std::error_code EC;
       OS = new llvm::raw_fd_ostream(getDriver().CCPrintOptionsFilename, EC,
                                     llvm::sys::fs::F_Append |
                                         llvm::sys::fs::F_Text);
       if (EC) {
         getDriver().Diag(diag::err_drv_cc_print_options_failure)
             << EC.message();
-        FailingCommand = &C;
         delete OS;
-        return 1;
+
+        S.P.ReturnCode = 1;
+        return S;
       }
     }
 
     if (getDriver().CCPrintOptions)
       *OS << "[Logging clang options]";
 
     C.Print(*OS, "\n", /*Quote=*/getDriver().CCPrintOptions);
 
     if (OS != &llvm::errs())
       delete OS;
   }
 
-  std::string Error;
-  bool ExecutionFailed;
-  int Res = C.Execute(Redirects, &Error, &ExecutionFailed);
-  if (!Error.empty()) {
-    assert(Res && "Error string set with 0 result code!");
-    getDriver().Diag(diag::err_drv_command_failure) << Error;
-  }
+  S = C.ExecuteAsync(Redirects);
+  return S;
 
-  if (Res)
-    FailingCommand = &C;
+void Compilation::waitOnJobs(
+    SmallVectorImpl<JobExecutionState> &ActiveJobs) const {
+  SmallVector<llvm::sys::ProcessInfo *, 16> Waitable;
+  Waitable.reserve(ActiveJobs.size());
 
-  return ExecutionFailed ? 1 : Res;
+  for (auto &S : ActiveJobs) {
+    if (!S.P.running())
+      continue;
+    Waitable.push_back(&S.P);
+  }
+  llvm::sys::WaitMany(Waitable, /*WaitOnAll*/ false, 0,
+                      /*WaitUntilChildTerminates*/ true);
 }
 
-using FailingCommandList = SmallVectorImpl<std::pair<int, const Command *>>;
+void Compilation::terminateAllJobs(
+    SmallVectorImpl<JobExecutionState> &ActiveJobs) const {
+  SmallVector<llvm::sys::ProcessInfo *, 16> Waitable;
+  Waitable.reserve(ActiveJobs.size());
+
+  for (auto &S : ActiveJobs) {
+    if (S.isFree() || S.P.completedSuccess())
+      continue;
+    Waitable.push_back(&S.P);
+  }
+  llvm::sys::Terminate(Waitable);
+}
+
+using FailingCommandList = SmallVectorImpl<JobExecutionState>;
 
 static bool ActionFailed(const Action *A,
                          const FailingCommandList &FailingCommands) {
   if (FailingCommands.empty())
     return false;
 
   // CUDA/HIP can have the same input source code compiled multiple times so do
   // not compiled again if there are already failures. It is OK to abort the
   // CUDA pipeline on errors.
   if (A->isOffloading(Action::OFK_Cuda) || A->isOffloading(Action::OFK_HIP))
     return true;
 
-  for (const auto &CI : FailingCommands)
-    if (A == &(CI.second->getSource()))
+  for (const auto &S : FailingCommands)
+    if (A == &(S.C->getSource()))
       return true;
 
   for (const auto *AI : A->inputs())
     if (ActionFailed(AI, FailingCommands))
       return true;
 
   return false;
 }
 
 static bool InputsOk(const Command &C,
                      const FailingCommandList &FailingCommands) {
   return !ActionFailed(&C.getSource(), FailingCommands);
 }
 
-void Compilation::ExecuteJobs(const JobList &Jobs,
+static bool pruneCompletedJobs(SmallVectorImpl<JobExecutionState> &ActiveJobs,
+                               int &JobsDone,
+                               FailingCommandList &FailingCommands) {
+  bool HasErrors = false;
+  for (auto &S : ActiveJobs) {
+    if (S.isFree() || S.P.running())
+      continue;
+
+    ++JobsDone;
+
+    if (S.P.completedError()) {
+      FailingCommands.push_back(S);
+      HasErrors = true;
+    }
+
+    // Free slot
+    S = JobExecutionState();
+  }
+  return HasErrors;
+}
+
+void Compilation::executeJobs(const JobList &Jobs,
                               FailingCommandList &FailingCommands) const {
-  // According to UNIX standard, driver need to continue compiling all the
-  // inputs on the command line even one of them failed.
-  // In all but CLMode, execute all the jobs unless the necessary inputs for the
-  // job is missing due to previous failures.
-  for (const auto &Job : Jobs) {
-    if (!InputsOk(Job, FailingCommands))
+
+  // Allocate one slot for each processor/core
+  SmallVector<JobExecutionState, 16> ActiveJobs(
+      std::min(CoresToUse, (unsigned)Jobs.size()));
+
+  int JobsDone = 0;
+  auto NextJob = Jobs.begin();
+
+  // The job allocation algorithm works as following, on each loop:
+  // 1. Fill up all processing slots (one per core) by spawning a new
+  // compilation job on each core
+  // 2. Wait on all cores, until at least one job finishes (waiting with
+  // system-level primitives)
+  // 3. Check all completed jobs for failures
+  // 4. Loop until the job list is exhausted
+  while (JobsDone < Jobs.size()) {
+
+    // Spawn new jobs on inactive processor cores
+    for (auto &S : ActiveJobs) {
+      if (!S.isFree())
+        continue;
+
+      if (NextJob == Jobs.end())
+        break;
+
+      Command &C = *NextJob++;
+      // According to UNIX standard, the driver needs to continue compiling
+      // all the inputs on the command line even one of them failed. In all
+      // but CLMode, execute all the jobs unless the necessary inputs for the
+      // job are missing due to previous failures.
+      if (!InputsOk(C, FailingCommands)) {
+        ++JobsDone;
         continue;
-    const Command *FailingCommand = nullptr;
-    if (int Res = ExecuteCommand(Job, FailingCommand)) {
-      FailingCommands.push_back(std::make_pair(Res, FailingCommand));
-      // Bail as soon as one command fails in cl driver mode.
-      if (TheDriver.IsCLMode())
-        return;
+      }
+      // Start new compilation job
+      S = ExecuteCommandAsync(C);
+    }
+
+    // Spin on active jobs with OS primitives, until one of them finishes
+    waitOnJobs(ActiveJobs);
+
+    // Complete job processing once finished
+    bool HasErrors = pruneCompletedJobs(ActiveJobs, JobsDone, FailingCommands);
+
+    if (HasErrors && TheDriver.IsCLMode()) {
+      terminateAllJobs(ActiveJobs);
+
+      // Ensure any killed job is queued as 'FailingCommands', to clean up
+      // temporary files
+      for (auto &S : ActiveJobs) {
+        if (S.isFree())
+          continue;
+        assert(!S.P.running());
+
+        if (S.P.completedError())
+          FailingCommands.push_back(S);
+      }
     }
   }
 }
 
 void Compilation::initCompilationForDiagnostics() {
   ForDiagnostics = true;
 
   // Free actions and jobs.

clang/trunk/lib/Driver/Driver.cpp

 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/PrettyStackTrace.h"
 #include "llvm/Support/Process.h"
 #include "llvm/Support/Program.h"
 #include "llvm/Support/StringSaver.h"
 #include "llvm/Support/TargetRegistry.h"
+#include "llvm/Support/Threading.h"
 #include "llvm/Support/raw_ostream.h"
 #include <map>
 #include <memory>
 #include <utility>
 #if LLVM_ON_UNIX
 #include <unistd.h> // getpid
 #endif
 
void Driver::generateCompilationDiagnostics(
   // If there were errors building the compilation, quit now.
   if (Trap.hasErrorOccurred()) {
     Diag(clang::diag::note_drv_command_failed_diag_msg)
         << "Error generating preprocessed source(s).";
     return;
   }
 
   // Generate preprocessed output.
-  SmallVector<std::pair<int, const Command *>, 4> FailingCommands;
-  C.ExecuteJobs(C.getJobs(), FailingCommands);
+  SmallVector<JobExecutionState, 4> FailingCommands;
+  C.executeJobs(C.getJobs(), FailingCommands);
 
   // If any of the preprocessing commands failed, clean up and exit.
   if (!FailingCommands.empty()) {
     Diag(clang::diag::note_drv_command_failed_diag_msg)
         << "Error generating preprocessed source(s).";
     return;
   }
 
void Driver::generateCompilationDiagnostics(
   for (const auto &A : C.getArgs().filtered(options::OPT_frewrite_map_file,
                                             options::OPT_frewrite_map_file_EQ))
     Diag(clang::diag::note_drv_command_failed_diag_msg) << A->getValue();
 
   Diag(clang::diag::note_drv_command_failed_diag_msg)
       << "\n\n********************";
 }
 
-void Driver::setUpResponseFiles(Compilation &C, Command &Cmd) {
+void Driver::setUpResponseFiles(Compilation &C, Command &Cmd) const {
   // Since commandLineFitsWithinSystemLimits() may underestimate system's capacity
   // if the tool does not support response files, there is a chance/ that things
   // will just work without a response file, so we silently just skip it.
   if (Cmd.getCreator().getResponseFilesSupport() == Tool::RF_None ||
       llvm::sys::commandLineFitsWithinSystemLimits(Cmd.getExecutable(), Cmd.getArguments()))
     return;
 
   std::string TmpName = GetTemporaryPath("response", "txt");
   Cmd.setResponseFile(C.addTempFile(C.getArgs().MakeArgString(TmpName)));
 }
 
 int Driver::ExecuteCompilation(
     Compilation &C,
-    SmallVectorImpl<std::pair<int, const Command *>> &FailingCommands) {
+    SmallVectorImpl<JobExecutionState> &FailingCommands) {
   // Just print if -### was present.
   if (C.getArgs().hasArg(options::OPT__HASH_HASH_HASH)) {
     C.getJobs().Print(llvm::errs(), "\n", true);
-    return 0;
+    return (int)llvm::sys::ProgramReturnCode::Success;
   }
 
   // If there were errors building the compilation, quit now.
   if (Diags.hasErrorOccurred())
-    return 1;
+    return (int)llvm::sys::ProgramReturnCode::Failure;
 
   // Set up response file names for each command, if necessary
   for (auto &Job : C.getJobs())
     setUpResponseFiles(C, Job);
 
-  C.ExecuteJobs(C.getJobs(), FailingCommands);
+  C.executeJobs(C.getJobs(), FailingCommands);
 
   // If the command succeeded, we are done.
   if (FailingCommands.empty())
-    return 0;
+    return (int)llvm::sys::ProgramReturnCode::Success;
 
   // Otherwise, remove result files and print extra information about abnormal
   // failures.
-  for (const auto &CmdPair : FailingCommands) {
-    int Res = CmdPair.first;
-    const Command *FailingCommand = CmdPair.second;
+  Optional<int> Res;
+  for (const auto &S : FailingCommands) {
+    const Command *FailingCommand = S.C;
 
     // Remove result files if we're not saving temps.
     if (!isSaveTempsEnabled()) {
       const JobAction *JA = cast<JobAction>(&FailingCommand->getSource());
       C.CleanupFileMap(C.getResultFiles(), JA, true);
 
       // Failure result files are valid unless we crashed.
-      if (Res < 0)
+      if (S.P.notCompletedOrFailed())
         C.CleanupFileMap(C.getFailureResultFiles(), JA, true);
     }
 
     // Print extra information about abnormal failures, if possible.
     //
     // This is ad-hoc, but we don't want to be excessively noisy. If the result
     // status was 1, assume the command failed normally. In particular, if it
     // was the compiler then assume it gave a reasonable error code. Failures
     // in other tools are less common, and they generally have worse
     // diagnostics, so always print the diagnostic there.
     const Tool &FailingTool = FailingCommand->getCreator();
 
-    if (!FailingCommand->getCreator().hasGoodDiagnostics() || Res != 1) {
-      // FIXME: See FIXME above regarding result code interpretation.
-      if (Res < 0)
+    if (!FailingCommand->getCreator().hasGoodDiagnostics()) {
+      if (S.P.notCompletedOrFailed())
         Diag(clang::diag::err_drv_command_signalled)
             << FailingTool.getShortName();
+      else if (S.P.completedError() &&
+               S.P.ReturnCode.getValue() != (int)llvm::sys::ProgramReturnCode::Failure) {
+        if (S.P.ErrMsg.hasValue())
+          Diag(diag::err_drv_command_failure)
+              << FailingTool.getShortName() << S.P.ErrMsg.getValue();
         else
-        Diag(clang::diag::err_drv_command_failed) << FailingTool.getShortName()
-                                                  << Res;
+          Diag(clang::diag::err_drv_command_failed)
+              << FailingTool.getShortName() << S.P.ReturnCode.getValue();
       }
     }
-  return 0;
+    Res = S.P.convertToParentReturnCode();
+  }
+  return Res.getValueOr((int)llvm::sys::ProgramReturnCode::Success);
 }
 
 void Driver::PrintHelp(bool ShowHidden) const {
   unsigned IncludedFlagsBitmask;
   unsigned ExcludedFlagsBitmask;
   std::tie(IncludedFlagsBitmask, ExcludedFlagsBitmask) =
       getIncludeExcludeOptionFlagMasks();
 
if (FinalPhase == phases::Preprocess || Args.hasArg(options::OPT__SLASH_Y_)) {
     // If only preprocessing or /Y- is used, all pch handling is disabled.
     // Rather than check for it everywhere, just remove clang-cl pch-related
     // flags here.
     Args.eraseArg(options::OPT__SLASH_Fp);
     Args.eraseArg(options::OPT__SLASH_Yc);
     Args.eraseArg(options::OPT__SLASH_Yu);
     YcArg = YuArg = nullptr;
   }
+  if (FinalPhase != phases::Preprocess) {
+    if (Arg *A = Args.getLastArg(options::OPT__SLASH_MP)) {
+        C.CoresToUse = llvm::hardware_concurrency();
+        StringRef(A->getValue()).getAsInteger(10, C.CoresToUse);
+    }
+  }
 
   // Builder to be used to build offloading actions.
   OffloadingActionBuilder OffloadBuilder(C, Args, Inputs);
 
   // Construct the actions to perform.
   ActionList LinkerInputs;
 
   llvm::SmallVector<phases::ID, phases::MaxNumberOfPhases> PL;

clang/trunk/lib/Driver/Job.cpp

 }
 
 void Command::setEnvironment(llvm::ArrayRef<const char *> NewEnvironment) {
   Environment.reserve(NewEnvironment.size() + 1);
   Environment.assign(NewEnvironment.begin(), NewEnvironment.end());
   Environment.push_back(nullptr);
 }
 
-int Command::Execute(ArrayRef<llvm::Optional<StringRef>> Redirects,
-                     std::string *ErrMsg, bool *ExecutionFailed) const {
-  SmallVector<const char*, 128> Argv;
+JobExecutionState
+Command::ExecuteAsync(ArrayRef<llvm::Optional<StringRef>> Redirects) const {
+  SmallVector<const char *, 128> Argv;
 
   Optional<ArrayRef<StringRef>> Env;
   std::vector<StringRef> ArgvVectorStorage;
   if (!Environment.empty()) {
     assert(Environment.back() == nullptr &&
            "Environment vector should be null-terminated by now");
     ArgvVectorStorage = llvm::toStringRefArray(Environment.data());
     Env = makeArrayRef(ArgvVectorStorage);
   }
 
+  JobExecutionState S;
+  S.C = this;
+
   if (ResponseFile == nullptr) {
     Argv.push_back(Executable);
     Argv.append(Arguments.begin(), Arguments.end());
     Argv.push_back(nullptr);
-
-    auto Args = llvm::toStringRefArray(Argv.data());
-    return llvm::sys::ExecuteAndWait(
-        Executable, Args, Env, Redirects, /*secondsToWait*/ 0,
-        /*memoryLimit*/ 0, ErrMsg, ExecutionFailed);
-  }
-
+  } else {
     // We need to put arguments in a response file (command is too large)
     // Open stream to store the response file contents
     std::string RespContents;
     llvm::raw_string_ostream SS(RespContents);
 
     // Write file contents and build the Argv vector
     writeResponseFile(SS);
     buildArgvForResponseFile(Argv);
     Argv.push_back(nullptr);
     SS.flush();
 
     // Save the response file in the appropriate encoding
     if (std::error_code EC = writeFileWithEncoding(
             ResponseFile, RespContents, Creator.getResponseFileEncoding())) {
-    if (ErrMsg)
-      *ErrMsg = EC.message();
-    if (ExecutionFailed)
-      *ExecutionFailed = true;
-    return -1;
+      S.P.ReturnCode = -1;
+      S.P.ErrMsg = toString(createFileError(ResponseFile, llvm::errorCodeToError(EC)));
+      return S;
+    }
   }
 
   auto Args = llvm::toStringRefArray(Argv.data());
-  return llvm::sys::ExecuteAndWait(Executable, Args, Env, Redirects,
-                                   /*secondsToWait*/ 0,
-                                   /*memoryLimit*/ 0, ErrMsg, ExecutionFailed);
+
+  S.P = llvm::sys::ExecuteNoWait(Executable, Args, Env, Redirects,
+                                 /*memoryLimit*/ 0);
+  return S;
+}
+
+bool Command::Execute(ArrayRef<llvm::Optional<StringRef>> Redirects,
+                      llvm::sys::ProcessInfo &PI) const {
+  auto S = ExecuteAsync(Redirects);
+  bool R = llvm::sys::Wait(S.P, 0, /*WaitUntilChildTerminates*/ true);
+  PI = S.P;
+  return R;
 }
 
 FallbackCommand::FallbackCommand(const Action &Source_, const Tool &Creator_,
                                  const char *Executable_,
                                  const ArgStringList &Arguments_,
                                  ArrayRef<InputInfo> Inputs,
                                  std::unique_ptr<Command> Fallback_)
     : Command(Source_, Creator_, Executable_, Arguments_, Inputs),
       Fallback(std::move(Fallback_)) {}
 
 void FallbackCommand::Print(raw_ostream &OS, const char *Terminator,
                             bool Quote, CrashReportInfo *CrashInfo) const {
   Command::Print(OS, "", Quote, CrashInfo);
   OS << " ||";
   Fallback->Print(OS, Terminator, Quote, CrashInfo);
 }
 
-static bool ShouldFallback(int ExitCode) {
+static bool ShouldFallback(llvm::sys::ProcessInfo &PI) {
   // FIXME: We really just want to fall back for internal errors, such
   // as when some symbol cannot be mangled, when we should be able to
   // parse something but can't, etc.
-  return ExitCode != 0;
+  return PI.completedError();
 }
 
-int FallbackCommand::Execute(ArrayRef<llvm::Optional<StringRef>> Redirects,
-                             std::string *ErrMsg, bool *ExecutionFailed) const {
-  int PrimaryStatus = Command::Execute(Redirects, ErrMsg, ExecutionFailed);
-  if (!ShouldFallback(PrimaryStatus))
-    return PrimaryStatus;
-
-  // Clear ExecutionFailed and ErrMsg before falling back.
-  if (ErrMsg)
-    ErrMsg->clear();
-  if (ExecutionFailed)
-    *ExecutionFailed = false;
+bool FallbackCommand::Execute(ArrayRef<llvm::Optional<StringRef>> Redirects,
+                              llvm::sys::ProcessInfo &PI) const {
+  Command::Execute(Redirects, PI);
+  if (!ShouldFallback(PI))
+    return false;
 
   const Driver &D = getCreator().getToolChain().getDriver();
   D.Diag(diag::warn_drv_invoking_fallback) << Fallback->getExecutable();
 
-  int SecondaryStatus = Fallback->Execute(Redirects, ErrMsg, ExecutionFailed);
-  return SecondaryStatus;
+  PI.clear();
+
+  Fallback->Execute(Redirects, PI);
+  return PI.completedSuccess();
 }
 
 ForceSuccessCommand::ForceSuccessCommand(const Action &Source_,
                                          const Tool &Creator_,
                                          const char *Executable_,
                                          const ArgStringList &Arguments_,
                                          ArrayRef<InputInfo> Inputs)
     : Command(Source_, Creator_, Executable_, Arguments_, Inputs) {}
 
 void ForceSuccessCommand::Print(raw_ostream &OS, const char *Terminator,
                             bool Quote, CrashReportInfo *CrashInfo) const {
   Command::Print(OS, "", Quote, CrashInfo);
   OS << " || (exit 0)" << Terminator;
 }
 
-int ForceSuccessCommand::Execute(ArrayRef<llvm::Optional<StringRef>> Redirects,
-                                 std::string *ErrMsg,
-                                 bool *ExecutionFailed) const {
-  int Status = Command::Execute(Redirects, ErrMsg, ExecutionFailed);
-  (void)Status;
-  if (ExecutionFailed)
-    *ExecutionFailed = false;
-  return 0;
+bool ForceSuccessCommand::Execute(ArrayRef<llvm::Optional<StringRef>> Redirects,
+                                  llvm::sys::ProcessInfo &PI) const {
+  Command::Execute(Redirects, PI);
+  return true;
 }
 
 void JobList::Print(raw_ostream &OS, const char *Terminator, bool Quote,
                     CrashReportInfo *CrashInfo) const {
   for (const auto &Job : *this)
     Job.Print(OS, Terminator, Quote, CrashInfo);
 }
 
 void JobList::clear() { Jobs.clear(); }

clang/trunk/tools/driver/cc1_main.cpp

 #include "llvm/ADT/Statistic.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/LinkAllPasses.h"
 #include "llvm/Option/ArgList.h"
 #include "llvm/Option/OptTable.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/ManagedStatic.h"
+#include "llvm/Support/Program.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/TargetSelect.h"
 #include "llvm/Support/Timer.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cstdio>
 
 #ifdef CLANG_HAVE_RLIMITS
 #include <sys/resource.h>
static void LLVMErrorHandler(void *UserData, const std::string &Message,
 
   // Run the interrupt handlers to make sure any special cleanups get done, in
   // particular that we remove files registered with RemoveFileOnSignal.
   llvm::sys::RunInterruptHandlers();
 
   // We cannot recover from llvm errors.  When reporting a fatal error, exit
   // with status 70 to generate crash diagnostics.  For BSD systems this is
   // defined as an internal software error.  Otherwise, exit with status 1.
-  exit(GenCrashDiag ? 70 : 1);
+  exit(int(GenCrashDiag ? llvm::sys::ProgramReturnCode::GenerateCrashDiagnostic
+                        : llvm::sys::ProgramReturnCode::Failure));
 }
 
 #ifdef LINK_POLLY_INTO_TOOLS
 namespace polly {
 void initializePollyPasses(llvm::PassRegistry &Registry);
 }
 #endif
 
static void ensureSufficientStack() {
   // We should now have a stack of size at least DesiredStackSize. Ensure
   // that we can actually use that much, if necessary.
   ensureStackAddressSpace();
 }
 #else
 static void ensureSufficientStack() {}
 #endif
 
-int cc1_main(ArrayRef<const char *> Argv, const char *Argv0, void *MainAddr) {
+bool cc1_main(ArrayRef<const char *> Argv, const char *Argv0, void *MainAddr) {
   ensureSufficientStack();
 
   std::unique_ptr<CompilerInstance> Clang(new CompilerInstance());
   IntrusiveRefCntPtr<DiagnosticIDs> DiagID(new DiagnosticIDs());
 
   // Register the support for object-file-wrapped Clang modules.
   auto PCHOps = Clang->getPCHContainerOperations();
   PCHOps->registerWriter(llvm::make_unique<ObjectFilePCHContainerWriter>());
#endif
   if (Clang->getHeaderSearchOpts().UseBuiltinIncludes &&
       Clang->getHeaderSearchOpts().ResourceDir.empty())
     Clang->getHeaderSearchOpts().ResourceDir =
       CompilerInvocation::GetResourcesPath(Argv0, MainAddr);
 
   // Create the actual diagnostics engine.
   Clang->createDiagnostics();
   if (!Clang->hasDiagnostics())
-    return 1;
+    return false;
 
   // Set an error handler, so that any LLVM backend diagnostics go through our
   // error handler.
   llvm::install_fatal_error_handler(LLVMErrorHandler,
                                   static_cast<void*>(&Clang->getDiagnostics()));
 
   DiagsBuffer->FlushDiagnostics(Clang->getDiagnostics());
   if (!Success)
-    return 1;
+    return false;
 
   // Execute the frontend actions.
   Success = ExecuteCompilerInvocation(Clang.get());
 
   // If any timers were active but haven't been destroyed yet, print their
   // results now.  This happens in -disable-free mode.
   llvm::TimerGroup::printAll(llvm::errs());
 
   // Our error handler depends on the Diagnostics object, which we're
   // potentially about to delete. Uninstall the handler now so that any
   // later errors use the default handling behavior instead.
   llvm::remove_fatal_error_handler();
 
   // When running with -disable-free, don't do any destruction or shutdown.
   if (Clang->getFrontendOpts().DisableFree) {
     BuryPointer(std::move(Clang));
-    return !Success;
   }
-
-  return !Success;
+  return Success;
 }

clang/trunk/tools/driver/cc1as_main.cpp

static void LLVMErrorHandler(void *UserData, const std::string &Message,
   DiagnosticsEngine &Diags = *static_cast<DiagnosticsEngine*>(UserData);
 
   Diags.Report(diag::err_fe_error_backend) << Message;
 
   // We cannot recover from llvm errors.
   exit(1);
 }
 
-int cc1as_main(ArrayRef<const char *> Argv, const char *Argv0, void *MainAddr) {
+bool cc1as_main(ArrayRef<const char *> Argv, const char *Argv0, void *MainAddr) {
   // Initialize targets and assembly printers/parsers.
   InitializeAllTargetInfos();
   InitializeAllTargetMCs();
   InitializeAllAsmParsers();
 
   // Construct our diagnostic client.
   IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new DiagnosticOptions();
   TextDiagnosticPrinter *DiagClient
     = new TextDiagnosticPrinter(errs(), &*DiagOpts);
   DiagClient->setPrefix("clang -cc1as");
   IntrusiveRefCntPtr<DiagnosticIDs> DiagID(new DiagnosticIDs());
   DiagnosticsEngine Diags(DiagID, &*DiagOpts, DiagClient);
 
   // Set an error handler, so that any LLVM backend diagnostics go through our
   // error handler.
   ScopedFatalErrorHandler FatalErrorHandler
     (LLVMErrorHandler, static_cast<void*>(&Diags));
 
   // Parse the arguments.
   AssemblerInvocation Asm;
   if (!AssemblerInvocation::CreateFromArgs(Asm, Argv, Diags))
-    return 1;
+    return false;
 
   if (Asm.ShowHelp) {
     std::unique_ptr<OptTable> Opts(driver::createDriverOptTable());
     Opts->PrintHelp(llvm::outs(), "clang -cc1as", "Clang Integrated Assembler",
                     /*Include=*/driver::options::CC1AsOption, /*Exclude=*/0,
                     /*ShowAllAliases=*/false);
-    return 0;
+    return true;
   }
 
   // Honor -version.
   //
   // FIXME: Use a better -version message?
   if (Asm.ShowVersion) {
     llvm::cl::PrintVersionMessage();
-    return 0;
+    return true;
   }
 
   // Honor -mllvm.
   //
   // FIXME: Remove this, one day.
   if (!Asm.LLVMArgs.empty()) {
     unsigned NumArgs = Asm.LLVMArgs.size();
     auto Args = llvm::make_unique<const char*[]>(NumArgs + 2);
     Args[0] = "clang (LLVM option parsing)";
     for (unsigned i = 0; i != NumArgs; ++i)
       Args[i + 1] = Asm.LLVMArgs[i].c_str();
     Args[NumArgs + 1] = nullptr;
     llvm::cl::ParseCommandLineOptions(NumArgs + 1, Args.get());
   }
 
   // Execute the invocation, unless there were parsing errors.
   bool Failed = Diags.hasErrorOccurred() || ExecuteAssembler(Asm, Diags);
 
   // If any timers were active but haven't been destroyed yet, print their
   // results now.
   TimerGroup::printAll(errs());
 
-  return !!Failed;
+  return !Failed;
 }

clang/trunk/tools/driver/cc1gen_reproducer_main.cpp

static void printReproducerInformation(
   for (const auto &File : llvm::enumerate(Report.TemporaryFiles)) {
     if (File.index())
       OS << ',';
     OS << '"' << File.value() << '"';
   }
   OS << "]\n}\n";
 }
 
-int cc1gen_reproducer_main(ArrayRef<const char *> Argv, const char *Argv0,
+bool cc1gen_reproducer_main(ArrayRef<const char *> Argv, const char *Argv0,
                            void *MainAddr) {
   if (Argv.size() < 1) {
     llvm::errs() << "error: missing invocation file\n";
-    return 1;
+    return false;
   }
   // Parse the invocation descriptor.
   StringRef Input = Argv[0];
   llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> Buffer =
       llvm::MemoryBuffer::getFile(Input);
   if (!Buffer) {
     llvm::errs() << "error: failed to read " << Input << ": "
                  << Buffer.getError().message() << "\n";
-    return 1;
+    return false;
   }
   llvm::yaml::Input YAML(Buffer.get()->getBuffer());
   ClangInvocationInfo InvocationInfo;
   YAML >> InvocationInfo;
   if (Argv.size() > 1 && Argv[1] == StringRef("-v"))
     InvocationInfo.Dump = true;
 
   // Create an invocation that will produce the reproducer.
   std::vector<const char *> DriverArgs;
   for (const auto &Arg : InvocationInfo.Arguments)
     DriverArgs.push_back(Arg.c_str());
   std::string Path = GetExecutablePath(Argv0, /*CanonicalPrefixes=*/true);
   DriverArgs[0] = Path.c_str();
   llvm::Optional<driver::Driver::CompilationDiagnosticReport> Report =
       generateReproducerForInvocationArguments(DriverArgs, InvocationInfo);
 
   // Emit the information about the reproduce files to stdout.
-  int Result = 1;
+  int Result = false;
   if (Report) {
     printReproducerInformation(llvm::outs(), InvocationInfo, *Report);
-    Result = 0;
+    Result = true;
   }
 
   // Remove the input file.
   llvm::sys::fs::remove(Input);
   return Result;
 }

clang/trunk/tools/driver/driver.cpp

while (*S) {
     if (End != S)
       ApplyOneQAOverride(*OS, Args, std::string(S, End), SavedStrings);
     S = End;
     if (*S != '\0')
       ++S;
   }
 }
 
-extern int cc1_main(ArrayRef<const char *> Argv, const char *Argv0,
+extern bool cc1_main(ArrayRef<const char *> Argv, const char *Argv0,
                     void *MainAddr);
-extern int cc1as_main(ArrayRef<const char *> Argv, const char *Argv0,
+extern bool cc1as_main(ArrayRef<const char *> Argv, const char *Argv0,
                       void *MainAddr);
-extern int cc1gen_reproducer_main(ArrayRef<const char *> Argv,
+extern bool cc1gen_reproducer_main(ArrayRef<const char *> Argv,
                                   const char *Argv0, void *MainAddr);
 
 static void insertTargetAndModeArgs(const ParsedClangName &NameParts,
                                     SmallVectorImpl<const char *> &ArgVector,
                                     std::set<std::string> &SavedStrings) {
   // Put target and mode arguments at the start of argument list so that
   // arguments specified in command line could override them. Avoid putting
   // them at index 0, as an option like '-cc1' must remain the first.
static void SetInstallDir(SmallVectorImpl<const char *> &argv,
   if (CanonicalPrefixes)
     llvm::sys::fs::make_absolute(InstalledPath);
 
   StringRef InstalledPathParent(llvm::sys::path::parent_path(InstalledPath));
   if (llvm::sys::fs::exists(InstalledPathParent))
     TheDriver.setInstalledDir(InstalledPathParent);
 }
 
-static int ExecuteCC1Tool(ArrayRef<const char *> argv, StringRef Tool) {
+static bool ExecuteCC1Tool(ArrayRef<const char *> argv, StringRef Tool) {
   void *GetExecutablePathVP = (void *)(intptr_t) GetExecutablePath;
   if (Tool == "")
     return cc1_main(argv.slice(2), argv[0], GetExecutablePathVP);
   if (Tool == "as")
     return cc1as_main(argv.slice(2), argv[0], GetExecutablePathVP);
   if (Tool == "gen-reproducer")
     return cc1gen_reproducer_main(argv.slice(2), argv[0], GetExecutablePathVP);
 
   // Reject unknown tools.
   llvm::errs() << "error: unknown integrated tool '" << Tool << "'. "
                << "Valid tools include '-cc1' and '-cc1as'.\n";
-  return 1;
+  return false;
 }
 
 int main(int argc_, const char **argv_) {
   llvm::InitLLVM X(argc_, argv_);
   SmallVector<const char *, 256> argv(argv_, argv_ + argc_);
 
   if (llvm::sys::Process::FixupStandardFileDescriptors())
-    return 1;
+    return (int)llvm::sys::ProgramReturnCode::Failure;
 
   llvm::InitializeAllTargets();
   auto TargetAndMode = ToolChain::getTargetAndModeFromProgramName(argv[0]);
 
   llvm::BumpPtrAllocator A;
   llvm::StringSaver Saver(A);
 
   // Parse response files using the GNU syntax, unless we're in CL mode. There
int main(int argc_, const char **argv_) {
   auto FirstArg = std::find_if(argv.begin() + 1, argv.end(),
                                [](const char *A) { return A != nullptr; });
   if (FirstArg != argv.end() && StringRef(*FirstArg).startswith("-cc1")) {
     // If -cc1 came from a response file, remove the EOL sentinels.
     if (MarkEOLs) {
       auto newEnd = std::remove(argv.begin(), argv.end(), nullptr);
       argv.resize(newEnd - argv.begin());
     }
-    return ExecuteCC1Tool(argv, argv[1] + 4);
+    bool R = ExecuteCC1Tool(argv, argv[1] + 4);
+    return int(R ? llvm::sys::ProgramReturnCode::Success
+                 : llvm::sys::ProgramReturnCode::Failure);
   }
 
   bool CanonicalPrefixes = true;
   for (int i = 1, size = argv.size(); i < size; ++i) {
     // Skip end-of-line response file markers
     if (argv[i] == nullptr)
       continue;
     if (StringRef(argv[i]) == "-no-canonical-prefixes") {
int main(int argc_, const char **argv_) {
   SetInstallDir(argv, TheDriver, CanonicalPrefixes);
   TheDriver.setTargetAndMode(TargetAndMode);
 
   insertTargetAndModeArgs(TargetAndMode, argv, SavedStrings);
 
   SetBackdoorDriverOutputsFromEnvVars(TheDriver);
 
   std::unique_ptr<Compilation> C(TheDriver.BuildCompilation(argv));
-  int Res = 1;
+  int Res = (int)llvm::sys::ProgramReturnCode::Failure;
   if (C && !C->containsError()) {
-    SmallVector<std::pair<int, const Command *>, 4> FailingCommands;
+    SmallVector<JobExecutionState, 4> FailingCommands;
     Res = TheDriver.ExecuteCompilation(*C, FailingCommands);
 
     // Force a crash to test the diagnostics.
     if (TheDriver.GenReproducer) {
       Diags.Report(diag::err_drv_force_crash)
         << !::getenv("FORCE_CLANG_DIAGNOSTICS_CRASH");
 
-      // Pretend that every command failed.
       FailingCommands.clear();
-      for (const auto &J : C->getJobs())
-        if (const Command *C = dyn_cast<Command>(&J))
-          FailingCommands.push_back(std::make_pair(-1, C));
-    }
-
-    for (const auto &P : FailingCommands) {
-      int CommandRes = P.first;
-      const Command *FailingCommand = P.second;
-      if (!Res)
-        Res = CommandRes;
-
-      // If result status is < 0, then the driver command signalled an error.
-      // If result status is 70, then the driver command reported a fatal error.
-      // On Windows, abort will return an exit code of 3.  In these cases,
-      // generate additional diagnostic information if possible.
-      bool DiagnoseCrash = CommandRes < 0 || CommandRes == 70;
-#ifdef _WIN32
-      DiagnoseCrash |= CommandRes == 3;
-#endif
-      if (DiagnoseCrash) {
+
+      Command &FirstC = *C->getJobs().begin();
+
+      JobExecutionState S;
+      S.C = &FirstC;
+      S.P.Status = llvm::sys::ExecutionStatus::Crashed;
+      FailingCommands.push_back(S);
+    }
+
+    for (const auto &S : FailingCommands) {
+      const Command *FailingCommand = S.C;
+
+      if (S.P.crashed()) {
         TheDriver.generateCompilationDiagnostics(*C, *FailingCommand);
+        Res = (int)llvm::sys::ProgramReturnCode::Failure;
         break;
       }
     }
   }
 
   Diags.getClient()->finish();
 
   // If any timers were active but haven't been destroyed yet, print their
   // results now.  This happens in -disable-free mode.
   llvm::TimerGroup::printAll(llvm::errs());
 
-#ifdef _WIN32
-  // Exit status should not be negative on Win32, unless abnormal termination.
-  // Once abnormal termiation was caught, negative status should not be
-  // propagated.
-  if (Res < 0)
-    Res = 1;
-#endif
-
   // If we have multiple failing commands, we return the result of the first
   // failing command.
   return Res;
 }

llvm/trunk/include/llvm/Support/Program.h

 #include "llvm/ADT/StringRef.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/Support/ErrorOr.h"
 #include <system_error>
 
 namespace llvm {
 namespace sys {
 
-  /// This is the OS-specific separator for PATH like environment variables:
-  // a colon on Unix or a semicolon on Windows.
+/// Some return codes that clang or other LLVM tools can return
+enum class ProgramReturnCode {
+  Success = 0,
+  Failure = 1,                  // General program failure.
+                                // Also used by Windows' TaskManager when a program is killed manually by the user.
+                                // Clang also uses this as time-out termination code for its children processes.
+#ifdef _WIN32
+  Abort = 3,                    // Returned by abort() on Windows.
+#endif
+  GenerateCrashDiagnostic = 70, // EX_SOFTWARE on BSD, used by Clang to forcibly generate a crash dump
+  BugpointClientFail = 255
+};
+
 #if defined(LLVM_ON_UNIX)
   const char EnvPathSeparator = ':';
 #elif defined (_WIN32)
   const char EnvPathSeparator = ';';
 #endif
 
 #if defined(_WIN32)
   typedef unsigned long procid_t; // Must match the type of DWORD on Windows.
   typedef void *process_t;        // Must match the type of HANDLE on Windows.
 #else
   typedef pid_t procid_t;
   typedef procid_t process_t;
 #endif
 
+  /// This represents the high-level state of the program, as steps of the
+  /// following DAG state machine:
+  ///
+  ///            /--> ExecutionFailure
+  /// NotStarted ---> StartedAndExecuting ---> CompletedSuccess
+  ///                                     \--> CompletedWithError
+  ///                                     \--> Crashed
+  ///                                     \--> Killed
+  enum class ExecutionStatus {
+    NotStarted = 0,
+    StartedAndExecuting, // process is up and running
+    CompletedSuccess,    // process completed with no error
+    CompletedWithError,  // process completed with error, see ReturnCode
+    ExecutionFailure,    // process can't be started
+    Crashed,             // process crashed
+    Killed               // process was forcibly terminated
+  };
+
   /// This struct encapsulates information about a process.
   struct ProcessInfo {
     enum : procid_t { InvalidPid = 0 };
 
     procid_t Pid;      /// The process identifier.
     process_t Process; /// Platform-dependent process object.
 
-    /// The return code, set after execution.
-    int ReturnCode;
+    /// The program execution status
+    ExecutionStatus Status;
+
+    /// The return code after execution, only set if the program terminates with
+    /// an error
+    Optional<int> ReturnCode;
+
+    /// A possible error message after execution
+    Optional<std::string> ErrMsg;
 
     ProcessInfo();
+
+    operator bool() {
+      return Status == ExecutionStatus::StartedAndExecuting ||
+             Status == ExecutionStatus::CompletedSuccess;
+    }
+    bool notStarted() const {
+        return Status == ExecutionStatus::NotStarted;
+    }
+    bool running() const {
+      return Status == ExecutionStatus::StartedAndExecuting;
+    }
+    bool completedError() const {
+      return Status >= ExecutionStatus::CompletedWithError;
+    }
+    bool completedSuccess() const {
+      return Status == ExecutionStatus::CompletedSuccess;
+    }
+    bool notCompletedOrFailed() const {
+      return Status >= ExecutionStatus::ExecutionFailure;
+    }
+    bool crashed(bool ExpectCrash = false) const {
+      if (Status == ExecutionStatus::Crashed)
+        return true;
+      if (!ReturnCode.hasValue())
+        return false;
+      if (ReturnCode.getValue() < 0 ||
+          ReturnCode.getValue() == (int)ProgramReturnCode::GenerateCrashDiagnostic)
+        return true;
+#ifdef _WIN32
+      // Handle abort() in msvcrt -- It has exit code as 3.  abort(), aka
+      // unreachable, should be recognized as a crash.  However, some binaries use
+      // exit code 3 on non-crash failure paths, so only do this if we expect a
+      // crash.
+      if (ExpectCrash && ReturnCode.getValue() == (int)ProgramReturnCode::Abort)
+        return true;
+#endif
+      return false;
+    }
+    bool killed() const { return Status == ExecutionStatus::Killed; }
+    int convertToParentReturnCode() const {
+      assert(Status >= ExecutionStatus::CompletedSuccess &&
+             "Can only be called after program completion.");
+      if (Status >= ExecutionStatus::ExecutionFailure)
+        return (int)ProgramReturnCode::Failure;
+      int R = ReturnCode.getValueOr((int)ProgramReturnCode::Success);
+#ifdef _WIN32
+      // Exit status should not be negative on Win32, unless abnormal
+      // termination. Once abnormal termination was caught, negative status
+      // should not be propagated.
+      if (R < 0)
+        return (int)ProgramReturnCode::Failure;
+#endif
+      return R;
+    }
+    void clear() { *this = ProcessInfo(); }
+    bool hasRet(ProgramReturnCode RC) const {
+        if (!ReturnCode.hasValue())
+            return false;
+        return ReturnCode.getValue() == (int)RC;
+    }
   };
 
   /// Find the first executable file \p Name in \p Paths.
   ///
   /// This does not perform hashing as a shell would but instead stats each PATH
   /// entry individually so should generally be avoided. Core LLVM library
   /// functions and options should instead require fully specified paths.
   ///
#endif
 
   /// This function executes the program using the arguments provided.  The
   /// invoked program will inherit the stdin, stdout, and stderr file
   /// descriptors, the environment and other configuration settings of the
   /// invoking program.
   /// This function waits for the program to finish, so should be avoided in
   /// library functions that aren't expected to block. Consider using
   /// ExecuteNoWait() instead.
-  /// \returns an integer result code indicating the status of the program.
-  /// A zero or positive value indicates the result code of the program.
-  /// -1 indicates failure to execute
-  /// -2 indicates a crash during execution or timeout
-  int ExecuteAndWait(
+  /// \returns true if the program executed with no errors.
+  /// If false, the program execution failed, check PI->Status, PI->ReturnCode
+  /// or PI->ErrMsg for further error information.
+  bool ExecuteAndWait(
       StringRef Program, ///< Path of the program to be executed. It is
       ///< presumed this is the result of the findProgramByName method.
       ArrayRef<StringRef> Args, ///< An array of strings that are passed to the
       ///< program.  The first element should be the name of the program.
       ///< The array should **not** be terminated by an empty StringRef.
       Optional<ArrayRef<StringRef>> Env = None, ///< An optional vector of
       ///< strings to use for the program's environment. If not provided, the
       ///< current program's environment will be used.  If specified, the
bool ExecuteAndWait(
       ///< of time to wait for the child process to exit. If the time
       ///< expires, the child is killed and this call returns. If zero,
       ///< this function will wait until the child finishes or forever if
       ///< it doesn't.
       unsigned MemoryLimit = 0, ///< If non-zero, this specifies max. amount
       ///< of memory can be allocated by process. If memory usage will be
       ///< higher limit, the child is killed and this call returns. If zero
       ///< - no memory limit.
-      std::string *ErrMsg = nullptr, ///< If non-zero, provides a pointer to a
-      ///< string instance in which error messages will be returned. If the
-      ///< string is non-empty upon return an error occurred while invoking the
-      ///< program.
-      bool *ExecutionFailed = nullptr);
+      ProcessInfo *PI = nullptr);
 
   /// Similar to ExecuteAndWait, but returns immediately.
-  /// @returns The \see ProcessInfo of the newly launced process.
+  /// @returns The \see ProcessInfo of the newly launched process.
   /// \note On Microsoft Windows systems, users will need to either call
   /// \see Wait until the process finished execution or win32 CloseHandle() API
   /// on ProcessInfo.ProcessHandle to avoid memory leaks.
   ProcessInfo ExecuteNoWait(StringRef Program, ArrayRef<StringRef> Args,
                             Optional<ArrayRef<StringRef>> Env,
                             ArrayRef<Optional<StringRef>> Redirects = {},
-                            unsigned MemoryLimit = 0,
-                            std::string *ErrMsg = nullptr,
-                            bool *ExecutionFailed = nullptr);
+                            unsigned MemoryLimit = 0);
 
   /// Return true if the given arguments fit within system-specific
   /// argument length limits.
   bool commandLineFitsWithinSystemLimits(StringRef Program,
                                          ArrayRef<StringRef> Args);
 
   /// Return true if the given arguments fit within system-specific
   /// argument length limits.
#endif
   /// should be changed as soon as binutils fix this to support UTF16 on mingw.
   ///
   /// \returns non-zero error_code if failed
   std::error_code
   writeFileWithEncoding(StringRef FileName, StringRef Contents,
                         WindowsEncodingMethod Encoding = WEM_UTF8);
 
   /// This function waits for the process specified by \p PI to finish.
-  /// \returns A \see ProcessInfo struct with Pid set to:
-  /// \li The process id of the child process if the child process has changed
-  /// state.
-  /// \li 0 if the child process has not changed state.
-  /// \note Users of this function should always check the ReturnCode member of
-  /// the \see ProcessInfo returned from this function.
-  ProcessInfo Wait(
-      const ProcessInfo &PI, ///< The child process that should be waited on.
+  /// \returns true if the program executed with no errors.
+  /// If false, the program execution failed, check PI.Status, PI.ReturnCode
+  /// or PI.ErrMsg for further error information.
+  bool Wait(
+      ProcessInfo &PI,        ///< The child process that should be waited on.
       unsigned SecondsToWait, ///< If non-zero, this specifies the amount of
       ///< time to wait for the child process to exit. If the time expires, the
       ///< child is killed and this function returns. If zero, this function
       ///< will perform a non-blocking wait on the child process.
-      bool WaitUntilTerminates, ///< If true, ignores \p SecondsToWait and waits
+      bool WaitUntilTerminates ///< If true, ignores \p SecondsToWait and waits
       ///< until child has terminated.
-      std::string *ErrMsg = nullptr ///< If non-zero, provides a pointer to a
-      ///< string instance in which error messages will be returned. If the
-      ///< string is non-empty upon return an error occurred while invoking the
-      ///< program.
-      );
 
+  /// This function waits for some/all processes specified by \p PIArray to
+  /// finish.
+  /// \returns Whether the wait fulfilled all conditions.
+  /// \note Users of this function should always check the ReturnCode member of
+  /// all ProcessInfo(s) passed in PIArray.
+  bool WaitMany(
+      MutableArrayRef<ProcessInfo *> PIArray,
+      ///< The children processes that should be waited on.
+      bool WaitOnAll, ///< If set, succeeds only when all processes complete.
+      ///< If not set, the function will succeed on the first completed process.
+      ///< Ignored if \p PIArray contains only one element.
+      unsigned SecondsToWait, ///< If non-zero, this specifies the amount of
+      ///< time to wait for the children processes to exit. If the time expires,
+      ///< all children are killed and this function returns. If zero, this
+      ///< function will perform a non-blocking wait on all children processes.
+      bool WaitUntilProcessTerminates ///< If true, ignores \p SecondsToWait and
+      ///< waits until at least one child has terminated, depending on
+      ///< \p WaitOnAll.
+  );
+
+  /// This function kills the processes specified by \p PIArray, and waits
+  /// until all resources are closed, including I/O streams. \returns Whether
+  /// all processes were successfully closed/killed/terminated. \note If the
+  /// function fails, users must check elements in \p PIArray to find out which
+  /// process failed.
+  bool Terminate(MutableArrayRef<ProcessInfo *> PIArray);
+
 #if defined(_WIN32)
   /// Given a list of command line arguments, quote and escape them as necessary
   /// to build a single flat command line appropriate for calling CreateProcess
   /// on
   /// Windows.
   std::string flattenWindowsCommandLine(ArrayRef<StringRef> Args);
 #endif
   }
 }
 
 #endif

llvm/trunk/lib/Support/GraphWriter.cpp

std::string llvm::createGraphFilename(const Twine &Name, int &FD) {
   }
 
   errs() << "Writing '" << Filename << "'... ";
   return Filename.str();
 }
 
 // Execute the graph viewer. Return true if there were errors.
 static bool ExecGraphViewer(StringRef ExecPath, std::vector<StringRef> &args,
-                            StringRef Filename, bool wait,
-                            std::string &ErrMsg) {
+                            StringRef Filename, bool wait) {
+  sys::ProcessInfo PI;
   if (wait) {
-    if (sys::ExecuteAndWait(ExecPath, args, None, {}, 0, 0, &ErrMsg)) {
-      errs() << "Error: " << ErrMsg << "\n";
+    if (!sys::ExecuteAndWait(ExecPath, args, None, {}, 0, 0, &PI)) {
+      errs() << "Error: " << PI.ErrMsg.getValueOr("unknown error") << "\n";
       return true;
     }
     sys::fs::remove(Filename);
     errs() << " done. \n";
   } else {
-    sys::ExecuteNoWait(ExecPath, args, None, {}, 0, &ErrMsg);
+    sys::ExecuteNoWait(ExecPath, args, None);
     errs() << "Remember to erase graph file: " << Filename << "\n";
   }
   return false;
 }
 
 namespace {
 
 struct GraphSession {
case GraphProgram::CIRCO:
     return "circo";
   }
   llvm_unreachable("bad kind");
 }
 
 bool llvm::DisplayGraph(StringRef FilenameRef, bool wait,
                         GraphProgram::Name program) {
   std::string Filename = FilenameRef;
-  std::string ErrMsg;
   std::string ViewerPath;
   GraphSession S;
 
 #ifdef __APPLE__
   wait &= !ViewBackground;
   if (S.TryFindProgram("open", ViewerPath)) {
     std::vector<StringRef> args;
     args.push_back(ViewerPath);
     if (wait)
       args.push_back("-W");
     args.push_back(Filename);
     errs() << "Trying 'open' program... ";
-    if (!ExecGraphViewer(ViewerPath, args, Filename, wait, ErrMsg))
+    if (!ExecGraphViewer(ViewerPath, args, Filename, wait))
       return false;
   }
 #endif
   if (S.TryFindProgram("xdg-open", ViewerPath)) {
     std::vector<StringRef> args;
     args.push_back(ViewerPath);
     args.push_back(Filename);
     errs() << "Trying 'xdg-open' program... ";
-    if (!ExecGraphViewer(ViewerPath, args, Filename, wait, ErrMsg))
+    if (!ExecGraphViewer(ViewerPath, args, Filename, wait))
       return false;
   }
 
   // Graphviz
   if (S.TryFindProgram("Graphviz", ViewerPath)) {
     std::vector<StringRef> args;
     args.push_back(ViewerPath);
     args.push_back(Filename);
 
     errs() << "Running 'Graphviz' program... ";
-    return ExecGraphViewer(ViewerPath, args, Filename, wait, ErrMsg);
+    return ExecGraphViewer(ViewerPath, args, Filename, wait);
   }
 
   // xdot
   if (S.TryFindProgram("xdot|xdot.py", ViewerPath)) {
     std::vector<StringRef> args;
     args.push_back(ViewerPath);
     args.push_back(Filename);
 
     args.push_back("-f");
     args.push_back(getProgramName(program));
 
     errs() << "Running 'xdot.py' program... ";
-    return ExecGraphViewer(ViewerPath, args, Filename, wait, ErrMsg);
+    return ExecGraphViewer(ViewerPath, args, Filename, wait);
   }
 
   enum ViewerKind {
     VK_None,
     VK_OSXOpen,
     VK_XDGOpen,
     VK_Ghostview,
     VK_CmdStart
if (Viewer &&
     args.push_back("-Nfontname=Courier");
     args.push_back("-Gsize=7.5,10");
     args.push_back(Filename);
     args.push_back("-o");
     args.push_back(OutputFilename);
 
     errs() << "Running '" << GeneratorPath << "' program... ";
 
-    if (ExecGraphViewer(GeneratorPath, args, Filename, true, ErrMsg))
+    if (ExecGraphViewer(GeneratorPath, args, Filename, true))
       return true;
 
     // The lifetime of StartArg must include the call of ExecGraphViewer
     // because the args are passed as vector of char*.
     std::string StartArg;
 
     args.clear();
     args.push_back(ViewerPath);
case VK_CmdStart:
       args.push_back("/C");
       StartArg =
           (StringRef("start ") + (wait ? "/WAIT " : "") + OutputFilename).str();
       args.push_back(StartArg);
       break;
     case VK_None:
       llvm_unreachable("Invalid viewer");
     }
-
-    ErrMsg.clear();
-    return ExecGraphViewer(ViewerPath, args, OutputFilename, wait, ErrMsg);
+    return ExecGraphViewer(ViewerPath, args, OutputFilename, wait);
   }
 
   // dotty
   if (S.TryFindProgram("dotty", ViewerPath)) {
     std::vector<StringRef> args;
     args.push_back(ViewerPath);
     args.push_back(Filename);
 
 // Dotty spawns another app and doesn't wait until it returns
 #ifdef _WIN32
     wait = false;
 #endif
     errs() << "Running 'dotty' program... ";
-    return ExecGraphViewer(ViewerPath, args, Filename, wait, ErrMsg);
+    return ExecGraphViewer(ViewerPath, args, Filename, wait);
   }
 
   errs() << "Error: Couldn't find a usable graph viewer program:\n";
   errs() << S.LogBuffer << "\n";
   return true;
 }

llvm/trunk/lib/Support/Program.cpp

 //===----------------------------------------------------------------------===//
 //=== WARNING: Implementation here must contain only TRULY operating system
 //===          independent code.
 //===----------------------------------------------------------------------===//
 
 static bool Execute(ProcessInfo &PI, StringRef Program,
                     ArrayRef<StringRef> Args, Optional<ArrayRef<StringRef>> Env,
                     ArrayRef<Optional<StringRef>> Redirects,
-                    unsigned MemoryLimit, std::string *ErrMsg);
+                    unsigned MemoryLimit);
 
-int sys::ExecuteAndWait(StringRef Program, ArrayRef<StringRef> Args,
+bool sys::ExecuteAndWait(StringRef Program, ArrayRef<StringRef> Args,
                         Optional<ArrayRef<StringRef>> Env,
                         ArrayRef<Optional<StringRef>> Redirects,
                         unsigned SecondsToWait, unsigned MemoryLimit,
-                        std::string *ErrMsg, bool *ExecutionFailed) {
+                        ProcessInfo *PI) {
   assert(Redirects.empty() || Redirects.size() == 3);
-  ProcessInfo PI;
-  if (Execute(PI, Program, Args, Env, Redirects, MemoryLimit, ErrMsg)) {
-    if (ExecutionFailed)
-      *ExecutionFailed = false;
-    ProcessInfo Result = Wait(
-        PI, SecondsToWait, /*WaitUntilTerminates=*/SecondsToWait == 0, ErrMsg);
-    return Result.ReturnCode;
-  }
-
-  if (ExecutionFailed)
-    *ExecutionFailed = true;
+  ProcessInfo LocalPI;
+  if (!PI)
+    PI = &LocalPI;
+  bool Ret = Execute(*PI, Program, Args, Env, Redirects, MemoryLimit);
+  if (!Ret)
+    return false;
 
-  return -1;
+  return Wait(*PI, SecondsToWait, /*WaitUntilTerminates=*/SecondsToWait == 0);
 }
 
 ProcessInfo sys::ExecuteNoWait(StringRef Program, ArrayRef<StringRef> Args,
                                Optional<ArrayRef<StringRef>> Env,
                                ArrayRef<Optional<StringRef>> Redirects,
-                               unsigned MemoryLimit, std::string *ErrMsg,
-                               bool *ExecutionFailed) {
+                               unsigned MemoryLimit) {
   assert(Redirects.empty() || Redirects.size() == 3);
   ProcessInfo PI;
-  if (ExecutionFailed)
-    *ExecutionFailed = false;
-  if (!Execute(PI, Program, Args, Env, Redirects, MemoryLimit, ErrMsg))
-    if (ExecutionFailed)
-      *ExecutionFailed = true;
-
+  Execute(PI, Program, Args, Env, Redirects, MemoryLimit);
   return PI;
 }
 
 bool sys::commandLineFitsWithinSystemLimits(StringRef Program,
                                             ArrayRef<const char *> Args) {
   SmallVector<StringRef, 8> StringRefArgs;
   StringRefArgs.reserve(Args.size());
   for (const char *A : Args)
     StringRefArgs.emplace_back(A);
   return commandLineFitsWithinSystemLimits(Program, StringRefArgs);
 }
 
+bool sys::Wait(ProcessInfo &PI, unsigned SecondsToWait,
+               bool WaitUntilChildTerminates) {
+  ProcessInfo *A[] = {&PI};
+  return WaitMany(makeMutableArrayRef(A, 1), true, SecondsToWait,
+                  WaitUntilChildTerminates);
+}
+
 // Include the platform-specific parts of this class.
 #ifdef LLVM_ON_UNIX
 #include "Unix/Program.inc"
 #endif
 #ifdef _WIN32
 #include "Windows/Program.inc"
 #endif

llvm/trunk/lib/Support/Windows/DynamicLibrary.inc

 }
 
 void *DynamicLibrary::HandleSet::DLOpen(const char *File, std::string *Err) {
   // Create the instance and return it to be the *Process* handle
   // simillar to dlopen(NULL, RTLD_LAZY|RTLD_GLOBAL)
   if (!File)
     return &(*OpenedHandles);
 
+  const char* E = nullptr;
+  HMODULE Handle = nullptr;
+
   SmallVector<wchar_t, MAX_PATH> FileUnicode;
   if (std::error_code ec = windows::UTF8ToUTF16(File, FileUnicode)) {
     SetLastError(ec.value());
-    MakeErrMsg(Err, std::string(File) + ": Can't convert to UTF-16");
-    return &DynamicLibrary::Invalid;
+    E = "Can't convert to UTF-16";
+  } else {
+    Handle = LoadLibraryW(FileUnicode.data());
+    if (Handle == NULL) {
+      E = "Can't open";
+    }
   }
 
-  HMODULE Handle = LoadLibraryW(FileUnicode.data());
-  if (Handle == NULL) {
-    MakeErrMsg(Err, std::string(File) + ": Can't open");
+  if (E) {
+    Optional<std::string> ErrMsg;
+    MakeErrMsg(ErrMsg, std::string(File) + ": " + E);
+    if (Err)
+      *Err = ErrMsg.getValue();
     return &DynamicLibrary::Invalid;
   }
-
   return reinterpret_cast<void*>(Handle);
 }
 
 static DynamicLibrary::HandleSet *IsOpenedHandlesInstance(void *Handle) {
   if (!OpenedHandles.isConstructed())
     return nullptr;
   DynamicLibrary::HandleSet &Inst = *OpenedHandles;
   return Handle == &Inst ? &Inst : nullptr;
static bool GetProcessModules(HANDLE H, DWORD &Bytes, HMODULE *Data = nullptr) {
   // MingW-32 don't have EnumProcessModulesEx.
   if (
 #ifdef _WIN64
       !EnumProcessModulesEx(H, Data, Bytes, &Bytes, LIST_MODULES_64BIT)
 #else
       !EnumProcessModules(H, Data, Bytes, &Bytes)
 #endif
      ) {
-    std::string Err;
-    if (MakeErrMsg(&Err, "EnumProcessModules failure"))
-      llvm::errs() << Err << "\n";
+    Optional<std::string> Err;
+    if (MakeErrMsg(Err, "EnumProcessModules failure"))
+      llvm::errs() << Err.getValue() << "\n";
     return false;
   }
   return true;
 }
 
 void *DynamicLibrary::HandleSet::DLSym(void *Handle, const char *Symbol) {
   HandleSet* HS = IsOpenedHandlesInstance(Handle);
   if (!HS)
     return (void *)uintptr_t(GetProcAddress((HMODULE)Handle, Symbol));
 
   // Could have done a dlclose on the *Process* handle
   if (!HS->Process)
     return nullptr;
 
-  // Trials indicate EnumProcessModulesEx is consistantly faster than using
+  // Trials indicate EnumProcessModulesEx is consistently faster than using
   // EnumerateLoadedModules64 or CreateToolhelp32Snapshot.
   //
   // | Handles | DbgHelp.dll | CreateSnapshot | EnumProcessModulesEx
   // |=========|=============|========================================
   // | 37      | 0.0000585 * | 0.0003031      | 0.0000152
   // | 1020    | 0.0026310 * | 0.0121598      | 0.0002683
   // | 2084    | 0.0149418 * | 0.0369936      | 0.0005610
   //

llvm/trunk/lib/Support/Windows/Process.inc

 const char *Process::ResetColor() {
   if (UseANSI) return "\033[0m";
   SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), defaultColors());
   return 0;
 }
 
 // Include GetLastError() in a fatal error message.
 static void ReportLastErrorFatal(const char *Msg) {
-  std::string ErrMsg;
-  MakeErrMsg(&ErrMsg, Msg);
-  report_fatal_error(ErrMsg);
+  llvm::Optional<std::string> ErrMsg;
+  MakeErrMsg(ErrMsg, Msg);
+  report_fatal_error(ErrMsg.getValue());
 }
 
 unsigned Process::GetRandomNumber() {
   HCRYPTPROV HCPC;
   if (!::CryptAcquireContextW(&HCPC, NULL, NULL, PROV_RSA_FULL,
                               CRYPT_VERIFYCONTEXT))
     ReportLastErrorFatal("Could not acquire a cryptographic context");
 
   ScopedCryptContext CryptoProvider(HCPC);
   unsigned Ret;
   if (!::CryptGenRandom(CryptoProvider, sizeof(Ret),
                         reinterpret_cast<BYTE *>(&Ret)))
     ReportLastErrorFatal("Could not generate a random number");
   return Ret;
 }

llvm/trunk/lib/Support/Windows/Program.inc

 
 //===----------------------------------------------------------------------===//
 //=== WARNING: Implementation here must contain only Win32 specific code
 //===          and must not be UNIX code
 //===----------------------------------------------------------------------===//
 
 namespace llvm {
 
-ProcessInfo::ProcessInfo() : Pid(0), Process(0), ReturnCode(0) {}
+ProcessInfo::ProcessInfo() : Pid(), Process(), Status(ExecutionStatus::NotStarted) {}
 
 ErrorOr<std::string> sys::findProgramByName(StringRef Name,
                                             ArrayRef<StringRef> Paths) {
   assert(!Name.empty() && "Must have a name!");
 
   if (Name.find_first_of("/\\") != StringRef::npos)
     return std::string(Name);
 
ErrorOr<std::string> sys::findProgramByName(StringRef Name,
   if (std::error_code EC =
           windows::UTF16ToUTF8(U16Result.data(), U16Result.size(), U8Result))
     return EC;
 
   return std::string(U8Result.begin(), U8Result.end());
 }
 
 static HANDLE RedirectIO(Optional<StringRef> Path, int fd,
-                         std::string *ErrMsg) {
+                         Optional<std::string>& ErrMsg) {
   HANDLE h;
   if (!Path) {
     if (!DuplicateHandle(GetCurrentProcess(), (HANDLE)_get_osfhandle(fd),
                          GetCurrentProcess(), &h,
                          0, TRUE, DUPLICATE_SAME_ACCESS))
       return INVALID_HANDLE_VALUE;
     return h;
   }
static HANDLE RedirectIO(Optional<StringRef> Path, int fd,
   } else {
     if (path::widenPath(fname, fnameUnicode))
       return INVALID_HANDLE_VALUE;
   }
   h = CreateFileW(fnameUnicode.data(), fd ? GENERIC_WRITE : GENERIC_READ,
                   FILE_SHARE_READ, &sa, fd == 0 ? OPEN_EXISTING : CREATE_ALWAYS,
                   FILE_ATTRIBUTE_NORMAL, NULL);
   if (h == INVALID_HANDLE_VALUE) {
-    MakeErrMsg(ErrMsg, fname + ": Can't open file for " +
-        (fd ? "input" : "output"));
+    MakeErrMsg(ErrMsg,
+               fname + ": Can't open file for " + (fd ? "input" : "output"));
   }
-
   return h;
 }
 
 }
 
 static bool Execute(ProcessInfo &PI, StringRef Program,
                     ArrayRef<StringRef> Args, Optional<ArrayRef<StringRef>> Env,
                     ArrayRef<Optional<StringRef>> Redirects,
-                    unsigned MemoryLimit, std::string *ErrMsg) {
+                    unsigned MemoryLimit) {
+  assert(PI.notStarted());
   if (!sys::fs::can_execute(Program)) {
-    if (ErrMsg)
-      *ErrMsg = "program not executable";
+    PI.Status = sys::ExecutionStatus::ExecutionFailure;
+    PI.ErrMsg.emplace("program not executable or not found");
     return false;
   }
 
   // can_execute may succeed by looking at Program + ".exe". CreateProcessW
   // will implicitly add the .exe if we provide a command line without an
   // executable path, but since we use an explicit executable, we have to add
   // ".exe" ourselves.
   SmallString<64> ProgramStorage;
static bool Execute(ProcessInfo &PI, StringRef Program,
   if (Env) {
     // An environment block consists of a null-terminated block of
     // null-terminated strings. Convert the array of environment variables to
     // an environment block by concatenating them.
     for (const auto E : *Env) {
       SmallVector<wchar_t, MAX_PATH> EnvString;
       if (std::error_code ec = windows::UTF8ToUTF16(E, EnvString)) {
         SetLastError(ec.value());
-        MakeErrMsg(ErrMsg, "Unable to convert environment variable to UTF-16");
+        PI.Status = sys::ExecutionStatus::ExecutionFailure;
+        MakeErrMsg(PI.ErrMsg, "unable to convert environment variable to UTF-16");
         return false;
       }
 
       EnvBlock.insert(EnvBlock.end(), EnvString.begin(), EnvString.end());
       EnvBlock.push_back(0);
     }
     EnvBlock.push_back(0);
   }
 
   // Create a child process.
   STARTUPINFOW si;
   memset(&si, 0, sizeof(si));
   si.cb = sizeof(si);
   si.hStdInput = INVALID_HANDLE_VALUE;
   si.hStdOutput = INVALID_HANDLE_VALUE;
   si.hStdError = INVALID_HANDLE_VALUE;
 
   if (!Redirects.empty()) {
     si.dwFlags = STARTF_USESTDHANDLES;
 
-    si.hStdInput = RedirectIO(Redirects[0], 0, ErrMsg);
+    si.hStdInput = RedirectIO(Redirects[0], 0, PI.ErrMsg);
     if (si.hStdInput == INVALID_HANDLE_VALUE) {
-      MakeErrMsg(ErrMsg, "can't redirect stdin");
+      PI.Status = sys::ExecutionStatus::ExecutionFailure;
+      MakeErrMsg(PI.ErrMsg, "can't redirect stdin");
       return false;
     }
-    si.hStdOutput = RedirectIO(Redirects[1], 1, ErrMsg);
+    si.hStdOutput = RedirectIO(Redirects[1], 1, PI.ErrMsg);
     if (si.hStdOutput == INVALID_HANDLE_VALUE) {
       CloseHandle(si.hStdInput);
-      MakeErrMsg(ErrMsg, "can't redirect stdout");
+      PI.Status = sys::ExecutionStatus::ExecutionFailure;
+      MakeErrMsg(PI.ErrMsg, "can't redirect stdout");
       return false;
     }
     if (Redirects[1] && Redirects[2] && *Redirects[1] == *Redirects[2]) {
       // If stdout and stderr should go to the same place, redirect stderr
       // to the handle already open for stdout.
       if (!DuplicateHandle(GetCurrentProcess(), si.hStdOutput,
                            GetCurrentProcess(), &si.hStdError,
                            0, TRUE, DUPLICATE_SAME_ACCESS)) {
         CloseHandle(si.hStdInput);
         CloseHandle(si.hStdOutput);
-        MakeErrMsg(ErrMsg, "can't dup stderr to stdout");
+        PI.Status = sys::ExecutionStatus::ExecutionFailure;
+        MakeErrMsg(PI.ErrMsg, "can't dup stderr to stdout");
         return false;
       }
     } else {
       // Just redirect stderr
-      si.hStdError = RedirectIO(Redirects[2], 2, ErrMsg);
+      si.hStdError = RedirectIO(Redirects[2], 2, PI.ErrMsg);
       if (si.hStdError == INVALID_HANDLE_VALUE) {
         CloseHandle(si.hStdInput);
         CloseHandle(si.hStdOutput);
-        MakeErrMsg(ErrMsg, "can't redirect stderr");
+        PI.Status = sys::ExecutionStatus::ExecutionFailure;
+        MakeErrMsg(PI.ErrMsg, "can't redirect stderr");
         return false;
       }
     }
   }
 
   PROCESS_INFORMATION pi;
   memset(&pi, 0, sizeof(pi));
 
   fflush(stdout);
   fflush(stderr);
 
   SmallVector<wchar_t, MAX_PATH> ProgramUtf16;
   if (std::error_code ec = path::widenPath(Program, ProgramUtf16)) {
     SetLastError(ec.value());
-    MakeErrMsg(ErrMsg,
-               std::string("Unable to convert application name to UTF-16"));
+    PI.Status = sys::ExecutionStatus::ExecutionFailure;
+    MakeErrMsg(PI.ErrMsg, "unable to convert application name to UTF-16");
     return false;
   }
 
   SmallVector<wchar_t, MAX_PATH> CommandUtf16;
   if (std::error_code ec = windows::UTF8ToUTF16(Command, CommandUtf16)) {
     SetLastError(ec.value());
-    MakeErrMsg(ErrMsg,
-               std::string("Unable to convert command-line to UTF-16"));
+    PI.Status = sys::ExecutionStatus::ExecutionFailure;
+    MakeErrMsg(PI.ErrMsg,"unable to convert command-line to UTF-16");
     return false;
   }
 
   BOOL rc = CreateProcessW(ProgramUtf16.data(), CommandUtf16.data(), 0, 0,
                            TRUE, CREATE_UNICODE_ENVIRONMENT,
                            EnvBlock.empty() ? 0 : EnvBlock.data(), 0, &si,
                            &pi);
   DWORD err = GetLastError();
 
   // Regardless of whether the process got created or not, we are done with
   // the handles we created for it to inherit.
   CloseHandle(si.hStdInput);
   CloseHandle(si.hStdOutput);
   CloseHandle(si.hStdError);
 
   // Now return an error if the process didn't get created.
   if (!rc) {
     SetLastError(err);
-    MakeErrMsg(ErrMsg, std::string("Couldn't execute program '") +
-               Program.str() + "'");
+    PI.Status = sys::ExecutionStatus::ExecutionFailure;
+    MakeErrMsg(PI.ErrMsg,
+               std::string("couldn't execute program '") + Program.str() + "'");
     return false;
   }
 
   PI.Pid = pi.dwProcessId;
   PI.Process = pi.hProcess;
 
   // Make sure these get closed no matter what.
   ScopedCommonHandle hThread(pi.hThread);
if (hJob) {
       if (SetInformationJobObject(hJob, JobObjectExtendedLimitInformation,
                                   &jeli, sizeof(jeli))) {
         if (AssignProcessToJobObject(hJob, pi.hProcess))
           success = true;
       }
     }
     if (!success) {
       SetLastError(GetLastError());
-      MakeErrMsg(ErrMsg, std::string("Unable to set memory limit"));
+      PI.Status = sys::ExecutionStatus::ExecutionFailure;
+      MakeErrMsg(PI.ErrMsg, "unable to set memory limit");
       TerminateProcess(pi.hProcess, 1);
       WaitForSingleObject(pi.hProcess, INFINITE);
+      PI.Pid = procid_t();
+      PI.Process = process_t();
       return false;
     }
   }
 
+  PI.Status = sys::ExecutionStatus::StartedAndExecuting;
   return true;
 }
 
 static bool argNeedsQuotes(StringRef Arg) {
   if (Arg.empty())
     return true;
   return StringRef::npos != Arg.find_first_of("\t \"&\'()*<>\\`^|");
 }
while (!Arg.empty()) {
     // Drop all the backslashes, plus the following character.
     Arg = Arg.drop_front(FirstNonBackslash + 1);
   }
 
   Result.push_back('"');
   return Result;
 }
 
+bool CompleteProcess(sys::ProcessInfo &PI) {
+  DWORD status;
+  BOOL rc = GetExitCodeProcess(PI.Process, &status);
+  if (rc && status == STILL_ACTIVE)
+    return true;
+
+  DWORD err = GetLastError();
+  if (err != ERROR_INVALID_HANDLE) {
+    CloseHandle(PI.Process);
+  }
+  PI.Pid = procid_t();
+  PI.Process = process_t();
+
+  if (!rc) {
+    SetLastError(err);
+
+    // indicates a crash or timeout as opposed to failure to execute.
+    PI.Status = ExecutionStatus::Crashed;
+    MakeErrMsg(PI.ErrMsg, "Failed getting status for program");
+    return false;
+  }
+
+  if (!status) {
+    PI.ReturnCode = 0;
+    PI.Status = ExecutionStatus::CompletedSuccess;
+    return true;
+  }
+
+  // Pass 10(Warning) and 11(Error) to the callee as negative value.
+  if ((status & 0xBFFF0000U) == 0x80000000U)
+    PI.ReturnCode = static_cast<int>(status);
+  else if (status & 0xFF)
+    PI.ReturnCode = status & 0x7FFFFFFF;
+  else
+    PI.ReturnCode = 1;
+
+  PI.Status = ExecutionStatus::CompletedWithError;
+  return true;
+}
+
 namespace llvm {
 std::string sys::flattenWindowsCommandLine(ArrayRef<StringRef> Args) {
   std::string Command;
   for (StringRef Arg : Args) {
     if (argNeedsQuotes(Arg))
       Command += quoteSingleArg(Arg);
     else
       Command += Arg;
 
     Command.push_back(' ');
   }
 
   return Command;
 }
 
-ProcessInfo sys::Wait(const ProcessInfo &PI, unsigned SecondsToWait,
-                      bool WaitUntilChildTerminates, std::string *ErrMsg) {
-  assert(PI.Pid && "invalid pid to wait on, process not started?");
-  assert((PI.Process && PI.Process != INVALID_HANDLE_VALUE) &&
-         "invalid process handle to wait on, process not started?");
+bool sys::WaitMany(MutableArrayRef<ProcessInfo *> PIArray, bool WaitOnAll,
+                   unsigned SecondsToWait, bool WaitUntilProcessTerminates) {
   DWORD milliSecondsToWait = 0;
-  if (WaitUntilChildTerminates)
+  if (WaitUntilProcessTerminates)
     milliSecondsToWait = INFINITE;
   else if (SecondsToWait > 0)
     milliSecondsToWait = SecondsToWait * 1000;
 
-  ProcessInfo WaitResult = PI;
-  DWORD WaitStatus = WaitForSingleObject(PI.Process, milliSecondsToWait);
-  if (WaitStatus == WAIT_TIMEOUT) {
-    if (SecondsToWait) {
-      if (!TerminateProcess(PI.Process, 1)) {
-        if (ErrMsg)
-          MakeErrMsg(ErrMsg, "Failed to terminate timed-out program");
+  SmallVector<HANDLE, 16> Handles;
+  Handles.reserve(PIArray.size());
 
-        // -2 indicates a crash or timeout as opposed to failure to execute.
-        WaitResult.ReturnCode = -2;
-        CloseHandle(PI.Process);
-        return WaitResult;
-      }
-      WaitForSingleObject(PI.Process, INFINITE);
-      CloseHandle(PI.Process);
-    } else {
-      // Non-blocking wait.
-      return ProcessInfo();
+  for (auto &PI : PIArray) {
+    if (!PI->running())
+      continue;
+    assert(PI->Pid);
+    assert(PI->Process && PI->Process != INVALID_HANDLE_VALUE);
+    Handles.push_back(PI->Process);
+
+  if (Handles.empty())
+    return false;
+
+  if (Handles.size() == 1)
+    WaitOnAll = true;
+
+  DWORD WaitStatus = WaitForMultipleObjects(Handles.size(), Handles.begin(),
+                                            WaitOnAll, milliSecondsToWait);
+
+  if (WaitStatus >= WAIT_OBJECT_0 &&
+      WaitStatus < (WAIT_OBJECT_0 + Handles.size())) {
+    // success
+    if (WaitStatus == WAIT_TIMEOUT) {
+      if (!SecondsToWait)
+        return true; // Non-blocking wait
+      Terminate(PIArray);
+    }
+    else
+    {
+      for (auto &PI : PIArray) {
+        if (!PI->running())
+          continue;
+        MakeErrMsg(PI->ErrMsg, "Failed to wait for processes");
+      }
     }
+    return false;
   }
 
-  // Get its exit status.
-  DWORD status;
-  BOOL rc = GetExitCodeProcess(PI.Process, &status);
-  DWORD err = GetLastError();
-  if (err != ERROR_INVALID_HANDLE)
-    CloseHandle(PI.Process);
+  bool Success = true;
+  // Finish collecting information from completed processes
+  for (auto &PI : PIArray) {
+    if (!PI->running())
+      continue;
 
-  if (!rc) {
-    SetLastError(err);
-    if (ErrMsg)
-      MakeErrMsg(ErrMsg, "Failed getting status for program");
+    if (!CompleteProcess(*PI))
+      Success = false;
+  }
+  return Success;
+}
+
+bool sys::Terminate(MutableArrayRef<ProcessInfo *> PIArray) {
+  bool AllSuccess = true;
 
-    // -2 indicates a crash or timeout as opposed to failure to execute.
-    WaitResult.ReturnCode = -2;
-    return WaitResult;
+  SmallVector<HANDLE, 16> Handles;
+  Handles.reserve(PIArray.size());
+
+  // First, terminate all process asynchronously to give them time to close the
+  // file streams
+  for (auto &PI : PIArray) {
+    if (!PI->running())
+      continue;
+
+    if (!TerminateProcess(PI->Process, (int)ProgramReturnCode::Failure)) {
+      // Failure - the process might have finished in the meanwhile
+      if (!CompleteProcess(*PI))
+        AllSuccess = false;
+      continue;
+    }
+    Handles.push_back(PI->Process);
   }
 
-  if (!status)
-    return WaitResult;
+  if (Handles.size() == 0)
+    return false;
 
-  // Pass 10(Warning) and 11(Error) to the callee as negative value.
-  if ((status & 0xBFFF0000U) == 0x80000000U)
-    WaitResult.ReturnCode = static_cast<int>(status);
-  else if (status & 0xFF)
-    WaitResult.ReturnCode = status & 0x7FFFFFFF;
-  else
-    WaitResult.ReturnCode = 1;
+  WaitForMultipleObjects(Handles.size(), Handles.begin(), TRUE, INFINITE);
 
-  return WaitResult;
+  for (auto &PI : PIArray) {
+    if (!PI->running())
+      continue;
+
+    // Collect the process' return code
+    if (!CompleteProcess(*PI))
+      AllSuccess = false;
+
+    PI->Status = sys::ExecutionStatus::Killed;
+  }
+  return AllSuccess;
 }
 
 std::error_code sys::ChangeStdinToBinary() {
   int result = _setmode(_fileno(stdin), _O_BINARY);
   if (result == -1)
     return std::error_code(errno, std::generic_category());
   return std::error_code();
 }

llvm/trunk/lib/Support/Windows/WindowsSupport.h

inline bool RunningWindows8OrGreater() {
   Mask = VerSetConditionMask(Mask, VER_MINORVERSION, VER_GREATER_EQUAL);
   Mask = VerSetConditionMask(Mask, VER_SERVICEPACKMAJOR, VER_GREATER_EQUAL);
 
   return VerifyVersionInfoW(&osvi, VER_MAJORVERSION | VER_MINORVERSION |
                                        VER_SERVICEPACKMAJOR,
                             Mask) != FALSE;
 }
 
-inline bool MakeErrMsg(std::string *ErrMsg, const std::string &prefix) {
-  if (!ErrMsg)
-    return true;
+inline bool MakeErrMsg(llvm::Optional<std::string> &ErrMsg, const std::string &prefix) {
   char *buffer = NULL;
   DWORD LastError = GetLastError();
   DWORD R = FormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER |
                                FORMAT_MESSAGE_FROM_SYSTEM |
                                FORMAT_MESSAGE_MAX_WIDTH_MASK,
                            NULL, LastError, 0, (LPSTR)&buffer, 1, NULL);
   if (R)
-    *ErrMsg = prefix + ": " + buffer;
+    ErrMsg.emplace(prefix + ": " + buffer);
   else
-    *ErrMsg = prefix + ": Unknown error";
+    ErrMsg.emplace(prefix + ": Unknown error");
   *ErrMsg += " (0x" + llvm::utohexstr(LastError) + ")";
 
   LocalFree(buffer);
   return R != 0;
 }
 
 template <typename HandleTraits>
 class ScopedHandle {

llvm/trunk/tools/bugpoint/ExecutionDriver.cpp

Expected<std::string> BugDriver::executeProgram(const Module &Program,
   }
   OutputFile = UniqueFile.str();
 
   // Figure out which shared objects to run, if any.
   std::vector<std::string> SharedObjs(AdditionalSOs);
   if (!SharedObj.empty())
     SharedObjs.push_back(SharedObj);
 
-  Expected<int> RetVal = AI->ExecuteProgram(BitcodeFile, InputArgv, InputFile,
-                                            OutputFile, AdditionalLinkerArgs,
-                                            SharedObjs, Timeout, MemoryLimit);
-  if (Error E = RetVal.takeError())
+  Expected<sys::ProcessInfo> R = AI->ExecuteProgram(
+      BitcodeFile, InputArgv, InputFile, OutputFile, AdditionalLinkerArgs,
+  if (Error E = R.takeError())
     return std::move(E);
 
-  if (*RetVal == -1) {
+  if (R->killed()) {
     errs() << "<timeout>";
     static bool FirstTimeout = true;
     if (FirstTimeout) {
       outs()
           << "\n"
              "*** Program execution timed out!  This mechanism is designed to "
              "handle\n"
              "    programs stuck in infinite loops gracefully.  The -timeout "
              "option\n"
              "    can be used to change the timeout threshold or disable it "
              "completely\n"
              "    (with -timeout=0).  This message is only displayed once.\n";
       FirstTimeout = false;
     }
   }
 
   if (AppendProgramExitCode) {
     std::ofstream outFile(OutputFile.c_str(), std::ios_base::app);
-    outFile << "exit " << *RetVal << '\n';
+    outFile << "exit " << R->ErrMsg.getValueOr("unknown error") << '\n';
     outFile.close();
   }
 
   // Return the filename we captured the output to.
   return OutputFile;
 }
 
 /// Used to create reference output with the "safe" backend, if reference output

llvm/trunk/tools/bugpoint/OptimizerDriver.cpp

bool BugDriver::runPasses(Module &Program,
 
   Optional<StringRef> Redirects[3] = {None, None, None};
   // Redirect stdout and stderr to nowhere if SilencePasses is given.
   if (SilencePasses) {
     Redirects[1] = "";
     Redirects[2] = "";
   }
 
-  std::string ErrMsg;
-  int result = sys::ExecuteAndWait(Prog, Args, None, Redirects, Timeout,
-                                   MemoryLimit, &ErrMsg);
+  sys::ProcessInfo PI;
+  bool R = sys::ExecuteAndWait(Prog, Args, None, Redirects, Timeout,
+                               MemoryLimit, &PI);
 
   // If we are supposed to delete the bitcode file or if the passes crashed,
   // remove it now.  This may fail if the file was never created, but that's ok.
-  if (DeleteOutput || result != 0)
+  if (DeleteOutput || PI.completedError())
     sys::fs::remove(OutputFilename);
 
   if (!Quiet) {
-    if (result == 0)
+    if (PI.killed())
+      outs() << "Killed or timed out!\n";
+    else if (PI.crashed())
+      outs() << "Crashed: " << PI.ErrMsg.getValue() << "\n";
+    else if (PI.notCompletedOrFailed())
+      outs() << "Execute failed: " << PI.ErrMsg.getValue() << "\n";
+    else if (PI.completedError())
+      outs() << "Exited with error code '" << PI.ReturnCode.getValue() << "'\n";
+    else if (PI.completedSuccess())
       outs() << "Success!\n";
-    else if (result > 0)
-      outs() << "Exited with error code '" << result << "'\n";
-    else if (result < 0) {
-      if (result == -1)
-        outs() << "Execute failed: " << ErrMsg << "\n";
-      else
-        outs() << "Crashed: " << ErrMsg << "\n";
-    }
-    if (result & 0x01000000)
-      outs() << "Dumped core\n";
   }
 
   // Was the child successful?
-  return result != 0;
+  return !R;
 }
 
 std::unique_ptr<Module>
 BugDriver::runPassesOn(Module *M, const std::vector<std::string> &Passes,
                        unsigned NumExtraArgs, const char *const *ExtraArgs) {
   std::string BitcodeResult;
   if (runPasses(*M, Passes, BitcodeResult, false /*delete*/, true /*quiet*/,
                 NumExtraArgs, ExtraArgs)) {
     return nullptr;
   }
 
   std::unique_ptr<Module> Ret = parseInputFile(BitcodeResult, Context);
   if (!Ret) {
     errs() << getToolName() << ": Error reading bitcode file '" << BitcodeResult
            << "'!\n";
-    exit(1);
+    exit((int)sys::ProgramReturnCode::Failure);
   }
   sys::fs::remove(BitcodeResult);
   return Ret;
 }

llvm/trunk/tools/bugpoint/ToolRunner.h

 
 #ifndef LLVM_TOOLS_BUGPOINT_TOOLRUNNER_H
 #define LLVM_TOOLS_BUGPOINT_TOOLRUNNER_H
 
 #include "llvm/ADT/Triple.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/Program.h"
 #include "llvm/Support/SystemUtils.h"
 #include <exception>
 #include <vector>
 
 namespace llvm {
 
 extern cl::opt<bool> SaveTemps;
 extern Triple TargetTriple;
public:
 
   /// ExecuteProgram - Execute the program specified by "ProgramFile" (which is
   /// either a .s file, or a .c file, specified by FileType), with the specified
   /// arguments.  Standard input is specified with InputFile, and standard
   /// Output is captured to the specified OutputFile location.  The SharedLibs
   /// option specifies optional native shared objects that can be loaded into
   /// the program for execution.
   ///
-  Expected<int> ExecuteProgram(
+  Expected<sys::ProcessInfo> ExecuteProgram(
       const std::string &ProgramFile, const std::vector<std::string> &Args,
       FileType fileType, const std::string &InputFile,
       const std::string &OutputFile,
       const std::vector<std::string> &CCArgs = std::vector<std::string>(),
       unsigned Timeout = 0, unsigned MemoryLimit = 0);
 
   /// MakeSharedObject - This compiles the specified file (which is either a .c
   /// file or a .s file) into a shared object.
return make_error<StringError>(
         inconvertibleErrorCode());
   }
 
   /// ExecuteProgram - Run the specified bitcode file, emitting output to the
   /// specified filename.  This sets RetVal to the exit code of the program or
   /// returns an Error if a problem was encountered that prevented execution of
   /// the program.
   ///
-  virtual Expected<int> ExecuteProgram(
+  virtual Expected<sys::ProcessInfo> ExecuteProgram(
       const std::string &Bitcode, const std::vector<std::string> &Args,
       const std::string &InputFile, const std::string &OutputFile,
       const std::vector<std::string> &CCArgs = std::vector<std::string>(),
       const std::vector<std::string> &SharedLibs = std::vector<std::string>(),
       unsigned Timeout = 0, unsigned MemoryLimit = 0) = 0;
 };
 
 //===---------------------------------------------------------------------===//
public:
   ~LLC() override { delete cc; }
 
   /// compileProgram - Compile the specified program from bitcode to executable
   /// code.  This does not produce any output, it is only used when debugging
   /// the code generator.  Returns false if the code generator fails.
   Error compileProgram(const std::string &Bitcode, unsigned Timeout = 0,
                        unsigned MemoryLimit = 0) override;
 
-  Expected<int> ExecuteProgram(
+  Expected<sys::ProcessInfo> ExecuteProgram(
       const std::string &Bitcode, const std::vector<std::string> &Args,
       const std::string &InputFile, const std::string &OutputFile,
       const std::vector<std::string> &CCArgs = std::vector<std::string>(),
       const std::vector<std::string> &SharedLibs = std::vector<std::string>(),
       unsigned Timeout = 0, unsigned MemoryLimit = 0) override;
 
   Expected<CC::FileType> OutputCode(const std::string &Bitcode,
                                     std::string &OutFile, unsigned Timeout = 0,
                                     unsigned MemoryLimit = 0) override;
 };
 
 } // End llvm namespace
 
 #endif

llvm/trunk/tools/bugpoint/ToolRunner.cpp

 cl::opt<std::string>
     RemoteExtra("remote-extra-options",
                 cl::desc("Remote execution (rsh/ssh) extra options"));
 }
 
 /// RunProgramWithTimeout - This function provides an alternate interface
 /// to the sys::Program::ExecuteAndWait interface.
 /// @see sys::Program::ExecuteAndWait
-static int RunProgramWithTimeout(StringRef ProgramPath,
+static sys::ProcessInfo RunProgramWithTimeout(StringRef ProgramPath,
                                  ArrayRef<StringRef> Args, StringRef StdInFile,
                                  StringRef StdOutFile, StringRef StdErrFile,
                                  unsigned NumSeconds = 0,
-                                 unsigned MemoryLimit = 0,
-                                 std::string *ErrMsg = nullptr) {
+                                 unsigned MemoryLimit = 0) {
   Optional<StringRef> Redirects[3] = {StdInFile, StdOutFile, StdErrFile};
-  return sys::ExecuteAndWait(ProgramPath, Args, None, Redirects, NumSeconds,
-                             MemoryLimit, ErrMsg);
+  sys::ProcessInfo PI;
+  sys::ExecuteAndWait(ProgramPath, Args, None, Redirects, NumSeconds,
+                      MemoryLimit, &PI);
+  return PI;
 }
 
 /// RunProgramRemotelyWithTimeout - This function runs the given program
 /// remotely using the given remote client and the sys::Program::ExecuteAndWait.
 /// Returns the remote program exit code or reports a remote client error if it
 /// fails. Remote client is required to return 255 if it failed or program exit
 /// code otherwise.
 /// @see sys::Program::ExecuteAndWait
-static int RunProgramRemotelyWithTimeout(
+static sys::ProcessInfo RunProgramRemotelyWithTimeout(
     StringRef RemoteClientPath, ArrayRef<StringRef> Args, StringRef StdInFile,
     StringRef StdOutFile, StringRef StdErrFile, unsigned NumSeconds = 0,
     unsigned MemoryLimit = 0) {
   Optional<StringRef> Redirects[3] = {StdInFile, StdOutFile, StdErrFile};
+  sys::ProcessInfo PI;
 
   // Run the program remotely with the remote client
-  int ReturnCode = sys::ExecuteAndWait(RemoteClientPath, Args, None, Redirects,
-                                       NumSeconds, MemoryLimit);
+  bool R = sys::ExecuteAndWait(RemoteClientPath, Args, None, Redirects,
+                               NumSeconds, MemoryLimit, &PI);
 
   // Has the remote client fail?
-  if (255 == ReturnCode) {
+  if (!R || PI.notCompletedOrFailed() ||
+      (PI.completedError() && PI.hasRet(sys::ProgramReturnCode::BugpointClientFail))) {
     std::ostringstream OS;
     OS << "\nError running remote client:\n ";
     for (StringRef Arg : Args)
       OS << " " << Arg.str();
     OS << "\n";
 
     // The error message is in the output file, let's print it out from there.
     std::string StdOutFileName = StdOutFile.str();
     std::ifstream ErrorFile(StdOutFileName.c_str());
     if (ErrorFile) {
       std::copy(std::istreambuf_iterator<char>(ErrorFile),
                 std::istreambuf_iterator<char>(),
                 std::ostreambuf_iterator<char>(OS));
       ErrorFile.close();
     }
 
     errs() << OS.str();
   }
 
-  return ReturnCode;
+  return PI;
 }
 
 static Error ProcessFailure(StringRef ProgPath, ArrayRef<StringRef> Args,
                             unsigned Timeout = 0, unsigned MemoryLimit = 0) {
   std::ostringstream OS;
   OS << "\nError running tool:\n ";
   for (StringRef Arg : Args)
     OS << " " << Arg.str();
   OS << "\n";
 
   // Rerun the compiler, capturing any error messages to print them.
   SmallString<128> ErrorFilename;
   std::error_code EC = sys::fs::createTemporaryFile(
       "bugpoint.program_error_messages", "", ErrorFilename);
   if (EC) {
     errs() << "Error making unique filename: " << EC.message() << "\n";
-    exit(1);
+    exit((int)llvm::sys::ProgramReturnCode::Failure);
   }
 
   RunProgramWithTimeout(ProgPath, Args, "", ErrorFilename.str(),
                         ErrorFilename.str(), Timeout, MemoryLimit);
   // FIXME: check return code ?
 
   // Print out the error messages generated by CC if possible...
   std::ifstream ErrorFile(ErrorFilename.c_str());
public:
   LLI(const std::string &Path, const std::vector<std::string> *Args)
       : LLIPath(Path) {
     ToolArgs.clear();
     if (Args) {
       ToolArgs = *Args;
     }
   }
 
-  Expected<int> ExecuteProgram(
+  Expected<sys::ProcessInfo> ExecuteProgram(
       const std::string &Bitcode, const std::vector<std::string> &Args,
       const std::string &InputFile, const std::string &OutputFile,
       const std::vector<std::string> &CCArgs,
       const std::vector<std::string> &SharedLibs = std::vector<std::string>(),
       unsigned Timeout = 0, unsigned MemoryLimit = 0) override;
 };
 }
 
-Expected<int> LLI::ExecuteProgram(const std::string &Bitcode,
+Expected<sys::ProcessInfo> LLI::ExecuteProgram(const std::string &Bitcode,
                                   const std::vector<std::string> &Args,
                                   const std::string &InputFile,
                                   const std::string &OutputFile,
                                   const std::vector<std::string> &CCArgs,
                                   const std::vector<std::string> &SharedLibs,
                                   unsigned Timeout, unsigned MemoryLimit) {
   std::vector<StringRef> LLIArgs;
   LLIArgs.push_back(LLIPath.c_str());

 #elif defined(_WIN32)
 const char EXESuffix[] = "exe";
 #endif
 
 /// Prepend the path to the program being executed
 /// to \p ExeName, given the value of argv[0] and the address of main()
 /// itself. This allows us to find another LLVM tool if it is built in the same
 /// directory. An empty string is returned on error; note that this function
-/// just mainpulates the path and doesn't check for executability.
+/// just manipulates the path and doesn't check for executability.
 /// Find a named executable.
 static std::string PrependMainExecutablePath(const std::string &ExeName,
                                              const char *Argv0,
                                              void *MainAddr) {
   // Check the directory that the calling program is in.  We can do
   // this if ProgramPath contains at least one / character, indicating that it
   // is a relative path to the executable itself.
   std::string Main = sys::fs::getMainExecutable(Argv0, MainAddr);

 public:
   CustomCompiler(const std::string &CompilerCmd,
                  std::vector<std::string> CompArgs)
       : CompilerCommand(CompilerCmd), CompilerArgs(std::move(CompArgs)) {}
 
   Error compileProgram(const std::string &Bitcode, unsigned Timeout = 0,
                        unsigned MemoryLimit = 0) override;
 
-  Expected<int> ExecuteProgram(
+  Expected<sys::ProcessInfo> ExecuteProgram(
       const std::string &Bitcode, const std::vector<std::string> &Args,
       const std::string &InputFile, const std::string &OutputFile,
       const std::vector<std::string> &CCArgs = std::vector<std::string>(),
       const std::vector<std::string> &SharedLibs = std::vector<std::string>(),
       unsigned Timeout = 0, unsigned MemoryLimit = 0) override {
     return make_error<StringError>(
         "Execution not supported with -compile-custom",
         inconvertibleErrorCode());
Error CustomCompiler::compileProgram(const std::string &Bitcode,
   for (std::size_t i = 0; i < CompilerArgs.size(); ++i)
     ProgramArgs.push_back(CompilerArgs.at(i).c_str());
   ProgramArgs.push_back(Bitcode);
 
   // Add optional parameters to the running program from Argv
   for (unsigned i = 0, e = CompilerArgs.size(); i != e; ++i)
     ProgramArgs.push_back(CompilerArgs[i].c_str());
 
-  if (RunProgramWithTimeout(CompilerCommand, ProgramArgs, "", "", "", Timeout,
+  if (!RunProgramWithTimeout(CompilerCommand, ProgramArgs, "", "", "", Timeout,
                             MemoryLimit))
     return ProcessFailure(CompilerCommand, ProgramArgs, Timeout, MemoryLimit);
   return Error::success();
 }
 
 //===---------------------------------------------------------------------===//
 // Custom execution command implementation of AbstractIntepreter interface
 //
 // Allows using a custom command for executing the bitcode, thus allows,
 // for example, to invoke a cross compiler for code generation followed by
 // a simulator that executes the generated binary.
 namespace {
 class CustomExecutor : public AbstractInterpreter {
   std::string ExecutionCommand;
   std::vector<std::string> ExecutorArgs;
 
 public:
   CustomExecutor(const std::string &ExecutionCmd,
                  std::vector<std::string> ExecArgs)
       : ExecutionCommand(ExecutionCmd), ExecutorArgs(std::move(ExecArgs)) {}
 
-  Expected<int> ExecuteProgram(
+  Expected<sys::ProcessInfo> ExecuteProgram(
       const std::string &Bitcode, const std::vector<std::string> &Args,
       const std::string &InputFile, const std::string &OutputFile,
       const std::vector<std::string> &CCArgs,
       const std::vector<std::string> &SharedLibs = std::vector<std::string>(),
       unsigned Timeout = 0, unsigned MemoryLimit = 0) override;
 };
 }
 
-Expected<int> CustomExecutor::ExecuteProgram(
+Expected<sys::ProcessInfo> CustomExecutor::ExecuteProgram(
     const std::string &Bitcode, const std::vector<std::string> &Args,
     const std::string &InputFile, const std::string &OutputFile,
     const std::vector<std::string> &CCArgs,
     const std::vector<std::string> &SharedLibs, unsigned Timeout,
     unsigned MemoryLimit) {
 
   std::vector<StringRef> ProgramArgs;
   ProgramArgs.push_back(ExecutionCommand);
Expected<CC::FileType> LLC::OutputCode(const std::string &Bitcode,
                                        unsigned Timeout, unsigned MemoryLimit) {
   const char *Suffix = (UseIntegratedAssembler ? ".llc.o" : ".llc.s");
 
   SmallString<128> UniqueFile;
   std::error_code EC =
       sys::fs::createUniqueFile(Bitcode + "-%%%%%%%" + Suffix, UniqueFile);
   if (EC) {
     errs() << "Error making unique filename: " << EC.message() << "\n";
-    exit(1);
+    exit((int)llvm::sys::ProgramReturnCode::Failure);
   }
   OutputAsmFile = UniqueFile.str();
   std::vector<StringRef> LLCArgs;
   LLCArgs.push_back(LLCPath);
 
   // Add any extra LLC args.
   for (unsigned i = 0, e = ToolArgs.size(); i != e; ++i)
     LLCArgs.push_back(ToolArgs[i]);
 
   LLCArgs.push_back("-o");
   LLCArgs.push_back(OutputAsmFile); // Output to the Asm file
   LLCArgs.push_back(Bitcode);       // This is the input bitcode
 
   if (UseIntegratedAssembler)
     LLCArgs.push_back("-filetype=obj");
 
   outs() << (UseIntegratedAssembler ? "<llc-ia>" : "<llc>");
   outs().flush();
   LLVM_DEBUG(errs() << "\nAbout to run:\t";
              for (unsigned i = 0, e = LLCArgs.size() - 1; i != e; ++i) errs()
              << " " << LLCArgs[i];
              errs() << "\n";);
-  if (RunProgramWithTimeout(LLCPath, LLCArgs, "", "", "", Timeout, MemoryLimit))
+  if (!RunProgramWithTimeout(LLCPath, LLCArgs, "", "", "", Timeout, MemoryLimit))
     return ProcessFailure(LLCPath, LLCArgs, Timeout, MemoryLimit);
   return UseIntegratedAssembler ? CC::ObjectFile : CC::AsmFile;
 }
 
 Error LLC::compileProgram(const std::string &Bitcode, unsigned Timeout,
                           unsigned MemoryLimit) {
   std::string OutputAsmFile;
   Expected<CC::FileType> Result =
       OutputCode(Bitcode, OutputAsmFile, Timeout, MemoryLimit);
   sys::fs::remove(OutputAsmFile);
   if (Error E = Result.takeError())
     return E;
   return Error::success();
 }
 
-Expected<int> LLC::ExecuteProgram(const std::string &Bitcode,
+Expected<sys::ProcessInfo> LLC::ExecuteProgram(const std::string &Bitcode,
                                   const std::vector<std::string> &Args,
                                   const std::string &InputFile,
                                   const std::string &OutputFile,
                                   const std::vector<std::string> &ArgsForCC,
                                   const std::vector<std::string> &SharedLibs,
                                   unsigned Timeout, unsigned MemoryLimit) {
 
   std::string OutputAsmFile;
LLC *AbstractInterpreter::createLLC(const char *Argv0, std::string &Message,
   if (LLCPath.empty()) {
     Message = "Cannot find `llc' in executable directory!\n";
     return nullptr;
   }
 
   CC *cc = CC::create(Message, CCBinary, CCArgs);
   if (!cc) {
     errs() << Message << "\n";
-    exit(1);
+    exit((int)llvm::sys::ProgramReturnCode::Failure);
   }
   Message = "Found llc: " + LLCPath + "\n";
   return new LLC(LLCPath, cc, Args, UseIntegratedAssembler);
 }
 
 //===---------------------------------------------------------------------===//
 // JIT Implementation of AbstractIntepreter interface
 //
 namespace {
 class JIT : public AbstractInterpreter {
   std::string LLIPath;               // The path to the LLI executable
   std::vector<std::string> ToolArgs; // Args to pass to LLI
 public:
   JIT(const std::string &Path, const std::vector<std::string> *Args)
       : LLIPath(Path) {
     ToolArgs.clear();
     if (Args) {
       ToolArgs = *Args;
     }
   }
 
-  Expected<int> ExecuteProgram(
+  Expected<sys::ProcessInfo> ExecuteProgram(
       const std::string &Bitcode, const std::vector<std::string> &Args,
       const std::string &InputFile, const std::string &OutputFile,
       const std::vector<std::string> &CCArgs = std::vector<std::string>(),
       const std::vector<std::string> &SharedLibs = std::vector<std::string>(),
       unsigned Timeout = 0, unsigned MemoryLimit = 0) override;
 };
 }
 
-Expected<int> JIT::ExecuteProgram(const std::string &Bitcode,
+Expected<sys::ProcessInfo> JIT::ExecuteProgram(const std::string &Bitcode,
                                   const std::vector<std::string> &Args,
                                   const std::string &InputFile,
                                   const std::string &OutputFile,
                                   const std::vector<std::string> &CCArgs,
                                   const std::vector<std::string> &SharedLibs,
                                   unsigned Timeout, unsigned MemoryLimit) {
   // Construct a vector of parameters, incorporating those from the command-line
   std::vector<StringRef> JITArgs;
if (I == Args.size())
       break;
     if (Args[I].startswith_lower("arm"))
       return true;
   }
 
   return false;
 }
 
-Expected<int> CC::ExecuteProgram(const std::string &ProgramFile,
+Expected<sys::ProcessInfo> CC::ExecuteProgram(const std::string &ProgramFile,
                                  const std::vector<std::string> &Args,
                                  FileType fileType,
                                  const std::string &InputFile,
                                  const std::string &OutputFile,
                                  const std::vector<std::string> &ArgsForCC,
                                  unsigned Timeout, unsigned MemoryLimit) {
   std::vector<StringRef> CCArgs;
 
Expected<sys::ProcessInfo> CC::ExecuteProgram(const std::string &ProgramFile,
   CCArgs.push_back("none");
   CCArgs.push_back("-o");
 
   SmallString<128> OutputBinary;
   std::error_code EC =
       sys::fs::createUniqueFile(ProgramFile + "-%%%%%%%.cc.exe", OutputBinary);
   if (EC) {
     errs() << "Error making unique filename: " << EC.message() << "\n";
-    exit(1);
+    exit((int)llvm::sys::ProgramReturnCode::Failure);
   }
   CCArgs.push_back(OutputBinary); // Output to the right file...
 
   // Add any arguments intended for CC. We locate them here because this is
   // most likely -L and -l options that need to come before other libraries but
   // after the source. Other options won't be sensitive to placement on the
   // command line, so this should be safe.
   for (unsigned i = 0, e = ArgsForCC.size(); i != e; ++i)
     CCArgs.push_back(ArgsForCC[i]);
 
   CCArgs.push_back("-lm"); // Hard-code the math library...
   CCArgs.push_back("-O2"); // Optimize the program a bit...
   if (TargetTriple.getArch() == Triple::sparc)
     CCArgs.push_back("-mcpu=v9");
 
   outs() << "<CC>";
   outs().flush();
   LLVM_DEBUG(errs() << "\nAbout to run:\t";
              for (unsigned i = 0, e = CCArgs.size() - 1; i != e; ++i) errs()
              << " " << CCArgs[i];
              errs() << "\n";);
-  if (RunProgramWithTimeout(CCPath, CCArgs, "", "", ""))
+  if (!RunProgramWithTimeout(CCPath, CCArgs, "", "", ""))
     return ProcessFailure(CCPath, CCArgs);
 
   std::vector<StringRef> ProgramArgs;
 
   // Declared here so that the destructor only runs after
   // ProgramArgs is used.
   std::string Exec;
 
LLVM_DEBUG(
       for (unsigned i = 0, e = ProgramArgs.size() - 1; i != e; ++i) errs()
       << " " << ProgramArgs[i];
       errs() << "\n";);
 
   FileRemover OutputBinaryRemover(OutputBinary.str(), !SaveTemps);
 
   if (RemoteClientPath.empty()) {
     LLVM_DEBUG(errs() << "<run locally>");
-    std::string Error;
-    int ExitCode = RunProgramWithTimeout(OutputBinary.str(), ProgramArgs,
-                                         InputFile, OutputFile, OutputFile,
-                                         Timeout, MemoryLimit, &Error);
+    sys::ProcessInfo PI =
+        RunProgramWithTimeout(OutputBinary.str(), ProgramArgs, InputFile,
+                              OutputFile, OutputFile, Timeout, MemoryLimit);
     // Treat a signal (usually SIGSEGV) or timeout as part of the program output
     // so that crash-causing miscompilation is handled seamlessly.
-    if (ExitCode < -1) {
+    if (PI.notCompletedOrFailed() || PI.crashed()) {
       std::ofstream outFile(OutputFile.c_str(), std::ios_base::app);
-      outFile << Error << '\n';
+      outFile << PI.ErrMsg.getValueOr("unknown error") << '\n';
       outFile.close();
     }
-    return ExitCode;
+    return PI;
   } else {
     outs() << "<run remotely>";
     outs().flush();
     return RunProgramRemotelyWithTimeout(RemoteClientPath, ProgramArgs,
                                          InputFile, OutputFile, OutputFile,
                                          Timeout, MemoryLimit);
   }
 }
 
 Error CC::MakeSharedObject(const std::string &InputFile, FileType fileType,
                            std::string &OutputFile,
                            const std::vector<std::string> &ArgsForCC) {
   SmallString<128> UniqueFilename;
   std::error_code EC = sys::fs::createUniqueFile(
       InputFile + "-%%%%%%%" + LTDL_SHLIB_EXT, UniqueFilename);
   if (EC) {
     errs() << "Error making unique filename: " << EC.message() << "\n";
-    exit(1);
+    exit((int)llvm::sys::ProgramReturnCode::Failure);
   }
   OutputFile = UniqueFilename.str();
 
   std::vector<StringRef> CCArgs;
 
   CCArgs.push_back(CCPath);
 
   if (TargetTriple.getArch() == Triple::x86)
for (unsigned i = 0, e = ArgsForCC.size(); i != e; ++i)
     CCArgs.push_back(ArgsForCC[i]);
 
   outs() << "<CC>";
   outs().flush();
   LLVM_DEBUG(errs() << "\nAbout to run:\t";
              for (unsigned i = 0, e = CCArgs.size() - 1; i != e; ++i) errs()
              << " " << CCArgs[i];
              errs() << "\n";);
-  if (RunProgramWithTimeout(CCPath, CCArgs, "", "", ""))
+  if (!RunProgramWithTimeout(CCPath, CCArgs, "", "", ""))
     return ProcessFailure(CCPath, CCArgs);
   return Error::success();
 }
 
 /// create - Try to find the CC executable
 ///
 CC *CC::create(std::string &Message, const std::string &CCBinary,
                const std::vector<std::string> *Args) {

llvm/trunk/tools/dsymutil/MachOUtils.cpp

static bool runLipo(StringRef SDKPath, SmallVectorImpl<StringRef> &Args) {
   if (!Path)
     Path = sys::findProgramByName("lipo");
 
   if (!Path) {
     WithColor::error() << "lipo: " << Path.getError().message() << "\n";
     return false;
   }
 
-  std::string ErrMsg;
-  int result = sys::ExecuteAndWait(*Path, Args, None, {}, 0, 0, &ErrMsg);
-  if (result) {
-    WithColor::error() << "lipo: " << ErrMsg << "\n";
+  sys::ProcessInfo PI;
+  bool R = sys::ExecuteAndWait(*Path, Args, None, {}, 0, 0, &PI);
+  if (!R) {
+    WithColor::error() << "lipo: " << PI.ErrMsg.getValueOr("unknown error") << "\n";
     return false;
   }
 
   return true;
 }
 
 bool generateUniversalBinary(SmallVectorImpl<ArchAndFile> &ArchFiles,
                              StringRef OutputFileName,

llvm/trunk/tools/llvm-cov/CodeCoverage.cpp

void CodeCoverageTool::demangleSymbols(const CoverageMapping &Coverage) {
   ToolOutputFile OutputTOF{OutputPath, OutputFD};
   OutputTOF.os().close();
 
   // Invoke the demangler.
   std::vector<StringRef> ArgsV;
   for (StringRef Arg : ViewOpts.DemanglerOpts)
     ArgsV.push_back(Arg);
   Optional<StringRef> Redirects[] = {InputPath.str(), OutputPath.str(), {""}};
-  std::string ErrMsg;
-  int RC = sys::ExecuteAndWait(ViewOpts.DemanglerOpts[0], ArgsV,
+  sys::ProcessInfo PI;
+  bool R = sys::ExecuteAndWait(ViewOpts.DemanglerOpts[0], ArgsV,
                                /*env=*/None, Redirects, /*secondsToWait=*/0,
-                               /*memoryLimit=*/0, &ErrMsg);
-  if (RC) {
-    error(ErrMsg, ViewOpts.DemanglerOpts[0]);
+                               /*memoryLimit=*/0, &PI);
+  if (!R) {
+    error(PI.ErrMsg.getValueOr("unknown error"), ViewOpts.DemanglerOpts[0]);
     return;
   }
 
   // Parse the demangler's output.
   auto BufOrError = MemoryBuffer::getFile(OutputPath);
   if (!BufOrError) {
     error(OutputPath, BufOrError.getError().message());
     return;

llvm/trunk/unittests/Support/ProgramTest.cpp

TEST_F(ProgramEnvTest, CreateProcessLongPath) {
   SmallString<128> TestDirectory;
   ASSERT_NO_ERROR(
     fs::createUniqueDirectory("program-redirect-test", TestDirectory));
   SmallString<256> LongPath(TestDirectory);
   LongPath.push_back('\\');
   // MAX_PATH = 260
   LongPath.append(260 - TestDirectory.size(), 'a');
 
-  std::string Error;
-  bool ExecutionFailed;
+  ProcessInfo PI;
   Optional<StringRef> Redirects[] = {None, LongPath.str(), None};
-  int RC = ExecuteAndWait(MyExe, ArgV, getEnviron(), Redirects,
-    /*secondsToWait=*/ 10, /*memoryLimit=*/ 0, &Error,
-    &ExecutionFailed);
-  EXPECT_FALSE(ExecutionFailed) << Error;
-  EXPECT_EQ(0, RC);
+  bool R = ExecuteAndWait(MyExe, ArgV, getEnviron(), Redirects,
+                          /*secondsToWait=*/10, /*memoryLimit=*/0, &PI);
+  EXPECT_TRUE(R) << PI.ErrMsg.getValueOr("execution error");
+  EXPECT_EQ(0, PI.ReturnCode.getValue());
 
   // Remove the long stdout.
   ASSERT_NO_ERROR(fs::remove(Twine(LongPath)));
   ASSERT_NO_ERROR(fs::remove(Twine(TestDirectory)));
 }
 #endif
 
 TEST_F(ProgramEnvTest, CreateProcessTrailingSlash) {
StringRef argv[] = {
       "-program-test-string-arg1",
       "has\\\\ trailing\\",
       "-program-test-string-arg2",
       "has\\\\ trailing\\"};
 
   // Add LLVM_PROGRAM_TEST_CHILD to the environment of the child.
   addEnvVar("LLVM_PROGRAM_TEST_CHILD=1");
 
-  std::string error;
-  bool ExecutionFailed;
+  ProcessInfo PI;
   // Redirect stdout and stdin to NUL, but let stderr through.
 #ifdef _WIN32
   StringRef nul("NUL");
 #else
   StringRef nul("/dev/null");
 #endif
   Optional<StringRef> redirects[] = { nul, nul, None };
-  int rc = ExecuteAndWait(my_exe, argv, getEnviron(), redirects,
-                          /*secondsToWait=*/ 10, /*memoryLimit=*/ 0, &error,
-                          &ExecutionFailed);
-  EXPECT_FALSE(ExecutionFailed) << error;
-  EXPECT_EQ(0, rc);
+  bool R = ExecuteAndWait(my_exe, argv, getEnviron(), redirects,
+                          /*secondsToWait=*/ 10, /*memoryLimit=*/ 0, &PI);
+  EXPECT_TRUE(R) << PI.ErrMsg.getValueOr("execution error");
+  EXPECT_EQ(0, PI.ReturnCode.getValue());
 }
 
 TEST_F(ProgramEnvTest, TestExecuteNoWait) {
   using namespace llvm::sys;
 
   if (getenv("LLVM_PROGRAM_TEST_EXECUTE_NO_WAIT")) {
     sleep_for(/*seconds*/ 1);
     exit(0);
   }
 
   std::string Executable =
       sys::fs::getMainExecutable(TestMainArgv0, &ProgramTestStringArg1);
   StringRef argv[] = {Executable,
                       "--gtest_filter=ProgramEnvTest.TestExecuteNoWait"};
 
   // Add LLVM_PROGRAM_TEST_EXECUTE_NO_WAIT to the environment of the child.
   addEnvVar("LLVM_PROGRAM_TEST_EXECUTE_NO_WAIT=1");
 
-  std::string Error;
-  bool ExecutionFailed;
-  ProcessInfo PI1 = ExecuteNoWait(Executable, argv, getEnviron(), {}, 0, &Error,
-                                  &ExecutionFailed);
-  ASSERT_FALSE(ExecutionFailed) << Error;
+  ProcessInfo PI1 = ExecuteNoWait(Executable, argv, getEnviron());
+  ASSERT_TRUE(PI1.running()) << PI1.ErrMsg.getValueOr("execution error");
   ASSERT_NE(PI1.Pid, ProcessInfo::InvalidPid) << "Invalid process id";
 
   unsigned LoopCount = 0;
 
   // Test that Wait() with WaitUntilTerminates=true works. In this case,
   // LoopCount should only be incremented once.
-  while (true) {
+  while (LoopCount < 10) {
     ++LoopCount;
-    ProcessInfo WaitResult = llvm::sys::Wait(PI1, 0, true, &Error);
-    ASSERT_TRUE(Error.empty());
-    if (WaitResult.Pid == PI1.Pid)
+    bool R = llvm::sys::Wait(PI1, 0, true);
+    ASSERT_TRUE(R) << PI1.ErrMsg.getValueOr("wait error");
+    if (PI1.completedSuccess())
       break;
   }
 
   EXPECT_EQ(LoopCount, 1u) << "LoopCount should be 1";
 
-  ProcessInfo PI2 = ExecuteNoWait(Executable, argv, getEnviron(), {}, 0, &Error,
-                                  &ExecutionFailed);
-  ASSERT_FALSE(ExecutionFailed) << Error;
+  ProcessInfo PI2 = ExecuteNoWait(Executable, argv, getEnviron());
+  ASSERT_TRUE(PI2.running()) << PI2.ErrMsg.getValueOr("execution error");
   ASSERT_NE(PI2.Pid, ProcessInfo::InvalidPid) << "Invalid process id";
 
   // Test that Wait() with SecondsToWait=0 performs a non-blocking wait. In this
   // cse, LoopCount should be greater than 1 (more than one increment occurs).
   while (true) {
     ++LoopCount;
-    ProcessInfo WaitResult = llvm::sys::Wait(PI2, 0, false, &Error);
-    ASSERT_TRUE(Error.empty());
-    if (WaitResult.Pid == PI2.Pid)
+    bool R = llvm::sys::Wait(PI2, 0, false);
+    ASSERT_TRUE(R) << PI2.ErrMsg.getValueOr("wait error");
+    ASSERT_FALSE(PI2.completedError());
+    if (PI1.completedSuccess())
       break;
   }
 
   ASSERT_GT(LoopCount, 1u) << "LoopCount should be >1";
 }
 
 TEST_F(ProgramEnvTest, TestExecuteAndWaitTimeout) {
   using namespace llvm::sys;
 
   if (getenv("LLVM_PROGRAM_TEST_TIMEOUT")) {
     sleep_for(/*seconds*/ 10);
     exit(0);
   }
 
   std::string Executable =
       sys::fs::getMainExecutable(TestMainArgv0, &ProgramTestStringArg1);
   StringRef argv[] = {
       Executable, "--gtest_filter=ProgramEnvTest.TestExecuteAndWaitTimeout"};
 
   // Add LLVM_PROGRAM_TEST_TIMEOUT to the environment of the child.
   addEnvVar("LLVM_PROGRAM_TEST_TIMEOUT=1");
 
-  std::string Error;
-  bool ExecutionFailed;
-  int RetCode =
-      ExecuteAndWait(Executable, argv, getEnviron(), {}, /*secondsToWait=*/1, 0,
-                     &Error, &ExecutionFailed);
-  ASSERT_EQ(-2, RetCode);
+  ProcessInfo PI;
+  bool R =
+      ExecuteAndWait(Executable, argv, getEnviron(), {}, /*secondsToWait=*/1, 0, &PI);
+  ASSERT_FALSE(R);
+  ASSERT_TRUE(PI.killed());
 }
 
 TEST(ProgramTest, TestExecuteNegative) {
   std::string Executable = "i_dont_exist";
   StringRef argv[] = {Executable};
 
   {
-    std::string Error;
-    bool ExecutionFailed;
-    int RetCode = ExecuteAndWait(Executable, argv, llvm::None, {}, 0, 0, &Error,
-                                 &ExecutionFailed);
-    ASSERT_TRUE(RetCode < 0) << "On error ExecuteAndWait should return 0 or "
-                                "positive value indicating the result code";
-    ASSERT_TRUE(ExecutionFailed);
-    ASSERT_FALSE(Error.empty());
+    ProcessInfo PI;
+    bool R = ExecuteAndWait(Executable, argv, llvm::None, {}, 0, 0, &PI);
+    ASSERT_FALSE(R);
+    ASSERT_TRUE(PI.notCompletedOrFailed());
+    ASSERT_TRUE(PI.ErrMsg.hasValue());
   }
 
   {
-    std::string Error;
-    bool ExecutionFailed;
-    ProcessInfo PI = ExecuteNoWait(Executable, argv, llvm::None, {}, 0, &Error,
-                                   &ExecutionFailed);
+    ProcessInfo PI = ExecuteNoWait(Executable, argv, llvm::None);
     ASSERT_EQ(PI.Pid, ProcessInfo::InvalidPid)
         << "On error ExecuteNoWait should return an invalid ProcessInfo";
-    ASSERT_TRUE(ExecutionFailed);
-    ASSERT_FALSE(Error.empty());
+    ASSERT_TRUE(PI.completedError());
+    ASSERT_FALSE(PI.ErrMsg.hasValue());
   }
 
 }
 
 #ifdef _WIN32
 const char utf16le_text[] =
     "\x6c\x00\x69\x00\x6e\x00\x67\x00\xfc\x00\x69\x00\xe7\x00\x61\x00";
 const char utf16be_text[] =

llvm/trunk/utils/not/not.cpp

int main(int argc, const char **argv) {
 
   if (argc > 0 && StringRef(argv[0]) == "--crash") {
     ++argv;
     --argc;
     ExpectCrash = true;
   }
 
   if (argc == 0)
-    return 1;
+    return (int)llvm::sys::ProgramReturnCode::Failure;
 
   auto Program = sys::findProgramByName(argv[0]);
   if (!Program) {
     errs() << "Error: Unable to find `" << argv[0]
            << "' in PATH: " << Program.getError().message() << "\n";
-    return 1;
+    return (int)llvm::sys::ProgramReturnCode::Failure;
   }
 
   std::vector<StringRef> Argv;
   Argv.reserve(argc);
   for (int i = 0; i < argc; ++i)
     Argv.push_back(argv[i]);
-  std::string ErrMsg;
-  int Result = sys::ExecuteAndWait(*Program, Argv, None, {}, 0, 0, &ErrMsg);
-#ifdef _WIN32
-  // Handle abort() in msvcrt -- It has exit code as 3.  abort(), aka
-  // unreachable, should be recognized as a crash.  However, some binaries use
-  // exit code 3 on non-crash failure paths, so only do this if we expect a
-  // crash.
-  if (ExpectCrash && Result == 3)
-    Result = -3;
-#endif
-  if (Result < 0) {
-    errs() << "Error: " << ErrMsg << "\n";
-    if (ExpectCrash)
-      return 0;
-    return 1;
-  }
   
-  if (ExpectCrash)
-    return 1;
+  sys::ProcessInfo PI;
+  sys::ExecuteAndWait(*Program, Argv, None, {}, 0, 0, &PI);
 
-  return Result == 0;
+  if (PI.crashed(ExpectCrash)) {
+    errs() << "Error: " << PI.ErrMsg.getValue() << "\n";
+    return int(ExpectCrash ? llvm::sys::ProgramReturnCode::Success
+                           : llvm::sys::ProgramReturnCode::Failure);
+  }
+  if (ExpectCrash)
+      return (int)llvm::sys::ProgramReturnCode::Failure;
+  return PI.completedSuccess();
 }