Use ninja pool to limit the number of concurrent compile/link jobs

[eugenis]

Summary

This change makes use of the new "job pool" capability in cmake 3.0 with ninja generator to allow limiting the number of concurrent jobs of a certain type.

For some reason, there is no way to define "JOB_POOLS" global property on the command line (or is there?).

We need this to limit the number of link jobs on the sanitizer-bootstrap bot, which runs out of memory linking multiple targets with debug info and sanitizer instrumentation.

[eugenis]

Removed the check for cmake version and generator type.

[eugenis]

-j sets the general limit on concurrent build jobs, and these two are
separate limits on compile/link jobs.

[eugenis]

Added a mention of the new variables to the docs.
Made the new variables cacheable.

[chandlerc]

This looks good as a an initial patch. If there are further refinements we want here, they can be post-commit IMO. Thanks!

[eugenis]

Thanks, committed as r222341.

CMakeLists.txt

 if(CMAKE_VERSION VERSION_LESS 3.1.20141117)
   set(cmake_3_2_USES_TERMINAL)
 else()
   set(cmake_3_2_USES_TERMINAL USES_TERMINAL)
 endif()
 
 project(LLVM)
 
+# The following only works with the Ninja generator in CMake >= 3.0.
+set(LLVM_PARALLEL_COMPILE_JOBS "" CACHE STRING
+  "Define the maximum number of concurrent compilation jobs.")
+if(LLVM_PARALLEL_COMPILE_JOBS)
+  set_property(GLOBAL APPEND PROPERTY JOB_POOLS compile_job_pool=${LLVM_PARALLEL_COMPILE_JOBS})
+  set(CMAKE_JOB_POOL_COMPILE compile_job_pool)
+endif()
+
+set(LLVM_PARALLEL_LINK_JOBS "" CACHE STRING
+  "Define the maximum number of concurrent link jobs.")
+if(LLVM_PARALLEL_LINK_JOBS)
+  set_property(GLOBAL APPEND PROPERTY JOB_POOLS link_job_pool=${LLVM_PARALLEL_LINK_JOBS})
+  set(CMAKE_JOB_POOL_LINK link_job_pool)
+endif()
+
 # Add path for custom modules
 set(CMAKE_MODULE_PATH
   ${CMAKE_MODULE_PATH}
   "${CMAKE_CURRENT_SOURCE_DIR}/cmake"
   "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules"
   )
 
 set(LLVM_VERSION_MAJOR 3)

docs/CMake.rst

**LLVM_ENABLE_ZLIB**:BOOL
   Build with zlib to support compression/uncompression in LLVM tools.
   Defaults to ON.
 
 **LLVM_USE_SANITIZER**:STRING
   Define the sanitizer used to build LLVM binaries and tests. Possible values
   are ``Address``, ``Memory``, ``MemoryWithOrigins`` and ``Undefined``.
   Defaults to empty string.
 
+**LLVM_PARALLEL_COMPILE_JOBS**:STRING
+  Define the maximum number of concurrent compilation jobs.
+
+**LLVM_PARALLEL_LINK_JOBS**:STRING
+  Define the maximum number of concurrent link jobs.
+
 **LLVM_BUILD_DOCS**:BOOL
   Enables all enabled documentation targets (i.e. Doxgyen and Sphinx targets) to
   be built as part of the normal build. If the ``install`` target is run then
   this also enables all built documentation targets to be installed. Defaults to
   OFF.
 
 **LLVM_ENABLE_DOXYGEN**:BOOL
   Enables the generation of browsable HTML documentation using doxygen.