[lit] Limit parallelism of sanitizer tests on Darwin

[kubamracek]

Summary

Running lit tests and unit tests of ASan and TSan on macOS has very bad performance when running with a high number of threads. This is caused by xnu (the macOS kernel), which currently doesn't handle mapping and unmapping of sanitizer shadow regions (reserved VM which are several terabytes large) very well. The situation is so bad that increasing the number of threads actually makes the total testing time larger. The macOS buildbots are affected by this. Note that we can't easily limit the number of sanitizer testing threads without affecting the rest of the tests.

This patch adds a special "group" into lit, and limits the number of concurrently running tests in this group. This helps solve the contention problem, while still allowing other tests to run in full, that means running lit with -j8 will still with 8 threads, and parallelism is only limited in sanitizer tests.

[dvyukov]

LGTM

We noticed that issues with slow mmaps/munmaps on darwin as well.
Looks fine to me, but I am not very experienced with lit/python.

[kubamracek]

LLVM part landed in r292231, compiler-rt part in r292232.

[MatzeB]

llvm/trunk/utils/lit/lit/main.py

321 ↗

(On Diff #84687)

Mentioning the sanitizer and the magic number 3 in the generic lit code seems wrong to me.
This should really be in the compiler-rt lit.site.cfg somehow.

[kubamracek]

Mentioning the sanitizer and the magic number 3 in the generic lit code seems wrong to me.
This should really be in the compiler-rt lit.site.cfg somehow.

Right. I've reverted this (there was also a failure on the Windows bot), let me submit another patch for review.

[kubamracek]

Updating patch. darwin_sanitizer_parallelism_group_func is now only used on Darwin (should fix Windows bot breakage) and the magic number 3 is moved to compiler-rt (into test/lit.common.cfg and unittests/lit.common.unit.cfg).

[MatzeB]

Generally looks good to me, nitpicks below.

You may consider using the word "pool" instead of "parallelism_group" to match the terminology of the ninja builder.

projects/compiler-rt/test/asan/lit.cfg

245–246

could be

if config.host_os == 'Darwin' and config.target_arch in ["x86_64", "x86_64h"]:

projects/compiler-rt/unittests/lit.common.unit.cfg

39–41

return "darwin-64bit-sanitizer" if "x86_64" in test.file_path else ""

utils/lit/lit/run.py

178

This should rather be dict() or left out

182

Supporting callbacks here feels overengineered, but if you have to...

185

I would move the name lookup into the try: part, so that for an invalid name lit doesn't shutdown completely.

[kubamracek]

Thanks for reviewing this! I'll submit another patch.

utils/lit/lit/run.py

182

I tried really hard not to use a callback, but unfortunately for compiler-rt unittests (gtest-style), we can't just set config.parallelism_group at lit launch time -- we only know the architecture after test discovery. I'm open to suggestions if you can think of a better way of handling this.

[MatzeB]

utils/lit/lit/run.py

182

I don't understand exactly what is happening with the gtests here. Do you expect unittests for architectures other than config.host_os being present?
I wouldn't expect us to have infrastructure here to even run tests for targets other than the host...

[kubamracek]

utils/lit/lit/run.py

182

We have i386 and x86_64 unittests. Finding which tests are 64-bit can only be done after test discovery.

[MatzeB]

utils/lit/lit/run.py

182

Ah. Well, I don't have any better suggestion than the callback right now.

[kubamracek]

Updating patch, addressing review comments.

[MatzeB]

LGTM

[kubamracek]

Thanks! LLVM part in r292548, compiler-rt part in r292549.

projects/compiler-rt/test/asan/Unit/lit.site.cfg.in

else:
   test_dir = "default"
 config.test_exec_root = os.path.join("@COMPILER_RT_BINARY_DIR@",
                                      "lib", "asan", "tests", test_dir)
 
 config.test_source_root = config.test_exec_root
 
 # Set LD_LIBRARY_PATH to pick dynamic runtime up properly.
 push_ld_library_path(config, config.compiler_rt_libdir)
+
+if config.host_os == 'Darwin':
+  config.parallelism_group = config.darwin_sanitizer_parallelism_group_func

projects/compiler-rt/test/asan/lit.cfg

 else:
   config.substitutions.append(('%fPIC', '-fPIC'))
   config.substitutions.append(('%fPIE', '-fPIE'))
   config.substitutions.append(('%pie', '-pie'))
 
 # Only run the tests on supported OSs.
 if config.host_os not in ['Linux', 'Darwin', 'FreeBSD', 'Windows']:
   config.unsupported = True
+
+if config.host_os == 'Darwin':
+  if config.target_arch in ["x86_64", "x86_64h"]:
+    config.parallelism_group = "darwin-64bit-sanitizer"

projects/compiler-rt/test/lit.common.cfg

 if re.search(r'ON', llvm_config_cmd.stdout.read().decode('ascii')):
   config.available_features.add('asserts')
 llvm_config_cmd.wait()
 
 # Sanitizer tests tend to be flaky on Windows due to PR24554, so add some
 # retries. We don't do this on otther platforms because it's slower.
 if platform.system() == 'Windows':
   config.test_retry_attempts = 2
+
+# Only run up to 3 64-bit sanitized processes simultaneously on Darwin.
+# Using more scales badly and hogs the system due to inefficient handling
+# of large mmap'd regions (terabytes) by the kernel.
+if platform.system() == 'Darwin':
+  lit_config.parallelism_groups["darwin-64bit-sanitizer"] = 3

projects/compiler-rt/test/tsan/Unit/lit.site.cfg.in

 @LIT_SITE_CFG_IN_HEADER@
 
 # Load common config for all compiler-rt unit tests.
 lit_config.load_config(config, "@COMPILER_RT_BINARY_DIR@/unittests/lit.common.unit.configured")
 
 # Setup config name.
 config.name = 'ThreadSanitizer-Unit'
 
 # Setup test source and exec root. For unit tests, we define
 # it as build directory with ASan unit tests.
 # FIXME: De-hardcode this path.
 config.test_exec_root = "@COMPILER_RT_BINARY_DIR@/lib/tsan/tests"
 config.test_source_root = config.test_exec_root
+
+if config.host_os == 'Darwin':
+  config.parallelism_group = config.darwin_sanitizer_parallelism_group_func

projects/compiler-rt/test/tsan/lit.cfg

 # ThreadSanitizer tests are currently supported on FreeBSD, Linux and Darwin.
 if config.host_os not in ['FreeBSD', 'Linux', 'Darwin']:
   config.unsupported = True
 
 # Allow tests to use REQUIRES=stable-runtime.  For use when you cannot use XFAIL
 # because the test hangs.
 if config.target_arch != 'aarch64':
   config.available_features.add('stable-runtime')
+
+if config.host_os == 'Darwin':
+  if config.target_arch in ["x86_64", "x86_64h"]:
+    config.parallelism_group = "darwin-64bit-sanitizer"

projects/compiler-rt/unittests/lit.common.unit.cfg

 config.environment['PATH'] = path
 
 # Propagate the temp directory. Windows requires this because it uses \Windows\
 # if none of these are present.
 if 'TMP' in os.environ:
     config.environment['TMP'] = os.environ['TMP']
 if 'TEMP' in os.environ:
     config.environment['TEMP'] = os.environ['TEMP']
+
+if config.host_os == 'Darwin':
+  # Only run up to 3 64-bit sanitized processes simultaneously on Darwin.
+  # Using more scales badly and hogs the system due to inefficient handling
+  # of large mmap'd regions (terabytes) by the kernel.
+  lit_config.parallelism_groups["darwin-64bit-sanitizer"] = 3
+
+  def darwin_sanitizer_parallelism_group_func(test):
+    if test.file_path.find("x86_64") != -1:
+      return "darwin-64bit-sanitizer"
+    return ""
+  config.darwin_sanitizer_parallelism_group_func = darwin_sanitizer_parallelism_group_func

utils/lit/lit/LitConfig.py

class LitConfig(object):
     easily.
     """
 
     def __init__(self, progname, path, quiet,
                  useValgrind, valgrindLeakCheck, valgrindArgs,
                  noExecute, debug, isWindows,
                  params, config_prefix = None,
                  maxIndividualTestTime = 0,
-                 maxFailures = None):
+                 maxFailures = None,
+                 parallelism_groups = []):
         # The name of the test runner.
         self.progname = progname
         # The items to add to the PATH environment variable.
         self.path = [str(p) for p in path]
         self.quiet = bool(quiet)
         self.useValgrind = bool(useValgrind)
         self.valgrindLeakCheck = bool(valgrindLeakCheck)
         self.valgrindUserArgs = list(valgrindArgs)
def __init__(self, progname, path, quiet,
                 self.valgrindArgs.append('--leak-check=full')
             else:
                 # The default is 'summary'.
                 self.valgrindArgs.append('--leak-check=no')
             self.valgrindArgs.extend(self.valgrindUserArgs)
 
         self.maxIndividualTestTime = maxIndividualTestTime
         self.maxFailures = maxFailures
+        self.parallelism_groups = parallelism_groups
 
     @property
     def maxIndividualTestTime(self):
         """
             Interface for getting maximum time to spend executing
             a single test
         """
         return self._maxIndividualTestTime

utils/lit/lit/TestingConfig.py

def load_from_path(self, path, litConfig):
                     path, traceback.format_exc()))
 
         self.finish(litConfig)
 
     def __init__(self, parent, name, suffixes, test_format,
                  environment, substitutions, unsupported,
                  test_exec_root, test_source_root, excludes,
                  available_features, pipefail, limit_to_features = [],
-                 is_early = False):
+                 is_early = False, parallelism_group = ""):
         self.parent = parent
         self.name = str(name)
         self.suffixes = set(suffixes)
         self.test_format = test_format
         self.environment = dict(environment)
         self.substitutions = list(substitutions)
         self.unsupported = unsupported
         self.test_exec_root = test_exec_root
         self.test_source_root = test_source_root
         self.excludes = set(excludes)
         self.available_features = set(available_features)
         self.pipefail = pipefail
         # This list is used by TestRunner.py to restrict running only tests that
         # require one of the features in this list if this list is non-empty.
         # Configurations can set this list to restrict the set of tests to run.
         self.limit_to_features = set(limit_to_features)
         # Whether the suite should be tested early in a given run.
         self.is_early = bool(is_early)
+        self.parallelism_group = parallelism_group
 
     def finish(self, litConfig):
         """finish() - Finish this config object, after loading is complete."""
 
         self.name = str(self.name)
         self.suffixes = set(self.suffixes)
         self.environment = dict(self.environment)
         self.substitutions = list(self.substitutions)

utils/lit/lit/main.py

litConfig = lit.LitConfig.LitConfig(
         valgrindLeakCheck = opts.valgrindLeakCheck,
         valgrindArgs = opts.valgrindArgs,
         noExecute = opts.noExecute,
         debug = opts.debug,
         isWindows = isWindows,
         params = userParams,
         config_prefix = opts.configPrefix,
         maxIndividualTestTime = maxIndividualTestTime,
-        maxFailures = opts.maxFailures)
+        maxFailures = opts.maxFailures,
+        parallelism_groups = {})
 
     # Perform test discovery.
     run = lit.run.Run(litConfig,
                       lit.discovery.find_tests_for_inputs(litConfig, inputs))
 
     # After test discovery the configuration might have changed
     # the maxIndividualTestTime. If we explicitly set this on the
     # command line then override what was set in the test configuration

utils/lit/lit/run.py

 class Run(object):
     """
     This class represents a concrete, configured testing run.
     """
 
     def __init__(self, lit_config, tests):
         self.lit_config = lit_config
         self.tests = tests
+        self.parallelism_semaphores = ""
 
     def execute_test(self, test):
+        pg = test.config.parallelism_group
+        if callable(pg): pg = pg(test)
+
         result = None
+        if pg: self.parallelism_semaphores[pg].acquire()
         start_time = time.time()
         try:
             result = test.config.test_format.execute(test, self.lit_config)
 
             # Support deprecated result from execute() which returned the result
             # code and additional output as a tuple.
             if isinstance(result, tuple):
                 code, output = result
                 result = lit.Test.Result(code, output)
             elif not isinstance(result, lit.Test.Result):
                 raise ValueError("unexpected result from test execution")
+
+            result.elapsed = time.time() - start_time
         except KeyboardInterrupt:
             raise
         except:
             if self.lit_config.debug:
                 raise
             output = 'Exception during script execution:\n'
             output += traceback.format_exc()
             output += '\n'
             result = lit.Test.Result(lit.Test.UNRESOLVED, output)
-        result.elapsed = time.time() - start_time
+        finally:
+            if pg: self.parallelism_semaphores[pg].release()
 
         test.setResult(result)
 
     def execute_tests(self, display, jobs, max_time=None,
                       use_processes=False):
         """
         execute_tests(display, jobs, [max_time])
 
def execute_tests(self, display, jobs, max_time=None,
         """
 
         # Choose the appropriate parallel execution implementation.
         consumer = None
         if jobs != 1 and use_processes and multiprocessing:
             try:
                 task_impl = multiprocessing.Process
                 queue_impl = multiprocessing.Queue
+                sem_impl = multiprocessing.Semaphore
                 canceled_flag =  multiprocessing.Value('i', 0)
                 consumer = MultiprocessResultsConsumer(self, display, jobs)
             except:
                 # multiprocessing fails to initialize with certain OpenBSD and
                 # FreeBSD Python versions: http://bugs.python.org/issue3770
                 # Unfortunately the error raised also varies by platform.
                 self.lit_config.note('failed to initialize multiprocessing')
                 consumer = None
         if not consumer:
             task_impl = threading.Thread
             queue_impl = queue.Queue
+            sem_impl = threading.Semaphore
             canceled_flag = LockedValue(0)
             consumer = ThreadResultsConsumer(display)
 
+        self.parallelism_semaphores = {k: sem_impl(v)
+            for k, v in self.lit_config.parallelism_groups.items()}
+
         # Create the test provider.
         provider = TestProvider(queue_impl, canceled_flag)
         handleFailures(provider, consumer, self.lit_config.maxFailures)
 
         # Queue the tests outside the main thread because we can't guarantee
         # that we can put() all the tests without blocking:
         # https://docs.python.org/2/library/multiprocessing.html
         # e.g: On Mac OS X, we will hang if we put 2^15 elements in the queue