Clean temp directories before running lit

[zturner]

Summary

Currently, lit creates a little "sandbox" for each test to use by doing two things:

1. Creating a directory in the build folder for each test directory in the source tree.
2. Within the folder in directory 1, creating a unique file prefix that can be appended to to create unique per-test filenames that an individual test can use.

No effort is made to clean these files up between test runs, which creates two problems.

First is that we just leak files. If a test changes, or gets removed, or moves to a different location, all the files remain in the old location. (Very) gradually this can contribute to out of disk space errors on build bots, but in general it's just bad practice.

More sinister is that leaked files from previous runs can lead to false positives in tests.

This patch makes lit clean up the output directory before each run. One existing design choice made this tricky: A tests' temporary folder is of the form <build-dir>/test/SourceDir/Output. We can't just rmtree the entire <build-dir>/test because there are some lit configuration files in there. So we would have to enumerate the top-level test directory deleting only certain ones.

To simplify this, I inverted the heirarchy, so that now it's <build-dir>/test/Output/SourceDir. this way we can just indiscriminately rmtree anything under Output without worrying about what it is.

A few tests on Windows show that the rmtree takes about 1.75 seconds with a full output tree, which is a negligible portion of the overall test suite run. On Linux I expect the rmtree to be even faster.

There is one test failure with this patch, in llvm-symbolizer.test. For some reason it's demangling symbols differently, which is quite bizarre. Still looking into this, but throwing up this patch early for high level comments.

[zturner]

Two files from an unrelated CL snuck in. Removed those.

[hfinkel]

More sinister is that leaked files from previous runs can lead to false positives in tests.

I've also seen this problem; thanks for working on a fix.

Does this always remove the entire Output directory if you're only running a subset of the tests?

[zturner]

In D34732#793239, @hfinkel wrote:

More sinister is that leaked files from previous runs can lead to false positives in tests.

I've also seen this problem; thanks for working on a fix.

Does this always remove the entire Output directory if you're only running a subset of the tests?

No, I actually tried to be smart about it. The following pseudocode describes the algorithm used

paths = list(unique_set(<all output paths>))
sort(paths, by number of components in path ascending)
for (path in paths) {
  if exists(path)
    rmtree(path)
  mkdir(path)
}

the unique'ifying is because you have have A/foo.test and A/bar.test which are both in A. Sorting by number of components in path is because otherwise you might do something like

rmtree("foo/bar")
mkdir("foo/bar")
rmtree("foo")
mkdir("foo")

and then foo/bar is gone too. So we rmtree the highest level folder that must be rmtree'd first, and we only ever remove trees that are necessary to remove based on the subset of tests you're running.

[hfinkel]

In D34732#793263, @zturner wrote:

In D34732#793239, @hfinkel wrote:

More sinister is that leaked files from previous runs can lead to false positives in tests.

I've also seen this problem; thanks for working on a fix.

Does this always remove the entire Output directory if you're only running a subset of the tests?

No, I actually tried to be smart about it. The following pseudocode describes the algorithm used

paths = list(unique_set(<all output paths>))
sort(paths, by number of components in path ascending)
for (path in paths) {
  if exists(path)
    rmtree(path)
  mkdir(path)
}

the unique'ifying is because you have have A/foo.test and A/bar.test which are both in A. Sorting by number of components in path is because otherwise you might do something like

rmtree("foo/bar")
mkdir("foo/bar")
rmtree("foo")
mkdir("foo")

and then foo/bar is gone too. So we rmtree the highest level folder that must be rmtree'd first, and we only ever remove trees that are necessary to remove based on the subset of tests you're running.

Sounds great.

[rnk]

In D34732#793263, @zturner wrote:

No, I actually tried to be smart about it. The following pseudocode describes the algorithm used

Nice, that's really cool. I figured we'd just pay 1.7s the first time we remove a test tree after 'check-all', but then pay less after running just llvm/test/MyAwesomeSuite because there were less output files. This is better, though. :)

[rnk]

llvm/utils/lit/lit/run.py

254 ↗

(On Diff #104331)

The time module is already available at global scope

260 ↗

(On Diff #104331)

Can you add a comment, something like "Sort the list by number of path components so that parent directories come before their children."

264 ↗

(On Diff #104331)

Let's just import shutil at global scope and do shutil.rmtree.

271 ↗

(On Diff #104331)

Should we print this at all? If we decide we want it, should we print with litConfig.note()?

[zturner]

Updated with fixes from dblaikie. All tests pass now.

[zturner]

Update with suggestions from rnk. With this patch all tests pass.

[rnk]

Code looks good to me as is. Improving the symbolizer test seems like something we can handle as needed or later.

llvm/trunk/test/DebugInfo/Inputs/split-dwarf-addr-object-relocation.cpp

+void f1();
+__attribute__((always_inline)) void f2() {
+  f1();
+}
+void f3() {
+  f2();
+}

llvm/trunk/test/DebugInfo/Inputs/split-dwarf-multiple-cu1.cpp

+extern int i;
+int i;

llvm/trunk/test/DebugInfo/Inputs/split-dwarf-multiple-cu2.cpp

+void f1();
+inline __attribute__((always_inline)) void f2() {
+  f1();
+}
+void f3() {
+  f2();
+}

llvm/trunk/test/DebugInfo/llvm-symbolizer.test

 RUN: echo "\"%p/Inputs/dwarfdump-test3.elf-x86-64 space\" 0x62d" >> %t.input
 RUN: echo "%p/Inputs/macho-universal 0x1f84" >> %t.input
 RUN: echo "%p/Inputs/macho-universal:i386 0x1f67" >> %t.input
 RUN: echo "%p/Inputs/macho-universal:x86_64 0x100000f05" >> %t.input
 RUN: echo "%p/Inputs/llvm-symbolizer-dwo-test 0x400514" >> %t.input
 RUN: echo "%p/Inputs/fission-ranges.elf-x86_64 0x720" >> %t.input
 RUN: echo "%p/Inputs/arange-overlap.elf-x86_64 0x714" >> %t.input
 RUN: cp %p/Inputs/split-dwarf-test.dwo %T
-RUN: echo "%p/Inputs/split-dwarf-test 0x4005d4" >> %t.input
-RUN: echo "%p/Inputs/split-dwarf-test 0x4005c4" >> %t.input
+RUN: echo "%p/Inputs/split-dwarf-test 0x400504" >> %t.input
+RUN: echo "%p/Inputs/split-dwarf-test 0x4004f0" >> %t.input
 RUN: echo "%p/Inputs/cross-cu-inlining.x86_64-macho.o 0x17" >> %t.input
 RUN: cp %p/Inputs/split-dwarf-multiple-cu.dwo %T
 RUN: echo "%p/Inputs/split-dwarf-multiple-cu.o 0x4" >> %t.input
 RUN: cp %p/Inputs/split-dwarf-addr-object-relocation.dwo %T
 RUN: echo "%p/Inputs/split-dwarf-addr-object-relocation.o 0x14" >> %t.input
 RUN: cp %p/Inputs/split-dwarf-dwp.o %T
 RUN: cp %p/Inputs/split-dwarf-dwp.o.dwp %T
 RUN: echo "%T/split-dwarf-dwp.o 0x4" >> %t.input

 ; func has been inlined into main by LTO. Check that the symbolizer is able
 ; to resolve the cross-cu reference and retrieve func's name
 CHECK: func
 CHECK-NEXT: /tmp{{[/\\]}}cross-cu-inlining.c:16:3
 CHECK-NEXT: main
 CHECK-NEXT: /tmp{{[/\\]}}cross-cu-inlining.c:11:0
 
 CHECK:      f2
-CHECK-NEXT: b.cpp:3:3
+CHECK-NEXT: split-dwarf-multiple-cu2.cpp:3:3
 CHECK-NEXT: f3
-CHECK-NEXT: b.cpp:6:0
+CHECK-NEXT: split-dwarf-multiple-cu2.cpp:6:0
 
 CHECK:      f2
 CHECK-NEXT: split-dwarf-addr-object-relocation.cpp:3:3
 CHECK-NEXT: f3
 CHECK-NEXT: split-dwarf-addr-object-relocation.cpp:6:0
 
 CHECK:      f2
 CHECK-NEXT: split-dwarf-dwp.cpp:3:3

llvm/trunk/test/Other/lit-globbing.ll

-RUN: echo TA > %T/TA.txt
-RUN: echo TB > %T/TB.txt
-RUN: echo TAB > %T/TAB.txt
-
-RUN: echo %T/TA* | FileCheck -check-prefix=STAR %s
-RUN: echo %T/'TA'* | FileCheck -check-prefix=STAR %s
-RUN: echo %T/T'A'* | FileCheck -check-prefix=STAR %s
-
-RUN: echo %T/T?.txt | FileCheck -check-prefix=QUESTION %s
-RUN: echo %T/'T'?.txt | FileCheck -check-prefix=QUESTION %s
-
-RUN: echo %T/T??.txt | FileCheck -check-prefix=QUESTION2 %s
-RUN: echo %T/'T'??.txt | FileCheck -check-prefix=QUESTION2 %s
-
-RUN: echo 'T*' 'T?.txt' 'T??.txt' | FileCheck -check-prefix=QUOTEDARGS %s
-
-STAR-NOT: TB.txt
-STAR: {{(TA.txt.*TAB.txt|TAB.txt.*TA.txt)}}
-
-QUESTION-NOT: TAB.txt
-QUESTION: {{(TA.txt.*TB.txt|TB.txt.*TA.txt)}}
-
-QUESTION2-NOT: TA.txt
-QUESTION2-NOT: TB.txt
-QUESTION2: TAB.txt
-
-QUOTEDARGS-NOT: .txt
-QUOTEDARGS: T* T?.txt T??.txt
+RUN: echo XXA > %T/XXA.txt
+RUN: echo XXB > %T/XXB.txt
+RUN: echo XXAB > %T/XXAB.txt
+
+RUN: echo %T/XXA* | FileCheck -check-prefix=STAR %s
+RUN: echo %T/'XXA'* | FileCheck -check-prefix=STAR %s
+RUN: echo %T/XX'A'* | FileCheck -check-prefix=STAR %s
+
+RUN: echo %T/XX?.txt | FileCheck -check-prefix=QUESTION %s
+RUN: echo %T/'XX'?.txt | FileCheck -check-prefix=QUESTION %s
+
+RUN: echo %T/XX??.txt | FileCheck -check-prefix=QUESTION2 %s
+RUN: echo %T/'XX'??.txt | FileCheck -check-prefix=QUESTION2 %s
+
+RUN: echo 'XX*' 'XX?.txt' 'XX??.txt' | FileCheck -check-prefix=QUOTEDARGS %s
+
+STAR-NOT: XXB.txt
+STAR: {{(XXA.txt.*XXAB.txt|XXAB.txt.*XXA.txt)}}
+
+QUESTION-NOT: XXAB.txt
+QUESTION: {{(XXA.txt.*XXB.txt|XXB.txt.*XXA.txt)}}
+
+QUESTION2-NOT: XXA.txt
+QUESTION2-NOT: XXB.txt
+QUESTION2: XXAB.txt
+
+QUOTEDARGS-NOT: .txt
+QUOTEDARGS: XX* XX?.txt XX??.txt

llvm/trunk/test/Unit/lit.cfg

if 'TEMP' in os.environ:
     config.environment['TEMP'] = os.environ['TEMP']
 
 # Propagate path to symbolizer for ASan/MSan.
 for symbolizer in ['ASAN_SYMBOLIZER_PATH', 'MSAN_SYMBOLIZER_PATH']:
     if symbolizer in os.environ:
         config.environment[symbolizer] = os.environ[symbolizer]
 
 # Win32 seeks DLLs along %PATH%.
-if sys.platform in ['win32', 'cygwin'] and os.path.isdir(config.shlibdir):
+if sys.platform in ['win32', 'cygwin']:
+  shlibdir = getattr(config, 'shlibdir', None)
+  if shlibdir is not None and os.path.isdir(shlibdir):
       config.environment['PATH'] = os.path.pathsep.join((
               config.shlibdir, config.environment['PATH']))
 
 # Win32 may use %SYSTEMDRIVE% during file system shell operations, so propogate.
 if sys.platform == 'win32' and 'SYSTEMDRIVE' in os.environ:
     config.environment['SYSTEMDRIVE'] = os.environ['SYSTEMDRIVE']
 
 ###
 
 # Check that the object root is known.

llvm/trunk/utils/lit/lit/Test.py

def __init__(self, name, source_root, exec_root, config):
         self.exec_root = exec_root
         # The test suite configuration.
         self.config = config
 
     def getSourcePath(self, components):
         return os.path.join(self.source_root, *components)
 
     def getExecPath(self, components):
-        return os.path.join(self.exec_root, *components)
+        return os.path.join(self.exec_root, "Output", *components)
 
 class Test:
     """Test - Information on a single test instance."""
 
     def __init__(self, suite, path_in_suite, config, file_path = None):
         self.suite = suite
         self.path_in_suite = path_in_suite
         self.config = config
def setResult(self, result):
                 if self.result.code == PASS:
                     self.result.code = XPASS
                 elif self.result.code == FAIL:
                     self.result.code = XFAIL
         except ValueError as e:
             # Syntax error in an XFAIL line.
             self.result.code = UNRESOLVED
             self.result.output = str(e)
-        
     def getFullName(self):
         return self.suite.config.name + ' :: ' + '/'.join(self.path_in_suite)
 
+    def getTestBaseName(self):
+        return self.path_in_suite[-1]
+
     def getFilePath(self):
         if self.file_path:
             return self.file_path
         return self.getSourcePath()
 
     def getSourcePath(self):
         return self.suite.getSourcePath(self.path_in_suite)
 
-    def getExecPath(self):
-        return self.suite.getExecPath(self.path_in_suite)
+    def getTempFilePrefix(self):
+        return self.suite.getExecPath(self.path_in_suite) + ".tmp"
+
+    def getTempFileDir(self):
+        return os.path.dirname(self.getTempFilePrefix())
 
     def isExpectedToFail(self):
         """
         isExpectedToFail() -> bool
 
         Check whether this test is expected to fail in the current
         configuration. This check relies on the test xfails property which by
         some test formats may not be computed until the test has first been
class Test:
 
     def getJUnitXML(self):
         test_name = self.path_in_suite[-1]
         test_path = self.path_in_suite[:-1]
         safe_test_path = [x.replace(".","_") for x in test_path]
         safe_name = self.suite.name.replace(".","-")
 
         if safe_test_path:
-            class_name = safe_name + "." + "/".join(safe_test_path) 
+            class_name = safe_name + "." + "/".join(safe_test_path)
         else:
             class_name = safe_name + "." + safe_name
 
         xml = "<testcase classname='" + class_name + "' name='" + \
             test_name + "'"
         xml += " time='%.2f'" % (self.result.elapsed,)
         if self.result.code.isFailure:
             xml += ">\n\t<failure >\n" + escape(self.result.output)
             xml += "\n\t</failure>\n</testcase>"
         else:
             xml += "/>"
         return xml

llvm/trunk/utils/lit/lit/TestRunner.py

class ShellCommandResult(object):
     def __init__(self, command, stdout, stderr, exitCode, timeoutReached,
                  outputFiles = []):
         self.command = command
         self.stdout = stdout
         self.stderr = stderr
         self.exitCode = exitCode
         self.timeoutReached = timeoutReached
         self.outputFiles = list(outputFiles)
-               
 def executeShCmd(cmd, shenv, results, timeout=0):
     """
         Wrapper around _executeShCmd that handles
         timeout
     """
     # Use the helper even when no timeout is required to make
     # other code simpler (i.e. avoid bunch of ``!= None`` checks)
     timeoutHelper = TimeoutHelper(timeout)
for i,(out,err) in enumerate(procData):
                 if path is not None and mode in ('w', 'a'):
                     try:
                         with open(path, 'rb') as f:
                             data = f.read()
                     except:
                         data = None
                     if data is not None:
                         output_files.append((name, path, data))
-            
         results.append(ShellCommandResult(
             cmd.commands[i], out, err, res, timeoutHelper.timeoutReached(),
             output_files))
         if cmd.pipe_err:
             # Take the last failing exit code from the pipeline.
             if not exitCode or res != 0:
                 exitCode = res
         else:
for i,result in enumerate(results):
                 out += "# redirected output from %r:\n" % (name,)
                 data = to_string(data.decode('utf-8', errors='replace'))
                 if len(data) > 1024:
                     out += data[:1024] + "\n...\n"
                     out += "note: data was truncated\n"
                 else:
                     out += data
                 out += "\n"
-                    
         if result.stdout.strip():
             out += '# command output:\n%s\n' % (result.stdout,)
         if result.stderr.strip():
             out += '# command stderr:\n%s\n' % (result.stderr,)
         if not result.stdout.strip() and not result.stderr.strip():
             out += "note: command had no output on stdout or stderr\n"
 
         # Show the error conditions:
try:
             # Python 2, to avoid other code having to differentiate between the
             # str and unicode types.
             keyword,ln = match.groups()
             yield (line_number, to_string(keyword.decode('utf-8')),
                    to_string(ln.decode('utf-8')))
     finally:
         f.close()
 
-def getTempPaths(test):
-    """Get the temporary location, this is always relative to the test suite
-    root, not test source root."""
-    execpath = test.getExecPath()
-    execdir,execbase = os.path.split(execpath)
-    tmpDir = os.path.join(execdir, 'Output')
-    tmpBase = os.path.join(tmpDir, execbase)
-    return tmpDir, tmpBase
-
-def getDefaultSubstitutions(test, tmpDir, tmpBase, normalize_slashes=False):
+def getDefaultSubstitutions(test, normalize_slashes=False):
     sourcepath = test.getSourcePath()
     sourcedir = os.path.dirname(sourcepath)
+    tmpDir = test.getTempFileDir()
+    tmpPrefix = test.getTempFilePrefix()
+    baseName = test.getTestBaseName()
 
     # Normalize slashes, if requested.
     if normalize_slashes:
         sourcepath = sourcepath.replace('\\', '/')
         sourcedir = sourcedir.replace('\\', '/')
         tmpDir = tmpDir.replace('\\', '/')
-        tmpBase = tmpBase.replace('\\', '/')
 
     # We use #_MARKER_# to hide %% while we do the other substitutions.
     substitutions = []
     substitutions.extend([('%%', '#_MARKER_#')])
     substitutions.extend(test.config.substitutions)
-    tmpName = tmpBase + '.tmp'
-    baseName = os.path.basename(tmpBase)
     substitutions.extend([('%s', sourcepath),
                           ('%S', sourcedir),
                           ('%p', sourcedir),
                           ('%{pathsep}', os.pathsep),
-                          ('%t', tmpName),
+                          ('%t', tmpPrefix),
                           ('%basename_t', baseName),
                           ('%T', tmpDir),
                           ('#_MARKER_#', '%')])
 
     # "%/[STpst]" should be normalized.
     substitutions.extend([
             ('%/s', sourcepath.replace('\\', '/')),
             ('%/S', sourcedir.replace('\\', '/')),
             ('%/p', sourcedir.replace('\\', '/')),
-            ('%/t', tmpBase.replace('\\', '/') + '.tmp'),
+            ('%/t', tmpPrefix.replace('\\', '/')),
             ('%/T', tmpDir.replace('\\', '/')),
             ])
 
     # "%:[STpst]" are paths without colons.
     if kIsWindows:
         substitutions.extend([
                 ('%:s', re.sub(r'^(.):', r'\1', sourcepath)),
                 ('%:S', re.sub(r'^(.):', r'\1', sourcedir)),
                 ('%:p', re.sub(r'^(.):', r'\1', sourcedir)),
-                ('%:t', re.sub(r'^(.):', r'\1', tmpBase) + '.tmp'),
+                ('%:t', re.sub(r'^(.):', r'\1', tmpPrefix)),
                 ('%:T', re.sub(r'^(.):', r'\1', tmpDir)),
                 ])
     else:
         substitutions.extend([
                 ('%:s', sourcepath),
                 ('%:S', sourcedir),
                 ('%:p', sourcedir),
-                ('%:t', tmpBase + '.tmp'),
+                ('%:t', tmpPrefix),
                 ('%:T', tmpDir),
                 ])
     return substitutions
 
 def applySubstitutions(script, substitutions):
     """Apply substitutions to the script.  Allow full regular expression syntax.
     Replace each matching occurrence of regular expression pattern a with
     substitution b in line ln."""

 class ParserKind(object):
     """
     An enumeration representing the style of an integrated test keyword or
     command.
 
     TAG: A keyword taking no value. Ex 'END.'
     COMMAND: A keyword taking a list of shell commands. Ex 'RUN:'
     LIST: A keyword taking a comma-separated list of values.
-    BOOLEAN_EXPR: A keyword taking a comma-separated list of 
+    BOOLEAN_EXPR: A keyword taking a comma-separated list of
         boolean expressions. Ex 'XFAIL:'
     CUSTOM: A keyword with custom parsing semantics.
     """
     TAG = 0
     COMMAND = 1
     LIST = 2
     BOOLEAN_EXPR = 3
     CUSTOM = 4
def parseIntegratedTestScript(test, additional_parsers=[],
     builtin_parsers = [
         IntegratedTestKeywordParser('RUN:', ParserKind.COMMAND,
                                     initial_value=script),
         IntegratedTestKeywordParser('XFAIL:', ParserKind.BOOLEAN_EXPR,
                                     initial_value=test.xfails),
         IntegratedTestKeywordParser('REQUIRES:', ParserKind.BOOLEAN_EXPR,
                                     initial_value=test.requires),
         IntegratedTestKeywordParser('REQUIRES-ANY:', ParserKind.CUSTOM,
-                                    IntegratedTestKeywordParser._handleRequiresAny, 
-                                    initial_value=test.requires), 
+                                    IntegratedTestKeywordParser._handleRequiresAny,
+                                    initial_value=test.requires),
         IntegratedTestKeywordParser('UNSUPPORTED:', ParserKind.BOOLEAN_EXPR,
                                     initial_value=test.unsupported),
         IntegratedTestKeywordParser('END.', ParserKind.TAG)
     ]
     keyword_parsers = {p.keyword: p for p in builtin_parsers}
-    
     # Install user-defined additional parsers.
     for parser in additional_parsers:
         if not isinstance(parser, IntegratedTestKeywordParser):
             raise ValueError('additional parser must be an instance of '
                              'IntegratedTestKeywordParser')
         if parser.keyword in keyword_parsers:
             raise ValueError("Parser for keyword '%s' already exists"
                              % parser.keyword)
         keyword_parsers[parser.keyword] = parser
-        
     # Collect the test lines from the script.
     sourcepath = test.getSourcePath()
     for line_number, command_type, ln in \
             parseIntegratedTestScriptCommands(sourcepath,
                                               keyword_parsers.keys()):
         parser = keyword_parsers[command_type]
         parser.parseLine(line_number, ln)
         if command_type == 'END.' and parser.getValue() is True:
def parseIntegratedTestScript(test, additional_parsers=[],
     if not test.isWithinFeatureLimits():
         msg = ', '.join(test.config.limit_to_features)
         return lit.Test.Result(Test.UNSUPPORTED,
                                "Test does not require any of the features "
                                "specified in limit_to_features: %s" % msg)
 
     return script
 
-
 def _runShTest(test, litConfig, useExternalSh, script, tmpBase):
-    # Create the output directory if it does not already exist.
-    lit.util.mkdir_p(os.path.dirname(tmpBase))
-
-    execdir = os.path.dirname(test.getExecPath())
+    execdir = os.path.dirname(test.getTempFileDir())
     if useExternalSh:
         res = executeScript(test, litConfig, tmpBase, script, execdir)
     else:
         res = executeScriptInternal(test, litConfig, tmpBase, script, execdir)
     if isinstance(res, lit.Test.Result):
         return res
 
     out,err,exitCode,timeoutInfo = res
def executeShTest(test, litConfig, useExternalSh,
     if test.config.unsupported:
         return lit.Test.Result(Test.UNSUPPORTED, 'Test is unsupported')
 
     script = parseIntegratedTestScript(test)
     if isinstance(script, lit.Test.Result):
         return script
     if litConfig.noExecute:
         return lit.Test.Result(Test.PASS)
-
-    tmpDir, tmpBase = getTempPaths(test)
     substitutions = list(extra_substitutions)
-    substitutions += getDefaultSubstitutions(test, tmpDir, tmpBase,
+    substitutions += getDefaultSubstitutions(test,
                                              normalize_slashes=useExternalSh)
     script = applySubstitutions(script, substitutions)
 
     # Re-run failed tests up to test_retry_attempts times.
     attempts = 1
     if hasattr(test.config, 'test_retry_attempts'):
         attempts += test.config.test_retry_attempts
     for i in range(attempts):
-        res = _runShTest(test, litConfig, useExternalSh, script, tmpBase)
+        res = _runShTest(test, litConfig, useExternalSh, script,
+                         test.getTempFilePrefix())
         if res.code != Test.FAIL:
             break
     # If we had to run the test more than once, count it as a flaky pass. These
     # will be printed separately in the test summary.
     if i > 0 and res.code == Test.PASS:
         res.code = Test.FLAKYPASS
     return res

llvm/trunk/utils/lit/lit/run.py

 import os
+import shutil
 import sys
 import threading
 import time
 import traceback
 try:
     import Queue as queue
 except ImportError:
     import queue
def execute_tests(self, display, jobs, max_time=None):
         Upon completion, each test in the run will have its result
         computed. Tests which were not actually executed (for any reason) will
         be given an UNRESOLVED result.
         """
         # Don't do anything if we aren't going to run any tests.
         if not self.tests or jobs == 0:
             return
 
+        # Create fresh output directories for each test we're going to run.
+        # This guarantees that test runs will not remnants of previous test
+        # runs' output.
+        clean_paths = set()
+        for test in self.tests:
+            clean_paths.add(os.path.normpath(test.getTempFileDir()))
+        clean_paths = list(clean_paths)
+        # Sort by number of path components, to ensure that parent directories
+        # get deleted and re-created before child directories.
+        clean_paths.sort(key=lambda x: len(x.split(os.sep)))
+        for base in clean_paths:
+            if os.path.exists(base):
+                if not os.path.islink(base) and os.path.isdir(base):
+                    shutil.rmtree(base, True)
+                else:
+                    os.unlink(os.path)
+            if not os.path.exists(base):
+                lit.util.mkdir_p(base)
+
         # Set up semaphores to limit parallelism of certain classes of tests.
         # For example, some ASan tests require lots of virtual memory and run
         # faster with less parallelism on OS X.
         self.parallelism_semaphores = \
                 {k: multiprocessing.Semaphore(v) for k, v in
                  self.lit_config.parallelism_groups.items()}
 
         # Install a console-control signal handler on Windows.