[lit] Add LIT_DEBUG environment variable to support passing options though ninja

[george.karpenkov]

Summary

Currently, there two useful techniques for debugging failing tests from LIT:

• -vv flag shows every line as it is executed
• FILECHECK_DUMP_INPUT_ON_FAILURE tells FileCheck to show the original input which failed the comparison.

Note that -vv flag can not be passed through ninja, and a manual LIT invocation is required.
This patch proposes a unification of such debugging features under a common umbrella, so that each flag would not have to be memorized separately.

[jdenny]

llvm/utils/lit/lit/TestingConfig.py

24–25

If LIT_DEBUG=2 and FILECHECK_DUMP_INPUT_ON_FAILURE=0 are specified in the environment, does the former clobber the latter? Should it?

llvm/utils/lit/lit/main.py

204–208

If LIT_DEBUG>2, does debug_level=0? It seems you should have a diagnostic for unexpected LIT_DEBUG values, or you should at least let larger than expected values imply the largest expected value.

llvm/utils/lit/tests/Inputs/test_verbose_filecheck

11–13

I haven't tried out your dump feature yet, but I would've expected that CHECK-NEXT makes sense for the last three directives.

llvm/utils/lit/tests/shtest-run-at-line.py

6–7

If we change this to LIT_DEBUG=2, will this test still pass? It looks like it would, but I might have missed something.

I'm thinking that, by adding a few more FileCheck commands here (with different prefixes), you could easily extend this test to ensure that LIT_DEBUG=1 doesn't dump the file and that LIT_DEBUG=2 does dump it. Other tests already test that dumping works correctly, so here you could just test that it's been activated or not activated by looking for existence or absence of "Full input was".

Or you could put those kinds of checks in your other test, which currently only checks for file dumping at LIT_DEBUG=2.

[jdenny]

@george.karpenkov: I don't know what your current plans are for this patch. I recall you once mentioned wanting it to set FileCheck's -v. D53517 could help with that, and it addresses other uses cases, such as color.

llvm/utils/lit/lit/LitConfig.py

class LitConfig(object):
 
     def __init__(self, progname, path, quiet,
                  useValgrind, valgrindLeakCheck, valgrindArgs,
                  noExecute, debug, isWindows, singleProcess,
                  params, config_prefix = None,
                  maxIndividualTestTime = 0,
                  maxFailures = None,
                  parallelism_groups = {},
-                 echo_all_commands = False):
+                 echo_all_commands = False,
+                 debug_level = 0):
         # The name of the test runner.
         self.progname = progname
         # The items to add to the PATH environment variable.
         self.path = [str(p) for p in path]
         self.quiet = bool(quiet)
         self.useValgrind = bool(useValgrind)
         self.valgrindLeakCheck = bool(valgrindLeakCheck)
         self.valgrindUserArgs = list(valgrindArgs)
def __init__(self, progname, path, quiet,
                 # The default is 'summary'.
                 self.valgrindArgs.append('--leak-check=no')
             self.valgrindArgs.extend(self.valgrindUserArgs)
 
         self.maxIndividualTestTime = maxIndividualTestTime
         self.maxFailures = maxFailures
         self.parallelism_groups = parallelism_groups
         self.echo_all_commands = echo_all_commands
+        self.debug_level = debug_level
 
     @property
     def maxIndividualTestTime(self):
         """
             Interface for getting maximum time to spend executing
             a single test
         """
         return self._maxIndividualTestTime

llvm/utils/lit/lit/TestingConfig.py

def fromdefaults(litConfig):
         """
         # Set the environment based on the command line arguments.
         environment = {
             'PATH' : os.pathsep.join(litConfig.path +
                                      [os.environ.get('PATH','')]),
             'LLVM_DISABLE_CRASH_REPORT' : '1',
             }
 
+        if (litConfig.debug_level >= 2):
+            environment['FILECHECK_DUMP_INPUT_ON_FAILURE'] = '1'
+
         pass_vars = ['LIBRARY_PATH', 'LD_LIBRARY_PATH', 'SYSTEMROOT', 'TERM',
                      'CLANG', 'LD_PRELOAD', 'ASAN_OPTIONS', 'UBSAN_OPTIONS',
                      'LSAN_OPTIONS', 'ADB', 'ANDROID_SERIAL',
                      'SANITIZER_IGNORE_CVE_2016_2143', 'TMPDIR', 'TMP', 'TEMP',
                      'TEMPDIR', 'AVRLIT_BOARD', 'AVRLIT_PORT',
                      'FILECHECK_DUMP_INPUT_ON_FAILURE']
         for var in pass_vars:
             val = os.environ.get(var, '')

llvm/utils/lit/lit/main.py

finally:
             try:
                 shutil.rmtree(lit_tmp)
             except:
                 # FIXME: Re-try after timeout on Windows.
                 pass
 
 def main_with_tmp(builtinParameters):
     parser = argparse.ArgumentParser()
+    debug_level = 0
+    if os.getenv("LIT_DEBUG") == "1":
+        debug_level = 1
+    if os.getenv("LIT_DEBUG") == "2":
+        debug_level = 2
     parser.add_argument('test_paths',
                         nargs='*',
                         help='Files or paths to include in the test suite')
-
     parser.add_argument("--version", dest="show_version",
                       help="Show version and exit",
                       action="store_true", default=False)
     parser.add_argument("-j", "--threads", dest="numThreads", metavar="N",
                       help="Number of testing threads",
                       type=int, default=None)
     parser.add_argument("--config-prefix", dest="configPrefix",
                       metavar="NAME", help="Prefix for 'lit' config files",
def main_with_tmp(builtinParameters):
     format_group.add_argument("-s", "--succinct",
                      help="Reduce amount of output",
                      action="store_true", default=False)
     format_group.add_argument("-v", "--verbose", dest="showOutput",
                      help="Show test output for failures",
                      action="store_true", default=False)
     format_group.add_argument("-vv", "--echo-all-commands",
                      dest="echoAllCommands",
-                     action="store_true", default=False,
+                     action="store_true", default=(debug_level >= 1),
                      help="Echo all commands as they are executed to stdout.\
                      In case of failure, last command shown will be the\
                      failing one.")
     format_group.add_argument("-a", "--show-all", dest="showAllOutput",
                      help="Display all commandlines and output",
                      action="store_true", default=False)
     format_group.add_argument("-o", "--output", dest="output_path",
                      help="Write test results to the provided path",
litConfig = lit.LitConfig.LitConfig(
         singleProcess = opts.singleProcess,
         debug = opts.debug,
         isWindows = isWindows,
         params = userParams,
         config_prefix = opts.configPrefix,
         maxIndividualTestTime = maxIndividualTestTime,
         maxFailures = opts.maxFailures,
         parallelism_groups = {},
-        echo_all_commands = opts.echoAllCommands)
+        echo_all_commands = opts.echoAllCommands,
+        debug_level = debug_level)
 
     # Perform test discovery.
     run = lit.run.Run(litConfig,
                       lit.discovery.find_tests_for_inputs(litConfig, inputs))
 
     # After test discovery the configuration might have changed
     # the maxIndividualTestTime. If we explicitly set this on the
     # command line then override what was set in the test configuration

llvm/utils/lit/tests/Inputs/test_verbose_filecheck

+; RUN: not FileCheck -input-file %s %s --check-prefix=CHECK1 --match-full-lines 2>&1 |  FileCheck %s --match-full-lines
+
+hello
+world
+
+; CHECK1: ciao
+; CHECK1-NEXT: world
+
+; CHECK: Full input was:
+; CHECK-NEXT: <<<<<<
+; CHECK: hello
+; CHECK: world
+; CHECK: >>>>>>

llvm/utils/lit/tests/debug.py

+# Check that LIT_DEBUG causes FileCheck to be verbose.
+
+# RUN: env LIT_DEBUG=2 %{lit} %{inputs}/test_verbose_filecheck

llvm/utils/lit/tests/shtest-run-at-line.py

 # Check that -vv makes the line number of the failing RUN command clear.
 # (-v is actually sufficient in the case of the internal shell.)
 #
 # RUN: not %{lit} -j 1 -vv %{inputs}/shtest-run-at-line > %t.out
 # RUN: FileCheck --input-file %t.out %s
+# RUN: env LIT_DEBUG=1 not %{lit} -j 1 %{inputs}/shtest-run-at-line > %t.out
+# RUN: FileCheck --input-file %t.out %s
 #
 # END.
 
 
 # CHECK: Testing: 4 tests
 
 
 # In the case of the external shell, we check for only RUN lines in stderr in