Add new lit argument: --exec-feature=FEATURE

[samsonov]

Summary

This patch adds a new option for lit test runner: --exec-feature=<feature>
This option allows to specify test execution mode and can be used in REQUIRES:
and XFAIL: lines. Particular use case: we are running LLVM/Clang tests under
AddressSanitizer to find memory errors, and there are some "known" buganized
failures like http://llvm.org/bugs/show_bug.cgi?id=15130. With this patch applied,
we can mark these tests "XFAIL: asan", specify LLVM_LIT_ARGS="--exec-feature=asan"
at configuration time, and keep the build clean.

[samsonov]

Ping.

[• ddunbar]

Hi Alexey,

This shouldn't require adding a new lit option.

Ideally, I would propose the right way to handle this would be to have some
way to detect that the compile is done with ASAN support enabled (if the
support is enabled with a configure flag, just set a new build variable).
For example, see how "llvm_use_intel_jit" gets added to the lit features
for LLVM. This way things will "just work" if people build with ASAN
enabled.

Is there a reason that approach doesn't work here? If not, then I still
would prefer using the single existing lit mechanism --param to pass in the
parameter.

• Daniel

[eugenis]

Daniel, thanks for the pointer. It should work for us.

We can detect asan-ized build (either by searching through cflags, or by a compile+link+objdump+grep __asan_init configuration test).

[samsonov]

Hi Daniel,

Thanks for pointing out at llvm_use_intel_jitevents! I think we can benefit from adding a build configuration options that would tell
the build system to build LLVM/Clang under ASan/MSan/whatever. Then we can setup the correct compile flags, alter llvm-lit options
etc.

Still, I think this is somewhat orthogonal to this patch: we want to mark certain tests as "xfailing" under ASan, not "unsupported".
It's also good if we have per-test granularity instead of per-directory offered by lit.site.cfg (which is far less discoverable).

Probably, there is another way to achieve the goal - specify "asan" in --param flag, but this would also require untrivial
changes to lit to make it respect --param values in XFAIL: lines.

P.S. One more random crazy idea: we can later pull --vg and --vg-leak under --exec-feature flag :)

docs/CommandGuide/lit.rst

 
 .. option:: --time-tests
 
  Track the wall time individual tests take to execute and includes the results
  in the summary output.  This is useful for determining which tests in a test
  suite take the most time to execute.  Note that this option is most useful
  with ``-j 1``.
 
+.. option:: --exec-feature=FEATURE
+
+ Specify an additional feature of test execution that can be matched in REQUIRES
+ and XFAIL lines. For example, if you run tests under AddressSanitizer, you can
+ specify ``--exec-feature=asan``.
+ 
 .. _selection-options:
 
 SELECTION OPTIONS
 -----------------
 
 .. option:: --max-tests=N
 
  Run at most ``N`` tests and then terminate.

utils/lit/lit/LitConfig.py

class LitConfig:
     # Provide access to built-in formats.
     import TestFormats as formats
 
     # Provide access to built-in utility functions.
     import Util as util
 
     def __init__(self, progname, path, quiet,
                  useValgrind, valgrindLeakCheck, valgrindArgs,
-                 noExecute, ignoreStdErr, debug, isWindows,
+                 noExecute, execFeatures, ignoreStdErr, debug, isWindows,
                  params, config_prefix = None):
         # The name of the test runner.
         self.progname = progname
         # The items to add to the PATH environment variable.
         self.path = list(map(str, path))
         self.quiet = bool(quiet)
         self.useValgrind = bool(useValgrind)
         self.valgrindLeakCheck = bool(valgrindLeakCheck)
         self.valgrindUserArgs = list(valgrindArgs)
         self.noExecute = noExecute
+        self.execFeatures = list(execFeatures)
         self.ignoreStdErr = ignoreStdErr
         self.debug = debug
         self.isWindows = bool(isWindows)
         self.params = dict(params)
         self.bashPath = None
 
         # Configuration files to look for when discovering test suites.
         self.config_prefix = config_prefix or 'lit'
         self.config_name = '%s.cfg' % (self.config_prefix,)
         self.site_config_name = '%s.site.cfg' % (self.config_prefix,)
         self.local_config_name = '%s.local.cfg' % (self.config_prefix,)
 
         self.numErrors = 0
         self.numWarnings = 0
 
         self.valgrindArgs = []
         if self.useValgrind:
+            self.execFeatures.append('valgrind')
             self.valgrindArgs = ['valgrind', '-q', '--run-libc-freeres=no',
                                  '--tool=memcheck', '--trace-children=yes',
                                  '--error-exitcode=123']
             if self.valgrindLeakCheck:
+                self.execFeatures.append('vg_leak')
                 self.valgrindArgs.append('--leak-check=full')
             else:
                 # The default is 'summary'.
                 self.valgrindArgs.append('--leak-check=no')
             self.valgrindArgs.extend(self.valgrindUserArgs)
 
 
     def load_config(self, config, path):

utils/lit/lit/TestingConfig.py

def frompath(path, parent, litConfig, mustExist, config = None):
                 environment.update({
                         'INCLUDE' : os.environ.get('INCLUDE',''),
                         'PATHEXT' : os.environ.get('PATHEXT',''),
                         'PYTHONUNBUFFERED' : '1',
                         'TEMP' : os.environ.get('TEMP',''),
                         'TMP' : os.environ.get('TMP',''),
                         })
 
-            # Set the default available features based on the LitConfig.
-            available_features = []
-            if litConfig.useValgrind:
-                available_features.append('valgrind')
-                if litConfig.valgrindLeakCheck:
-                    available_features.append('vg_leak')
-
             config = TestingConfig(parent,
                                    name = '<unnamed>',
                                    suffixes = set(),
                                    test_format = None,
                                    environment = environment,
                                    substitutions = [],
                                    unsupported = False,
                                    on_clone = None,
                                    test_exec_root = None,
                                    test_source_root = None,
                                    excludes = [],
-                                   available_features = available_features)
+                                   available_features = litConfig.execFeatures)
 
         if os.path.exists(path):
             # FIXME: Improve detection and error reporting of errors in the
             # config file.
             f = open(path)
             cfg_globals = dict(globals())
             cfg_globals['config'] = config
             cfg_globals['lit'] = litConfig

utils/lit/lit/discovery.py

def load_test_suite(inputs):
     # Create the global config object.
     litConfig = LitConfig.LitConfig(progname = 'lit',
                                     path = [],
                                     quiet = False,
                                     useValgrind = False,
                                     valgrindLeakCheck = False,
                                     valgrindArgs = [],
                                     noExecute = False,
+                                    execFeatures = [],
                                     ignoreStdErr = False,
                                     debug = False,
                                     isWindows = (platform.system()=='Windows'),
                                     params = {})
 
     tests = find_tests_for_inputs(litConfig, inputs)
 
     # Return a unittest test suite which just runs the tests in order.
     return unittest.TestSuite([LitTestCase(test, litConfig) for test in tests])
 

utils/lit/lit/main.py

group.add_option("", "--vg-arg", dest="valgrindArgs", metavar="ARG",
                      help="Specify an extra argument for valgrind",
                      type=str, action="append", default=[])
     group.add_option("", "--time-tests", dest="timeTests",
                      help="Track elapsed wall time for each test",
                      action="store_true", default=False)
     group.add_option("", "--no-execute", dest="noExecute",
                      help="Don't execute any tests (assume PASS)",
                      action="store_true", default=False)
+    group.add_option("", "--exec-feature", dest="execFeatures",
+                     metavar="FEATURE",
+                     help="Add user-defined feature of test execution",
+                     type=str, action="append", default=[])
     parser.add_option_group(group)
 
     group = OptionGroup(parser, "Test Selection")
     group.add_option("", "--max-tests", dest="maxTests", metavar="N",
                      help="Maximum number of tests to run",
                      action="store", type=int, default=None)
     group.add_option("", "--max-time", dest="maxTime", metavar="N",
                      help="Maximum time to spend testing (in seconds)",
# threads by default there.
     # Create the global config object.
     litConfig = LitConfig.LitConfig(progname = os.path.basename(sys.argv[0]),
                                     path = opts.path,
                                     quiet = opts.quiet,
                                     useValgrind = opts.useValgrind,
                                     valgrindLeakCheck = opts.valgrindLeakCheck,
                                     valgrindArgs = opts.valgrindArgs,
                                     noExecute = opts.noExecute,
+                                    execFeatures = opts.execFeatures,
                                     ignoreStdErr = False,
                                     debug = opts.debug,
                                     isWindows = (platform.system()=='Windows'),
                                     params = userParams,
                                     config_prefix = opts.configPrefix)
 
     tests = lit.discovery.find_tests_for_inputs(litConfig, inputs)