[lit] Add knob --run-prefix to allow developers to run LIT on a simulator when real machine is not available

[skan]

Summary

https://llvm.org/docs/TestSuiteGuide.html

LLVM test-suite leverages the LIT infrastructure for testing. It's a run-time test suite but sometimes we do
not have a real machine for the test. So, I'd like to introduce a mechanism to run it on a simulator by this
patch.

According to the documentations:

Intel SDE:
https://www.intel.com/content/www/us/en/developer/articles/tool/software-development-emulator.html

QEMU:
https://www.qemu.org/docs/master/user/main.html

Here are some usage examples:

llvm-lit ./llvm/test/MC/X86/abs8.s --run-prefix "sde --"
llvm-lit ./llvm/test/MC/X86/abs8.s --run-prefix "qemu-x86_64"

Although LLVM test suites supports CMAKE variable TEST_SUITE_RUN_UNDER, it does not work for me.

When I built llvm-test-suite with command like cmake -DTEST_SUITE_RUN_UNDER="sde --", and then ran llvm-lit, the result looked like

bash$ llvm-lit -v llvm-test-suite/build/SingleSource/Regression/C++/Regression-C++-2011-03-28-Bitfield.test -a 

llvm-test-suite/build/tools/timeit-target --limit-core 0 --limit-cpu 7200 --timeout 7200 --limit-file-size 104857600 --limit-rss-size 838860800 --append-exitstatus 
--redirect-output llvm-test-suite/build/SingleSource/Regression/C++/Output/Regression-C++-2011-03-28-Bitfield.test.out 
--redirect-input /dev/null --chdir llvm-test-suite/build/SingleSource/Regression/C++ --summary 
llvm-test-suite/build/SingleSource/Regression/C++/Output/Regression-C++-2011-03-28-Bitfield.test.time sde -- 
llvm-test-suite/build/SingleSource/Regression/C++/Regression-C++-2011-03-28-Bitfield

It tried to run Regression-C++-2011-03-28-Bitfield on the simulator, however the tool timeit-target was built with same flags, which has illegal instructions too. So the LIT ran fail.

In addition, it looks more reasonable to add the prefix during the run process than build process.

Discussion: https://discourse.llvm.org/t/rfc-prefix-test-invocations-with-the-given-tool-for-llvm-lit/70611

[skan]

Fix LIT test fail by setting default=''

[skan]

Abandoned this b/c I found we can utilize the CMAKE variable TEST_SUITE_RUN_UNDER to achieve this.

[skan]

Reopen this revision b/c TEST_SUITE_RUN_UNDER does not work for me.

When I build llvm-test-suite with command like cmake -DTEST_SUITE_RUN_UNDER="sde --", and then run llvm-lit, the result looks like

bash$ llvm-lit -v llvm-test-suite/build/SingleSource/Regression/C++/Regression-C++-2011-03-28-Bitfield.test -a 

llvm-test-suite/build/tools/timeit-target --limit-core 0 --limit-cpu 7200 --timeout 7200 --limit-file-size 104857600 --limit-rss-size 838860800 --append-exitstatus 
--redirect-output llvm-test-suite/build/SingleSource/Regression/C++/Output/Regression-C++-2011-03-28-Bitfield.test.out 
--redirect-input /dev/null --chdir llvm-test-suite/build/SingleSource/Regression/C++ --summary 
llvm-test-suite/build/SingleSource/Regression/C++/Output/Regression-C++-2011-03-28-Bitfield.test.time sde -- 
llvm-test-suite/build/SingleSource/Regression/C++/Regression-C++-2011-03-28-Bitfield

It tries to run Regression-C++-2011-03-28-Bitfield on the simulator, however the tool timeit-target is built with same flags, which has illegal instructions too. So the LIT run fail.

And with the knob --simulator-cmd, the test can pass.

[skan]

Rename the knob to --run-prefix to make it more generic

[pengfei]

We had a buildbot that run with sde before, though it was removed by D143552. I think current LLVM test-suite infrastructure should have supported sde already. Maybe @yinyangsx can help with your problem?

[skan]

In D150431#4339970, @pengfei wrote:

We had a buildbot that run with sde before, though it was removed by D143552. I think current LLVM test-suite infrastructure should have supported sde already. Maybe @yinyangsx can help with your problem?

Great link! It seems the variable USER_MODE_EMULATION=1 may help. Let me try.

# litsupport/modules/timeit.py

def _mutateCommandLine(context, commandline):
    timefile = context.tmpBase + ".time"
    config = context.config
    cmd = shellcommand.parse(commandline)

 

    if config.user_mode_emulation:
        # user_mode_emulation should be true if tests are being run via
        # user-mode emulation (e.g. Qemu) and thus the host version of timeit
        # should be used.
        timeit_name = "timeit"
    else:
        timeit_name = "timeit-target"

[skan]

USER_MODE_EMULATION=1 did resolve the problem. I will keep this revision open for a while in case someone expects this functionality in LIT.

llvm/utils/lit/lit/LitConfig.py

class LitConfig(object):
 
     The LitConfig object is also used to communicate with client configuration
     files, it is always passed in as the global variable 'lit' so that
     configuration files can access common functionality and internal components
     easily.
     """
 
     def __init__(self, progname, path, quiet,
-                 useValgrind, valgrindLeakCheck, valgrindArgs,
+                 useValgrind, valgrindLeakCheck, valgrindArgs, runPrefix,
                  noExecute, debug, isWindows, order,
                  params, config_prefix = None,
                  maxIndividualTestTime = 0,
                  parallelism_groups = {},
                  echo_all_commands = False):
         # The name of the test runner.
         self.progname = progname
         # The items to add to the PATH environment variable.
         self.path = [str(p) for p in path]
         self.quiet = bool(quiet)
         self.useValgrind = bool(useValgrind)
         self.valgrindLeakCheck = bool(valgrindLeakCheck)
         self.valgrindUserArgs = list(valgrindArgs)
+        self.runPrefix = str(runPrefix)
         self.noExecute = noExecute
         self.debug = debug
         self.isWindows = bool(isWindows)
         self.order = order
         self.params = dict(params)
         self.bashPath = None
 
         # Configuration files to look for when discovering test suites.

llvm/utils/lit/lit/LitTestCase.py

def load_test_suite(inputs):
     # Create the global config object.
     lit_config = lit.LitConfig.LitConfig(
         progname='lit',
         path=[],
         quiet=False,
         useValgrind=False,
         valgrindLeakCheck=False,
         valgrindArgs=[],
+        runPrefix='',
         noExecute=False,
         debug=False,
         isWindows=windows,
         order='smart',
         params={})
 
     # Perform test discovery.
     tests = lit.discovery.find_tests_for_inputs(lit_config, inputs, False)
     test_adaptors = [LitTestCase(t, lit_config) for t in tests]
 
     # Return a unittest test suite which just runs the tests in order.
     return unittest.TestSuite(test_adaptors)

llvm/utils/lit/lit/TestRunner.py

else:
             command = [bashPath, script]
         else:
             command = ['/bin/sh', script]
         if litConfig.useValgrind:
             # FIXME: Running valgrind on sh is overkill. We probably could just
             # run on clang with no real loss.
             command = litConfig.valgrindArgs + command
 
+    if litConfig.runPrefix:
+        command = litConfig.runPrefix.split() + command
+
     try:
         out, err, exitCode = lit.util.executeCommand(command, cwd=cwd,
                                        env=test.config.environment,
                                        timeout=litConfig.maxIndividualTestTime)
         return (out, err, exitCode, None)
     except lit.util.ExecuteCommandTimeoutException as e:
         return (e.out, e.err, e.exitCode, e.msg)
 

llvm/utils/lit/lit/cl_arguments.py

execution_group.add_argument("--vg-leak",
             help="Check for memory leaks under valgrind",
             action="store_true")
     execution_group.add_argument("--vg-arg",
             dest="valgrindArgs",
             metavar="ARG",
             help="Specify an extra argument for valgrind",
             action="append",
             default=[])
+    execution_group.add_argument("--run-prefix",
+            dest="runPrefix",
+            default='',
+            metavar="PREFIX",
+            help="Prefix test invocations with the given tool. This is typically "
+                 "used to run cross-compiled tests within a simulator tool.")
     execution_group.add_argument("--time-tests",
             help="Track elapsed wall time for each test",
             action="store_true")
     execution_group.add_argument("--no-execute",
             dest="noExecute",
             help="Don't execute any tests (assume PASS)",
             action="store_true")
     execution_group.add_argument("--xunit-xml-output",

llvm/utils/lit/lit/formats/googletest.py

def execute(self, test, litConfig):
         }
         test.config.environment.update(shard_env)
 
         cmd = [testPath]
         cmd = self.prepareCmd(cmd)
         if litConfig.useValgrind:
             cmd = litConfig.valgrindArgs + cmd
 
+        if litConfig.runPrefix:
+            cmd = litConfig.runPrefix.split() + cmd
+
         if litConfig.noExecute:
             return lit.Test.PASS, ''
 
         def get_shard_header(shard_env):
             shard_envs = ' '.join([k + '=' + v for k, v in shard_env.items()])
             return f"Script(shard):\n--\n%s %s\n--\n" % (shard_envs, ' '.join(cmd))
 
         shard_header = get_shard_header(shard_env)

llvm/utils/lit/lit/main.py

def main(builtin_params={}):
 
     lit_config = lit.LitConfig.LitConfig(
         progname=os.path.basename(sys.argv[0]),
         path=opts.path,
         quiet=opts.quiet,
         useValgrind=opts.useValgrind,
         valgrindLeakCheck=opts.valgrindLeakCheck,
         valgrindArgs=opts.valgrindArgs,
+        runPrefix=opts.runPrefix,
         noExecute=opts.noExecute,
         debug=opts.debug,
         isWindows=is_windows,
         order=opts.order,
         params=params,
         config_prefix=opts.configPrefix,
         echo_all_commands=opts.echoAllCommands)
 

llvm/utils/lit/tests/unit/TestRunner.py

def load_keyword_parser_lit_tests():
         """
         # Create the global config object.
         lit_config = lit.LitConfig.LitConfig(progname='lit',
                                              path=[],
                                              quiet=False,
                                              useValgrind=False,
                                              valgrindLeakCheck=False,
                                              valgrindArgs=[],
+                                             runPrefix='',
                                              noExecute=False,
                                              debug=False,
                                              isWindows=(
                                                platform.system() == 'Windows'),
                                              order='smart',
                                              params={})
         TestIntegratedTestKeywordParser.litConfig = lit_config
         # Perform test discovery.