[test] Split LLDB tests into filecheck, unittest and dotest.

[JDevlieghere]

Summary

LLDB has three major testing strategies: unit tests, tests that exercise the SB API though dotest.py and what we currently call lit tests. The later is rather confusing as we're now using lit as the driver for all three types of tests. As most of this grew organically, the directory structure in the lldb repository doesn't really make this clear. The 'lit' tests are part of the root and among these tests there's a Unit and Suite folder for the unit and dotest-tests. This makes it impossible to run just the lit tests.

This patch changes the directory layout to match the 3 testing strategies, each with their own directory and their own configuration file. I've opted to name the directories after the testing tool, so dotest for the dotest.py tests and filecheck for the lit tests. This means there are now 3 directories under lit with 3 corresponding targets in addition to the usual check-lldb:

• check-lldb-unittests (replaces check-lldb-unit)
• check-lldb-dotest
• check-lldb-filecheck

Each directory defines its own lit test suite (which was already the case). The top level directory also has a lit.cfg.py which is there for check-lldb and is mostly empty.

Current layout:

lit
├── CMakeLists.txt
├── lit-lldb-init.in
├── lit.cfg.py
├── lit.site.cfg.py.in
├── Breakpoint
├── BuildScript
├── Commands
├── Driver
├── ExecControl
├── Expr
├── Heap
├── Host
├── Minidump
├── Modules
├── ObjectFile
├── Process
├── Python
├── Quit
├── Register
├── Reproducer
├── Settings
├── Suite          <- dotest
├── SymbolFile
├── Unit           <- unittests
├── Unwind
├── Watchpoint
├── dotest
├── filecheck
├── helper
└── lldb

New layout:

lit
├── CMakeLists.txt
├── lit.cfg.py
├── dotest
├── filecheck
└── unittests

My goal is to move the unit test and dotest files into their corresponding subdirectories, remove them from the root and finally rename lit to tests. This is the first step in that direction.

[jingham]

This seems much clearer to me.

[JDevlieghere]

Essentially this patch just moves all the filecheck tests in a directory called filecheck and renames the Suite and Unit directories to dotest and unittests respectively.

[thopre]

Shouldn't the CMake target remain check-lldb-unit to be consistant with check-llvm-unit/check-clang-unit etc.?

[mstorsjo]

and finally rename lit to tests

Shouldn't that be test, to be consistent with llvm/lld/clang and others?

[labath]

In D68606#1699197, @thopre wrote:

Shouldn't the CMake target remain check-lldb-unit to be consistant with check-llvm-unit/check-clang-unit etc.?

In D68606#1699219, @mstorsjo wrote:

and finally rename lit to tests

Shouldn't that be test, to be consistent with llvm/lld/clang and others?

+1 to both of these.

More generally, there are other things here which are not consistent with llvm layouts, and I am wondering how much should try to be compatible with that. On one hand, consistency is good, but on the other, llvm does not have _three_ kinds of tests to worry about. For instance, test/Unit *is* consistent with the llvm location of unit tests, and by extension test/Suite could be considered the "right place" for non-filecheck tests too (though maybe it shouldn't be called "Suite"). Speaking of that, I am not completely sure about the name "filecheck" either -- technically this should be "shtest" as that is their main distinguishing feature, though that may end up being too pedantic.

[tatyana-krasnukha]

I have longed for this change, thank you!

[JDevlieghere]

We discussed this on IRC a bit and this is my updated proposal for the directory names:

• API
• Shell
• Unit

The corresponding targets are lldb-check-unit (unchanged), lldb-check-shell (runs just the FileCheck / ShTests) and lldb-check-api (runs just the dotest.py tests). I've also added a README to both API and Unit explaining where the actual tests can be found.

[JDevlieghere]

Rename lit to test to decrease the number of renames in the git history.

[jankratochvil]

Before this patch I was getting:

#rm -rf *
cmake ~/redhat/llvm-monorepo2/llvm/ -DCMAKE_BUILD_TYPE=Release  -DLLVM_USE_LINKER=gold -DLLVM_ENABLE_PROJECTS="lldb;clang;lld"  -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DLLVM_ENABLE_ASSERTIONS=ON
make check-lldb
...
llvm-lit: /home/jkratoch/redhat/llvm-monorepo2/llvm/utils/lit/lit/llvm/subst.py:134: fatal: Did not find count in /home/jkratoch/redhat/llvm-monorepo2-clangassert/./bin

After this patch I am getting:

[100%] Running lldb lit test suite
llvm-lit: /home/jkratoch/redhat/llvm-monorepo2/llvm/utils/lit/lit/discovery.py:133: warning: unable to find test suite for '/home/jkratoch/redhat/llvm-monorepo2-clangassert/tools/lldb/test'
llvm-lit: /home/jkratoch/redhat/llvm-monorepo2/llvm/utils/lit/lit/discovery.py:244: warning: input '/home/jkratoch/redhat/llvm-monorepo2-clangassert/tools/lldb/test' contained no tests
Testing Time: 0.00s
2 warning(s) in tests.
[100%] Built target check-lldb-lit
Scanning dependencies of target check-lldb
[100%] Built target check-lldb

A Fedora buildbot has not yet built this release: http://lab.llvm.org:8014/builders/lldb-x86_64-fedora?numbuilds=1000

lldb/lit/CMakeLists.txt

   LLVM_ENABLE_ZLIB
   LLVM_ENABLE_SHARED_LIBS
   LLDB_IS_64_BITS)
 
 configure_lit_site_cfg(
-  ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.py.in
-  ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg.py
+  ${CMAKE_CURRENT_SOURCE_DIR}/filecheck/lit.site.cfg.py.in
+  ${CMAKE_CURRENT_BINARY_DIR}/filecheck/lit.site.cfg.py
   MAIN_CONFIG
-  ${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.py)
+  ${CMAKE_CURRENT_SOURCE_DIR}/filecheck/lit.cfg.py)
 configure_lit_site_cfg(
-  ${CMAKE_CURRENT_SOURCE_DIR}/Unit/lit.site.cfg.py.in
-  ${CMAKE_CURRENT_BINARY_DIR}/Unit/lit.site.cfg.py
+  ${CMAKE_CURRENT_SOURCE_DIR}/unittests/lit.site.cfg.py.in
+  ${CMAKE_CURRENT_BINARY_DIR}/unittests/lit.site.cfg.py
   MAIN_CONFIG
-  ${CMAKE_CURRENT_SOURCE_DIR}/Unit/lit.cfg.py)
+  ${CMAKE_CURRENT_SOURCE_DIR}/unittests/lit.cfg.py)
 configure_lit_site_cfg(
-  ${CMAKE_CURRENT_SOURCE_DIR}/Suite/lit.site.cfg.in
-  ${CMAKE_CURRENT_BINARY_DIR}/Suite/lit.site.cfg)
+  ${CMAKE_CURRENT_SOURCE_DIR}/dotest/lit.site.cfg.in
+  ${CMAKE_CURRENT_BINARY_DIR}/dotest/lit.site.cfg)
 
 configure_file(
-  ${CMAKE_CURRENT_SOURCE_DIR}/lit-lldb-init.in
-  ${CMAKE_CURRENT_BINARY_DIR}/lit-lldb-init)
+  ${CMAKE_CURRENT_SOURCE_DIR}/filecheck/lit-lldb-init.in
+  ${CMAKE_CURRENT_BINARY_DIR}/filecheck/lit-lldb-init)
 
 if(NOT LLDB_BUILT_STANDALONE)
   list(APPEND LLDB_TEST_DEPS
     FileCheck
     count

lldb/lit/dotest/lit.cfg

 import shlex
 
 import lit.formats
 
 # name: The name of this test suite.
-config.name = 'lldb-Suite'
+config.name = 'lldb-dotest'
 
 # suffixes: A list of file extensions to treat as test files.
 config.suffixes = ['.py']
 
 # test_source_root: The root path where tests are located.

 
 if config.lldb_module_cache:
   dotest_cmd += ['--module-cache-dir', config.lldb_module_cache]
 
 # Load LLDB test format.
-sys.path.append(os.path.join(config.lldb_src_root, "lit", "Suite"))
+sys.path.append(os.path.join(config.lldb_src_root, "lit", "dotest"))
 import lldbtest
 
 # testFormat: The test format to use to interpret tests.
 config.test_format = lldbtest.LLDBTest(dotest_cmd)

lldb/lit/dotest/lit.site.cfg.in

 except KeyError as e:
     key, = e.args
     lit_config.fatal("unable to find %r parameter, use '--param=%s=VALUE'" % (key,key))
 
 # Let the main config do the real work.
-lit_config.load_config(config, "@LLDB_SOURCE_DIR@/lit/Suite/lit.cfg")
+lit_config.load_config(config, "@LLDB_SOURCE_DIR@/lit/dotest/lit.cfg")

lldb/lit/filecheck/Python/crashlog.test

 #                                                                 -*- python -*-
 # REQUIRES: system-darwin
-# RUN: cd %S/../../examples/python && cat %s | %lldb | FileCheck %s
+# RUN: cd %S/../../../examples/python && cat %s | %lldb | FileCheck %s
 # CHECK-LABEL: {{S}}KIP BEYOND CHECKS
 script
 import crashlog
 cl = crashlog.CrashLog
 images = [

 # CHECK: 0x22222
 # CHECK: MyApp Pro
 # CHECK: (0)
 # CHECK: 01234
 # CHECK: /tmp/MyApp Pro.app/MyApp Pro
-    
 "0x7fff63f20000 - 0x7fff63f77ff7  libc++.1.dylib (400.9.4) /usr/lib/libc++.1.dylib"
 # CHECK: 0x7fff63f20000
 # CHECK: 0x7fff63f77ff7
 # CHECK: libc++.1.dylib
 # CHECK: (400.9.4)

lldb/lit/filecheck/helper/toolchain.py

     if config.lldb_tools_dir:
         build_script_args.append('--tools-dir={0}'.format(config.lldb_tools_dir))
     if config.llvm_libs_dir:
         build_script_args.append('--libs-dir={0}'.format(config.llvm_libs_dir))
 
-    lldb_init = os.path.join(config.test_exec_root, 'lit-lldb-init')
+    lldb_init = os.path.join(config.test_exec_root, 'filecheck', 'lit-lldb-init')
 
     primary_tools = [
         ToolSubst('%lldb',
                   command=FindTool('lldb'),
                   extra_args=['--no-lldbinit', '-S', lldb_init]),

lldb/lit/filecheck/lit.cfg.py

+# -*- Python -*-
+
+import os
+import platform
+import re
+import shutil
+import site
+import sys
+
+import lit.formats
+from lit.llvm import llvm_config
+from lit.llvm.subst import FindTool
+from lit.llvm.subst import ToolSubst
+from distutils.spawn import find_executable
+
+site.addsitedir(os.path.dirname(__file__))
+from helper import toolchain
+
+# name: The name of this test suite.
+config.name = 'lldb-filecheck'
+
+# testFormat: The test format to use to interpret tests.
+config.test_format = lit.formats.ShTest(not llvm_config.use_lit_shell)
+
+# suffixes: A list of file extensions to treat as test files. This is overriden
+# by individual lit.local.cfg files in the test subdirectories.
+config.suffixes = ['.test', '.cpp', '.s']
+
+# excludes: A list of directories to exclude from the testsuite. The 'Inputs'
+# subdirectories contain auxiliary inputs for various tests in their parent
+# directories.
+config.excludes = ['Inputs', 'CMakeLists.txt', 'README.txt', 'LICENSE.txt']
+
+# test_source_root: The root path where tests are located.
+config.test_source_root = os.path.dirname(__file__)
+
+# test_exec_root: The root path where tests should be run.
+config.test_exec_root = os.path.join(config.lldb_obj_root, 'lit')
+
+
+llvm_config.use_default_substitutions()
+
+toolchain.use_lldb_substitutions(config)
+
+toolchain.use_support_substitutions(config)
+
+
+if re.match(r'^arm(hf.*-linux)|(.*-linux-gnuabihf)', config.target_triple):
+    config.available_features.add("armhf-linux")
+
+def calculate_arch_features(arch_string):
+    # This will add a feature such as x86, arm, mips, etc for each built
+    # target
+    features = []
+    for arch in arch_string.split():
+        features.append(arch.lower())
+    return features
+
+# Run llvm-config and add automatically add features for whether we have
+# assertions enabled, whether we are in debug mode, and what targets we
+# are built for.
+llvm_config.feature_config(
+    [('--assertion-mode', {'ON': 'asserts'}),
+     ('--build-mode', {'DEBUG': 'debug'}),
+     ('--targets-built', calculate_arch_features)
+     ])
+
+# Clean the module caches in the test build directory. This is necessary in an
+# incremental build whenever clang changes underneath, so doing it once per
+# lit.py invocation is close enough.
+for cachedir in [config.clang_module_cache, config.lldb_module_cache]:
+    if os.path.isdir(cachedir):
+        print("Deleting module cache at %s."%cachedir)
+        shutil.rmtree(cachedir)
+
+# Set a default per-test timeout of 10 minutes. Setting a timeout per test
+# requires that killProcessAndChildren() is supported on the platform and
+# lit complains if the value is set but it is not supported.
+supported, errormsg = lit_config.maxIndividualTestTimeIsSupported
+if supported:
+    lit_config.maxIndividualTestTime = 600
+else:
+    lit_config.warning("Could not set a default per-test timeout. " + errormsg)
+
+
+# If running tests natively, check for CPU features needed for some tests.
+
+if 'native' in config.available_features:
+    cpuid_exe = lit.util.which('lit-cpuid', config.lldb_tools_dir)
+    if cpuid_exe is None:
+        lit_config.warning("lit-cpuid not found, tests requiring CPU extensions will be skipped")
+    else:
+        out, err, exitcode = lit.util.executeCommand([cpuid_exe])
+        if exitcode == 0:
+            for x in out.split():
+                config.available_features.add('native-cpu-%s' % x)
+        else:
+            lit_config.warning("lit-cpuid failed: %s" % err)
+
+if not config.lldb_disable_python:
+    config.available_features.add('python')
+
+if config.lldb_enable_lzma:
+    config.available_features.add('lzma')
+
+if find_executable('xz') != None:
+    config.available_features.add('xz')

lldb/lit/filecheck/lit.site.cfg.py.in

 
 import lit.llvm
 lit.llvm.initialize(lit_config, config)
 
 # Let the main config do the real work.
-lit_config.load_config(config, "@LLDB_SOURCE_DIR@/lit/lit.cfg.py")
+lit_config.load_config(config, "@LLDB_SOURCE_DIR@/lit/filecheck/lit.cfg.py")

lldb/lit/lit.cfg.py

 from lit.llvm import llvm_config
 from lit.llvm.subst import FindTool
 from lit.llvm.subst import ToolSubst
 from distutils.spawn import find_executable
 
-site.addsitedir(os.path.dirname(__file__))
-from helper import toolchain
+# This is the top level configuration. Most of these configuration options will
+# be overriden by individual lit configuration files in the test
+# subdirectories.
 
 # name: The name of this test suite.
 config.name = 'LLDB'
 
 # testFormat: The test format to use to interpret tests.

 config.test_source_root = os.path.dirname(__file__)
 
 # test_exec_root: The root path where tests should be run.
 config.test_exec_root = os.path.join(config.lldb_obj_root, 'lit')
 
-
 llvm_config.use_default_substitutions()
-
-toolchain.use_lldb_substitutions(config)
-
-toolchain.use_support_substitutions(config)
-
-
-if re.match(r'^arm(hf.*-linux)|(.*-linux-gnuabihf)', config.target_triple):
-    config.available_features.add("armhf-linux")
-
-def calculate_arch_features(arch_string):
-    # This will add a feature such as x86, arm, mips, etc for each built
-    # target
-    features = []
-    for arch in arch_string.split():
-        features.append(arch.lower())
-    return features
-
-# Run llvm-config and add automatically add features for whether we have
-# assertions enabled, whether we are in debug mode, and what targets we
-# are built for.
-llvm_config.feature_config(
-    [('--assertion-mode', {'ON': 'asserts'}),
-     ('--build-mode', {'DEBUG': 'debug'}),
-     ('--targets-built', calculate_arch_features)
-     ])
-
-# Clean the module caches in the test build directory. This is necessary in an
-# incremental build whenever clang changes underneath, so doing it once per
-# lit.py invocation is close enough.
-for cachedir in [config.clang_module_cache, config.lldb_module_cache]:
-    if os.path.isdir(cachedir):
-        print("Deleting module cache at %s."%cachedir)
-        shutil.rmtree(cachedir)
-
-# Set a default per-test timeout of 10 minutes. Setting a timeout per test
-# requires that killProcessAndChildren() is supported on the platform and
-# lit complains if the value is set but it is not supported.
-supported, errormsg = lit_config.maxIndividualTestTimeIsSupported
-if supported:
-    lit_config.maxIndividualTestTime = 600
-else:
-    lit_config.warning("Could not set a default per-test timeout. " + errormsg)
-
-
-# If running tests natively, check for CPU features needed for some tests.
-
-if 'native' in config.available_features:
-    cpuid_exe = lit.util.which('lit-cpuid', config.lldb_tools_dir)
-    if cpuid_exe is None:
-        lit_config.warning("lit-cpuid not found, tests requiring CPU extensions will be skipped")
-    else:
-        out, err, exitcode = lit.util.executeCommand([cpuid_exe])
-        if exitcode == 0:
-            for x in out.split():
-                config.available_features.add('native-cpu-%s' % x)
-        else:
-            lit_config.warning("lit-cpuid failed: %s" % err)
-
-if not config.lldb_disable_python:
-    config.available_features.add('python')
-
-if config.lldb_enable_lzma:
-    config.available_features.add('lzma')
-
-if find_executable('xz') != None:
-    config.available_features.add('xz')

lldb/lit/unittests/lit.cfg.py

 
 import lit.formats
 from lit.llvm import llvm_config
 
 # name: The name of this test suite.
-config.name = 'lldb-Unit'
+config.name = 'lldb-unittests'
 
 # suffixes: A list of file extensions to treat as test files.
 config.suffixes =  []
 
 # test_source_root: The root path where unit test binaries are located.

lldb/lit/unittests/lit.site.cfg.py.in

 
 import lit.llvm
 lit.llvm.initialize(lit_config, config)
 
 # Let the main config do the real work.
-lit_config.load_config(config, "@LLDB_SOURCE_DIR@/lit/Unit/lit.cfg.py")
+lit_config.load_config(config, "@LLDB_SOURCE_DIR@/lit/unittests/lit.cfg.py")