Properly hook debuginfo-tests up to lit and CMake

[zturner]

Summary

This is all a bit hairy and magical, and I don't entirely understand how this works, but here is what I am trying to achieve:

1. debuginfo-tests needs to be able to depend on both clang and lld.
2. It needs to work both with and without the monorepo
3. It should be opt-in

Currently the assumption is that you clone it under the clang/test folder, but the fact that it needs to depend on lld means that clang isn't really the best place for it. Besides, this is already broken in the mono-repo unless you manually symlink it from clang/test which seems odd.

It currently runs automatically as part of ninja check-clang, and this satisfies the opt in requirement by saying that you opt in by cloning the repo into the clang tree the first place. With this patch, the opt-in requirement is satisfied by creating a new cmake target for it, so if you want to run it you just run ninja check-debuginfo.

In order for all of this to work, I had to give it its own lit configs, and this is where things start going crazy. These lit configs are a bit of magic to me, so I'm not sure if I did this right. I copied the clang lit configs as the starting poitn, and deleted a bunch of stuff from them that I don't think we need for the purposes of debug info tests. sanitizer stuff, clang tool stuff, etc. in any case, on windows I can now run ninja check-debuginfo and it actually runs the test suite.

They all fail for me, but this is because it's trying to run a perl script to kick off the test, and my machine doesn't have perl on it.

What i need help with is:

1. Am I going about this all wrong? Am I close?
2. Am I missing anything obvious with the lit configs? Or not obvious?
3. I don't have a non mono-repo anymore, I would appreciate it if someone could help me test the non mono-repo case.

[zturner]

Remove some debug print statements from cmake

[zturner]

print statements didn't get removed, try again.

[amccarth]

debuginfo-tests/CMakeLists.txt

30

I'm concerned that the difference between ninja check-debuginfo and ninja check-llvm-debuginfo isn't very obvious. Can we tweak the name of this one, like check-debuginfo-integration?

[aprantl]

Currently the assumption is that you clone it under the clang/test folder, but the fact that it needs to depend on lld means that clang isn't really the best place for it. Besides, this is already broken in the mono-repo unless you manually symlink it from clang/test which seems odd.

• It is supposed to test clang (the product), so being (logically) part of the clang tests seems reasonable to me.

• If you are planning to change this assumption, you might also want to change the Readme to reflect the new situation.

• There is also code/configuration in clang/test/lit.cfg that is specific to these tests that would need to be (re)moved.

All in all I am really not sure whether this is a good idea. This patch is duplicating a lot of code from clang's lit configuration and it seems likely that the two will diverge in the future when people update one but forget to update the other, leading to potentially very confusing bugs. What problem is this solving that it is worth the extra maintenance cost?

debuginfo-tests/CMakeLists.txt

23

why does it need lld? Shouldn't the system linker be good enough?

[aprantl]

debuginfo-tests/CMakeLists.txt

30

the target should be called check-clang-debuginfo. It is testing clang, not (or only indirectly) LLVM.

[rnk]

debuginfo-tests/CMakeLists.txt

30

That's the first I've heard of check-llvm-debuginfo. It looks like @beanz added that in 2015. Personally I like check-debuginfo as is.

[zturner]

debuginfo-tests/CMakeLists.txt

23

No, lld is a hard requirement. Part of what we're trying to do is test that we emit PDBs that can work with Microsoft tools. If we use the system linker, that defeats the entire purpose.

As an aside, "system linker" on Windows isn't really a thing, since systems don't really have linkers unless you install them. Which in this case would be MSVC's linker, which we already know works for obvious reasons

[rnk]

debuginfo-tests/CMakeLists.txt

23

Don't make this unconditional, do it like ASan does: add the dependency iff LLD is checked out. See how compiler-rt tests for it. We can feed lld's existence as a variable into the test suite. Besides, LLD doesn't have a functioning MachO port today. We need to disable any LLD-depending tests on Darwin.

debuginfo-tests/lit.cfg

331

This duplication sucks. We should install it in llvm's share directory next to its cmake or something, so that we can import it.

[zturner]

debuginfo-tests/CMakeLists.txt

30

In our case, it is also testing LLD's debug info, since LLD is what emits the PDB

[zturner]

debuginfo-tests/lit.cfg

331

Indeed it does. But at the same time, debug info tests don't need half of the stuff clang's is doing. Is there a good way to extract out the useful bits into something that is more general than just "these two specific projects need to share some specific functions"? Is there a way to make it useful for all projects?

[probinson]

debuginfo-tests/CMakeLists.txt

23

Don't see why lld is a "hard requirement." Surely you want clang's concept of CodeView output to work with the MSVC linker as well as with lld. Or is clang-cl no longer viewed as a drop-in replacement for cl?

debuginfo-tests/lit.cfg

290

See D37604.

[jroelofs]

Unless you're going for "make this lit.cfg as close as possible to clang's", then...

debuginfo-tests/lit.cfg

142

Probably don't need %llvmshlibdir.

143

Probably don't need %pluginext.

189

Probably don't need clang_analyze_cc1.

283

Probably don't need libgcc as an available feature.

293

Probably don't need def-fd-fs as an available feature.

[aprantl]

debuginfo-tests/CMakeLists.txt

23

The bots on green dragon, for example, run the debuginfo-tests and don't even check out the lld repository. It may be a hard requirement for windows, but certainly not on every platform.

[aprantl]

debuginfo-tests/lit.cfg

331

Is it possible to import the clang configuration somehow?

[zturner]

debuginfo-tests/CMakeLists.txt

23

Definitely we want that. For example, if you look back in the thread on llvm-dev, I posted this as an "example":

// RUN: %clangcl /Z7 %i /c /Fo%t.obj
// RUN: %lld-link /DEBUG %t.obj /out:%t.lld.exe
// RUN: %run_windbg %t.lld.exe %s
// RUN: %ms-link /DEBUG:FASTLINK %t.obj /out:%t.fastlink.exe
// RUN: %run_windbg %t.fastlink.exe %s
// DEBUGGER: bp 22
// DEBUGGER: g
// DEBUGGER: dt v
// CHECK: Local var {{.*}} Type SVal
// CHECK:  +0x000 Data : (null) 
// CHECK: +0x004 Kind : 0x85e

In this hypothetical example, I am testing clang-cl's codeview with MSVC's linker, but I'm *also* testing clang-cl's codeview with lld. In all honesty, CodeView is not that complicated. PDB is, and because of that it is actually the primary motivating factor behind getting debuginfo-tests working on Windows. Without LLD, it's impossible to test that LLVM's PDBs are valid.

So again, yes we want to test clang-cl's CodeView with MS's linker, but it's still a hard requirement that we need to test LLD's PDBs against MS's tools.

[zturner]

debuginfo-tests/CMakeLists.txt

23

That's a fair point. Reid's suggestion of making the lld dependency conditional on whether it's checked out seems like it would address that.

[zturner]

I think this addresses most of the concerns raised so far. The logic to conditionally add a dependency on CMake depends on D37637. I took rnk's advice and looked at what compiler-rt is doing, and was a little bit horrified. I can always go back to that if needed, but I decided to generalize what compiler-rt was doing and add it to LLVM's main CMake library instead, and then build off of that. I think it makes for cleaner / more easily understandable code this way.

I also talked with rnk about how to mitigate the effects of the code duplication. He can correct me if I'm misrepresenting him, but I believe we both came to the agreement that this duplication is something that has been ongoing with LLVM's lit infrastructure and configuration files for some time. That said, we both agreed that it should be possible to move a lot of this functionality into the core lit utility library. I think that's a good improvement which I'm willing to attempt in a follow-up.

In any case, +chandlerc because it seems like the # of people who understand this lit stuff is pretty small, so hopefully this elicits some additional feedback.

[zturner]

s/conditionally add a dependency on CMake/conditionally add a dependency on LLD/

[beanz]

debuginfo-tests/CMakeLists.txt

30

In LLVM & Clang we auto-generate lots of sub-check targets so they follow a convention check-${project}-${test subdirectory}. That's where the check-llvm-debuginfo target comes from, it is the <llvm>/test/DebugInfo directory of lit tests.

This is all done with the CMake function add_lit_testsuites.

[aprantl]

In D37602#865266, @zturner wrote:

I think this addresses most of the concerns raised so far.

Here's a couple of concerns that were mentioned before that are outstanding:

• The Readme.txt needs to be updated, it currently states that the repository is supposed to be cloned into tools/clang/test.
• The clang lit configuration needs to be updated to remove the debuginfo-tests-specific parts.
• This patch is adding >300 lines of extra configuration. I still don't understand what problem this patch is solving that is worth the extra maintenance effort.

[zturner]

In D37602#865313, @aprantl wrote:

In D37602#865266, @zturner wrote:

I think this addresses most of the concerns raised so far.

Here's a couple of concerns that were mentioned before that are outstanding:

• The Readme.txt needs to be updated, it currently states that the repository is supposed to be cloned into tools/clang/test.
• The clang lit configuration needs to be updated to remove the debuginfo-tests-specific parts.

Thanks, I'll take a look.

• This patch is adding >300 lines of extra configuration. I still don't understand what problem this patch is solving that is worth the extra maintenance effort.

Part of what I was hoping to achieve with this code review is someone who understands clang's lit config to come along and say "oh you can delete all of this stuff, it's not necessary for this." I also agree there's too much. I'm currently thinking about ways to sink some of it into llvm/utils/lit, since much of it is already duplicated across multiple projects.

debuginfo-tests/CMakeLists.txt

+# Test runner infrastructure for debug info integration tests. This
+# configures the test trees for use by Lit, and delegates to LLVM's
+# lit test handlers.
+
+if (CMAKE_CFG_INTDIR STREQUAL ".")
+  set(LLVM_BUILD_MODE ".")
+else ()
+  set(LLVM_BUILD_MODE "%(build_mode)s")
+endif ()
+
+set(DEBUGINFO_TESTS_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR})
+set(DEBUGINFO_TESTS_BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR})
+
+
+configure_lit_site_cfg(
+  ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
+  ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
+  )
+
+set(DEBUGINFO_TESTS_DEPS
+  clang
+  llvm-config
+  FileCheck
+  count
+  not
+  )
+
+if (LLVM_PROJECT_LLD_ENABLED)
+  list(APPEND DEBUGINFO_TESTS_DEPS lld)
+endif()
+
+add_lit_testsuite(check-debuginfo "Running debug info integration tests"
+  ${CMAKE_CURRENT_BINARY_DIR}
+  DEPENDS ${DEBUGINFO_TESTS_DEPS}
+  ARGS ${DEBUGINFO_TEST_EXTRA_ARGS}
+  )
+
+set_target_properties(check-debuginfo PROPERTIES FOLDER "Debug info tests")

debuginfo-tests/lit.cfg

+# -*- Python -*-
+
+import os
+import platform
+import re
+import subprocess
+import tempfile
+
+import lit.formats
+import lit.util
+
+# Configuration file for the 'lit' test runner.
+
+# name: The name of this test suite.
+config.name = 'Debug Info Tests'
+
+# Choose between lit's internal shell pipeline runner and a real shell.  If
+# LIT_USE_INTERNAL_SHELL is in the environment, we use that as an override.
+use_lit_shell = os.environ.get("LIT_USE_INTERNAL_SHELL")
+if use_lit_shell:
+    # 0 is external, "" is default, and everything else is internal.
+    execute_external = (use_lit_shell == "0")
+else:
+    # Otherwise we default to internal on Windows and external elsewhere, as
+    # bash on Windows is usually very slow.
+    execute_external = (not sys.platform in ['win32'])
+
+# testFormat: The test format to use to interpret tests.
+#
+# For now we require '&&' between commands, until they get globally killed and
+# the test runner updated.
+config.test_format = lit.formats.ShTest(execute_external)
+
+# suffixes: A list of file extensions to treat as test files.
+config.suffixes = ['.c', '.cpp', '.cppm', '.m', '.mm', '.cu', '.ll', '.cl', '.s', '.S', '.modulemap', '.test', '.rs']
+
+# excludes: A list of directories to exclude from the testsuite. The 'Inputs'
+# subdirectories contain auxiliary inputs for various tests in their parent
+# directories.
+config.excludes = ['Inputs', 'CMakeLists.txt', 'README.txt', 'LICENSE.txt']
+
+# test_source_root: The root path where tests are located.
+config.test_source_root = os.path.dirname(__file__)
+
+# test_exec_root: The root path where tests should be run.
+debuginfo_obj_root = getattr(config, 'debuginfo_obj_root', None)
+if debuginfo_obj_root is not None:
+    config.test_exec_root = debuginfo_obj_root
+
+# Set llvm_{src,obj}_root for use by others.
+config.llvm_src_root = getattr(config, 'llvm_src_root', None)
+config.llvm_obj_root = getattr(config, 'llvm_obj_root', None)
+
+# Clear some environment variables that might affect Clang.
+#
+# This first set of vars are read by Clang, but shouldn't affect tests
+# that aren't specifically looking for these features, or are required
+# simply to run the tests at all.
+#
+# FIXME: Should we have a tool that enforces this?
+
+# safe_env_vars = ('TMPDIR', 'TEMP', 'TMP', 'USERPROFILE', 'PWD',
+#                  'MACOSX_DEPLOYMENT_TARGET', 'IPHONEOS_DEPLOYMENT_TARGET',
+#                  'VCINSTALLDIR', 'VC100COMNTOOLS', 'VC90COMNTOOLS',
+#                  'VC80COMNTOOLS')
+possibly_dangerous_env_vars = ['COMPILER_PATH', 'RC_DEBUG_OPTIONS',
+                               'CINDEXTEST_PREAMBLE_FILE', 'LIBRARY_PATH',
+                               'CPATH', 'C_INCLUDE_PATH', 'CPLUS_INCLUDE_PATH',
+                               'OBJC_INCLUDE_PATH', 'OBJCPLUS_INCLUDE_PATH',
+                               'LIBCLANG_TIMING', 'LIBCLANG_OBJTRACKING',
+                               'LIBCLANG_LOGGING', 'LIBCLANG_BGPRIO_INDEX',
+                               'LIBCLANG_BGPRIO_EDIT', 'LIBCLANG_NOTHREADS',
+                               'LIBCLANG_RESOURCE_USAGE',
+                               'LIBCLANG_CODE_COMPLETION_LOGGING']
+# Clang/Win32 may refer to %INCLUDE%. vsvarsall.bat sets it.
+if platform.system() != 'Windows':
+    possibly_dangerous_env_vars.append('INCLUDE')
+for name in possibly_dangerous_env_vars:
+  if name in config.environment:
+    del config.environment[name]
+
+# Tweak the PATH to include the tools dir and the scripts dir.
+if debuginfo_obj_root is not None:
+    llvm_tools_dir = getattr(config, 'llvm_tools_dir', None)
+    if not llvm_tools_dir:
+        lit_config.fatal('No LLVM tools dir set!')
+    path = os.path.pathsep.join((llvm_tools_dir, config.environment['PATH']))
+    config.environment['PATH'] = path
+    llvm_libs_dir = getattr(config, 'llvm_libs_dir', None)
+    if not llvm_libs_dir:
+        lit_config.fatal('No LLVM libs dir set!')
+    path = os.path.pathsep.join((llvm_libs_dir,
+                                 config.environment.get('LD_LIBRARY_PATH','')))
+    config.environment['LD_LIBRARY_PATH'] = path
+
+###
+
+# Discover the 'clang' and 'clangcc' to use.
+
+import os
+
+def inferClang(PATH):
+    # Determine which clang to use.
+    clang = os.getenv('CLANG')
+
+    # If the user set clang in the environment, definitely use that and don't
+    # try to validate.
+    if clang:
+        return clang
+
+    # Otherwise look in the path.
+    clang = lit.util.which('clang', PATH)
+
+    if not clang:
+        lit_config.fatal("couldn't find 'clang' program, try setting "
+                         "CLANG in your environment")
+
+    return clang
+
+config.clang = inferClang(config.environment['PATH']).replace('\\', '/')
+if not lit_config.quiet:
+    lit_config.note('using clang: %r' % config.clang)
+
+config.substitutions.append( ('%PATH%', config.environment['PATH']) )
+
+# Note that when substituting %clang_cc1 also fill in the include directory of
+# the builtin headers. Those are part of even a freestanding environment, but
+# Clang relies on the driver to locate them.
+def getClangBuiltinIncludeDir(clang):
+    # FIXME: Rather than just getting the version, we should have clang print
+    # out its resource dir here in an easy to scrape form.
+    cmd = subprocess.Popen([clang, '-print-file-name=include'],
+                           stdout=subprocess.PIPE,
+                           env=config.environment)
+    if not cmd.stdout:
+      lit_config.fatal("Couldn't find the include dir for Clang ('%s')" % clang)
+    dir = cmd.stdout.read().strip()
+    if sys.platform in ['win32'] and execute_external:
+        # Don't pass dosish path separator to msys bash.exe.
+        dir = dir.replace('\\', '/')
+    # Ensure the result is an ascii string, across Python2.5+ - Python3.
+    return str(dir.decode('ascii'))
+
+def makeItaniumABITriple(triple):
+    m = re.match(r'(\w+)-(\w+)-(\w+)', triple)
+    if not m:
+      lit_config.fatal("Could not turn '%s' into Itanium ABI triple" % triple)
+    if m.group(3).lower() != 'win32':
+      # All non-win32 triples use the Itanium ABI.
+      return triple
+    return m.group(1) + '-' + m.group(2) + '-mingw32'
+
+def makeMSABITriple(triple):
+    m = re.match(r'(\w+)-(\w+)-(\w+)', triple)
+    if not m:
+      lit_config.fatal("Could not turn '%s' into MS ABI triple" % triple)
+    isa = m.group(1).lower()
+    vendor = m.group(2).lower()
+    os = m.group(3).lower()
+    if os == 'win32':
+      # If the OS is win32, we're done.
+      return triple
+    if isa.startswith('x86') or isa == 'amd64' or re.match(r'i\d86', isa):
+      # For x86 ISAs, adjust the OS.
+      return isa + '-' + vendor + '-win32'
+    # -win32 is not supported for non-x86 targets; use a default.
+    return 'i686-pc-win32'
+
+config.substitutions.append( ('%clang_cc1',
+                              '%s -cc1 -internal-isystem %s -nostdsysteminc'
+                              % (config.clang, getClangBuiltinIncludeDir(config.clang))) )
+config.substitutions.append( ('%clang_cpp', ' ' + config.clang + ' --driver-mode=cpp '))
+config.substitutions.append( ('%clang_cl', ' ' + config.clang + ' --driver-mode=cl '))
+config.substitutions.append( ('%clangxx', ' ' + config.clang + ' --driver-mode=g++ '))
+config.substitutions.append( ('%clang', ' ' + config.clang + ' ') )
+config.substitutions.append( ('%test_debuginfo', ' ' + config.llvm_src_root + '/utils/test_debuginfo.pl ') )
+config.substitutions.append( ('%itanium_abi_triple', makeItaniumABITriple(config.target_triple)) )
+config.substitutions.append( ('%ms_abi_triple', makeMSABITriple(config.target_triple)) )
+config.substitutions.append( ('%resource_dir', getClangBuiltinIncludeDir(config.clang)) )
+config.substitutions.append( ('%python', config.python_executable) )
+
+# The host triple might not be set, at least if we're compiling clang from
+# an already installed llvm.
+if config.host_triple and config.host_triple != '@LLVM_HOST_TRIPLE@':
+    config.substitutions.append( ('%target_itanium_abi_host_triple', '--target=%s' % makeItaniumABITriple(config.host_triple)) )
+else:
+    config.substitutions.append( ('%target_itanium_abi_host_triple', '') )
+
+# FIXME: Find nicer way to prohibit this.
+config.substitutions.append(
+    (' clang ', """*** Do not use 'clang' in tests, use '%clang'. ***""") )
+config.substitutions.append(
+    (' clang\+\+ ', """*** Do not use 'clang++' in tests, use '%clangxx'. ***"""))
+config.substitutions.append(
+    (' clang-cc ',
+     """*** Do not use 'clang-cc' in tests, use '%clang_cc1'. ***""") )
+config.substitutions.append(
+    (' clang -cc1 -analyze ',
+     """*** Do not use 'clang -cc1 -analyze' in tests, use '%clang_analyze_cc1'. ***""") )
+config.substitutions.append(
+    (' clang -cc1 ',
+     """*** Do not use 'clang -cc1' in tests, use '%clang_cc1'. ***""") )
+config.substitutions.append(
+    (' %clang-cc1 ',
+     """*** invalid substitution, use '%clang_cc1'. ***""") )
+config.substitutions.append(
+    (' %clang-cpp ',
+     """*** invalid substitution, use '%clang_cpp'. ***""") )
+config.substitutions.append(
+    (' %clang-cl ',
+     """*** invalid substitution, use '%clang_cl'. ***""") )
+
+# For each occurrence of a clang tool name as its own word, replace it
+# with the full path to the build directory holding that tool.  This
+# ensures that we are testing the tools just built and not some random
+# tools that might happen to be in the user's PATH.
+
+# Regex assertions to reject neighbor hyphens/dots (seen in some tests).
+# For example, don't match 'clang-check-' or '.clang-format'.
+NoPreHyphenDot = r"(?<!(-|\.))"
+NoPostHyphenDot = r"(?!(-|\.))"
+NoPostBar = r"(?!(/|\\))"
+
+tool_patterns = [r"\bFileCheck\b",
+                 r"\| \bcount\b",
+                 r"\| \bnot\b"]
+
+for pattern in tool_patterns:
+    # Extract the tool name from the pattern.  This relies on the tool
+    # name being surrounded by \b word match operators.  If the
+    # pattern starts with "| ", include it in the string to be
+    # substituted.
+    tool_match = re.match(r"^(\\)?((\| )?)\W+b([0-9A-Za-z-_]+)\\b\W*$",
+                          pattern)
+    tool_pipe = tool_match.group(2)
+    tool_name = tool_match.group(4)
+    tool_path = lit.util.which(tool_name, llvm_tools_dir)
+    if not tool_path:
+        # Warn.
+        lit_config.note('Did not find ' + tool_name + ' in ' + llvm_tools_dir)
+    else:
+        config.substitutions.append((pattern, tool_pipe + tool_path))
+
+###
+
+# Shell execution
+if execute_external:
+    config.available_features.add('shell')
+
+if config.has_lld:
+  config.available_features.add('lld')
+
+# For tests that require Darwin to run.
+# This is used by *block*.m and foreach.m.
+if platform.system() in ['Darwin']:
+    config.available_features.add('system-darwin')
+elif platform.system() in ['Windows']:
+    # For tests that require Windows to run.
+    config.available_features.add('system-windows')
+
+# debuginfo-tests are not expected to pass in a cross-compilation setup
+if config.host_triple != config.target_triple:
+    config.unsupported = true
+
+# Returns set of available features, registered-target(s), asserts and
+# compile definitions.
+def get_llvm_config_props():
+    set_of_features = set()
+    path = os.path.join(llvm_tools_dir, 'llvm-config')
+    path = os.path.normpath(path)
+    args = [path,
+            '--assertion-mode',
+            '--targets-built',
+            '--cxxflags'
+            ]
+    try:
+        cmd = subprocess.Popen(args,
+            stdout=subprocess.PIPE,
+            env=config.environment)
+    except:
+        import traceback
+        lit_config.fatal("Error executing subprocess.Popen, traceback: %s" % traceback.format_exc())
+
+    # 1st line corresponds to --assertion-mode, "ON" or "OFF".
+    line = cmd.stdout.readline().strip().decode('ascii')
+    if line == "ON":
+        set_of_features.add('asserts')
+
+    # 2nd line corresponds to --targets-built, like;
+    # AArch64 ARM CppBackend X86
+    for arch in cmd.stdout.readline().decode('ascii').split():
+        set_of_features.add(arch.lower() + '-registered-target')
+
+    # 3rd line contains compile definitions, search it to define if
+    # libstdc++ safe mode is set.
+    if re.search(r'-D_GLIBCXX_DEBUG\b', cmd.stdout.readline().decode('ascii')):
+        set_of_features.add('libstdcxx-safe-mode')
+
+    return set_of_features
+
+config.available_features.update(get_llvm_config_props())

debuginfo-tests/lit.site.cfg.in

+@LIT_SITE_CFG_IN_HEADER@
+
+import lit.util
+import sys
+
+config.llvm_src_root = "@LLVM_SOURCE_DIR@"
+config.llvm_obj_root = "@LLVM_BINARY_DIR@"
+config.llvm_tools_dir = "@LLVM_TOOLS_DIR@"
+config.llvm_libs_dir = "@LLVM_LIBS_DIR@"
+config.debuginfo_obj_root = "@DEBUGINFO_TESTS_BINARY_DIR@"
+config.debuginfo_tests_dir = "@DEBUGINFO_TESTS_SOURCE_DIR@"
+config.has_lld = lit.util.fromCmakeBoolean("@LLVM_PROJECT_LLD_ENABLED@")
+config.host_triple = "@LLVM_HOST_TRIPLE@"
+config.target_triple = "@TARGET_TRIPLE@"
+config.host_arch = "@HOST_ARCH@"
+config.python_executable = "@PYTHON_EXECUTABLE@"
+
+# Support substitution of the tools and libs dirs with user parameters. This is
+# used when we can't determine the tool dir at configuration time.
+try:
+    config.llvm_tools_dir = config.llvm_tools_dir % lit_config.params
+    config.llvm_libs_dir = config.llvm_libs_dir % lit_config.params
+except KeyError:
+    e = sys.exc_info()[1]
+    key, = e.args
+    lit_config.fatal("unable to find %r parameter, use '--param=%s=VALUE'" % (key,key))
+
+# Let the main config do the real work.
+lit_config.load_config(config, "@DEBUGINFO_TESTS_SOURCE_DIR@/lit.cfg")

llvm/CMakeLists.txt

 # one for llvm+clang+... using the same sources.
 set(LLVM_ALL_PROJECTS "clang;libcxx;libcxxabi;lldb;compiler-rt;lld;polly")
 set(LLVM_ENABLE_PROJECTS "" CACHE STRING
   "Semicolon-separated list of projects to build (${LLVM_ALL_PROJECTS}), or \"all\".")
 if( LLVM_ENABLE_PROJECTS STREQUAL "all" )
   set( LLVM_ENABLE_PROJECTS ${LLVM_ALL_PROJECTS})
 endif()
 foreach(proj ${LLVM_ENABLE_PROJECTS})
+  message("Found project ${proj}")
   set(PROJ_DIR "${CMAKE_CURRENT_SOURCE_DIR}/../${proj}")
   if(NOT EXISTS "${PROJ_DIR}" OR NOT IS_DIRECTORY "${PROJ_DIR}")
     message(FATAL_ERROR "LLVM_ENABLE_PROJECTS requests ${proj} but directory not found: ${PROJ_DIR}")
   endif()
   string(TOUPPER "${proj}" upper_proj)
   STRING(REGEX REPLACE "-" "_" upper_proj ${upper_proj})
   set(LLVM_EXTERNAL_${upper_proj}_SOURCE_DIR   "${CMAKE_CURRENT_SOURCE_DIR}/../${proj}")
   # There is a widely spread opinion that clang-tools-extra should be merged

 endif()
 
 foreach( binding ${LLVM_BINDINGS_LIST} )
   if( EXISTS "${LLVM_MAIN_SRC_DIR}/bindings/${binding}/CMakeLists.txt" )
     add_subdirectory(bindings/${binding})
   endif()
 endforeach()
 
-add_subdirectory(projects)
-
 if( LLVM_INCLUDE_TOOLS )
   add_subdirectory(tools)
 endif()
 
+add_subdirectory(projects)
+
 if( LLVM_INCLUDE_RUNTIMES )
   add_subdirectory(runtimes)
 endif()
 
 if( LLVM_INCLUDE_EXAMPLES )
   add_subdirectory(examples)
 endif()
 

llvm/projects/CMakeLists.txt

if(${LLVM_BUILD_RUNTIME})
   if(NOT LLVM_BUILD_EXTERNAL_COMPILER_RT)
     add_llvm_external_project(compiler-rt)
   endif()
 endif()
 
 add_llvm_external_project(dragonegg)
 add_llvm_external_project(parallel-libs)
 add_llvm_external_project(openmp)
+add_llvm_external_project(debuginfo-tests)

llvm/utils/lit/lit/util.py

def which(command, paths = None):
     for path in paths.split(os.pathsep):
         for ext in pathext:
             p = os.path.join(path, command + ext)
             if os.path.exists(p) and not os.path.isdir(p):
                 return p
 
     return None
 
+def fromCmakeBoolean(value):
+  # If it's already a boolean, just return it
+  if type(value) is bool:
+    return value
+
+  # If it's an integer, 0 = false, everything else is true
+  if type(value) is int:
+    return value == 0
+
+  # Only other type we handle is string.  If it's not string, return false
+  if type(value) is not string:
+    return False
+
+  return value.lower() in ["on", "true", "yes"]
+
 def checkToolsPath(dir, tools):
     for tool in tools:
         if not os.path.exists(os.path.join(dir, tool)):
             return False
     return True
 
 def whichTools(tools, paths):
     for path in paths.split(os.pathsep):