Refactor LLDB lit configuration files

[zturner]

Summary

A year or so ago, I re-wrote most of the lit infrastructure in LLVM so that it wasn't so boilerplate-y. I added lots of common helper type stuff, simplifed usage patterns, and made the code more elegant and maintainable.

We migrated to this in LLVM, clang, and lld's lit files, but not in LLDBs. This started to bite me recently, as the 4 most recent times I tried to run the lit test suite in LLDB on a fresh checkout the first thing that would happen is that python would just start crashing with unhelpful backtraces and I would have to spend time investigating.

You can reproduce this today by doing a fresh cmake generation, doing ninja lldb and then python bin/llvm-lit.py -sv ~/lldb/lit/SymbolFile at which point you'll get a segfault that tells you nothing about what your problem is.

I started trying to fix the issues with bandaids, but it became clear that the proper solution was to just bring in the work I did in the rest of the projects. The side benefit of this is that the lit configuration files become much cleaner and more understandable as a result.

[stella.stamenova]

Looks good. The formatting in lit.cfg.py is a bit messy (indentations, especially), but as long as the tests pass, this is an improvement :). Thanks!

[zturner]

In D54009#1284827, @stella.stamenova wrote:

Looks good. The formatting in lit.cfg.py is a bit messy (indentations, especially), but as long as the tests pass, this is an improvement :). Thanks!

Is there something like clang-format for Python? Happy to fix it if so. (I don't do a lot of Python so it's hard for me to eyeball it and figure out what's wrong, so I have to say it looks fine to me :))

[JDevlieghere]

Hi Zachary, looks like this broke GreenDragon: http://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/12087/console

Can you have a look please?

[zturner]

Fix incoming, sorry about that.

[stella.stamenova]

In D54009#1284839, @zturner wrote:

In D54009#1284827, @stella.stamenova wrote:

Looks good. The formatting in lit.cfg.py is a bit messy (indentations, especially), but as long as the tests pass, this is an improvement :). Thanks!

Is there something like clang-format for Python? Happy to fix it if so. (I don't do a lot of Python so it's hard for me to eyeball it and figure out what's wrong, so I have to say it looks fine to me :))

There's no clang-format for Python that I know of. Since python cares about indentation, it's always a good idea to make sure that we use the same number of spaces for all of the 'if's and such as it will occasionally lead to hard to track down bugs. There are (in the final version) of lit.cfg.py a couple of places where two spaces make the indentation and everything else uses four. It's not a big deal because of where they are, but something to look out for.

[stella.stamenova]

Several of the windows tests that invoke clang-cl have started failing recently (I am not sure exactly when) and I suspect this change is the culprit. Were you able to run the tests successfully with this change?

[zturner]

I have not run the dotest suite recently, is that where you’re seeing the
failures? I successfully ran the lit suite and unit tests though

[stella.stamenova]

In D54009#1290644, @zturner wrote:

I have not run the dotest suite recently, is that where you’re seeing the
failures? I successfully ran the lit suite and unit tests though

Yes, they are primarily in the lldb-suite but not only:

http://lab.llvm.org:8014/builders/lldb-x64-windows-ninja/builds/1129/steps/test/logs/stdio

There are some other failures, but what I suspect happened after this change is that the tests are now not picking up clang-cl correctly:

$ "clang-cl" "/Zi" "E:\build_slave\lldb-x64-windows-ninja\llvm\tools\lldb\lit\Expr/Inputs/call-function.cpp" "-o" "E:\build_slave\lldb-x64-windows-ninja\build\tools\lldb\lit\Expr\Output\TestIRMemoryMapWindows.test.tmp"
# command stderr:
clang-cl: warning: unable to find a Visual Studio installation; try running Clang from a developer command prompt [-Wmsvc-not-found]

clang-cl: error: unable to execute command: program not executable

clang-cl: error: linker command failed with exit code 1 (use -v to see invocation)

I will have some time to have a look in more detail tomorrow.

[zturner]

It’s possible we lost some environment variable propagation, that would do
it. But I’m curious how it was finding the visual studio installation
before my patch.

It also looks like it’s failing finding link.exe (we really should make
lld-link the default). Another fix is to pass -fuse-ld=lld or split the
clang-cl line to separate compiler and linker invocations

[stella.stamenova]

I haven't verified this yet - but I suspect it is now picking up the clang-cl that comes with VS rather than the one that was just built.

[zturner]

Actually maybe it’s the other way around. Are you specifying
LLDB_TEST_COMPILER? If it’s picking up VS’s version, it would definitely be
able to find link.exe

[stella.stamenova]

The builds don't specify the two options for LLDB_TEST_C/CXX_COMPILER, so they should be picking up the freshly build compilers. We don't have an option for clang-cl though - so it's never been explicitly specified.

[aleksandr.urakov]

I also have some strange failures on Windows x86 (I run tests from x64_86 MSVC command line). If I locally revert this commit and 3 fix commits right after this one, then all seems to work. Here is the failure:

C:\Work\llvm\build_x86\bin>llvm-lit.py -v C:\Work\llvm\tools\lldb\lit\SymbolFile\PDB\enums-layout.test
llvm-lit.py: C:/Work/llvm\utils\lit\lit\llvm\config.py:333: note: using C:/Work/llvm/build_x86/./bin/clang.exe: C:\Work\llvm\build_x86\bin\clang.exe
llvm-lit.py: C:/Work/llvm\utils\lit\lit\llvm\config.py:333: note: using C:/Work/llvm/build_x86/./bin/clang++.exe: C:\Work\llvm\build_x86\bin\clang++.exe
config.cc = C:\Work\llvm\build_x86\bin\clang.exe
-- Testing: 1 tests, 1 threads --
FAIL: LLDB :: SymbolFile/PDB/enums-layout.test (1 of 1)
******************** TEST 'LLDB :: SymbolFile/PDB/enums-layout.test' FAILED ********************
Script:
--
: 'RUN: at line 2';   clang-cl -m32 /Z7 /c /GS- C:\Work\llvm\tools\lldb\lit\SymbolFile\PDB/Inputs/SimpleTypesTest.cpp /o C:\Work\llvm\build_x86\tools\lldb\lit\SymbolFile\PDB\Output/SimpleTypesTest.cpp.enums.obj
: 'RUN: at line 3';   link C:\Work\llvm\build_x86\tools\lldb\lit\SymbolFile\PDB\Output/SimpleTypesTest.cpp.enums.obj /DEBUG /nodefaultlib /ENTRY:main /OUT:C:\Work\llvm\build_x86\tools\lldb\lit\SymbolFile\PDB\Output/SimpleTypesTest.cpp.enums.exe
: 'RUN: at line 4';   C:\Work\llvm\build_x86\bin\lldb-test.EXE symbols C:\Work\llvm\build_x86\tools\lldb\lit\SymbolFile\PDB\Output/SimpleTypesTest.cpp.enums.exe | C:\Work\llvm\build_x86\bin\FileCheck.EXE C:\Work\llvm\tools\lldb\lit\SymbolFile\PDB\enums-layout.test
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 2"
$ "clang-cl" "-m32" "/Z7" "/c" "/GS-" "C:\Work\llvm\tools\lldb\lit\SymbolFile\PDB/Inputs/SimpleTypesTest.cpp" "/o" "C:\Work\llvm\build_x86\tools\lldb\lit\SymbolFile\PDB\Output/SimpleTypesTest.cpp.enums.obj"
# command stderr:
C:\Work\llvm\tools\lldb\lit\SymbolFile\PDB/Inputs/SimpleTypesTest.cpp(41,9) :  error: unknown type name 'char16_t'
typedef char16_t WChar16Typedef;
        ^
C:\Work\llvm\tools\lldb\lit\SymbolFile\PDB/Inputs/SimpleTypesTest.cpp(44,9) :  error: unknown type name 'char32_t'
typedef char32_t WChar32Typedef;
        ^
2 errors generated.

error: command failed with exit status: 1

--

********************
Testing Time: 0.33s
********************
Failing Tests (1):
    LLDB :: SymbolFile/PDB/enums-layout.test

  Unexpected Failures: 1

I can't figure out yet what can be wrong with it. May be you have some ideas how to fix this?

[aleksandr.urakov]

But all compiles without errors if I run this manually:

clang-cl -m32 /Z7 /c /GS- C:\Work\llvm\tools\lldb\lit\SymbolFile\PDB/Inputs/SimpleTypesTest.cpp /o C:\Work\llvm\build_x86\tools\lldb\lit\SymbolFile\PDB\Output/SimpleTypesTest.cpp.enums.obj

[zturner]

I think it must be related to setting up the environment in which to run
clang. In all other projects we call llvm_config.use_clang() which is in
llvm/utils/lit/lit/llvm/config.py, but because here we have an exact path
of a clang we are trying to use, we skip this function in LLDB's lit
configuration files. But there is also a lot of other logic in that
function, so perhaps it's some of that logic that's necessary.

[stella.stamenova]

I took a brief look and I have a question about the usage of clang (rather than clang-cl).

In general I would agree that we have an exact path of clang (or gcc) that we are trying to use and they’re specified by using %cc and %cxx in the test files, but there are a number of test files that simply use clang e.g.:

SymbolFile\DWARF\find-variable-dwo.cpp:3:// RUN: clang %s -g -gsplit-dwarf -c -emit-llvm -o - --target=x86_64-pc-linux -DONE

In this case, are we not going to pick up whatever clang happens to be in the path instead of one that was explicitly specified? Is this intentional?

Thanks,
-Stella

From: Zachary Turner <[e-mail]>
Sent: Tuesday, November 13, 2018 2:46 PM
To: [e-mail]
Cc: Stella Stamenova <[e-mail]>; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]
Subject: Re: [PATCH] D54009: Refactor LLDB lit configuration files

I think it must be related to setting up the environment in which to run clang. In all other projects we call llvm_config.use_clang() which is in llvm/utils/lit/lit/llvm/config.py, but because here we have an exact path of a clang we are trying to use, we skip this function in LLDB's lit configuration files. But there is also a lot of other logic in that function, so perhaps it's some of that logic that's necessary.

lldb/lit/Breakpoint/case-insensitive.test

-# REQUIRES: windows
-# XFAIL: windows
+# REQUIRES: system-windows
+# XFAIL: system-windows
 # -> llvm.org/pr24528
 #
 # RUN: %cc %p/Inputs/case-sensitive.c -g -o %t
 # RUN: lldb-test breakpoints %t %s | FileCheck %s
 
 breakpoint set -f case-sensitive.c -l 3
 # CHECK-LABEL: breakpoint set -f case-sensitive.c -l 3
 # CHECK: At least one location.

lldb/lit/Breakpoint/lit.local.cfg

-config.suffixes = ['.test']

lldb/lit/CMakeLists.txt

 endif(BUILD_SHARED_LIBS)
 
 # the value is not canonicalized within LLVM
 llvm_canonicalize_cmake_booleans(
   LLDB_DISABLE_PYTHON
   LLVM_ENABLE_ZLIB)
 
 configure_lit_site_cfg(
-  ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
-  ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg)
+  ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.py.in
+  ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg.py
+  MAIN_CONFIG
+  ${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.py)
 configure_lit_site_cfg(
-  ${CMAKE_CURRENT_SOURCE_DIR}/Unit/lit.site.cfg.in
-  ${CMAKE_CURRENT_BINARY_DIR}/Unit/lit.site.cfg)
+  ${CMAKE_CURRENT_SOURCE_DIR}/Unit/lit.site.cfg.py.in
+  ${CMAKE_CURRENT_BINARY_DIR}/Unit/lit.site.cfg.py)
 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/Suite/lit.site.cfg.in
   ${CMAKE_CURRENT_BINARY_DIR}/Suite/lit.site.cfg)
 
 if(NOT LLDB_BUILT_STANDALONE)
   list(APPEND LLDB_TEST_DEPS
     FileCheck
     not
     )
 endif()
 
-set(LLDB_TEST_PARAMS
-  lldb_site_config=${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
-  )
-
 add_lit_testsuite(check-lldb-lit "Running lldb lit test suite"
   ${CMAKE_CURRENT_BINARY_DIR}
-  PARAMS lldb_site_config=${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
-         lldb_unit_site_config=${CMAKE_CURRENT_BINARY_DIR}/Unit/lit.site.cfg
   DEPENDS ${LLDB_TEST_DEPS}
   )
 
 set_target_properties(check-lldb-lit PROPERTIES FOLDER "LLDB tests")
 
 # If we're building with an in-tree clang, then list clang as a dependency
 # to run tests.
 if (TARGET clang)
   add_dependencies(check-lldb-lit clang)
 endif()
 
 add_lit_testsuites(LLDB
   ${CMAKE_CURRENT_BINARY_DIR}
-  PARAMS lldb_site_config=${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
-         lldb_unit_site_config=${CMAKE_CURRENT_BINARY_DIR}/Unit/lit.site.cfg
   DEPENDS ${LLDB_TEST_DEPS}
   )

lldb/lit/Expr/TestIRMemoryMapWindows.test

-# REQUIRES: windows
+# REQUIRES: system-windows
 
 # RUN: clang-cl /Zi %p/Inputs/call-function.cpp -o %t
 
 # RUN: lldb-test ir-memory-map %t %S/Inputs/ir-memory-map-basic
 # RUN: lldb-test ir-memory-map -host-only %t %S/Inputs/ir-memory-map-basic
 
 # RUN: lldb-test ir-memory-map %t %S/Inputs/ir-memory-map-overlap1
 # RUN: lldb-test ir-memory-map -host-only %t %S/Inputs/ir-memory-map-overlap1
 
 # RUN: lldb-test ir-memory-map %t %S/Inputs/ir-memory-map-mix-malloc-free
 # RUN: lldb-test ir-memory-map -host-only %t %S/Inputs/ir-memory-map-mix-malloc-free

lldb/lit/Expr/lit.local.cfg

-config.suffixes = ['.test']

lldb/lit/Quit/lit.local.cfg

-config.suffixes = ['.test']

lldb/lit/Settings/lit.local.cfg

-config.suffixes = ['.test']

lldb/lit/SymbolFile/NativePDB/lit.local.cfg

-config.suffixes = ['.test', '.cpp']

lldb/lit/SymbolFile/PDB/ast-restore.test

-REQUIRES: windows
+REQUIRES: system-windows
 RUN: cl /Zi /GS- /c %S/Inputs/AstRestoreTest.cpp /Fo%t.obj
 RUN: link /debug:full /nodefaultlib /entry:main %t.obj /out:%t.exe
 RUN: lldb-test symbols -dump-ast %t.exe | FileCheck --check-prefix=ENUM %s
 RUN: lldb-test symbols -dump-ast %t.exe | FileCheck --check-prefix=GLOBAL %s
 RUN: lldb-test symbols -dump-ast %t.exe | FileCheck --check-prefix=BASE %s
 RUN: lldb-test symbols -dump-ast %t.exe | FileCheck --check-prefix=CLASS %s
 RUN: lldb-test symbols -dump-ast %t.exe | FileCheck --check-prefix=INNER %s
 RUN: lldb-test symbols -dump-ast %t.exe | FileCheck --check-prefix=FOO %s

lldb/lit/SymbolFile/PDB/calling-conventions.test

-REQUIRES: windows, lld
+REQUIRES: system-windows, lld
 RUN: clang-cl -m32 /Zi /GS- /c %S/Inputs/CallingConventionsTest.cpp /o %t.obj
 RUN: lld-link /debug:full /nodefaultlib /entry:main %t.obj /out:%t.exe
 RUN: lldb-test symbols -dump-ast %t.exe | FileCheck %s
 
 CHECK: Module: {{.*}}
 CHECK-DAG: int (*FuncCCallPtr)();
 CHECK-DAG: int (*FuncStdCallPtr)() __attribute__((stdcall));
 CHECK-DAG: int (*FuncFastCallPtr)() __attribute__((fastcall));
 CHECK-DAG: int (*FuncVectorCallPtr)() __attribute__((vectorcall));
 CHECK-DAG: int (S::*FuncThisCallPtr)() __attribute__((thiscall));

lldb/lit/SymbolFile/PDB/class-layout.test

-REQUIRES: windows
+REQUIRES: system-windows
 RUN: clang-cl -m32 /Z7 /c /GS- %S/Inputs/ClassLayoutTest.cpp /o %T/ClassLayoutTest.cpp.obj
 RUN: link %T/ClassLayoutTest.cpp.obj /DEBUG /nodefaultlib /ENTRY:main /OUT:%T/ClassLayoutTest.cpp.exe
 RUN: lldb-test symbols %T/ClassLayoutTest.cpp.exe | FileCheck %s
 RUN: lldb-test symbols %T/ClassLayoutTest.cpp.exe | FileCheck --check-prefix=ENUM %s
 RUN: lldb-test symbols %T/ClassLayoutTest.cpp.exe | FileCheck --check-prefix=UNION %s
 RUN: lldb-test symbols %T/ClassLayoutTest.cpp.exe | FileCheck --check-prefix=STRUCT %s
 RUN: lldb-test symbols %T/ClassLayoutTest.cpp.exe | FileCheck --check-prefix=COMPLEX %s
 RUN: lldb-test symbols %T/ClassLayoutTest.cpp.exe | FileCheck --check-prefix=LIST %s

lldb/lit/SymbolFile/PDB/compilands.test

-REQUIRES: windows
+REQUIRES: system-windows
 RUN: clang-cl /Z7 %S/Inputs/CompilandsTest.cpp /o %T/CompilandsTest.cpp.exe
 RUN: lldb-test symbols %T/CompilandsTest.cpp.exe | FileCheck %s
 
 ; Link default libraries
 
 CHECK: Module [[CU:.*]]
 CHECK: {{^[0-9A-F]+}}: SymbolVendor ([[CU]])
 CHECK: {{^[0-9A-F]+}}:   CompileUnit{{[{]0x[0-9a-f]+[}]}}, language = "c++", file = '{{.*}}\CompilandsTest.cpp'

lldb/lit/SymbolFile/PDB/enums-layout.test

-REQUIRES: windows
+REQUIRES: system-windows
 RUN: clang-cl -m32 /Z7 /c /GS- %S/Inputs/SimpleTypesTest.cpp /o %T/SimpleTypesTest.cpp.enums.obj
 RUN: link %T/SimpleTypesTest.cpp.enums.obj /DEBUG /nodefaultlib /ENTRY:main /OUT:%T/SimpleTypesTest.cpp.enums.exe
 RUN: lldb-test symbols %T/SimpleTypesTest.cpp.enums.exe | FileCheck %s
 
 ; FIXME: PDB does not have information about scoped enumeration (Enum class) so the  
 ; compiler type used is the same as the one for unscoped enumeration.
 
 CHECK: Module [[CU:.*]]

lldb/lit/SymbolFile/PDB/func-symbols.test

-REQUIRES: windows
+REQUIRES: system-windows
 RUN: clang-cl -m32 /Z7 /c /GS- %S/Inputs/FuncSymbolsTestMain.cpp /o %T/FuncSymbolsTestMain.cpp.obj
 RUN: clang-cl -m32 /Z7 /c /GS- %S/Inputs/FuncSymbols.cpp /o %T/FuncSymbols.cpp.obj
 RUN: link %T/FuncSymbolsTestMain.cpp.obj %T/FuncSymbols.cpp.obj /DEBUG /nodefaultlib /Entry:main /OUT:%T/FuncSymbolsTest.exe
 RUN: lldb-test symbols %T/FuncSymbolsTest.exe | FileCheck --check-prefix=CHECK-ONE %s
 RUN: lldb-test symbols %T/FuncSymbolsTest.exe | FileCheck --check-prefix=CHECK-TWO %s
 
 ; Link multiple objects
 ; In this test, We don't check demangled name of a mangled function.

lldb/lit/SymbolFile/PDB/function-level-linking.test

-REQUIRES: windows, lld
+REQUIRES: system-windows, lld
 RUN: clang-cl /c /Zi /Gy %S/Inputs/FunctionLevelLinkingTest.cpp /o %t.obj
 RUN: lld-link /debug:full /nodefaultlib /entry:main /order:@%S/Inputs/FunctionLevelLinkingTest.ord %t.obj /out:%t.exe
 RUN: lldb-test symbols -verify %t.exe

lldb/lit/SymbolFile/PDB/function-nested-block.test

-REQUIRES: windows, lld
+REQUIRES: system-windows, lld
 RUN: clang-cl /c /Zi %S/Inputs/FunctionNestedBlockTest.cpp /o %t.obj
 RUN: lld-link /debug:full /nodefaultlib /entry:main %t.obj /out:%t.exe
 RUN: lldb-test symbols -find=function -file FunctionNestedBlockTest.cpp -line 4 %t.exe | FileCheck --check-prefix=CHECK-FUNCTION %s
 RUN: lldb-test symbols -find=block -file FunctionNestedBlockTest.cpp -line 4 %t.exe | FileCheck --check-prefix=CHECK-BLOCK %s
 
 CHECK-FUNCTION: Found 1 functions:
 CHECK-FUNCTION: name = "{{.*}}", mangled = "{{_?}}main"
 
 CHECK-BLOCK: Found 1 blocks:
 CHECK-BLOCK: Blocks: id = {{.*}}, range = {{.*}}
 CHECK-BLOCK:         id = {{.*}}, range = {{.*}}

lldb/lit/SymbolFile/PDB/lit.local.cfg

-config.suffixes = ['.test']

lldb/lit/SymbolFile/PDB/pointers.test

-REQUIRES: windows
+REQUIRES: system-windows
 RUN: clang-cl -m32 /Z7 /c /GS- %S/Inputs/PointerTypeTest.cpp /o %T/PointerTypeTest.cpp.obj
 RUN: link %T/PointerTypeTest.cpp.obj /DEBUG /nodefaultlib /ENTRY:main /OUT:%T/PointerTypeTest.cpp.exe
 RUN: lldb-test symbols %T/PointerTypeTest.cpp.exe | FileCheck %s
 RUN: lldb-test symbols %T/PointerTypeTest.cpp.exe | FileCheck --check-prefix=MAIN-ST-F %s
 RUN: lldb-test symbols %T/PointerTypeTest.cpp.exe | FileCheck --check-prefix=MAIN-ST %s
 RUN: lldb-test symbols %T/PointerTypeTest.cpp.exe | FileCheck --check-prefix=MAIN %s
 RUN: lldb-test symbols %T/PointerTypeTest.cpp.exe | FileCheck --check-prefix=F %s
 

lldb/lit/SymbolFile/PDB/type-quals.test

-REQUIRES: windows
+REQUIRES: system-windows
 RUN: clang-cl -m32 /Z7 /c /GS- %S/Inputs/TypeQualsTest.cpp /o %T/TypeQualsTest.cpp.obj
 RUN: link %T/TypeQualsTest.cpp.obj /DEBUG /nodefaultlib /ENTRY:main /OUT:%T/TypeQualsTest.cpp.exe
 RUN: lldb-test symbols %T/TypeQualsTest.cpp.exe | FileCheck %s
 
 CHECK: Module [[MOD:.*]]
 CHECK-DAG: {{^[0-9A-F]+}}: SymbolVendor ([[MOD]])
 CHECK:      Type{{.*}} , name = "const int", size = 4, compiler_type = {{.*}} const int
 CHECK:      Type{{.*}} , size = 4, compiler_type = {{.*}} const int *

lldb/lit/SymbolFile/PDB/typedefs.test

-REQUIRES: windows
+REQUIRES: system-windows
 RUN: clang-cl -m32 /Z7 /c /GS- %S/Inputs/SimpleTypesTest.cpp /o %T/SimpleTypesTest.cpp.typedefs.obj
 RUN: link %T/SimpleTypesTest.cpp.typedefs.obj /DEBUG /nodefaultlib /ENTRY:main /OUT:%T/SimpleTypesTest.cpp.typedefs.exe
 RUN: lldb-test symbols %T/SimpleTypesTest.cpp.typedefs.exe | FileCheck %s
 
 ; Generate 32-bit target
 
 ; FIXME: PDB does not have line information for typedef statements so source
 ; and line information for them is not tested.

lldb/lit/SymbolFile/PDB/udt-layout.test

-REQUIRES: windows
+REQUIRES: system-windows
 RUN: clang-cl /Zi %S/Inputs/UdtLayoutTest.cpp /o %t.exe
 RUN: %lldb -b -s %S/Inputs/UdtLayoutTest.script -- %t.exe | FileCheck %s
 
 CHECK:(int) int C::abc = 123
 CHECK:(List [16]) ls = {
 CHECK:  [15] = {
 CHECK:    Prev = 0x00000000
 CHECK:    Next = 0x00000000

lldb/lit/SymbolFile/PDB/variables-locations.test

-REQUIRES: windows
+REQUIRES: system-windows
 RUN: clang-cl /Zi %S/Inputs/VariablesLocationsTest.cpp /o %t.exe
 RUN: %lldb -b -s %S/Inputs/VariablesLocationsTest.script -- %t.exe | FileCheck %s
 
 CHECK: g_var = 2222
 
 CHECK: arg_0 = 1111
 CHECK: arg_1 = 0.123
 
 CHECK: loc_0 = 'x'
 CHECK: loc_1 = 0.567
 
 CHECK: loc_0 = true
 CHECK: loc_1 = 3333

lldb/lit/SymbolFile/PDB/variables.test

-REQUIRES: windows
+REQUIRES: system-windows
 RUN: clang-cl -m64 /Z7 /c /GS- %S/Inputs/VariablesTest.cpp /o %T/VariablesTest.cpp.obj
 RUN: link %T/VariablesTest.cpp.obj /DEBUG /nodefaultlib /ENTRY:main /OUT:%T/VariablesTest.cpp.exe
 RUN: lldb-test symbols %T/VariablesTest.cpp.exe | FileCheck %s
 
 CHECK: Module [[MOD:.*]]
 CHECK: SymbolVendor ([[MOD]])
 CHECK:     CompileUnit{{.*}}, language = "c++", file = '{{.*}}\VariablesTest.cpp'
 CHECK-DAG:   Variable{{.*}}, name = "g_IntVar"

 CHECK-DAG:     Variable{{.*}}, name = "a"
 CHECK-SAME:    scope = parameter
 
 CHECK-DAG:   Function{[[FID4:.*]]}, mangled = ?Func@Class@@QEAAXXZ
 CHECK-NEXT:  Block{[[FID4]]}
 CHECK-DAG:     Variable{{.*}}, name = "this"
 CHECK-SAME:    scope = parameter
 CHECK-SAME:    artificial
 No newline at end of file

lldb/lit/Unit/lit.cfg.py

 # -*- Python -*-
 
 # Configuration file for the 'lit' test runner.
 
 import os
+import sys
 
 import lit.formats
+from lit.llvm import llvm_config
 
 # name: The name of this test suite.
 config.name = 'lldb-Unit'
 
 # suffixes: A list of file extensions to treat as test files.
 config.suffixes =  []
 
 # test_source_root: The root path where unit test binaries are located.
 # test_exec_root: The root path where tests should be run.
 config.test_source_root = os.path.join(config.lldb_obj_root, 'unittests')
 config.test_exec_root = config.test_source_root
 
+# One of our unit tests dynamically links against python.dll, and on Windows
+# it needs to be able to find it at runtime.  This is fine if Python is on your
+# system PATH, but if it's not, then this unit test executable will fail to run.
+# We can solve this by forcing the Python directory onto the system path here.
+llvm_config.with_system_environment('PATH')
+llvm_config.with_environment('PATH', os.path.dirname(sys.executable), append_path=True)
+
 # testFormat: The test format to use to interpret tests.
 config.test_format = lit.formats.GoogleTest(config.llvm_build_mode, 'Tests')

lldb/lit/Unit/lit.site.cfg.py.in

 try:
     config.llvm_tools_dir = config.llvm_tools_dir % lit_config.params
     config.llvm_libs_dir = config.llvm_libs_dir % lit_config.params
     config.llvm_build_mode = config.llvm_build_mode % lit_config.params
 except KeyError as e:
     key, = e.args
     lit_config.fatal("unable to find %r parameter, use '--param=%s=VALUE'" % (key,key))
 
+import lit.llvm
+lit.llvm.initialize(lit_config, config)
+
 # Let the main config do the real work.
-lit_config.load_config(config, "@LLDB_SOURCE_DIR@/lit/Unit/lit.cfg")
+lit_config.load_config(config, "@LLDB_SOURCE_DIR@/lit/Unit/lit.cfg.py")

lldb/lit/lit.cfg

-# -*- Python -*-
-
-import os
-import platform
-import re
-import subprocess
-import locale
-
-import lit.formats
-import lit.util
-
-def binary_feature(on, feature, off_prefix):
-  return feature if on else off_prefix + feature
-
-# Configuration file for the 'lit' test runner.
-
-# name: The name of this test suite.
-config.name = 'lldb'
-
-# testFormat: The test format to use to interpret tests.
-#
-# For now we require '&&' between commands, until they get globally killed and
-# the test runner updated.
-execute_external = (platform.system() != 'Windows'
-                    or lit_config.getBashPath() not in [None, ""])
-config.test_format = lit.formats.ShTest(execute_external)
-
-# suffixes: We only support unit tests
-config.suffixes = []
-
-config.excludes = ['Inputs']
-
-# test_source_root: The root path where tests are located.
-config.test_source_root = os.path.dirname(__file__)
-
-# test_exec_root: The root path where tests should be run.
-config.test_exec_root = os.path.join(config.lldb_obj_root, 'lit')
-
-# Tweak the PATH to include the tools dir and the scripts dir.
-lldb_tools_dir = config.lldb_tools_dir
-llvm_tools_dir = config.llvm_tools_dir
-path = os.path.pathsep.join((config.lldb_tools_dir, config.llvm_tools_dir, config.environment['PATH']))
-
-config.environment['PATH'] = path
-
-path = os.path.pathsep.join((config.lldb_libs_dir, config.llvm_libs_dir,
-                              config.environment.get('LD_LIBRARY_PATH','')))
-config.environment['LD_LIBRARY_PATH'] = path
-
-# Propagate LLVM_SRC_ROOT into the environment.
-config.environment['LLVM_SRC_ROOT'] = getattr(config, 'llvm_src_root', '')
-
-# Propagate PYTHON_EXECUTABLE into the environment
-config.environment['PYTHON_EXECUTABLE'] = getattr(config, 'python_executable', '')
-
-# Register substitutions
-config.substitutions.append(('%python', '"%s"' % (config.python_executable)))
-
-if platform.system() in ['Darwin']:
-    debugserver = lit.util.which('debugserver', lldb_tools_dir)
-else:
-    debugserver = lit.util.which('lldb-server', lldb_tools_dir)
-lldb = "%s -S %s/lit-lldb-init" % (lit.util.which('lldb', lldb_tools_dir),
-                               config.test_source_root)
-
-lldbmi = lit.util.which('lldb-mi', lldb_tools_dir)
-if lldbmi:
-    config.available_features.add('lldb-mi')
-
-if not os.path.exists(config.cc):
-    config.cc = lit.util.which(config.cc, config.environment['PATH'])
-
-if not os.path.exists(config.cxx):
-    config.cxx = lit.util.which(config.cxx, config.environment['PATH'])
-
-if platform.system() in ['Darwin']:
-    try:
-        out = subprocess.check_output(['xcrun', '--show-sdk-path']).strip()
-        res = 0
-    except OSError:
-        res = -1
-    if res == 0 and out:
-        sdk_path = lit.util.to_string(out)
-        lit_config.note('using SDKROOT: %r' % sdk_path)
-        config.cc += " -isysroot %s" % sdk_path
-        config.cxx += " -isysroot %s" % sdk_path
-
-if platform.system() in ['OpenBSD']:
-    config.cc += " -pthread"
-    config.cxx += " -pthread"
-
-config.substitutions.append(('%cc', config.cc))
-config.substitutions.append(('%cxx', config.cxx))
-
-if lldbmi:
-  config.substitutions.append(('%lldbmi', lldbmi + " --synchronous"))
-config.substitutions.append(('%lldb', lldb))
-
-if debugserver is not None:
-    if platform.system() in ['Darwin']:
-        config.substitutions.append(('%debugserver', debugserver))
-    else:
-        config.substitutions.append(('%debugserver', debugserver + ' gdbserver'))
-
-for pattern in [r"\bFileCheck\b",
-                r"\blldb-test\b",
-                r"\byaml2obj\b",
-                r"\| \bnot\b"]:
-    tool_match = re.match(r"^(\\)?((\| )?)\W+b([0-9A-Za-z-_]+)\\b\W*$",
-                          pattern)
-    tool_pipe = tool_match.group(2)
-    tool_name = tool_match.group(4)
-    tool_path = lit.util.which(tool_name, config.environment['PATH'])
-    if not tool_path:
-        # Warn, but still provide a substitution.
-        lit_config.note(
-            'Did not find ' + tool_name + ' in ' + config.environment['PATH'])
-    config.substitutions.append((pattern, tool_pipe + tool_path))
-
-# Shell execution
-if platform.system() not in ['Windows'] or lit_config.getBashPath() != '':
-    config.available_features.add('shell')
-
-# Running on Darwin OS
-if platform.system() in ['Darwin']:
-    config.available_features.add('darwin')
-    config.available_features.add('system-linker-mach-o')
-
-# Running on ELF based *nix
-if platform.system() in ['FreeBSD', 'Linux']:
-    config.available_features.add('system-linker-elf')
-    if platform.system() in ['FreeBSD']:
-        config.available_features.add('freebsd')
-    else:
-        config.available_features.add('linux')
-
-config.available_features.add(
-    binary_feature(platform.system() in ['Windows'], 'windows', 'no'))
-
-if re.match(r'^arm(hf.*-linux)|(.*-linux-gnuabihf)', config.target_triple):
-    config.available_features.add("armhf-linux")
-
-if re.match(r'icc', config.cc):
-    config.available_features.add("compiler-icc")
-elif re.match(r'clang', config.cc):
-    config.available_features.add("compiler-clang")
-elif re.match(r'gcc', config.cc):
-    config.available_features.add("compiler-gcc")
-elif re.match(r'cl', config.cc):
-    config.available_features.add("compiler-msvc")
-
-config.available_features.add(binary_feature(config.have_zlib, "zlib", "no"))
-if config.have_lld:
-  config.available_features.add("lld")
-
-# llvm-config knows whether it is compiled with asserts (and)
-# whether we are operating in release/debug mode.
-import subprocess
-try:
-    llvm_config_cmd = \
-     subprocess.Popen([os.path.join(llvm_tools_dir, 'llvm-config'),
-                     '--build-mode', '--assertion-mode', '--targets-built'],
-                      stdout = subprocess.PIPE)
-except OSError as why:
-    print("Could not find llvm-config in " + llvm_tools_dir)
-    exit(42)
-
-llvm_config_output = llvm_config_cmd.stdout.read().decode('utf_8')
-llvm_config_output_list = llvm_config_output.split("\n")
-
-if re.search(r'DEBUG', llvm_config_output_list[0]):
-    config.available_features.add('debug')
-if re.search(r'ON', llvm_config_output_list[1]):
-    config.available_features.add('asserts')
-if re.search(r'ARM', llvm_config_output_list[2]):
-    config.available_features.add('arm')
-if re.search(r'Mips', llvm_config_output_list[2]):
-    config.available_features.add('mips')
-if re.search(r'PowerPC', llvm_config_output_list[2]):
-    config.available_features.add('powerpc')
-if re.search(r'X86', llvm_config_output_list[2]):
-    config.available_features.add('x86')
-llvm_config_cmd.wait()

lldb/lit/lit.cfg.py

+# -*- Python -*-
+
+import os
+import sys
+import re
+import platform
+import subprocess
+
+
+import lit.util
+import lit.formats
+from lit.llvm import llvm_config
+from lit.llvm.subst import FindTool
+from lit.llvm.subst import ToolSubst
+
+# name: The name of this test suite.
+config.name = 'LLDB'
+
+# testFormat: The test format to use to interpret tests.
+config.test_format = lit.formats.ShTest(not llvm_config.use_lit_shell)
+
+# suffixes: A list of file extensions to treat as test files. This is overriden
+# by individual lit.local.cfg files in the test subdirectories.
+config.suffixes = ['.test', '.cpp', '.s']
+
+# excludes: A list of directories to exclude from the testsuite. The 'Inputs'
+# subdirectories contain auxiliary inputs for various tests in their parent
+# directories.
+config.excludes = ['Inputs', 'CMakeLists.txt', 'README.txt', 'LICENSE.txt']
+
+# test_source_root: The root path where tests are located.
+config.test_source_root = os.path.dirname(__file__)
+
+# test_exec_root: The root path where tests should be run.
+config.test_exec_root = os.path.join(config.lldb_obj_root, 'lit')
+
+# Tweak the PATH to include the tools dir.
+llvm_config.with_system_environment('PATH')
+llvm_config.with_environment('PATH', config.lldb_tools_dir, append_path=True)
+llvm_config.with_environment('PATH', config.llvm_tools_dir, append_path=True)
+
+llvm_config.with_environment('LD_LIBRARY_PATH', config.lldb_libs_dir, append_path=True)
+llvm_config.with_environment('LD_LIBRARY_PATH', config.llvm_libs_dir, append_path=True)
+llvm_config.with_system_environment('LD_LIBRARY_PATH', append_path=True)
+
+
+llvm_config.use_default_substitutions()
+
+if platform.system() in ['Darwin']:
+    debugserver = lit.util.which('debugserver', config.lldb_tools_dir)
+else:
+    debugserver = lit.util.which('lldb-server', config.lldb_tools_dir)
+lldb = "%s -S %s/lit-lldb-init" % (lit.util.which('lldb', config.lldb_tools_dir),
+                               config.test_source_root)
+
+lldbmi = lit.util.which('lldb-mi', config.lldb_tools_dir)
+if lldbmi:
+    config.available_features.add('lldb-mi')
+
+config.cc = llvm_config.use_llvm_tool(config.cc, required=True)
+config.cxx = llvm_config.use_llvm_tool(config.cxx, required=True)
+
+if platform.system() in ['Darwin']:
+    try:
+        out = subprocess.check_output(['xcrun', '--show-sdk-path']).strip()
+        res = 0
+    except OSError:
+        res = -1
+    if res == 0 and out:
+        sdk_path = lit.util.to_string(out)
+        lit_config.note('using SDKROOT: %r' % sdk_path)
+        config.cc += " -isysroot %s" % sdk_path
+        config.cxx += " -isysroot %s" % sdk_path
+
+if platform.system() in ['OpenBSD']:
+    config.cc += " -pthread"
+    config.cxx += " -pthread"
+
+config.substitutions.append(('%cc', config.cc))
+config.substitutions.append(('%cxx', config.cxx))
+
+if lldbmi:
+  config.substitutions.append(('%lldbmi', lldbmi + " --synchronous"))
+config.substitutions.append(('%lldb', lldb))
+
+if debugserver is not None:
+    if platform.system() in ['Darwin']:
+        config.substitutions.append(('%debugserver', debugserver))
+    else:
+        config.substitutions.append(('%debugserver', debugserver + ' gdbserver'))
+
+tools = ['lldb-test', 'yaml2obj', 'obj2yaml', 'llvm-pdbutil']
+llvm_config.add_tool_substitutions(tools, [config.llvm_tools_dir, config.lldb_tools_dir])
+
+if re.match(r'^arm(hf.*-linux)|(.*-linux-gnuabihf)', config.target_triple):
+    config.available_features.add("armhf-linux")
+
+print("config.cc = {}".format(config.cc))
+if re.match(r'icc', config.cc):
+    config.available_features.add("compiler-icc")
+elif re.match(r'clang', config.cc):
+    config.available_features.add("compiler-clang")
+elif re.match(r'gcc', config.cc):
+    config.available_features.add("compiler-gcc")
+elif re.match(r'cl', config.cc):
+    config.available_features.add("compiler-msvc")
+
+if config.have_lld:
+  config.available_features.add("lld")
+
+
+def calculate_arch_features(arch_string):
+    features = []
+    for arch in arch_string.split():
+        features.append(arch.lower() + '-registered-target')
+    return features
+
+def calculate_arch_features(arch_string):
+    # This will add a feature such as x86, arm, mips, etc
+    # for each built target
+    features = []
+    for arch in arch_string.split():
+        features.append(arch.lower())
+    return features
+
+llvm_config.feature_config(
+    [('--assertion-mode', {'ON': 'asserts'}),
+     ('--build-mode', {'DEBUG': 'debug'}),
+     ('--targets-built', calculate_arch_features)
+     ])

lldb/lit/lit.site.cfg.py.in

try:
     config.lldb_tools_dir = config.lldb_tools_dir % lit_config.params
     config.cc = config.cc % lit_config.params
     config.cxx = config.cxx % lit_config.params
 except KeyError as e:
     key, = e.args
     lit_config.fatal("unable to find %r parameter, use '--param=%s=VALUE'" % (key,key))
 
 # Let the main config do the real work.
-lit_config.load_config(config, "@LLDB_SOURCE_DIR@/lit/lit.cfg")
+lit_config.load_config(config, "@LLDB_SOURCE_DIR@/lit/lit.cfg.py")

lldb/lit/tools/lldb-mi/breakpoint/break-insert-enable-pending.test

-# XFAIL: windows
+# XFAIL: system-windows
 # -> llvm.org/pr24452
 #
 # RUN: %cc -o %t %p/inputs/break-insert-pending.c -g
 # RUN: %lldbmi %t < %s | FileCheck %s
 
 # Test for enabling pending breakpoints globally
 
 -break-insert printf

lldb/lit/tools/lldb-mi/breakpoint/break-insert.test

-# XFAIL: windows
+# XFAIL: system-windows
 # -> llvm.org/pr24452
 #
 # RUN: %cc -o a.exe %p/inputs/break-insert.c -g
 # RUN: %lldbmi < %s | FileCheck %s
 
 # Test that a breakpoint can be inserted before creating a target.
 
 -break-insert breakpoint

lldb/lit/tools/lldb-mi/data/data-info-line.test

-# XFAIL: windows
+# XFAIL: system-windows
 # -> llvm.org/pr24452
 #
 # RUN: %cc -o %t %p/inputs/data-info-line.c -g
 # RUN: %lldbmi %t < %s | FileCheck %s
 
 # Test lldb-mi -data-info-line command.
 
 # Check that we have a valid target created via '%lldbmi %t'.

lldb/lit/tools/lldb-mi/exec/exec-continue.test

-# XFAIL: windows
+# XFAIL: system-windows
 # -> llvm.org/pr24452
 #
 # RUN: %cc -o %t %p/inputs/main.c -g
 # RUN: %lldbmi %t < %s | FileCheck %s
 
 # Test lldb-mi -exec-continue command.
 
 # Check that we have a valid target created via '%lldbmi %t'.

lldb/lit/tools/lldb-mi/exec/exec-finish.test

-# XFAIL: windows
+# XFAIL: system-windows
 # -> llvm.org/pr24452
 #
 # RUN: %cc -o %t %p/inputs/main.c -g
 # RUN: %lldbmi %t < %s | FileCheck %s
 
 # Test lldb-mi -exec-finish command.
 
 # Check that we have a valid target created via '%lldbmi %t'.

lldb/lit/tools/lldb-mi/exec/exec-interrupt.test

-# XFAIL: windows
+# XFAIL: system-windows
 # -> llvm.org/pr24452
 #
 # RUN: %cc -o %t %p/inputs/main.c -g
 # RUN: %lldbmi %t < %s | FileCheck %s
 
 # Test lldb-mi -exec-interrupt command.
 
 # Check that we have a valid target created via '%lldbmi %t'.

lldb/lit/tools/lldb-mi/exec/exec-next-instruction.test

-# XFAIL: windows
+# XFAIL: system-windows
 # -> llvm.org/pr24452
 #
 # RUN: %cc -o %t %p/inputs/main.c -g
 # RUN: %lldbmi %t < %s | FileCheck %s
 
 # Test lldb-mi -exec-next-instruction command.
 
 # Check that we have a valid target created via '%lldbmi %t'.

lldb/lit/tools/lldb-mi/exec/exec-next.test

-# XFAIL: windows
+# XFAIL: system-windows
 # -> llvm.org/pr24452
 #
 # RUN: %cc -o %t %p/inputs/main.c -g
 # RUN: %lldbmi %t < %s | FileCheck %s
 
 # Test lldb-mi -exec-next command.
 
 # Check that we have a valid target created via '%lldbmi %t'.

lldb/lit/tools/lldb-mi/exec/exec-step-instruction.test

-# XFAIL: windows
+# XFAIL: system-windows
 # -> llvm.org/pr24452
 #
 # RUN: %cc -o %t %p/inputs/main.c -g
 # RUN: %lldbmi %t < %s | FileCheck %s
 
 # Test lldb-mi -exec-step-instruction command.
 
 # Check that we have a valid target created via '%lldbmi %t'.

lldb/lit/tools/lldb-mi/exec/exec-step.test

-# XFAIL: windows
+# XFAIL: system-windows
 # -> llvm.org/pr24452
 #
 # RUN: %cc -o %t %p/inputs/main.c -g
 # RUN: %lldbmi %t < %s | FileCheck %s
 
 # Test lldb-mi -exec-step command.
 
 # Check that we have a valid target created via '%lldbmi %t'.

lldb/lit/tools/lldb-mi/symbol/symbol-list-lines.test

-# XFAIL: windows
+# XFAIL: system-windows
 # -> llvm.org/pr24452
 #
 # RUN: %cc -o %t %p/inputs/main.c %p/inputs/symbol-list-lines.c %p/inputs/list-lines-helper.c -g
 # RUN: %lldbmi %t < %s | FileCheck %s
 
 # Test lldb-mi -symbol-list-lines command.
 
 # Check that we have a valid target created via '%lldbmi %t'.

llvm/utils/lit/lit/llvm/config.py

def __init__(self, lit_config, config):
             # We should standardize on the former.
             features.add('system-linker-mach-o')
             features.add('system-darwin')
         elif platform.system() == 'Windows':
             # For tests that require Windows to run.
             features.add('system-windows')
         elif platform.system() == "Linux":
             features.add('system-linux')
+        elif platform.system() in ['FreeBSD']:
+            config.available_features.add('system-freebsd')
 
         # Native compilation: host arch == default triple arch
         # Both of these values should probably be in every site config (e.g. as
         # part of the standard header.  But currently they aren't)
         host_triple = getattr(config, 'host_triple', None)
         target_triple = getattr(config, 'target_triple', None)
         if host_triple and host_triple == target_triple:
             features.add('native')