[lit] Fix testing of standalone clang and lld builds

[jhenderson]

Summary

In such cases, the executables are not in the llvm_tools_dir directory, so we need to look in the other search locations. Previously, they were found via the PATH, but this was disabled by default in commit rGa1e6565.

Depends on D103154.

[teemperor]

Depends on D101354.

Typo?

[jhenderson]

In D103156#2781804, @teemperor wrote:

Depends on D101354.

Typo?

Thanks. Fixed.

[tstellar]

D103154 seems to be making a lot of the same changes as this patch.

[jhenderson]

In D103156#2783447, @tstellar wrote:

D103154 seems to be making a lot of the same changes as this patch.

Sorry, I stuffed up the diff in that patch - it shouldn't include the config.py changes. Will fix.

[jhenderson]

Remove unrelated parts of diff.

[jhenderson]

Upload correct patch.

[thopre]

LGTM

[jhenderson]

Rebase. I'll try landing this once the pre-merge checks complete, if there are no objections.

FYI: I'm off next week, so if this doesn't fix things, and the issue is an issue that needs resolving faster, I'd recommend reverting the original change.

[kwk]

This comment was removed by kwk.

[kwk]

I've applied this patch but for me clang still fails to build. I've build python-lit and clang in revision 1546c52d971292ed4145b6d41aaca0d02229ebff. And applied this patch on top.

[340/341] cd /builddir/build/BUILD/clang-13.0.0.src/x86_64-redhat-linux-gnu && /usr/bin/python3.9 /usr/bin/lit -sv --param USE_Z3_SOLVER=0 /builddir/build/BUILD/clang-13.0.0.src/x86_64-redhat-linux-gnu/tools/extra/test /builddir/build/BUILD/clang-13.0.0.src/x86_64-redhat-linux-gnu/tools/extra/clangd/test/../unittests /builddir/build/BUILD/clang-13.0.0.src/x86_64-redhat-linux-gnu/tools/extra/clangd/test /builddir/build/BUILD/clang-13.0.0.src/x86_64-redhat-linux-gnu/test
lit: /usr/lib/python3.9/site-packages/lit/llvm/config.py:431: fatal: couldn't find 'clang' program, try setting CLANG in your environment
FAILED: CMakeFiles/check-all

These are the clang files in my mock environment:

$ find / -name "clang"
/builddir/build/BUILDROOT/clang-13.0.0~pre20210527.g1546c52d971292-2.fc35.x86_64/usr/lib64/cmake/clang
/builddir/build/BUILDROOT/clang-13.0.0~pre20210527.g1546c52d971292-2.fc35.x86_64/usr/lib64/clang
/builddir/build/BUILDROOT/clang-13.0.0~pre20210527.g1546c52d971292-2.fc35.x86_64/usr/lib/python3.9/site-packages/clang
/builddir/build/BUILDROOT/clang-13.0.0~pre20210527.g1546c52d971292-2.fc35.x86_64/usr/bin/clang
/builddir/build/BUILDROOT/clang-13.0.0~pre20210527.g1546c52d971292-2.fc35.x86_64/usr/include/clang
/builddir/build/BUILDROOT/clang-13.0.0~pre20210527.g1546c52d971292-2.fc35.x86_64/usr/share/doc/clang
/builddir/build/BUILDROOT/clang-13.0.0~pre20210527.g1546c52d971292-2.fc35.x86_64/usr/share/clang
/builddir/build/BUILD/clang-13.0.0.src/test/Tooling/Inputs/mock-libcxx/bin/clang
/builddir/build/BUILD/clang-13.0.0.src/test/Driver/Inputs/mips_mti_linux/lib/clang
/builddir/build/BUILD/clang-13.0.0.src/include/clang
/builddir/build/BUILD/clang-13.0.0.src/x86_64-redhat-linux-gnu/lib64/cmake/clang
/builddir/build/BUILD/clang-13.0.0.src/x86_64-redhat-linux-gnu/lib64/clang
/builddir/build/BUILD/clang-13.0.0.src/x86_64-redhat-linux-gnu/cmake/modules/CMakeFiles/Export/lib64/cmake/clang
/builddir/build/BUILD/clang-13.0.0.src/x86_64-redhat-linux-gnu/bin/clang
/builddir/build/BUILD/clang-13.0.0.src/x86_64-redhat-linux-gnu/include/clang
/builddir/build/BUILD/clang-13.0.0.src/bindings/python/clang
/builddir/build/BUILD/clang-tools-extra-13.0.0.src/test/clang-tidy/checkers/Inputs/Headers/clang
/builddir/build/BUILD/clang-tools-extra-13.0.0.src/test/clang-tidy/infrastructure/Inputs/mock-libcxx/bin/clang

[jhenderson]

Ah, I see I missed that paths is reused in use_clang, before the call to use_llvm_tool. I'll fix that now.

[jhenderson]

Use the right set of paths in use_clang. This should fix the issue. If it doesn't I'll need someone who can reproduce to dig more into why it doesn't.

[jhenderson]

@kwk, could you try again please with the latest version of the patch? The code now looks at the same paths as are added to the PATH environment variable, so it should work.

[thopre]

New change LGTM, provided @kwk confirms it works for him.

[kwk]

@jhenderson @thopre it works for me now! Thank you!

[jhenderson]

In D103156#2787777, @kwk wrote:

@jhenderson @thopre it works for me now! Thank you!

Great, thanks for letting me know. I'm not going to be at a computer to be able to land this for the next week, so someone else can on my behalf if they wish. Otherwise it will be in a bit over a week's time before I do.

llvm/utils/lit/tests/Inputs/use-llvm-tool-required/lit.cfg

+import lit.formats
+config.name = 'use-llvm-tool-required'
+config.suffixes = ['.txt']
+config.test_format = lit.formats.ShTest()
+config.test_source_root = None
+config.test_exec_root = None
+import os.path
+config.llvm_tools_dir = os.path.realpath(os.path.dirname(__file__))
+import lit.llvm
+lit.llvm.initialize(lit_config, config)
+lit.llvm.llvm_config.use_llvm_tool('found', required=True)
+lit.llvm.llvm_config.use_llvm_tool('not-found', required=True)

llvm/utils/lit/tests/Inputs/use-llvm-tool/lit.cfg

+import lit.formats
+config.name = 'use-llvm-tool'
+config.suffixes = ['.txt']
+config.test_format = lit.formats.ShTest()
+config.test_source_root = None
+config.test_exec_root = None
+import os.path
+this_dir = os.path.realpath(os.path.dirname(__file__))
+config.llvm_tools_dir = os.path.join(this_dir, 'build')
+import lit.llvm
+lit.llvm.initialize(lit_config, config)
+lit.llvm.llvm_config.with_environment('CASE1', os.path.join(this_dir, 'env-case1'))
+lit.llvm.llvm_config.with_environment('CASE6', os.path.join(this_dir, 'env-case6'))
+lit.llvm.llvm_config.with_environment('PATH', os.path.join(this_dir, 'path'), append_path=True)
+lit.llvm.llvm_config.use_llvm_tool('case1', search_env='CASE1')
+lit.llvm.llvm_config.use_llvm_tool('case2', search_env='CASE2')
+lit.llvm.llvm_config.use_llvm_tool('case3')
+lit.llvm.llvm_config.use_llvm_tool('case4', use_installed=True)
+lit.llvm.llvm_config.use_llvm_tool('case5')
+lit.llvm.llvm_config.use_llvm_tool('case6', search_env='CASE6', use_installed=True)
+lit.llvm.llvm_config.use_llvm_tool('case7', use_installed=True)
+lit.llvm.llvm_config.use_llvm_tool('case8', use_installed=True)

llvm/utils/lit/tests/Inputs/use-llvm-tool/true.txt

+# RUN: true

llvm/utils/lit/tests/Inputs/use-tool-search-env/lit.cfg

-import lit.formats
-config.name = 'search-env'
-config.suffixes = ['.txt']
-config.test_format = lit.formats.ShTest()
-config.test_source_root = None
-config.test_exec_root = None
-import lit.llvm
-lit.llvm.initialize(lit_config, config)
-import os.path
-path = os.path.realpath(os.path.join(os.path.dirname(__file__), 'test.tool'))
-lit.llvm.llvm_config.with_environment('TOOL_LOCATION', path)
-lit.llvm.llvm_config.use_llvm_tool('test-tool', search_env='TOOL_LOCATION')

llvm/utils/lit/tests/use-llvm-tool.py

+## Show that lit reports the path of tools found via use_llvm_tool.
+## Additionally show that use_llvm_tool uses in order of preference:
+## 1) The path specified in an environment variable,
+## 2) The LLVM tools build directory,
+## 3) The PATH, if requested.
+
+# RUN: %{lit} %{inputs}/use-llvm-tool 2>&1 | \
+# RUN:   FileCheck %s -DDIR=%p
+
+## The exact breakdown of cases is:
+## Case | Env | Build Dir | PATH |
+##   1  |  /  |     X     |  N/S | <- Can be found via env
+##   2  |  X  |     /     |  N/S | <- Can be found via build dir if env specified
+##   3  | N/S |     /     |  N/S | <- Can be found via build dir
+##   4  | N/S |     X     |   /  | <- Can be found via PATH, if requested
+##   5  | N/S |     X     |  N/S | <- Cannot be found via PATH, if not requested
+##   6  |  /  |     /     |   /  | <- Env is preferred over build, PATH
+##   7  | N/S |     /     |   /  | <- Build dir is preferred over PATH
+##   8  |  X  |     X     |   X  | <- Say nothing if cannot be found if not required
+
+## Check the exact path reported for the first case, but don't bother for the
+## others.
+# CHECK:      note: using case1: [[DIR]]{{[\\/]}}Inputs{{[\\/]}}use-llvm-tool{{[\\/]}}env-case1
+# CHECK-NEXT: note: using case2: {{.*}}build{{[\\/]}}case2
+# CHECK-NEXT: note: using case3: {{.*}}build{{[\\/]}}case3
+# CHECK-NEXT: note: using case4: {{.*}}path{{[\\/]}}case4
+# CHECK-NOT:  case5
+# CHECK-NEXT: note: using case6: {{.*}}env-case6
+# CHECK-NEXT: note: using case7: {{.*}}build{{[\\/]}}case7
+# CHECK-NOT:  case8
+
+## Test that if required is True, lit errors if the tool is not found.
+# RUN: not %{lit} %{inputs}/use-llvm-tool-required 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=ERROR
+# ERROR:      note: using found: {{.*}}found
+# ERROR-NEXT: fatal: couldn't find 'not-found' program

llvm/utils/lit/tests/use-tool-search-env.py

-## Show that lit reports the path to tools picked up via the use_llvm_tool
-## function when the tool is found via an environment variable.
-
-# RUN: %{lit} %{inputs}/use-tool-search-env 2>&1 | \
-# RUN:   FileCheck %s -DDIR=%p
-
-# CHECK: note: using test-tool: [[DIR]]{{[\\/]}}Inputs{{[\\/]}}use-tool-search-env{{[\\/]}}test.tool