[Utils] Refactor update_cc_test_checks.py to use shutil

[jmciver]

Summary

The package distutils is deprecated and removal is planned for Python 3.12. All calls to distutils.spawn.find_executable are replaced with local version of find_executable which makes use of shutils.which.

[arichardson]

LGTM

[jmciver]

@arichardson thanks for taking the time to review!

Can you please commit on my behalf?

Name: John McIver
Email: [e-mail]

[dyung]

Hi @jmciver and @nlopes, this change is causing failures on the PS5 Windows bot, can you take a look and revert if it will take a while to investigate?

https://lab.llvm.org/buildbot/#/builders/216/builds/10030

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "rm" "-rf" "Z:\test\build\tools\clang\test\utils\update_cc_test_checks\Output\check-globals.test.tmp"
$ "mkdir" "Z:\test\build\tools\clang\test\utils\update_cc_test_checks\Output\check-globals.test.tmp"
$ ":" "RUN: at line 5"
$ "cp" "Z:\test\llvm-project\clang\test\utils\update_cc_test_checks/Inputs/check-globals.c" "Z:\test\build\tools\clang\test\utils\update_cc_test_checks\Output\check-globals.test.tmp/norm.c"
$ ":" "RUN: at line 6"
$ "C:/Program Files/Python310/python.exe" "Z:\test\llvm-project\llvm\utils\update_cc_test_checks.py" "--clang" "Z:\test\build\bin\clang" "--opt" "Z:\test\build\bin\opt" "Z:\test\build\tools\clang\test\utils\update_cc_test_checks\Output\check-globals.test.tmp/norm.c" "--check-globals"
# command stderr:
Please specify --llvm-bin or --clang

[abrachet]

We are also seeing errors on Windows, see https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci/clang-windows-x64/b8802513693562827489/overview

[bd1976llvm]

I reverted this change as it breaks the tests on windows. It seems that distutils.spawn.find_executable is not exactly equivalent to shutil.which on Windows as the former adds ".exe" and the latter adds the extensions from PATHEXT. Furthermore we are invoking these functions on a full path e.g. "C:\p\15\vs2019\bin\clang" and looking at the docs I'm not sure what the behaviour of either function should be in that case.

[thakis]

Also broke my windows bot: http://45.33.8.238/win/66470/step_7.txt

Thanks for the revert!

[jmciver]

Thanks all for the feedback and the revert. I'll look into this more.

[jmciver]

Add a local find_executable function to update_cc_test_checks.py that takes the necessary functionality from distutils.spawn.find_executable and makes use of shutil.which.

The local find_executable:

• Appends .exe to the executable name when the environment is win32
• Test if executable is specified by an absolute or relative path and if iit is return the absolute or relative executable

I have tested in both Windows 11 and Linux using PATH, --clang (both absolute and relative), and --llvm-bin (both absolute and relative).

https://github.com/python/cpython/blob/main/Lib/distutils/spawn.py#L95
https://github.com/python/cpython/blob/main/Lib/shutil.py#L1441

[jmciver]

Add a local function find_executable to update_cc_test_checks.py, which takes the necessary functionality from distutils.spawn.find_executable and makes use of shutil.which.

The local find_executable appends .exe to the executable name when the environment is win32. shutil.which correctly handles executables that are specified by absolute or relative path.

I have tested in both Windows 11 and Linux using PATH, --clang (both absolute and relative), and --llvm-bin (both absolute and relative).

https://github.com/python/cpython/blob/main/Lib/distutils/spawn.py#L95
https://github.com/python/cpython/blob/main/Lib/shutil.py#L1441

[bd1976llvm]

Thanks. LGTM.

llvm/utils/update_cc_test_checks.py

 % utils/update_cc_test_checks.py --llvm-bin=release/bin test/a.cc
 % utils/update_cc_test_checks.py --clang=release/bin/clang /tmp/c/a.cc
 '''
 
 from __future__ import print_function
 
 import argparse
 import collections
-import distutils.spawn
 import json
 import os
 import re
 import shlex
+import shutil
 import subprocess
 import sys
 import tempfile
 
 from UpdateTestChecks import common
 
 SUBST = {
     '%clang': [],
else:
       args.clang = os.path.join(args.llvm_bin, 'clang')
   if not args.opt:
     if not args.llvm_bin:
       args.opt = 'opt'
     else:
       args.opt = os.path.join(args.llvm_bin, 'opt')
 
 
+def find_executable(executable):
+  _, ext = os.path.splitext(executable)
+  if sys.platform == 'win32' and ext != '.exe':
+    executable = executable + '.exe'
+
+  return shutil.which(executable)
+
+
 def config():
   parser = argparse.ArgumentParser(
       description=__doc__,
       formatter_class=argparse.RawTextHelpFormatter)
   parser.add_argument('--llvm-bin', help='llvm $prefix/bin path')
   parser.add_argument('--clang',
                       help='"clang" executable, defaults to $llvm_bin/clang')
   parser.add_argument('--clang-args', default=[], type=str_to_commandline,
def config():
   parser.add_argument('--check-attributes', action='store_true',
                       help='Check "Function Attributes" for functions')
   parser.add_argument('--check-globals', action='store_true',
                       help='Check global entries (global variables, metadata, attribute sets, ...) for functions')
   parser.add_argument('tests', nargs='+')
   args = common.parse_commandline_args(parser)
   infer_dependent_args(args)
 
-  if not distutils.spawn.find_executable(args.clang):
+  if not find_executable(args.clang):
     print('Please specify --llvm-bin or --clang', file=sys.stderr)
     sys.exit(1)
 
   # Determine the builtin includes directory so that we can update tests that
   # depend on the builtin headers. See get_clang_builtin_include_dir() and
   # use_clang() in llvm/utils/lit/lit/llvm/config.py.
   try:
     builtin_include_dir = subprocess.check_output(
       [args.clang, '-print-file-name=include']).decode().strip()
     SUBST['%clang_cc1'] = ['-cc1', '-internal-isystem', builtin_include_dir,
                            '-nostdsysteminc']
   except subprocess.CalledProcessError:
     common.warn('Could not determine clang builtins directory, some tests '
                 'might not update correctly.')
 
-  if not distutils.spawn.find_executable(args.opt):
+  if not find_executable(args.opt):
     # Many uses of this tool will not need an opt binary, because it's only
     # needed for updating a test that runs clang | opt | FileCheck. So we
     # defer this error message until we find that opt is actually needed.
     args.opt = None
 
   return args, parser