[lit] Keep original cfg file case around.

[thakis]

Summary

There's been some back and forth if the cfg paths in the
config_map should be normcase()d. The argument for is that
it allows using all-lower spelling in cmd on Windows, the
argument against that doing so is lossy. (See r313918.)

Before the relative-paths-in-generated-lit.site.cfg.py work,
there was no downside to calling normcase(), but with it
we need a hack to recover the original case.

This time, normcase() the hashtable key, but store the original
cased key in addition to the value. This fixes both cons, at the
cost of a few bytes more memory.

[thakis]

llvm/utils/lit/lit/formats/googletest.py

15

This is somewhat unrelated; I believe this call is unneeded after https://reviews.llvm.org/D34855

llvm/utils/lit/lit/util.py

19

This is also unrelated; this function seems to be dead.

[hans]

Seems fine to me. Maybe try landing the unrelated changes separately to minimize things.

[Meinersbur]

llvm/utils/lit/lit/discovery.py

67

Although it turned out to be unrelated to this patch, I leave this comment here:

check-clang started to fail with these kind of error messages:

FAIL: Clang :: SemaObjC/selector-4.m (1 of 1)
******************** TEST 'Clang :: SemaObjC/selector-4.m' FAILED ********************
Script:
--
: 'RUN: at line 1';   c:\users\meinersbur\build\llvm-project\release\bin\clang.exe -cc1 -internal-isystem c:\users\meinersbur\build\llvm-project\release\lib\clang\11.0.0\include -nostdsysteminc -Wselector -x objective-c C:\users\meinersbur\src\llvm-project\clang\test\SemaObjC\selector-4.m -include C:\users\meinersbur\src\llvm-project\clang\test\SemaObjC\selector-4.m -verify
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "c:\users\meinersbur\build\llvm-project\release\bin\clang.exe" "-cc1" "-internal-isystem" "c:\users\meinersbur\build\llvm-project\release\lib\clang\11.0.0\include" "-nostdsysteminc" "-Wselector" "-x" "objective-c" "C:\users\meinersbur\src\llvm-project\clang\test\SemaObjC\selector-4.m" "-include" "C:\users\meinersbur\src\llvm-project\clang\test\SemaObjC\selector-4.m" "-verify"
# command stderr:
error: 'warning' diagnostics seen but not expected:
  Line 1: non-portable path to file '"C\\Users\meinersbur\src\llvm-project\clang\test\SemaObjC\selector-4.m"'; specified path differs in case from file name on disk
1 error generated.

error: command failed with exit status: 1

The C\\ resulted from some printing problem, but what cause the issue was -include C:\users\.... with lower-case 'U' while the folder is upper case in the file system. It is the expansion of %s by lit.

Why would lit pass the wrong case? I suspected this patch.

The lower-case "user" originates from __file__ in Clang's lit.cfg.py assigned to config.test_source_root, which is used by Test.getSourcePath() to use as the substitution of %s. No idea why __file__ would not use the proper file's case, bit this call of os.path.realpath might have been intended to return the correct case.

Except, it doesn't. At least in the Python 3.7 that ships with the current Visual Studio. However, Python 3.8 changes the case from user to User. So my solution is to use Python 3.8 from now on, but I have no idea why it worked before or why nobody else seem to have the same problem. I leave this here in case might be useful to someone else.

llvm/cmake/modules/AddLLVM.cmake

sys.stdout.write(';'.join(relpath(p) for p in sys.argv[2].split(';')))"
     OUTPUT_VARIABLE pathlist_relative)
   set(${out_pathlist} "${pathlist_relative}" PARENT_SCOPE)
 endfunction()
 
 # Converts a file that's relative to the current python file to an absolute
 # path. Since this uses __file__, it has to be emitted into python files that
 # use it and can't be in a lit module. Use with make_paths_relative().
 string(CONCAT LLVM_LIT_PATH_FUNCTION
-  # Lit converts config paths to lower case in discovery.py, before
-  # loading the config. This causes __file__ to be all lower-case (including
-  # the drive letter), but several clang tests pass -include %s and a
-  # clang warning checks that passed case matches on-disk cache. So it's
-  # important that this restores the on-disk case of the prefix.
   "# Allow generated file to be relocatable.\n"
   "def path(p):\n"
   "    if not p: return ''\n"
   "    p = os.path.join(os.path.dirname(os.path.abspath(__file__)), p)\n"
-  "    if os.name == 'nt' and os.path.isabs(p): return p[0].upper() + p[1:]\n"
   "    return p\n"
   )
 
 # This function provides an automatic way to 'configure'-like generate a file
 # based on a set of common and custom variables, specifically targeting the
 # variables needed for the 'lit.site.cfg' files. This function bundles the
 # common variables that any Lit instance is likely to need, and custom
 # variables can be passed in.

llvm/utils/lit/lit/discovery.py

def search1(path):
         # translation of configuration paths.  Specifically, this parameter
         # can be set to a dictionary that the discovery process will consult
         # when it finds a configuration it is about to load.  If the given
         # path is in the map, the value of that key is a path to the
         # configuration to load instead.
         config_map = litConfig.params.get('config_map')
         if config_map:
             cfgpath = os.path.realpath(cfgpath)
-            cfgpath = os.path.normcase(cfgpath)
-            target = config_map.get(cfgpath)
+            t, target = config_map.get(os.path.normcase(cfgpath), (None, None))
+            assert t is None or t == cfgpath
             if target:
                 cfgpath = target
 
         # We found a test suite, create a new config for it and load it.
         if litConfig.debug:
             litConfig.note('loading suite config %r' % cfgpath)
 
         cfg = TestingConfig.fromdefaults(litConfig)
         cfg.load_from_path(cfgpath, litConfig)
         source_root = os.path.realpath(cfg.test_source_root or path)
         exec_root = os.path.realpath(cfg.test_exec_root or path)
         return Test.TestSuite(cfg.name, source_root, exec_root, cfg), ()
 
     def search(path):
         # Check for an already instantiated test suite.
         real_path = os.path.realpath(path)
         res = cache.get(real_path)
         if res is None:

llvm/utils/lit/lit/formats/googletest.py

 from __future__ import absolute_import
 import os
 import subprocess
 import sys
 
 import lit.Test
 import lit.TestRunner
 import lit.util
 from .base import TestFormat
 
 kIsWindows = sys.platform in ['win32', 'cygwin']
 
 class GoogleTest(TestFormat):
     def __init__(self, test_sub_dirs, test_suffix):
-        self.test_sub_dirs = os.path.normcase(str(test_sub_dirs)).split(';')
+        self.test_sub_dirs = str(test_sub_dirs).split(';')
 
         # On Windows, assume tests will also end in '.exe'.
         exe_suffix = str(test_suffix)
         if kIsWindows:
             exe_suffix += '.exe'
 
         # Also check for .py files for testing purposes.
         self.test_suffixes = {exe_suffix, test_suffix + '.py'}

llvm/utils/lit/lit/llvm/config.py

def __init__(self, lit_config, config):
             gmalloc_path_str = lit_config.params.get('gmalloc_path',
                                                      '/usr/lib/libgmalloc.dylib')
             if gmalloc_path_str is not None:
                 self.with_environment(
                     'DYLD_INSERT_LIBRARIES', gmalloc_path_str)
 
     def with_environment(self, variable, value, append_path=False):
         if append_path:
-            # For paths, we should be able to take a list of them and process all
-            # of them.
+            # For paths, we should be able to take a list of them and process
+            # all of them.
             paths_to_add = value
             if lit.util.is_string(paths_to_add):
                 paths_to_add = [paths_to_add]
 
             def norm(x):
                 return os.path.normcase(os.path.normpath(x))
 
             current_paths = self.config.environment.get(variable, None)
             if current_paths:
                 current_paths = current_paths.split(os.path.pathsep)
                 paths = [norm(p) for p in current_paths]
             else:
                 paths = []
 
             # If we are passed a list [a b c], then iterating this list forwards
             # and adding each to the beginning would result in b c a.  So we
             # need to iterate in reverse to end up with the original ordering.
             for p in reversed(paths_to_add):
-                # Move it to the front if it already exists, otherwise insert it at the
-                # beginning.
+                # Move it to the front if it already exists, otherwise insert
+                # it at the beginning.
                 p = norm(p)
                 try:
                     paths.remove(p)
                 except ValueError:
                     pass
                 paths = [p] + paths
             value = os.pathsep.join(paths)
         self.config.environment[variable] = value

llvm/utils/lit/lit/util.py

 from __future__ import print_function
 
 import errno
 import itertools
 import math
 import numbers
 import os
 import platform
 import signal
 import subprocess
 import sys
 import threading
 
 
-def norm_path(path):
-    path = os.path.realpath(path)
-    path = os.path.normpath(path)
-    path = os.path.normcase(path)
-    return path
-
-
 def is_string(value):
     try:
         # Python 2 and Python 3 are different here.
         return isinstance(value, basestring)
     except NameError:
         return isinstance(value, str)
 
 

llvm/utils/lit/tests/Inputs/config-map-discovery/driver.py

 import lit.util
 import os
 import sys
 
 main_config = sys.argv[1]
 main_config = os.path.realpath(main_config)
 main_config = os.path.normcase(main_config)
 
-config_map = {main_config : sys.argv[2]}
+config_map = {os.path.normcase(main_config) : (main_config, sys.argv[2])}
 builtin_parameters = {'config_map' : config_map}
 
 if __name__=='__main__':
     from lit.main import main
     main_config_dir = os.path.dirname(main_config)
     sys.argv = [sys.argv[0]] + sys.argv[3:] + [main_config_dir]
     main(builtin_parameters)

llvm/utils/llvm-lit/llvm-lit.in

 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 import os
 import sys
 
 config_map = {}
 
 def map_config(source_dir, site_config):
     global config_map
     source_dir = os.path.realpath(source_dir)
-    source_dir = os.path.normcase(source_dir)
     site_config = os.path.normpath(site_config)
-    config_map[source_dir] = site_config
+    config_map[os.path.normcase(source_dir)] = source_dir, site_config
 
 # Set up some builtin parameters, so that by default the LLVM test suite
 # configuration file knows how to find the object tree.
 builtin_parameters = { 'build_mode' : "@BUILD_MODE@" }
 
 @LLVM_LIT_CONFIG_MAP@
 
 builtin_parameters['config_map'] = config_map
 
 # Make sure we can find the lit package.
 llvm_source_root = path("@LLVM_SOURCE_DIR@")
 sys.path.insert(0, os.path.join(llvm_source_root, 'utils', 'lit'))
 
 if __name__=='__main__':
     from lit.main import main
     main(builtin_parameters)