[lit] Avoid os.path.realpath in lit.py due to MAX_PATH limitations on Windows

[MrTrillian]

Summary

lit.py uses os.path.realpath on file paths. Somewhere between Python 3.7 and 3.9, os.path.realpath was updated to resolve substitute drives on Windows (subst S: C:\Long\Path\To\My\Code). This is a problem because it prevents using substitute drives to work around MAX_PATH path length limitations on Windows.

We run into this while building & testing, the Swift compiler on Windows, which uses a substitute drive in CI to shorten the workspace directory. cmake builds without resolving the substitute drive and can apply its logic to avoid output files exceeding MAX_PATH. However, when running tests, lit.py's use of os.path.realpath will resolve the substitute drive (with newer Python versions), resulting in some paths being longer than MAX_PATH, which cause all kinds of failures (for example rd in tests fails, or link.exe fails, etc).

How tested: Ran check-all, and lit tests, saw no failures

> ninja -C build check-all
Testing Time: 262.63s
  Skipped          :    24
  Unsupported      :  2074
  Passed           : 51812
  Expectedly Failed:   167

> python utils\lit\lit.py --path ..\build\bin utils\lit\tests
Testing Time: 12.17s
  Unsupported:  6
  Passed     : 47

[rnk]

lgtm

llvm/utils/lit/lit/discovery.py

59

I see this uses realpath and that must be intentional.

[MrTrillian]

llvm/utils/lit/lit/discovery.py

59

@rnk The reason was that this is looking up a param from outside the script and I thought it might regress something, but let me try locally if I can convert this and the one in def search below as well.

[MrTrillian]

Converted final two instances of realpath to abspath. Reran tests.

[nlopes]

This commit started failing running llvm-lit outside the llvm directory:

$ ~/llvm/build/bin/llvm-lit -vv ~/llvm/llvm/test/Transforms
llvm-lit: /bitbucket/nlopes/llvm/llvm/utils/lit/lit/TestingConfig.py:151: fatal: unable to parse config file '/home/nlopes/llvm/llvm/test/lit.cfg.py', traceback: Traceback (most recent call last):
  File "/bitbucket/nlopes/llvm/llvm/utils/lit/lit/TestingConfig.py", line 139, in load_from_path
    exec(compile(data, path, "exec"), cfg_globals, None)
  File "/home/nlopes/llvm/llvm/test/lit.cfg.py", line 21, in <module>
    config.test_format = lit.formats.ShTest(not llvm_config.use_lit_shell)
AttributeError: 'NoneType' object has no attribute 'use_lit_shell'

Any idea of what's going on?

[MrTrillian]

In D152709#4423897, @nlopes wrote:

This commit started failing running llvm-lit outside the llvm directory:

$ ~/llvm/build/bin/llvm-lit -vv ~/llvm/llvm/test/Transforms
llvm-lit: /bitbucket/nlopes/llvm/llvm/utils/lit/lit/TestingConfig.py:151: fatal: unable to parse config file '/home/nlopes/llvm/llvm/test/lit.cfg.py', traceback: Traceback (most recent call last):
  File "/bitbucket/nlopes/llvm/llvm/utils/lit/lit/TestingConfig.py", line 139, in load_from_path
    exec(compile(data, path, "exec"), cfg_globals, None)
  File "/home/nlopes/llvm/llvm/test/lit.cfg.py", line 21, in <module>
    config.test_format = lit.formats.ShTest(not llvm_config.use_lit_shell)
AttributeError: 'NoneType' object has no attribute 'use_lit_shell'

Any idea of what's going on?

I suspect it has to do with the remaining os.path.realpath in llvm\utils\llvm-lit\llvm-lit.in. How can I repro this?

[Jake-Egan]

This caused a failure on AIX
https://lab.llvm.org/buildbot/#/builders/214/builds/7990/steps/6/logs/FAIL__lit___use-llvm-tool_py

It looks like the test expects /home but on AIX we use a temporary directory /scratch

[nlopes]

In D152709#4424510, @MrTrillian wrote:

In D152709#4423897, @nlopes wrote:

This commit started failing running llvm-lit outside the llvm directory:

$ ~/llvm/build/bin/llvm-lit -vv ~/llvm/llvm/test/Transforms
llvm-lit: /bitbucket/nlopes/llvm/llvm/utils/lit/lit/TestingConfig.py:151: fatal: unable to parse config file '/home/nlopes/llvm/llvm/test/lit.cfg.py', traceback: Traceback (most recent call last):
  File "/bitbucket/nlopes/llvm/llvm/utils/lit/lit/TestingConfig.py", line 139, in load_from_path
    exec(compile(data, path, "exec"), cfg_globals, None)
  File "/home/nlopes/llvm/llvm/test/lit.cfg.py", line 21, in <module>
    config.test_format = lit.formats.ShTest(not llvm_config.use_lit_shell)
AttributeError: 'NoneType' object has no attribute 'use_lit_shell'

Any idea of what's going on?

I suspect it has to do with the remaining os.path.realpath in llvm\utils\llvm-lit\llvm-lit.in. How can I repro this?

my ~/llvm dir is a symlink

[MrTrillian]

In D152709#4424726, @nlopes wrote:

my ~/llvm dir is a symlink

This change doesn't break symlinks. It would only cause issues if some code resolves symlinks to real paths whereas other code does not and we expect the two paths to be the same.

llvm/utils/lit/lit/TestRunner.py

def __init__(self, cwd, env):
         self.cwd = cwd
         self.env = dict(env)
         self.dirStack = []
 
     def change_dir(self, newdir):
         if os.path.isabs(newdir):
             self.cwd = newdir
         else:
-            self.cwd = os.path.realpath(os.path.join(self.cwd, newdir))
+            self.cwd = os.path.abspath(os.path.join(self.cwd, newdir))
 
 
 class TimeoutHelper(object):
     """
     Object used to helper manage enforcing a timeout in
     _executeShCmd(). It is passed through recursive calls
     to collect processes that have been executed so that when
     the timeout happens they can be killed.
def executeBuiltinMkdir(cmd, cmd_shenv):
 
     stderr = StringIO()
     exitCode = 0
     for dir in args:
         cwd = cmd_shenv.cwd
         dir = to_unicode(dir) if kIsWindows else to_bytes(dir)
         cwd = to_unicode(cwd) if kIsWindows else to_bytes(cwd)
         if not os.path.isabs(dir):
-            dir = os.path.realpath(os.path.join(cwd, dir))
+            dir = os.path.abspath(os.path.join(cwd, dir))
         if parent:
             lit.util.mkdir_p(dir)
         else:
             try:
                 lit.util.mkdir(dir)
             except OSError as err:
                 stderr.write("Error: 'mkdir' command failed, %s\n" % str(err))
                 exitCode = 1
def executeBuiltinRm(cmd, cmd_shenv):
 
     stderr = StringIO()
     exitCode = 0
     for path in args:
         cwd = cmd_shenv.cwd
         path = to_unicode(path) if kIsWindows else to_bytes(path)
         cwd = to_unicode(cwd) if kIsWindows else to_bytes(cwd)
         if not os.path.isabs(path):
-            path = os.path.realpath(os.path.join(cwd, path))
+            path = os.path.abspath(os.path.join(cwd, path))
         if force and not os.path.exists(path):
             continue
         try:
             if os.path.isdir(path):
                 if not recursive:
                     stderr.write("Error: %s is a directory\n" % path)
                     exitCode = 1
                 if platform.system() == "Windows":

llvm/utils/lit/lit/builtin_commands/diff.py

def main(argv):
     if len(args) != 2:
         sys.stderr.write("Error: missing or extra operand\n")
         sys.exit(1)
 
     exitCode = 0
     try:
         for file in args:
             if file != "-" and not os.path.isabs(file):
-                file = os.path.realpath(os.path.join(os.getcwd(), file))
+                file = os.path.abspath(os.path.join(os.getcwd(), file))
 
             if flags.recursive_diff:
                 if file == "-":
                     sys.stderr.write("Error: cannot recursively compare '-'\n")
                     sys.exit(1)
                 dir_trees.append(getDirTree(file))
             else:
                 filepaths.append(file)

llvm/utils/lit/lit/discovery.py

def search1(path):
         # This is a private builtin parameter which can be used to perform
         # translation of configuration paths.  Specifically, this parameter
         # can be set to a dictionary that the discovery process will consult
         # when it finds a configuration it is about to load.  If the given
         # path is in the map, the value of that key is a path to the
         # configuration to load instead.
         config_map = litConfig.params.get("config_map")
         if config_map:
-            cfgpath = os.path.realpath(cfgpath)
-            target = config_map.get(os.path.normcase(cfgpath))
+            target = config_map.get(os.path.normcase(os.path.abspath(cfgpath)))
             if target:
                 cfgpath = target
 
         # We found a test suite, create a new config for it and load it.
         if litConfig.debug:
             litConfig.note("loading suite config %r" % cfgpath)
 
         cfg = TestingConfig.fromdefaults(litConfig)
         cfg.load_from_path(cfgpath, litConfig)
-        source_root = os.path.realpath(cfg.test_source_root or path)
-        exec_root = os.path.realpath(cfg.test_exec_root or path)
+        source_root = os.path.abspath(cfg.test_source_root or path)
+        exec_root = os.path.abspath(cfg.test_exec_root or path)
         return Test.TestSuite(cfg.name, source_root, exec_root, cfg), ()
 
     def search(path):
         # Check for an already instantiated test suite.
-        real_path = os.path.realpath(path)
-        res = cache.get(real_path)
+        full_path = os.path.normcase(os.path.abspath(path))
+        res = cache.get(full_path)
         if res is None:
-            cache[real_path] = res = search1(path)
+            cache[full_path] = res = search1(path)
         return res
 
     # Canonicalize the path.
     item = os.path.normpath(os.path.join(os.getcwd(), item))
 
     # Skip files and virtual components.
     components = []
     while not os.path.isdir(item):

llvm/utils/lit/tests/Inputs/config-map-discovery/driver.py

 import lit.util
 import os
 import sys
 
 main_config = sys.argv[1]
-main_config = os.path.realpath(main_config)
+main_config = os.path.abspath(main_config)
 main_config = os.path.normcase(main_config)
 
 config_map = {main_config: sys.argv[2]}
 builtin_parameters = {"config_map": config_map}
 
 if __name__ == "__main__":
     from lit.main import main
 
     main_config_dir = os.path.dirname(main_config)
     sys.argv = [sys.argv[0]] + sys.argv[3:] + [main_config_dir]
     main(builtin_parameters)

llvm/utils/lit/tests/Inputs/config-map-discovery/lit.alt.cfg

 import lit.formats
 import lit.util
 config.name = 'config-map'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest()
 
 import os
-config.test_exec_root = os.path.realpath(os.path.dirname(__file__))
+config.test_exec_root = os.path.abspath(os.path.dirname(__file__))
 config.test_source_root = os.path.join(config.test_exec_root, "tests")

llvm/utils/lit/tests/Inputs/use-llvm-tool-required/lit.cfg

 import lit.formats
 
 config.name = "use-llvm-tool-required"
 config.suffixes = [".txt"]
 config.test_format = lit.formats.ShTest()
 config.test_source_root = None
 config.test_exec_root = None
 import os.path
 
-config.llvm_tools_dir = os.path.realpath(os.path.dirname(__file__))
+config.llvm_tools_dir = os.path.abspath(os.path.dirname(__file__))
 import lit.llvm
 
 lit.llvm.initialize(lit_config, config)
 lit.llvm.llvm_config.use_llvm_tool("found", required=True)
 lit.llvm.llvm_config.use_llvm_tool("not-found", required=True)

llvm/utils/lit/tests/Inputs/use-llvm-tool/lit.cfg

 import lit.formats
 
 config.name = "use-llvm-tool"
 config.suffixes = [".txt"]
 config.test_format = lit.formats.ShTest()
 config.test_source_root = None
 config.test_exec_root = None
 import os.path
 
-this_dir = os.path.realpath(os.path.dirname(__file__))
+this_dir = os.path.abspath(os.path.dirname(__file__))
 config.llvm_tools_dir = os.path.join(this_dir, "build")
 import lit.llvm
 
 lit.llvm.initialize(lit_config, config)
 lit.llvm.llvm_config.with_environment("CASE1", os.path.join(this_dir, "env-case1"))
 lit.llvm.llvm_config.with_environment("CASE6", os.path.join(this_dir, "env-case6"))
 lit.llvm.llvm_config.with_environment(
     "PATH", os.path.join(this_dir, "path"), append_path=True