[lit] Correctly expand globs relative to their intended directory.

[graydon]

Summary

Lit emulates shell semantics in its RUN: line interpretation. As
part of this emulation, it sniffs 'cd' commands in the command
to execute (such as "cd /some/path && ...") and then uses the
argument to 'cd' as a directory in which it interprets subsequent
elements of the command, in particular globs.

When expanding a non-absolute (relative) glob, however, it currently
takes a naive and (I argue) incorrect approach to them: it prepends
the inferred working directory's absolute path to the glob and then
expands the glob as an absolute path. This identifies the correct
_files_ but it names them absolutely, where the glob was written
as a relative glob. In a few corner cases, the difference between
an absolute and a relative filename is _meaningful_ to the program
under test, and should be preserved even when expanding globs.

This change works by stripping the absolute prefix back off the
expansion of a relative glob (using relpath), which makes the
behaviour at least more shell-like.

[graydon]

Add test, fix typo.

[graydon]

For context, the motivation here is a somewhat subtle testcase over in swift driver land in which we need to construct an argv for the driver that is not longer than kern.maxarg, but that will be transformed into such an over-long argv when composed with an absolute output directory and passed to a frontend. This test has been flaky on CI because we were globbing the input to the driver (under a "cd ... && ...") and lit was sticking the absolute path of the testsuite-on-the-CI-machine on the front of all the globbed inputs. Sometimes this meant the input argv to the driver would overflow, other times not; depends on which CI host and subdir it's running inside of. Obviously not great.

See https://github.com/apple/swift/pull/19149 for current workaround on swift side. I'm also fixing it here so that nobody has to hunt this particular family of errors down again.

utils/lit/lit/ShCommands.py

def __eq__(self, other):
             return False
 
         return (self.pattern == other.pattern)
 
     def resolve(self, cwd):
         import glob
         import os
         if os.path.isabs(self.pattern):
-           abspath = self.pattern
+            results = glob.glob(self.pattern)
         else:
-            abspath = os.path.join(cwd, self.pattern)
-        results = glob.glob(abspath)
+            # For a non-absolute pattern, we want to match "relative to
+            # cwd"; unfortunately glob.glob() will only match relative to
+            # the process working directory, or else an absolute path. So
+            # rather than change working directory to and from cwd in order
+            # to glob, we extend the relative pattern to an absolute
+            # pattern, then glob, then map the results back to relative
+            # paths using relpath.
+            abs_pat = os.path.join(cwd, self.pattern)
+            abs_glob = glob.glob(abs_pat)
+            results = [os.path.relpath(f, cwd) for f in abs_glob]
         return [self.pattern] if len(results) == 0 else results
 
 class Pipeline:
     def __init__(self, commands, negate=False, pipe_err=False):
         self.commands = commands
         self.negate = negate
         self.pipe_err = pipe_err
 

utils/lit/tests/shtest-glob-relative.py

+# Here we're checking that glob-expanding a relative glob within a "cd ..." does not
+# wind up prepending the working directory absolute path to the glob expansion.
+#
+# RUN: mkdir -p %t.dir/test/deep/path
+# RUN: touch %t.dir/test/deep/path/file.txt
+# RUN: cd %t.dir/test/deep/path && ls file.* > %t.out
+# RUN: FileCheck --input-file %t.out %s
+# CHECK-NOT: deep/path