[dotest] Make a missing FileCheck binary a warning, not an error

[vsk]

Summary

This allows bots which haven't updated to pass in --filecheck to dotest.py to run more tests. FileCheck-dependent tests will continue to fail.

[zturner]

Why is FileCheck missing in the first place?

[stella.stamenova]

I spent a bit more time on this and I think I have a better idea of what's happening.

Vedant added functionality so that we can use FileCheck-style checks inside non-lit LLDB tests. Part of that change was to require a parameter --filecheck to be passed when calling the scripts (such as dotests.py). Also as part of that change, the CMakeFiles now propagate a default location for FileCheck as an input to the scripts, so that if you just run the tests by default using any of the generated outputs (for ninja, or make, or VS), they work correctly

The bots (at least some bots) don't just use the generated solutions to run the tests. There's some complicated logic in the zorg repo to create the list of parameters that are passed to the bots and they do not include the new --filecheck argument. So it's not individual bots that need to be updated, but the test harness. I haven't really looked at it before, so I'm still trying to understand it - especially how we would get FileCheck to any of the bots that run the tests remotely using lldbserver today.

[stella.stamenova]

The failing bots are not windows bots but Linux bots. It looks like you only updated the configurations for xcode.

I think the file that needs to be updated is:

zorg\buildbot\builders\LLDBBuilder.py

[jasonmolenda]

I've been meaning to look at this - for the command line style "cd test; ./dotest.py" approach, we could use similar logic to what packages/Python/lldbsuite/test/dotest.py does in getOutputPaths() where it finds an lldb binary in the "typical" location. for an Xcode style build, this would be in a directory like ../llvm-build/Release+Asserts/x86_64/bin/FileCheck where the build-style ("Release+Asserts") is going to vary but I'm guessing that the build configuration of the FileCheck binary picked is not important.

[jasonmolenda]

(obviously this would only be the behavior if a filecheck path was not specified)

[stella.stamenova]

@jasonmolenda That's not a bad idea. If you have a change ready, you could send it for review ;)

[stella.stamenova]

@vsk Could you also make the filecheck function explicitly fail when it doesn't have a path to the binary? That way it will be fairly obvious what the problem is and the error combined with the warning should be clear enough.

[vsk]

• Address comments from @stella.stamenova

[stella.stamenova]

Thanks!

[stella.stamenova]

The bots are now failing because lexists doesn't handle NoneType correctly:

  File "/lldb-buildbot/lldbSlave/buildWorkingDir/llvm/tools/lldb/packages/Python/lldbsuite/test/configuration.py", line 191, in get_filecheck_path
    if os.path.lexists(filecheck):
  File "/lldb-buildbot/virenv/lib/python2.7/posixpath.py", line 152, in lexists
    os.lstat(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found

I think get_filecheck_path should check if filecheck is set first before calling lexists:

if filecheck and os.path.lexists(filecheck):
        return filecheck

[stella.stamenova]

Progress! The tests are running again and only a handful fail (some because of FileCheck, others for various other reasons).

http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/29816

It looks like there a number of tests that have failures unrelated to FileCheck, so anyone who has committed something recently should probably check if they broke a test or two.

lldb/trunk/packages/Python/lldbsuite/test/configuration.py

def get_absolute_path_to_root_test_dir():
 
     return os.path.dirname(os.path.realpath(__file__))
 
 
 def get_filecheck_path():
     """
     Get the path to the FileCheck testing tool.
     """
-    assert os.path.lexists(filecheck)
+    if os.path.lexists(filecheck):
         return filecheck

lldb/trunk/packages/Python/lldbsuite/test/dotest.py

def parseOptionsAndInitTestdirs():
 
     if args.dsymutil:
       os.environ['DSYMUTIL'] = args.dsymutil
     elif platform_system == 'Darwin':
       os.environ['DSYMUTIL'] = seven.get_command_output(
           'xcrun -find -toolchain default dsymutil')
 
     if args.filecheck:
-        # The CMake build passes in a path to a working FileCheck binary.
+        # The lldb-dotest script produced by the CMake build passes in a path
+        # to a working FileCheck binary. So does one specific Xcode project
+        # target. However, when invoking dotest.py directly, a valid --filecheck
+        # option needs to be given.
         configuration.filecheck = os.path.abspath(args.filecheck)
-    else:
-        logging.error('No valid FileCheck executable; aborting...')
-        sys.exit(-1)
+
+    if not configuration.get_filecheck_path():
+        logging.warning('No valid FileCheck executable; some tests may fail...')
+        logging.warning('(Double-check the --filecheck argument to dotest.py)')
 
     if args.channels:
         lldbtest_config.channels = args.channels
 
     if args.log_success:
         lldbtest_config.log_success = args.log_success
 
     if args.out_of_tree_debugserver:

lldb/trunk/packages/Python/lldbsuite/test/lldbtest.py

def filecheck(
         # LLDB inline tests, assume that the check file is a relative path to
         # a file within the inline test directory.
         if check_file.endswith('.pyc'):
             check_file = check_file[:-1]
         check_file_abs = os.path.abspath(check_file)
 
         # Run FileCheck.
         filecheck_bin = configuration.get_filecheck_path()
+        if not filecheck_bin:
+            self.assertTrue(False, "No valid FileCheck executable specified")
         filecheck_args = [filecheck_bin, check_file_abs]
         if filecheck_options:
             filecheck_args.append(filecheck_options)
         subproc = Popen(filecheck_args, stdin=PIPE, stdout=PIPE, stderr=PIPE, universal_newlines = True)
         cmd_stdout, cmd_stderr = subproc.communicate(input=output)
         cmd_status = subproc.returncode
 
         filecheck_cmd = " ".join(filecheck_args)