Allow use of self.filecheck in LLDB tests (c.f self.expect)

[vsk]

Summary

Add a "filecheck" method to the LLDB test base. This allows test authors
to pattern match command output using FileCheck, making it possible to
write stricter tests than what self.expect allows.

For context (motivation, examples of stricter checking, etc), see the
lldb-dev thread: "Using FileCheck in lldb inline tests".

[teemperor]

lldb/packages/Python/lldbsuite/test/configuration.py

194

This is true if we build LLDB as part of LLVM, but this doesn't seem correct if LLDB is built against an existing LLVM (which is in general supported in LLDB's CMake if IIUC). I think the best way is to pass the binary directory for this to dotest and configure it in the lit.cfg.

204

Windows itself can probably handle the fact that the .exe suffix is missing for FileCheck, but this assert will fail in this situation.

lldb/packages/Python/lldbsuite/test/lldbtest.py

51

I think that list is supposed to be sorted alphabetically

2166

Making a temporary file and opening it later by name is not really supported (see the NamedTemporaryFile docs). Could we maybe pass the output to FileCheck via stdin? That would also save us from the detour over the fs.

2181

We should quote the paths in here. Or at least, the temporary file path should be quoted as it will contain spaces on some setups.

[stella.stamenova]

lldb/packages/Python/lldbsuite/test/configuration.py

194

Rather than making the change here, as @teemperor pointed out, you should use the existing mechanisms we have in the CMake files. Look for LLDB_DEFAULT_TEST_DSYMUTIL to see how we handle this (and the extension too!) for dsymutil.

[vsk]

Sorry for the delay, I was busy with other work. I think I've addressed the review feedback. PTAL.

[stella.stamenova]

LGTM, but I'd wait till @teemperor approves as well.

[JDevlieghere]

Thanks Vedant, this looks very useful!

[teemperor]

LGTM minus one small detail (see inline comment). Thanks Vedant!

lldb/packages/Python/lldbsuite/test/lldbtest.py

51

I think we don't need that import anymore.

lldb/packages/Python/lldbsuite/test/configuration.py

def get_absolute_path_to_root_test_dir():
     If an exclusive test subdirectory is specified, return its absolute path.
     Otherwise, return the absolute path of the root test directory.
     """
     test_subdir = get_absolute_path_to_exclusive_test_subdir()
     if test_subdir:
         return test_subdir
 
     return os.path.dirname(os.path.realpath(__file__))
+
+
+def get_llvm_build_dir():
+    """
+    Get the LLVM build directory path.
+    """
+    # The Xcode build defines an environment variable pointing to the LLVM
+    # build dir.
+    llvm_build_dir = os.environ.get('LLVM_BUILD_DIR')
+    if not llvm_build_dir:
+        # If a suitable environment variable isn't set, assume that the parent
+        # of the test build dir is the LLVM build dir.
+        llvm_build_dir = os.path.dirname(os.path.abspath(test_build_dir))
+    assert os.path.lexists(llvm_build_dir)
+    return llvm_build_dir
+
+
+def get_filecheck_path():
+    """
+    Get the path to the FileCheck testing tool.
+    """
+    filecheck_path = os.path.join(get_llvm_build_dir(), "bin", "FileCheck")
+    assert os.path.lexists(filecheck_path)
+    return filecheck_path

lldb/packages/Python/lldbsuite/test/expression_command/formatters/TestFormatters.py

def test(self):
         self.runCmd("command script import formatters.py")
         self.runCmd("command script import foosynth.py")
 
         if self.TraceOn():
             self.runCmd("frame variable foo1 --show-types")
             self.runCmd("frame variable foo1.b --show-types")
             self.runCmd("frame variable foo1.b.b_ref --show-types")
 
-        self.expect(
-            "expression --show-types -- *(new foo(47))",
-            substrs=[
-                '(int) a = 47',
-                '(bar) b = {',
-                '(int) i = 94',
-                '(baz) b = {',
-                '(int) k = 99'])
+        self.filecheck("expression --show-types -- *(new foo(47))", __file__,
+                '-check-prefix=EXPR-TYPES-NEW-FOO')
+        # EXPR-TYPES-NEW-FOO: (foo) ${{.*}} = {
+        # EXPR-TYPES-NEW-FOO-NEXT:   (int) a = 47
+        # EXPR-TYPES-NEW-FOO-NEXT:   (int *) a_ptr = 0x
+        # EXPR-TYPES-NEW-FOO-NEXT:   (bar) b = {
+        # EXPR-TYPES-NEW-FOO-NEXT:     (int) i = 94
+        # EXPR-TYPES-NEW-FOO-NEXT:     (int *) i_ptr = 0x
+        # EXPR-TYPES-NEW-FOO-NEXT:     (baz) b = {
+        # EXPR-TYPES-NEW-FOO-NEXT:       (int) h = 97
+        # EXPR-TYPES-NEW-FOO-NEXT:       (int) k = 99
+        # EXPR-TYPES-NEW-FOO-NEXT:     }
+        # EXPR-TYPES-NEW-FOO-NEXT:     (baz &) b_ref = 0x
+        # EXPR-TYPES-NEW-FOO-NEXT:   }
+        # EXPR-TYPES-NEW-FOO-NEXT: }
 
         self.runCmd("type summary add -F formatters.foo_SummaryProvider foo")
 
         self.expect("expression new int(12)",
                     substrs=['(int *) $', ' = 0x'])
 
         self.runCmd(
             "type summary add -s \"${var%pointer} -> ${*var%decimal}\" \"int *\"")
 
         self.expect("expression new int(12)",
                     substrs=['(int *) $', '= 0x', ' -> 12'])
 
         self.expect("expression foo1.a_ptr",
                     substrs=['(int *) $', '= 0x', ' -> 13'])
 
-        self.expect(
-            "expression foo1",
-            substrs=[
-                '(foo) $',
-                ' a = 12',
-                'a_ptr = ',
-                ' -> 13',
-                'i = 24',
-                'i_ptr = ',
-                ' -> 25'])
-
-        self.expect(
-            "expression --ptr-depth=1 -- new foo(47)",
-            substrs=[
-                '(foo *) $',
-                'a = 47',
-                'a_ptr = ',
-                ' -> 48',
-                'i = 94',
-                'i_ptr = ',
-                ' -> 95'])
-
-        self.expect(
-            "expression foo2",
-            substrs=[
-                '(foo) $',
-                'a = 121',
-                'a_ptr = ',
-                ' -> 122',
-                'i = 242',
-                'i_ptr = ',
-                ' -> 243'])
+        self.filecheck("expression foo1", __file__, '-check-prefix=EXPR-FOO1')
+        # EXPR-FOO1: (foo) $
+        # EXPR-FOO1-SAME: a = 12
+        # EXPR-FOO1-SAME: a_ptr = {{[0-9]+}} -> 13
+        # EXPR-FOO1-SAME: i = 24
+        # EXPR-FOO1-SAME: i_ptr = {{[0-9]+}} -> 25
+        # EXPR-FOO1-SAME: b_ref = {{[0-9]+}}
+        # EXPR-FOO1-SAME: h = 27
+        # EXPR-FOO1-SAME: k = 29
+
+        self.filecheck("expression --ptr-depth=1 -- new foo(47)", __file__,
+                '-check-prefix=EXPR-PTR-DEPTH1')
+        # EXPR-PTR-DEPTH1: (foo *) $
+        # EXPR-PTR-DEPTH1-SAME: a = 47
+        # EXPR-PTR-DEPTH1-SAME: a_ptr = {{[0-9]+}} -> 48
+        # EXPR-PTR-DEPTH1-SAME: i = 94
+        # EXPR-PTR-DEPTH1-SAME: i_ptr = {{[0-9]+}} -> 95
+
+        self.filecheck("expression foo2", __file__, '-check-prefix=EXPR-FOO2')
+        # EXPR-FOO2: (foo) $
+        # EXPR-FOO2-SAME: a = 121
+        # EXPR-FOO2-SAME: a_ptr = {{[0-9]+}} -> 122
+        # EXPR-FOO2-SAME: i = 242
+        # EXPR-FOO2-SAME: i_ptr = {{[0-9]+}} -> 243
+        # EXPR-FOO2-SAME: h = 245
+        # EXPR-FOO2-SAME: k = 247
 
         object_name = self.res.GetOutput()
         object_name = object_name[7:]
         object_name = object_name[0:object_name.find(' =')]
 
-        self.expect(
-            "frame variable foo2",
-            substrs=[
-                '(foo)',
-                'foo2',
-                'a = 121',
-                'a_ptr = ',
-                ' -> 122',
-                'i = 242',
-                'i_ptr = ',
-                ' -> 243'])
-
-        self.expect(
-            "expression $" +
-            object_name,
-            substrs=[
-                '(foo) $',
-                'a = 121',
-                'a_ptr = ',
-                ' -> 122',
-                'i = 242',
-                'i_ptr = ',
-                ' -> 243',
-                'h = 245',
-                'k = 247'])
+        self.filecheck("frame variable foo2", __file__, '-check-prefix=VAR-FOO2')
+        # VAR-FOO2: (foo) foo2
+        # VAR-FOO2-SAME: a = 121
+        # VAR-FOO2-SAME: a_ptr = {{[0-9]+}} -> 122
+        # VAR-FOO2-SAME: i = 242
+        # VAR-FOO2-SAME: i_ptr = {{[0-9]+}} -> 243
+        # VAR-FOO2-SAME: h = 245
+        # VAR-FOO2-SAME: k = 247
+
+        # The object is the same as foo2, so use the EXPR-FOO2 checks.
+        self.filecheck("expression $" + object_name, __file__,
+                '-check-prefix=EXPR-FOO2')
 
         self.runCmd("type summary delete foo")
         self.runCmd(
             "type synthetic add --python-class foosynth.FooSyntheticProvider foo")
 
         self.expect("expression --show-types -- $" + object_name,
                     substrs=['(foo) $', ' = {', '(int) *i_ptr = 243'])
 

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/typedef_array/main.cpp

 //===-- main.cpp --------------------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 typedef int Foo;
 
 int main() {
+    // CHECK: (Foo [3]) array = {
+    // CHECK-NEXT: (Foo) [0] = 1
+    // CHECK-NEXT: (Foo) [1] = 2
+    // CHECK-NEXT: (Foo) [2] = 3
+    // CHECK-NEXT: }
     Foo array[3] = {1,2,3};
-    return 0; //% self.expect("frame variable array --show-types --", substrs = ['(Foo [3]) array = {','(Foo) [0] = 1','(Foo) [1] = 2','(Foo) [2] = 3'])
+    return 0; //% self.filecheck("frame variable array --show-types --", 'main.cpp')
 }

lldb/packages/Python/lldbsuite/test/lldbtest.py

 import inspect
 import io
 import os.path
 import re
 import shutil
 import signal
 from subprocess import *
 import sys
+import commands
+import tempfile
 import time
 import traceback
 import types
 import distutils.spawn
 
 # Third-party modules
 import unittest2
 from six import add_metaclass

             if matched:
                 break
 
         self.assertTrue(matched if matching else not matched,
                         msg if msg else EXP_MSG(str, output, exe))
 
         return match_object
 
+    def filecheck(
+            self,
+            command,
+            check_file,
+            filecheck_options = '',
+            trace = False):
+        # Run the command.
+        self.runCmd(
+                command,
+                msg="FileCheck'ing result of `{0}`".format(command))
+
+        # Get the error text if there was an error, and the regular text if not.
+        output = self.res.GetOutput() if self.res.Succeeded() \
+                else self.res.GetError()
+
+        # Write the output to a temporary file.
+        input_file = tempfile.NamedTemporaryFile()
+        input_file.write(output)
+        input_file.flush()
+
+        # Assemble the absolute path to the check file. As a convenience for
+        # LLDB inline tests, assume that the check file is a relative path to
+        # a file within the inline test directory.
+        if hasattr(self, 'test_filename'):
+            check_file_abs = os.path.join(os.path.dirname(self.test_filename),
+                    check_file)
+        else:
+            check_file_abs = os.path.abspath(check_file)
+
+        # Run FileCheck.
+        filecheck_bin = configuration.get_filecheck_path()
+        filecheck_cmd = "{0} {1} -input-file {2} {3}".format(filecheck_bin,
+                check_file_abs, input_file.name, filecheck_options)
+        with recording(self, trace) as sbuf:
+            print("filecheck:", filecheck_cmd, file=sbuf)
+
+        (cmd_status, cmd_output) = commands.getstatusoutput(filecheck_cmd)
+
+        self.assertTrue(cmd_status == 0,
+                "FileCheck failed (code={0}):\n\n{1}\n\n{2}".format(cmd_status,
+                    cmd_output, output))
+
     def expect(
             self,
             str,
             msg=None,
             patterns=None,
             startstr=None,
             endstr=None,
             substrs=None,