[lldb] Extend and document TestIRInterpreter.py

[teemperor]

Summary

There are a bunch of arithmetic and comparison instructions supported by the IRInterpreter and we
currently don't test these at all in the test suit. This patch extends the IRInterpreter test to try out these operations
with different types and values. Also documents the test case a bit while I'm at it.

[shafik]

lldb/packages/Python/lldbsuite/test/commands/expression/ir-interpreter/TestIRInterpreter.py

55

I may be missing something here but this looks like it should be 32 instead of 7 or rather sizeof(int)*8

When we say doesn't work do we mean undefined behavior?

[teemperor]

lldb/packages/Python/lldbsuite/test/commands/expression/ir-interpreter/TestIRInterpreter.py

55

Good catch, originally that was supposed to be > 7 so that we don't overflow any data type (assuming we ever extend the test to char). But I can change it to >= 32 until we actually use any 8-bit type.

And 'doesn't work' means that it will literally cause the test to fail and this test stops working. The interpreter will do something else than the JIT in these cases which is a bug. We probably should detect UB when interpreting these expressions and throw an error, but that's a whole new story. This is more about adding testing to the existing code.

[shafik]

lldb/packages/Python/lldbsuite/test/commands/expression/ir-interpreter/TestIRInterpreter.py

55

if I am not missing anything here in C++ and C the operands of expressions undergo the usual arithmetic conversions and for integral types they undergo integer promotions. This means that the smallest type should be int or unsigned int.

More details.

[teemperor]

• Relaxed can_handle_operands that it allows shifting unsigned large values (produced by initialising them with negative signed values).
• Now checking for 32bit instead of 7 (thanks Shafik!)

[shafik]

lldb/packages/Python/lldbsuite/test/commands/expression/ir-interpreter/TestIRInterpreter.py

49

Maybe be worth noting here that for the unsigned case we are relying in the fact that converting an signed value to an unsigned value works as we would expect with twos complement numbers.

119

You don't see to use interp_result here

135

In the unsigned case should we be converting the python value which is signed manually to an unsigned value via a mask like it is discussed here

[teemperor]

• Addressed feedback.

lldb/packages/Python/lldbsuite/test/commands/expression/ir-interpreter/TestIRInterpreter.py

55

Done, thanks!

119

Yeah, seems that declaring a $variable in LLDB just returns some nonsensical value and a meaningless error... We will know if we have an error later, but at the moment there doesn't seem to be any early error checking we can do here.

[jankratochvil]

On Fedora 31 x86_64 (that is with python3-3.7.4-5.fc31.x86_64) I get:

FAIL: LLDB (/quad/home/jkratoch/redhat/llvm-monorepo-clangassertpython3/bin/clang-10-x86_64) :: test_ir_interpreter_int_ops (TestIRInterpreter.IRInterpreterTestCase)
PASS: LLDB (/quad/home/jkratoch/redhat/llvm-monorepo-clangassertpython3/bin/clang-10-x86_64) :: test_type_conversions (TestIRInterpreter.IRInterpreterTestCase)
======================================================================
ERROR: test_ir_interpreter_int_ops (TestIRInterpreter.IRInterpreterTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jkratoch/redhat/llvm-monorepo/lldb/packages/Python/lldbsuite/test/decorators.py", line 112, in wrapper
    func(*args, **kwargs)
  File "/home/jkratoch/redhat/llvm-monorepo/lldb/packages/Python/lldbsuite/test/decorators.py", line 112, in wrapper
    func(*args, **kwargs)
  File "/home/jkratoch/redhat/llvm-monorepo/lldb/packages/Python/lldbsuite/test/decorators.py", line 112, in wrapper
    func(*args, **kwargs)
  File "/home/jkratoch/redhat/llvm-monorepo/lldb/packages/Python/lldbsuite/test/commands/expression/ir-interpreter/TestIRInterpreter.py", line 130, in test_ir_interpreter_int_ops
    if not op.can_handle_operands(var1, var2):
  File "/home/jkratoch/redhat/llvm-monorepo/lldb/packages/Python/lldbsuite/test/commands/expression/ir-interpreter/TestIRInterpreter.py", line 71, in can_handle_operands
    if rhs.value <= 0 or rhs.value >= 32:
TypeError: '<=' not supported between instances of 'c_uint' and 'int'
Config=x86_64-/quad/home/jkratoch/redhat/llvm-monorepo-clangassertpython3/bin/clang-10
----------------------------------------------------------------------

lldb/packages/Python/lldbsuite/test/commands/expression/ir-interpreter/TestIRInterpreter.py

69

Why not < 0?

71

Why not < 0?

[tatyana-krasnukha]

Another problem of the test_ir_interpreter is that it fails even when JIT fails despite it should test the interpreter.

[teemperor]

In D67227#1730144, @tatyana-krasnukha wrote:

Another problem of the test_ir_interpreter is that it fails even when JIT fails despite it should test the interpreter.

Well, it also tests that the IRInterpreter is equivalent to what we would JIT (i.e., our interpreting-vs-JITing optimisation is actually working as intended).

[tatyana-krasnukha]

I mean, when a process is not able to allocate memory, it cannot JIT, but still can interpret. Should the test report failure in this case?

lldb/packages/Python/lldbsuite/test/commands/expression/ir-interpreter/TestIRInterpreter.py

 """
 Test the IR interpreter
 """
 
 from __future__ import print_function
 
 import unittest2
+import ctypes
 
 import lldb
 from lldbsuite.test.decorators import *
 from lldbsuite.test.lldbtest import *
 from lldbsuite.test import lldbutil
 
 
 class IRInterpreterTestCase(TestBase):
def build_and_run(self):
 
         self.runCmd("file " + self.getBuildArtifact("a.out"), CURRENT_EXECUTABLE_SET)
 
         lldbutil.run_break_set_by_file_and_line(
             self, "main.c", self.line, num_expected_locations=1, loc_exact=False)
 
         self.runCmd("run", RUN_SUCCEEDED)
 
+    class Variable:
+        def __init__(self, uid, type, value):
+            self.name = "$i_" + str(uid) + "_" + type.replace(" ", "_")
+            self.type = type
+            self.value = value
+            # The value may be different if it's negative and we have an
+            # unsigned type. Let's ask Python for the actual converted value on
+            # the current platform.
+            if type == "unsigned int":
+                self.value = ctypes.c_uint(self.value)
+            elif "unsigned" in type:
+                assert False, "Unsupported unsigned type: " + type
+            self.decl_expr = type + " " + self.name + " = " + str(self.value)
+
+    class Operator:
+        def __init__(self, name):
+           self.name = name
+
+        def can_handle_operands(self, lhs, rhs):
+            """True iff this operator can handle the variables as operands."""
+            if self.name in ['<<', '>>']:
+                # Shifting negative values, shifting by negative values and
+                # shifting by too large values will make this test fail (as
+                # the interpreter will evaluate this differently than the JIT).
+                # FIXME: This is probably a bug in the IRInterpreter.
+                if lhs.value <= 0:
+                    return False
+                if rhs.value <= 0 or rhs.value >= 32:
+                    return False
+            return True
+
     @add_test_categories(['pyapi'])
     # getpid() is POSIX, among other problems, see bug
     @expectedFailureAll(
         oslist=['windows'],
         bugnumber="http://llvm.org/pr21765")
     @expectedFailureNetBSD
     @expectedFailureAll(
         oslist=['linux'],
         archs=['arm'],
         bugnumber="llvm.org/pr27868")
-    def test_ir_interpreter(self):
+    def test_ir_interpreter_int_ops(self):
         self.build_and_run()
 
+        # Normal expression options we use for JITed expressions.
         options = lldb.SBExpressionOptions()
         options.SetLanguage(lldb.eLanguageTypeC_plus_plus)
 
-        set_up_expressions = ["int $i = 9", "int $j = 3", "int $k = 5"]
-
-        expressions = ["$i + $j",
-                       "$i - $j",
-                       "$i * $j",
-                       "$i / $j",
-                       "$i % $k",
-                       "$i << $j",
-                       "$i & $j",
-                       "$i | $j",
-                       "$i ^ $j"]
-
-        for expression in set_up_expressions:
-            self.frame().EvaluateExpression(expression, options)
-
-        for expression in expressions:
+        # Expression options that prevent that we use the JIT.
+        nojit_options = lldb.SBExpressionOptions()
+        nojit_options.SetLanguage(lldb.eLanguageTypeC_plus_plus)
+        nojit_options.SetAllowJIT(False)
+
+        # List of operators the interpreter supports and we want to test.
+        operators = ['+', '-', '*', '%', '/',
+                     '<<', '>>', '&', '|', '^',
+                     '>', '<', '>=', '<=', '!=', '==']
+        operator_list = []
+        for opname in operators:
+            operator_list.append(self.Operator(opname))
+
+        # Variable types that should be tested with the operators.
+        types_to_test = ['int', 'unsigned int']
+
+        # Values these variables can have.
+        values_to_test = [255, 256, -1, 0, 1, 255, 256]
+
+        # Id for every variable to give them unique names.
+        uid = 0
+        # Define a variable for every type and for every value in LLDB.
+        variable_list = []
+        for t in types_to_test:
+            for value in values_to_test:
+                v = self.Variable(uid, t, value)
+                variable_list.append(v)
+                # FIXME: We should check for errors here, but declaring
+                # $variables just returns some empty error it seems...
+                self.frame().EvaluateExpression(v.decl_expr, nojit_options)
+                uid += 1
+
+        # Create a list of expressions that use every operator.
+        exprs_to_run = []
+        for op in operator_list:
+            # Try all combinations of variables with the operator.
+            for var1 in variable_list:
+                for var2 in variable_list:
+                    if not op.can_handle_operands(var1, var2):
+                        continue
+                    # Create an expression using the operator.
+                    expr = var1.name + " " + op.name + " " + var2.name
+                    # Resolve the variable values and add that as a comment
+                    # to the expression. This will be shown when the test fails.
+                    expr += " // " + str(var1.value) + " " + op.name + " " + str(var2.value)
+                    exprs_to_run.append(expr)
+
+        # Make sure we actually did generate a list of expressions.
+        assert len(exprs_to_run) != 0
+        for expression in exprs_to_run:
             interp_expression = expression
+            # We call 'getpid()' as a trick to force the expression to JITed.
+            # FIXME: Find a non-hacky way to tell LLDB to enforce this.
             jit_expression = "(int)getpid(); " + expression
 
+            # Run the expression in the interpreter and in the JIT. Both
+            # should produce the same result.
             interp_result = self.frame().EvaluateExpression(
-                interp_expression, options).GetValueAsSigned()
+                interp_expression, nojit_options).GetValueAsUnsigned()
             jit_result = self.frame().EvaluateExpression(
-                jit_expression, options).GetValueAsSigned()
+                jit_expression, options).GetValueAsUnsigned()
 
             self.assertEqual(
                 interp_result,
                 jit_result,
                 "While evaluating " +
                 expression)
 
     def test_type_conversions(self):
         target = self.dbg.GetDummyTarget()
         short_val = target.EvaluateExpression("(short)-1")
         self.assertEqual(short_val.GetValueAsSigned(), -1)
         long_val = target.EvaluateExpression("(long) "+ short_val.GetName())
         self.assertEqual(long_val.GetValueAsSigned(), -1)