Add "lldb-test breakpoint" command and convert the case-sensitivity test to use it

[labath]

Summary

The command takes two input arguments: a module to use as a debug target
and a file containing a list of commands. The command will execute each
of the breakpoint commands in the file and dump the breakpoint state
after each one.

The commands are expected to be breakpoint set/remove/etc. commands, but
I explicitly allow any lldb command here, so you can do things like
change setting which impact breakpoint resolution, etc. There is also a
"-persistent" flag, which causes lldb-test to *not* automatically clear
the breakpoint list after each command. Right now I don't use it, but
the idea behind it was that it could be used to test more complex
combinations of breakpoint commands (set+modify, set+disable, etc.).

Right now the command prints out only the basic breakpoint state, but
more information can be easily added there. To enable easy matching of
the "at least one breakpoint location found" state, the command
explicitly prints out the string "At least one breakpoint location.".

To enable testing of breakpoints set with an absolute paths, I add the
ability to perform rudimentary substitutions on the commands: right now
the string %p is replaced by the directory which contains the command
file (so, under normal circumstances, this will perform the same
substitution as lit would do for %p).

I use this command to rewrite the TestBreakpointCaseSensitivity test --
the test was checking about a dozen breakpoint commands, but it was
launching a new process for each one, so it took about 90 seconds to
run. The new test takes about 0.3 seconds for me, which is approximately
a 300x speedup.

[zturner]

Curious if we need lldb-test for this. Could we get by with just using lldb in batch mode? Obviously I'm not opposed to adding whatever we need to lldb-test, just want to make sure it's something that can't already be done with just lldb.

lit/Breakpoint/case-insensitive.test

1 ↗

(On Diff #135666)

OSX has a case insensitive file system too. Should we run this test on OSX as well?

[jingham]

This test has the advantage over running lldb in batch mode that it isolates the tests you write using it from the output format of the break command.

The lldb test suite used to check the output of break all over the place, and that meant when I went to change that output to add some info and many tests broke for no good reason. That's why I changed all the tests to use the lldbutil.run_break_set_* commands so I wouldn't have to deal with this again. Not having to do that for these tests some time in the future to me justifies the effort of supporting this small amount of code.

[labath]

In D43686#1017577, @zturner wrote:

Curious if we need lldb-test for this. Could we get by with just using lldb in batch mode? Obviously I'm not opposed to adding whatever we need to lldb-test, just want to make sure it's something that can't already be done with just lldb.

There are a couple of things which make this tricky. Neither of them in insurmountable I think, but together they add up:

• without the %p substitution it would be hard to perform an absolute-path match
• the "auto-clear" and "auto-dump" behavior is quite handy to avoid repetition in the tests
• the output format can be tweaked to enable easy FileCheck-ing ("At least one breakpoint location"), and can be independent of the actual format we want to present to the users.

lit/Breakpoint/case-insensitive.test

1 ↗

(On Diff #135666)

Yes, but they don't do case-insensitive matches by default. This matches the logic of the original test, which matches the logic of the code inside lldb. If the lldb implementation changes, then the test will need to be updated.

[davide]

I was going to suggest the same thing Zach suggested, but I think this fine as is.
LGTM. The fact the test is more concise is definitely a win, but I don't think this is the main reason for doing the conversion.
The 300x speedup makes this change a no-brainer.

[davide]

(and thanks for saving 1 minutes and 30 seconds of my life multiplied by the many times I run the test suite per day).

[vsk]

I like that we can customize the command substitutions and the way we dump internal state in this test mode. This offers some added flexibility over the batch mode, IMO. LGTM, thanks!

lldb/trunk/lit/Breakpoint/Inputs/case-sensitive.c

+int main() {
+  int x = 47; // REGEX-THIS
+  return x;
+}

lldb/trunk/lit/Breakpoint/case-insensitive.test

+# REQUIRES: windows
+#
+# RUN: %cc %p/Inputs/case-sensitive.c -g -o %t
+# RUN: lldb-test breakpoints %t %s | FileCheck %s
+
+breakpoint set -f case-sensitive.c -l 3
+# CHECK-LABEL: breakpoint set -f case-sensitive.c -l 3
+# CHECK: At least one location.
+
+breakpoint set -f %p/Inputs/case-sensitive.c -l 3
+# CHECK-LABEL: breakpoint set -f {{.*}}/Inputs/case-sensitive.c -l 3
+# CHECK: At least one location.
+
+breakpoint set -f %p/INPUTS/case-sensitive.c -l 3
+# CHECK-LABEL: breakpoint set -f {{.*}}/INPUTS/case-sensitive.c -l 3
+# CHECK: At least one location.
+
+breakpoint set -f Case-Sensitive.c -l 3
+# CHECK-LABEL: breakpoint set -f Case-Sensitive.c -l 3
+# CHECK: At least one location.
+
+breakpoint set -f %p/INPUTS/Case-Sensitive.c -l 3
+# CHECK-LABEL: breakpoint set -f {{.*}}/INPUTS/Case-Sensitive.c -l 3
+# CHECK: At least one location.
+
+breakpoint set -f case-sensitive.c -p REGEX-THIS
+# CHECK-LABEL: breakpoint set -f case-sensitive.c -p REGEX-THIS
+# CHECK: At least one location.
+
+breakpoint set -f %p/Inputs/case-sensitive.c -p REGEX-THIS
+# CHECK-LABEL: breakpoint set -f {{.*}}/Inputs/case-sensitive.c -p REGEX-THIS
+# CHECK: At least one location.
+
+breakpoint set -f %p/INPUTS/case-sensitive.c -p REGEX-THIS
+# CHECK-LABEL: breakpoint set -f {{.*}}/INPUTS/case-sensitive.c -p REGEX-THIS
+# CHECK: At least one location.
+
+breakpoint set -f Case-Sensitive.c -p REGEX-THIS
+# CHECK-LABEL: breakpoint set -f Case-Sensitive.c -p REGEX-THIS
+# CHECK: At least one location.
+
+breakpoint set -f %p/INPUTS/Case-Sensitive.c -p REGEX-THIS
+# CHECK-LABEL: breakpoint set -f {{.*}}/INPUTS/Case-Sensitive.c -p REGEX-THIS
+# CHECK: At least one location.

lldb/trunk/lit/Breakpoint/case-sensitive.test

+# REQUIRES: nowindows
+#
+# RUN: %cc %p/Inputs/case-sensitive.c -g -o %t
+# RUN: lldb-test breakpoints %t %s | FileCheck %s
+
+breakpoint set -f case-sensitive.c -l 3
+# CHECK-LABEL: breakpoint set -f case-sensitive.c -l 3
+# CHECK: At least one location.
+
+breakpoint set -f %p/Inputs/case-sensitive.c -l 3
+# CHECK-LABEL: breakpoint set -f {{.*}}/Inputs/case-sensitive.c -l 3
+# CHECK: At least one location.
+
+breakpoint set -f %p/INPUTS/case-sensitive.c -l 3
+# CHECK-LABEL: breakpoint set -f {{.*}}/INPUTS/case-sensitive.c -l 3
+# CHECK: 0 locations.
+
+breakpoint set -f Case-Sensitive.c -l 3
+# CHECK-LABEL: breakpoint set -f Case-Sensitive.c -l 3
+# CHECK: 0 locations.
+
+breakpoint set -f %p/INPUTS/Case-Sensitive.c -l 3
+# CHECK-LABEL: breakpoint set -f {{.*}}/INPUTS/Case-Sensitive.c -l 3
+# CHECK: 0 locations.
+
+breakpoint set -f case-sensitive.c -p REGEX-THIS
+# CHECK-LABEL: breakpoint set -f case-sensitive.c -p REGEX-THIS
+# CHECK: At least one location.
+
+breakpoint set -f %p/Inputs/case-sensitive.c -p REGEX-THIS
+# CHECK-LABEL: breakpoint set -f {{.*}}/Inputs/case-sensitive.c -p REGEX-THIS
+# CHECK: At least one location.
+
+breakpoint set -f %p/INPUTS/case-sensitive.c -p REGEX-THIS
+# CHECK-LABEL: breakpoint set -f {{.*}}/INPUTS/case-sensitive.c -p REGEX-THIS
+# CHECK: 0 locations.
+
+breakpoint set -f Case-Sensitive.c -p REGEX-THIS
+# CHECK-LABEL: breakpoint set -f Case-Sensitive.c -p REGEX-THIS
+# CHECK: 0 locations.
+
+breakpoint set -f %p/INPUTS/Case-Sensitive.c -p REGEX-THIS
+# CHECK-LABEL: breakpoint set -f {{.*}}/INPUTS/Case-Sensitive.c -p REGEX-THIS
+# CHECK: 0 locations.
+
+

lldb/trunk/lit/Breakpoint/lit.local.cfg

+config.suffixes = ['.test']

lldb/trunk/lit/lit.cfg

 # Running on ELF based *nix
 if platform.system() in ['FreeBSD', 'Linux']:
     config.available_features.add('system-linker-elf')
     if platform.system() in ['FreeBSD']:
         config.available_features.add('freebsd')
     else:
         config.available_features.add('linux')
 
-if platform.system() in ['Windows']:
-    config.available_features.add('windows')
+config.available_features.add(
+    binary_feature(platform.system() in ['Windows'], 'windows', 'no'))
 
 if re.match(r'^arm(hf.*-linux)|(.*-linux-gnuabihf)', config.target_triple):
     config.available_features.add("armhf-linux")
 
 if re.match(r'icc', config.cc):
     config.available_features.add("compiler-icc")
 elif re.match(r'clang', config.cc):
     config.available_features.add("compiler-clang")

lldb/trunk/packages/Python/lldbsuite/test/functionalities/breakpoint/breakpoint_case_sensitivity/Makefile

-LEVEL = ../../../make
-
-C_SOURCES := main.c
-CFLAGS_EXTRAS += -std=c99
-
-include $(LEVEL)/Makefile.rules

lldb/trunk/packages/Python/lldbsuite/test/functionalities/breakpoint/breakpoint_case_sensitivity/TestBreakpointCaseSensitivity.py

-"""
-Test case sensitivity of paths on Windows / POSIX
-llvm.org/pr22667
-"""
-
-import os
-import lldb
-from lldbsuite.test.lldbtest import *
-from lldbsuite.test.decorators import *
-from lldbsuite.test import lldbplatform, lldbplatformutil
-
-
-class BreakpointCaseSensitivityTestCase(TestBase):
-    mydir = TestBase.compute_mydir(__file__)
-    BREAKPOINT_TEXT = 'Set a breakpoint here'
-
-    def setUp(self):
-        # Call super's setUp().
-        TestBase.setUp(self)
-        self.line = line_number('main.c', self.BREAKPOINT_TEXT)
-
-    @skipIf(hostoslist=no_match(['windows']))  # Skip for non-windows platforms
-    def test_breakpoint_matches_file_with_different_case(self):
-        """Set breakpoint on file, should match files with different case on Windows"""
-        self.build()
-        self.case_sensitivity_breakpoint(True)
-
-    @skipIf(hostoslist=['windows'])  # Skip for windows platforms
-    def test_breakpoint_doesnt_match_file_with_different_case(self):
-        """Set breakpoint on file, shouldn't match files with different case on POSIX systems"""
-        self.build()
-        self.case_sensitivity_breakpoint(False)
-
-    def case_sensitivity_breakpoint(self, case_insensitive):
-        """Set breakpoint on file, should match files with different case if case_insensitive is True"""
-
-        # use different case to check CreateTarget
-        exe = 'a.out'
-        if case_insensitive:
-            exe = exe.upper()
-
-        exe = self.getBuildArtifact(exe)
-
-        # Create a target by the debugger.
-        self.target = self.dbg.CreateTarget(exe)
-        self.assertTrue(self.target, VALID_TARGET)
-        srcdir = self.getSourceDir()
-
-        # try both BreakpointCreateByLocation and BreakpointCreateBySourceRegex
-        for regex in [False, True]:
-            # should always hit
-            self.check_breakpoint('main.c', regex, True)
-            # should always hit
-            self.check_breakpoint(os.path.join(srcdir, 'main.c'), regex, True)
-            # different case for directory
-            self.check_breakpoint(os.path.join(srcdir.upper(), 'main.c'),
-                                  regex,
-                                  case_insensitive)
-            # different case for file
-            self.check_breakpoint('Main.c',
-                                  regex,
-                                  case_insensitive)
-            # different case for both
-            self.check_breakpoint(os.path.join(srcdir.upper(), 'Main.c'),
-                                  regex,
-                                  case_insensitive)
-
-    def check_breakpoint(self, file, source_regex, should_hit):
-        """
-        Check breakpoint hit at given file set by given method
-
-        file:
-            File where insert the breakpoint
-
-        source_regex:
-            True for testing using BreakpointCreateBySourceRegex,
-            False for  BreakpointCreateByLocation
-
-        should_hit:
-            True if the breakpoint should hit, False otherwise
-        """
-
-        desc = ' file %s set by %s' % (
-            file, 'regex' if source_regex else 'location')
-        if source_regex:
-            breakpoint = self.target.BreakpointCreateBySourceRegex(
-                self.BREAKPOINT_TEXT, lldb.SBFileSpec(file))
-        else:
-            breakpoint = self.target.BreakpointCreateByLocation(
-                file, self.line)
-
-        self.assertEqual(breakpoint and breakpoint.GetNumLocations() == 1,
-                         should_hit,
-                         VALID_BREAKPOINT + desc)
-
-        # Get the breakpoint location from breakpoint after we verified that,
-        # indeed, it has one location.
-        location = breakpoint.GetLocationAtIndex(0)
-
-        self.assertEqual(location.IsValid(),
-                         should_hit,
-                         VALID_BREAKPOINT_LOCATION + desc)
-
-        process = self.target.LaunchSimple(
-            None, None, self.get_process_working_directory())
-        self.assertTrue(process, PROCESS_IS_VALID + desc)
-
-        if should_hit:
-            # Did we hit our breakpoint?
-            from lldbsuite.test.lldbutil import get_threads_stopped_at_breakpoint
-            threads = get_threads_stopped_at_breakpoint(process, breakpoint)
-            self.assertEqual(
-                len(threads),
-                1,
-                "There should be a thread stopped at breakpoint" +
-                desc)
-            # The hit count for the breakpoint should be 1.
-            self.assertEqual(breakpoint.GetHitCount(), 1)
-
-        else:
-            # check the breakpoint was not hit
-            self.assertEqual(lldb.eStateExited, process.GetState())
-            self.assertEqual(breakpoint.GetHitCount(), 0)
-
-        # let process finish
-        process.Continue()
-
-        # cleanup
-        self.target.BreakpointDelete(breakpoint.GetID())

lldb/trunk/packages/Python/lldbsuite/test/functionalities/breakpoint/breakpoint_case_sensitivity/main.c

-#include <stdio.h>
-
-int
-main()
-{
-    printf("Set a breakpoint here.\n");
-    return 0;
-}

lldb/trunk/tools/lldb-test/lldb-test.cpp

 //===- lldb-test.cpp ------------------------------------------ *- C++ --*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "FormatUtil.h"
 #include "SystemInitializerTest.h"
 
 #include "Plugins/SymbolFile/DWARF/SymbolFileDWARF.h"
+#include "lldb/Breakpoint/BreakpointLocation.h"
 #include "lldb/Core/Debugger.h"
 #include "lldb/Core/Module.h"
 #include "lldb/Core/Section.h"
 #include "lldb/Initialization/SystemLifetimeManager.h"
+#include "lldb/Interpreter/CommandInterpreter.h"
+#include "lldb/Interpreter/CommandReturnObject.h"
 #include "lldb/Symbol/ClangASTContext.h"
 #include "lldb/Symbol/ClangASTImporter.h"
 #include "lldb/Utility/DataExtractor.h"
 #include "lldb/Utility/StreamString.h"
 
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ManagedStatic.h"
+#include "llvm/Support/Path.h"
 #include "llvm/Support/PrettyStackTrace.h"
 #include "llvm/Support/Signals.h"
 #include <thread>
 
 using namespace lldb;
 using namespace lldb_private;
 using namespace llvm;
 
 namespace opts {
+static cl::SubCommand BreakpointSubcommand("breakpoints",
+                                           "Test breakpoint resolution");
 cl::SubCommand ModuleSubcommand("module-sections",
                                 "Display LLDB Module Information");
 cl::SubCommand SymbolsSubcommand("symbols", "Dump symbols for an object file");
 
+namespace breakpoint {
+static cl::opt<std::string> Target(cl::Positional, cl::desc("<target>"),
+                                   cl::Required, cl::sub(BreakpointSubcommand));
+static cl::opt<std::string> CommandFile(cl::Positional,
+                                        cl::desc("<command-file>"),
+                                        cl::init("-"),
+                                        cl::sub(BreakpointSubcommand));
+static cl::opt<bool> Persistent(
+    "persistent",
+    cl::desc("Don't automatically remove all breakpoints before each command"),
+    cl::sub(BreakpointSubcommand));
+
+static llvm::StringRef plural(uintmax_t value) { return value == 1 ? "" : "s"; }
+static void dumpState(const BreakpointList &List, LinePrinter &P);
+static std::string substitute(StringRef Cmd);
+static void evaluateBreakpoints(Debugger &Dbg);
+} // namespace breakpoint
+
 namespace module {
 cl::opt<bool> SectionContents("contents",
                               cl::desc("Dump each section's contents"),
                               cl::sub(ModuleSubcommand));
 cl::list<std::string> InputFilenames(cl::Positional, cl::desc("<input files>"),
                                      cl::OneOrMore, cl::sub(ModuleSubcommand));
 } // namespace module
 
 namespace symbols {
 cl::list<std::string> InputFilenames(cl::Positional, cl::desc("<input files>"),
                                      cl::OneOrMore, cl::sub(SymbolsSubcommand));
 }
 } // namespace opts
 
 static llvm::ManagedStatic<SystemLifetimeManager> DebuggerLifetime;
 
+void opts::breakpoint::dumpState(const BreakpointList &List, LinePrinter &P) {
+  P.formatLine("{0} breakpoint{1}", List.GetSize(), plural(List.GetSize()));
+  if (List.GetSize() > 0)
+    P.formatLine("At least one breakpoint.");
+  for (size_t i = 0, e = List.GetSize(); i < e; ++i) {
+    BreakpointSP BP = List.GetBreakpointAtIndex(i);
+    P.formatLine("Breakpoint ID {0}:", BP->GetID());
+    AutoIndent Indent(P, 2);
+    P.formatLine("{0} location{1}.", BP->GetNumLocations(),
+                 plural(BP->GetNumLocations()));
+    if (BP->GetNumLocations() > 0)
+      P.formatLine("At least one location.");
+    P.formatLine("{0} resolved location{1}.", BP->GetNumResolvedLocations(),
+                 plural(BP->GetNumResolvedLocations()));
+    if (BP->GetNumResolvedLocations() > 0)
+      P.formatLine("At least one resolved location.");
+    for (size_t l = 0, le = BP->GetNumLocations(); l < le; ++l) {
+      BreakpointLocationSP Loc = BP->GetLocationAtIndex(l);
+      P.formatLine("Location ID {0}:", Loc->GetID());
+      AutoIndent Indent(P, 2);
+      P.formatLine("Enabled: {0}", Loc->IsEnabled());
+      P.formatLine("Resolved: {0}", Loc->IsResolved());
+      P.formatLine("Address: {0}+{1:x}",
+                   Loc->GetAddress().GetSection()->GetName(),
+                   Loc->GetAddress().GetOffset());
+    }
+  }
+  P.NewLine();
+}
+
+std::string opts::breakpoint::substitute(StringRef Cmd) {
+  std::string Result;
+  raw_string_ostream OS(Result);
+  while (!Cmd.empty()) {
+    switch (Cmd[0]) {
+    case '%':
+      if (Cmd.consume_front("%p") && (Cmd.empty() || !isalnum(Cmd[0]))) {
+        OS << sys::path::parent_path(CommandFile);
+        break;
+      }
+      // fall through
+    default:
+      size_t pos = Cmd.find('%');
+      OS << Cmd.substr(0, pos);
+      Cmd = Cmd.substr(pos);
+      break;
+    }
+  }
+  return std::move(OS.str());
+}
+
+void opts::breakpoint::evaluateBreakpoints(Debugger &Dbg) {
+  TargetSP Target;
+  Status ST =
+      Dbg.GetTargetList().CreateTarget(Dbg, breakpoint::Target, /*triple*/ "",
+                                       /*get_dependent_modules*/ false,
+                                       /*platform_options*/ nullptr, Target);
+  if (ST.Fail()) {
+    errs() << formatv("Failed to create target '{0}: {1}\n", breakpoint::Target,
+                      ST);
+    exit(1);
+  }
+
+  auto MB = MemoryBuffer::getFileOrSTDIN(CommandFile);
+  if (!MB) {
+    errs() << formatv("Could not open file '{0}: {1}\n", CommandFile,
+                      MB.getError().message());
+    exit(1);
+  }
+
+  LinePrinter P(4, outs());
+  StringRef Rest = (*MB)->getBuffer();
+  while (!Rest.empty()) {
+    StringRef Line;
+    std::tie(Line, Rest) = Rest.split('\n');
+    Line = Line.ltrim();
+    if (Line.empty() || Line[0] == '#')
+      continue;
+
+    if (!Persistent)
+      Target->RemoveAllBreakpoints(/*internal_also*/ true);
+
+    std::string Command = substitute(Line);
+    P.formatLine("Command: {0}", Command);
+    CommandReturnObject Result;
+    if (!Dbg.GetCommandInterpreter().HandleCommand(
+            Command.c_str(), /*add_to_history*/ eLazyBoolNo, Result)) {
+      P.formatLine("Failed: {0}", Result.GetErrorData());
+      continue;
+    }
+
+    dumpState(Target->GetBreakpointList(/*internal*/ false), P);
+  }
+}
+
 static void dumpSymbols(Debugger &Dbg) {
   for (const auto &File : opts::symbols::InputFilenames) {
     ModuleSpec Spec{FileSpec(File, false)};
     Spec.GetSymbolFileSpec().SetFile(File, false);
 
     auto ModulePtr = std::make_shared<lldb_private::Module>(Spec);
 
     StreamString Stream;
int main(int argc, const char *argv[]) {
 
   cl::ParseCommandLineOptions(argc, argv, "LLDB Testing Utility\n");
 
   DebuggerLifetime->Initialize(llvm::make_unique<SystemInitializerTest>(),
                                nullptr);
 
   auto Dbg = lldb_private::Debugger::CreateInstance();
 
+  if (opts::BreakpointSubcommand)
+    opts::breakpoint::evaluateBreakpoints(*Dbg);
   if (opts::ModuleSubcommand)
     dumpModules(*Dbg);
   else if (opts::SymbolsSubcommand)
     dumpSymbols(*Dbg);
 
   DebuggerLifetime->Terminate();
   return 0;
 }