Build each testcase variant in its own subdirectory and remove the srcdir lock file

[aprantl]

Summary

This patch creates a <test>.dwarf, <test>.dwo, etc., build directory for each testcase variant.
Most importantly, this eliminates the need for the per-test lock file in the source directory.

Tests that are marked as NO_DEBUG_INFO_TESTCASE and build with buildDefault() are built in a <test>.default build directory.

The implementation changes self.mydir to a tuple (test_subdir, test_name). I chose not to rename the variable as to not break anyone's out-of-tree testcases any more than necessary.

It would be awesome if someone could test this on Windows / Linux / Android again.

[xiaobai]

packages/Python/lldbsuite/test/functionalities/breakpoint/comp_dir_symlink/TestCompDirSymLink.py

39–41

Instead of making a temporary variable src_path that gets used once in each instance, why not just insert self.getBuildArtifact(_SRC_FILE) directly into lldbutil.run_break_set_by_file_and_line()?

packages/Python/lldbsuite/test/functionalities/watchpoint/watchpoint_on_vectors/TestValueOfVectorVariable.py

25–29

You should be able to insert the call to getBuildArtifact() into the construction of d, since exe is used nowhere else.

40

Is this d used?

packages/Python/lldbsuite/test/lldbinline.py

95–97

Good opportunity to move away from the old style of formatting strings to a newer style? It doesn't make a huge difference, I just think it'd be nice to do. :)
return "-N dwarf {}".format(testdir)
return "-N dwarf {}".format(testdir)

This is supported in Python 2.7, but I'm not sure if we can assume that version or greater to be present.

packages/Python/lldbsuite/test/lldbtest.py

533–539

Might I suggest this?

(rel_test_path, extension) = os.path.splitext(rel_prefix)
if extension not in [".py", ".pyc"]:
    raise Exception("test_file is not a python file")
return rel_test_path

Feels cleaner to me, at least. :)

706

stem is unused, you could make this a _.

711

subdir and stem are kind of confusing to me. It looks like you used test_subdir below, so it might be good to use that here too, I think. Also, I think the name test_name is less confusing than test_stem. What do you think?

712

I recommend if variant is None instead of if not variant

packages/Python/lldbsuite/test/plugins/builder_base.py

63

I find test_stem kind of confusing. How do you feel about test_name?

[xiaobai]

Oh, FWIW I tested this on Linux (specifically CentOS) and the test result summary showed no difference in Success/Failure rate.

[labath]

I am not sure this actually creates enough separation. Plenty of tests have more then one test method per file:

TestFoo.py:
class FooTestCase(TestBase):
  NO_DEBUG_INFO_TESTCASE = True
  def test_bar(self):
    ...
  def test_baz(self):
    ...

If I understand correctly, you patch will run both test_bar and test_baz in the same build folder (TestFoo.default). Shouldn't we separate those as well ? (e.g. TestFoo.test_bar, TestFoo.test_baz) ?

The good thing about this is that you don't need to worry about debug info variants in this code at all. You just take the test name, and the debug info will be encoded in there (if I drop the NO_DEBUG_INFO_TESTCASE, the metaclass will "magically" create test methods "test_bar_dwarf", "test_bar_dwo", ...)

(Technically that still leaves the possibility that a single file will have two classes, and each of them will have the same method name, but no test currently does that, so we can just disallow that).

packages/Python/lldbsuite/test/lldbinline.py

95–97

2.7 is definitely fine, and we use {} format in plently of places already. (But I don't have any beef with the %s format either).

[aprantl]

Address review feedback from Alex.

[aprantl]

I am not sure this actually creates enough separation.

That's a good point. If I manage to extract the testname somehow via Python reflection magic, I could also get rid of the unintuitive self.mydir tuple. I'll give that a try.

packages/Python/lldbsuite/test/functionalities/breakpoint/comp_dir_symlink/TestCompDirSymLink.py

39–41

While most tests have their sources in the source directory, some use generated source file, which are located in the build directory, so that is unfortunately not an option.

packages/Python/lldbsuite/test/lldbinline.py

95–97

In this case I think this should just be return "-N dsym " + testdir

[labath]

packages/Python/lldbsuite/test/lldbinline.py

95–97

:D

packages/Python/lldbsuite/test/python_api/hello_world/TestHelloWorld.py

85–89

I think self.testMethodName is what you're looking for. (There also seems to be a self._testMethodName but I'm not sure what's the difference between the two.)

[aprantl]

This comment was removed by aprantl.

[aprantl]

Update the diff with Pavel's awesome suggestion of using self.testMethodName.

Note that that I haven't fixed all the resulting test error yet, I'm working through those now.

[aprantl]

@jingham wrote:

Now that we aren't mixing variants, would it be possible to have a test class claim that all the tests use the same binary file? At present to get self-contained tests you often need to do roughly the same thing many times, on the same binary. You only need to build it once per variant in that case. Of course, not all tests would behave this way, some play tricks with the binary products mid-test. But I bet the majority of test classes with more than one test method just build exactly the same thing as all the other tests, and don't modify it.

Yes, but I would like to keep this for a separate patch, since it is nontrivial:

• For NO_DEBUG_INFO_TESTCASEs this is as easy as invoking buildDefault() in self.setUp().
• For tests with variants, we first would need to define a method setUpVariant() that behaves like setUp() in that is invoked for each variant but before any tests are executed.

[aprantl]

Fix broken testcases.

[labath]

packages/Python/lldbsuite/test/api/listeners/TestListener.py

26

I'm confused by these changes. I was under the impression that setUp() runs before each test method (and hence this should be NFC). Can you explain the reasoning behind this?

[aprantl]

packages/Python/lldbsuite/test/api/listeners/TestListener.py

26

(see also my previous comment)
setUp runs before everything else, but it runs once per testcase and self.testMethodName is not initialized yet. Therefore we can only run self.build() in self.setUp() in NO_DEBUG_INFO_TESTCASEs.

[labath]

packages/Python/lldbsuite/test/api/listeners/TestListener.py

26

I'm not sure where you got the idea that testMethodName is not initialized. It's initialized as one of the first things in Base.setUp (lldbtest.py:776). This also demonstrates that the setUp method is run once *per test function* (as otherwise testMethodName would make no sense; also check out third_party/Python/module/unittest2/unittest2/case.py:379 to see how tests are run).

The problem I think you are having is that self.debug_info is not initialized. I was curious at hard it would be to make this work, so I played around with it a bit and the result was D42836.

It's not that I think this change is that bad (although I would prefer if we could keep doing these things in setUp), but I think this could also help make happen the ideas about building binaries once per test case (class) -- the way I'd do that is that I would enumerate all variants that are needed to build in setUpClass (and build them); then the actual test could fetch the corresponding binary according to the variant.

packages/Python/lldbsuite/test/lldbtest.py

721–728

Can we use the (new) lldbutil.mkdir_p here? If there's some circular dependency, maybe we could move the function somewhere else?

1510–1511

Why was this necessary? Could we either always pass the name by argument, or always fetch it from self? Maybe it will be possible after D42836?

[JDevlieghere]

packages/Python/lldbsuite/test/lldbinline.py

95–97

+1. I agree with Pavel that we should adopt the new formatters. Let's be idiomatic :-)

packages/Python/lldbsuite/test/lldbtest.py

728

"{} is not a directory".format(path)

[aprantl]

Updated to query self.getDebugInfo() in getBuildDir().

[aprantl]

Cleanup and address outstanding review feedback.

[labath]

This looks much better. Thanks.

packages/Python/lldbsuite/test/api/listeners/TestListener.py

 
 class ListenToModuleLoadedEvents (TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
-        self.build()
 
     def test_receiving_breakpoint_added(self):
         """Test that we get breakpoint added events, waiting on event classes on the debugger"""
+        self.build()
 
         my_listener = lldb.SBListener("test_listener")
 
         my_listener.StartListeningForEventClass(
             self.dbg,
             lldb.SBTarget.GetBroadcasterClassName(),
             lldb.SBTarget.eBroadcastBitBreakpointChanged)
 

packages/Python/lldbsuite/test/functionalities/breakpoint/comp_dir_symlink/TestCompDirSymLink.py

class CompDirSymLinkTestCase(TestBase):
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Find the line number to break inside main().
         self.line = line_number(
             os.path.join(self.getSourceDir(), "main.cpp"),
             '// Set break point at this line.')
-        self.src_path = self.getBuildArtifact(_SRC_FILE)
-
 
     @skipIf(hostoslist=["windows"])
     def test_symlink_paths_set(self):
         pwd_symlink = self.create_src_symlink()
         self.doBuild(pwd_symlink)
         self.runCmd(
             "settings set %s %s" %
             (_COMP_DIR_SYM_LINK_PROP, pwd_symlink))
-        lldbutil.run_break_set_by_file_and_line(self, self.src_path, self.line)
+        src_path = self.getBuildArtifact(_SRC_FILE)
+        lldbutil.run_break_set_by_file_and_line(self, src_path, self.line)
 
     @skipIf(hostoslist=no_match(["linux"]))
     def test_symlink_paths_set_procselfcwd(self):
         os.chdir(self.getBuildDir())
         pwd_symlink = '/proc/self/cwd'
         self.doBuild(pwd_symlink)
         self.runCmd(
             "settings set %s %s" %
             (_COMP_DIR_SYM_LINK_PROP, pwd_symlink))
-        lldbutil.run_break_set_by_file_and_line(self, self.src_path, self.line)
+        src_path = self.getBuildArtifact(_SRC_FILE)
+        lldbutil.run_break_set_by_file_and_line(self, src_path, self.line)
 
     @skipIf(hostoslist=["windows"])
     def test_symlink_paths_unset(self):
         pwd_symlink = self.create_src_symlink()
         self.doBuild(pwd_symlink)
         self.runCmd('settings clear ' + _COMP_DIR_SYM_LINK_PROP)
+        src_path = self.getBuildArtifact(_SRC_FILE)
         self.assertRaises(
             AssertionError,
             lldbutil.run_break_set_by_file_and_line,
             self,
-            self.src_path,
+            src_path,
             self.line)
 
     def create_src_symlink(self):
+        self.makeBuildDir()
         pwd_symlink = self.getBuildArtifact('pwd_symlink')
         if os.path.exists(pwd_symlink):
             os.unlink(pwd_symlink)
         os.symlink(self.getBuildDir(), pwd_symlink)
         self.addTearDownHook(lambda: os.remove(pwd_symlink))
         return pwd_symlink
 
     def doBuild(self, pwd_symlink):
         self.build(None, None, {'PWD': pwd_symlink}, True)
 
         exe = self.getBuildArtifact(_EXE_NAME)
         self.runCmd('file ' + exe, CURRENT_EXECUTABLE_SET)

packages/Python/lldbsuite/test/functionalities/load_unload/TestLoadUnload.py

 @skipIfWindows  # Windows doesn't have dlopen and friends, dynamic libraries work differently
 class LoadUnloadTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
-        lldbutil.mkdir_p(self.getBuildArtifact("hidden"))
         # Find the line number to break for main.cpp.
         self.line = line_number(
             'main.cpp',
             '// Set break point at this line for test_lldb_process_load_and_unload_commands().')
         self.line_d_function = line_number(
             'd.cpp', '// Find this line number within d_dunction().')
+
+    def setup_test(self):
+        lldbutil.mkdir_p(self.getBuildArtifact("hidden"))
         if not self.platformIsDarwin():
             if not lldb.remote_platform and "LD_LIBRARY_PATH" in os.environ:
                 self.runCmd(
                     "settings set target.env-vars " +
                     self.dylibPath +
                     "=" +
                     os.environ["LD_LIBRARY_PATH"] +
                     ":" +
class LoadUnloadTestCase(TestBase):
     # libloadunload_d.so does not appear in the image list because executable
     # dependencies are resolved relative to the debuggers PWD. Bug?
     @expectedFailureAll(oslist=["linux"])
     @skipIfFreeBSD  # llvm.org/pr14424 - missing FreeBSD Makefiles/testcase support
     @not_remote_testsuite_ready
     @skipIfWindows  # Windows doesn't have dlopen and friends, dynamic libraries work differently
     def test_modules_search_paths(self):
         """Test target modules list after loading a different copy of the library libd.dylib, and verifies that it works with 'target modules search-paths add'."""
-
+        self.setup_test()
         # Invoke the default build rule.
         self.build()
 
         if self.platformIsDarwin():
             dylibName = 'libloadunload_d.dylib'
         else:
             dylibName = 'libloadunload_d.so'
 
class LoadUnloadTestCase(TestBase):
     # libloadunload_d.so does not appear in the image list because executable
     # dependencies are resolved relative to the debuggers PWD. Bug?
     @expectedFailureAll(oslist=["linux"])
     @skipIfFreeBSD  # llvm.org/pr14424 - missing FreeBSD Makefiles/testcase support
     @expectedFailureAndroid  # wrong source file shows up for hidden library
     @skipIfWindows  # Windows doesn't have dlopen and friends, dynamic libraries work differently
     def test_dyld_library_path(self):
         """Test (DY)LD_LIBRARY_PATH after moving libd.dylib, which defines d_function, somewhere else."""
-
+        self.setup_test()
         # Invoke the default build rule.
         self.build()
         self.copy_shlibs_to_remote(hidden_dir=True)
 
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         # Shut off ANSI color usage so we don't get ANSI escape sequences
class LoadUnloadTestCase(TestBase):
     @expectedFailureAll(
         bugnumber="llvm.org/pr25805",
         hostoslist=["windows"],
         triple='.*-android')
     @skipIfFreeBSD  # llvm.org/pr14424 - missing FreeBSD Makefiles/testcase support
     @skipIfWindows  # Windows doesn't have dlopen and friends, dynamic libraries work differently
     def test_lldb_process_load_and_unload_commands(self):
         """Test that lldb process load/unload command work correctly."""
-
+        self.setup_test()
         # Invoke the default build rule.
         self.build()
         self.copy_shlibs_to_remote()
 
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         # Break at main.cpp before the call to dlopen().
def test_lldb_process_load_and_unload_commands(self):
                 "Unloading .* with index %s.*ok" %
                 index])
 
         self.runCmd("process continue")
 
     @skipIfFreeBSD  # llvm.org/pr14424 - missing FreeBSD Makefiles/testcase support
     def test_load_unload(self):
         """Test breakpoint by name works correctly with dlopen'ing."""
-
+        self.setup_test()
         # Invoke the default build rule.
         self.build()
         self.copy_shlibs_to_remote()
 
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         # Break by function name a_function (not yet loaded).
def test_load_unload(self):
         # The breakpoint should have a hit count of 2.
         self.expect("breakpoint list -f", BREAKPOINT_HIT_ONCE,
                     substrs=[' resolved, hit count = 2'])
 
     @skipIfFreeBSD  # llvm.org/pr14424 - missing FreeBSD Makefiles/testcase support
     @skipIfWindows  # Windows doesn't have dlopen and friends, dynamic libraries work differently
     def test_step_over_load(self):
         """Test stepping over code that loads a shared library works correctly."""
-
+        self.setup_test()
         # Invoke the default build rule.
         self.build()
         self.copy_shlibs_to_remote()
 
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         # Break by function name a_function (not yet loaded).
class LoadUnloadTestCase(TestBase):
 
     # We can't find a breakpoint location for d_init before launching because
     # executable dependencies are resolved relative to the debuggers PWD. Bug?
     @expectedFailureAll(oslist=["linux"])
     @skipIfFreeBSD  # llvm.org/pr14424 - missing FreeBSD Makefiles/testcase support
     @skipIfWindows  # Windows doesn't have dlopen and friends, dynamic libraries work differently
     def test_static_init_during_load(self):
         """Test that we can set breakpoints correctly in static initializers"""
-
+        self.setup_test()
         self.build()
         self.copy_shlibs_to_remote()
 
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         a_init_bp_num = lldbutil.run_break_set_by_symbol(
             self, "a_init", num_expected_locations=0)

packages/Python/lldbsuite/test/functionalities/watchpoint/watchpoint_on_vectors/TestValueOfVectorVariable.py

class TestValueOfVectorVariableTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @expectedFailureAll(
         oslist=["windows"],
         bugnumber="llvm.org/pr24446: WINDOWS XFAIL TRIAGE - Watchpoints not supported on Windows")
     def test_value_of_vector_variable_using_watchpoint_set(self):
         """Test verify displayed value of vector variable."""
-        self.build(dictionary=self.d)
-        self.setTearDownCleanup(dictionary=self.d)
+        exe = self.getBuildArtifact("a.out")
+        d = {'C_SOURCES': self.source, 'EXE': exe}
+        self.build(dictionary=d)
+        self.setTearDownCleanup(dictionary=d)
         self.value_of_vector_variable_with_watchpoint_set()
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Our simple source filename.
         self.source = 'main.c'
-        self.exe_name = self.getBuildArtifact("a.out")
-        self.d = {'C_SOURCES': self.source, 'EXE': self.exe_name}
 
     def value_of_vector_variable_with_watchpoint_set(self):
         """Test verify displayed value of vector variable"""
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         # Set break to get a frame
         self.runCmd("b main")
 
         # Run the program.
         self.runCmd("run", RUN_SUCCEEDED)
 
         # Value of a vector variable should be displayed correctly
         self.expect(
             "watchpoint set variable global_vector",
             WATCHPOINT_CREATED,
             substrs=['new value: (1, 2, 3, 4)'])

packages/Python/lldbsuite/test/lldbinline.py

class InlineTest(TestBase):
     def getRerunArgs(self):
         # The -N option says to NOT run a if it matches the option argument, so
         # if we are using dSYM we say to NOT run dwarf (-N dwarf) and vice
         # versa.
         if self.using_dsym is None:
             # The test was skipped altogether.
             return ""
         elif self.using_dsym:
-            return "-N dwarf %s" % (self.mydir)
+            return "-N dwarf " + self.mydir
         else:
-            return "-N dsym %s" % (self.mydir)
+            return "-N dsym " + self.mydir
 
     def BuildMakefile(self):
         self.makeBuildDir()
         makefilePath = self.getBuildArtifact("Makefile")
         if os.path.exists(makefilePath):
             return
 
         categories = {}

packages/Python/lldbsuite/test/lldbtest.py

class Base(unittest2.TestCase):
     # The concrete subclass should override this attribute.
     mydir = None
 
     # Keep track of the old current working directory.
     oldcwd = None
 
     @staticmethod
     def compute_mydir(test_file):
-        '''Subclasses should call this function to correctly calculate the required "mydir" attribute as follows:
+        '''Subclasses should call this function to correctly calculate the
+           required "mydir" attribute as follows:
 
-            mydir = TestBase.compute_mydir(__file__)'''
-        test_dir = os.path.dirname(test_file)
-        return test_dir[len(os.environ["LLDB_TEST"]) + 1:]
+            mydir = TestBase.compute_mydir(__file__)
+        '''
+        # /abs/path/to/packages/group/subdir/mytest.py -> group/subdir
+        rel_prefix = test_file[len(os.environ["LLDB_TEST"]) + 1:]
+        return os.path.dirname(rel_prefix)
 
     def TraceOn(self):
         """Returns True if we are in trace mode (tracing detailed test execution)."""
         return traceAlways
 
     @classmethod
     def setUpClass(cls):
         """
         Python unittest framework class setup fixture.
         Do current directory manipulation.
         """
         # Fail fast if 'mydir' attribute is not overridden.
         if not cls.mydir or len(cls.mydir) == 0:
             raise Exception("Subclasses must override the 'mydir' attribute.")
 
         # Save old working directory.
         cls.oldcwd = os.getcwd()
 
         # Change current working directory if ${LLDB_TEST} is defined.
         # See also dotest.py which sets up ${LLDB_TEST}.
         if ("LLDB_TEST" in os.environ):
-            full_dir = os.path.join(os.environ["LLDB_TEST"], cls.mydir)
+            full_dir = os.path.join(os.environ["LLDB_TEST"],
+                                    cls.mydir)
             if traceAlways:
                 print("Change dir to:", full_dir, file=sys.stderr)
-            os.chdir(os.path.join(os.environ["LLDB_TEST"], cls.mydir))
-
-        # TODO: Obsolete this by creating one working dir per configuration.
-        if debug_confirm_directory_exclusivity:
-            import lock
-            cls.dir_lock = lock.Lock(os.path.join(full_dir, ".dirlock"))
-            try:
-                cls.dir_lock.try_acquire()
-                # write the class that owns the lock into the lock file
-                cls.dir_lock.handle.write(cls.__name__)
-            except IOError as ioerror:
-                # nothing else should have this directory lock
-                # wait here until we get a lock
-                cls.dir_lock.acquire()
-                # read the previous owner from the lock file
-                lock_id = cls.dir_lock.handle.read()
-                print(
-                    "LOCK ERROR: {} wants to lock '{}' but it is already locked by '{}'".format(
-                        cls.__name__,
-                        full_dir,
-                        lock_id),
-                    file=sys.stderr)
-                raise ioerror
+            os.chdir(full_dir)
 
         # Set platform context.
         cls.platformContext = lldbplatformutil.createPlatformContext()
 
     @classmethod
     def tearDownClass(cls):
         """
         Python unittest framework class teardown fixture.
def setPlatformWorkingDir(self):
             # list of files we have to remove during test runs.
             shell_cmd = lldb.SBPlatformShellCommand(
                 "rm %s/*" % remote_test_dir)
             lldb.remote_platform.Run(shell_cmd)
         self.addTearDownHook(clean_working_directory)
 
     def getSourceDir(self):
         """Return the full path to the current test."""
         return os.path.join(os.environ["LLDB_TEST"], self.mydir)
 
     def getBuildDir(self):
         """Return the full path to the current test."""
-        return os.path.join(os.environ["LLDB_BUILD"], self.mydir)
+        variant = self.getDebugInfo()
+        if variant is None:
+            variant = 'default'
+        return os.path.join(os.environ["LLDB_BUILD"], self.mydir,
+                            self.testMethodName)
     
      
     def makeBuildDir(self):
         """Create the test-specific working directory."""
         # See also dotest.py which sets up ${LLDB_BUILD}.
-        try: os.makedirs(self.getBuildDir())
-        except: pass
+        lldbutil.mkdir_p(self.getBuildDir())
  
     def getBuildArtifact(self, name="a.out"):
         """Return absolute path to an artifact in the test's build directory."""
         return os.path.join(self.getBuildDir(), name)
  
     def getSourcePath(self, name):
         """Return absolute path to a file in the test's source directory."""
         return os.path.join(self.getSourceDir(), name)
 
     def setUp(self):
         """Fixture for unittest test case setup.
 
         It works with the test driver to conditionally skip tests and does other
         initializations."""
         #import traceback
         # traceback.print_stack()
def buildProgram(self, sources, exe_name):
         self.buildDefault(dictionary=d)
 
     def buildDefault(
             self,
             architecture=None,
             compiler=None,
             dictionary=None,
             clean=True,
-            testdir=None):
+            testdir=None,
+            testname=None):
         """Platform specific way to build the default binaries."""
         if not testdir:
             testdir = self.mydir
+        if not testname:
+            testname = self.testMethodName
         if self.getDebugInfo():
             raise Exception("buildDefault tests must set NO_DEBUG_INFO_TESTCASE")
         module = builder_module()
         self.makeBuildDir()
         dictionary = lldbplatformutil.finalize_build_dictionary(dictionary)
         if not module.buildDefault(self, architecture, compiler,
-                                   dictionary, clean, testdir):
+                                   dictionary, clean, testdir, testname):
             raise Exception("Don't know how to build default binary")
 
     def buildDsym(
             self,
             architecture=None,
             compiler=None,
             dictionary=None,
             clean=True,
-            testdir=None):
+            testdir=None,
+            testname=None):
         """Platform specific way to build binaries with dsym info."""
         if not testdir:
             testdir = self.mydir
+        if not testname:
+            testname = self.testMethodName
         if self.getDebugInfo() != "dsym":
             raise Exception("NO_DEBUG_INFO_TESTCASE must build with buildDefault")
 
         module = builder_module()
         dictionary = lldbplatformutil.finalize_build_dictionary(dictionary)
         if not module.buildDsym(self, architecture, compiler,
-                                dictionary, clean, testdir):
+                                dictionary, clean, testdir, testname):
             raise Exception("Don't know how to build binary with dsym")
 
     def buildDwarf(
             self,
             architecture=None,
             compiler=None,
             dictionary=None,
             clean=True,
-            testdir=None):
+            testdir=None,
+            testname=None):
         """Platform specific way to build binaries with dwarf maps."""
         if not testdir:
             testdir = self.mydir
+        if not testname:
+            testname = self.testMethodName
         if self.getDebugInfo() != "dwarf":
             raise Exception("NO_DEBUG_INFO_TESTCASE must build with buildDefault")
 
         module = builder_module()
         dictionary = lldbplatformutil.finalize_build_dictionary(dictionary)
         if not module.buildDwarf(self, architecture, compiler,
-                                   dictionary, clean, testdir):
+                                   dictionary, clean, testdir, testname):
             raise Exception("Don't know how to build binary with dwarf")
 
     def buildDwo(
             self,
             architecture=None,
             compiler=None,
             dictionary=None,
             clean=True,
-            testdir=None):
+            testdir=None,
+            testname=None):
         """Platform specific way to build binaries with dwarf maps."""
         if not testdir:
             testdir = self.mydir
+        if not testname:
+            testname = self.testMethodName
         if self.getDebugInfo() != "dwo":
             raise Exception("NO_DEBUG_INFO_TESTCASE must build with buildDefault")
 
         module = builder_module()
         dictionary = lldbplatformutil.finalize_build_dictionary(dictionary)
         if not module.buildDwo(self, architecture, compiler,
-                                   dictionary, clean, testdir):
+                                   dictionary, clean, testdir, testname):
             raise Exception("Don't know how to build binary with dwo")
 
     def buildGModules(
             self,
             architecture=None,
             compiler=None,
             dictionary=None,
             clean=True,
-            testdir=None):
+            testdir=None,
+            testname=None):
         """Platform specific way to build binaries with gmodules info."""
         if not testdir:
             testdir = self.mydir
+        if not testname:
+            testname = self.testMethodName
         if self.getDebugInfo() != "gmodules":
             raise Exception("NO_DEBUG_INFO_TESTCASE must build with buildDefault")
 
         module = builder_module()
         dictionary = lldbplatformutil.finalize_build_dictionary(dictionary)
         if not module.buildGModules(self, architecture, compiler,
-                                   dictionary, clean, testdir):
+                                    dictionary, clean, testdir, testname):
             raise Exception("Don't know how to build binary with gmodules")
 
     def buildGo(self):
         """Build the default go binary.
         """
         exe = self.getBuildArtifact("a.out")
         system([[which('go'), 'build -gcflags "-N -l" -o %s main.go' % exe]])
 
def __new__(cls, name, bases, attrs):
                         "categories",
                         [])) & all_dbginfo_categories
                 if not categories:
                     categories = all_dbginfo_categories
 
                 supported_categories = [
                     x for x in categories if test_categories.is_supported_on_platform(
                         x, target_platform, configuration.compiler)]
+                
                 if "dsym" in supported_categories:
                     @decorators.add_test_categories(["dsym"])
                     @wraps(attrvalue)
                     def dsym_test_method(self, attrvalue=attrvalue):
                         return attrvalue(self)
                     dsym_method_name = attrname + "_dsym"
                     dsym_test_method.__name__ = dsym_method_name
                     dsym_test_method.debug_info = "dsym"

packages/Python/lldbsuite/test/plugins/builder_base.py

elif "gcc" in compiler:
         archflag = "-m"
     elif "clang" in compiler:
         archflag = "-arch"
     else:
         archflag = None
 
     return ("ARCHFLAG=" + archflag) if archflag else ""
 
-def getMake(test_subdir):
+def getMake(test_subdir, test_name):
     """Returns the invocation for GNU make.
-       The argument test_subdir is the relative path to the testcase."""
+       The first argument is a tuple of the relative path to the testcase
+       and its filename stem."""
     if platform.system() == "FreeBSD" or platform.system() == "NetBSD":
         make = "gmake"
     else:
         make = "make"
 
     # Construct the base make invocation.
     lldb_test = os.environ["LLDB_TEST"]
     lldb_build = os.environ["LLDB_BUILD"]
-    if not (lldb_test and lldb_build and test_subdir and
+    if not (lldb_test and lldb_build and test_subdir and test_name and
             (not os.path.isabs(test_subdir))):
         raise Exception("Could not derive test directories")
-    build_dir = os.path.join(lldb_build, test_subdir)
-    test_dir = os.path.join(lldb_test, test_subdir)
+    build_dir = os.path.join(lldb_build, test_subdir, test_name)
+    src_dir = os.path.join(lldb_test, test_subdir)
     # This is a bit of a hack to make inline testcases work.
-    makefile = os.path.join(test_dir, "Makefile")
+    makefile = os.path.join(src_dir, "Makefile")
     if not os.path.isfile(makefile):
         makefile = os.path.join(build_dir, "Makefile")
     return [make,
-            "VPATH="+test_dir,
+            "VPATH="+src_dir,
             "-C", build_dir,
-            "-I", test_dir,
+            "-I", src_dir,
             "-f", makefile]
 
 
 def getArchSpec(architecture):
     """
     Helper function to return the key-value string to specify the architecture
     used for the make system.
     """

 
 
 def buildDefault(
         sender=None,
         architecture=None,
         compiler=None,
         dictionary=None,
         clean=True,
-        testdir=None):
+        testdir=None,
+        testname=None):
     """Build the binaries the default way."""
     commands = []
     if clean:
-        commands.append(getMake(testdir) + ["clean", getCmdLine(dictionary)])
-    commands.append(getMake(testdir) + ["all", getArchSpec(architecture),
+        commands.append(getMake(testdir, testname) + ["clean", getCmdLine(dictionary)])
+    commands.append(getMake(testdir, testname) + ["all", getArchSpec(architecture),
                      getCCSpec(compiler), getCmdLine(dictionary)])
 
     runBuildCommands(commands, sender=sender)
 
     # True signifies that we can handle building default.
     return True
 
 
 def buildDwarf(
         sender=None,
         architecture=None,
         compiler=None,
         dictionary=None,
         clean=True,
-        testdir=None):
+        testdir=None,
+        testname=None):
     """Build the binaries with dwarf debug info."""
     commands = []
     if clean:
-        commands.append(getMake(testdir) + ["clean", getCmdLine(dictionary)])
-    commands.append(getMake(testdir) + ["MAKE_DSYM=NO", getArchSpec(
-        architecture), getCCSpec(compiler), getCmdLine(dictionary)])
+        commands.append(getMake(testdir, testname) +
+                        ["clean", getCmdLine(dictionary)])
+    commands.append(getMake(testdir, testname) +
+                    ["MAKE_DSYM=NO", getArchSpec(architecture),
+                     getCCSpec(compiler), getCmdLine(dictionary)])
 
     runBuildCommands(commands, sender=sender)
     # True signifies that we can handle building dwarf.
     return True
 
 
 def buildDwo(
         sender=None,
         architecture=None,
         compiler=None,
         dictionary=None,
         clean=True,
-        testdir=None):
+        testdir=None,
+        testname=None):
     """Build the binaries with dwarf debug info."""
     commands = []
     if clean:
-        commands.append(getMake(testdir) + ["clean", getCmdLine(dictionary)])
-    commands.append(getMake(testdir) +
+        commands.append(getMake(testdir, testname) +
+                        ["clean", getCmdLine(dictionary)])
+    commands.append(getMake(testdir, testname) +
                     ["MAKE_DSYM=NO", "MAKE_DWO=YES",
                      getArchSpec(architecture),
                      getCCSpec(compiler),
                      getCmdLine(dictionary)])
 
     runBuildCommands(commands, sender=sender)
     # True signifies that we can handle building dwo.
     return True
 
 
 def buildGModules(
         sender=None,
         architecture=None,
         compiler=None,
         dictionary=None,
         clean=True,
-        testdir=None):
+        testdir=None,
+        testname=None):
     """Build the binaries with dwarf debug info."""
     commands = []
     if clean:
-        commands.append(getMake(testdir) + ["clean", getCmdLine(dictionary)])
-    commands.append(getMake(testdir) +
+        commands.append(getMake(testdir, testname) +
+                        ["clean", getCmdLine(dictionary)])
+    commands.append(getMake(testdir, testname) +
                     ["MAKE_DSYM=NO",
                      "MAKE_GMODULES=YES",
                      getArchSpec(architecture),
                      getCCSpec(compiler),
                      getCmdLine(dictionary)])
 
     lldbtest.system(commands, sender=sender)
     # True signifies that we can handle building with gmodules.
     return True
 
 
 def cleanup(sender=None, dictionary=None):
     """Perform a platform-specific cleanup after the test."""
     return True

packages/Python/lldbsuite/test/plugins/builder_darwin.py

 
 from __future__ import print_function
 import os
 import lldbsuite.test.lldbtest as lldbtest
 
 from builder_base import *
 
 def buildDsym(
         sender=None,
         architecture=None,
         compiler=None,
         dictionary=None,
         clean=True,
-        testdir=None):
+        testdir=None,
+        testname=None):
     """Build the binaries with dsym debug info."""
     commands = []
-
     if clean:
-        commands.append(getMake(testdir) + ["clean", getCmdLine(dictionary)])
-    commands.append(getMake(testdir) +
+        commands.append(getMake(testdir, testname) +
+                        ["clean", getCmdLine(dictionary)])
+    commands.append(getMake(testdir, testname) +
                     ["MAKE_DSYM=YES",
                      getArchSpec(architecture),
                      getCCSpec(compiler),
                      "all", getCmdLine(dictionary)])
 
     runBuildCommands(commands, sender=sender)
 
     # True signifies that we can handle building dsym.
     return True

packages/Python/lldbsuite/test/python_api/hello_world/TestHelloWorld.py

 
 class HelloWorldTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
-        # Get the full path to our executable to be attached/debugged.
-        self.exe = self.getBuildArtifact(self.testMethodName)
-        self.d = {'EXE': self.testMethodName}
         # Find a couple of the line numbers within main.c.
         self.line1 = line_number('main.c', '// Set break point at this line.')
         self.line2 = line_number('main.c', '// Waiting to be attached...')
 
     def tearDown(self):
         # Destroy process before TestBase.tearDown()
         self.dbg.GetSelectedTarget().GetProcess().Destroy()
         # Call super's tearDown().
         TestBase.tearDown(self)
 
     @add_test_categories(['pyapi'])
     @skipIfiOSSimulator
     def test_with_process_launch_api(self):
         """Create target, breakpoint, launch a process, and then kill it."""
-        self.build(dictionary=self.d)
-        self.setTearDownCleanup(dictionary=self.d)
-        target = self.dbg.CreateTarget(self.exe)
+        # Get the full path to our executable to be attached/debugged.
+        exe = self.getBuildArtifact(self.testMethodName)
+        d = {'EXE': exe}
+        self.build(dictionary=d)
+        self.setTearDownCleanup(dictionary=d)
+        target = self.dbg.CreateTarget(exe)
 
         breakpoint = target.BreakpointCreateByLocation("main.c", self.line1)
 
         # The default state after breakpoint creation should be enabled.
         self.assertTrue(breakpoint.IsEnabled(),
                         "Breakpoint should be enabled after creation")
 
         breakpoint.SetEnabled(False)
def test_with_process_launch_api(self):
         self.assertEqual(breakpoint.GetHitCount(), 1, BREAKPOINT_HIT_ONCE)
 
     @add_test_categories(['pyapi'])
     @expectedFailureAll(oslist=["windows"], bugnumber="llvm.org/pr24600")
     @skipIfiOSSimulator
     @expectedFailureAll(oslist=['ios', 'watchos', 'tvos', 'bridgeos'], bugnumber="<rdar://problem/34538611>") # old lldb-server has race condition, launching an inferior and then launching debugserver in quick succession sometimes fails
     def test_with_attach_to_process_with_id_api(self):
         """Create target, spawn a process, and attach to it with process id."""
-        self.build(dictionary=self.d)
-        self.setTearDownCleanup(dictionary=self.d)
-        target = self.dbg.CreateTarget(self.exe)
+        exe = self.getBuildArtifact(self.testMethodName)
+        d = {'EXE': exe}
+        self.build(dictionary=d)
+        self.setTearDownCleanup(dictionary=d)
+        target = self.dbg.CreateTarget(exe)
 
         # Spawn a new process
-        popen = self.spawnSubprocess(self.exe, ["abc", "xyz"])
+        popen = self.spawnSubprocess(exe, ["abc", "xyz"])
         self.addTearDownHook(self.cleanupSubprocesses)
 
         # Give the subprocess time to start and wait for user input
         time.sleep(0.25)
 
         listener = lldb.SBListener("my.attach.listener")
         error = lldb.SBError()
         process = target.AttachToProcessWithID(listener, popen.pid, error)
 
         self.assertTrue(error.Success() and process, PROCESS_IS_VALID)
 
         # Let's check the stack traces of the attached process.
         import lldbsuite.test.lldbutil as lldbutil
         stacktraces = lldbutil.print_stacktraces(process, string_buffer=True)
         self.expect(stacktraces, exe=False,
                     substrs=['main.c:%d' % self.line2,
                              '(int)argc=3'])
 
     @add_test_categories(['pyapi'])
     @expectedFailureAll(oslist=["windows"], bugnumber="llvm.org/pr24600")
     @skipIfiOSSimulator
     @expectedFailureAll(oslist=['ios', 'watchos', 'tvos', 'bridgeos'], bugnumber="<rdar://problem/34538611>") # old lldb-server has race condition, launching an inferior and then launching debugserver in quick succession sometimes fails
     def test_with_attach_to_process_with_name_api(self):
         """Create target, spawn a process, and attach to it with process name."""
-        self.build(dictionary=self.d)
-        self.setTearDownCleanup(dictionary=self.d)
-        target = self.dbg.CreateTarget(self.exe)
+        exe = self.getBuildArtifact(self.testMethodName)
+        d = {'EXE': exe}
+        self.build(dictionary=d)
+        self.setTearDownCleanup(dictionary=d)
+        target = self.dbg.CreateTarget(exe)
 
         # Spawn a new process
-        popen = self.spawnSubprocess(self.exe, ["abc", "xyz"])
+        popen = self.spawnSubprocess(exe, ["abc", "xyz"])
         self.addTearDownHook(self.cleanupSubprocesses)
 
         # Give the subprocess time to start and wait for user input
         time.sleep(0.25)
 
         listener = lldb.SBListener("my.attach.listener")
         error = lldb.SBError()
         # Pass 'False' since we don't want to wait for new instance of
         # "hello_world" to be launched.
-        name = os.path.basename(self.exe)
+        name = os.path.basename(exe)
 
         # While we're at it, make sure that passing a None as the process name
         # does not hang LLDB.
         target.AttachToProcessWithName(listener, None, False, error)
         # Also boundary condition test ConnectRemote(), too.
         target.ConnectRemote(listener, None, None, error)
 
         process = target.AttachToProcessWithName(listener, name, False, error)

packages/Python/lldbsuite/test/python_api/process/io/TestProcessIO.py

 
 class ProcessIOTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
+
+    def setup_test(self):
         # Get the full path to our executable to be debugged.
         self.exe = self.getBuildArtifact("process_io")
         self.local_input_file = self.getBuildArtifact("input.txt")
         self.local_output_file = self.getBuildArtifact("output.txt")
         self.local_error_file = self.getBuildArtifact("error.txt")
 
         self.input_file = os.path.join(
             self.get_process_working_directory(), "input.txt")
         self.output_file = os.path.join(
             self.get_process_working_directory(), "output.txt")
         self.error_file = os.path.join(
             self.get_process_working_directory(), "error.txt")
         self.lines = ["Line 1", "Line 2", "Line 3"]
 
     @skipIfWindows  # stdio manipulation unsupported on Windows
     @add_test_categories(['pyapi'])
     @expectedFlakeyLinux(bugnumber="llvm.org/pr26437")
     def test_stdin_by_api(self):
         """Exercise SBProcess.PutSTDIN()."""
+        self.setup_test()
         self.build()
         self.create_target()
         self.run_process(True)
         output = self.process.GetSTDOUT(1000)
         self.check_process_output(output, output)
 
     @skipIfWindows  # stdio manipulation unsupported on Windows
     @add_test_categories(['pyapi'])
     @expectedFlakeyLinux(bugnumber="llvm.org/pr26437")
     def test_stdin_redirection(self):
         """Exercise SBLaunchInfo::AddOpenFileAction() for STDIN without specifying STDOUT or STDERR."""
+        self.setup_test()
         self.build()
         self.create_target()
         self.redirect_stdin()
         self.run_process(False)
         output = self.process.GetSTDOUT(1000)
         self.check_process_output(output, output)
 
     @skipIfWindows  # stdio manipulation unsupported on Windows
     @add_test_categories(['pyapi'])
     @expectedFlakeyLinux(bugnumber="llvm.org/pr26437")
     @skipIfDarwinEmbedded # debugserver can't create/write files on the device
     def test_stdout_redirection(self):
         """Exercise SBLaunchInfo::AddOpenFileAction() for STDOUT without specifying STDIN or STDERR."""
+        self.setup_test()
         self.build()
         self.create_target()
         self.redirect_stdout()
         self.run_process(True)
         output = self.read_output_file_and_delete()
         error = self.process.GetSTDOUT(1000)
         self.check_process_output(output, error)
 
     @skipIfWindows  # stdio manipulation unsupported on Windows
     @add_test_categories(['pyapi'])
     @expectedFlakeyLinux(bugnumber="llvm.org/pr26437")
     @skipIfDarwinEmbedded # debugserver can't create/write files on the device
     def test_stderr_redirection(self):
         """Exercise SBLaunchInfo::AddOpenFileAction() for STDERR without specifying STDIN or STDOUT."""
+        self.setup_test()
         self.build()
         self.create_target()
         self.redirect_stderr()
         self.run_process(True)
         output = self.process.GetSTDOUT(1000)
         error = self.read_error_file_and_delete()
         self.check_process_output(output, error)
 
     @skipIfWindows  # stdio manipulation unsupported on Windows
     @add_test_categories(['pyapi'])
     @expectedFlakeyLinux(bugnumber="llvm.org/pr26437")
     @skipIfDarwinEmbedded # debugserver can't create/write files on the device
     def test_stdout_stderr_redirection(self):
         """Exercise SBLaunchInfo::AddOpenFileAction() for STDOUT and STDERR without redirecting STDIN."""
+        self.setup_test()
         self.build()
         self.create_target()
         self.redirect_stdout()
         self.redirect_stderr()
         self.run_process(True)
         output = self.read_output_file_and_delete()
         error = self.read_error_file_and_delete()
         self.check_process_output(output, error)

packages/Python/lldbsuite/test/python_api/symbol-context/TestSymbolContext.py

def test(self):
         self.expect(desc, "The module should match", exe=False,
                     substrs=[self.getBuildArtifact("a.out")])
 
         compileUnit = context.GetCompileUnit()
         self.expect(
             str(compileUnit),
             "The compile unit should match",
             exe=False,
-            substrs=[
-                os.path.join(
-                    self.mydir,
-                    'main.c')])
+            substrs=[self.getSourcePath('main.c')])
 
         function = context.GetFunction()
         self.assertTrue(function)
         #print("function:", function)
 
         block = context.GetBlock()
         self.assertTrue(block)
         #print("block:", block)
 
         lineEntry = context.GetLineEntry()
         #print("line entry:", lineEntry)
         self.expect(
             lineEntry.GetFileSpec().GetDirectory(),
             "The line entry should have the correct directory",
             exe=False,
-            substrs=[
-                self.mydir])
+            substrs=[self.mydir])
         self.expect(
             lineEntry.GetFileSpec().GetFilename(),
             "The line entry should have the correct filename",
             exe=False,
             substrs=['main.c'])
         self.assertTrue(lineEntry.GetLine() == self.line,
                         "The line entry's line number should match ")
 
         symbol = context.GetSymbol()
         self.assertTrue(
             function.GetName() == symbol.GetName() and symbol.GetName() == 'c',
             "The symbol name should be 'c'")

packages/Python/lldbsuite/test/python_api/symbol-context/two-files/TestSymbolContextTwoFiles.py

class SymbolContextTwoFilesTestCase(TestBase):
     mydir = TestBase.compute_mydir(__file__)
 
     @add_test_categories(['pyapi'])
     @expectedFailureAll(oslist=["windows"])
     def test_lookup_by_address(self):
         """Test lookup by address in a module with multiple compilation units"""
         self.build()
         exe = self.getBuildArtifact("a.out")
-
         target = self.dbg.CreateTarget(exe)
         self.assertTrue(target, VALID_TARGET)
 
         module = target.GetModuleAtIndex(0)
         self.assertTrue(module.IsValid())
         for symbol_name in ["struct1::f()", "struct2::f()"]:
             sc_list = module.FindFunctions(symbol_name, lldb.eSymbolTypeCode)
             self.assertTrue(1, sc_list.GetSize())
             symbol_address = sc_list.GetContextAtIndex(
                 0).GetSymbol().GetStartAddress()
             self.assertTrue(symbol_address.IsValid())
             sc_by_address = module.ResolveSymbolContextForAddress(
                 symbol_address, lldb.eSymbolContextFunction)
             self.assertEqual(symbol_name,
                              sc_by_address.GetFunction().GetName())
 
     @add_test_categories(['pyapi'])
     def test_ranges_in_multiple_compile_unit(self):
         """This test verifies that we correctly handle the case when multiple
         compile unit contains DW_AT_ranges and DW_AT_ranges_base attributes."""
         self.build()
         exe = self.getBuildArtifact("a.out")
-
         target = self.dbg.CreateTarget(exe)
         self.assertTrue(target, VALID_TARGET)
 
         source1 = "file1.cpp"
         line1 = line_number(source1, '// Break1')
         breakpoint1 = target.BreakpointCreateByLocation(source1, line1)
         self.assertIsNotNone(breakpoint1)
         self.assertTrue(breakpoint1.IsValid())

packages/Python/lldbsuite/test/python_api/value_var_update/TestValueVarUpdate.py

 
 class HelloWorldTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
-        # Get the full path to our executable to be attached/debugged.
-        self.exe = self.getBuildArtifact(self.testMethodName)
-        self.d = {'EXE': self.testMethodName}
 
     @add_test_categories(['pyapi'])
     def test_with_process_launch_api(self):
         """Test SBValue::GetValueDidChange"""
-        self.build(dictionary=self.d)
-        self.setTearDownCleanup(dictionary=self.d)
-        target = self.dbg.CreateTarget(self.exe)
+        # Get the full path to our executable to be attached/debugged.
+        exe = self.getBuildArtifact(self.testMethodName)
+        d = {'EXE': exe}
+        self.build(dictionary=d)
+        self.setTearDownCleanup(dictionary=d)
+        target = self.dbg.CreateTarget(exe)
 
         breakpoint = target.BreakpointCreateBySourceRegex(
             "break here", lldb.SBFileSpec("main.c"))
 
         self.runCmd("run", RUN_SUCCEEDED)
 
         # The stop reason of the thread should be breakpoint.
         self.expect("thread list", STOPPED_DUE_TO_BREAKPOINT,