[LIT] Added an option to llvm-lit to emit the necessary test coverage data, divided per test case

[xgupta]

Summary

This patch is the first part of https://llvm.org/OpenProjects.html#llvm_patch_coverage.

We have first define a new variable LLVM_TEST_COVERAGE which when set, pass --emit-coverage option to
llvm-lit which will help in setting a unique value to LLVM_PROFILE_FILE for each RUN. So for example
coverage data for test case llvm/test/Analysis/AliasSet/memtransfer.ll will be emitted as
build/test/Analysis/AliasSet/memtransfer.profraw

[xgupta]

minor update

[xgupta]

.

[hnrklssn]

Please also add a test of lit itself (see llvm/utils/lit/tests). I think something using --emit-coverage on a test that echoes $LLVM_PROFILE_FILE should be enough.

llvm/utils/lit/lit/TestRunner.py

1139–1140

This call seems redundant

llvm/utils/lit/lit/cl_arguments.py

188

Since this option doesn't actually affect whether the tests emit coverage we should probably name it something more descriptive, e.g. --per-test-coverage or something to that regard.

[xgupta]

Address comments
Need to fix test case [WIP]

[xgupta]

Update test case [WIP]

[hnrklssn]

llvm/utils/lit/lit/TestRunner.py

1148

This looks like it might remove other existing env vars. Could you add a test case checking that other variables remain also?

[xgupta]

Update to avoid overwriting env

[xgupta]

Please also add a test of lit itself (see llvm/utils/lit/tests). I think something using --emit-coverage on a test that echoes $LLVM_PROFILE_FILE should be enough.

Hi Henrik,

Currently, I am getting LLVM_PROFILE_FILE = None instead of LVM_PROFILE_FILE = per-test-coverage.profraw so the test case is failing. And I can't use .ll test case as the opt and other tools should not have a dependency to lit test cases.

Can you suggest if this is the right way to have test case?

llvm/utils/lit/lit/TestRunner.py

1148

I have updated the code that will not overwrite env.

[hnrklssn]

llvm/utils/lit/tests/Inputs/per-test-coverage/lit.cfg

8

Instead of fetching the environment variable here (which seems like it's probably happening before the test is executed?) as part of the test framework, try running a script that reads the env var on its own

[xgupta]

Update test case [WIP]

[xgupta]

llvm/utils/lit/tests/Inputs/per-test-coverage/lit.cfg

8

Tried with a script per-test-coverage.py but not working it prints None, same for other env variables. Also same thing happen with .sh script attached in the patch as wip.

[xgupta]

llvm/utils/lit/tests/Inputs/per-test-coverage/lit.cfg

8

But the patch is working as expected I could see -

llvm/llvm-upstream/build $ bin/llvm-lit -a -v --per-test-coverage  ../llvm/test/Analysis/AliasSet

llvm/llvm-upstream/build  $ find .  -type f -name "*.profraw" | grep -i ".profraw$"  
./test/Analysis/AliasSet/unknown-inst-tracking.profraw
./test/Analysis/AliasSet/memtransfer.profraw
./test/Analysis/AliasSet/argmemonly.profraw
./test/Analysis/AliasSet/saturation.profraw
./test/Analysis/AliasSet/guards.profraw
./test/Analysis/AliasSet/memset.profraw
./test/Analysis/AliasSet/intrinsics.profraw

[delcypher]

llvm/docs/CommandGuide/lit.rst

188

You may want to rephrase this. Lit can (and is) used outside of the llvm project so the <build>/test directory isn't guaranteed to be the name of a test directory. Instead I think you're saying that the coverage data files will be located in the directory that config.test_exec_root is set to in a test suite.

A concrete example under the llvm-project itself would be clang. It's lit shell tests are usually located in <build>/tools/clang/test which is different to the path you give above.

llvm/utils/lit/lit/LitConfig.py

91

Consider adding a @property getter and a setter (see maxIndividualTestTime) so that when setting the data it can be validated.

llvm/utils/lit/lit/TestRunner.py

1147

How is this unique to every RUN line? A test case could have multiple run lines and I don't see how that would be handled here.

[delcypher]

@xgupta Could you add a test case where a lit.cfg file sets config.per_test_coverage directly? A test suite may want to set it there rather than on the command line.

[delcypher]

llvm/docs/CMake.rst

378

This documentation seems inaccurate right now. Reading the implementation, enabling this option doesn't actually cause any coverage to be collected. I suspect you will do this in later patches but I would suggest you update the documentation when the feature actually works.

Nits:

• LLVM_TEST_COVERAGE is a vague name. LLVM_INDIVIDUAL_TEST_COVERAGE might be more specific.
• Consider being more specific about where the code coverage data is written (e.g. under the config.test_exec_root directory.
• Documentation doesn't typically use the phrase "This allows you to...". Instead consider writing " This allows code coverage analysis of each individual test case"

Super nit:

• Missing fullstop (period) at the end of the last sentence.

[xgupta]

Address comments (Test case is not passing yet)

[xgupta]

@xgupta Could you add a test case where a lit.cfg file sets config.per_test_coverage directly? A test suite may want to set it there rather than on the command line.

I have added one test case but it is not passing, I am investigating that.

llvm/docs/CMake.rst

378

This documentation seems inaccurate right now. Reading the implementation, enabling this option doesn't actually cause any coverage to be collected. I suspect you will do this in later patches but I would suggest you update the documentation when the feature actually works.

Yeah, I was thinking of future name, agree, updated now, Thanks.

llvm/docs/CommandGuide/lit.rst

188

Thanks for pointing it, updated.

llvm/utils/lit/lit/TestRunner.py

1147

I updated the documentation to say it is set for each test file.

[xgupta]

Update TestRunner.py for each RUN rahter than per test case ( Testcase WIP)

llvm/utils/lit/lit/LitConfig.py

91

I added that but thinking about the use of that property as "echo_all_commands" don't have it. Can you please clarify?

llvm/utils/lit/lit/TestRunner.py

1147

Kept the documentation for each RUN and updated the code in that regard.

[xgupta]

Put export LLVM_PROFILE_FILE={llvm_profile_file} before the RUN command execute.

[xgupta]

Minor update

[xgupta]

remove test case

[xgupta]

Hi,

writing test case in llvm/utils/lit/tests seems difficult, as when we run them, there is no .script file generated as in case with other llvm or clang test cases. So the patch is not working there and the env variable is not setting. I am thinking if we can submit in this way without testcase since lit code is very stable so chances of breaking something is less.

shivam@linux ~/.llvm/llvm-upstream/build (arcpatch-D154280) $ bin/llvm-lit -a -v --per-test-coverage  ../clang/test/SemaCXX/bool.cpp

llvm-lit: /home/shivam/.llvm/llvm-upstream/llvm/utils/lit/lit/llvm/config.py:484: note: using clang: /home/shivam/.llvm/llvm-upstream/build/bin/clang
-- Testing: 1 tests, 1 workers --
PASS: Clang :: SemaCXX/bool.cpp (1 of 1)
Script:
--
export LLVM_PROFILE_FILE=bool0.profraw && : 'RUN: at line 1';   /home/shivam/.llvm/llvm-upstream/build/bin/clang -cc1 -internal-isystem /home/shivam/.llvm/llvm-upstream/build/lib/clang/17/include -nostdsysteminc -std=c++14 -fsyntax-only -verify=expected,precxx17 -Wno-constant-conversion /home/shivam/.llvm/llvm-upstream/clang/test/SemaCXX/bool.cpp
export LLVM_PROFILE_FILE=bool1.profraw && : 'RUN: at line 2';   /home/shivam/.llvm/llvm-upstream/build/bin/clang -cc1 -internal-isystem /home/shivam/.llvm/llvm-upstream/build/lib/clang/17/include -nostdsysteminc -std=c++14 -fsyntax-only -verify=expected,precxx17 -Wno-constant-conversion -Wno-deprecated -Wdeprecated-increment-bool /home/shivam/.llvm/llvm-upstream/clang/test/SemaCXX/bool.cpp
export LLVM_PROFILE_FILE=bool2.profraw && : 'RUN: at line 3';   /home/shivam/.llvm/llvm-upstream/build/bin/clang -cc1 -internal-isystem /home/shivam/.llvm/llvm-upstream/build/lib/clang/17/include -nostdsysteminc -std=c++2b -fsyntax-only -verify=expected,cxx17 -Wno-constant-conversion -Wno-deprecated -Wdeprecated-increment-bool /home/shivam/.llvm/llvm-upstream/clang/test/SemaCXX/bool.cpp
--
Exit Code: 0


********************

Testing Time: 0.76s
  Passed: 1
shivam@linux ~/.llvm/llvm-upstream/build (arcpatch-D154280) $ find . -type f -name "*.profraw" | grep -i "profraw"

./default.profraw
./tools/clang/test/SemaCXX/bool1.profraw
./tools/clang/test/SemaCXX/bool0.profraw
./tools/clang/test/SemaCXX/bool2.profraw

[hnrklssn]

In D154280#4519029, @xgupta wrote:

Hi,

writing test case in llvm/utils/lit/tests seems difficult, as when we run them, there is no .script file generated as in case with other llvm or clang test cases. So the patch is not working there and the env variable is not setting. I am thinking if we can submit in this way without testcase since lit code is very stable so chances of breaking something is less.

shivam@linux ~/.llvm/llvm-upstream/build (arcpatch-D154280) $ bin/llvm-lit -a -v --per-test-coverage  ../clang/test/SemaCXX/bool.cpp

llvm-lit: /home/shivam/.llvm/llvm-upstream/llvm/utils/lit/lit/llvm/config.py:484: note: using clang: /home/shivam/.llvm/llvm-upstream/build/bin/clang
-- Testing: 1 tests, 1 workers --
PASS: Clang :: SemaCXX/bool.cpp (1 of 1)
Script:
--
export LLVM_PROFILE_FILE=bool0.profraw && : 'RUN: at line 1';   /home/shivam/.llvm/llvm-upstream/build/bin/clang -cc1 -internal-isystem /home/shivam/.llvm/llvm-upstream/build/lib/clang/17/include -nostdsysteminc -std=c++14 -fsyntax-only -verify=expected,precxx17 -Wno-constant-conversion /home/shivam/.llvm/llvm-upstream/clang/test/SemaCXX/bool.cpp
export LLVM_PROFILE_FILE=bool1.profraw && : 'RUN: at line 2';   /home/shivam/.llvm/llvm-upstream/build/bin/clang -cc1 -internal-isystem /home/shivam/.llvm/llvm-upstream/build/lib/clang/17/include -nostdsysteminc -std=c++14 -fsyntax-only -verify=expected,precxx17 -Wno-constant-conversion -Wno-deprecated -Wdeprecated-increment-bool /home/shivam/.llvm/llvm-upstream/clang/test/SemaCXX/bool.cpp
export LLVM_PROFILE_FILE=bool2.profraw && : 'RUN: at line 3';   /home/shivam/.llvm/llvm-upstream/build/bin/clang -cc1 -internal-isystem /home/shivam/.llvm/llvm-upstream/build/lib/clang/17/include -nostdsysteminc -std=c++2b -fsyntax-only -verify=expected,cxx17 -Wno-constant-conversion -Wno-deprecated -Wdeprecated-increment-bool /home/shivam/.llvm/llvm-upstream/clang/test/SemaCXX/bool.cpp
--
Exit Code: 0


********************

Testing Time: 0.76s
  Passed: 1
shivam@linux ~/.llvm/llvm-upstream/build (arcpatch-D154280) $ find . -type f -name "*.profraw" | grep -i "profraw"

./default.profraw
./tools/clang/test/SemaCXX/bool1.profraw
./tools/clang/test/SemaCXX/bool0.profraw
./tools/clang/test/SemaCXX/bool2.profraw

It sounds like it’s because the test is not using external ShTest. Later it might be worth looking into supporting internal lit shell, but I haven’t checked what that would imply yet.

[xgupta]

Added test cases

[hnrklssn]

LGTM

[DavidSpickett]

llvm/utils/lit/lit/TestRunner.py

1087

This appears to get set but never used. Or I am missing something later in the code, if that's the case, consider making that more obvious e.g. declaring it outside the for loop.

[DavidSpickett]

Side note: I have been wanting to try some things where we run only the tests we think might fail after a given patch, and this could be very useful there. Thank you for working on this.

[DavidSpickett]

llvm/utils/lit/tests/Inputs/per-test-coverage-by-lit-cfg/per-test-coverage-by-lit-cfg.py

9

If I understand correctly, LLVM_PROFILE_FILE should be set by lit itself. So this test could be more robust if it did env -u LLVM_PROFILE_FILE python.... The new value for LLVM_PROFILE_FILE should be redefined by lit as part of the test.

[DavidSpickett]

llvm/utils/lit/tests/Inputs/per-test-coverage-by-lit-cfg/per-test-coverage-by-lit-cfg.py

9

Actually ignore me, doing that would remove the value set by LIT because lit is the one that runs the RUN line.

[xgupta]

update python to %{python} in test cases

[xgupta]

llvm/utils/lit/lit/TestRunner.py

1087

Actually, this variable is being set for each RUN line in a test case so it is in the loop.

[DavidSpickett]

llvm/utils/lit/lit/TestRunner.py

1087

That explains it being in the loop, but where is env actually used?

It makes a copy of the current environment, and sets a new value in it:

env = dict(test.config.environment)  # Create a copy of the environment
env["LLVM_PROFILE_FILE"] = llvm_profile_file

However it doesn't use env again, instead it sets llvm_profile_file directly:

commands[j] = f"export LLVM_PROFILE_FILE={llvm_profile_file} && {commands[j]}"
coverage_index += 1

Which seems correct, so is env actually unused?

If it's used later outside the loop it would be whatever the last env set, which would likely create some strange results.

[xgupta]

Updated lit.cfg of test cases for %{python}

[xgupta]

Don't set env variable

[xgupta]

llvm/utils/lit/lit/TestRunner.py

1087

Thanks, @DavidSpickett, yes you are correct, we are not using env variable in this code.
Actually that we were using it before when setting LLVM_PROFILE_FILE for each file (.script), but now it should be for each RUN line so use a different approach.

[DavidSpickett]

Were you able to reproduce the issue and check it was fixed, or do you need me to try it again to confirm?

[xgupta]

In D154280#4534490, @DavidSpickett wrote:

Were you able to reproduce the issue and check it was fixed, or do you need me to try it again to confirm?

Yes, I reproduce it on docker and checked it was fixed by replacing python to %{python} in test cases (Actually there is only python3, not python so it was failing).
I am waiting for pre-merge checks to be completed and then will reland it.

llvm/CMakeLists.txt

 endif()
 
 # Define the default arguments to use with 'lit', and an option for the user to
 # override.
 set(LIT_ARGS_DEFAULT "-sv")
 if (MSVC_IDE OR XCODE)
   set(LIT_ARGS_DEFAULT "${LIT_ARGS_DEFAULT} --no-progress-bar")
 endif()
+if(LLVM_INDIVIDUAL_TEST_COVERAGE)
+   set(LIT_ARGS_DEFAULT "${LIT_ARGS_DEFAULT} --per-test-coverage")
+endif()
 set(LLVM_LIT_ARGS "${LIT_ARGS_DEFAULT}" CACHE STRING "Default options for lit")
 
 # On Win32 hosts, provide an option to specify the path to the GnuWin32 tools.
 if( WIN32 AND NOT CYGWIN )
   set(LLVM_LIT_TOOLS_DIR "" CACHE PATH "Path to GnuWin32 tools")
 endif()
 set(LLVM_NATIVE_TOOL_DIR "" CACHE PATH "Path to a directory containing prebuilt matching native tools (such as llvm-tblgen)")
 

 if (CMAKE_BUILD_TYPE AND uppercase_CMAKE_BUILD_TYPE MATCHES "^(RELWITHDEBINFO|DEBUG)$")
   set(LLVM_OMIT_DAGISEL_COMMENTS OFF)
 endif()
 
 if (MSVC_IDE)
   option(LLVM_ADD_NATIVE_VISUALIZERS_TO_SOLUTION "Configure project to use Visual Studio native visualizers" TRUE)
 endif()
 
-if (LLVM_BUILD_INSTRUMENTED OR LLVM_BUILD_INSTRUMENTED_COVERAGE OR
-    LLVM_ENABLE_IR_PGO)
+if(NOT LLVM_INDIVIDUAL_TEST_COVERAGE)
+  if(LLVM_BUILD_INSTRUMENTED OR LLVM_BUILD_INSTRUMENTED_COVERAGE OR LLVM_ENABLE_IR_PGO)
     if(NOT LLVM_PROFILE_MERGE_POOL_SIZE)
-    # A pool size of 1-2 is probably sufficient on a SSD. 3-4 should be fine
-    # for spining disks. Anything higher may only help on slower mediums.
+      # A pool size of 1-2 is probably sufficient on an SSD. 3-4 should be fine
+      # for spinning disks. Anything higher may only help on slower mediums.
       set(LLVM_PROFILE_MERGE_POOL_SIZE "4")
     endif()
     if(NOT LLVM_PROFILE_FILE_PATTERN)
       if(NOT LLVM_PROFILE_DATA_DIR)
         file(TO_NATIVE_PATH "${LLVM_BINARY_DIR}/profiles" LLVM_PROFILE_DATA_DIR)
       endif()
       file(TO_NATIVE_PATH "${LLVM_PROFILE_DATA_DIR}/%${LLVM_PROFILE_MERGE_POOL_SIZE}m.profraw" LLVM_PROFILE_FILE_PATTERN)
     endif()
     if(NOT LLVM_CSPROFILE_FILE_PATTERN)
       if(NOT LLVM_CSPROFILE_DATA_DIR)
         file(TO_NATIVE_PATH "${LLVM_BINARY_DIR}/csprofiles" LLVM_CSPROFILE_DATA_DIR)
       endif()
       file(TO_NATIVE_PATH "${LLVM_CSPROFILE_DATA_DIR}/%${LLVM_PROFILE_MERGE_POOL_SIZE}m.profraw" LLVM_CSPROFILE_FILE_PATTERN)
     endif()
   endif()
+endif()
 
 if (LLVM_BUILD_STATIC)
   set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -static")
   # Remove shared library suffixes from use in find_library
   foreach (shared_lib_suffix ${CMAKE_SHARED_LIBRARY_SUFFIX} ${CMAKE_IMPORT_LIBRARY_SUFFIX})
     list(FIND CMAKE_FIND_LIBRARY_SUFFIXES ${shared_lib_suffix} shared_lib_suffix_idx)
     if(NOT ${shared_lib_suffix_idx} EQUAL -1)
       list(REMOVE_AT CMAKE_FIND_LIBRARY_SUFFIXES ${shared_lib_suffix_idx})

llvm/cmake/modules/HandleLLVMOptions.cmake

if(NOT LINKER_IS_LLD_LINK)
         CMAKE_SHARED_LINKER_FLAGS)
     endif()
   else()
     message(FATAL_ERROR "LLVM_PROFDATA_FILE can only be specified when compiling with clang")
   endif()
 endif()
 
 option(LLVM_BUILD_INSTRUMENTED_COVERAGE "Build LLVM and tools with Code Coverage instrumentation" Off)
+option(LLVM_INDIVIDUAL_TEST_COVERAGE "Emit individual coverage file for each test case." OFF)
 mark_as_advanced(LLVM_BUILD_INSTRUMENTED_COVERAGE)
 append_if(LLVM_BUILD_INSTRUMENTED_COVERAGE "-fprofile-instr-generate=\"${LLVM_PROFILE_FILE_PATTERN}\" -fcoverage-mapping"
   CMAKE_CXX_FLAGS
   CMAKE_C_FLAGS
   CMAKE_EXE_LINKER_FLAGS
   CMAKE_SHARED_LINKER_FLAGS)
 
 if (LLVM_BUILD_INSTRUMENTED AND LLVM_BUILD_INSTRUMENTED_COVERAGE)

llvm/docs/CMake.rst

**LLVM_CODE_COVERAGE_TARGETS**:STRING
   to drive the code coverage reports. If unset, the target list will be
   constructed using the LLVM build's CMake export list.
 
 **LLVM_COVERAGE_SOURCE_DIRS**:STRING
   If set to a semicolon separated list of directories, the coverage reports
   will limit code coverage summaries to just the listed directories. If unset,
   coverage reports will include all sources identified by the tooling.
 
+ **LLVM_INDIVIDUAL_TEST_COVERAGE**: BOOL
+  Enable individual test case coverage. When set to ON, code coverage data for
+  each test case will be generated and stored in a separate directory under the
+  config.test_exec_root path. This feature allows code coverage analysis of each
+  individual test case. Defaults to OFF.
+
 **LLVM_BUILD_LLVM_DYLIB**:BOOL
   If enabled, the target for building the libLLVM shared library is added.
   This library contains all of LLVM's components in a single shared library.
   Defaults to OFF. This cannot be used in conjunction with BUILD_SHARED_LIBS.
   Tools will only be linked to the libLLVM shared library if LLVM_LINK_LLVM_DYLIB
   is also ON.
   The components in the library can be customised by setting LLVM_DYLIB_COMPONENTS
   to a list of the desired components.

llvm/docs/CommandGuide/lit.rst

 `.lit_test_times.txt`. If this file does not exist, then `lit` checks the
 `test_source_root` for the file to optionally accelerate clean builds.
 
 .. option:: --shuffle
 
  Run the tests in a random order, not failing/slowest first. Deprecated,
  use :option:`--order` instead.
 
+.. option:: --per-test-coverage
+
+ Emit the necessary test coverage data, divided per test case (involves
+ setting a unique value to LLVM_PROFILE_FILE for each RUN). The coverage
+ data files will be emitted in the directory specified by `config.test_exec_root`.
+
 .. option:: --max-failures N
 
  Stop execution after the given number ``N`` of failures.
  An integer argument should be passed on the command line
  prior to execution.
 
 .. option:: --max-tests=N
 

llvm/utils/lit/lit/LitConfig.py

def __init__(
         debug,
         isWindows,
         order,
         params,
         config_prefix=None,
         maxIndividualTestTime=0,
         parallelism_groups={},
         echo_all_commands=False,
+        per_test_coverage=False,
     ):
         # The name of the test runner.
         self.progname = progname
         # The items to add to the PATH environment variable.
         self.path = [str(p) for p in path]
         self.quiet = bool(quiet)
         self.useValgrind = bool(useValgrind)
         self.valgrindLeakCheck = bool(valgrindLeakCheck)
):
             else:
                 # The default is 'summary'.
                 self.valgrindArgs.append("--leak-check=no")
             self.valgrindArgs.extend(self.valgrindUserArgs)
 
         self.maxIndividualTestTime = maxIndividualTestTime
         self.parallelism_groups = parallelism_groups
         self.echo_all_commands = echo_all_commands
+        self.per_test_coverage = per_test_coverage
 
     @property
     def maxIndividualTestTime(self):
         """
         Interface for getting maximum time to spend executing
         a single test
         """
         return self._maxIndividualTestTime
def maxIndividualTestTime(self, value):
             # a timeout per test. Check it's available.
             # See lit.util.killProcessAndChildren()
             supported, errormsg = self.maxIndividualTestTimeIsSupported
             if not supported:
                 self.fatal("Setting a timeout per test not supported. " + errormsg)
         elif self.maxIndividualTestTime < 0:
             self.fatal("The timeout per test must be >= 0 seconds")
 
+    @property
+    def per_test_coverage(self):
+        """
+        Interface for getting the per_test_coverage value
+        """
+        return self._per_test_coverage
+
+    @per_test_coverage.setter
+    def per_test_coverage(self, value):
+        """
+        Interface for setting the per_test_coverage value
+        """
+        if not isinstance(value, bool):
+            self.fatal("per_test_coverage must set to a value of type bool.")
+        self._per_test_coverage = value
+
     def load_config(self, config, path):
         """load_config(config, path) - Load a config object from an alternate
         path."""
         if self.debug:
             self.note("load_config from %r" % path)
         config.load_from_path(path, self)
         return config
 

llvm/utils/lit/lit/TestRunner.py

for i, result in enumerate(results):
             )
 
     return out, err, exitCode, timeoutInfo
 
 
 def executeScript(test, litConfig, tmpBase, commands, cwd):
     bashPath = litConfig.getBashPath()
     isWin32CMDEXE = litConfig.isWindows and not bashPath
+    coverage_index = 0  # Counter for coverage file index
     script = tmpBase + ".script"
     if isWin32CMDEXE:
         script += ".bat"
 
+    # Set unique LLVM_PROFILE_FILE for each run command
+    for j, ln in enumerate(commands):
+        match = re.match(kPdbgRegex, ln)
+        if match:
+            command = match.group(2)
+            commands[j] = match.expand(": '\\1'; \\2" if command else ": '\\1'")
+            if litConfig.per_test_coverage:
+                # Extract the test case name from the test object
+                test_case_name = test.path_in_suite[-1]
+                test_case_name = test_case_name.rsplit(".", 1)[0]  # Remove the file extension
+                llvm_profile_file = f"{test_case_name}{coverage_index}.profraw"
+                commands[j] = f"export LLVM_PROFILE_FILE={llvm_profile_file} && {commands[j]}"
+                coverage_index += 1
+
     # Write script file
     mode = "w"
     open_kwargs = {}
     if litConfig.isWindows and not isWin32CMDEXE:
         mode += "b"  # Avoid CRLFs when writing bash scripts.
     elif sys.version_info > (3, 0):
         open_kwargs["encoding"] = "utf-8"
     f = open(script, mode, **open_kwargs)
else:
             command = [bashPath, script]
         else:
             command = ["/bin/sh", script]
         if litConfig.useValgrind:
             # FIXME: Running valgrind on sh is overkill. We probably could just
             # run on clang with no real loss.
             command = litConfig.valgrindArgs + command
 
     try:
         out, err, exitCode = lit.util.executeCommand(
             command,
             cwd=cwd,
             env=test.config.environment,
             timeout=litConfig.maxIndividualTestTime,
         )
         return (out, err, exitCode, None)
     except lit.util.ExecuteCommandTimeoutException as e:
         return (e.out, e.err, e.exitCode, e.msg)
 
 
 def parseIntegratedTestScriptCommands(source_path, keywords):
     """
     parseIntegratedTestScriptCommands(source_path) -> commands
 
     Parse the commands in an integrated test script file into a list of
     (line_number, command_type, line).

llvm/utils/lit/lit/cl_arguments.py

execution_group.add_argument(
         type=_positive_int,
     )
     execution_group.add_argument(
         "--allow-empty-runs",
         help="Do not fail the run if all tests are filtered out",
         action="store_true",
     )
     execution_group.add_argument(
+        "--per-test-coverage",
+        dest="per_test_coverage",
+        action="store_true",
+        help="Enable individual test case coverage",
+    )
+    execution_group.add_argument(
         "--ignore-fail",
         dest="ignoreFail",
         action="store_true",
         help="Exit with status zero even if some tests fail",
     )
 
     selection_group = parser.add_argument_group("Test Selection")
     selection_group.add_argument(

llvm/utils/lit/lit/main.py

lit_config = lit.LitConfig.LitConfig(
         valgrindArgs=opts.valgrindArgs,
         noExecute=opts.noExecute,
         debug=opts.debug,
         isWindows=is_windows,
         order=opts.order,
         params=params,
         config_prefix=opts.configPrefix,
         echo_all_commands=opts.echoAllCommands,
+        per_test_coverage=opts.per_test_coverage,
     )
 
     discovered_tests = lit.discovery.find_tests_for_inputs(
         lit_config, opts.test_paths
     )
     if not discovered_tests:
         sys.stderr.write("error: did not discover any tests for provided path(s)\n")
         sys.exit(2)

llvm/utils/lit/tests/Inputs/per-test-coverage-by-lit-cfg/lit.cfg

+import lit.formats
+import os
+
+config.name = "per-test-coverage-by-lit-cfg"
+config.suffixes = [".py"]
+config.test_format = lit.formats.ShTest(execute_external=True)
+lit_config.per_test_coverage = True
+config.substitutions.append(("%{python}", '"%s"' % (sys.executable)))

llvm/utils/lit/tests/Inputs/per-test-coverage-by-lit-cfg/per-test-coverage-by-lit-cfg.py

+# Check that the environment variable is set correctly
+# RUN: %{python} %s | FileCheck %s
+
+# Python script to read the environment variable
+# and print its value
+import os
+
+llvm_profile_file = os.environ.get('LLVM_PROFILE_FILE')
+print(llvm_profile_file)
+
+# CHECK: per-test-coverage-by-lit-cfg0.profraw

llvm/utils/lit/tests/Inputs/per-test-coverage/lit.cfg

+import lit.formats
+import os
+
+config.name = "per-test-coverage"
+config.suffixes = [".py"]
+config.test_format = lit.formats.ShTest(execute_external=True)
+config.substitutions.append(("%{python}", '"%s"' % (sys.executable)))
+

llvm/utils/lit/tests/Inputs/per-test-coverage/per-test-coverage.py

+# Check that the environment variable is set correctly
+# RUN: %{python} %s | FileCheck %s
+
+# Python script to read the environment variable
+# and print its value
+import os
+
+llvm_profile_file = os.environ.get('LLVM_PROFILE_FILE')
+print(llvm_profile_file)
+
+# CHECK: per-test-coverage0.profraw

llvm/utils/lit/tests/per-test-coverage-by-lit-cfg.py

+# Test if lit_config.per_test_coverage in lit.cfg sets individual test case coverage.
+
+# RUN: %{lit} -a -v %{inputs}/per-test-coverage-by-lit-cfg/per-test-coverage-by-lit-cfg.py \
+# RUN: | FileCheck -match-full-lines %s
+#
+# CHECK: PASS: per-test-coverage-by-lit-cfg :: per-test-coverage-by-lit-cfg.py ({{[^)]*}})

llvm/utils/lit/tests/per-test-coverage.py

+# Test LLVM_PROFILE_FILE is set when --per-test-coverage is passed to command line.
+
+# RUN: %{lit} -a -v --per-test-coverage %{inputs}/per-test-coverage/per-test-coverage.py \
+# RUN: | FileCheck -match-full-lines %s
+#
+# CHECK: PASS: per-test-coverage :: per-test-coverage.py ({{[^)]*}})