A new hidden option exec-on-ir-change=exe that calls exe after each time IR changes

[jamieschmeiser]

Summary

A new option exec-on-ir-changed is defined that allows one to specify an
exe that is called after each pass in the opt pipeline that changes the IR.

I recently was investigating a situation where some IR was being
mis-optimized in the IR pipeline and I needed to determine after
which pass the IR was no longer producing the correct results.
In order to determine this, I created the following option which calls
to an external exe (or script) each time the IR changes in the opt pipeline.
In my case, the exe was a script that called llc to compile the IR and ran
the result so that I could determine when it became incorrect. This is
a generally useful technique for debugging optimization bugs.

The exec-on-ir-change=exe option saves the IR in a temporary file and calls exe
with the name of the file and the name of the pass that just changed it after
each pass alters the IR. exe is also called with the initial IR. This
can be used, for example, to determine which pass corrupts the IR by having
exe as a script that calls llc and runs a test to see after which pass the
results change. The print-changed filtering options are respected.

[aeubanks]

super nit, the title doesn't need to specify that the option is "hidden", that is definitely not important enough to be in the title

I think this could be useful, although I believe opt-bisect would also work for your use case

llvm/lib/Passes/StandardInstrumentations.cpp

73

This is way too simple of a function to use templates. They're very hard to read and are bad for compile times.
I'd say to avoid C arrays and use SmallVector/ArrayRef whenever possible.

[jamieschmeiser]

Respond to review comments: remove templates in functions for temporary file creation and cleanup.

[jamieschmeiser]

ping

[aeubanks]

looks mostly good

llvm/lib/Passes/StandardInstrumentations.cpp

104

these should probably be returning llvm::Error instead of std::string

[jamieschmeiser]

llvm/lib/Passes/StandardInstrumentations.cpp

104

Is llvm::Error applicable? This isn't an error in user code but rather a failure in debug output and the error text returned is just reported as debug information.

[aeubanks]

llvm/lib/Passes/StandardInstrumentations.cpp

104

llvm::Error is better typing than using a empty vs non-empty string, and it enforces that you check the return value
but doesn't really matter that much

[thakis]

Looks like this breaks tests on Windows: http://45.33.8.238/win/50453/step_11.txt

Please take a look, and revert for now if it takes a while to fix.

[aeubanks]

I've reverted this for now. We probably shouldn't be using /bin/sh in tests. $ rg /bin/sh llvm/test shows that it's not really a thing

[jamieschmeiser]

@aeubanks Thanks for reverting it. I'll take a look.

[jamieschmeiser]

Replace the shell script with just calling cat directly. cat is used in many lit tests and will cat the IR file and will give an error that no file exists with the name of the pass and this can be checked. @thakis, can you please verify that this does not break the windows build?

[jrtc27]

Why not just use bugpoint? It'll bisect the pipeline and the IR itself to give you the reduced IR+misbehaving opt pass.

[jamieschmeiser]

Yes, bugpoint will find a misbehaving class but that is just one example usage for this. test-changed is more general in that any script/exe can be called each time the IR changes. Consider the situation where a performance degradation is noticed so the code still compiles but is not as well optimized. bugpoint (as I understand from the documation) will not detect this because the IR still compiles and the exe still runs. Using test-changed and a script that calls llc followed by a timed execution, one could determine how much impact the various passes have on the performance of the code as it gets changed in the pipeline. This could point out the source(s) of speed-ups and slow-downs in the pipeline and could be used as an exploration tool.

[jrtc27]

You can use a custom run script that does whatever you like with bugpoint

[aeubanks]

bugpoint doesn't work with the new pass manager so it's effectively deprecated

[jamieschmeiser]

renamed option to -exec-on-ir-change

[jamieschmeiser]

It was suggested by @reames that -exec-on-ir-change=exe is a better name for the option. This sounds like a good suggestion to me so I have renamed it.

[jamieschmeiser]

Failing build: https://lab.llvm.org/buildbot/#/builders/42/builds/2800

[jamieschmeiser]

@aeubanks When I first reverted this because it failed on windows, I wasn't sure what to do to fix it and it fell off my radar.
I have updated it to reflect changes since then (the tests were updated to reflect changes to specifying filtering, functions have moved, etc). I've made a few minor changes, such as the functions for managing the temp files return Optional strings now and the tempfile name is different to reflect the multiple usage. I've also added a lit.local.cfg file to ensure that '/bin/cat' exists, which should fix the original windows failure that caused the revert. Can you please take a look to see that everything is still okay? Thanks.

[aeubanks]

one coding comment

llvm/lib/IR/PrintPasses.cpp

159–177

this (and below) should use ErrorOr

[jamieschmeiser]

ErrorOr isn't really appropriate here because the string was produced when the error occurred and it was None otherwise, which is the opposite of what happens with ErrorOr. It isn't desirable to create the array of filenames using ErrorOr since you want to reuse the existing ones or a large trace will open too many temporary files. However, your comment does point out that this return is inconsistent and awkward. I've changed these functions to return an std::error_code, which is more consistent. It loses the different error strings previously produced, but this code is only used for debugging anyway. Also, I changed the function for creating the temporary files to clean up any it succeeded in creating if an error occurs.

[aeubanks]

lg

llvm/include/llvm/IR/PrintPasses.h

 
 // Return true if -filter-passes is empty or contains the pass name.
 bool isPassInPrintList(StringRef PassName);
 bool isFilterPassesEmpty();
 
 // Returns true if we should print the function.
 bool isFunctionInPrintList(StringRef FunctionName);
 
+// Ensure temporary files exist, creating or re-using them.  \p FD contains
+// file descriptors (-1 indicates that the file should be created) and
+// \p SR contains the corresponding initial content.  \p FileName will have
+// the filenames filled in when creating files.  Return first error code (if
+// any) and stop.
+std::error_code prepareTempFiles(SmallVector<int> &FD, ArrayRef<StringRef> SR,
+                                 SmallVector<std::string> &FileName);
+
+// Remove the temporary files in \p FileName.  Typically used in conjunction
+// with prepareTempFiles.  Return first error code (if any) and stop..
+std::error_code cleanUpTempFiles(ArrayRef<std::string> FileName);
+
 // Perform a system based diff between \p Before and \p After, using \p
 // OldLineFormat, \p NewLineFormat, and \p UnchangedLineFormat to control the
 // formatting of the output. Return an error message for any failures instead
 // of the diff.
 std::string doSystemDiff(StringRef Before, StringRef After,
                          StringRef OldLineFormat, StringRef NewLineFormat,
                          StringRef UnchangedLineFormat);
 
 } // namespace llvm
 
 #endif // LLVM_IR_PRINTPASSES_H

llvm/include/llvm/Passes/StandardInstrumentations.h

protected:
   void generateIRRepresentation(Any IR, StringRef PassID,
                                 std::string &Output) override;
   // Called when an interesting IR has changed.
   void handleAfter(StringRef PassID, std::string &Name,
                    const std::string &Before, const std::string &After,
                    Any) override;
 };
 
+class IRChangedTester : public IRChangedPrinter {
+public:
+  IRChangedTester() : IRChangedPrinter(true) {}
+  ~IRChangedTester() override;
+  void registerCallbacks(PassInstrumentationCallbacks &PIC);
+
+protected:
+  void handleIR(const std::string &IR, StringRef PassID);
+
+  // Check initial IR
+  void handleInitialIR(Any IR) override;
+  // Do nothing.
+  void omitAfter(StringRef PassID, std::string &Name) override;
+  // Do nothing.
+  void handleInvalidated(StringRef PassID) override;
+  // Do nothing.
+  void handleFiltered(StringRef PassID, std::string &Name) override;
+  // Do nothing.
+  void handleIgnored(StringRef PassID, std::string &Name) override;
+
+  // Call test as interesting IR has changed.
+  void handleAfter(StringRef PassID, std::string &Name,
+                   const std::string &Before, const std::string &After,
+                   Any) override;
+};
+
 // Information that needs to be saved for a basic block in order to compare
 // before and after the pass to determine if it was changed by a pass.
 template <typename T> class BlockDataT {
 public:
   BlockDataT(const BasicBlock &B) : Label(B.getName().str()), Data(B) {
     raw_string_ostream SS(Body);
     B.print(SS, nullptr, true, true);
   }
class StandardInstrumentations {
   OptNoneInstrumentation OptNone;
   OptPassGateInstrumentation OptPassGate;
   PreservedCFGCheckerInstrumentation PreservedCFGChecker;
   IRChangedPrinter PrintChangedIR;
   PseudoProbeVerifier PseudoProbeVerification;
   InLineChangePrinter PrintChangedDiff;
   DotCfgChangeReporter WebsiteChangeReporter;
   PrintCrashIRInstrumentation PrintCrashIR;
+  IRChangedTester ChangeTester;
   VerifyInstrumentation Verify;
 
   bool VerifyEach;
 
 public:
   StandardInstrumentations(LLVMContext &Context, bool DebugLogging,
                            bool VerifyEach = false,
                            PrintPassOptions PrintPassOpts = PrintPassOptions());

llvm/lib/IR/PrintPasses.cpp

 //===- PrintPasses.cpp ----------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/IR/PrintPasses.h"
 #include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Errc.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Program.h"
 #include <unordered_set>
 
 using namespace llvm;
 
 // Print IR out before/after specified passes.

 
 bool llvm::isFunctionInPrintList(StringRef FunctionName) {
   static std::unordered_set<std::string> PrintFuncNames(PrintFuncsList.begin(),
                                                         PrintFuncsList.end());
   return PrintFuncNames.empty() ||
          PrintFuncNames.count(std::string(FunctionName));
 }
 
-std::string llvm::doSystemDiff(StringRef Before, StringRef After,
-                               StringRef OldLineFormat, StringRef NewLineFormat,
-                               StringRef UnchangedLineFormat) {
-  StringRef SR[2]{Before, After};
-  // Store the 2 bodies into temporary files and call diff on them
-  // to get the body of the node.
-  const unsigned NumFiles = 3;
-  static std::string FileName[NumFiles];
-  static int FD[NumFiles]{-1, -1, -1};
-  for (unsigned I = 0; I < NumFiles; ++I) {
+std::error_code cleanUpTempFilesImpl(ArrayRef<std::string> FileName,
+                                     unsigned N) {
+  std::error_code RC;
+  for (unsigned I = 0; I < N; ++I) {
+    std::error_code EC = sys::fs::remove(FileName[I]);
+    if (EC)
+      RC = EC;
+  }
+  return RC;
+}
+
+std::error_code llvm::prepareTempFiles(SmallVector<int> &FD,
+                                       ArrayRef<StringRef> SR,
+                                       SmallVector<std::string> &FileName) {
+  assert(FD.size() >= SR.size() && FileName.size() == FD.size() &&
+         "Unexpected array sizes");
+  std::error_code EC;
+  unsigned I = 0;
+  for (; I < FD.size(); ++I) {
     if (FD[I] == -1) {
       SmallVector<char, 200> SV;
-      std::error_code EC =
-          sys::fs::createTemporaryFile("tmpdiff", "txt", FD[I], SV);
+      EC = sys::fs::createTemporaryFile("tmpfile", "txt", FD[I], SV);
       if (EC)
-        return "Unable to create temporary file.";
+        break;
       FileName[I] = Twine(SV).str();
     }
-    // The third file is used as the result of the diff.
-    if (I == NumFiles - 1)
-      break;
-
-    std::error_code EC = sys::fs::openFileForWrite(FileName[I], FD[I]);
+    if (I < SR.size()) {
+      EC = sys::fs::openFileForWrite(FileName[I], FD[I]);
       if (EC)
-      return "Unable to open temporary file for writing.";
-
+        break;
       raw_fd_ostream OutStream(FD[I], /*shouldClose=*/true);
-    if (FD[I] == -1)
-      return "Error opening file for writing.";
+      if (FD[I] == -1) {
+        EC = make_error_code(errc::io_error);
+        break;
+      }
       OutStream << SR[I];
     }
+  }
+  if (EC && I > 0)
+    // clean up created temporary files
+    cleanUpTempFilesImpl(FileName, I);
+  return EC;
+}
+
+std::error_code llvm::cleanUpTempFiles(ArrayRef<std::string> FileName) {
+  return cleanUpTempFilesImpl(FileName, FileName.size());
+}
+
+std::string llvm::doSystemDiff(StringRef Before, StringRef After,
+                               StringRef OldLineFormat, StringRef NewLineFormat,
+                               StringRef UnchangedLineFormat) {
+  // Store the 2 bodies into temporary files and call diff on them
+  // to get the body of the node.
+  static SmallVector<int> FD{-1, -1, -1};
+  SmallVector<StringRef> SR{Before, After};
+  static SmallVector<std::string> FileName{"", "", ""};
+  if (auto Err = prepareTempFiles(FD, SR, FileName))
+    return "Unable to create temporary file.";
 
   static ErrorOr<std::string> DiffExe = sys::findProgramByName(DiffBinary);
   if (!DiffExe)
     return "Unable to find diff executable.";
 
   SmallString<128> OLF, NLF, ULF;
   ("--old-line-format=" + OldLineFormat).toVector(OLF);
   ("--new-line-format=" + NewLineFormat).toVector(NLF);
   ("--unchanged-line-format=" + UnchangedLineFormat).toVector(ULF);
 
   StringRef Args[] = {DiffBinary, "-w", "-d",        OLF,
                       NLF,        ULF,  FileName[0], FileName[1]};
   std::optional<StringRef> Redirects[] = {std::nullopt, StringRef(FileName[2]),
                                           std::nullopt};
   int Result = sys::ExecuteAndWait(*DiffExe, Args, std::nullopt, Redirects);
   if (Result < 0)
     return "Error executing system diff.";
   std::string Diff;
   auto B = MemoryBuffer::getFile(FileName[2]);
   if (B && *B)
     Diff = (*B)->getBuffer().str();
   else
     return "Unable to read result.";
 
-  // Clean up.
-  for (const std::string &I : FileName) {
-    std::error_code EC = sys::fs::remove(I);
-    if (EC)
-      return "Unable to remove temporary file.";
-  }
+  if (auto Err = cleanUpTempFiles(FileName))
+
   return Diff;
 }

llvm/lib/Passes/StandardInstrumentations.cpp

 // An option for specifying the dot used by
 // print-changed=[dot-cfg | dot-cfg-quiet]
 static cl::opt<std::string>
     DotBinary("print-changed-dot-path", cl::Hidden, cl::init("dot"),
               cl::desc("system dot used by change reporters"));
 
 // An option that determines the colour used for elements that are only
 // in the before part.  Must be a colour named in appendix J of
 // https://graphviz.org/pdf/dotguide.pdf
 static cl::opt<std::string>
     BeforeColour("dot-cfg-before-color",
                  cl::desc("Color for dot-cfg before elements"), cl::Hidden,
                  cl::init("red"));
 // An option that determines the colour used for elements that are only
 // in the after part.  Must be a colour named in appendix J of
 // https://graphviz.org/pdf/dotguide.pdf
 static cl::opt<std::string>
static cl::opt<std::string> DotCfgDir(
     "dot-cfg-dir",
     cl::desc("Generate dot files into specified directory for changed IRs"),
     cl::Hidden, cl::init("./"));
 
 // An option to print the IR that was being processed when a pass crashes.
 static cl::opt<bool>
     PrintCrashIR("print-on-crash",
                  cl::desc("Print the last form of the IR before crash"),
                  cl::Hidden);
 
 static cl::opt<std::string> OptBisectPrintIRPath(
     "opt-bisect-print-ir-path",
     cl::desc("Print IR to path when opt-bisect-limit is reached"), cl::Hidden);
 
 namespace {
 
+// An option for specifying an executable that will be called with the IR
+// everytime it changes in the opt pipeline.  It will also be called on
+// the initial IR as it enters the pipeline.  The executable will be passed
+// the name of a temporary file containing the IR and the PassID.  This may
+// be used, for example, to call llc on the IR and run a test to determine
+// which pass makes a change that changes the functioning of the IR.
+// The usual modifier options work as expected.
+static cl::opt<std::string>
+    TestChanged("exec-on-ir-change", cl::Hidden, cl::init(""),
+                cl::desc("exe called with module IR after each pass that "
+                         "changes it"));
+
 /// Extract Module out of \p IR unit. May return nullptr if \p IR does not match
 /// certain global filters. Will never return nullptr if \p Force is true.
 const Module *unwrapModule(Any IR, bool Force = false) {
   if (any_isa<const Module *>(IR))
     return any_cast<const Module *>(IR);
 
   if (any_isa<const Function *>(IR)) {
     const Function *F = any_cast<const Function *>(IR);
void IRChangedPrinter::handleAfter(StringRef PassID, std::string &Name,
   if (After.empty()) {
     Out << "*** IR Deleted After " << PassID << " on " << Name << " ***\n";
     return;
   }
 
   Out << "*** IR Dump After " << PassID << " on " << Name << " ***\n" << After;
 }
 
+IRChangedTester::~IRChangedTester() {}
+
+void IRChangedTester::registerCallbacks(PassInstrumentationCallbacks &PIC) {
+  if (TestChanged != "")
+    TextChangeReporter<std::string>::registerRequiredCallbacks(PIC);
+}
+
+void IRChangedTester::handleIR(const std::string &S, StringRef PassID) {
+  // Store the body into a temporary file
+  static SmallVector<int> FD{-1};
+  SmallVector<StringRef> SR{S};
+  static SmallVector<std::string> FileName{""};
+  if (auto Err = prepareTempFiles(FD, SR, FileName)) {
+    dbgs() << "Unable to create temporary file.";
+    return;
+  }
+  static ErrorOr<std::string> Exe = sys::findProgramByName(TestChanged);
+  if (!Exe) {
+    dbgs() << "Unable to find test-changed executable.";
+    return;
+  }
+
+  StringRef Args[] = {TestChanged, FileName[0], PassID};
+  int Result = sys::ExecuteAndWait(*Exe, Args);
+  if (Result < 0) {
+    dbgs() << "Error executing test-changed executable.";
+    return;
+  }
+
+  if (auto Err = cleanUpTempFiles(FileName))
+    dbgs() << "Unable to remove temporary file.";
+}
+
+void IRChangedTester::handleInitialIR(Any IR) {
+  // Always test the initial module.
+  // Unwrap and print directly to avoid filtering problems in general routines.
+  std::string S;
+  generateIRRepresentation(IR, "Initial IR", S);
+  handleIR(S, "Initial IR");
+}
+
+void IRChangedTester::omitAfter(StringRef PassID, std::string &Name) {}
+void IRChangedTester::handleInvalidated(StringRef PassID) {}
+void IRChangedTester::handleFiltered(StringRef PassID, std::string &Name) {}
+void IRChangedTester::handleIgnored(StringRef PassID, std::string &Name) {}
+void IRChangedTester::handleAfter(StringRef PassID, std::string &Name,
+                                  const std::string &Before,
+                                  const std::string &After, Any) {
+  handleIR(After, PassID);
+}
+
 template <typename T>
 void OrderedChangedData<T>::report(
     const OrderedChangedData &Before, const OrderedChangedData &After,
     function_ref<void(const T *, const T *)> HandlePair) {
   const auto &BFD = Before.getData();
   const auto &AFD = After.getData();
   std::vector<std::string>::const_iterator BI = Before.getOrder().begin();
   std::vector<std::string>::const_iterator BE = Before.getOrder().end();
void StandardInstrumentations::registerCallbacks(
   if (FAM)
     PreservedCFGChecker.registerCallbacks(PIC, *FAM);
   PrintChangedIR.registerCallbacks(PIC);
   PseudoProbeVerification.registerCallbacks(PIC);
   if (VerifyEach)
     Verify.registerCallbacks(PIC);
   PrintChangedDiff.registerCallbacks(PIC);
   WebsiteChangeReporter.registerCallbacks(PIC);
+
+  ChangeTester.registerCallbacks(PIC);
+
   PrintCrashIR.registerCallbacks(PIC);
   // TimeProfiling records the pass running time cost.
   // Its 'BeforePassCallback' can be appended at the tail of all the
   // BeforeCallbacks by calling `registerCallbacks` in the end.
   // Its 'AfterPassCallback' is put at the front of all the
   // AfterCallbacks by its `registerCallbacks`. This is necessary
   // to ensure that other callbacks are not included in the timings.
   TimeProfilingPasses.registerCallbacks(PIC);

llvm/test/Other/ChangeTesters/exec-on-ir-change.ll

+; Simple checks of -exec-on-ir-change=cat functionality
+;
+; Simple functionality check.
+; RUN: opt -S -exec-on-ir-change=cat -passes=instsimplify 2>&1 -o /dev/null < %s | FileCheck %s --check-prefix=CHECK-SIMPLE
+;
+; Check that only the passes that change the IR are printed and that the
+; others (including g) are filtered out.
+; RUN: opt -S -exec-on-ir-change=cat -passes=instsimplify -filter-print-funcs=f  2>&1 -o /dev/null < %s | FileCheck %s --check-prefix=CHECK-FUNC-FILTER
+;
+; Check that the reporting of IRs respects -print-module-scope
+; RUN: opt -S -exec-on-ir-change=cat -passes=instsimplify -print-module-scope 2>&1 -o /dev/null < %s | FileCheck %s --check-prefix=CHECK-PRINT-MOD-SCOPE
+;
+; Check that the reporting of IRs respects -print-module-scope
+; RUN: opt -S -exec-on-ir-change=cat -passes=instsimplify -filter-print-funcs=f -print-module-scope 2>&1 -o /dev/null < %s | FileCheck %s --check-prefix=CHECK-FUNC-FILTER-MOD-SCOPE
+;
+; Check that reporting of multiple functions happens
+; RUN: opt -S -exec-on-ir-change=cat -passes=instsimplify -filter-print-funcs="f,g" 2>&1 -o /dev/null < %s | FileCheck %s --check-prefix=CHECK-FILTER-MULT-FUNC
+;
+; Check that the reporting of IRs respects -filter-passes
+; RUN: opt -S -exec-on-ir-change=cat -passes="instsimplify,no-op-function" -filter-passes="no-op-function" 2>&1 -o /dev/null < %s | FileCheck %s --check-prefix=CHECK-FILTER-PASSES
+;
+; Check that the reporting of IRs respects -filter-passes with multiple passes
+; RUN: opt -S -exec-on-ir-change=cat -passes="instsimplify,no-op-function" -filter-passes="no-op-function,instsimplify" 2>&1 -o /dev/null < %s | FileCheck %s --check-prefix=CHECK-FILTER-MULT-PASSES
+;
+; Check that the reporting of IRs respects both -filter-passes and -filter-print-funcs
+; RUN: opt -S -exec-on-ir-change=cat -passes="instsimplify,no-op-function" -filter-passes="no-op-function,instsimplify" -filter-print-funcs=f 2>&1 -o /dev/null < %s | FileCheck %s --check-prefix=CHECK-FILTER-FUNC-PASSES
+;
+; Check that the reporting of IRs respects -filter-passes, -filter-print-funcs and -print-module-scope
+; RUN: opt -S -exec-on-ir-change=cat -passes="instsimplify,no-op-function" -filter-passes="no-op-function,instsimplify" -filter-print-funcs=f -print-module-scope 2>&1 -o /dev/null < %s | FileCheck %s --check-prefix=CHECK-FILTER-FUNC-PASSES-MOD-SCOPE
+;
+; Check that repeated passes that change the IR are printed and that the
+; others (including g) are filtered out.  Note that the second time
+; instsimplify is run on f, it does not change the IR
+; RUN: opt -S -exec-on-ir-change=cat -passes="instsimplify,instsimplify" -filter-print-funcs=f  2>&1 -o /dev/null < %s | FileCheck %s --check-prefix=CHECK-MULT-PASSES-FILTER-FUNC
+;
+
+define i32 @g() {
+entry:
+  %a = add i32 2, 3
+  ret i32 %a
+}
+
+define i32 @f() {
+entry:
+  %a = add i32 2, 3
+  ret i32 %a
+}
+
+; CHECK-SIMPLE: ; ModuleID = {{.+}}
+; CHECK-SIMPLE: cat:{{.*}}Initial IR
+; CHECK-SIMPLE: define i32 @g()
+; CHECK-SIMPLE: cat:{{.*}}InstSimplifyPass
+; CHECK-SIMPLE: define i32 @f()
+; CHECK-SIMPLE: cat:{{.*}}InstSimplifyPass
+
+; CHECK-FUNC-FILTER: define i32 @f()
+; CHECK-FUNC-FILTER: cat:{{.*}}Initial IR
+; CHECK-FUNC-FILTER: define i32 @f()
+; CHECK-FUNC-FILTER: cat:{{.*}}InstSimplifyPass
+
+; CHECK-PRINT-MOD-SCOPE: ModuleID = {{.+}}
+; CHECK-PRINT-MOD-SCOPE: cat:{{.*}}Initial IR
+; CHECK-PRINT-MOD-SCOPE: ModuleID = {{.+}}
+; CHECK-PRINT-MOD-SCOPE: cat:{{.*}}InstSimplifyPass
+; CHECK-PRINT-MOD-SCOPE: ModuleID = {{.+}}
+; CHECK-PRINT-MOD-SCOPE: cat:{{.*}}InstSimplifyPass
+
+; CHECK-FUNC-FILTER-MOD-SCOPE: ; ModuleID = {{.+}}
+; CHECK-FUNC-FILTER-MOD-SCOPE: cat:{{.*}}Initial IR
+; CHECK-FUNC-FILTER-MOD-SCOPE: ModuleID = {{.+}}
+; CHECK-FUNC-FILTER-MOD-SCOPE: cat:{{.*}}InstSimplifyPass
+
+; CHECK-FILTER-MULT-FUNC: define i32 @g()
+; CHECK-FILTER-MULT-FUNC: cat:{{.*}}Initial IR
+; CHECK-FILTER-MULT-FUNC: define i32 @g()
+; CHECK-FILTER-MULT-FUNC: cat:{{.*}}InstSimplifyPass
+; CHECK-FILTER-MULT-FUNC: define i32 @f()
+; CHECK-FILTER-MULT-FUNC: cat:{{.*}}InstSimplifyPass
+
+; CHECK-FILTER-PASSES: define i32 @g()
+; CHECK-FILTER-PASSES: cat:{{.*}}Initial IR
+
+; CHECK-FILTER-MULT-PASSES: define i32 @g()
+; CHECK-FILTER-MULT-PASSES: cat:{{.*}}Initial IR
+; CHECK-FILTER-MULT-PASSES: define i32 @g()
+; CHECK-FILTER-MULT-PASSES: cat:{{.*}}InstSimplifyPass
+; CHECK-FILTER-MULT-PASSES: define i32 @f()
+; CHECK-FILTER-MULT-PASSES: cat:{{.*}}InstSimplifyPass
+
+; CHECK-FILTER-FUNC-PASSES: define i32 @f()
+; CHECK-FILTER-FUNC-PASSES: cat:{{.*}}Initial IR
+; CHECK-FILTER-FUNC-PASSES: define i32 @f()
+; CHECK-FILTER-FUNC-PASSES: cat:{{.*}}InstSimplifyPass
+
+; CHECK-FILTER-FUNC-PASSES-MOD-SCOPE: ; ModuleID = {{.+}}
+; CHECK-FILTER-FUNC-PASSES-MOD-SCOPE: cat:{{.*}}Initial IR
+; CHECK-FILTER-FUNC-PASSES-MOD-SCOPE: ModuleID = {{.+}}
+; CHECK-FILTER-FUNC-PASSES-MOD-SCOPE: cat:{{.*}}InstSimplifyPass
+
+; CHECK-MULT-PASSES-FILTER-FUNC: define i32 @f()
+; CHECK-MULT-PASSES-FILTER-FUNC: cat:{{.*}}Initial IR
+; CHECK-MULT-PASSES-FILTER-FUNC: define i32 @f()
+; CHECK-MULT-PASSES-FILTER-FUNC: cat:{{.*}}InstSimplifyPass

llvm/test/Other/ChangeTesters/lit.local.cfg

+if not os.path.exists('/bin/cat'):
+  config.unsupported = True