[Bugpoint redesign] Added option to run only certain delta passes

[diegotf]

Summary

This diff has been implemented to make llvm-reduce tests modular. Before, all the delta passes had to be run, thus interesting-ness tests had to add breaks so other passes didn't remove to-be-tested items.

It should be noted that this option is hidden, therefore it should only be accessed by tests!

[dblaikie]

I still feel like it'd probably be better to try to come up with some reliable/simple-enough idioms in the interestingness tests - rather than having more flags for the tool (even if they're only intended for test purposes).

I understand golden files in tests are usually not a great idea - but what would happen if the interestingness test was an exact IR file of the expected output, I wonder? Specifically the purpose being to demonstrate whether the reduction tool can get from the input to the output. Unlike other golden file situations that become hard to maintain as the tool changes its behavior, etc, in llvm-reduce the goal is that no matter how much better/different the tool gets, it can still manage every specific mapping from input to output.

[diegotf]

In D66110#1630478, @dblaikie wrote:

I still feel like it'd probably be better to try to come up with some reliable/simple-enough idioms in the interestingness tests - rather than having more flags for the tool (even if they're only intended for test purposes).

I understand golden files in tests are usually not a great idea - but what would happen if the interestingness test was an exact IR file of the expected output, I wonder? Specifically the purpose being to demonstrate whether the reduction tool can get from the input to the output. Unlike other golden file situations that become hard to maintain as the tool changes its behavior, etc, in llvm-reduce the goal is that no matter how much better/different the tool gets, it can still manage every specific mapping from input to output.

Fair enough. My main focus was simplifying the interesting-ness tests, but I can't argue with the last point. I'll go ahead and close this diff

llvm/test/Reduce/Inputs/remove-args.py

 #!/usr/bin/env python
 
 import sys
 
 UsedArgumentPresent = False
-# So ReduceFunctions pass doesn't remove anything
-FunctionCount = 0
 
 input = open(sys.argv[1], "r")
 for line in input:
   if "%used" in line:
-    UsedArgumentPresent = True
-  if "define" in line:
-    FunctionCount += 1
-
-if UsedArgumentPresent and FunctionCount == 3:
     sys.exit(0) # Interesting!
-else:
-  sys.exit(1)
+

llvm/test/Reduce/remove-args.ll

 ; Test that llvm-reduce can remove uninteresting function arguments from function definitions as well as their calls.
 ; Note: This pass also removes InstAllocs of uninteresting args
 ;
-; RUN: llvm-reduce --test %p/Inputs/remove-args.py %s
+; RUN: llvm-reduce --test %p/Inputs/remove-args.py %s --run-only args
 ; RUN: cat reduced.ll | FileCheck %s
 ; REQUIRES: plugins
 
 @x = global i32 0, align 4
 
 ; CHECK: %used
 ; CHECK-NOT: %unused
 define void @func(i32* dereferenceable(4) %used, i32 %unused, i32* dereferenceable(4) %unused2) {

llvm/test/Reduce/remove-funcs.ll

 ; Test that llvm-reduce can remove uninteresting functions as well as
 ; their InstCalls.
 ;
-; RUN: llvm-reduce --test %p/Inputs/remove-funcs.py %s
+; RUN: llvm-reduce --test %p/Inputs/remove-funcs.py %s --run-only funcs
 ; RUN: cat reduced.ll | FileCheck %s
 ; REQUIRES: plugins
 
 ; CHECK-NOT: uninteresting1()
 define i32 @uninteresting1() {
 entry:
   ret i32 0
 }

llvm/test/Reduce/remove-global-vars.ll

 ; Test that llvm-reduce can remove uninteresting Global Variables as well as
 ; their direct uses.
 ;
-; RUN: llvm-reduce --test %p/Inputs/remove-global-vars.py %s
+; RUN: llvm-reduce --test %p/Inputs/remove-global-vars.py %s --run-only globals
 ; RUN: cat reduced.ll | FileCheck %s
 ; REQUIRES: plugins
 
 @uninteresting1 = global i32 0, align 4
 ; CHECK: @interesting = global
 @interesting = global i32 5, align 4
 ; CHECK-NOT: global
 @uninteresting2 = global i32 25, align 4

llvm/test/Reduce/remove-metadata.ll

 ; Test that llvm-reduce can remove uninteresting metadata from an IR file.
 ; The Metadata pass erases named & unnamed metadata nodes, as well as debug functions.
 ;
-; RUN: llvm-reduce --test %p/Inputs/remove-metadata.py %s
+; RUN: llvm-reduce --test %p/Inputs/remove-metadata.py %s --run-only metadata
 ; RUN: cat reduced.ll | FileCheck %s
 ; REQUIRES: plugins
 
 ; CHECK: @llvm.dbg.interesting
 declare void @llvm.dbg.interesting(metadata %0, metadata %1, metadata %2)
 declare void @llvm.dbg.uninteresting(metadata %0, metadata %1, metadata %2)
 
 ; CHECK-NOT: !dbg

llvm/tools/llvm-reduce/DeltaManager.h

 #include "deltas/Delta.h"
 #include "deltas/ReduceArguments.h"
 #include "deltas/ReduceFunctions.h"
 #include "deltas/ReduceGlobalVars.h"
 #include "deltas/ReduceMetadata.h"
 
 namespace llvm {
 
-// TODO: Add CLI option to run only specified Passes (for unit tests)
 inline void runDeltaPasses(TestRunner &Tester) {
   reduceFunctionsDeltaPass(Tester);
   reduceGlobalsDeltaPass(Tester);
   reduceMetadataDeltaPass(Tester);
   reduceArgumentsDeltaPass(Tester);
   // TODO: Implement the remaining Delta Passes
 }
 
+inline void runSpecifiedDeltaPasses(TestRunner &Tester,
+                                    const std::vector<std::string> &Passes) {
+  for (auto &Pass : Passes) {
+    if (Pass == "funcs")
+      reduceFunctionsDeltaPass(Tester);
+    else if (Pass == "globals")
+      reduceGlobalsDeltaPass(Tester);
+    else if (Pass == "metadata")
+      reduceMetadataDeltaPass(Tester);
+    else if (Pass == "args")
+      reduceArgumentsDeltaPass(Tester);
+  }
+}
+
 } // namespace llvm

llvm/tools/llvm-reduce/llvm-reduce.cpp

 static cl::alias OutputFileAlias("o", cl::desc("Alias for -output"),
                                  cl::aliasopt(OutputFilename));
 
 static cl::opt<bool>
     ReplaceInput("in-place",
                  cl::desc("WARNING: This option will replace your input file"
                           "with the reduced version!"));
 
-static cl::opt<bool>
-    RunOnlySpecifiedPasses("run-only", 
-                           cl::desc("option used for testing purposes,"
-                                    "runs only the specified delta passes"),
-                           cl::Hidden);
+static cl::opt<std::string>
+    SpecifiedPasses("run-only",
 
+static std::vector<std::string> parseDeltaPassNames(std::string PassNames) {
+  std::vector<std::string> DeltaPasses;
+  if (PassNames.empty())
+    return DeltaPasses;
+
+  SmallVector<StringRef, 8> Deltas;
+  StringRef(PassNames).split(Deltas, ',', -1, false);
+  for (auto &D : Deltas)
+    DeltaPasses.push_back(D);
+
+  return DeltaPasses;
+}
+
 // Parses IR into a Module and verifies it
 static std::unique_ptr<Module> parseInputFile(StringRef Filename,
                                               LLVMContext &Ctxt) {
   SMDiagnostic Err;
   std::unique_ptr<Module> Result = parseIRFile(Filename, Err, Ctxt);
   if (!Result) {
     Err.print("llvm-reduce", errs());
     return Result;
int main(int argc, char **argv) {
   LLVMContext Context;
   std::unique_ptr<Module> OriginalProgram =
       parseInputFile(InputFilename, Context);
 
   // Initialize test environment
   TestRunner Tester(TestFilename, TestArguments, InputFilename);
   Tester.setProgram(std::move(OriginalProgram));
 
-  // Try to reduce code
+  // No passes specified, run all of them
+  if (SpecifiedPasses.empty())
     runDeltaPasses(Tester);
+  else {
+    std::vector<std::string> DeltaPasses = parseDeltaPassNames(SpecifiedPasses);
+    runSpecifiedDeltaPasses(Tester, DeltaPasses);
+  }
+
   StringRef ReducedFilename = sys::path::filename(Tester.getReducedFilepath());
 
   if (ReducedFilename == sys::path::filename(InputFilename)) {
     outs() << "\nCouldnt reduce input :/\n";
   } else {
     if (ReplaceInput) // In-place
       OutputFilename = InputFilename.c_str();
     else if (OutputFilename.empty())