[mlir] Clear running passes in crashreporter

[jpienaar]

Summary

Clear active contexts and running passes whenever finalizing crash
report message. Ran into segfault where a failure in dynamic pipeline
resulted in querying a pass whose passmanager had already been destroyed
come time for creating summary of running passes. Conservatively clear
both running states as I don't think there is recovery intended from
pass pipeline failure.

Additionally restrict to one reproducer per report - else we end up
clobbering the same reproducer file over and over again. So instead of
ending with last reproducer we now end up with the first reproducer
while not creating and clobbering reproducers over and over again.

[rriddle]

Seems reasonable to me. Is there a way we could test this at all? Okay if it requires really weird interactions.

[jpienaar]

I couldn't find a good one yet, but I'm going to keep on trying (submit to avoid segfault and will follow up with adding test if I can get one that isn't terrible/fails reliably without relying on thread interleaving).

[stellaraccident]

Thank you Jacques/River -- I found this one in the field when only running release binaries on windows. It was not fun to get the repro that triggered it. Really glad we're continuing to improve reprocucers as they are a life saver.

mlir/lib/Pass/PassCrashRecovery.cpp

if (!impl->localReproducer) {
 
     // Emit an error to the user.
     Diagnostic &note = diag.attachNote() << "Pipeline failed while executing [";
     llvm::interleaveComma(impl->runningPasses, note,
                           [&](const std::pair<Pass *, Operation *> &value) {
                             formatPassOpReproducerMessage(note, value);
                           });
     note << "]: " << description;
+    impl->runningPasses.clear();
+    impl->activeContexts.clear();
     return;
   }
 
   // If we were generating a local reproducer, we generate a reproducer for the
   // most recently executing pass using the matching entry from  `runningPasses`
   // to generate a localized diagnostic message.
   assert(impl->activeContexts.size() == impl->runningPasses.size() &&
          "expected running passes to match active contexts");
 
   // Generate the reproducer.
   RecoveryReproducerContext &reproducerContext = *impl->activeContexts.back();
   std::string description;
   reproducerContext.generate(description);
 
   // Emit an error to the user.
   Diagnostic &note = diag.attachNote() << "Pipeline failed while executing ";
   formatPassOpReproducerMessage(note, impl->runningPasses.back());
   note << ": " << description;
 
   impl->activeContexts.clear();
+  impl->runningPasses.clear();
 }
 
 void PassCrashReproducerGenerator::prepareReproducerFor(Pass *pass,
                                                         Operation *op) {
   // If not tracking local reproducers, we simply remember that this pass is
   // running.
   impl->runningPasses.insert(std::make_pair(pass, op));
   if (!impl->localReproducer)
struct CrashReproducerInstrumentation : public PassInstrumentation {
   }
 
   void runAfterPass(Pass *pass, Operation *op) override {
     if (!isa<OpToOpPassAdaptor>(pass))
       generator.removeLastReproducerFor(pass, op);
   }
 
   void runAfterPassFailed(Pass *pass, Operation *op) override {
+    // Only generate one reproducer per crash reproducer instrumentation.
+    if (alreadyFailed)
+      return;
+
+    alreadyFailed = true;
     generator.finalize(op, /*executionResult=*/failure());
   }
 
 private:
   /// The generator used to create crash reproducers.
   PassCrashReproducerGenerator &generator;
+  bool alreadyFailed = false;
 };
 } // namespace
 
 //===----------------------------------------------------------------------===//
 // FileReproducerStream
 //===----------------------------------------------------------------------===//
 
 namespace {