[MLIR] Temporary workaround for calling the LLD ELF driver as-a-lib

[aganea]

Summary

This fixes the situation described in https://github.com/llvm/llvm-project/issues/53475 with a repro exposed by https://github.com/ROCmSoftwarePlatform/D108850-lld-bug-reproduction

This is purposely just a workaround to unblock users. This could be transplanted to the release/14.x branch if need be. A proper fix will later be provided in D119049

[krzysz00]

Other than a minor typo, fixes the problem and looks good

mlir/lib/Dialect/GPU/Transforms/SerializeToHsaco.cpp

446

Missing namespace causing compilation failure

[aganea]

namespace fix.

[aganea]

mlir/lib/Dialect/GPU/Transforms/SerializeToHsaco.cpp

446

Fixed, thanks for testing! I don't have the ROCm libs installed. I only tested with the repro you provided in https://github.com/ROCmSoftwarePlatform/D108850-lld-bug-reproduction

[krzysz00]

LGTM, we can poke at long-term solutions later

This at least gets everything testable again and lets us update downstream to LLVM tip.

[MaskRay]

lld/ELF/Driver.cpp

139 ↗

(On Diff #406945)

If this is not ELF-specific, seems that it needs to live in lld::cleanup ? If the API (perhaps not intended as an API), we can give it an long discouraging name.

[aganea]

Use more generic lld::cleanup() instead of lld::elf::cleanup()

lld/Common/CommonLinkerContext.cpp

 //===- CommonLinkerContext.cpp --------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "lld/Common/CommonLinkerContext.h"
+#include "lld/Common/Driver.h"
 #include "lld/Common/ErrorHandler.h"
 #include "lld/Common/Memory.h"
 
 using namespace llvm;
 using namespace lld;
 
 // Reference to the current LLD instance. This is a temporary situation, until
 // we pass this context everywhere by reference, or we make it a thread_local,

 
 bool lld::hasContext() { return lctx != nullptr; }
 
 void CommonLinkerContext::destroy() {
   if (lctx == nullptr)
     return;
   delete lctx;
 }
+
+// Temporary API that forces global state cleanup between explicit calls to
+// drivers. See discussion in https://reviews.llvm.org/D119049.
+void lld::cleanup() {
+  // Delete the global context and clear the global context pointer, so that it
+  // cannot be accessed anymore.
+  CommonLinkerContext::destroy();
+}

lld/include/lld/Common/Driver.h

 bool link(llvm::ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
           llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput);
 }
 
 namespace wasm {
 bool link(llvm::ArrayRef<const char *> args, llvm::raw_ostream &stdoutOS,
           llvm::raw_ostream &stderrOS, bool exitEarly, bool disableOutput);
 }
-}
+
+// Temporary API that forces global state cleanup between explicit calls to
+// drivers above. DO NOT USE - this will be replaced by safeLldMain(). See
+// discussion in https://reviews.llvm.org/D119049.
+void cleanup();
+} // namespace lld
 
 #endif

mlir/lib/Dialect/GPU/Transforms/SerializeToHsaco.cpp

if (llvm::sys::fs::createTemporaryFile("kernel", "hsaco", tempHsacoFD,
     return {};
   }
   llvm::FileRemover cleanupHsaco(tempHsacoFilename);
 
   {
     static std::mutex mutex;
     const std::lock_guard<std::mutex> lock(mutex);
     // Invoke lld. Expect a true return value from lld.
-    if (!lld::elf::link({"ld.lld", "-shared", tempIsaBinaryFilename.c_str(),
+    bool r = lld::elf::link({"ld.lld", "-shared", tempIsaBinaryFilename.c_str(),
                              "-o", tempHsacoFilename.c_str()},
-                        llvm::outs(), llvm::errs(), /*exitEarly=*/true,
-                        /*disableOutput=*/false)) {
+                            llvm::outs(), llvm::errs(), /*exitEarly=*/false,
+                            /*disableOutput=*/false);
+    // Allow for calling the driver again in the same process.
+    lld::cleanup();
+    if (!r) {
       emitError(loc, "lld invocation error");
       return {};
     }
   }
 
   // Load the HSA code object.
   auto hsacoFile = openInputFile(tempHsacoFilename);
   if (!hsacoFile) {