[mlir][LLVM] Add delayed intrinsic import support

[Dinistro]

Summary

This commit introduces support for specific intrinsics imports to be
delayed after all instructions of a function were imported. This is
necesarry to import debug intrinsics correctly.

Debug intrinsics in LLVM are excluded from the define-before-use
invariant, i.e., they can reference SSA values that do not dominate
them. So far, we had implemented checks to stop the import of such
intrinsics, but there were always additional cases that were not
covered (the latest being terminators that define such used values).
To resolve this cleanly, the delayed intrinsic conversion functionality
is necessary. It allows the debug intrinsic conversion to place the
newly created MLIR operation in the correct place, while being certain
that the defining operation was already converted.

So far, this is only relevant for two LLVM debug intrinsics, that's
why the tablegen support was not added.

[zero9178]

mlir/include/mlir/Target/LLVMIR/LLVMImportInterface.h

82

Returning ArrayRef here is imo a bit of a premature optimization where it doesn't really matter and raises a lot of questions about object lifetime. I'd prefer returning SmallVector<unsigned> to always be safe from stack-use-after-return.

mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp

56–58

105–111

You can improve on the logic here by not trying to manually find a block that dominates op in its successor list, but rather querying the dominator tree to give you blocks that it dominates.
Calling domInfo.getNode(op->getBlock())->children() would give you a list of all blocks immediately dominated by the op's block. You could place the intrinsic in any of these.

One thing I am curious about, is it okay to place the debug intrinsic in the exception handling block of an invoke, right before the LandingPadOp? If not, you might have to exclude these or choose a different insertion point than the start.

347

mlir/test/Target/LLVMIR/Import/debug-info.ll

417

Could we add CHECK: lines to make sure that the @llvm.dbg.value exists?

[Dinistro]

address review comments and remove usage of dominance.

mlir/include/mlir/Target/LLVMIR/LLVMImportInterface.h

82

I agree that this isn't great, but I followed the design of the other interface functions.
Changing all the interface member functions would require changes in all implementation of it.

mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp

105–111

After internal discussion, we decided to drop the potentially expensive dominance usage with a simpler check if one of the block has a unique predecessor.
This ensures that the dominance tree is no recomputed for each terminator.

Regarding the LandingPad: You are right, this is indeed invalid. We decided to now insert all the intrinsics right before the terminator of the selected block.

mlir/test/Target/LLVMIR/Import/debug-info.ll

417

Good catch, I forgot to do that as I initially used this test to check if this still causes crashes ^^

[Dinistro]

I've implemented a less intrusive change that is more specific to the two debug intrinsics: https://reviews.llvm.org/D157496

[Dinistro]

Abandoned in favor of the simpler solution.

mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td

llvm::Function *fn =
        # !subst(".", "_", name) # [{);
     builder.CreateCall(fn, {
         llvm::MetadataAsValue::get(ctx,
             llvm::ValueAsMetadata::get(moduleTranslation.lookupValue(opInst.getOperand(0)))),
         llvm::MetadataAsValue::get(ctx, moduleTranslation.translateDebugInfo($varInfo)),
         llvm::MetadataAsValue::get(ctx, llvm::DIExpression::get(ctx, std::nullopt)),
       });
   }];
-  let mlirBuilder = [{
-    // Drop debug intrinsics with a non-empty debug expression.
-    // TODO: Support debug intrinsics that evaluate a debug expression.
-    auto *dbgIntr = cast<llvm::DbgVariableIntrinsic>(inst);
-    if (dbgIntr->hasArgList() || dbgIntr->getExpression()->getNumElements() != 0)
-      return success();
-    // Convert the value/address operand late since it cannot be a debug
-    // metadata argument list at this stage. Generating the conversion using an
-    // argument variable would not work here, since the builder variables are
-    // converted before entering the builder, which would result in an error
-    // when attempting to convert an argument list.
-
-    FailureOr<Value> argOperand = moduleImport.convertMetadataValue(llvmOperands[0]);
-    // Drop the intrinsic when its operand could not be converted. This can
-    // happen for use before definition cases that are allowed for debug
-    // intrinsics.
-    // TODO: Implement a two pass solution that translates the debug intrinsics
-    // after the entire function as been translated.
-    if (failed(argOperand))
-      return success();
-
-    // Ensure that the debug instrinsic is inserted right after its operand is
-    // defined. Otherwise, the operand might not necessarily dominate the
-    // intrinsic.
-    OpBuilder::InsertionGuard guard($_builder);
-    $_builder.setInsertionPointAfterValue(*argOperand);
-    $_op = $_builder.create<$_qualCppClassName>($_location,
-        *argOperand, $_var_attr($varInfo));
-  }];
+  let delayedImport = 1;
   let assemblyFormat = [{
     qualified($varInfo) `=` $}] # argName #
       [{ `:` qualified(type($}] # argName # [{)) attr-dict
   }];
 }
 
 def LLVM_DbgDeclareOp : LLVM_DbgIntrOp<"dbg.declare", "addr",
     [DeclareOpInterfaceMethods<PromotableOpInterface>]> {

mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td

 // `llvm::Intrinsic` enum; one usually wants these to be related. Additionally,
 // the base class also defines the "mlirBuilder" field to support the inverse
 // translation starting from an LLVM IR intrinsic. The "requiresAccessGroup",
 // "requiresAliasAnalysis", and "requiresFastmath" flags specify which
 // interfaces the intrinsic implements. If the corresponding flags are set, the
 // "aliasAttrs" list contains the arguments required by the access group and
 // alias analysis interfaces. Derived intrinsics should append the "aliasAttrs"
 // to their argument list if they set one of the flags.
+// If the `delayedImport` bit is set to 1, the intrinsic's `mlirBuilder` will
+// be ignored and the intrinsic needs to be manually added to the delayed
+// intrinsic import.
 class LLVM_IntrOpBase<Dialect dialect, string opName, string enumName,
                       list<int> overloadedResults, list<int> overloadedOperands,
                       list<Trait> traits, int numResults,
                       bit requiresAccessGroup = 0, bit requiresAliasAnalysis = 0,
                       bit requiresFastmath = 0>
     : LLVM_OpBase<dialect, opName, !listconcat(
         !if(!gt(requiresAccessGroup, 0),
             [DeclareOpInterfaceMethods<AccessGroupOpInterface>], []),
if (failed(mlirOperands))
       return failure();
     SmallVector<Type> resultTypes =
     }] # !if(!gt(numResults, 0), "{$_resultType};", "{};") # [{
     auto op = $_builder.create<$_qualCppClassName>(
       $_location, resultTypes, *mlirOperands);
     }] # !if(!gt(requiresFastmath, 0),
       "moduleImport.setFastmathFlagsAttr(inst, op);", "")
     # !if(!gt(numResults, 0), "$res = op;", "$_op = op;");
+  bit delayedImport = 0;
 }
 
 // Base class for LLVM intrinsic operations, should not be used directly. Places
 // the intrinsic into the LLVM dialect and prefixes its name with "intr.".
 class LLVM_IntrOp<string mnem, list<int> overloadedResults,
                   list<int> overloadedOperands, list<Trait> traits,
                   int numResults, bit requiresAccessGroup = 0,
                   bit requiresAliasAnalysis = 0, bit requiresFastmath = 0>

mlir/include/mlir/Target/LLVMIR/LLVMImportInterface.h

 #include "mlir/IR/Builders.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/Diagnostics.h"
 #include "mlir/IR/DialectInterface.h"
 #include "mlir/IR/Location.h"
 #include "mlir/Support/LogicalResult.h"
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
 #include "llvm/Support/FormatVariadic.h"
 
 namespace llvm {
 class IRBuilderBase;
 } // namespace llvm
 
 namespace mlir {
 namespace LLVM {
public:
   /// Hook for derived dialect interfaces to implement the import of
   /// intrinsics into MLIR.
   virtual LogicalResult
   convertIntrinsic(OpBuilder &builder, llvm::CallInst *inst,
                    LLVM::ModuleImport &moduleImport) const {
     return failure();
   }
 
+  /// Hook for derived dialect interfaces to implement the delayed import of
+  /// intrinsics into MLIR.
+  virtual LogicalResult
+  convertDelayedIntrinsic(OpBuilder &builder, llvm::CallInst *inst,
+                          LLVM::ModuleImport &moduleImport) const {
+    return failure();
+  }
+
   /// Hook for derived dialect interfaces to implement the import of metadata
   /// into MLIR. Attaches the converted metadata kind and node to the provided
   /// operation.
   virtual LogicalResult
   setMetadataAttrs(OpBuilder &builder, unsigned kind, llvm::MDNode *node,
                    Operation *op, LLVM::ModuleImport &moduleImport) const {
     return failure();
   }
 
   /// Hook for derived dialect interfaces to publish the supported intrinsics.
   /// As every LLVM IR intrinsic has a unique integer identifier, the function
   /// returns the list of supported intrinsic identifiers.
   virtual ArrayRef<unsigned> getSupportedIntrinsics() const { return {}; }
 
+  /// Hook for derived dialect interfaces to publish the supported intrinsics
+  /// that should be imported after all instructions of a function were
+  /// imported. As every LLVM IR intrinsic has a unique integer identifier, the
+  /// function returns the list of supported intrinsic identifiers.
+  virtual ArrayRef<unsigned> getSupportedDelayedIntrinsics() const {
+    return {};
+  }
+
   /// Hook for derived dialect interfaces to publish the supported metadata
   /// kinds. As every metadata kind has a unique integer identifier, the
   /// function returns the list of supported metadata identifiers.
   virtual ArrayRef<unsigned> getSupportedMetadata() const { return {}; }
 };
 
 /// Interface collection for the import of LLVM IR that dispatches to a concrete
 /// dialect interface implementation. Queries the dialect interfaces to obtain a
for (const LLVMImportDialectInterface &iface : *this) {
             llvm::formatv("expected unique conversion for intrinsic ({0}), but "
                           "got conflicting {1} and {2} conversions",
                           *it, iface.getDialect()->getNamespace(),
                           intrinsicToDialect.lookup(*it)->getNamespace()));
       }
       // Add a mapping for all supported intrinsic identifiers.
       for (unsigned id : iface.getSupportedIntrinsics())
         intrinsicToDialect[id] = iface.getDialect();
+      // Add a mapping for all supported intrinsic identifiers whose conversion
+      // should be delayed.
+      for (unsigned id : iface.getSupportedDelayedIntrinsics())
+        delayedIntrinsicToDialect[id] = iface.getDialect();
       // Add a mapping for all supported metadata kinds.
       for (unsigned kind : iface.getSupportedMetadata())
         metadataToDialect[kind].push_back(iface.getDialect());
     }
 
     return success();
   }
 
   /// Converts the LLVM intrinsic to an MLIR operation if a conversion exists.
   /// Returns failure otherwise.
   LogicalResult convertIntrinsic(OpBuilder &builder, llvm::CallInst *inst,
                                  LLVM::ModuleImport &moduleImport) const {
     // Lookup the dialect interface for the given intrinsic.
     Dialect *dialect = intrinsicToDialect.lookup(inst->getIntrinsicID());
     if (!dialect)
       return failure();
 
     // Dispatch the conversion to the dialect interface.
     const LLVMImportDialectInterface *iface = getInterfaceFor(dialect);
     assert(iface && "expected to find a dialect interface");
     return iface->convertIntrinsic(builder, inst, moduleImport);
   }
 
+  /// Converts the LLVM intrinsic to an MLIR operation assuming all other
+  /// instructions in the function were already converted. Returns failure, if
+  /// no conversion exists.
+  LogicalResult
+  convertDelayedIntrinsic(OpBuilder &builder, llvm::CallInst *inst,
+                          LLVM::ModuleImport &moduleImport) const {
+    // Lookup the dialect interface for the given intrinsic.
+    Dialect *dialect = delayedIntrinsicToDialect.lookup(inst->getIntrinsicID());
+    if (!dialect)
+      return failure();
+
+    // Dispatch the conversion to the dialect interface.
+    const LLVMImportDialectInterface *iface = getInterfaceFor(dialect);
+    assert(iface && "expected to find a dialect interface");
+    return iface->convertDelayedIntrinsic(builder, inst, moduleImport);
+  }
+
   /// Returns true if the given LLVM IR intrinsic is convertible to an MLIR
   /// operation.
   bool isConvertibleIntrinsic(llvm::Intrinsic::ID id) {
     return intrinsicToDialect.count(id);
   }
 
+  /// Returns true if the given LLVM IR intrinsic's conversion should be
+  /// delayed.
+  bool isDelayedConvertibleIntrinsic(llvm::Intrinsic::ID id) {
+    return delayedIntrinsicToDialect.count(id);
+  }
+
+  /// Returns true if `inst` is an intrinsic and its conversion should be
+  /// delayed.
+  bool isDelayedConvertibleIntrinsic(const llvm::Instruction *inst) {
+    auto *intrinsic = dyn_cast<llvm::IntrinsicInst>(inst);
+    return intrinsic &&
+           isDelayedConvertibleIntrinsic(intrinsic->getIntrinsicID());
+  }
+
   /// Attaches the given LLVM metadata to the imported operation if a conversion
   /// to one or more MLIR dialect attributes exists and succeeds. Returns
   /// success if at least one of the conversions is successful and failure if
   /// all of them fail.
   LogicalResult setMetadataAttrs(OpBuilder &builder, unsigned kind,
                                  llvm::MDNode *node, Operation *op,
                                  LLVM::ModuleImport &moduleImport) const {
     // Lookup the dialect interfaces for the given metadata.
public:
   /// Returns true if the given LLVM IR metadata is convertible to an MLIR
   /// attribute.
   bool isConvertibleMetadata(unsigned kind) {
     return metadataToDialect.count(kind);
   }
 
 private:
   DenseMap<unsigned, Dialect *> intrinsicToDialect;
+  DenseMap<unsigned, Dialect *> delayedIntrinsicToDialect;
   DenseMap<unsigned, SmallVector<Dialect *, 1>> metadataToDialect;
 };
 
 } // namespace mlir
 
 #endif // MLIR_TARGET_LLVMIR_LLVMIMPORTINTERFACE_H

mlir/include/mlir/Target/LLVMIR/ModuleImport.h

 #ifndef MLIR_TARGET_LLVMIR_MODULEIMPORT_H
 #define MLIR_TARGET_LLVMIR_MODULEIMPORT_H
 
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/Target/LLVMIR/Import.h"
 #include "mlir/Target/LLVMIR/LLVMImportInterface.h"
 #include "mlir/Target/LLVMIR/TypeFromLLVM.h"
+#include "llvm/ADT/SmallVector.h"
 
 namespace llvm {
 class BasicBlock;
 class CallBase;
 class Function;
 class Instruction;
 class Value;
 } // namespace llvm
private:
 
   /// Converts an LLVM global variable into an MLIR LLVM dialect global
   /// operation if a conversion exists. Otherwise, returns failure.
   LogicalResult convertGlobal(llvm::GlobalVariable *globalVar);
   /// Imports the magic globals "global_ctors" and "global_dtors".
   LogicalResult convertGlobalCtorsAndDtors(llvm::GlobalVariable *globalVar);
   /// Returns personality of `func` as a FlatSymbolRefAttr.
   FlatSymbolRefAttr getPersonalityAsAttr(llvm::Function *func);
-  /// Imports `bb` into `block`, which must be initially empty.
-  LogicalResult processBasicBlock(llvm::BasicBlock *bb, Block *block);
+  /// Imports `bb` into `block`, which must be initially empty. Additionally,
+  /// collects all intrinsic instructions, whose conversion should be delayed,
+  /// and adds them to `delayedIntrinsics`.
+  LogicalResult
+  processBasicBlock(llvm::BasicBlock *bb, Block *block,
+                    SmallVector<llvm::CallInst *> &delayedIntrinsics);
+  /// Converts all intrinsics in `delayedIntrinsics`. Assumes that the function
+  /// that these intrinsics are part of was already fully converted to MLIR.
+  LogicalResult
+  processDelayedIntrinsics(SmallVector<llvm::CallInst *> &delayedIntrinsics);
   /// Converts an LLVM intrinsic to an MLIR LLVM dialect operation if an MLIR
   /// counterpart exists. Otherwise, returns failure.
   LogicalResult convertIntrinsic(llvm::CallInst *inst);
   /// Converts an LLVM instruction to an MLIR LLVM dialect operation if an MLIR
   /// counterpart exists. Otherwise, returns failure.
   LogicalResult convertInstruction(llvm::Instruction *inst);
   /// Converts the metadata attached to the original instruction `inst` if
   /// a dialect interfaces supports the specific kind of metadata and attaches

mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp

 /// MLIR LLVM dialect intrinsics.
 static ArrayRef<unsigned> getSupportedIntrinsicsImpl() {
   static const SmallVector<unsigned> convertibleIntrinsics = {
 #include "mlir/Dialect/LLVMIR/LLVMConvertibleLLVMIRIntrinsics.inc"
   };
   return convertibleIntrinsics;
 }
 
+/// Returns the list of LLVM IR intrinsic identifiers that can be converted to
+/// MLIR LLVM dialect after the function they are part of is converted.
+static ArrayRef<unsigned> getSupportedDelayedIntrinsicsImpl() {
+  static const SmallVector<unsigned> convertibleDelayedIntrinsics = {
+      llvm::Intrinsic::dbg_declare, llvm::Intrinsic::dbg_value};
+  return convertibleDelayedIntrinsics;
+}
+
 /// Converts the LLVM intrinsic to an MLIR LLVM dialect operation if a
 /// conversion exits. Returns failure otherwise.
 static LogicalResult convertIntrinsicImpl(OpBuilder &odsBuilder,
                                           llvm::CallInst *inst,
                                           LLVM::ModuleImport &moduleImport) {
   llvm::Intrinsic::ID intrinsicID = inst->getIntrinsicID();
 
   // Check if the intrinsic is convertible to an MLIR dialect counterpart and
   // copy the arguments to an an LLVM operands array reference for conversion.
   if (isConvertibleIntrinsic(intrinsicID)) {
     SmallVector<llvm::Value *> args(inst->args());
     ArrayRef<llvm::Value *> llvmOperands(args);
 #include "mlir/Dialect/LLVMIR/LLVMIntrinsicFromLLVMIRConversions.inc"
   }
 
   return failure();
 }
 
+/// Converts the LLVM intrinsic to an MLIR LLVM dialect operation after all
+/// other instructions in the function were converted. Returns failure if no
+/// conversion exists.
+static LogicalResult
+convertDelayedIntrinsicImpl(OpBuilder &odsBuilder, llvm::CallInst *inst,
+                            LLVM::ModuleImport &moduleImport) {
+  auto *dbgIntr = dyn_cast<llvm::DbgVariableIntrinsic>(inst);
+  if (!dbgIntr)
+    return failure();
+  // Drop debug intrinsics with a non-empty debug expression.
+  // TODO: Support debug intrinsics that evaluate a debug expression.
+  if (dbgIntr->hasArgList() || dbgIntr->getExpression()->getNumElements() != 0)
+    return success();
+  FailureOr<Value> argOperand =
+      moduleImport.convertMetadataValue(dbgIntr->getArgOperand(0));
+  if (failed(argOperand))
+    return failure();
+
+  // Ensure that the debug instrinsic is inserted right after its operand is
+  // defined. Otherwise, the operand might not necessarily dominate the
+  // intrinsic. If the defining operation is a terminator, insert the intrinsic
+  // into a successor block that is dominated by the terminator.
+  OpBuilder::InsertionGuard guard(odsBuilder);
+  if (Operation *op = argOperand->getDefiningOp();
+      op && op->hasTrait<OpTrait::IsTerminator>()) {
+    // Find a successor that has only a single predecessor, to ensure it is
+    // dominated by this terminator.
+    auto it = llvm::find_if(op->getSuccessors(), [&](Block *successor) {
+      return successor->getUniquePredecessor();
+    });
+    // If no successor is dominated, this intrinisc cannot be converted.
+    if (it == op->getSuccessors().end())
+      return success();
+    // Set insertion point before the terminator, to avoid inserting something
+    // before landingpads.
+    odsBuilder.setInsertionPoint((*it)->getTerminator());
+  } else {
+    odsBuilder.setInsertionPointAfterValue(*argOperand);
+  }
+  Location loc = moduleImport.translateLoc(dbgIntr->getDebugLoc());
+  DILocalVariableAttr localVariableAttr =
+      moduleImport.matchLocalVariableAttr(dbgIntr->getArgOperand(1));
+  return llvm::TypeSwitch<llvm::DbgVariableIntrinsic *, LogicalResult>(dbgIntr)
+      .Case([&](llvm::DbgDeclareInst *t) {
+        odsBuilder.create<LLVM::DbgDeclareOp>(loc, *argOperand,
+                                              localVariableAttr);
+        return success();
+      })
+      .Case([&](llvm::DbgValueInst *dbgValue) {
+        odsBuilder.create<LLVM::DbgValueOp>(loc, *argOperand,
+                                            localVariableAttr);
+        return success();
+      })
+      .Default([](auto) { return failure(); });
+}
+
 /// Returns the list of LLVM IR metadata kinds that are convertible to MLIR LLVM
 /// dialect attributes.
 static ArrayRef<unsigned> getSupportedMetadataImpl() {
   static const SmallVector<unsigned> convertibleMetadata = {
       llvm::LLVMContext::MD_prof,         llvm::LLVMContext::MD_tbaa,
       llvm::LLVMContext::MD_access_group, llvm::LLVMContext::MD_loop,
       llvm::LLVMContext::MD_noalias,      llvm::LLVMContext::MD_alias_scope};
   return convertibleMetadata;
public:
 
   /// Converts the LLVM intrinsic to an MLIR LLVM dialect operation if a
   /// conversion exits. Returns failure otherwise.
   LogicalResult convertIntrinsic(OpBuilder &builder, llvm::CallInst *inst,
                                  LLVM::ModuleImport &moduleImport) const final {
     return convertIntrinsicImpl(builder, inst, moduleImport);
   }
 
+  /// Converts the LLVM intrinsic to an MLIR LLVM dialect operation, assuming
+  /// that all other instructions in the function containing the intrinsics were
+  /// already converted. Returns failure, if no conversion exists.
+  LogicalResult
+  convertDelayedIntrinsic(OpBuilder &builder, llvm::CallInst *inst,
+                          LLVM::ModuleImport &moduleImport) const final {
+    return convertDelayedIntrinsicImpl(builder, inst, moduleImport);
+  }
+
   /// Attaches the given LLVM metadata to the imported operation if a conversion
   /// to an LLVM dialect attribute exists and succeeds. Returns failure
   /// otherwise.
   LogicalResult setMetadataAttrs(OpBuilder &builder, unsigned kind,
                                  llvm::MDNode *node, Operation *op,
                                  LLVM::ModuleImport &moduleImport) const final {
     // Call metadata specific handlers.
     if (kind == llvm::LLVMContext::MD_prof)
public:
   }
 
   /// Returns the list of LLVM IR intrinsic identifiers that are convertible to
   /// MLIR LLVM dialect intrinsics.
   ArrayRef<unsigned> getSupportedIntrinsics() const final {
     return getSupportedIntrinsicsImpl();
   }
 
+  ArrayRef<unsigned> getSupportedDelayedIntrinsics() const final {
+    return getSupportedDelayedIntrinsicsImpl();
+  }
+
   /// Returns the list of LLVM IR metadata kinds that are convertible to MLIR
   /// LLVM dialect attributes.
   ArrayRef<unsigned> getSupportedMetadata() const final {
     return getSupportedMetadataImpl();
   }
 };
 } // namespace
 

mlir/lib/Target/LLVMIR/ModuleImport.cpp

LogicalResult ModuleImport::convertInstruction(llvm::Instruction *inst) {
 
   return emitError(loc) << "unhandled instruction: " << diag(*inst);
 }
 
 LogicalResult ModuleImport::processInstruction(llvm::Instruction *inst) {
   // FIXME: Support uses of SubtargetData.
   // FIXME: Add support for call / operand attributes.
   // FIXME: Add support for the indirectbr, cleanupret, catchret, catchswitch,
-  // callbr, vaarg, landingpad, catchpad, cleanuppad instructions.
+  // callbr, vaarg, catchpad, cleanuppad instructions.
 
   // Convert LLVM intrinsics calls to MLIR intrinsics.
-  if (auto *callInst = dyn_cast<llvm::CallInst>(inst)) {
-    llvm::Function *callee = callInst->getCalledFunction();
-    if (callee && callee->isIntrinsic())
-      return convertIntrinsic(callInst);
-  }
+  if (auto *intrinsic = dyn_cast<llvm::IntrinsicInst>(inst))
+    return convertIntrinsic(intrinsic);
 
   // Convert all remaining LLVM instructions to MLIR operations.
   return convertInstruction(inst);
 }
 
 FlatSymbolRefAttr ModuleImport::getPersonalityAsAttr(llvm::Function *f) {
   if (!f->hasPersonalityFn())
     return nullptr;
for (const auto &it : llvm::enumerate(func->args())) {
     mapValue(&it.value(), blockArg);
   }
 
   // Process the blocks in topological order. The ordered traversal ensures
   // operands defined in a dominating block have a valid mapping to an MLIR
   // value once a block is translated.
   SetVector<llvm::BasicBlock *> blocks = getTopologicallySortedBlocks(func);
   setConstantInsertionPointToStart(lookupBlock(blocks.front()));
-  for (llvm::BasicBlock *bb : blocks) {
-    if (failed(processBasicBlock(bb, lookupBlock(bb))))
-      return failure();
-  }
+  SmallVector<llvm::CallInst *> delayedIntrinsics;
+  for (llvm::BasicBlock *bb : blocks)
+    if (failed(processBasicBlock(bb, lookupBlock(bb), delayedIntrinsics)))
+      return failure();
+
+  // Process the intrinsics that require a delayed conversion after everything
+  // else was converted.
+  if (failed(processDelayedIntrinsics(delayedIntrinsics)))
+
+  return success();
 
+LogicalResult ModuleImport::processDelayedIntrinsics(
+    SmallVector<llvm::CallInst *> &delayedIntrinsics) {
+  for (llvm::CallInst *intrCall : delayedIntrinsics) {
+    if (failed(iface.convertDelayedIntrinsic(builder, intrCall, *this)))
+      return failure();
+    if (Operation *op = lookupOperation(intrCall)) {
+      setNonDebugMetadataAttrs(intrCall, op);
+    } else if (emitExpensiveWarnings) {
+      Location loc = debugImporter->translateLoc(intrCall->getDebugLoc());
+      emitWarning(loc) << "dropped intrinsic: " << diag(*intrCall);
+    }
+  }
   return success();
 }
 
-LogicalResult ModuleImport::processBasicBlock(llvm::BasicBlock *bb,
-                                              Block *block) {
+LogicalResult ModuleImport::processBasicBlock(
+    llvm::BasicBlock *bb, Block *block,
+    SmallVector<llvm::CallInst *> &delayedIntrinsics) {
   builder.setInsertionPointToStart(block);
   for (llvm::Instruction &inst : *bb) {
+    if (iface.isDelayedConvertibleIntrinsic(&inst)) {
+      // Collect all intrinsics whose conversion should be delayed.
+      delayedIntrinsics.push_back(cast<llvm::CallInst>(&inst));
+      continue;
+    }
     if (failed(processInstruction(&inst)))
       return failure();
 
     // Set the non-debug metadata attributes on the imported operation and emit
     // a warning if an instruction other than a phi instruction is dropped
     // during the import.
     if (Operation *op = lookupOperation(&inst)) {
       setNonDebugMetadataAttrs(&inst, op);

mlir/test/Target/LLVMIR/Import/debug-info.ll

 !2 = !DIFile(filename: "debug-info.ll", directory: "/")
 !3 = !DISubprogram(name: "variadic_func", scope: !2, file: !2, flags: DIFlagPrototyped, spFlags: DISPFlagOptimized, type: !4)
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, null}
 
 ; // -----
 
 define void @dbg_use_before_def(ptr %arg) {
+  ; CHECK: llvm.getelementptr
+  ; CHECK-NEXT: llvm.intr.dbg.value
   call void @llvm.dbg.value(metadata ptr %dbg_arg, metadata !7, metadata !DIExpression()), !dbg !9
   %dbg_arg = getelementptr double, ptr %arg, i64 16
   ret void
 }
 
 declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !llvm.dbg.cu = !{!1}

 !5 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !3, flags: DIFlagArtificial | DIFlagObjectPointer)
 !6 = !DIDerivedType(tag: DW_TAG_member, name: "call_field", file: !2, baseType: !5)
 !7 = !DILocalVariable(scope: !8, name: "var", file: !2, type: !5);
 !8 = distinct !DISubprogram(name: "dbg_use_before_def", scope: !2, file: !2, spFlags: DISPFlagDefinition, unit: !1)
 !9 = !DILocation(line: 1, column: 2, scope: !8)
 
 ; // -----
 
+declare i64 @callee()
+declare i32 @personality(...)
+
+; CHECK-LABEL: @dbg_broken_dominance_invoke
+define void @dbg_broken_dominance_invoke() personality ptr @personality {
+  %1 = invoke i64 @callee()
+          to label %b1 unwind label %b2
+b1:
+; CHECK: llvm.intr.dbg.value
+  call void @llvm.dbg.value(metadata i64 %1, metadata !7, metadata !DIExpression()), !dbg !9
+  ret void
+b2:
+  %2 = landingpad { ptr, i32 }
+          cleanup
+  ret void
+}
+
+declare void @llvm.dbg.value(metadata, metadata, metadata)
+
+!llvm.dbg.cu = !{!1}
+!llvm.module.flags = !{!0}
+!0 = !{i32 2, !"Debug Info Version", i32 3}
+!1 = distinct !DICompileUnit(language: DW_LANG_C, file: !2)
+!2 = !DIFile(filename: "debug-info.ll", directory: "/")
+!7 = !DILocalVariable(scope: !8, name: "var", file: !2);
+!8 = distinct !DISubprogram(name: "dbg_broken_dominance_invoke", scope: !2, file: !2, spFlags: DISPFlagDefinition, unit: !1)
+!9 = !DILocation(line: 1, column: 2, scope: !8)
+
+; // -----
+
+declare i64 @callee()
+declare i32 @personality(...)
+
+; CHECK-LABEL: @dbg_broken_dominance_invoke_reordered
+define void @dbg_broken_dominance_invoke_reordered() personality ptr @personality {
+  %1 = invoke i64 @callee()
+          to label %b2 unwind label %b1
+b1:
+; CHECK: landingpad
+; CHECK: llvm.intr.dbg.value
+  %2 = landingpad { ptr, i32 }
+          cleanup
+  call void @llvm.dbg.value(metadata i64 %1, metadata !7, metadata !DIExpression()), !dbg !9
+  ret void
+b2:
+  ret void
+}
+
+declare void @llvm.dbg.value(metadata, metadata, metadata)
+
+!llvm.dbg.cu = !{!1}
+!llvm.module.flags = !{!0}
+!0 = !{i32 2, !"Debug Info Version", i32 3}
+!1 = distinct !DICompileUnit(language: DW_LANG_C, file: !2)
+!2 = !DIFile(filename: "debug-info.ll", directory: "/")
+!7 = !DILocalVariable(scope: !8, name: "var", file: !2);
+!8 = distinct !DISubprogram(name: "dbg_broken_dominance_invoke", scope: !2, file: !2, spFlags: DISPFlagDefinition, unit: !1)
+!9 = !DILocation(line: 1, column: 2, scope: !8)
+
+; // -----
+
 ; CHECK-DAG: #[[NAMESPACE:.+]] = #llvm.di_namespace<name = "std", exportSymbols = false>
 ; CHECK-DAG: #[[SUBPROGRAM:.+]] =  #llvm.di_subprogram<compileUnit = #{{.*}}, scope = #[[NAMESPACE]], name = "namespace"
 
 define void @namespace(ptr %arg) {
   call void @llvm.dbg.value(metadata ptr %arg, metadata !7, metadata !DIExpression()), !dbg !9
   ret void
 }
 

mlir/test/Target/LLVMIR/Import/import-failure.ll

define void @unhandled_intrinsic() gc "example" {
   ret void
 }
 
 ; // -----
 
 declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 ; CHECK:      import-failure.ll
-; CHECK-SAME: warning: dropped instruction: call void @llvm.dbg.value(metadata i64 %arg1, metadata !3, metadata !DIExpression(DW_OP_plus_uconst, 42, DW_OP_stack_value)), !dbg !5
+; CHECK-SAME: warning: dropped intrinsic: call void @llvm.dbg.value(metadata i64 %arg1, metadata !3, metadata !DIExpression(DW_OP_plus_uconst, 42, DW_OP_stack_value)), !dbg !5
 ; CHECK:      import-failure.ll
-; CHECK-SAME: warning: dropped instruction: call void @llvm.dbg.value(metadata !DIArgList(i64 %arg1, i64 undef), metadata !3, metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_constu, 1, DW_OP_mul, DW_OP_plus, DW_OP_stack_value)), !dbg !5
+; CHECK-SAME: warning: dropped intrinsic: call void @llvm.dbg.value(metadata !DIArgList(i64 %arg1, i64 undef), metadata !3, metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_constu, 1, DW_OP_mul, DW_OP_plus, DW_OP_stack_value)), !dbg !5
 define void @dropped_instruction(i64 %arg1) {
   call void @llvm.dbg.value(metadata i64 %arg1, metadata !3, metadata !DIExpression(DW_OP_plus_uconst, 42, DW_OP_stack_value)), !dbg !5
   call void @llvm.dbg.value(metadata !DIArgList(i64 %arg1, i64 undef), metadata !3, metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_constu, 1, DW_OP_mul, DW_OP_plus, DW_OP_stack_value)), !dbg !5
   ret void
 }
 
 !llvm.dbg.cu = !{!1}
 !llvm.module.flags = !{!0}

mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp

static bool emitIntrMLIRBuilders(const RecordKeeper &recordKeeper,
                                  raw_ostream &os) {
   // Emit condition to check if "llvmEnumName" matches the intrinsic id.
   auto emitIntrCond = [](const Record &record) {
     return "intrinsicID == llvm::Intrinsic::" +
            record.getValueAsString("llvmEnumName");
   };
   for (const Record *def :
        recordKeeper.getAllDerivedDefinitions("LLVM_IntrOpBase")) {
+    // Intrinsics whose import should be delayed do not need the emission of an
+    // mlirBuilder.
+    if (def->getValueAsBit("delayedImport"))
+      continue;
     if (failed(emitOneMLIRBuilder(*def, os, emitIntrCond)))
       return true;
   }
   return false;
 }
 
 // Emit all op builders. Returns false on success because of the
 // generator registration requirements.
static void emitOneIntrinsic(const Record &record, raw_ostream &os) {
   os << "llvm::Intrinsic::" << record.getValueAsString("llvmEnumName") << ",\n";
 }
 
 // Emit the list of LLVM IR intrinsics identifiers that are convertible to a
 // matching MLIR LLVM dialect intrinsic operation.
 static bool emitConvertibleIntrinsics(const RecordKeeper &recordKeeper,
                                       raw_ostream &os) {
   for (const Record *def :
-       recordKeeper.getAllDerivedDefinitions("LLVM_IntrOpBase"))
+       recordKeeper.getAllDerivedDefinitions("LLVM_IntrOpBase")) {
+    // Intrinsics with a a delayed import are not convertible by the normal
+    // intrinsic conversion, so skip them.
+    if (def->getValueAsBit("delayedImport"))
+      continue;
     emitOneIntrinsic(*def, os);
+  }
 
   return false;
 }
 
 static mlir::GenRegistration
     genLLVMIRConversions("gen-llvmir-conversions",
                          "Generate LLVM IR conversions", emitBuilders);