[mlir][llvmir] Import intrinsics with attributes from LLVMIR.

[gysit]

Summary

The revision adds support to specify custom import functions for
LLVM IR intrinsics with immediate arguments that translate to MLIR
attributes. It takes an approach similar to the MLIR to LLVM translation
that uses a tablegen defined build method. The default implementation
of this newly introduced "mlirBuilder" assumes all intrinsic arguments
translate to operands. Specific intrinsics, such as
llvm.lifetime.start/stop then define a custom builder that converts
their immediate arguments to MLIR attributes.

Depends on D135349

[ftynse]

mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td

121

It feels like having a separate builder for start and end using the normal builder.create would not be longer, but would be more robust...

mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td

338

338

Nit: "pattern" is pretty strongly associated with a rewrite pattern, which this is not, consider a different term.

352

mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp

263

I think we are allowed to use MLIRSupport here, so prefer LogicalResult ot bools.

280–281

Nit: expand auto.

292–293

I don't see any check for this being an argument or a result name, so this error message is incorrect.

300–303

I'm confused about insertion here. The injected code, from what I can see, only calls Operation::create which only creates an detached operation but doesn't put it anywhere in the surrounding IR. One would normally need to push/splice it into some block, and I don't see where that would happen. But the test seems to pass... Can you point me to where the insertion happens?

313

Please expand auto here and everywhere else where the type is not clear from the immediate context, except for lambdas and annoyingly long types such as iterators.

[gysit]

mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td

121

Ah that could be an interesting approach for all intrinsics. I could essentially, define a normal builder that constructs an MLIR LLVM dialect intrinsic given the LLVM IR call arguments? At least, if it is possible to define a builder on the intrinsic base class and then overwrite it for specific intrinsics.

mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp

300–303

If not mistaken, I did not change that part before and after the revision the code calls:

Operation *op = b.create(state);

I thought this actually inserts the operation at the current position.

Would it be preferable to call b.create<QualifiedCPPName>(...)? This would be possible in tablgen.

[ftynse]

mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td

121

I meant for these two specifically, builder.create<LLVM::LifetimeStartOp>($_location, lookupIntegerAttr(inst->getArgOperand(0)), lookupValue(inst->getArgOperand(1))) looks more readable to an average MLIR user than going through OperationState. builder.create<LLVM::LifetimeStartOp>($_location, $_int_attr($size), $ptr) would be even more concise, but unclear if it's worth the complexity.

For the base class, you can likely call the default-generated builder function that takes a list of results, a list of operands and a list of attributes. Intrinsics that need attributes will have to override that or need some mechanism to tell the base class which ArgOperands correspond to which attribute names/types.

mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp

300–303

Oh, I missed that it was b.create, that does insert, thanks.

Yes, prefer using the templated version to populating the state manually. Also, at minimum document that "b" is an OpBuilder variable available in the builder snippet and that it is set up to the correct insertion point. Better yet, make it use $_builder instead and, internally, rename "b" to something that the user is less likely to clash with, like "odsBuilder".

[gysit]

Address comments:

• nicer builders
• avoid auto

[gysit]

mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td

121

I tried to implement the later solution. Operations with complex operands / attributes that differ too much from the LLVM implementation will still need a handwritten mapping from call instruction arguments to mlir builder arguments.

[ftynse]

mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td

121

Nice!

mlir/include/mlir/Dialect/LLVMIR/CMakeLists.txt

 set(LLVM_TARGET_DEFINITIONS LLVMOps.td)
 mlir_tablegen(LLVMConversions.inc -gen-llvmir-conversions)
 mlir_tablegen(LLVMConversionEnumsToLLVM.inc -gen-enum-to-llvmir-conversions)
 mlir_tablegen(LLVMConversionEnumsFromLLVM.inc -gen-enum-from-llvmir-conversions)
 add_public_tablegen_target(MLIRLLVMConversionsIncGen)
 
 set(LLVM_TARGET_DEFINITIONS LLVMIntrinsicOps.td)
 mlir_tablegen(LLVMIntrinsicConversions.inc -gen-llvmir-conversions)
-mlir_tablegen(LLVMIntrinsicToLLVMIROpPairs.inc -gen-llvmintrinsic-to-llvmirop-pairs)
+mlir_tablegen(LLVMIntrinsicFromLLVMIRConversions.inc -gen-intr-from-llvmir-conversions)
+mlir_tablegen(LLVMConvertibleLLVMIRIntrinsics.inc -gen-convertible-llvmir-intrinsics)
 add_public_tablegen_target(MLIRLLVMIntrinsicConversionsIncGen)
 
 add_mlir_dialect(NVVMOps nvvm)
 add_mlir_doc(NVVMOps NVVMDialect Dialects/ -gen-dialect-doc -dialect=nvvm)
 set(LLVM_TARGET_DEFINITIONS NVVMOps.td)
 mlir_tablegen(NVVMConversions.inc -gen-llvmir-conversions)
 mlir_tablegen(NVVMOpsEnums.h.inc -gen-enum-decls)
 mlir_tablegen(NVVMOpsEnums.cpp.inc -gen-enum-defs)
 mlir_tablegen(NVVMOpsAttributes.h.inc -gen-attrdef-decls -attrdefs-dialect=nvvm)
 mlir_tablegen(NVVMOpsAttributes.cpp.inc -gen-attrdef-defs -attrdefs-dialect=nvvm)
 add_public_tablegen_target(MLIRNVVMConversionsIncGen)
 
 add_mlir_dialect(ROCDLOps rocdl)
 add_mlir_doc(ROCDLOps ROCDLDialect Dialects/ -gen-dialect-doc -dialect=rocdl)
 set(LLVM_TARGET_DEFINITIONS ROCDLOps.td)
 mlir_tablegen(ROCDLConversions.inc -gen-llvmir-conversions)
 add_public_tablegen_target(MLIRROCDLConversionsIncGen)
-

mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td

let llvmBuilder = [{
     llvm::Function *fn = llvm::Intrinsic::getDeclaration(
         module, llvm::Intrinsic::}] # llvmEnumName # [{, {}] #
         !interleave(ListIntSubst<LLVM_IntrPatterns.operand, [0]>.lst, ", ")
         # [{});
     builder.CreateCall(fn, {builder.getInt64(op.getSizeAttr().getInt()),
                             moduleTranslation.lookupValue(op.getPtr())});
   }];
 
   let assemblyFormat = "$size `,` $ptr attr-dict `:` type($ptr)";
 }
 
-def LLVM_LifetimeStartOp : LLVM_LifetimeBaseOp<"lifetime.start">;
-def LLVM_LifetimeEndOp : LLVM_LifetimeBaseOp<"lifetime.end">;
+def LLVM_LifetimeStartOp : LLVM_LifetimeBaseOp<"lifetime.start"> {
+  // Custom builder to convert the size argument to an attribute.
+  string mlirBuilder = [{
+    $_builder.create<LLVM::LifetimeStartOp>(
+      $_location, $_int_attr($size), $ptr);
+  }];
+}
+def LLVM_LifetimeEndOp : LLVM_LifetimeBaseOp<"lifetime.end"> {
+  // Custom builder to convert the size argument to an attribute.
+  string mlirBuilder = [{
+    $_builder.create<LLVM::LifetimeEndOp>(
+      $_location, $_int_attr($size), $ptr);
+  }];
+}
 
 // Intrinsics with multiple returns.
 
 def LLVM_SAddWithOverflowOp
     : LLVM_IntrOp<"sadd.with.overflow", [0], [], [], 2> {
   let arguments = (ins LLVM_Type, LLVM_Type);
 }
 def LLVM_UAddWithOverflowOp

mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td

 // and operands that are overloadable in the LLVM sense, that is their types
 // must be passed in during the construction of the intrinsic declaration to
 // differentiate between differently-typed versions of the intrinsic.
 // If the intrinsic has multiple results, this will eventually be packed into a
 // single struct result. In this case, the types of any overloaded results need
 // to be accessed via the LLVMStructType, instead of directly via the result.
 // "opName" contains the name of the operation to be associated with the
 // intrinsic and "enumName" contains the name of the intrinsic as appears in
-// `llvm::Intrinsic` enum; one usually wants these to be related.
+// `llvm::Intrinsic` enum; one usually wants these to be related. Additionally,
+// the base class also defines the "mlirBuilder" field to support the inverse
+// translation starting from an LLVM IR intrinsic.
 class LLVM_IntrOpBase<Dialect dialect, string opName, string enumName,
                       list<int> overloadedResults, list<int> overloadedOperands,
                       list<Trait> traits, int numResults,
                       bit requiresAccessGroup = 0, bit requiresAliasScope = 0>
     : LLVM_OpBase<dialect, opName, traits>,
       Results<!if(!gt(numResults, 0), (outs LLVM_Type:$res), (outs))> {
   string resultPattern = !if(!gt(numResults, 1),
                              LLVM_IntrPatterns.structResult,
                              LLVM_IntrPatterns.result);
-  string llvmEnumName = enumName;                           
+  string llvmEnumName = enumName;
   let llvmBuilder = [{
     llvm::Module *module = builder.GetInsertBlock()->getModule();
     llvm::Function *fn = llvm::Intrinsic::getDeclaration(
         module,
         llvm::Intrinsic::}] # enumName # [{,
         { }] # !interleave(!listconcat(
             ListIntSubst<resultPattern, overloadedResults>.lst,
             ListIntSubst<LLVM_IntrPatterns.operand,
                          overloadedOperands>.lst), ", ") # [{
         });
     auto operands = moduleTranslation.lookupValues(opInst.getOperands());
     }] # [{auto *inst = builder.CreateCall(fn, operands);
     }] # !if(!gt(requiresAccessGroup, 0),
       "moduleTranslation.setAccessGroupsMetadata(op, inst);",
       "(void) inst;")
     # !if(!gt(requiresAliasScope, 0),
       "moduleTranslation.setAliasScopeMetadata(op, inst);",
       "(void) inst;")
     # !if(!gt(numResults, 0), "$res = inst;", "");
+
+  // A builder to construct the MLIR LLVM dialect operation given the matching
+  // LLVM IR intrinsic instruction `callInst`. The following $-variables exist:
+  //   - $name - substituted by the remapped `callInst` argument using the
+  //             the index of the MLIR operation argument with the given name;
+  //   - $_int_attr - substituted by a call to an integer attribute matcher;
+  //   - $_resultType - substituted with the MLIR result type;
+  //   - $_location - substituted with the MLIR location;
+  //   - $_builder - substituted with the MLIR builder;
+  //   - $_qualCppClassName - substitiuted with the MLIR operation class name.
+  // Additionally, `$$` can be used to produce the dollar character.
+  // NOTE: The $name variable resolution assumes the MLIR and LLVM argument
+  // orders match and there are no optional or variadic arguments.
+  string mlirBuilder = [{
+    SmallVector<llvm::Value *> operands(callInst->args());
+    SmallVector<Type> resultTypes =
+    }] # !if(!gt(numResults, 0),
+      "{$_resultType};", "{};") # [{
+    Operation *op = $_builder.create<$_qualCppClassName>(
+      $_location,
+      resultTypes,
+      processValues(operands));
+    }] # !if(!gt(numResults, 0),
+      "instMap[callInst] = op->getResult(0);", "(void)op;");
 }
 
 // Base class for LLVM intrinsic operations, should not be used directly. Places
 // the intrinsic into the LLVM dialect and prefixes its name with "intr.".
 class LLVM_IntrOp<string mnem, list<int> overloadedResults,
                   list<int> overloadedOperands, list<Trait> traits,
                   int numResults, bit requiresAccessGroup = 0,
                   bit requiresAliasScope = 0>

mlir/lib/Target/LLVMIR/ConvertFromLLVMIR.cpp

 #include "mlir/Target/LLVMIR/Import.h"
 
 #include "mlir/Dialect/DLTI/DLTI.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/MLIRContext.h"
+#include "mlir/IR/Matchers.h"
 #include "mlir/Interfaces/DataLayoutInterfaces.h"
 #include "mlir/Target/LLVMIR/TypeFromLLVM.h"
 #include "mlir/Tools/mlir-translate/Translation.h"
 
 #include "llvm/ADT/StringSet.h"
 #include "llvm/ADT/TypeSwitch.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/InlineAsm.h"
 #include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/Type.h"
 #include "llvm/IRReader/IRReader.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/SourceMgr.h"
 
 using namespace mlir;
 using namespace mlir::LLVM;
 
 #include "mlir/Dialect/LLVMIR/LLVMConversionEnumsFromLLVM.inc"
 
+/// Returns true if the LLVM IR intrinsic is convertible to an MLIR LLVM dialect
+/// intrinsic, or false if no counterpart exists.
+static bool isConvertibleIntrinsic(llvm::Intrinsic::ID id) {
+  static const DenseSet<unsigned> convertibleIntrinsics = {
+#include "mlir/Dialect/LLVMIR/LLVMConvertibleLLVMIRIntrinsics.inc"
+  };
+  return convertibleIntrinsics.contains(id);
+}
+
 // Utility to print an LLVM value as a string for passing to emitError().
 // FIXME: Diagnostic should be able to natively handle types that have
 // operator << (raw_ostream&) defined.
 static std::string diag(llvm::Value &v) {
   std::string s;
   llvm::raw_string_ostream os(s);
   os << v;
   return os.str();
public:
   /// remapped later if the defining instruction has not yet been visited.
   Value processValue(llvm::Value *value);
 
   /// Calls `processValue` for a range of `values` and returns their remapped
   /// values or placeholders if the defining instructions have not yet been
   /// visited.
   SmallVector<Value> processValues(ArrayRef<llvm::Value *> values);
 
+  /// Converts `value` to an integer attribute. Asserts if the conversion fails.
+  IntegerAttr matchIntegerAttr(Value value);
+
   /// Translate the debug location to a FileLineColLoc, if `loc` is non-null.
   /// Otherwise, return UnknownLoc.
   Location translateLoc(llvm::DILocation *loc);
 
   /// Converts the type from LLVM to MLIR LLVM dialect.
   Type convertType(llvm::Type *type);
 
+  /// Converts an LLVM intrinsic to an MLIR LLVM dialect operation if an MLIR
+  /// counterpart exists. Otherwise, returns failure.
+  LogicalResult convertIntrinsic(OpBuilder &odsBuilder,
+                                 llvm::CallInst *callInst);
+
   /// Imports `f` into the current module.
   LogicalResult processFunction(llvm::Function *f);
 
   /// Converts function attributes of LLVM Function \p f
   /// into LLVM dialect attributes of LLVMFuncOp \p funcOp.
   void processFunctionAttributes(llvm::Function *f, LLVMFuncOp funcOp);
 
   /// Imports GV as a GlobalOp, creating it if it doesn't exist.
Location Importer::translateLoc(llvm::DILocation *loc) {
   return FileLineColLoc::get(context, loc->getFilename(), loc->getLine(),
                              loc->getColumn());
 }
 
 Type Importer::convertType(llvm::Type *type) {
   return typeTranslator.translateType(type);
 }
 
+LogicalResult Importer::convertIntrinsic(OpBuilder &odsBuilder,
+                                         llvm::CallInst *callInst) {
+  llvm::Function *callee = callInst->getCalledFunction();
+  if (!callee || !callee->isIntrinsic())
+    return failure();
+
+  // Check if the intrinsic is convertible to an MLIR dialect counterpart.
+  llvm::Intrinsic::ID intrinsicID = callee->getIntrinsicID();
+  if (!isConvertibleIntrinsic(intrinsicID))
+    return failure();
+
+#include "mlir/Dialect/LLVMIR/LLVMIntrinsicFromLLVMIRConversions.inc"
+
+  return failure();
+}
+
 // We only need integers, floats, doubles, and vectors and tensors thereof for
 // attributes. Scalar and vector types are converted to the standard
 // equivalents. Array types are converted to ranked tensors; nested array types
 // are converted to multi-dimensional tensors or vectors, depending on the
 // innermost type being a scalar or a vector.
 Type Importer::getStdTypeForAttr(Type type) {
   if (!type)
     return nullptr;

 SmallVector<Value> Importer::processValues(ArrayRef<llvm::Value *> values) {
   SmallVector<Value> remapped;
   remapped.reserve(values.size());
   for (llvm::Value *value : values)
     remapped.push_back(processValue(value));
   return remapped;
 }
 
+IntegerAttr Importer::matchIntegerAttr(Value value) {
+  IntegerAttr integerAttr;
+  bool success = matchPattern(value, m_Constant(&integerAttr));
+  assert(success && "expected a constant value");
+  (void)success;
+  return integerAttr;
+}
+
 /// Return the MLIR OperationName for the given LLVM opcode.
 static StringRef lookupOperationNameFromOpcode(unsigned opcode) {
 // Maps from LLVM opcode to MLIR OperationName. This is deliberately ordered
 // as in llvm/IR/Instructions.def to aid comprehension and spot missing
 // instructions.
 #define INST(llvm_n, mlir_n)                                                   \
   { llvm::Instruction::llvm_n, LLVM::mlir_n##Op::getOperationName() }
   static const DenseMap<unsigned, StringRef> opcMap = {
static const DenseMap<unsigned, StringRef> opcMap = {
       // FIXME: cleanuppad
       // clang-format on
   };
 #undef INST
 
   return opcMap.lookup(opcode);
 }
 
-/// Return the MLIR OperationName for the given LLVM intrinsic ID.
-static StringRef lookupOperationNameFromIntrinsicID(unsigned id) {
-  // Maps from LLVM intrinsic ID to MLIR OperationName.
-  static const DenseMap<unsigned, StringRef> intrMap = {
-#include "mlir/Dialect/LLVMIR/LLVMIntrinsicToLLVMIROpPairs.inc"
-  };
-  return intrMap.lookup(id);
-}
-
 static ICmpPredicate getICmpPredicate(llvm::CmpInst::Predicate p) {
   switch (p) {
   default:
     llvm_unreachable("incorrect comparison predicate");
   case llvm::CmpInst::Predicate::ICMP_EQ:
     return LLVM::ICmpPredicate::eq;
   case llvm::CmpInst::Predicate::ICMP_NE:
     return LLVM::ICmpPredicate::ne;
LogicalResult Importer::processInstruction(llvm::Instruction *inst) {
   case llvm::Instruction::PHI: {
     Type type = convertType(inst->getType());
     instMap[inst] = b.getInsertionBlock()->addArgument(
         type, translateLoc(inst->getDebugLoc()));
     return success();
   }
   case llvm::Instruction::Call: {
     llvm::CallInst *ci = cast<llvm::CallInst>(inst);
+
+    // For all intrinsics, try to generate to the corresponding op.
+    if (succeeded(convertIntrinsic(b, ci)))
+      return success();
+
     SmallVector<llvm::Value *> args(ci->args());
     SmallVector<Value> ops = processValues(args);
     SmallVector<Type, 2> tys;
     if (!ci->getType()->isVoidTy()) {
       Type type = convertType(inst->getType());
       tys.push_back(type);
     }
     Operation *op;
     if (llvm::Function *callee = ci->getCalledFunction()) {
-      // For all intrinsics, try to generate to the corresponding op.
-      if (callee->isIntrinsic()) {
-        auto id = callee->getIntrinsicID();
-        StringRef opName = lookupOperationNameFromIntrinsicID(id);
-        if (!opName.empty()) {
-          OperationState state(loc, opName);
-          state.addOperands(ops);
-          state.addTypes(tys);
-          Operation *op = b.create(state);
-          if (!inst->getType()->isVoidTy())
-            instMap[inst] = op->getResult(0);
-          return success();
-        }
-      }
       op = b.create<CallOp>(
           loc, tys, SymbolRefAttr::get(b.getContext(), callee->getName()), ops);
     } else {
       Value calledValue = processValue(ci->getCalledOperand());
       ops.insert(ops.begin(), calledValue);
       op = b.create<CallOp>(loc, tys, ops);
     }
     if (!ci->getType()->isVoidTy())

 
 LogicalResult Importer::processFunction(llvm::Function *f) {
   blocks.clear();
   instMap.clear();
   unknownInstMap.clear();
 
   auto functionType =
       convertType(f->getFunctionType()).dyn_cast<LLVMFunctionType>();
-  if (f->isIntrinsic()) {
-    StringRef opName = lookupOperationNameFromIntrinsicID(f->getIntrinsicID());
-    // Skip the intrinsic decleration if we could found a corresponding op.
-    if (!opName.empty())
-      return success();
-  }
+  if (f->isIntrinsic() && isConvertibleIntrinsic(f->getIntrinsicID()))
 
   bool dsoLocal = f->hasLocalLinkage();
   CConv cconv = convertCConvFromLLVM(f->getCallingConv());
 
   b.setInsertionPoint(module.getBody(), getFuncInsertPt());
   LLVMFuncOp fop = b.create<LLVMFuncOp>(
       UnknownLoc::get(context), f->getName(), functionType,
       convertLinkageFromLLVM(f->getLinkage()), dsoLocal, cconv);

mlir/test/Target/LLVMIR/Import/intrinsic.ll

define void @umin_test(i32 %0, i32 %1, <8 x i32> %2, <8 x i32> %3) {
   ; CHECK:   "llvm.intr.umin"(%{{.*}}, %{{.*}}) : (vector<8xi32>, vector<8xi32>) -> vector<8xi32>
   %6 = call <8 x i32> @llvm.umin.v8i32(<8 x i32> %2, <8 x i32> %3)
   ret void
 }
 ; CHECK-LABEL:  llvm.func @vector_reductions
 define void @vector_reductions(float %0, <8 x float> %1, <8 x i32> %2) {
   ; CHECK: "llvm.intr.vector.reduce.add"(%{{.*}}) : (vector<8xi32>) -> i32
   %4 = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> %2)
-  ; CHECK: "llvm.intr.vector.reduce.and"(%{{.*}}) : (vector<8xi32>) -> i32  
+  ; CHECK: "llvm.intr.vector.reduce.and"(%{{.*}}) : (vector<8xi32>) -> i32
   %5 = call i32 @llvm.vector.reduce.and.v8i32(<8 x i32> %2)
   ; CHECK: "llvm.intr.vector.reduce.fmax"(%{{.*}}) : (vector<8xf32>) -> f32
   %6 = call float @llvm.vector.reduce.fmax.v8f32(<8 x float> %1)
   ; CHECK: "llvm.intr.vector.reduce.fmin"(%{{.*}}) : (vector<8xf32>) -> f32
   %7 = call float @llvm.vector.reduce.fmin.v8f32(<8 x float> %1)
-  ; CHECK: "llvm.intr.vector.reduce.mul"(%{{.*}}) : (vector<8xi32>) -> i32  
+  ; CHECK: "llvm.intr.vector.reduce.mul"(%{{.*}}) : (vector<8xi32>) -> i32
   %8 = call i32 @llvm.vector.reduce.mul.v8i32(<8 x i32> %2)
   ; CHECK: "llvm.intr.vector.reduce.or"(%{{.*}}) : (vector<8xi32>) -> i32
   %9 = call i32 @llvm.vector.reduce.or.v8i32(<8 x i32> %2)
   ; CHECK: "llvm.intr.vector.reduce.smax"(%{{.*}}) : (vector<8xi32>) -> i32
   %10 = call i32 @llvm.vector.reduce.smax.v8i32(<8 x i32> %2)
   ; CHECK: "llvm.intr.vector.reduce.smin"(%{{.*}}) : (vector<8xi32>) -> i32
   %11 = call i32 @llvm.vector.reduce.smin.v8i32(<8 x i32> %2)
   ; CHECK: "llvm.intr.vector.reduce.umax"(%{{.*}}) : (vector<8xi32>) -> i32
define void @coro_suspend(i32 %0, i1 %1, i8* %2) {
   %4 = call token @llvm.coro.id(i32 %0, i8* %2, i8* %2, i8* null)
   ; CHECK: llvm.intr.coro.suspend %{{.*}}, %{{.*}} : i8
   %5 = call i8 @llvm.coro.suspend(token %4, i1 %1)
   ret void
 }
 
 ; CHECK-LABEL:  llvm.func @coro_end
 define void @coro_end(i8* %0, i1 %1) {
-  ; CHECK:  llvm.intr.coro.end 
+  ; CHECK:  llvm.intr.coro.end
   call i1 @llvm.coro.end(i8* %0, i1 %1)
   ret void
 }
 
 ; CHECK-LABEL:  llvm.func @coro_free
 define void @coro_free(i32 %0, i8* %1) {
   ; CHECK: llvm.intr.coro.id %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}} : !llvm.token
   %3 = call token @llvm.coro.id(i32 %0, i8* %1, i8* %1, i8* null)

 
 ; CHECK-LABEL:  llvm.func @stack_restore
 define void @stack_restore(i8* %0) {
   ; CHECK: llvm.intr.stackrestore %{{.*}}
   call void @llvm.stackrestore(i8* %0)
   ret void
 }
 
+; CHECK-LABEL: llvm.func @lifetime
+define void @lifetime(i8* %0) {
+  ; CHECK: llvm.intr.lifetime.start 16, %{{.*}} : !llvm.ptr<i8>
+  call void @llvm.lifetime.start.p0i8(i64 16, i8* %0)
+  ; CHECK: llvm.intr.lifetime.end 32, %{{.*}} : !llvm.ptr<i8>
+  call void @llvm.lifetime.end.p0i8(i64 32, i8* %0)
+  ret void
+}
+
 ; CHECK-LABEL:  llvm.func @vector_predication_intrinsics
 define void @vector_predication_intrinsics(<8 x i32> %0, <8 x i32> %1, <8 x float> %2, <8 x float> %3, <8 x i64> %4, <8 x double> %5, <8 x i32*> %6, i32 %7, float %8, i32* %9, float* %10, <8 x i1> %11, i32 %12) {
   ; CHECK: "llvm.intr.vp.add"(%{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) : (vector<8xi32>, vector<8xi32>, vector<8xi1>, i32) -> vector<8xi32>
   %14 = call <8 x i32> @llvm.vp.add.v8i32(<8 x i32> %0, <8 x i32> %1, <8 x i1> %11, i32 %12)
-  ; CHECK: "llvm.intr.vp.sub"(%{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) : (vector<8xi32>, vector<8xi32>, vector<8xi1>, i32) -> vector<8xi32>  
+  ; CHECK: "llvm.intr.vp.sub"(%{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) : (vector<8xi32>, vector<8xi32>, vector<8xi1>, i32) -> vector<8xi32>
   %15 = call <8 x i32> @llvm.vp.sub.v8i32(<8 x i32> %0, <8 x i32> %1, <8 x i1> %11, i32 %12)
   ; CHECK: "llvm.intr.vp.mul"(%{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) : (vector<8xi32>, vector<8xi32>, vector<8xi1>, i32) -> vector<8xi32>
   %16 = call <8 x i32> @llvm.vp.mul.v8i32(<8 x i32> %0, <8 x i32> %1, <8 x i1> %11, i32 %12)
   ; CHECK: "llvm.intr.vp.sdiv"(%{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) : (vector<8xi32>, vector<8xi32>, vector<8xi1>, i32) -> vector<8xi32>
   %17 = call <8 x i32> @llvm.vp.sdiv.v8i32(<8 x i32> %0, <8 x i32> %1, <8 x i1> %11, i32 %12)
   ; CHECK: "llvm.intr.vp.udiv"(%{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) : (vector<8xi32>, vector<8xi32>, vector<8xi1>, i32) -> vector<8xi32>
   %18 = call <8 x i32> @llvm.vp.udiv.v8i32(<8 x i32> %0, <8 x i32> %1, <8 x i1> %11, i32 %12)
   ; CHECK: "llvm.intr.vp.srem"(%{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) : (vector<8xi32>, vector<8xi32>, vector<8xi1>, i32) -> vector<8xi32>
define void @vector_predication_intrinsics(<8 x i32> %0, <8 x i32> %1, <8 x float> %2, <8 x float> %3, <8 x i64> %4, <8 x double> %5, <8 x i32*> %6, i32 %7, float %8, i32* %9, float* %10, <8 x i1> %11, i32 %12) {
   ; CHECK: "llvm.intr.vp.fpext"(%{{.*}}, %{{.*}}, %{{.*}}) : (vector<8xf32>, vector<8xi1>, i32) -> vector<8xf64>
   %55 = call <8 x double> @llvm.vp.fpext.v8f64.v8f32(<8 x float> %2, <8 x i1> %11, i32 %12)
   ; CHECK: "llvm.intr.vp.fptoui"(%{{.*}}, %{{.*}}, %{{.*}}) : (vector<8xf64>, vector<8xi1>, i32) -> vector<8xi64>
   %56 = call <8 x i64> @llvm.vp.fptoui.v8i64.v8f64(<8 x double> %5, <8 x i1> %11, i32 %12)
   ; CHECK: "llvm.intr.vp.fptosi"(%{{.*}}, %{{.*}}, %{{.*}}) : (vector<8xf64>, vector<8xi1>, i32) -> vector<8xi64>
   %57 = call <8 x i64> @llvm.vp.fptosi.v8i64.v8f64(<8 x double> %5, <8 x i1> %11, i32 %12)
   ; CHECK: "llvm.intr.vp.ptrtoint"(%{{.*}}, %{{.*}}, %{{.*}}) : (!llvm.vec<8 x ptr<i32>>, vector<8xi1>, i32) -> vector<8xi64>
   %58 = call <8 x i64> @llvm.vp.ptrtoint.v8i64.v8p0i32(<8 x i32*> %6, <8 x i1> %11, i32 %12)
-  ; CHECK: "llvm.intr.vp.inttoptr"(%{{.*}}, %{{.*}}, %{{.*}}) : (vector<8xi64>, vector<8xi1>, i32) -> !llvm.vec<8 x ptr<i32>>  
+  ; CHECK: "llvm.intr.vp.inttoptr"(%{{.*}}, %{{.*}}, %{{.*}}) : (vector<8xi64>, vector<8xi1>, i32) -> !llvm.vec<8 x ptr<i32>>
   %59 = call <8 x i32*> @llvm.vp.inttoptr.v8p0i32.v8i64(<8 x i64> %4, <8 x i1> %11, i32 %12)
   ret void
 }
 
 declare float @llvm.fmuladd.f32(float, float, float)
 declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>)
 declare float @llvm.fma.f32(float, float, float)
 declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>)

 declare float @llvm.vector.reduce.fmul.v8f32(float, <8 x float>)
 declare i32 @llvm.vector.reduce.xor.v8i32(<8 x i32>)
 declare <12 x float> @llvm.matrix.multiply.v12f32.v64f32.v48f32(<64 x float>, <48 x float>, i32 immarg, i32 immarg, i32 immarg)
 declare <48 x float> @llvm.matrix.transpose.v48f32(<48 x float>, i32 immarg, i32 immarg)
 declare <48 x float> @llvm.matrix.column.major.load.v48f32.i64(float* nocapture, i64, i1 immarg, i32 immarg, i32 immarg)
 declare void @llvm.matrix.column.major.store.v48f32.i64(<48 x float>, float* nocapture writeonly, i64, i1 immarg, i32 immarg, i32 immarg)
 declare <7 x i1> @llvm.get.active.lane.mask.v7i1.i64(i64, i64)
 declare <7 x float> @llvm.masked.load.v7f32.p0v7f32(<7 x float>*, i32 immarg, <7 x i1>, <7 x float>)
-declare void @llvm.masked.store.v7f32.p0v7f32(<7 x float>, <7 x float>*, i32 immarg, <7 x i1>) 
+declare void @llvm.masked.store.v7f32.p0v7f32(<7 x float>, <7 x float>*, i32 immarg, <7 x i1>)
 declare <7 x float> @llvm.masked.gather.v7f32.v7p0f32(<7 x float*>, i32 immarg, <7 x i1>, <7 x float>)
 declare void @llvm.masked.scatter.v7f32.v7p0f32(<7 x float>, <7 x float*>, i32 immarg, <7 x i1>)
 declare <7 x float> @llvm.masked.expandload.v7f32(float*, <7 x i1>, <7 x float>)
 declare void @llvm.masked.compressstore.v7f32(<7 x float>, float*, <7 x i1>)
 declare void @llvm.memcpy.p0i8.p0i8.i32(i8* noalias nocapture writeonly, i8* noalias nocapture readonly, i32, i1 immarg)
 declare void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* noalias nocapture writeonly, i8* noalias nocapture readonly, i64 immarg, i1 immarg)
 declare void @llvm.memmove.p0i8.p0i8.i32(i8* nocapture writeonly, i8* nocapture readonly, i32, i1 immarg)
 declare void @llvm.memset.p0i8.i32(i8* nocapture writeonly, i8, i32, i1 immarg)

 declare <8 x i64> @llvm.vp.zext.v8i64.v8i32(<8 x i32>, <8 x i1>, i32)
 declare <8 x i64> @llvm.vp.sext.v8i64.v8i32(<8 x i32>, <8 x i1>, i32)
 declare <8 x float> @llvm.vp.fptrunc.v8f32.v8f64(<8 x double>, <8 x i1>, i32)
 declare <8 x double> @llvm.vp.fpext.v8f64.v8f32(<8 x float>, <8 x i1>, i32)
 declare <8 x i64> @llvm.vp.fptoui.v8i64.v8f64(<8 x double>, <8 x i1>, i32)
 declare <8 x i64> @llvm.vp.fptosi.v8i64.v8f64(<8 x double>, <8 x i1>, i32)
 declare <8 x i64> @llvm.vp.ptrtoint.v8i64.v8p0i32(<8 x i32*>, <8 x i1>, i32)
 declare <8 x i32*> @llvm.vp.inttoptr.v8p0i32.v8i64(<8 x i64>, <8 x i1>, i32)
+declare void @llvm.lifetime.start.p0i8(i64 immarg, i8* nocapture)
+declare void @llvm.lifetime.end.p0i8(i64 immarg, i8* nocapture)

mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp

 //===- LLVMIRConversionGen.cpp - MLIR LLVM IR builder generator -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file uses tablegen definitions of the LLVM IR Dialect operations to
 // generate the code building the LLVM IR from it.
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Support/LogicalResult.h"
+#include "mlir/TableGen/Argument.h"
 #include "mlir/TableGen/Attribute.h"
 #include "mlir/TableGen/GenInfo.h"
 #include "mlir/TableGen/Operator.h"
 
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/TableGen/Record.h"
 #include "llvm/TableGen/TableGenBackend.h"
 
 using namespace llvm;
 using namespace mlir;
 
-static bool emitError(const Twine &message) {
+static LogicalResult emitError(const Twine &message) {
   llvm::errs() << message << "\n";
-  return false;
+  return failure();
 }
 
 namespace {
 // Helper structure to return a position of the substring in a string.
 struct StringLoc {
   size_t pos;
   size_t length;
 

 // Check if `name` is a known name of an operand of `op`.
 static bool isOperandName(const tblgen::Operator &op, StringRef name) {
   for (int i = 0, e = op.getNumOperands(); i < e; ++i)
     if (op.getOperand(i).name == name)
       return true;
   return false;
 }
 
+// Return the `op` argument index of the argument with the given `name`.
+static FailureOr<int> getArgumentIndex(const tblgen::Operator &op,
+                                       StringRef name) {
+  for (int i = 0, e = op.getNumArgs(); i != e; ++i)
+    if (op.getArgName(i) == name)
+      return i;
+  return failure();
+}
+
 // Emit to `os` the operator-name driven check and the call to LLVM IRBuilder
-// for one definition of an LLVM IR Dialect operation.  Return true on success.
-static bool emitOneBuilder(const Record &record, raw_ostream &os) {
+// for one definition of an LLVM IR Dialect operation.
+static LogicalResult emitOneBuilder(const Record &record, raw_ostream &os) {
   auto op = tblgen::Operator(record);
 
   if (!record.getValue("llvmBuilder"))
     return emitError("no 'llvmBuilder' field for op " + op.getOperationName());
 
   // Return early if there is no builder specified.
-  auto builderStrRef = record.getValueAsString("llvmBuilder");
+  StringRef builderStrRef = record.getValueAsString("llvmBuilder");
   if (builderStrRef.empty())
-    return true;
+    return success();
 
   // Progressively create the builder string by replacing $-variables with
   // value lookups.  Keep only the not-yet-traversed part of the builder pattern
   // to avoid re-traversing the string multiple times.
   std::string builder;
   llvm::raw_string_ostream bs(builder);
-  while (auto loc = findNextVariable(builderStrRef)) {
+  while (StringLoc loc = findNextVariable(builderStrRef)) {
     auto name = loc.in(builderStrRef).drop_front();
     auto getterName = op.getGetterName(name);
     // First, insert the non-matched part as is.
     bs << builderStrRef.substr(0, loc.pos);
     // Then, rewrite the name based on its kind.
     bool isVariadicOperand = isVariadicOperandName(op, name);
     if (isOperandName(op, name)) {
       auto result =
static LogicalResult emitOneBuilder(const Record &record, raw_ostream &os) {
 
   // Output the check and the rewritten builder string.
   os << "if (auto op = dyn_cast<" << op.getQualCppClassName()
      << ">(opInst)) {\n";
   os << bs.str() << builderStrRef << "\n";
   os << "  return success();\n";
   os << "}\n";
 
-  return true;
+  return success();
 }
 
 // Emit all builders.  Returns false on success because of the generator
 // registration requirements.
 static bool emitBuilders(const RecordKeeper &recordKeeper, raw_ostream &os) {
-  for (const auto *def : recordKeeper.getAllDerivedDefinitions("LLVM_OpBase")) {
-    if (!emitOneBuilder(*def, os))
+  for (const Record *def :
+       recordKeeper.getAllDerivedDefinitions("LLVM_OpBase")) {
+    if (failed(emitOneBuilder(*def, os)))
+      return true;
+  }
+  return false;
+}
+
+// Emit an intrinsic identifier driven check and a call to the builder of the
+// MLIR LLVM dialect intrinsic operation to build for the given LLVM IR
+// intrinsic identifier.
+static LogicalResult emitOneIntrBuilder(const Record &record, raw_ostream &os) {
+  auto op = tblgen::Operator(record);
+
+  if (!record.getValue("mlirBuilder"))
+    return emitError("no 'mlirBuilder' field for op " + op.getOperationName());
+
+  // Return early if there is no builder specified.
+  StringRef builderStrRef = record.getValueAsString("mlirBuilder");
+  if (builderStrRef.empty())
+    return success();
+
+  // Progressively create the builder string by replacing $-variables. Keep only
+  // the not-yet-traversed part of the builder pattern to avoid re-traversing
+  // the string multiple times.
+  std::string builder;
+  llvm::raw_string_ostream bs(builder);
+  while (StringLoc loc = findNextVariable(builderStrRef)) {
+    auto name = loc.in(builderStrRef).drop_front();
+    // First, insert the non-matched part as is.
+    bs << builderStrRef.substr(0, loc.pos);
+    // Then, rewrite the name based on its kind.
+    FailureOr<int> argIndex = getArgumentIndex(op, name);
+    if (succeeded(argIndex)) {
+      // Process the argument value assuming the MLIR and LLVM operand orders
+      // match and there are no optional or variadic arguments.
+      bs << formatv("processValue(callInst->getArgOperand({0}))", *argIndex);
+    } else if (name == "_int_attr") {
+      bs << "matchIntegerAttr";
+    } else if (name == "_resultType") {
+      bs << "convertType(callInst->getType())";
+    } else if (name == "_location") {
+      bs << "translateLoc(callInst->getDebugLoc())";
+    } else if (name == "_builder") {
+      bs << "odsBuilder";
+    } else if (name == "_qualCppClassName") {
+      bs << op.getQualCppClassName();
+    } else if (name == "$") {
+      bs << '$';
+    } else {
+      return emitError(name +
+                       " is neither a known keyword nor an argument of " +
+                       op.getOperationName());
+    }
+    // Finally, only keep the untraversed part of the string.
+    builderStrRef = builderStrRef.substr(loc.pos + loc.length);
+  }
+
+  // Output the check and the builder string.
+  os << "if (intrinsicID == llvm::Intrinsic::"
+     << record.getValueAsString("llvmEnumName") << ") {\n";
+  os << bs.str() << builderStrRef << "\n";
+  os << "  return success();\n";
+  os << "}\n";
+
+  return success();
+}
+
+// Emit all intrinsic builders. Returns false on success because of the
+// generator registration requirements.
+static bool emitIntrBuilders(const RecordKeeper &recordKeeper,
+                             raw_ostream &os) {
+  for (const Record *def :
+       recordKeeper.getAllDerivedDefinitions("LLVM_IntrOpBase")) {
+    if (failed(emitOneIntrBuilder(*def, os)))
       return true;
   }
   return false;
 }
 
 namespace {
 // Wrapper class around a Tablegen definition of an LLVM enum attribute case.
 class LLVMEnumAttrCase : public tblgen::EnumAttrCase {
 public:
   using tblgen::EnumAttrCase::EnumAttrCase;
 
   // Constructs a case from a non LLVM-specific enum attribute case.
   explicit LLVMEnumAttrCase(const tblgen::EnumAttrCase &other)
       : tblgen::EnumAttrCase(&other.getDef()) {}
 
   // Returns the C++ enumerant for the LLVM API.
   StringRef getLLVMEnumerant() const {
     return def->getValueAsString("llvmEnumerant");
   }
 };
 
 // Wraper class around a Tablegen definition of an LLVM enum attribute.
 class LLVMEnumAttr : public tblgen::EnumAttr {
 public:
   using tblgen::EnumAttr::EnumAttr;
 
   // Returns the C++ enum name for the LLVM API.
   StringRef getLLVMClassName() const {
     return def->getValueAsString("llvmClassName");
   }
 
   // Returns all associated cases viewed as LLVM-specific enum cases.
   std::vector<LLVMEnumAttrCase> getAllCases() const {
     std::vector<LLVMEnumAttrCase> cases;
 
     for (auto &c : tblgen::EnumAttr::getAllCases())
       cases.emplace_back(c);
 
     return cases;
   }
 };
 
 // Wraper class around a Tablegen definition of a C-style LLVM enum attribute.
 class LLVMCEnumAttr : public tblgen::EnumAttr {
 public:
   using tblgen::EnumAttr::EnumAttr;
 
   // Returns the C++ enum name for the LLVM API.
   StringRef getLLVMClassName() const {
     return def->getValueAsString("llvmClassName");
   }
 
   // Returns all associated cases viewed as LLVM-specific enum cases.
   std::vector<LLVMEnumAttrCase> getAllCases() const {
     std::vector<LLVMEnumAttrCase> cases;
 
     for (auto &c : tblgen::EnumAttr::getAllCases())
       cases.emplace_back(c);
 
     return cases;
   }
 };
 } // namespace
 
 // Emits conversion function "LLVMClass convertEnumToLLVM(Enum)" and containing
 // switch-based logic to convert from the MLIR LLVM dialect enum attribute case
 // (Enum) to the corresponding LLVM API enumerant
static void emitOneCEnumFromConversion(const llvm::Record *record,
   os << "}\n\n";
 }
 
 // Emits conversion functions between MLIR enum attribute case and corresponding
 // LLVM API enumerants for all registered LLVM dialect enum attributes.
 template <bool ConvertTo>
 static bool emitEnumConversionDefs(const RecordKeeper &recordKeeper,
                                    raw_ostream &os) {
-  for (const auto *def : recordKeeper.getAllDerivedDefinitions("LLVM_EnumAttr"))
+  for (const Record *def :
+       recordKeeper.getAllDerivedDefinitions("LLVM_EnumAttr"))
     if (ConvertTo)
       emitOneEnumToConversion(def, os);
     else
       emitOneEnumFromConversion(def, os);
 
-  for (const auto *def :
+  for (const Record *def :
        recordKeeper.getAllDerivedDefinitions("LLVM_CEnumAttr"))
     if (ConvertTo)
       emitOneCEnumToConversion(def, os);
     else
       emitOneCEnumFromConversion(def, os);
 
   return false;
 }
 
-static void emitIntrOpPair(const Record &record, raw_ostream &os) {
+static void emitOneIntrinsic(const Record &record, raw_ostream &os) {
   auto op = tblgen::Operator(record);
-  os << "{llvm::Intrinsic::" << record.getValueAsString("llvmEnumName") << ", "
-     << op.getQualCppClassName() << "::getOperationName()},\n";
+  os << "llvm::Intrinsic::" << record.getValueAsString("llvmEnumName") << ",\n";
 }
 
-static bool emitIntrOpPairs(const RecordKeeper &recordKeeper, raw_ostream &os) {
-  for (const auto *def :
+// Emit the list of LLVM IR intrinsics identifiers that are convertible to a
+// matching MLIR LLVM dialect intrinsic operation.
+static bool emitConvertibleIntrinsics(const RecordKeeper &recordKeeper,
+                                      raw_ostream &os) {
+  for (const Record *def :
        recordKeeper.getAllDerivedDefinitions("LLVM_IntrOpBase"))
-    emitIntrOpPair(*def, os);
+    emitOneIntrinsic(*def, os);
 
   return false;
 }
 
 static mlir::GenRegistration
     genLLVMIRConversions("gen-llvmir-conversions",
                          "Generate LLVM IR conversions", emitBuilders);
 
 static mlir::GenRegistration
+    genIntrFromLLVMIRConversions("gen-intr-from-llvmir-conversions",
+                                 "Generate intrinsic conversions from LLVM IR",
+                                 emitIntrBuilders);
+
+static mlir::GenRegistration
     genEnumToLLVMConversion("gen-enum-to-llvmir-conversions",
                             "Generate conversions of EnumAttrs to LLVM IR",
                             emitEnumConversionDefs</*ConvertTo=*/true>);
 
 static mlir::GenRegistration
     genEnumFromLLVMConversion("gen-enum-from-llvmir-conversions",
                               "Generate conversions of EnumAttrs from LLVM IR",
                               emitEnumConversionDefs</*ConvertTo=*/false>);
 
-static mlir::GenRegistration
-    genLLVMIntrinsicToOpPairs("gen-llvmintrinsic-to-llvmirop-pairs",
-                              "Generate LLVM intrinsic to LLVMIR op pairs",
-                              emitIntrOpPairs);
+static mlir::GenRegistration genConvertibleLLVMIRIntrinsics(
+    "gen-convertible-llvmir-intrinsics",
+    "Generate list of convertible LLVM IR intrinsics",
+    emitConvertibleIntrinsics);