[flang][hlfir] move intrinsic lowering out of BufferizeHLFIR

[tblah]

Summary

This move is useful for a few reasons:

• It is easier to see what the intrinsic lowering is doing when the operations it creates are not immediately lowered
• When lowering a HLFIR intrinsic generates an operation which needs to be lowered by another pattern matcher in the same pass, MLIR will run that other substitution before validating and finalizing the original changes. This means that the erasure of operations is not yet visible to subsequent matchers, which hugely complicates transformations (in this case, hlfir.exprs cannot be rewritten because they are still used by the now-erased HLFIR intrinsic op.

[tblah]

Update with rebase on upstream (a header moved)

[jeanPerier]

I think the cause of the issue mentioned in the second bullet point is that the patterns were actually not using the OpAdaptor adaptor to get the operands as they should have in a full translation pass (sorry for not catching that, I always miss this....). E.g: sum.getArray() sould have been getBufferizedExprStorage(adaptor.getArray()). And the hlfir::AsExprOp in processReturnValue could have been repaced by packageBufferizedExpr(loc, builder, *resultEntity, mustBeFreed). This would have avoided using/introducing again hlfir.expr in the pass that gets rid of them. This would also address the first point in some way (but I agree the hlfir.expr operands would still also be transformed).

Now, I am not against having this in a separate pass for modularity. The down side is that this has to be a moduleOp pass again because of the runtime calls creation, and these passes cannot be multithreaded, so they are more expensive on multicore machines compiling big Fortran files (where cmake multithreading is not the most useful).

However, if this becomes a separate pass, then it should not be a conversion pass (using the OpConversionPattern): this should use an OpRewritePattern since the operation is rewritten with the a value with the same IR type as the previous operation result (and there is not need for the adaptor in this case).

I do not have a strong take. So I am OK with your patch if you make if an rewrite pass instead of a conversion, and I am also OK with keeping it in the same pass and use the adaptor/remove the created as_expr op.

flang/lib/Optimizer/HLFIR/Transforms/LowerHLFIRIntrinsics.cpp

2

There seems to be a clang-format issue in this file.

[tblah]

Thanks for review, that's very helpful to know.

I will keep it like this because I think it is easier to understand what is
going on when bufferization and intrinsic implementation are separated (in the
future some intrinsic calls might be implemented by local fortran functions like
in the simplify intrinsic pass). Calling the particular runtime methods we
currently use does bufferize expressions now, but that could change in the
future if the non-allocating runtime functions (e.g. MatmulDirect) are used.

Changes:

• clang-format (sort headers)
• Switch base class to mlir::OpRewritePattern

[tschuett]

flang/lib/Optimizer/HLFIR/Transforms/LowerHLFIRIntrinsics.cpp

262

Nit. You claim it to be a ModuleOp pass. Could please spell out the auto.

[tblah]

Changes:

• Spell out auto
• Allow unknown operations - this prevents the pass from having to add every MLIR dialect used in Flang (and from breaking when new ones are added).

[jeanPerier]

Thanks, LGTM. Please beware there is a bot failure, but I think it is unrelated to your patch (you could rebase if this was fixed to check that).

[tblah]

In D145502#4201384, @jeanPerier wrote:

Thanks, LGTM. Please beware there is a bot failure, but I think it is unrelated to your patch (you could rebase if this was fixed to check that).

Thanks for pointing this out. The failing test was a driver test for RISC-V and it passes on the later patches in this series so I think it is unrelated. I'll revert if there are buildbot failures post commit.

flang/include/flang/Optimizer/HLFIR/Passes.h

 #include <memory>
 
 namespace hlfir {
 #define GEN_PASS_DECL
 #include "flang/Optimizer/HLFIR/Passes.h.inc"
 
 std::unique_ptr<mlir::Pass> createConvertHLFIRtoFIRPass();
 std::unique_ptr<mlir::Pass> createBufferizeHLFIRPass();
+std::unique_ptr<mlir::Pass> createLowerHLFIRIntrinsicsPass();
 
 #define GEN_PASS_REGISTRATION
 #include "flang/Optimizer/HLFIR/Passes.h.inc"
 } // namespace hlfir
 
 #endif // FORTRAN_OPTIMIZER_HLFIR_PASSES_H

flang/include/flang/Optimizer/HLFIR/Passes.td

def ConvertHLFIRtoFIR : Pass<"convert-hlfir-to-fir", "::mlir::ModuleOp"> {
   let constructor = "hlfir::createConvertHLFIRtoFIRPass()";
 }
 
 def BufferizeHLFIR : Pass<"bufferize-hlfir", "::mlir::ModuleOp"> {
   let summary = "Convert HLFIR operations operating on hlfir.expr into operations on memory";
   let constructor = "hlfir::createBufferizeHLFIRPass()";
 }
 
+def LowerHLFIRIntrinsics : Pass<"lower-hlfir-intrinsics", "::mlir::ModuleOp"> {
+  let summary = "Lower HLFIR transformational intrinsic operations";
+  let constructor = "hlfir::createLowerHLFIRIntrinsicsPass()";
+}
+
 #endif //FORTRAN_DIALECT_HLFIR_PASSES

flang/lib/Optimizer/HLFIR/Transforms/BufferizeHLFIR.cpp

 // An hlfir.expr is translated to a tuple<variable address, cleanupflag>
 // where cleanupflag is set to true if storage for the expression was allocated
 // on the heap.
 //===----------------------------------------------------------------------===//
 
 #include "flang/Optimizer/Builder/Character.h"
 #include "flang/Optimizer/Builder/FIRBuilder.h"
 #include "flang/Optimizer/Builder/HLFIRTools.h"
-#include "flang/Optimizer/Builder/IntrinsicCall.h"
-#include "flang/Optimizer/Builder/MutableBox.h"
 #include "flang/Optimizer/Builder/Runtime/Assign.h"
 #include "flang/Optimizer/Builder/Todo.h"
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "flang/Optimizer/Dialect/FIROps.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
 #include "flang/Optimizer/Dialect/Support/FIRContext.h"
 #include "flang/Optimizer/HLFIR/HLFIRDialect.h"
 #include "flang/Optimizer/HLFIR/HLFIROps.h"
 #include "flang/Optimizer/HLFIR/Passes.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/Pass/Pass.h"
 #include "mlir/Pass/PassManager.h"
 #include "mlir/Transforms/DialectConversion.h"
 #include <mlir/Support/LogicalResult.h>
-#include <optional>
 
 namespace hlfir {
 #define GEN_PASS_DEF_BUFFERIZEHLFIR
 #include "flang/Optimizer/HLFIR/Passes.h.inc"
 } // namespace hlfir
 
 namespace {
 
matchAndRewrite(hlfir::ElementalOp elemental, OpAdaptor adaptor,
 
     mlir::Value bufferizedExpr =
         packageBufferizedExpr(loc, builder, temp, cleanup);
     rewriter.replaceOp(elemental, bufferizedExpr);
     return mlir::success();
   }
 };
 
-/// Base class for passes converting transformational intrinsic operations into
-/// runtime calls
-template <class OP>
-class HlfirIntrinsicConversion : public mlir::OpConversionPattern<OP> {
-  using mlir::OpConversionPattern<OP>::OpConversionPattern;
-
-protected:
-  struct IntrinsicArgument {
-    mlir::Value val; // allowed to be null if the argument is absent
-    mlir::Type desiredType;
-  };
-
-  /// Lower the arguments to the intrinsic: adding nesecarry boxing and
-  /// conversion to match the signature of the intrinsic in the runtime library.
-  llvm::SmallVector<fir::ExtendedValue, 3>
-  lowerArguments(mlir::Operation *op,
-                 const llvm::ArrayRef<IntrinsicArgument> &args,
-                 mlir::ConversionPatternRewriter &rewriter,
-                 const fir::IntrinsicArgumentLoweringRules *argLowering) const {
-    mlir::Location loc = op->getLoc();
-    fir::KindMapping kindMapping{rewriter.getContext()};
-    fir::FirOpBuilder builder{rewriter, kindMapping};
-
-    llvm::SmallVector<fir::ExtendedValue, 3> ret;
-
-    for (size_t i = 0; i < args.size(); ++i) {
-      mlir::Value arg = args[i].val;
-      mlir::Type desiredType = args[i].desiredType;
-      if (!arg) {
-        ret.emplace_back(fir::getAbsentIntrinsicArgument());
-        continue;
-      }
-      hlfir::Entity entity{arg};
-
-      fir::ArgLoweringRule argRules =
-          fir::lowerIntrinsicArgumentAs(*argLowering, i);
-      switch (argRules.lowerAs) {
-      case fir::LowerIntrinsicArgAs::Value: {
-        if (args[i].desiredType != arg.getType()) {
-          arg = builder.createConvert(loc, desiredType, arg);
-          entity = hlfir::Entity{arg};
-        }
-        auto [exv, cleanup] = hlfir::convertToValue(loc, builder, entity);
-        if (cleanup)
-          TODO(loc, "extended value cleanup");
-        ret.emplace_back(exv);
-      } break;
-      case fir::LowerIntrinsicArgAs::Addr: {
-        auto [exv, cleanup] =
-            hlfir::convertToAddress(loc, builder, entity, desiredType);
-        if (cleanup)
-          TODO(loc, "extended value cleanup");
-        ret.emplace_back(exv);
-      } break;
-      case fir::LowerIntrinsicArgAs::Box: {
-        auto [box, cleanup] =
-            hlfir::convertToBox(loc, builder, entity, desiredType);
-        if (cleanup)
-          TODO(loc, "extended value cleanup");
-        ret.emplace_back(box);
-      } break;
-      case fir::LowerIntrinsicArgAs::Inquired: {
-        if (args[i].desiredType != arg.getType()) {
-          arg = builder.createConvert(loc, desiredType, arg);
-          entity = hlfir::Entity{arg};
-        }
-        // Place hlfir.expr in memory, and unbox fir.boxchar. Other entities
-        // are translated to fir::ExtendedValue without transofrmation (notably,
-        // pointers/allocatable are not dereferenced).
-        // TODO: once lowering to FIR retires, UBOUND and LBOUND can be
-        // simplified since the fir.box lowered here are now guarenteed to
-        // contain the local lower bounds thanks to the hlfir.declare (the extra
-        // rebox can be removed).
-        auto [exv, cleanup] =
-            hlfir::translateToExtendedValue(loc, builder, entity);
-        if (cleanup)
-          TODO(loc, "extended value cleanup");
-        ret.emplace_back(exv);
-      } break;
-      }
-    }
-
-    return ret;
-  }
-
-  void processReturnValue(mlir::Operation *op,
-                          const fir::ExtendedValue &resultExv, bool mustBeFreed,
-                          fir::FirOpBuilder &builder,
-                          mlir::PatternRewriter &rewriter) const {
-    mlir::Location loc = op->getLoc();
-
-    mlir::Value firBase = fir::getBase(resultExv);
-    mlir::Type firBaseTy = firBase.getType();
-
-    std::optional<hlfir::EntityWithAttributes> resultEntity;
-    if (fir::isa_trivial(firBaseTy)) {
-      resultEntity = hlfir::EntityWithAttributes{firBase};
-    } else {
-      resultEntity =
-          hlfir::genDeclare(loc, builder, resultExv, ".tmp.intrinsic_result",
-                            fir::FortranVariableFlagsAttr{});
-    }
-
-    if (resultEntity->isVariable()) {
-      hlfir::AsExprOp asExpr = builder.create<hlfir::AsExprOp>(
-          loc, *resultEntity, builder.createBool(loc, mustBeFreed));
-      resultEntity = hlfir::EntityWithAttributes{asExpr.getResult()};
-    }
-
-    rewriter.replaceOp(op, resultEntity->getBase());
-  }
-};
-
-struct SumOpConversion : public HlfirIntrinsicConversion<hlfir::SumOp> {
-  using HlfirIntrinsicConversion<hlfir::SumOp>::HlfirIntrinsicConversion;
-
-  mlir::LogicalResult
-  matchAndRewrite(hlfir::SumOp sum, OpAdaptor adaptor,
-                  mlir::ConversionPatternRewriter &rewriter) const override {
-    fir::KindMapping kindMapping{rewriter.getContext()};
-    fir::FirOpBuilder builder{rewriter, kindMapping};
-    const mlir::Location &loc = sum->getLoc();
-    HLFIRListener listener{builder, rewriter};
-    builder.setListener(&listener);
-
-    mlir::Type i32 = builder.getI32Type();
-    mlir::Type logicalType = fir::LogicalType::get(
-        builder.getContext(), builder.getKindMap().defaultLogicalKind());
-
-    llvm::SmallVector<IntrinsicArgument, 3> inArgs;
-    inArgs.push_back({sum.getArray(), sum.getArray().getType()});
-    inArgs.push_back({sum.getDim(), i32});
-    inArgs.push_back({sum.getMask(), logicalType});
-
-    auto *argLowering = fir::getIntrinsicArgumentLowering("sum");
-    llvm::SmallVector<fir::ExtendedValue, 3> args =
-        lowerArguments(sum, inArgs, rewriter, argLowering);
-
-    mlir::Type scalarResultType = hlfir::getFortranElementType(sum.getType());
-
-    auto [resultExv, mustBeFreed] =
-        fir::genIntrinsicCall(builder, loc, "sum", scalarResultType, args);
-
-    processReturnValue(sum, resultExv, mustBeFreed, builder, rewriter);
-    return mlir::success();
-  }
-};
-
-struct MatmulOpConversion : public HlfirIntrinsicConversion<hlfir::MatmulOp> {
-  using HlfirIntrinsicConversion<hlfir::MatmulOp>::HlfirIntrinsicConversion;
-
-  mlir::LogicalResult
-  matchAndRewrite(hlfir::MatmulOp matmul, OpAdaptor adaptor,
-                  mlir::ConversionPatternRewriter &rewriter) const override {
-    fir::KindMapping kindMapping{rewriter.getContext()};
-    fir::FirOpBuilder builder{rewriter, kindMapping};
-    const mlir::Location &loc = matmul->getLoc();
-    HLFIRListener listener{builder, rewriter};
-    builder.setListener(&listener);
-
-    mlir::Value lhs = matmul.getLhs();
-    mlir::Value rhs = matmul.getRhs();
-    llvm::SmallVector<IntrinsicArgument, 2> inArgs;
-    inArgs.push_back({lhs, lhs.getType()});
-    inArgs.push_back({rhs, rhs.getType()});
-
-    auto *argLowering = fir::getIntrinsicArgumentLowering("matmul");
-    llvm::SmallVector<fir::ExtendedValue, 2> args =
-        lowerArguments(matmul, inArgs, rewriter, argLowering);
-
-    mlir::Type scalarResultType =
-        hlfir::getFortranElementType(matmul.getType());
-
-    auto [resultExv, mustBeFreed] =
-        fir::genIntrinsicCall(builder, loc, "matmul", scalarResultType, args);
-
-    processReturnValue(matmul, resultExv, mustBeFreed, builder, rewriter);
-    return mlir::success();
-  }
-};
-
-class TransposeOpConversion
-    : public HlfirIntrinsicConversion<hlfir::TransposeOp> {
-  using HlfirIntrinsicConversion<hlfir::TransposeOp>::HlfirIntrinsicConversion;
-
-  mlir::LogicalResult
-  matchAndRewrite(hlfir::TransposeOp transpose, OpAdaptor adaptor,
-                  mlir::ConversionPatternRewriter &rewriter) const override {
-    fir::KindMapping kindMapping{rewriter.getContext()};
-    fir::FirOpBuilder builder{rewriter, kindMapping};
-    const mlir::Location &loc = transpose->getLoc();
-    HLFIRListener listener{builder, rewriter};
-    builder.setListener(&listener);
-
-    mlir::Value arg = transpose.getArray();
-    llvm::SmallVector<IntrinsicArgument, 1> inArgs;
-    inArgs.push_back({arg, arg.getType()});
-
-    auto *argLowering = fir::getIntrinsicArgumentLowering("transpose");
-    llvm::SmallVector<fir::ExtendedValue, 1> args =
-        lowerArguments(transpose, inArgs, rewriter, argLowering);
-
-    mlir::Type scalarResultType =
-        hlfir::getFortranElementType(transpose.getType());
-
-    auto [resultExv, mustBeFreed] = fir::genIntrinsicCall(
-        builder, loc, "transpose", scalarResultType, args);
-
-    processReturnValue(transpose, resultExv, mustBeFreed, builder, rewriter);
-    return mlir::success();
-  }
-};
-
 class BufferizeHLFIR : public hlfir::impl::BufferizeHLFIRBase<BufferizeHLFIR> {
 public:
   void runOnOperation() override {
     // TODO: make this a pass operating on FuncOp. The issue is that
     // FirOpBuilder helpers may generate new FuncOp because of runtime/llvm
     // intrinsics calls creation. This may create race conflict if the pass is
     // scheduled on FuncOp. A solution could be to provide an optional mutex
     // when building a FirOpBuilder and locking around FuncOp and GlobalOp
     // creation, but this needs a bit more thinking, so at this point the pass
     // is scheduled on the moduleOp.
     auto module = this->getOperation();
     auto *context = &getContext();
     mlir::RewritePatternSet patterns(context);
     patterns
         .insert<ApplyOpConversion, AsExprOpConversion, AssignOpConversion,
                 AssociateOpConversion, ConcatOpConversion, DestroyOpConversion,
                 ElementalOpConversion, EndAssociateOpConversion,
-                MatmulOpConversion, NoReassocOpConversion,
-                SetLengthOpConversion, SumOpConversion, TransposeOpConversion>(
-            context);
+                NoReassocOpConversion, SetLengthOpConversion>(context);
     mlir::ConversionTarget target(*context);
     target.addIllegalOp<hlfir::ApplyOp, hlfir::AssociateOp, hlfir::ElementalOp,
                         hlfir::EndAssociateOp, hlfir::SetLengthOp,
                         hlfir::YieldElementOp>();
     target.markUnknownOpDynamicallyLegal([](mlir::Operation *op) {
       return llvm::all_of(
                  op->getResultTypes(),
                  [](mlir::Type ty) { return !ty.isa<hlfir::ExprType>(); }) &&

flang/lib/Optimizer/HLFIR/Transforms/CMakeLists.txt

 get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
 
 add_flang_library(HLFIRTransforms
   BufferizeHLFIR.cpp
   ConvertToFIR.cpp
+  LowerHLFIRIntrinsics.cpp
 
   DEPENDS
   FIRDialect
   HLFIROpsIncGen
   ${dialect_libs}
 
   LINK_LIBS
   FIRDialect

flang/lib/Optimizer/HLFIR/Transforms/LowerHLFIRIntrinsics.cpp

+//===- LowerHLFIRIntrinsics.cpp - Bufferize HLFIR  ------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "flang/Optimizer/Builder/FIRBuilder.h"
+#include "flang/Optimizer/Builder/HLFIRTools.h"
+#include "flang/Optimizer/Builder/IntrinsicCall.h"
+#include "flang/Optimizer/Builder/Todo.h"
+#include "flang/Optimizer/Dialect/FIRDialect.h"
+#include "flang/Optimizer/Dialect/FIROps.h"
+#include "flang/Optimizer/Dialect/FIRType.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/HLFIR/HLFIRDialect.h"
+#include "flang/Optimizer/HLFIR/HLFIROps.h"
+#include "flang/Optimizer/HLFIR/Passes.h"
+#include "mlir/IR/BuiltinDialect.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Pass/Pass.h"
+#include "mlir/Pass/PassManager.h"
+#include "mlir/Support/LogicalResult.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include <mlir/IR/MLIRContext.h>
+#include <optional>
+
+namespace hlfir {
+#define GEN_PASS_DEF_LOWERHLFIRINTRINSICS
+#include "flang/Optimizer/HLFIR/Passes.h.inc"
+} // namespace hlfir
+
+namespace {
+
+/// Base class for passes converting transformational intrinsic operations into
+/// runtime calls
+template <class OP>
+class HlfirIntrinsicConversion : public mlir::OpRewritePattern<OP> {
+  using mlir::OpRewritePattern<OP>::OpRewritePattern;
+
+protected:
+  struct IntrinsicArgument {
+    mlir::Value val; // allowed to be null if the argument is absent
+    mlir::Type desiredType;
+  };
+
+  /// Lower the arguments to the intrinsic: adding nesecarry boxing and
+  /// conversion to match the signature of the intrinsic in the runtime library.
+  llvm::SmallVector<fir::ExtendedValue, 3>
+  lowerArguments(mlir::Operation *op,
+                 const llvm::ArrayRef<IntrinsicArgument> &args,
+                 mlir::PatternRewriter &rewriter,
+                 const fir::IntrinsicArgumentLoweringRules *argLowering) const {
+    mlir::Location loc = op->getLoc();
+    fir::KindMapping kindMapping{rewriter.getContext()};
+    fir::FirOpBuilder builder{rewriter, kindMapping};
+
+    llvm::SmallVector<fir::ExtendedValue, 3> ret;
+
+    for (size_t i = 0; i < args.size(); ++i) {
+      mlir::Value arg = args[i].val;
+      mlir::Type desiredType = args[i].desiredType;
+      if (!arg) {
+        ret.emplace_back(fir::getAbsentIntrinsicArgument());
+        continue;
+      }
+      hlfir::Entity entity{arg};
+
+      fir::ArgLoweringRule argRules =
+          fir::lowerIntrinsicArgumentAs(*argLowering, i);
+      switch (argRules.lowerAs) {
+      case fir::LowerIntrinsicArgAs::Value: {
+        if (args[i].desiredType != arg.getType()) {
+          arg = builder.createConvert(loc, desiredType, arg);
+          entity = hlfir::Entity{arg};
+        }
+        auto [exv, cleanup] = hlfir::convertToValue(loc, builder, entity);
+        if (cleanup)
+          TODO(loc, "extended value cleanup");
+        ret.emplace_back(exv);
+      } break;
+      case fir::LowerIntrinsicArgAs::Addr: {
+        auto [exv, cleanup] =
+            hlfir::convertToAddress(loc, builder, entity, desiredType);
+        if (cleanup)
+          TODO(loc, "extended value cleanup");
+        ret.emplace_back(exv);
+      } break;
+      case fir::LowerIntrinsicArgAs::Box: {
+        auto [box, cleanup] =
+            hlfir::convertToBox(loc, builder, entity, desiredType);
+        if (cleanup)
+          TODO(loc, "extended value cleanup");
+        ret.emplace_back(box);
+      } break;
+      case fir::LowerIntrinsicArgAs::Inquired: {
+        if (args[i].desiredType != arg.getType()) {
+          arg = builder.createConvert(loc, desiredType, arg);
+          entity = hlfir::Entity{arg};
+        }
+        // Place hlfir.expr in memory, and unbox fir.boxchar. Other entities
+        // are translated to fir::ExtendedValue without transofrmation (notably,
+        // pointers/allocatable are not dereferenced).
+        // TODO: once lowering to FIR retires, UBOUND and LBOUND can be
+        // simplified since the fir.box lowered here are now guarenteed to
+        // contain the local lower bounds thanks to the hlfir.declare (the extra
+        // rebox can be removed).
+        auto [exv, cleanup] =
+            hlfir::translateToExtendedValue(loc, builder, entity);
+        if (cleanup)
+          TODO(loc, "extended value cleanup");
+        ret.emplace_back(exv);
+      } break;
+      }
+    }
+
+    return ret;
+  }
+
+  void processReturnValue(mlir::Operation *op,
+                          const fir::ExtendedValue &resultExv, bool mustBeFreed,
+                          fir::FirOpBuilder &builder,
+                          mlir::PatternRewriter &rewriter) const {
+    mlir::Location loc = op->getLoc();
+
+    mlir::Value firBase = fir::getBase(resultExv);
+    mlir::Type firBaseTy = firBase.getType();
+
+    std::optional<hlfir::EntityWithAttributes> resultEntity;
+    if (fir::isa_trivial(firBaseTy)) {
+      resultEntity = hlfir::EntityWithAttributes{firBase};
+    } else {
+      resultEntity =
+          hlfir::genDeclare(loc, builder, resultExv, ".tmp.intrinsic_result",
+                            fir::FortranVariableFlagsAttr{});
+    }
+
+    if (resultEntity->isVariable()) {
+      hlfir::AsExprOp asExpr = builder.create<hlfir::AsExprOp>(
+          loc, *resultEntity, builder.createBool(loc, mustBeFreed));
+      resultEntity = hlfir::EntityWithAttributes{asExpr.getResult()};
+    }
+
+    mlir::Value base = resultEntity->getBase();
+    if (!mlir::isa<hlfir::ExprType>(base.getType())) {
+      for (mlir::Operation *use : op->getResult(0).getUsers()) {
+        if (mlir::isa<hlfir::DestroyOp>(use))
+          rewriter.eraseOp(use);
+      }
+    }
+    rewriter.replaceAllUsesWith(op->getResults(), {base});
+    rewriter.replaceOp(op, base);
+  }
+};
+
+struct SumOpConversion : public HlfirIntrinsicConversion<hlfir::SumOp> {
+  using HlfirIntrinsicConversion<hlfir::SumOp>::HlfirIntrinsicConversion;
+
+  mlir::LogicalResult
+  matchAndRewrite(hlfir::SumOp sum,
+                  mlir::PatternRewriter &rewriter) const override {
+    fir::KindMapping kindMapping{rewriter.getContext()};
+    fir::FirOpBuilder builder{rewriter, kindMapping};
+    const mlir::Location &loc = sum->getLoc();
+
+    mlir::Type i32 = builder.getI32Type();
+    mlir::Type logicalType = fir::LogicalType::get(
+        builder.getContext(), builder.getKindMap().defaultLogicalKind());
+
+    llvm::SmallVector<IntrinsicArgument, 3> inArgs;
+    inArgs.push_back({sum.getArray(), sum.getArray().getType()});
+    inArgs.push_back({sum.getDim(), i32});
+    inArgs.push_back({sum.getMask(), logicalType});
+
+    auto *argLowering = fir::getIntrinsicArgumentLowering("sum");
+    llvm::SmallVector<fir::ExtendedValue, 3> args =
+        lowerArguments(sum, inArgs, rewriter, argLowering);
+
+    mlir::Type scalarResultType = hlfir::getFortranElementType(sum.getType());
+
+    auto [resultExv, mustBeFreed] =
+        fir::genIntrinsicCall(builder, loc, "sum", scalarResultType, args);
+
+    processReturnValue(sum, resultExv, mustBeFreed, builder, rewriter);
+    return mlir::success();
+  }
+};
+
+struct MatmulOpConversion : public HlfirIntrinsicConversion<hlfir::MatmulOp> {
+  using HlfirIntrinsicConversion<hlfir::MatmulOp>::HlfirIntrinsicConversion;
+
+  mlir::LogicalResult
+  matchAndRewrite(hlfir::MatmulOp matmul,
+                  mlir::PatternRewriter &rewriter) const override {
+    fir::KindMapping kindMapping{rewriter.getContext()};
+    fir::FirOpBuilder builder{rewriter, kindMapping};
+    const mlir::Location &loc = matmul->getLoc();
+
+    mlir::Value lhs = matmul.getLhs();
+    mlir::Value rhs = matmul.getRhs();
+    llvm::SmallVector<IntrinsicArgument, 2> inArgs;
+    inArgs.push_back({lhs, lhs.getType()});
+    inArgs.push_back({rhs, rhs.getType()});
+
+    auto *argLowering = fir::getIntrinsicArgumentLowering("matmul");
+    llvm::SmallVector<fir::ExtendedValue, 2> args =
+        lowerArguments(matmul, inArgs, rewriter, argLowering);
+
+    mlir::Type scalarResultType =
+        hlfir::getFortranElementType(matmul.getType());
+
+    auto [resultExv, mustBeFreed] =
+        fir::genIntrinsicCall(builder, loc, "matmul", scalarResultType, args);
+
+    processReturnValue(matmul, resultExv, mustBeFreed, builder, rewriter);
+    return mlir::success();
+  }
+};
+
+class TransposeOpConversion
+    : public HlfirIntrinsicConversion<hlfir::TransposeOp> {
+  using HlfirIntrinsicConversion<hlfir::TransposeOp>::HlfirIntrinsicConversion;
+
+  mlir::LogicalResult
+  matchAndRewrite(hlfir::TransposeOp transpose,
+                  mlir::PatternRewriter &rewriter) const override {
+    fir::KindMapping kindMapping{rewriter.getContext()};
+    fir::FirOpBuilder builder{rewriter, kindMapping};
+    const mlir::Location &loc = transpose->getLoc();
+
+    mlir::Value arg = transpose.getArray();
+    llvm::SmallVector<IntrinsicArgument, 1> inArgs;
+    inArgs.push_back({arg, arg.getType()});
+
+    auto *argLowering = fir::getIntrinsicArgumentLowering("transpose");
+    llvm::SmallVector<fir::ExtendedValue, 1> args =
+        lowerArguments(transpose, inArgs, rewriter, argLowering);
+
+    mlir::Type scalarResultType =
+        hlfir::getFortranElementType(transpose.getType());
+
+    auto [resultExv, mustBeFreed] = fir::genIntrinsicCall(
+        builder, loc, "transpose", scalarResultType, args);
+
+    processReturnValue(transpose, resultExv, mustBeFreed, builder, rewriter);
+    return mlir::success();
+  }
+};
+
+class LowerHLFIRIntrinsics
+    : public hlfir::impl::LowerHLFIRIntrinsicsBase<LowerHLFIRIntrinsics> {
+public:
+  void runOnOperation() override {
+    // TODO: make this a pass operating on FuncOp. The issue is that
+    // FirOpBuilder helpers may generate new FuncOp because of runtime/llvm
+    // intrinsics calls creation. This may create race conflict if the pass is
+    // scheduled on FuncOp. A solution could be to provide an optional mutex
+    // when building a FirOpBuilder and locking around FuncOp and GlobalOp
+    // creation, but this needs a bit more thinking, so at this point the pass
+    // is scheduled on the moduleOp.
+    mlir::ModuleOp module = this->getOperation();
+    mlir::MLIRContext *context = &getContext();
+    mlir::RewritePatternSet patterns(context);
+    patterns.insert<MatmulOpConversion, SumOpConversion, TransposeOpConversion>(
+        context);
+    mlir::ConversionTarget target(*context);
+    target.addLegalDialect<mlir::BuiltinDialect, mlir::arith::ArithDialect,
+                           mlir::func::FuncDialect, fir::FIROpsDialect,
+                           hlfir::hlfirDialect>();
+    target.addIllegalOp<hlfir::MatmulOp, hlfir::SumOp, hlfir::TransposeOp>();
+    target.markUnknownOpDynamicallyLegal(
+        [](mlir::Operation *) { return true; });
+    if (mlir::failed(
+            mlir::applyFullConversion(module, target, std::move(patterns)))) {
+      mlir::emitError(mlir::UnknownLoc::get(context),
+                      "failure in HLFIR intrinsic lowering");
+      signalPassFailure();
+    }
+  }
+};
+} // namespace
+
+std::unique_ptr<mlir::Pass> hlfir::createLowerHLFIRIntrinsicsPass() {
+  return std::make_unique<LowerHLFIRIntrinsics>();
+}

flang/test/HLFIR/matmul-lowering.fir

 // Test hlfir.matmul operation lowering to fir runtime call
-// RUN: fir-opt %s -bufferize-hlfir | FileCheck %s
+// RUN: fir-opt %s -lower-hlfir-intrinsics | FileCheck %s
 
 func.func @_QPmatmul1(%arg0: !fir.box<!fir.array<?x?xi32>> {fir.bindc_name = "lhs"}, %arg1: !fir.box<!fir.array<?x?xi32>> {fir.bindc_name = "rhs"}, %arg2: !fir.box<!fir.array<?x?xi32>> {fir.bindc_name = "res"}) {
   %0:2 = hlfir.declare %arg0 {uniq_name = "_QFmatmul1Elhs"} : (!fir.box<!fir.array<?x?xi32>>) -> (!fir.box<!fir.array<?x?xi32>>, !fir.box<!fir.array<?x?xi32>>)
   %1:2 = hlfir.declare %arg2 {uniq_name = "_QFmatmul1Eres"} : (!fir.box<!fir.array<?x?xi32>>) -> (!fir.box<!fir.array<?x?xi32>>, !fir.box<!fir.array<?x?xi32>>)
   %2:2 = hlfir.declare %arg1 {uniq_name = "_QFmatmul1Erhs"} : (!fir.box<!fir.array<?x?xi32>>) -> (!fir.box<!fir.array<?x?xi32>>, !fir.box<!fir.array<?x?xi32>>)
   %3 = hlfir.matmul %0#0 %2#0 {fastmath = #arith.fastmath<contract>} : (!fir.box<!fir.array<?x?xi32>>, !fir.box<!fir.array<?x?xi32>>) -> !hlfir.expr<?x?xi32>
   hlfir.assign %3 to %1#0 : !hlfir.expr<?x?xi32>, !fir.box<!fir.array<?x?xi32>>
   hlfir.destroy %3 : !hlfir.expr<?x?xi32>

 // CHECK:         %[[NONE:.*]] = fir.call @_FortranAMatmul(%[[RET_ARG]], %[[LHS_ARG]], %[[RHS_ARG]], %[[LOC_STR:.*]], %[[LOC_N:.*]])
 
 // CHECK:         %[[RET:.*]] = fir.load %[[RET_BOX]]
 // CHECK-DAG:     %[[BOX_DIMS:.*]]:3 = fir.box_dims %[[RET]]
 // CHECK-DAG:     %[[ADDR:.*]] = fir.box_addr %[[RET]]
 // CHECK-NEXT:    %[[SHIFT:.*]] = fir.shape_shift %[[BOX_DIMS]]#0, %[[BOX_DIMS]]#1
 // TODO: fix alias analysis in hlfir.assign bufferization
 // CHECK-NEXT:    %[[TMP:.*]]:2 = hlfir.declare %[[ADDR]](%[[SHIFT]]) {uniq_name = ".tmp.intrinsic_result"}
-// CHECK:         %[[TUPLE0:.*]] = fir.undefined tuple<!fir.box<!fir.array<?x?xi32>>, i1>
-// CHECK:         %[[TUPLE1:.*]] = fir.insert_value %[[TUPLE0]], %[[TRUE:.*]], [1 : index]
-// CHECK:         %[[TUPLE2:.*]] = fir.insert_value %[[TUPLE1]], %[[TMP]]#0, [0 : index]
-// CHECK:         hlfir.assign %[[TMP]]#0 to %[[RES_VAR]]#0
-// CHECK:         fir.freemem %[[TMP]]#1
+// TODO: add shape information from original intrinsic op
+// CHECK:         %[[TRUE:.*]] = arith.constant true
+// CHECK:         %[[ASEXPR:.*]] = hlfir.as_expr %[[TMP]]#0 move %[[TRUE]] : (!fir.box<!fir.array<?x?xi32>>, i1) -> !hlfir.expr<?x?xi32>
+// CHECK:         hlfir.assign %[[ASEXPR]] to %[[RES_VAR]]#0
+// CHECK:         hlfir.destroy %[[ASEXPR]]
 // CHECK-NEXT:    return
 // CHECK-NEXT:  }

flang/test/HLFIR/sum-lowering.fir

 // Test hlfir.sum operation lowering to fir runtime call
-// RUN: fir-opt %s -bufferize-hlfir | FileCheck %s
+// RUN: fir-opt %s -lower-hlfir-intrinsics | FileCheck %s
 
 // simple one argument sum
 func.func @_QPsum1(%arg0: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "a"}, %arg1: !fir.ref<i32> {fir.bindc_name = "s"}) {
   %0:2 = hlfir.declare %arg0 {uniq_name = "_QFsum1Ea"} : (!fir.box<!fir.array<?xi32>>) -> (!fir.box<!fir.array<?xi32>>, !fir.box<!fir.array<?xi32>>)
   %1:2 = hlfir.declare %arg1 {uniq_name = "_QFsum1Es"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
   %2 = hlfir.sum %0#0 {fastmath = #arith.fastmath<contract>} : (!fir.box<!fir.array<?xi32>>) -> !hlfir.expr<i32>
   hlfir.assign %2 to %1#0 : !hlfir.expr<i32>, !fir.ref<i32>
   hlfir.destroy %2 : !hlfir.expr<i32>

 // CHECK:         %[[NONE:.*]] = fir.call @_FortranASumDim(%[[RET_ARG]], %[[ARRAY_ARG]], %[[DIM]], %[[LOC_STR:.*]], %[[LOC_N:.*]], %[[MASK_ARG]]) : (!fir.ref<!fir.box<none>>, !fir.box<none>, i32, !fir.ref<i8>, i32, !fir.box<none>) -> none
 
 // CHECK:         %[[RET:.*]] = fir.load %[[RET_BOX]]
 // CHECK:         %[[BOX_DIMS:.*]]:3 = fir.box_dims %[[RET]]
 // CHECK-NEXT:    %[[ADDR:.*]] = fir.box_addr %[[RET]]
 // CHECK-NEXT:    %[[SHIFT:.*]] = fir.shape_shift %[[BOX_DIMS]]#0, %[[BOX_DIMS]]#1
 // TODO: fix alias analysis in hlfir.assign bufferization
 // CHECK-NEXT:    %[[TMP:.*]]:2 = hlfir.declare %[[ADDR]](%[[SHIFT]]) {uniq_name = ".tmp.intrinsic_result"}
-// CHECK:         %[[TUPLE0:.*]] = fir.undefined tuple<!fir.box<!fir.array<?xi32>>, i1>
-// CHECK:         %[[TUPLE1:.*]] = fir.insert_value %[[TUPLE0]], %[[TRUE:.*]], [1 : index]
-// CHECK:         %[[TUPLE2:.*]] fir.insert_value %[[TUPLE1]], %[[TMP]]#0, [0 : index]
-// CHECK:         hlfir.assign %[[TMP]]#0 to %[[RES]]#0
-// CHECK:         fir.freemem %[[TMP]]#1
+// CHECK:         %[[TRUE:.*]] = arith.constant true
+// CHECK:         %[[ASEXPR:.*]] = hlfir.as_expr %[[TMP]]#0 move %[[TRUE]] : (!fir.box<!fir.array<?xi32>>, i1) -> !hlfir.expr<?xi32>
+// CHECK:         hlfir.assign %[[ASEXPR]] to %[[RES]]#0
+// CHECK:         hlfir.destroy %[[ASEXPR]]
 // CHECK-NEXT:    return
 // CHECK-NEXT:  }
 
 // sum with scalar mask
 func.func @_QPsum3(%arg0: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "a"}, %arg1: !fir.ref<i32> {fir.bindc_name = "s"}, %arg2: !fir.ref<!fir.logical<4>> {fir.bindc_name = "m"}) {
   %0:2 = hlfir.declare %arg0 {uniq_name = "_QFsum3Ea"} : (!fir.box<!fir.array<?xi32>>) -> (!fir.box<!fir.array<?xi32>>, !fir.box<!fir.array<?xi32>>)
   %1:2 = hlfir.declare %arg2 {uniq_name = "_QFsum3Em"} : (!fir.ref<!fir.logical<4>>) -> (!fir.ref<!fir.logical<4>>, !fir.ref<!fir.logical<4>>)
   %2:2 = hlfir.declare %arg1 {uniq_name = "_QFsum3Es"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)

flang/test/HLFIR/transpose-lowering.fir

 // Test hlfir.transpose operation lowering to fir runtime call
-// RUN: fir-opt %s -bufferize-hlfir | FileCheck %s
+// RUN: fir-opt %s -lower-hlfir-intrinsics | FileCheck %s
 
 func.func @_QPtranspose1(%arg0: !fir.ref<!fir.array<1x2xi32>> {fir.bindc_name = "m"}, %arg1: !fir.ref<!fir.array<2x1xi32>> {fir.bindc_name = "res"}) {
   %c1 = arith.constant 1 : index
   %c2 = arith.constant 2 : index
   %0 = fir.shape %c1, %c2 : (index, index) -> !fir.shape<2>
   %1:2 = hlfir.declare %arg0(%0) {uniq_name = "_QFtranspose1Em"} : (!fir.ref<!fir.array<1x2xi32>>, !fir.shape<2>) -> (!fir.ref<!fir.array<1x2xi32>>, !fir.ref<!fir.array<1x2xi32>>)
   %c2_0 = arith.constant 2 : index
   %c1_1 = arith.constant 1 : index

 // CHECK:         %[[NONE:.*]] = fir.call @_FortranATranspose(%[[RET_ARG]], %[[M_ARG]], %[[LOC_STR:.*]], %[[LOC_N:.*]])
 
 // CHECK:         %[[RET:.*]] = fir.load %[[RET_BOX]]
 // CHECK-DAG:     %[[BOX_DIMS:.*]]:3 = fir.box_dims %[[RET]]
 // CHECK-DAG:     %[[ADDR:.*]] = fir.box_addr %[[RET]]
 // CHECK-NEXT:    %[[SHIFT:.*]] = fir.shape_shift %[[BOX_DIMS]]#0, %[[BOX_DIMS]]#1
 // TODO: fix alias analysis in hlfir.assign bufferization
 // CHECK-NEXT:    %[[TMP:.*]]:2 = hlfir.declare %[[ADDR]](%[[SHIFT]]) {uniq_name = ".tmp.intrinsic_result"}
-// CHECK:         %[[TUPLE0:.*]] = fir.undefined tuple<!fir.box<!fir.array<?x?xi32>>, i1>
-// CHECK:         %[[TUPLE1:.*]] = fir.insert_value %[[TUPLE0]], %[[TRUE:.*]], [1 : index]
-// CHECK:         %[[TUPLE2:.*]] = fir.insert_value %[[TUPLE1]], %[[TMP]]#0, [0 : index]
-// CHECK:         hlfir.assign %[[TMP]]#0 to %[[RES_VAR]]#0
-// CHECK:         fir.freemem %[[TMP]]#1
+// CHECK:         %[[TRUE:.*]] = arith.constant true
+// CHECK:         %[[ASEXPR:.*]] = hlfir.as_expr %[[TMP]]#0 move %[[TRUE]] : (!fir.box<!fir.array<?x?xi32>>, i1) -> !hlfir.expr<?x?xi32>
+// CHECK:         hlfir.assign %[[ASEXPR]] to %[[RES_VAR]]#0
+// CHECK:         hlfir.destroy %[[ASEXPR]]
 // CHECK-NEXT:    return
 // CHECK-NEXT:  }