[flang][hlfir] Add pass to inline elementals

[tblah]

Summary

Implement hlfir.elemental inlining as proposed in
flang/docs/HighLevelFIR.md.

This is a separate pass to make the code easier to understand. One
alternative would have been to modify the hlfir.elemental lowering in
the HLFIR bufferization pass.

Inlining is done by splicing the mlir operations form the operations
list of the old block to the new block. We cannot inline by cloning the
operations because this causes pending rewrites and erasures of these
operations not to be performed on the cloned operations.

A hlfir.elemental can only be inlined once; if there are more uses, the
existing bufferization is used instead. It is not possible to splice the
operations to be in multiple places at once - that would require inlining
by cloning the operations.

[tschuett]

flang/lib/Optimizer/HLFIR/Transforms/InlineElementals.cpp

37

anonymous namespace + static is redundant.

53

std::nullopt

70

I believe you first have to check the optional:

std::optional<...> uses = getTwoUses(elemental);
if (!uses)
  return mlir::failure();

auto [apply, destroy] = *uses;

[tblah]

flang/lib/Optimizer/HLFIR/Transforms/InlineElementals.cpp

37

The style guide says that it is hard to look at a function defined inside of an anonymous namespace and notice that it is static without scrolling: https://llvm.org/docs/CodingStandards.html#anonymous-namespaces

The style guide recommends only putting declarations in the anonymous namespace and then defining things separately. I did not follow that advice because it felt like overkill for such short methods.

In this case, it is obvious that the function is inside an anonymous namespace. Would you rather I move it outside of the namespace, or to remove the static specifier?

70

This is checked by MLIR when determining if this hlfir.elemental is legal after the current pass. See line 97

target.addDynamicallyLegalOp<hlfir::ElementalOp>(
     [](hlfir::ElementalOp elemental) { return !getTwoUses(elemental); });

This means that if getTwoUses would have returned a std::nullopt, the elemental operation will be legal (for the purposes of this pass) and so InlineElementalConversion::matchAndRewrite would not be called.

It is unfortunate that getTwoUses is called twice, but the MLIR mechanism to avoid this (adding getTwoUses as an analysis pass) feels like overkill for such a tiny function.

[jeanPerier]

Thanks a lot, looks great!

Just to document a few things here that justify this pass is OK:

This optimization assumes that no operations between the hlfir.elemental/hlfir.apply is affected by or affects the hlfir.elemental evaluation.

Fortran/Lowering justification:
This is true with the code produced by lowering given Fortran "10.1.4 Evaluation of operations" that says that an expression evaluation shall not impact/be impacted by other expression evaluation in the statement. HLFIR hlfir.expr are not exported in the code lowering statements other than assignment/forall/where: if the value are needed (e.g. by an associate construct) they are placed in memory.

What rules should the optimizations running on HLFIR respect so that this assumption remains valid:

This optimization is possible, provided we do not start propagating hlfir.elemental (like by blindly rewriting an hlfir.associate + hlfir.as_expr to a no-op (We are not doing this, we are doing the reverse in bufferization, which is OK)). Any motion of an hlfir.apply outside of this pass should likely place its old operand in memory if no further analysis.

The implicit rule exported from Fortran into HLFIR is: any operation between an hlfir.elemental and a related hlfir.apply, or indirect hlfir.apply (*) shall not affect or be affected by the evaluation of the hlfir.elemental.
(*) an indirect hlfir.apply of hlfir.elemental A is an hlfir.apply of an hlfir.elemental B that contains a direct or indirect hlir.apply to A.

Given "10.1.4" cannot be enforced at compile time, there is also no way this can be enforced in the HLFIR on the code generated by lowering (we cannot have an MLIR verifier here). But any compiler motion of hlfir.elemental and hlfir.apply should prove it is not going passed an operation with that would affect/be affected by the hlfir.elemental.

How can this rule be exported to MLIR and the passes we do not control:

I think we may want to remove the "NoMemoryEffect" and replace it by an interface where hlfir.apply memory effects are the same as the ones inside the related hlfir.expr producer just to be safe. That way, any motion of the hlfir.apply by some MLIR compliant pass that we do not control will have to verify it that the hlfir.elemental content could be moved too. After all, an hlfir.elemental is a bit like a call to a lambda here, so it makes sense to consider the lambda side effects when looking at the effect of a call to it.

If we hit unexpected issues with this approach, I think we will need to add a concept to strongly highlight the parts where hlfir.elemental can be inlined with no issues (the program parts describes in 10.1.4). This may have other benefits outside of hlfir.apply/hlfir.elemental case, like providing the ability to share pointer fir.box loads if they appear in several places in a same expression, even when they escape in a call inside that expressions. But I am a bit unclear with how to best do this. So best to focus on getting HLFIR complete feature wise before doing this.

If you agree with this description, could you:

• add the rule in italic above to hlfir.apply documentation, just so we do not forget about this?
• remove the "NoMemoryEffect" of hlfir::ApplyOp (we can deal with adding an DeclareOpInterfaceMethods<MemoryEffectsOpInterface> that says it has the side effect of its producer later. I am not sure it will pessimize HLFIR performance at that point to just drop it for now).

[jeanPerier]

flang/lib/Optimizer/HLFIR/Transforms/InlineElementals.cpp

70

I would still apply the suggestion here, although I agree there is no reason the result should change, the hlfir.apply are being rewritten (the hlfir.apply using an hlfir.elemental may have been inlined in another hlfir.elemental).
The logic to bail here in case of unforeseen bugs looks cheap.

[tschuett]

flang/lib/Optimizer/HLFIR/Transforms/InlineElementals.cpp

37

You could move getTwoUses above the anonymous namespace. With the Conversion and Pass classes you also diverge from the style guide.

[tschuett]

flang/lib/Optimizer/HLFIR/Transforms/InlineElementals.cpp

70

I am always surprised when you have a function returning an optional and you don't need to check for nullopt due to some side-channel knowledge. Why does it need to return an optional at all?

[tblah]

Thanks for the review so far. I have marked this WIP because I have realized it needs more work.

When compiling a long chain of elemental operations e.g.

subroutine reproducer(ain, aout)
  real, dimension(:) :: ain, aout
  aout = sqrt(ain * (ain - 1))
end subroutine

MLIR fails to erase one of the hlfir.elemental operations, thinking that there it is still used by a hlfir.apply. I believe original hlfir.apply will have been marked as erased so I suspect the erased status of operations in the body of one elemental is not carried over when those operations are cloned into another elemental. I suspect I am using MLIR wrong. I will update once I have a solution.

Removing any one of the elemental operations in the example above results in correct code. It does not matter which elemental operations are used.

flang/lib/Optimizer/HLFIR/Transforms/InlineElementals.cpp

70

Because on the first call to getTwoUses (to determine if the elemental operation is legal) it does need to be checked. getTwoUses was factored out to allow both of these calls to share the code.

I will check here with an assertion and write a test inlining elemental operations inside of elemental operations

[jeanPerier]

In D149258#4302444, @tblah wrote:

Thanks for the review so far. I have marked this WIP because I have realized it needs more work.
...
MLIR fails to erase one of the hlfir.elemental operations, thinking that there it is still used by a hlfir.apply. I believe original hlfir.apply will have been marked as erased so I suspect the erased status of operations in the body of one elemental is not carried over when those operations are cloned into another elemental. I suspect I am using MLIR wrong. I will update once I have a solution.

One reason this could failed is that using the fir::FirOpBuilder in rewrite pass prevents the pass rewriter from being notified on new operation being created. This is "OK in many cases because we replace patterns by IR that do not contain these patterns. But here what could happen is that MLIR is not working top-down, and that one of the call to hlfir::inlineElementalOp clone an hlfir.apply that has not been rewritten yet. This new hlfir.apply, because it is created without notifying the pass rewritter, will not be considered for rewrites.

The solution here is to notify the pass rewriter through a fir::FirOpBuilder listener. I had to do it in bufferization: see https://github.com/llvm/llvm-project/blob/683a6e1c9e5396f64086c07bec334a38acd0ec7a/flang/lib/Optimizer/HLFIR/Transforms/BufferizeHLFIR.cpp#L501

If that is indeed the issue, you may want to move the HLFIRListener in a header to share it.

[tblah]

Sorry for the delay in updating this.

Changes:

• Inline by splicing instead of cloning to get around a limitation (or my missunderstanding) in MLIR. Add a test for this.
• Better document assumptions made by the pass (TL;DR: the contents of the hlfir.elemental should be pure, which is already asserted by lowering code because hlfir.elemental's can be evaluated in arbitrary order).
• Fix nits from review

The MLIR problem is that MLIR does not actually erase or rewrite operations when
instructed to do so during a pass. Instead it stores a log of all of these
changes then batches them at the end of the pass. This log is indexed by the
address of the mlir::Operation which is mutated. If I erase an operation (e.g.
a hlfir.yeild operation from an inlined hlfir.elemental), then clone the
instructions in that block into another hlfir.elemental, a new hlfir.yeild
operation with a different address is created and so MLIR looses track of the
fact that the source operation was erased. I work around this by splicing
operations from the linked list of one block to the destination block: avoiding
allocating any new operations.

I suspect the problem here is that I am using MLIR incorrectly. I asked about
this on the MLIR discord channel but got no responses. Feedback is welcome.

[tblah]

Ping for review on the changes, especially the use of splicing for inlining.

[jeanPerier]

Thanks for finding the issue here!

After reading MLIR documentation a bit, I think we are using the wrong driver here. We are doing a canonicalization like pattern rewrite that we want fully applied before doing the next pattern. So the greedy-pattern-rewrite-driver is probably what is needed here ([1]). I tried to use it successfully with your previous code (see [2]), a few changes are needed in the expected output because the greedy driver does DCE and moves constant up in the regions.

[1] https://mlir.llvm.org/docs/PatternRewriter/#greedy-pattern-rewrite-driver
[2] https://github.com/jeanPerier/llvm-project/commit/d58fdcbe4cc2ab4d3065c3071a68412dc0442012

flang/test/HLFIR/inline-elemental.fir

85

Why did you use fir.array_load/fir.array_update/fir.array_merge_store operations here? Were they generated from some Fortran code with HLFIR enabled?

I do not think it is incorrect to have them used inside HLFIR, but it is not planned, so if they are currently generated somewhere in lowering to HLFIR, this could be a bug.

[tblah]

Thanks for the suggestion - that's hugeley helpful. I think a lot of Flang's passes ought to be using that driver.

Changes: drop the list splicing hack and use the greedy pattern rewrite driver.

[tblah]

flang/test/HLFIR/inline-elemental.fir

85

Don't worry, I was just lazy when I wrote the test. Copying some FIR code to get a do_loop over an array.

[jeanPerier]

Thanks!

flang/lib/Optimizer/HLFIR/Transforms/InlineElementals.cpp

70

What does rewriter.notifyMatchFailure do, I am not familiar with it?

[tblah]

flang/lib/Optimizer/HLFIR/Transforms/InlineElementals.cpp

70

It calls the notifyMatchFailure hook on the listener (which by default returns mlir::failure()), but api users can customize the listener to take other actions. For flang this will always inline to a mlir::failure(), but looking at upstream mlir and at the documentation [1] I think this is the idomatic thing to do.

[1] https://mlir.llvm.org/docs/PatternRewriter/

[tblah]

The CI failure looks like it is because of a CMake version mismatch.

[jeanPerier]

In D149258#4349114, @tblah wrote:

The CI failure looks like it is because of a CMake version mismatch.

Yes, it is not related to your patch, all patches are failing with this since yesterday or so.

flang/lib/Optimizer/HLFIR/Transforms/InlineElementals.cpp

70

Thanks for the explanation.

[vzakhari]

Hi @tblah, could you please take a look at this regression?

subroutine test
  interface
     function func(i,j,k)
       character(5),allocatable :: func(:,:,:)
     end function func
  end interface
  character(13),allocatable :: a(:,:,:)
  print *, (func(2,5,3)//reshape([(char(ichar('a')+n),n=1,2*5*3)], &
       & [2,5,3]))
end subroutine test

It fails verification after InlineElementals:

error: loc("test.f90":8:12): 'hlfir.no_reassoc' op requires the same type for all operands and results

The problem operation is:

%58 = "hlfir.concat"(%56, %57, %1) : (!fir.ref<!fir.char<1,5>>, !fir.ref<!fir.char<1>>, index) -> !hlfir.expr<!fir.char<1,6>>
%59 = "hlfir.no_reassoc"(%58) : (!hlfir.expr<!fir.char<1,6>>) -> !hlfir.expr<!fir.char<1,?>>

Please let me know if I need to create a github issue.

[tblah]

In D149258#4361630, @vzakhari wrote:

Hi @tblah, could you please take a look at this regression?

subroutine test
  interface
     function func(i,j,k)
       character(5),allocatable :: func(:,:,:)
     end function func
  end interface
  character(13),allocatable :: a(:,:,:)
  print *, (func(2,5,3)//reshape([(char(ichar('a')+n),n=1,2*5*3)], &
       & [2,5,3]))
end subroutine test

It fails verification after InlineElementals:

error: loc("test.f90":8:12): 'hlfir.no_reassoc' op requires the same type for all operands and results

The problem operation is:

%58 = "hlfir.concat"(%56, %57, %1) : (!fir.ref<!fir.char<1,5>>, !fir.ref<!fir.char<1>>, index) -> !hlfir.expr<!fir.char<1,6>>
%59 = "hlfir.no_reassoc"(%58) : (!hlfir.expr<!fir.char<1,6>>) -> !hlfir.expr<!fir.char<1,?>>

Please let me know if I need to create a github issue.

Thanks for reporting this. I have reproduced the issue and I will start work on this today.

flang/include/flang/Optimizer/HLFIR/HLFIROps.td

let description = [{
     based indices iterating over the elemental expression shape.
     Given these indices, the element value for the given iteration
     can be computed in the region and yielded with the hlfir.yield_element
     operation.
 
     The shape and typeparams operands represent the extents and type
     parameters of the resulting array value.
 
+    Currently there is no way to control the iteration order of a hlfir
+    elemental operation and so operations in the body of the elemental must
+    not have side effects. If this is changed, an attribute must be added so
+    that the elemental inlining pass can skip these impure elementals.
+
 
     Example: Y + X,  with Integer :: X(10, 20), Y(10,20)
     ```
       %0 = fir.shape %c10, %c20 : (index, index) -> !fir.shape<2>
       %5 = hlfir.elemental %0 : (!fir.shape<2>) -> !hlfir.expr<10x20xi32> {
       ^bb0(%i: index, %j: index):
         %6 = hlfir.designate %x (%i, %j)  : (!fir.ref<!fir.array<10x20xi32>>, index, index) -> !fir.ref<i32>
         %7 = hlfir.designate %y (%i, %j)  : (!fir.ref<!fir.array<10x20xi32>>, index, index) -> !fir.ref<i32>
def hlfir_YieldElementOp : hlfir_Op<"yield_element", [Terminator, HasParent<"ElementalOp">, Pure]> {
   let assemblyFormat = "$element_value attr-dict `:` type($element_value)";
 }
 
 def hlfir_ApplyOp : hlfir_Op<"apply", [NoMemoryEffect, AttrSizedOperandSegments]> {
   let summary = "get the element value of an expression";
   let description = [{
     Given an hlfir.expr array value, hlfir.apply allow retrieving
     the value for an element given one based indices.
+
     When hlfir.apply is used on an hlfir.elemental, and if the hlfir.elemental
     operation evaluation can be moved to the location of the hlfir.apply, it is
     as if the hlfir.elemental body was evaluated given the hlfir.apply indices.
+    Therefore, apply operations on hlfir.elemental expressions should be located
+    such that evaluating the hlfir.elemental at the position of the hlfir.apply
+    operation produces the same result as evaluating the hlfir.elemental at its
+    location in the instruction stream. Attention should be paid to
+    hlfir.elemental memory side effects (in practice these are unlikely).
+    "10.1.4 Evaluation of operations" says that expression evaluation shall not
+    impact/be impacted by other expression evaluation in the statement.
   }];
 
   let arguments = (ins hlfir_ExprType:$expr,
                    Variadic<Index>:$indices,
                    Variadic<AnyIntegerType>:$typeparams
                   );
   let results = (outs AnyFortranValue:$element_value);
 

flang/include/flang/Optimizer/HLFIR/Passes.h

 namespace hlfir {
 #define GEN_PASS_DECL
 #include "flang/Optimizer/HLFIR/Passes.h.inc"
 
 std::unique_ptr<mlir::Pass> createConvertHLFIRtoFIRPass();
 std::unique_ptr<mlir::Pass> createBufferizeHLFIRPass();
 std::unique_ptr<mlir::Pass> createLowerHLFIRIntrinsicsPass();
 std::unique_ptr<mlir::Pass> createSimplifyHLFIRIntrinsicsPass();
+std::unique_ptr<mlir::Pass> createInlineElementalsPass();
 std::unique_ptr<mlir::Pass> createLowerHLFIROrderedAssignmentsPass();
 
 #define GEN_PASS_REGISTRATION
 #include "flang/Optimizer/HLFIR/Passes.h.inc"
 } // namespace hlfir
 
 #endif // FORTRAN_OPTIMIZER_HLFIR_PASSES_H

flang/include/flang/Optimizer/HLFIR/Passes.td

def LowerHLFIROrderedAssignments : Pass<"lower-hlfir-ordered-assignments", "::mlir::ModuleOp"> {
   ];
 }
 
 def SimplifyHLFIRIntrinsics : Pass<"simplify-hlfir-intrinsics", "::mlir::func::FuncOp"> {
   let summary = "Simplify HLFIR intrinsic operations that don't need to result in runtime calls";
   let constructor = "hlfir::createSimplifyHLFIRIntrinsicsPass()";
 }
 
+def InlineElementals : Pass<"inline-elementals", "::mlir::func::FuncOp"> {
+  let summary = "Inline chained hlfir.elemental operations";
+  let constructor = "hlfir::createInlineElementalsPass()";
+}
+
 #endif //FORTRAN_DIALECT_HLFIR_PASSES

flang/include/flang/Tools/CLOptions.inc

 /// \param optLevel - optimization level used for creating FIR optimization
 ///   passes pipeline
 inline void createHLFIRToFIRPassPipeline(
     mlir::PassManager &pm, llvm::OptimizationLevel optLevel = defaultOptLevel) {
   if (optLevel.isOptimizingForSpeed()) {
     addCanonicalizerPassWithoutRegionSimplification(pm);
     pm.addPass(hlfir::createSimplifyHLFIRIntrinsicsPass());
   }
+  pm.addPass(hlfir::createInlineElementalsPass());
   pm.addPass(hlfir::createLowerHLFIROrderedAssignmentsPass());
   pm.addPass(hlfir::createLowerHLFIRIntrinsicsPass());
   pm.addPass(hlfir::createBufferizeHLFIRPass());
   pm.addPass(hlfir::createConvertHLFIRtoFIRPass());
 }
 
 #if !defined(FLANG_EXCLUDE_CODEGEN)
 inline void createDebugPasses(

flang/lib/Optimizer/HLFIR/Transforms/CMakeLists.txt

 get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
 
 add_flang_library(HLFIRTransforms
   BufferizeHLFIR.cpp
   ConvertToFIR.cpp
+  InlineElementals.cpp
   LowerHLFIRIntrinsics.cpp
   LowerHLFIROrderedAssignments.cpp
   ScheduleOrderedAssignments.cpp
   SimplifyHLFIRIntrinsics.cpp
 
   DEPENDS
   FIRDialect
   HLFIROpsIncGen

flang/lib/Optimizer/HLFIR/Transforms/InlineElementals.cpp

+//===- InlineElementals.cpp - Inline chained hlfir.elemental ops ----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// Chained elemental operations like a + b + c can inline the first elemental
+// at the hlfir.apply in the body of the second one (as described in
+// docs/HighLevelFIR.md). This has to be done in a pass rather than in lowering
+// so that it happens after the HLFIR intrinsic simplification pass.
+//===----------------------------------------------------------------------===//
+
+#include "flang/Optimizer/Builder/FIRBuilder.h"
+#include "flang/Optimizer/Builder/HLFIRTools.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
+#include "flang/Optimizer/HLFIR/HLFIROps.h"
+#include "flang/Optimizer/HLFIR/Passes.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Pass/Pass.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+#include "llvm/ADT/TypeSwitch.h"
+#include <iterator>
+
+namespace hlfir {
+#define GEN_PASS_DEF_INLINEELEMENTALS
+#include "flang/Optimizer/HLFIR/Passes.h.inc"
+} // namespace hlfir
+
+/// If the elemental has only two uses and those two are an apply operation and
+/// a destory operation, return those two, otherwise return {}
+static std::optional<std::pair<hlfir::ApplyOp, hlfir::DestroyOp>>
+getTwoUses(hlfir::ElementalOp elemental) {
+  mlir::Operation::user_range users = elemental->getUsers();
+  // don't inline anything with more than one use (plus hfir.destroy)
+  if (std::distance(users.begin(), users.end()) != 2) {
+    return std::nullopt;
+  }
+
+  hlfir::ApplyOp apply;
+  hlfir::DestroyOp destroy;
+  for (mlir::Operation *user : users)
+    mlir::TypeSwitch<mlir::Operation *, void>(user)
+        .Case([&](hlfir::ApplyOp op) { apply = op; })
+        .Case([&](hlfir::DestroyOp op) { destroy = op; });
+
+  if (!apply || !destroy)
+    return std::nullopt;
+  return std::pair{apply, destroy};
+}
+
+namespace {
+class InlineElementalConversion
+    : public mlir::OpRewritePattern<hlfir::ElementalOp> {
+public:
+  using mlir::OpRewritePattern<hlfir::ElementalOp>::OpRewritePattern;
+
+  mlir::LogicalResult
+  matchAndRewrite(hlfir::ElementalOp elemental,
+                  mlir::PatternRewriter &rewriter) const override {
+    std::optional<std::pair<hlfir::ApplyOp, hlfir::DestroyOp>> maybeTuple =
+        getTwoUses(elemental);
+    if (!maybeTuple) {
+      return rewriter.notifyMatchFailure(elemental.getLoc(),
+                                         [](mlir::Diagnostic &) {});
+    }
+    auto [apply, destroy] = *maybeTuple;
+
+    assert(elemental.getRegion().hasOneBlock() &&
+           "expect elemental region to have one block");
+
+    fir::FirOpBuilder builder{rewriter,
+                              fir::KindMapping{rewriter.getContext()}};
+    builder.setInsertionPointAfter(apply);
+    hlfir::YieldElementOp yield = hlfir::inlineElementalOp(
+        elemental.getLoc(), builder, elemental, apply.getIndices());
+
+    // remove the old elemental and all of the bookkeeping
+    rewriter.replaceAllUsesWith(apply.getResult(), yield.getElementValue());
+    rewriter.eraseOp(yield);
+    rewriter.eraseOp(apply);
+    rewriter.eraseOp(destroy);
+    rewriter.eraseOp(elemental);
+
+    return mlir::success();
+  }
+};
+
+class InlineElementalsPass
+    : public hlfir::impl::InlineElementalsBase<InlineElementalsPass> {
+public:
+  void runOnOperation() override {
+    mlir::func::FuncOp func = getOperation();
+    mlir::MLIRContext *context = &getContext();
+
+    mlir::GreedyRewriteConfig config;
+    // Prevent the pattern driver from merging blocks.
+    config.enableRegionSimplification = false;
+
+    mlir::RewritePatternSet patterns(context);
+    patterns.insert<InlineElementalConversion>(context);
+
+    if (mlir::failed(mlir::applyPatternsAndFoldGreedily(
+            func, std::move(patterns), config))) {
+      mlir::emitError(func->getLoc(), "failure in HLFIR elemental inlining");
+      signalPassFailure();
+    }
+  }
+};
+} // namespace
+
+std::unique_ptr<mlir::Pass> hlfir::createInlineElementalsPass() {
+  return std::make_unique<InlineElementalsPass>();
+}

flang/test/Driver/mlir-debug-pass-pipeline.f90

 ! DEBUG-DIRECTIVES: warning: Unsupported debug option: line-directives-only
 !
 ! DEBUG-ERR: error: invalid value 'invalid' in '-debug-info-kind=invalid'
 ! DEBUG-ERR-NOT: Pass statistics report
 
 ! ALL: Pass statistics report
 
 ! ALL: Fortran::lower::VerifierPass
+! ALL-NEXT: 'func.func' Pipeline
+! ALL-NEXT:   InlineElementals
 ! ALL-NEXT: LowerHLFIROrderedAssignments
 ! ALL-NEXT: LowerHLFIRIntrinsics
 ! ALL-NEXT: BufferizeHLFIR
 ! ALL-NEXT: ConvertHLFIRtoFIR
 ! ALL-NEXT: CSE
 ! Ideally, we need an output with only the pass names, but
 ! there is currently no way to get that, so in order to
 ! guarantee that the passes are in the expected order

flang/test/Driver/mlir-pass-pipeline.f90

 end program
 
 ! ALL: Pass statistics report
 
 ! ALL: Fortran::lower::VerifierPass
 ! O2-NEXT: Canonicalizer
 ! O2-NEXT: 'func.func' Pipeline
 ! O2-NEXT:   SimplifyHLFIRIntrinsics
-! ALL-NEXT: LowerHLFIROrderedAssignments
+! ALL:       InlineElementals
+! ALL: LowerHLFIROrderedAssignments
 ! ALL-NEXT: LowerHLFIRIntrinsics
 ! ALL-NEXT: BufferizeHLFIR
 ! ALL-NEXT: ConvertHLFIRtoFIR
 ! ALL-NEXT: CSE
 ! Ideally, we need an output with only the pass names, but
 ! there is currently no way to get that, so in order to
 ! guarantee that the passes are in the expected order
 ! (i.e. use -NEXT) we have to check the statistics output as well.

flang/test/Fir/basic-program.fir

 // CHECK-LABEL: define void @_QQmain()
 // CHECK:       ret void
 
 // PASSES: Pass statistics report
 
 // PASSES:        Canonicalizer
 // PASSES-NEXT: 'func.func' Pipeline
 // PASSES-NEXT:   SimplifyHLFIRIntrinsics
+// PASSES-NEXT:   InlineElementals
 // PASSES-NEXT:   LowerHLFIROrderedAssignments
 // PASSES-NEXT:   LowerHLFIRIntrinsics
 // PASSES-NEXT:   BufferizeHLFIR
 // PASSES-NEXT:   ConvertHLFIRtoFIR
 // PASSES-NEXT:   CSE
 // PASSES-NEXT:   (S) 0 num-cse'd - Number of operations CSE'd
 // PASSES-NEXT:   (S) 0 num-dce'd - Number of operations DCE'd
 

flang/test/HLFIR/inline-elemental.fir

+// RUN: fir-opt --inline-elementals %s | FileCheck %s
+
+// check inlining one elemental into another
+// a = b * c + d
+func.func @inline_to_elemental(%arg0: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "a"}, %arg1: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "b"}, %arg2: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "c"}, %arg3: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "d"}) {
+  %0:2 = hlfir.declare %arg0 {uniq_name = "a"} : (!fir.box<!fir.array<?xi32>>) -> (!fir.box<!fir.array<?xi32>>, !fir.box<!fir.array<?xi32>>)
+  %1:2 = hlfir.declare %arg1 {uniq_name = "b"} : (!fir.box<!fir.array<?xi32>>) -> (!fir.box<!fir.array<?xi32>>, !fir.box<!fir.array<?xi32>>)
+  %2:2 = hlfir.declare %arg2 {uniq_name = "c"} : (!fir.box<!fir.array<?xi32>>) -> (!fir.box<!fir.array<?xi32>>, !fir.box<!fir.array<?xi32>>)
+  %3:2 = hlfir.declare %arg3 {uniq_name = "d"} : (!fir.box<!fir.array<?xi32>>) -> (!fir.box<!fir.array<?xi32>>, !fir.box<!fir.array<?xi32>>)
+  %c0 = arith.constant 0 : index
+  %4:3 = fir.box_dims %1#0, %c0 : (!fir.box<!fir.array<?xi32>>, index) -> (index, index, index)
+  %5 = fir.shape %4#1 : (index) -> !fir.shape<1>
+  %6 = hlfir.elemental %5 : (!fir.shape<1>) -> !hlfir.expr<?xi32> {
+  ^bb0(%arg4: index):
+    %8 = hlfir.designate %1#0 (%arg4)  : (!fir.box<!fir.array<?xi32>>, index) -> !fir.ref<i32>
+    %9 = hlfir.designate %2#0 (%arg4)  : (!fir.box<!fir.array<?xi32>>, index) -> !fir.ref<i32>
+    %10 = fir.load %8 : !fir.ref<i32>
+    %11 = fir.load %9 : !fir.ref<i32>
+    %12 = arith.muli %10, %11 : i32
+    hlfir.yield_element %12 : i32
+  }
+  %7 = hlfir.elemental %5 : (!fir.shape<1>) -> !hlfir.expr<?xi32> {
+  ^bb0(%arg4: index):
+    %8 = hlfir.apply %6, %arg4 : (!hlfir.expr<?xi32>, index) -> i32
+    %9 = hlfir.designate %3#0 (%arg4)  : (!fir.box<!fir.array<?xi32>>, index) -> !fir.ref<i32>
+    %10 = fir.load %9 : !fir.ref<i32>
+    %11 = arith.addi %8, %10 : i32
+    hlfir.yield_element %11 : i32
+  }
+  hlfir.assign %7 to %0#0 : !hlfir.expr<?xi32>, !fir.box<!fir.array<?xi32>>
+  hlfir.destroy %7 : !hlfir.expr<?xi32>
+  hlfir.destroy %6 : !hlfir.expr<?xi32>
+  return
+}
+// CHECK-LABEL: func.func @inline_to_elemental
+// CHECK-SAME:      %[[A_ARG:.*]]: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "a"}
+// CHECK-SAME:      %[[B_ARG:.*]]: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "b"}
+// CHECK-SAME:      %[[C_ARG:.*]]: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "c"}
+// CHECK-SAME:      %[[D_ARG:.*]]: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "d"}
+// CHECK-NEXT:    %[[C0:.*]] = arith.constant 0 : index
+// CHECK-DAG:     %[[A:.*]]:2 = hlfir.declare %[[A_ARG]]
+// CHECK-DAG:     %[[B:.*]]:2 = hlfir.declare %[[B_ARG]]
+// CHECK-DAG:     %[[C:.*]]:2 = hlfir.declare %[[C_ARG]]
+// CHECK-DAG:     %[[D:.*]]:2 = hlfir.declare %[[D_ARG]]
+// CHECK-NEXT:    %[[B_DIM0:.*]]:3 = fir.box_dims %[[B]]#0, %[[C0]]
+// CHECK-NEXT:    %[[B_SHAPE:.*]] = fir.shape %[[B_DIM0]]#1
+// CHECK-NEXT:    %[[EXPR:.*]] = hlfir.elemental %[[B_SHAPE]]
+// CHECK-NEXT:    ^bb0(%[[I:.*]]: index):
+// inline the first elemental:
+// CHECK-NEXT:      %[[B_I_REF:.*]] = hlfir.designate %[[B]]#0 (%[[I]])
+// CHECK-NEXT:      %[[C_I_REF:.*]] = hlfir.designate %[[C]]#0 (%[[I]])
+// CHECK-NEXT:      %[[B_I:.*]] = fir.load %[[B_I_REF]]
+// CHECK-NEXT:      %[[C_I:.*]] = fir.load %[[C_I_REF]]
+// CHECK-NEXT:      %[[MUL:.*]] = arith.muli %[[B_I]], %[[C_I]]
+// second elemental:
+// CHECK-NEXT:      %[[D_I_REF:.*]] = hlfir.designate %[[D]]#0 (%[[I]])
+// CHECK-NEXT:      %[[D_I:.*]] = fir.load %[[D_I_REF]]
+// CHECK-NEXT:      %[[ADD:.*]] = arith.addi %[[MUL]], %[[D_I]]
+// CHECK-NEXT:      hlfir.yield_element %[[ADD]]
+// CHECK-NEXT:    }
+// CHECK-NEXT:    hlfir.assign %[[EXPR]] to %[[A]]#0
+// CHECK-NEXT:    hlfir.destroy %[[EXPR]]
+// CHECK-NEXT:    return
+// CHECK-NEXT:  }
+
+// check inlining into a do_loop
+func.func @inline_to_loop(%arg0: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "a"}, %arg1: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "b"}, %arg2: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "c"}, %arg3: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "d"}) {
+  %0:2 = hlfir.declare %arg0 {uniq_name = "a"} : (!fir.box<!fir.array<?xi32>>) -> (!fir.box<!fir.array<?xi32>>, !fir.box<!fir.array<?xi32>>)
+  %1:2 = hlfir.declare %arg1 {uniq_name = "b"} : (!fir.box<!fir.array<?xi32>>) -> (!fir.box<!fir.array<?xi32>>, !fir.box<!fir.array<?xi32>>)
+  %2:2 = hlfir.declare %arg2 {uniq_name = "c"} : (!fir.box<!fir.array<?xi32>>) -> (!fir.box<!fir.array<?xi32>>, !fir.box<!fir.array<?xi32>>)
+  %3:2 = hlfir.declare %arg3 {uniq_name = "d"} : (!fir.box<!fir.array<?xi32>>) -> (!fir.box<!fir.array<?xi32>>, !fir.box<!fir.array<?xi32>>)
+  %c0 = arith.constant 0 : index
+  %4:3 = fir.box_dims %1#0, %c0 : (!fir.box<!fir.array<?xi32>>, index) -> (index, index, index)
+  %5 = fir.shape %4#1 : (index) -> !fir.shape<1>
+  %6 = hlfir.elemental %5 : (!fir.shape<1>) -> !hlfir.expr<?xi32> {
+  ^bb0(%arg4: index):
+    %8 = hlfir.designate %1#0 (%arg4)  : (!fir.box<!fir.array<?xi32>>, index) -> !fir.ref<i32>
+    %9 = hlfir.designate %2#0 (%arg4)  : (!fir.box<!fir.array<?xi32>>, index) -> !fir.ref<i32>
+    %10 = fir.load %8 : !fir.ref<i32>
+    %11 = fir.load %9 : !fir.ref<i32>
+    %12 = arith.muli %10, %11 : i32
+    hlfir.yield_element %12 : i32
+  }
+  %array = fir.array_load %0#0 : (!fir.box<!fir.array<?xi32>>) -> !fir.array<?xi32>
+  %c1 = arith.constant 1 : index
+  %max = arith.subi %4#1, %c1 : index
+  %7 = fir.do_loop %arg4 = %c0 to %max step %c1 unordered iter_args(%arg5 = %array) -> (!fir.array<?xi32>) {
+    %8 = hlfir.apply %6, %arg4 : (!hlfir.expr<?xi32>, index) -> i32
+    %9 = hlfir.designate %3#0 (%arg4)  : (!fir.box<!fir.array<?xi32>>, index) -> !fir.ref<i32>
+    %10 = fir.load %9 : !fir.ref<i32>
+    %11 = arith.addi %8, %10 : i32
+    %12 = fir.array_update %arg5, %11, %arg4 : (!fir.array<?xi32>, i32, index) -> !fir.array<?xi32>
+    fir.result %12 : !fir.array<?xi32>
+  }
+  fir.array_merge_store %array, %7 to %arg0 : !fir.array<?xi32>, !fir.array<?xi32>, !fir.box<!fir.array<?xi32>>
+  hlfir.destroy %6 : !hlfir.expr<?xi32>
+  return
+}
+// CHECK-LABEL: func.func @inline_to_loop
+// CHECK-SAME:      %[[A_ARG:.*]]: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "a"}
+// CHECK-SAME:      %[[B_ARG:.*]]: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "b"}
+// CHECK-SAME:      %[[C_ARG:.*]]: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "c"}
+// CHECK-SAME:      %[[D_ARG:.*]]: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "d"}
+// CHECK-DAG:     %[[C1:.*]] = arith.constant 1 : index
+// CHECK-DAG:     %[[C0:.*]] = arith.constant 0 : index
+// CHECK-DAG:     %[[A:.*]]:2 = hlfir.declare %[[A_ARG]]
+// CHECK-DAG:     %[[B:.*]]:2 = hlfir.declare %[[B_ARG]]
+// CHECK-DAG:     %[[C:.*]]:2 = hlfir.declare %[[C_ARG]]
+// CHECK-DAG:     %[[D:.*]]:2 = hlfir.declare %[[D_ARG]]
+// CHECK-NEXT:    %[[B_DIM0:.*]]:3 = fir.box_dims %[[B]]#0, %[[C0]]
+// CHECK-NEXT:    %[[ARRAY:.*]] = fir.array_load %[[A]]#0
+// CHECK-NEXT:    %[[MAX:.*]] = arith.subi %[[B_DIM0]]#1, %[[C1]]
+// CHECK-NEXT:    %[[LOOP:.*]] = fir.do_loop %[[I:.*]] = %[[C0]] to %[[MAX]] step %[[C1]] unordered iter_args(%[[LOOP_ARRAY:.*]] = %[[ARRAY]])
+// inline the elemental:
+// CHECK-NEXT:      %[[B_I_REF:.*]] = hlfir.designate %[[B]]#0 (%[[I]])
+// CHECK-NEXT:      %[[C_I_REF:.*]] = hlfir.designate %[[C]]#0 (%[[I]])
+// CHECK-NEXT:      %[[B_I:.*]] = fir.load %[[B_I_REF]]
+// CHECK-NEXT:      %[[C_I:.*]] = fir.load %[[C_I_REF]]
+// CHECK-NEXT:      %[[MUL:.*]] = arith.muli %[[B_I]], %[[C_I]]
+// loop body:
+// CHECK-NEXT:      %[[D_I_REF:.*]] = hlfir.designate %[[D]]#0 (%[[I]])
+// CHECK-NEXT:      %[[D_I:.*]] = fir.load %[[D_I_REF]]
+// CHECK-NEXT:      %[[ADD:.*]] = arith.addi %[[MUL]], %[[D_I]]
+// CHECK-NEXT:      %[[ARRAY_UPD:.*]] = fir.array_update %[[LOOP_ARRAY]], %[[ADD]], %[[I]]
+// CHECK-NEXT:      fir.result %[[ARRAY_UPD]]
+// CHECK-NEXT:    }
+// CHECK-NEXT:    fir.array_merge_store %[[ARRAY]], %[[LOOP]] to %[[A_ARG]]
+// CHECK-NEXT:    return
+// CHECK-NEXT:  }
+
+// inlining into a single hlfir.apply
+// a = (b * c)[1]
+func.func @inline_to_apply(%arg0: !fir.ref<i32> {fir.bindc_name = "a"}, %arg1: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "b"}, %arg2: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "c"}) {
+  %0:2 = hlfir.declare %arg0 {uniq_name = "a"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+  %1:2 = hlfir.declare %arg1 {uniq_name = "b"} : (!fir.box<!fir.array<?xi32>>) -> (!fir.box<!fir.array<?xi32>>, !fir.box<!fir.array<?xi32>>)
+  %2:2 = hlfir.declare %arg2 {uniq_name = "c"} : (!fir.box<!fir.array<?xi32>>) -> (!fir.box<!fir.array<?xi32>>, !fir.box<!fir.array<?xi32>>)
+  %c0 = arith.constant 0 : index
+  %4:3 = fir.box_dims %1#0, %c0 : (!fir.box<!fir.array<?xi32>>, index) -> (index, index, index)
+  %5 = fir.shape %4#1 : (index) -> !fir.shape<1>
+  %6 = hlfir.elemental %5 : (!fir.shape<1>) -> !hlfir.expr<?xi32> {
+  ^bb0(%arg4: index):
+    %8 = hlfir.designate %1#0 (%arg4)  : (!fir.box<!fir.array<?xi32>>, index) -> !fir.ref<i32>
+    %9 = hlfir.designate %2#0 (%arg4)  : (!fir.box<!fir.array<?xi32>>, index) -> !fir.ref<i32>
+    %10 = fir.load %8 : !fir.ref<i32>
+    %11 = fir.load %9 : !fir.ref<i32>
+    %12 = arith.muli %10, %11 : i32
+    hlfir.yield_element %12 : i32
+  }
+  %c1 = arith.constant 1 : index
+  %res = hlfir.apply %6, %c1 : (!hlfir.expr<?xi32>, index) -> i32
+  fir.store %res to %0#0 : !fir.ref<i32>
+  hlfir.destroy %6 : !hlfir.expr<?xi32>
+  return
+}
+// CHECK-LABEL: func.func @inline_to_apply
+// CHECK-SAME:      %[[A_ARG:.*]]: !fir.ref<i32> {fir.bindc_name = "a"}
+// CHECK-SAME:      %[[B_ARG:.*]]: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "b"}
+// CHECK-SAME:      %[[C_ARG:.*]]: !fir.box<!fir.array<?xi32>> {fir.bindc_name = "c"}
+// CHECK-DAG:     %[[C1:.*]] = arith.constant 1 : index
+// CHECK-DAG:     %[[A:.*]]:2 = hlfir.declare %[[A_ARG]]
+// CHECK-DAG:     %[[B:.*]]:2 = hlfir.declare %[[B_ARG]]
+// CHECK-DAG:     %[[C:.*]]:2 = hlfir.declare %[[C_ARG]]
+// inline the elemental:
+// CHECK-NEXT:    %[[B_1_REF:.*]] = hlfir.designate %[[B]]#0 (%[[C1]])
+// CHECK-NEXT:    %[[C_1_REF:.*]] = hlfir.designate %[[C]]#0 (%[[C1]])
+// CHECK-NEXT:    %[[B_1:.*]] = fir.load %[[B_1_REF]]
+// CHECK-NEXT:    %[[C_1:.*]] = fir.load %[[C_1_REF]]
+// CHECK-NEXT:    %[[MUL:.*]] = arith.muli %[[B_1]], %[[C_1]]
+// store:
+// CHECK-NEXT:    fir.store %[[MUL]] to %0#0 : !fir.ref<i32>
+// CHECK-NEXT:    return
+// CHECK-NEXT:  }
+
+// Check long chains of elementals
+// subroutine reproducer(a)
+//   real, dimension(:) :: a
+//   a = sqrt(a * (a - 1))
+// end subroutine
+func.func @_QPreproducer(%arg0: !fir.box<!fir.array<?xf32>> {fir.bindc_name = "a"}) {
+  %0:2 = hlfir.declare %arg0 {uniq_name = "_QFreproducerEa"} : (!fir.box<!fir.array<?xf32>>) -> (!fir.box<!fir.array<?xf32>>, !fir.box<!fir.array<?xf32>>)
+  %cst = arith.constant 1.000000e+00 : f32
+  %c0 = arith.constant 0 : index
+  %1:3 = fir.box_dims %0#0, %c0 : (!fir.box<!fir.array<?xf32>>, index) -> (index, index, index)
+  %2 = fir.shape %1#1 : (index) -> !fir.shape<1>
+  %3 = hlfir.elemental %2 : (!fir.shape<1>) -> !hlfir.expr<?xf32> {
+  ^bb0(%arg1: index):
+    %9 = hlfir.designate %0#0 (%arg1)  : (!fir.box<!fir.array<?xf32>>, index) -> !fir.ref<f32>
+    %10 = fir.load %9 : !fir.ref<f32>
+    %11 = arith.subf %10, %cst fastmath<contract> : f32
+    hlfir.yield_element %11 : f32
+  }
+  %4 = hlfir.elemental %2 : (!fir.shape<1>) -> !hlfir.expr<?xf32> {
+  ^bb0(%arg1: index):
+    %9 = hlfir.apply %3, %arg1 : (!hlfir.expr<?xf32>, index) -> f32
+    %10 = hlfir.no_reassoc %9 : f32
+    hlfir.yield_element %10 : f32
+  }
+  %c0_0 = arith.constant 0 : index
+  %5:3 = fir.box_dims %0#0, %c0_0 : (!fir.box<!fir.array<?xf32>>, index) -> (index, index, index)
+  %6 = fir.shape %5#1 : (index) -> !fir.shape<1>
+  %7 = hlfir.elemental %6 : (!fir.shape<1>) -> !hlfir.expr<?xf32> {
+  ^bb0(%arg1: index):
+    %9 = hlfir.designate %0#0 (%arg1)  : (!fir.box<!fir.array<?xf32>>, index) -> !fir.ref<f32>
+    %10 = hlfir.apply %4, %arg1 : (!hlfir.expr<?xf32>, index) -> f32
+    %11 = fir.load %9 : !fir.ref<f32>
+    %12 = arith.mulf %11, %10 fastmath<contract> : f32
+    hlfir.yield_element %12 : f32
+  }
+  %8 = hlfir.elemental %6 : (!fir.shape<1>) -> !hlfir.expr<?xf32> {
+  ^bb0(%arg1: index):
+    %9 = hlfir.apply %7, %arg1 : (!hlfir.expr<?xf32>, index) -> f32
+    %10 = math.sqrt %9 fastmath<contract> : f32
+    hlfir.yield_element %10 : f32
+  }
+  hlfir.assign %8 to %0#0 : !hlfir.expr<?xf32>, !fir.box<!fir.array<?xf32>>
+  hlfir.destroy %8 : !hlfir.expr<?xf32>
+  hlfir.destroy %7 : !hlfir.expr<?xf32>
+  hlfir.destroy %4 : !hlfir.expr<?xf32>
+  hlfir.destroy %3 : !hlfir.expr<?xf32>
+  return
+}
+// CHECK-LABEL: func.func @_QPreproducer
+// CHECK-SAME:      %[[A_ARG:.*]]: !fir.box<!fir.array<?xf32>> {fir.bindc_name = "a"}
+// CHECK-DAG:     %[[CST:.*]] = arith.constant 1.0000
+// CHECK-DAG:     %[[C0:.*]] = arith.constant 0
+// CHECK-DAG:     %[[A_VAR:.*]]:2 = hlfir.declare %[[A_ARG]]
+// CHECK-NEXT:    %[[A_DIMS_0:.*]]:3 = fir.box_dims %[[A_VAR]]#0, %[[C0]]
+// CHECK-NEXT:    %[[SHAPE_0:.*]] = fir.shape %[[A_DIMS_0]]#1
+// all in one elemental:
+// CHECK-NEXT:    %[[EXPR:.*]] = hlfir.elemental %[[SHAPE_0]]
+// CHECK-NEXT:    ^bb0(%[[I:.*]]: index):
+// CHECK-NEXT:      %[[A_I0:.*]] = hlfir.designate %[[A_VAR]]#0 (%[[I]])
+// CHECK-NEXT:      %[[A_I1:.*]] = hlfir.designate %[[A_VAR]]#0 (%[[I]])
+// CHECK-NEXT:      %[[A_I1_VAL:.*]] = fir.load %[[A_I1]]
+// CHECK-NEXT:      %[[SUB:.*]] = arith.subf %[[A_I1_VAL]], %[[CST]]
+// CHECK-NEXT:      %[[SUB0:.*]] = hlfir.no_reassoc %[[SUB]] : f32
+// CHECK-NEXT:      %[[A_I0_VAL:.*]] = fir.load %[[A_I0]]
+// CHECK-NEXT:      %[[MUL:.*]] = arith.mulf %[[A_I0_VAL]], %[[SUB0]]
+// CHECK-NEXT:      %[[SQRT:.*]] = math.sqrt %[[MUL]]
+// CHECK-NEXT:      hlfir.yield_element %[[SQRT]]
+// CHECK-NEXT:    }
+// CHECK-NEXT:    hlfir.assign %[[EXPR]] to %[[A_VAR]]#0
+// CHECK-NEXT:    hlfir.destroy %[[EXPR]]
+// CHECK-NEXT:    return
+// CHECK-NEXT:  }