[flang] add -flang-experimental-hlfir flag to flang-new

[tblah]

Summary

This flag instructs flang-new to use the new HLFIR lowering. It is
marked as experimental and not included in --help.

This was added to make it more convenient to test the performance of
code generated by the HLFIR lowering.

[awarzynski]

Could you add some tests?

[vzakhari]

Thank you for working on this!

flang/include/flang/Tools/CLOptions.inc

249

Would you mind also calling this in bbc driver?

252

Is this check and the option really needed? Except for the extra canonicalizer pass the newly added passes will be no-ops, so maybe we should just run them unconditionally.

It may be too much to ask for, but will it make sense not to bundle these passes into MLIRToLLVM pipeline and have the possibility to let driver emit post-HLFIR-lowering MLIR (under -emit-fir) and pre-HLFIR-lowering (under some new option)?

[vzakhari]

flang/include/flang/Tools/CLOptions.inc

252

I would imagine we may not want to optimize MATMUL(TRANSPOSE) into MATMUL_TRANSPOSE at O0. What is the best way to control this? We may either disable canonicalization or let LowerHLFIRIntrinsicsPass lower MATMUL_TRANSPOSE differently based on the optimization level. Or is it always okay to implement it as a combined operation?

[tschuett]

You can either pass parameters to passes or use different pass pipelines for different optimisation levels (in MLIR territory).

[tblah]

Thanks for the review. Changes are as follows:

• Move createHLFIRToFIRPassPipeline before #if !defined(FLANG_EXCCLUDE_CODEGEN
• Only canonicalize HLFIR (creating hlfir.matmul_transpose) if we are optimizing for speed
• Unconditionally run HLFIR passes
• Add tests

[awarzynski]

flang/test/Driver/mlir-pass-pipeline.f90

15–18

It looks like these passes are run unconditionally - what's the -flang-experimental-hlfir flag for then?

[tblah]

flang/include/flang/Tools/CLOptions.inc

249

Adding this to bbc will have to wait until after -emit-fir and -emit-hlfir are different flags. Otherwise hlfir ops will be lowered to fir, breaking some tests (and presumably people's workflows).

252

So far as I know, there should be no loss to precision from implementing it as a combined operation. Memory usage should be reduced as we need one fewer temporary.

If static linking is used, including MATMUL_TRANSPOSE in the runtime library will increase code size (so long as both matmul and transpose are also called elsewhere). I haven't measured this, but I wouldn't expect this to be a large change relative to the size of a real world application.

If dynamic linking is used, whether or not this pass runs, MATMUL_TRANSPOSE will make the runtime library a little larger (there are a lot of template instantiations, but MATMUL_TRANSPOSE is only one of many similar functions so the effect as a proportion of the whole shouldn't be much).

But I'll set the canonicalization pass to only run when we are optimizing for speed. Later canonicalisation passes (after createLowerHLFIRIntrinsicsPass) won't find any hlfir.matmul operations to canonicalise and so won't create a hlfir.matmul_transpose operation.

252

Okay I'll run the HLFIR passes unconditionally.

I made -emit-fir output HLFIR to match what bbc already does, but I can see the usefulness of having both -emit-fir and -emit-hlfir. I won't do it in this patch but I will get around to this at some point (but feel free to jump in sooner if you need the flag for something).

flang/test/Driver/mlir-pass-pipeline.f90

15–18

Yes. @vzakhari requested the passes should run unconditionally.

-flang-experimental-hlfir controls whether the HLFIR lowering is used.

[awarzynski]

flang/test/Driver/mlir-pass-pipeline.f90

15–18

Ok, so so far you have updated the default pass pipelines and that's what you are testing here. However, this patch is about -flang-experimental-hlfir - can you add a test for this flag?

[vzakhari]

LGTM
Please address @awarzynski's comment about the test.

flang/include/flang/Tools/CLOptions.inc

249

Okay! Thank you for considering it!

252

Thank you!

[tblah]

Added a test

[awarzynski]

flang/test/HLFIR/flang-experimental-hlfir-flag.f90

3

Could you also add a RUN line like this:

! RUN: %flang_fc1 -emit-fir -o - %s | FileCheck %s --prefix="NO-HLFIR"

! CHECK-NOT: hlfir

[tblah]

Thanks for the suggestion

• Added a test to check that hlfir is not output without the flag
• clang-formatted CLOptions.inc

[awarzynski]

LGTM, thanks!

flang/test/HLFIR/flang-experimental-hlfir-flag.f90

20

[ultra nit] I would separate with an empty line for better readability.

[tblah]

flang/include/flang/Tools/CLOptions.inc

252

-emit-fir and -emit-hlfir are separated in https://reviews.llvm.org/D151088

clang/include/clang/Driver/Options.td

 //===----------------------------------------------------------------------===//
 // FLangOption + NoXarchOption
 //===----------------------------------------------------------------------===//
 
 def flang_experimental_exec : Flag<["-"], "flang-experimental-exec">,
   Flags<[FlangOption, FlangOnlyOption, NoXarchOption, HelpHidden]>,
   HelpText<"Enable support for generating executables (experimental)">;
 
+def flang_experimental_hlfir : Flag<["-"], "flang-experimental-hlfir">,
+  Flags<[FlangOption, FC1Option, FlangOnlyOption, NoXarchOption, HelpHidden]>,
+  HelpText<"Use HLFIR lowering (experimental)">;
+
 //===----------------------------------------------------------------------===//
 // FLangOption + CoreOption + NoXarchOption
 //===----------------------------------------------------------------------===//
 
 def Xflang : Separate<["-"], "Xflang">,
   HelpText<"Pass <arg> to the flang compiler">, MetaVarName<"<arg>">,
   Flags<[FlangOption, FlangOnlyOption, NoXarchOption, CoreOption]>,
   Group<CompileOnly_Group>;

clang/lib/Driver/ToolChains/Flang.cpp

Args.AddAllArgs(CmdArgs,
                    options::OPT_funderscoring, options::OPT_fno_underscoring});
 
   Arg *stackArrays =
       Args.getLastArg(options::OPT_Ofast, options::OPT_fstack_arrays,
                       options::OPT_fno_stack_arrays);
   if (stackArrays &&
       !stackArrays->getOption().matches(options::OPT_fno_stack_arrays))
     CmdArgs.push_back("-fstack-arrays");
+
+  if (Args.hasArg(options::OPT_flang_experimental_hlfir))
+    CmdArgs.push_back("-flang-experimental-hlfir");
 }
 
 void Flang::addPicOptions(const ArgList &Args, ArgStringList &CmdArgs) const {
   // ParsePICArgs parses -fPIC/-fPIE and their variants and returns a tuple of
   // (RelocationModel, PICLevel, IsPIE).
   llvm::Reloc::Model RelocationModel;
   unsigned PICLevel;
   bool IsPIE;

flang/include/flang/Tools/CLOptions.inc

 //===-- CLOptions.inc -- command line options -------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 /// This file defines some shared command-line options that can be used when
 /// debugging the test tools. This file must be included into the tool.
 
 #include "mlir/Conversion/SCFToControlFlow/SCFToControlFlow.h"
 #include "mlir/Pass/PassManager.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "mlir/Transforms/Passes.h"
 #include "flang/Optimizer/CodeGen/CodeGen.h"
+#include "flang/Optimizer/HLFIR/Passes.h"
 #include "flang/Optimizer/Transforms/Passes.h"
 #include "llvm/Passes/OptimizationLevel.h"
 #include "llvm/Support/CommandLine.h"
 
 #define DisableOption(DOName, DOOption, DODescription) \
   static llvm::cl::opt<bool> disable##DOName("disable-" DOOption, \
       llvm::cl::desc("disable " DODescription " pass"), llvm::cl::init(false), \
       llvm::cl::Hidden)

 DisableOption(TargetRewrite, "target-rewrite", "rewrite FIR for target");
 DisableOption(DebugFoundation, "debug-foundation", "Add debug foundation");
 DisableOption(FirToLlvmIr, "fir-to-llvmir", "FIR to LLVM-IR dialect");
 DisableOption(LlvmIrToLlvm, "llvm", "conversion to LLVM");
 DisableOption(BoxedProcedureRewrite, "boxed-procedure-rewrite",
     "rewrite boxed procedures");
 #endif
 
-DisableOption(ExternalNameConversion, "external-name-interop", "convert names with external convention");
+DisableOption(ExternalNameConversion, "external-name-interop",
+    "convert names with external convention");
 
 /// Generic for adding a pass to the pass manager if it is not disabled.
 template <typename F>
 void addPassConditionally(
     mlir::PassManager &pm, llvm::cl::opt<bool> &disabled, F ctor) {
   if (!disabled)
     pm.addPass(ctor());
 }
inline void createDefaultFIROptimizerPassPipeline(mlir::PassManager &pm,
   fir::addCfgConversionPass(pm);
   pm.addPass(mlir::createConvertSCFToCFPass());
 
   pm.addPass(mlir::createCanonicalizerPass(config));
   pm.addPass(fir::createSimplifyRegionLitePass());
   pm.addPass(mlir::createCSEPass());
 }
 
+/// Create a pass pipeline for lowering from HLFIR to FIR
+///
+/// \param pm - MLIR pass manager that will hold the pipeline definition
+/// \param optLevel - optimization level used for creating FIR optimization
+///   passes pipeline
+inline void createHLFIRToFIRPassPipeline(
+    mlir::PassManager &pm, llvm::OptimizationLevel optLevel = defaultOptLevel) {
+  if (optLevel.isOptimizingForSpeed())
+    pm.addPass(mlir::createCanonicalizerPass());
+  pm.addPass(hlfir::createLowerHLFIRIntrinsicsPass());
+  pm.addPass(hlfir::createBufferizeHLFIRPass());
+  pm.addPass(hlfir::createConvertHLFIRtoFIRPass());
+}
+
 #if !defined(FLANG_EXCLUDE_CODEGEN)
 inline void createDefaultFIRCodeGenPassPipeline(mlir::PassManager &pm,
     llvm::OptimizationLevel optLevel = defaultOptLevel,
     bool underscoring = true) {
   fir::addBoxedProcedurePass(pm);
   pm.addNestedPass<mlir::func::FuncOp>(
       fir::createAbstractResultOnFuncOptPass());
-  pm.addNestedPass<fir::GlobalOp>(
-      fir::createAbstractResultOnGlobalOptPass());
+  pm.addNestedPass<fir::GlobalOp>(fir::createAbstractResultOnGlobalOptPass());
   fir::addCodeGenRewritePass(pm);
   fir::addTargetRewritePass(pm);
   fir::addExternalNameConversionPass(pm, underscoring);
   fir::addFIRToLLVMPass(pm, optLevel);
 }
 
 /// Create a pass pipeline for lowering from MLIR to LLVM IR
 ///
 /// \param pm - MLIR pass manager that will hold the pipeline definition
 /// \param optLevel - optimization level used for creating FIR optimization
 ///   passes pipeline
 inline void createMLIRToLLVMPassPipeline(mlir::PassManager &pm,
     llvm::OptimizationLevel optLevel = defaultOptLevel,
     bool stackArrays = false, bool underscoring = true) {
+  fir::createHLFIRToFIRPassPipeline(pm, optLevel);
+
   // Add default optimizer pass pipeline.
   fir::createDefaultFIROptimizerPassPipeline(pm, optLevel, stackArrays);
 
   // Add codegen pass pipeline.
   fir::createDefaultFIRCodeGenPassPipeline(pm, optLevel, underscoring);
 }
 #undef FLANG_EXCLUDE_CODEGEN
 #endif
 
 } // namespace fir

flang/lib/Frontend/CompilerInvocation.cpp

for (const auto *a : args.filtered(clang::driver::options::OPT_UNKNOWN)) {
     if (opts.findNearest(argString, nearest, includedFlagsBitmask) > 1)
       diags.Report(clang::diag::err_drv_unknown_argument) << argString;
     else
       diags.Report(clang::diag::err_drv_unknown_argument_with_suggestion)
           << argString << nearest;
     success = false;
   }
 
+  // -flang-experimental-hlfir
+  if (args.hasArg(clang::driver::options::OPT_flang_experimental_hlfir)) {
+    res.loweringOpts.setLowerToHighLevelFIR(true);
+  }
+
   success &= parseFrontendArgs(res.getFrontendOpts(), args, diags);
   parseTargetArgs(res.getTargetOpts(), args);
   parsePreprocessorArgs(res.getPreprocessorOpts(), args);
   parseCodeGenArgs(res.getCodeGenOpts(), args, diags);
   success &= parseSemaArgs(res, args, diags);
   success &= parseDialectArgs(res, args, diags);
   success &= parseDiagArgs(res, args, diags);
   res.frontendOpts.llvmArgs =

flang/test/Driver/driver-help-hidden.f90

 ! CHECK-NEXT: -ffp-contract=<value> Form fused FP ops (e.g. FMAs)
 ! CHECK-NEXT: -ffree-form            Process source files in free form
 ! CHECK-NEXT: -fimplicit-none        No implicit typing allowed unless overridden by IMPLICIT statements
 ! CHECK-NEXT: -finput-charset=<value> Specify the default character set for source files
 ! CHECK-NEXT: -fintrinsic-modules-path <dir>
 ! CHECK-NEXT:                        Specify where to find the compiled intrinsic modules
 ! CHECK-NEXT: -flang-experimental-exec
 ! CHECK-NEXT:                        Enable support for generating executables (experimental)
+! CHECK-NEXT: -flang-experimental-hlfir
+! CHECK-NEXT:                        Use HLFIR lowering (experimental)
 ! CHECK-NEXT: -flarge-sizes          Use INTEGER(KIND=8) for the result type in size-related intrinsics
 ! CHECK-NEXT: -flogical-abbreviations Enable logical abbreviations
 ! CHECK-NEXT: -flto=<value> Set LTO mode
 ! CHECK-NEXT: -flto Enable LTO in 'full' mode
 ! CHECK-NEXT: -fno-automatic         Implies the SAVE attribute for non-automatic local objects in subprograms unless RECURSIVE
 ! CHECK-NEXT: -fno-color-diagnostics  Disable colors in diagnostics
 ! CHECK-NEXT: -fno-integrated-as     Disable the integrated assembler
 ! CHECK-NEXT: -fno-signed-zeros      Allow optimizations that ignore the sign of floating point zeros

flang/test/Driver/mlir-pass-pipeline.f90

 ! Test the MLIR pass pipeline
 
 ! RUN: %flang_fc1 -S -mmlir --mlir-pass-statistics -mmlir --mlir-pass-statistics-display=pipeline -o /dev/null %s 2>&1 | FileCheck --check-prefixes=ALL %s
 ! -O0 is the default:
 ! RUN: %flang_fc1 -S -mmlir --mlir-pass-statistics -mmlir --mlir-pass-statistics-display=pipeline %s -O0 -o /dev/null 2>&1 | FileCheck --check-prefixes=ALL %s
 ! RUN: %flang_fc1 -S -mmlir --mlir-pass-statistics -mmlir --mlir-pass-statistics-display=pipeline %s -O2 -o /dev/null 2>&1 | FileCheck --check-prefixes=ALL,O2 %s
 
 ! REQUIRES: asserts
 
 end program
 
 ! ALL: Pass statistics report
 
 ! ALL: Fortran::lower::VerifierPass
+! O2-NEXT: Canonicalizer
+! ALL-NEXT: LowerHLFIRIntrinsics
+! ALL-NEXT: BufferizeHLFIR
+! ALL-NEXT: ConvertHLFIRtoFIR
 ! ALL-NEXT: CSE
 ! Ideally, we need an output with only the pass names, but
 ! there is currently no way to get that, so in order to
 ! guarantee that the passes are in the expected order
 ! (i.e. use -NEXT) we have to check the statistics output as well.
 ! ALL-NEXT:   (S) 0 num-cse'd - Number of operations CSE'd
 ! ALL-NEXT:   (S) 0 num-dce'd - Number of operations DCE'd
 

flang/test/Fir/basic-program.fir

 }
 
 // CHECK: ; ModuleID = 'FIRModule'
 // CHECK-LABEL: define void @_QQmain()
 // CHECK:       ret void
 
 // PASSES: Pass statistics report
 
-// PASSES:      CSE
+// PASSES:        Canonicalizer
+// PASSES-NEXT:   LowerHLFIRIntrinsics
+// PASSES-NEXT:   BufferizeHLFIR
+// PASSES-NEXT:   ConvertHLFIRtoFIR
+// PASSES-NEXT:   CSE
 // PASSES-NEXT:   (S) 0 num-cse'd - Number of operations CSE'd
 // PASSES-NEXT:   (S) 0 num-dce'd - Number of operations DCE'd
 
 // PASSES-NEXT: 'func.func' Pipeline
 // PASSES-NEXT:   ArrayValueCopy
 // PASSES-NEXT:   CharacterConversion
 
 // PASSES-NEXT: Canonicalizer

flang/test/HLFIR/flang-experimental-hlfir-flag.f90

+! Test -flang-experimental-hlfir flag
+! RUN: %flang_fc1 -flang-experimental-hlfir -emit-fir -o - %s | FileCheck %s
+! RUN: %flang_fc1 -emit-fir -o - %s | FileCheck %s --check-prefix NO-HLFIR
+
+subroutine test(a, res)
+  real :: a(:), res
+  res = SUM(a)
+end subroutine
+! CHECK-LABEL: func.func @_QPtest
+! CHECK:           %[[A:.*]]: !fir.box<!fir.array<?xf32>>
+! CHECK:           %[[RES:.*]]: !fir.ref<f32>
+! CHECK-DAG:     %[[A_VAR:.*]]:2 = hlfir.declare %[[A]]
+! CHECK-DAG:     %[[RES_VAR:.*]]:2 = hlfir.declare %[[RES]]
+! CHECK-NEXT:    %[[SUM_RES:.*]] = hlfir.sum %[[A_VAR]]#0
+! CHECK-NEXT:    hlfir.assign %[[SUM_RES]] to %[[RES_VAR]]#0
+! CHECK-NEXT:    hlfir.destroy %[[SUM_RES]]
+! CHECK-NEXT:    return
+! CHECK-NEXT:  }
+
+! NO-HLFIR-NOT: hlfir.