[flang] Use mlir complex dialect for supported operations

[DavidTruby]

Summary

This patch lowers the complex operations supported by the MLIR complex
dialect to those operations rather than libm. When the math runtime flag
is set to precise, libm lowering is used instead.

[DavidTruby]

Remove unnecessary convert.

[DavidTruby]

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3354

I think this variable is mis-spelt. I didn't change it here because I didn't think it was in scope for this patch. I guess this should be fixed as a separate commit? Or I can just tack it on to this one.

[vzakhari]

flang/lib/Lower/IntrinsicCall.cpp

1324

Can you please put this whole change under an engineering option? You may enable it by default, but I would like to have a way to switch to the old behavior easily to deal with any issues in our testing. I may take an AR to remove the option, when eveything works well.

1326

It is better to just return result here and avoid the else clause.

1335

1347

Does it mean a Complex dialect operation will return fir::ComplexType result? Shouldn't it be mlir::ComplexType?

This works for abs, but it does not look correct in general.

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3354

My bad. I am okay with fixing it here.

3356

Alternatively, you may call populateComplexToStandardConversionPatterns around line 3395, since ComplexToStandard convertor does not modify parent operations.

[DavidTruby]

flang/lib/Lower/IntrinsicCall.cpp

1324

Happy to do this, the only reason I didn't is that I thought it was already covered by setting the fp-contract to precise. Did you want something more fine-grained for just this option?

1347

I think you're right, we should take this result as the mlir::ComplexType and convert it to a fir::ComplexType. I'll update with this change.

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3356

Originally I did this, but it interacts with MathToFuncsPass and so needs to be run in the same pipeline as that. I couldn't see a better way of doing that than putting it here. When I did put it around 3395 as populateComplexToLLVMConversionPatterns it didn't work for this reason.

[vzakhari]

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3356

Hmm, it should not interact with MathToFuncs, because MathToFuncs currently only work for math::IPowIOp. I wonder in which way it did not work. Is it possible that due to the ordering we ended up converting some complex ops into LLVM rather than to Standard?

[vzakhari]

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3356

Just want to clarify that my previous comment did not intend to hold back the change. I have no remarks besides line 1342.

[DavidTruby]

Add flag to control new behaviour.
Take results of operations as mlir::complex and convert to fir::complex.

[DavidTruby]

clang-format

[vzakhari]

Thank you for the changes! LGTM with minor remarks.

flang/include/flang/Optimizer/Support/InitFIR.h

24

Why use '<...>'? Typo?

flang/lib/Lower/IntrinsicCall.cpp

42

Same here - why '<...>'?

1339

Still recommend specifying some fixed allocation size for SmallVector, e.g. 2.

flang/lib/Optimizer/CodeGen/CodeGen.cpp

40

'<...>' again.

[DavidTruby]

Fix includes
Add size suggestion to SmallVector

flang/include/flang/Optimizer/Support/InitFIR.h

24

Yep, my mistake

flang/lib/Lower/IntrinsicCall.cpp

1339

I actually thought I'd made this change but it must have slipped past my commit!

[vzakhari]

Thanks! LGTM

[DavidTruby]

In D135882#3883183, @vzakhari wrote:

Thanks! LGTM

Where does D134364 stand after this? Do we still need that design document?
Thanks

[vzakhari]

This broke my build:

ld.lld: error: undefined symbol: mlir::createConvertComplexToStandardPass()
>>> referenced by CodeGen.cpp:3540 (/llvm-project/flang/lib/Optimizer/CodeGen/CodeGen.cpp:3540)
>>>               CMakeFiles/obj.FIRCodeGen.dir/CodeGen.cpp.o:((anonymous namespace)::FIRToLLVMLowering::runOnOperation())

ld.lld: error: undefined symbol: mlir::populateComplexToLLVMConversionPatterns(mlir::LLVMTypeConverter&, mlir::RewritePatternSet&)
>>> referenced by CodeGen.cpp:3605 (/llvm-project/flang/lib/Optimizer/CodeGen/CodeGen.cpp:3605)
>>>               CMakeFiles/obj.FIRCodeGen.dir/CodeGen.cpp.o:((anonymous namespace)::FIRToLLVMLowering::runOnOperation())
collect2: error: ld returned 1 exit status

There are missing dependencies in CMake files. David, I can fix it, if you do not have time to do this today.

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3356

This requires linking MLIRComplexToStandard in CMakeLists.txt.

3395

This requires linking MLIRComplexToLLVM in CMakeLists.txt.

[vzakhari]

In D135882#3885384, @DavidTruby wrote:

In D135882#3883183, @vzakhari wrote:

Thanks! LGTM

Where does D134364 stand after this? Do we still need that design document?
Thanks

I think we should still merge the document. We need to update it mentioning special handling for some operations (like power).

[kiranchandramohan]

In D135882#3885895, @vzakhari wrote:

This broke my build:

ld.lld: error: undefined symbol: mlir::createConvertComplexToStandardPass()
>>> referenced by CodeGen.cpp:3540 (/llvm-project/flang/lib/Optimizer/CodeGen/CodeGen.cpp:3540)
>>>               CMakeFiles/obj.FIRCodeGen.dir/CodeGen.cpp.o:((anonymous namespace)::FIRToLLVMLowering::runOnOperation())

ld.lld: error: undefined symbol: mlir::populateComplexToLLVMConversionPatterns(mlir::LLVMTypeConverter&, mlir::RewritePatternSet&)
>>> referenced by CodeGen.cpp:3605 (/llvm-project/flang/lib/Optimizer/CodeGen/CodeGen.cpp:3605)
>>>               CMakeFiles/obj.FIRCodeGen.dir/CodeGen.cpp.o:((anonymous namespace)::FIRToLLVMLowering::runOnOperation())
collect2: error: ld returned 1 exit status

There are missing dependencies in CMake files. David, I can fix it, if you do not have time to do this today.

I created https://reviews.llvm.org/D136773 to fix this.

flang/include/flang/Optimizer/Support/InitFIR.h

 //===-- Optimizer/Support/InitFIR.h -----------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Coding style: https://mlir.llvm.org/getting_started/DeveloperGuide/
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef FORTRAN_OPTIMIZER_SUPPORT_INITFIR_H
 #define FORTRAN_OPTIMIZER_SUPPORT_INITFIR_H
 
 #include "flang/Optimizer/Dialect/FIRDialect.h"
 #include "mlir/Conversion/Passes.h"
 #include "mlir/Dialect/Affine/Passes.h"
 #include "mlir/InitAllDialects.h"
 #include "mlir/Pass/Pass.h"
 #include "mlir/Pass/PassRegistry.h"
 #include "mlir/Transforms/LocationSnapshot.h"
 #include "mlir/Transforms/Passes.h"
+#include <mlir/Dialect/Complex/IR/Complex.h>
 
 namespace fir::support {
 
 #define FLANG_NONCODEGEN_DIALECT_LIST                                          \
   mlir::AffineDialect, FIROpsDialect, mlir::acc::OpenACCDialect,               \
       mlir::omp::OpenMPDialect, mlir::scf::SCFDialect,                         \
       mlir::arith::ArithDialect, mlir::cf::ControlFlowDialect,                 \
       mlir::func::FuncDialect, mlir::vector::VectorDialect,                    \
-      mlir::math::MathDialect
+      mlir::math::MathDialect, mlir::complex::ComplexDialect
 
 // The definitive list of dialects used by flang.
 #define FLANG_DIALECT_LIST                                                     \
   FLANG_NONCODEGEN_DIALECT_LIST, FIRCodeGenDialect, mlir::LLVM::LLVMDialect
 
 inline void registerNonCodegenDialects(mlir::DialectRegistry &registry) {
   registry.insert<FLANG_NONCODEGEN_DIALECT_LIST>();
 }

flang/lib/Lower/IntrinsicCall.cpp

 //===-- IntrinsicCall.cpp -------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 #include "flang/Optimizer/Builder/Todo.h"
 #include "flang/Optimizer/Dialect/FIROpsSupport.h"
 #include "flang/Optimizer/Support/FatalError.h"
 #include "flang/Runtime/entry-names.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/Math/IR/Math.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
+#include <mlir/Dialect/Complex/IR/Complex.h>
 
 #define DEBUG_TYPE "flang-lower-intrinsic"
 
 #define PGMATH_DECLARE
 #include "flang/Evaluate/pgmath.h.inc"
 
 /// This file implements lowering of Fortran intrinsic procedures and Fortran
 /// intrinsic module procedures.  A call may be inlined with a mix of FIR and

 llvm::cl::opt<MathRuntimeVersion> mathRuntimeVersion(
     "math-runtime", llvm::cl::desc("Select math operations' runtime behavior:"),
     llvm::cl::values(
         clEnumValN(fastVersion, "fast", "use fast runtime behavior"),
         clEnumValN(relaxedVersion, "relaxed", "use relaxed runtime behavior"),
         clEnumValN(preciseVersion, "precise", "use precise runtime behavior")),
     llvm::cl::init(fastVersion));
 
+static llvm::cl::opt<bool> disableMlirComplex(
+  "disable-mlir-complex",
+  llvm::cl::desc("Use libm instead of the MLIR complex dialect to lower complex operations"),
+  llvm::cl::init(false));
+
 struct RuntimeFunction {
   // llvm::StringRef comparison operator are not constexpr, so use string_view.
   using Key = std::string_view;
   // Needed for implicit compare with keys.
   constexpr operator Key() const { return key; }
   Key key; // intrinsic name
 
   // Name of a runtime function that implements the operation.
LLVM_DEBUG(llvm::dbgs() << "Generating '" << mathLibFuncName
                             << "' operation with type ";
                mathLibFuncType.dump(); llvm::dbgs() << "\n");
     result = builder.create<T>(loc, args);
   }
   LLVM_DEBUG(result.dump(); llvm::dbgs() << "\n");
   return result;
 }
 
+template <typename T>
+static mlir::Value genComplexMathOp(fir::FirOpBuilder &builder,
+                                    mlir::Location loc,
+                                    llvm::StringRef mathLibFuncName,
+                                    mlir::FunctionType mathLibFuncType,
+                                    llvm::ArrayRef<mlir::Value> args) {
+  mlir::Value result;
+  if (disableMlirComplex || (mathRuntimeVersion == preciseVersion && !mathLibFuncName.empty())) {
+    result = genLibCall(builder, loc, mathLibFuncName, mathLibFuncType, args);
+    LLVM_DEBUG(result.dump(); llvm::dbgs() << "\n");
+    return result;
+  }
+
+  LLVM_DEBUG(llvm::dbgs() << "Generating '" << mathLibFuncName
+                          << "' operation with type ";
+              mathLibFuncType.dump(); llvm::dbgs() << "\n");
+  auto type = mathLibFuncType.getInput(0).cast<fir::ComplexType>();
+  auto kind = type.getElementType().cast<fir::RealType>().getFKind();
+  auto realTy = builder.getRealType(kind);
+  auto mComplexTy = mlir::ComplexType::get(realTy);
+
+  llvm::SmallVector<mlir::Value> cargs;
+  for (const mlir::Value &arg : args) {
+    // Convert the fir.complex to a mlir::complex
+    cargs.push_back(builder.createConvert(loc, mComplexTy, arg));
+  }
+
+  // Builder expects an extra return type to be provided if different to
+  // the argument types for an operation
+  if constexpr (T::template hasTrait<
+                    mlir::OpTrait::SameOperandsAndResultType>()) {
+    result = builder.create<T>(loc, cargs);
+    result = builder.createConvert(loc, mathLibFuncType.getResult(0), result);
+  } else {
+    result = builder.create<T>(loc, realTy, cargs);
+    result = builder.createConvert(loc, mathLibFuncType.getResult(0), result);
+  }
+
+  LLVM_DEBUG(result.dump(); llvm::dbgs() << "\n");
+  return result;
+}
+
+
 /// Mapping between mathematical intrinsic operations and MLIR operations
 /// of some appropriate dialect (math, complex, etc.) or libm calls.
 /// TODO: support remaining Fortran math intrinsics.
 ///       See https://gcc.gnu.org/onlinedocs/gcc-12.1.0/gfortran/\
 ///       Intrinsic-Procedures.html for a reference.
 static constexpr MathOperation mathOperations[] = {
     {"abs", "fabsf", genF32F32FuncType, genMathOp<mlir::math::AbsFOp>},
     {"abs", "fabs", genF64F64FuncType, genMathOp<mlir::math::AbsFOp>},
     {"abs", "llvm.fabs.f128", genF128F128FuncType,
      genMathOp<mlir::math::AbsFOp>},
-    {"abs", "cabsf", genF32ComplexFuncType, genLibCall},
-    {"abs", "cabs", genF64ComplexFuncType, genLibCall},
+    {"abs", "cabsf", genF32ComplexFuncType, genComplexMathOp<mlir::complex::AbsOp>},
+    {"abs", "cabs", genF64ComplexFuncType, genComplexMathOp<mlir::complex::AbsOp>},
     {"acos", "acosf", genF32F32FuncType, genLibCall},
     {"acos", "acos", genF64F64FuncType, genLibCall},
     {"acos", "cacosf", genComplexComplexFuncType<4>, genLibCall},
     {"acos", "cacos", genComplexComplexFuncType<8>, genLibCall},
     {"acosh", "acoshf", genF32F32FuncType, genLibCall},
     {"acosh", "acosh", genF64F64FuncType, genLibCall},
     {"acosh", "cacoshf", genComplexComplexFuncType<4>, genLibCall},
     {"acosh", "cacosh", genComplexComplexFuncType<8>, genLibCall},
static constexpr MathOperation mathOperations[] = {
     {"bessel_y1", "y1", genF64F64FuncType, genLibCall},
     {"bessel_yn", "ynf", genF32IntF32FuncType<32>, genLibCall},
     {"bessel_yn", "yn", genF64IntF64FuncType<32>, genLibCall},
     // math::CeilOp returns a real, while Fortran CEILING returns integer.
     {"ceil", "ceilf", genF32F32FuncType, genMathOp<mlir::math::CeilOp>},
     {"ceil", "ceil", genF64F64FuncType, genMathOp<mlir::math::CeilOp>},
     {"cos", "cosf", genF32F32FuncType, genMathOp<mlir::math::CosOp>},
     {"cos", "cos", genF64F64FuncType, genMathOp<mlir::math::CosOp>},
-    {"cos", "ccosf", genComplexComplexFuncType<4>, genLibCall},
-    {"cos", "ccos", genComplexComplexFuncType<8>, genLibCall},
+    {"cos", "ccosf", genComplexComplexFuncType<4>, genComplexMathOp<mlir::complex::CosOp>},
+    {"cos", "ccos", genComplexComplexFuncType<8>, genComplexMathOp<mlir::complex::CosOp>},
     {"cosh", "coshf", genF32F32FuncType, genLibCall},
     {"cosh", "cosh", genF64F64FuncType, genLibCall},
     {"cosh", "ccoshf", genComplexComplexFuncType<4>, genLibCall},
     {"cosh", "ccosh", genComplexComplexFuncType<8>, genLibCall},
     {"erf", "erff", genF32F32FuncType, genMathOp<mlir::math::ErfOp>},
     {"erf", "erf", genF64F64FuncType, genMathOp<mlir::math::ErfOp>},
     {"erfc", "erfcf", genF32F32FuncType, genLibCall},
     {"erfc", "erfc", genF64F64FuncType, genLibCall},
     {"exp", "expf", genF32F32FuncType, genMathOp<mlir::math::ExpOp>},
     {"exp", "exp", genF64F64FuncType, genMathOp<mlir::math::ExpOp>},
-    {"exp", "cexpf", genComplexComplexFuncType<4>, genLibCall},
-    {"exp", "cexp", genComplexComplexFuncType<8>, genLibCall},
+    {"exp", "cexpf", genComplexComplexFuncType<4>, genComplexMathOp<mlir::complex::ExpOp>},
+    {"exp", "cexp", genComplexComplexFuncType<8>, genComplexMathOp<mlir::complex::ExpOp>},
     // math::FloorOp returns a real, while Fortran FLOOR returns integer.
     {"floor", "floorf", genF32F32FuncType, genMathOp<mlir::math::FloorOp>},
     {"floor", "floor", genF64F64FuncType, genMathOp<mlir::math::FloorOp>},
     {"gamma", "tgammaf", genF32F32FuncType, genLibCall},
     {"gamma", "tgamma", genF64F64FuncType, genLibCall},
     {"hypot", "hypotf", genF32F32F32FuncType, genLibCall},
     {"hypot", "hypot", genF64F64F64FuncType, genLibCall},
     {"log", "logf", genF32F32FuncType, genMathOp<mlir::math::LogOp>},
     {"log", "log", genF64F64FuncType, genMathOp<mlir::math::LogOp>},
-    {"log", "clogf", genComplexComplexFuncType<4>, genLibCall},
-    {"log", "clog", genComplexComplexFuncType<8>, genLibCall},
+    {"log", "clogf", genComplexComplexFuncType<4>, genComplexMathOp<mlir::complex::LogOp>},
+    {"log", "clog", genComplexComplexFuncType<8>, genComplexMathOp<mlir::complex::LogOp>},
     {"log10", "log10f", genF32F32FuncType, genMathOp<mlir::math::Log10Op>},
     {"log10", "log10", genF64F64FuncType, genMathOp<mlir::math::Log10Op>},
     {"log_gamma", "lgammaf", genF32F32FuncType, genLibCall},
     {"log_gamma", "lgamma", genF64F64FuncType, genLibCall},
     // llvm.lround behaves the same way as libm's lround.
     {"nint", "llvm.lround.i64.f64", genIntF64FuncType<64>, genLibCall},
     {"nint", "llvm.lround.i64.f32", genIntF32FuncType<64>, genLibCall},
     {"nint", "llvm.lround.i32.f64", genIntF64FuncType<32>, genLibCall},
     {"nint", "llvm.lround.i32.f32", genIntF32FuncType<32>, genLibCall},
     {"pow", {}, genIntIntIntFuncType<8>, genMathOp<mlir::math::IPowIOp>},
     {"pow", {}, genIntIntIntFuncType<16>, genMathOp<mlir::math::IPowIOp>},
     {"pow", {}, genIntIntIntFuncType<32>, genMathOp<mlir::math::IPowIOp>},
     {"pow", {}, genIntIntIntFuncType<64>, genMathOp<mlir::math::IPowIOp>},
     {"pow", "powf", genF32F32F32FuncType, genMathOp<mlir::math::PowFOp>},
     {"pow", "pow", genF64F64F64FuncType, genMathOp<mlir::math::PowFOp>},
-    {"pow", "cpowf", genComplexComplexComplexFuncType<4>, genLibCall},
-    {"pow", "cpow", genComplexComplexComplexFuncType<8>, genLibCall},
+    {"pow", "cpowf", genComplexComplexComplexFuncType<4>, genComplexMathOp<mlir::complex::PowOp>},
+    {"pow", "cpow", genComplexComplexComplexFuncType<8>, genComplexMathOp<mlir::complex::PowOp>},
     // TODO: add PowIOp in math and complex dialects.
     {"pow", "llvm.powi.f32.i32", genF32F32IntFuncType<32>, genLibCall},
     {"pow", "llvm.powi.f64.i32", genF64F64IntFuncType<32>, genLibCall},
     {"pow", RTNAME_STRING(cpowi), genComplexComplexIntFuncType<4, 32>,
      genLibCall},
     {"pow", RTNAME_STRING(zpowi), genComplexComplexIntFuncType<8, 32>,
      genLibCall},
     {"pow", RTNAME_STRING(cpowk), genComplexComplexIntFuncType<4, 64>,
      genLibCall},
     {"pow", RTNAME_STRING(zpowk), genComplexComplexIntFuncType<8, 64>,
      genLibCall},
     {"sign", "copysignf", genF32F32F32FuncType,
      genMathOp<mlir::math::CopySignOp>},
     {"sign", "copysign", genF64F64F64FuncType,
      genMathOp<mlir::math::CopySignOp>},
     {"sign", "copysignl", genF80F80F80FuncType,
      genMathOp<mlir::math::CopySignOp>},
     {"sign", "llvm.copysign.f128", genF128F128F128FuncType,
      genMathOp<mlir::math::CopySignOp>},
     {"sin", "sinf", genF32F32FuncType, genMathOp<mlir::math::SinOp>},
     {"sin", "sin", genF64F64FuncType, genMathOp<mlir::math::SinOp>},
-    {"sin", "csinf", genComplexComplexFuncType<4>, genLibCall},
-    {"sin", "csin", genComplexComplexFuncType<8>, genLibCall},
+    {"sin", "csinf", genComplexComplexFuncType<4>, genComplexMathOp<mlir::complex::SinOp>},
+    {"sin", "csin", genComplexComplexFuncType<8>, genComplexMathOp<mlir::complex::SinOp>},
     {"sinh", "sinhf", genF32F32FuncType, genLibCall},
     {"sinh", "sinh", genF64F64FuncType, genLibCall},
     {"sinh", "csinhf", genComplexComplexFuncType<4>, genLibCall},
     {"sinh", "csinh", genComplexComplexFuncType<8>, genLibCall},
     {"sqrt", "sqrtf", genF32F32FuncType, genMathOp<mlir::math::SqrtOp>},
     {"sqrt", "sqrt", genF64F64FuncType, genMathOp<mlir::math::SqrtOp>},
-    {"sqrt", "csqrtf", genComplexComplexFuncType<4>, genLibCall},
-    {"sqrt", "csqrt", genComplexComplexFuncType<8>, genLibCall},
+    {"sqrt", "csqrtf", genComplexComplexFuncType<4>, genComplexMathOp<mlir::complex::SqrtOp>},
+    {"sqrt", "csqrt", genComplexComplexFuncType<8>, genComplexMathOp<mlir::complex::SqrtOp>},
     {"tan", "tanf", genF32F32FuncType, genMathOp<mlir::math::TanOp>},
     {"tan", "tan", genF64F64FuncType, genMathOp<mlir::math::TanOp>},
-    {"tan", "ctanf", genComplexComplexFuncType<4>, genLibCall},
-    {"tan", "ctan", genComplexComplexFuncType<8>, genLibCall},
+    {"tan", "ctanf", genComplexComplexFuncType<4>, genComplexMathOp<mlir::complex::TanOp>},
+    {"tan", "ctan", genComplexComplexFuncType<8>, genComplexMathOp<mlir::complex::TanOp>},
     {"tanh", "tanhf", genF32F32FuncType, genMathOp<mlir::math::TanhOp>},
     {"tanh", "tanh", genF64F64FuncType, genMathOp<mlir::math::TanhOp>},
-    {"tanh", "ctanhf", genComplexComplexFuncType<4>, genLibCall},
-    {"tanh", "ctanh", genComplexComplexFuncType<8>, genLibCall},
+    {"tanh", "ctanhf", genComplexComplexFuncType<4>, genComplexMathOp<mlir::complex::TanhOp>},
+    {"tanh", "ctanh", genComplexComplexFuncType<8>, genComplexMathOp<mlir::complex::TanhOp>},
 };
 
 // This helper class computes a "distance" between two function types.
 // The distance measures how many narrowing conversions of actual arguments
 // and result of "from" must be made in order to use "to" instead of "from".
 // For instance, the distance between ACOS(REAL(10)) and ACOS(REAL(8)) is
 // greater than the one between ACOS(REAL(10)) and ACOS(REAL(16)). This means
 // if no implementation of ACOS(REAL(10)) is available, it is better to use

flang/lib/Optimizer/CodeGen/CodeGen.cpp

 //===-- CodeGen.cpp -- bridge to lower to LLVM ----------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 #include "flang/Semantics/runtime-type-info.h"
 #include "mlir/Conversion/ArithToLLVM/ArithToLLVM.h"
 #include "mlir/Conversion/ControlFlowToLLVM/ControlFlowToLLVM.h"
 #include "mlir/Conversion/FuncToLLVM/ConvertFuncToLLVM.h"
 #include "mlir/Conversion/LLVMCommon/Pattern.h"
 #include "mlir/Conversion/MathToFuncs/MathToFuncs.h"
 #include "mlir/Conversion/MathToLLVM/MathToLLVM.h"
 #include "mlir/Conversion/MathToLibm/MathToLibm.h"
+#include <mlir/Conversion/ComplexToLLVM/ComplexToLLVM.h>
+#include <mlir/Conversion/ComplexToStandard/ComplexToStandard.h>
 #include "mlir/Conversion/OpenMPToLLVM/ConvertOpenMPToLLVM.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/Pass/Pass.h"
 #include "mlir/Pass/PassManager.h"
 #include "mlir/Target/LLVMIR/ModuleTranslation.h"
 #include "llvm/ADT/ArrayRef.h"
 
 namespace fir {
 #define GEN_PASS_DEF_FIRTOLLVMLOWERING
 #include "flang/Optimizer/CodeGen/CGPasses.h.inc"
 } // namespace fir
 
 #define DEBUG_TYPE "flang-codegen"
 
 // fir::LLVMTypeConverter for converting to LLVM IR dialect types.
void runOnOperation() override final {
 
     // Run dynamic pass pipeline for converting Math dialect
     // operations into other dialects (llvm, func, etc.).
     // Some conversions of Math operations cannot be done
     // by just using conversion patterns. This is true for
     // conversions that affect the ModuleOp, e.g. create new
     // function operations in it. We have to run such conversions
     // as passes here.
     mlir::OpPassManager mathConvertionPM("builtin.module");
     mathConvertionPM.addPass(mlir::createConvertMathToFuncsPass());
+    mathConvertionPM.addPass(mlir::createConvertComplexToStandardPass());
     if (mlir::failed(runPipeline(mathConvertionPM, mod)))
       return signalPassFailure();
 
     auto *context = getModule().getContext();
     fir::LLVMTypeConverter typeConverter{getModule()};
     mlir::RewritePatternSet pattern(context);
     pattern.insert<
         AbsentOpConversion, AddcOpConversion, AddrOfOpConversion,
void runOnOperation() override final {
     mlir::populateOpenMPToLLVMConversionPatterns(typeConverter, pattern);
     mlir::arith::populateArithToLLVMConversionPatterns(typeConverter, pattern);
     mlir::cf::populateControlFlowToLLVMConversionPatterns(typeConverter,
                                                           pattern);
     // Convert math-like dialect operations, which can be produced
     // when late math lowering mode is used, into llvm dialect.
     mlir::populateMathToLLVMConversionPatterns(typeConverter, pattern);
     mlir::populateMathToLibmConversionPatterns(pattern, /*benefit=*/0);
+    mlir::populateComplexToLLVMConversionPatterns(typeConverter, pattern);
     mlir::ConversionTarget target{*context};
     target.addLegalDialect<mlir::LLVM::LLVMDialect>();
     // The OpenMP dialect is legal for Operations without regions, for those
     // which contains regions it is legal if the region contains only the
     // LLVM dialect. Add OpenMP dialect as a legal dialect for conversion and
     // legalize conversion of OpenMP operations without regions.
     mlir::configureOpenMPToLLVMConversionLegality(target, typeConverter);
     target.addLegalDialect<mlir::omp::OpenMPDialect>();

flang/test/Lower/Intrinsics/abs.f90

-! RUN: bbc -emit-fir %s -o - | FileCheck %s
-! RUN: %flang_fc1 -emit-fir %s -o - | FileCheck %s
+! RUN: bbc -emit-fir %s -o - | FileCheck %s --check-prefixes="CHECK,CMPLX,CMPLX-FAST"
+! RUN: bbc -emit-fir --math-runtime=precise %s -o - | FileCheck %s --check-prefixes="CMPLX,CMPLX-PRECISE"
+! RUN: bbc --disable-mlir-complex -emit-fir %s -o - | FileCheck %s --check-prefixes="CMPLX,CMPLX-PRECISE"
+! RUN: %flang_fc1 -emit-fir %s -o - | FileCheck %s --check-prefixes="CHECK,CMPLX,CMPLX-FAST"
+! RUN: %flang_fc1 -emit-fir -mllvm --math-runtime=precise %s -o - | FileCheck %s --check-prefixes="CMPLX,CMPLX-PRECISE"
+! RUN: %flang_fc1 -emit-fir -mllvm --disable-mlir-complex %s -o - | FileCheck %s --check-prefixes="CMPLX,CMPLX-PRECISE"
 
 ! Test abs intrinsic for various types (int, float, complex)
 
 ! CHECK-LABEL: func @_QPabs_testi
 ! CHECK-SAME: %[[VAL_0:.*]]: !fir.ref<i32>{{.*}}, %[[VAL_1:.*]]: !fir.ref<i32>
 subroutine abs_testi(a, b)
 ! CHECK:  %[[VAL_2:.*]] = fir.load %[[VAL_0]] : !fir.ref<i32>
 ! CHECK:  %[[VAL_3:.*]] = arith.constant 31 : i32

 ! CHECK: %[[VAL_2:.*]] = fir.load %[[VAL_0]] : !fir.ref<f128>
 ! CHECK: %[[VAL_3:.*]] = math.absf %[[VAL_2]] : f128
 ! CHECK: fir.store %[[VAL_3]] to %[[VAL_1]] : !fir.ref<f128>
 ! CHECK: return
   real(kind=16) :: a, b
   b = abs(a)
 end subroutine
 
-! CHECK-LABEL: func @_QPabs_testzr(
-! CHECK-SAME:  %[[VAL_0:.*]]: !fir.ref<!fir.complex<4>>{{.*}}, %[[VAL_1:.*]]: !fir.ref<f32>{{.*}}) {
+! CMPLX-LABEL: func @_QPabs_testzr(
+! CMPLX-SAME:  %[[VAL_0:.*]]: !fir.ref<!fir.complex<4>>{{.*}}, %[[VAL_1:.*]]: !fir.ref<f32>{{.*}}) {
 subroutine abs_testzr(a, b)
-! CHECK:  %[[VAL_2:.*]] = fir.load %[[VAL_0]] : !fir.ref<!fir.complex<4>>
-! CHECK:  %[[VAL_3:.*]] = fir.call @cabsf(%[[VAL_2]]) : (!fir.complex<4>) -> f32
-! CHECK:  fir.store %[[VAL_3]] to %[[VAL_1]] : !fir.ref<f32>
-! CHECK:  return
+! CMPLX:  %[[VAL_2:.*]] = fir.load %[[VAL_0]] : !fir.ref<!fir.complex<4>>
+! CMPLX-FAST: %[[VAL_3:.*]] = fir.convert %[[VAL_2]] : (!fir.complex<4>) -> complex<f32>
+! CMPLX-FAST: %[[VAL_4:.*]] = complex.abs %[[VAL_3]] : complex<f32>
+! CMPLX-PRECISE:  %[[VAL_4:.*]] = fir.call @cabsf(%[[VAL_2]]) : (!fir.complex<4>) -> f32
+! CMPLX:  fir.store %[[VAL_4]] to %[[VAL_1]] : !fir.ref<f32>
+! CMPLX:  return
   complex :: a
   real :: b
   b = abs(a)
 end subroutine abs_testzr
 
-! CHECK-LABEL: func @_QPabs_testzd(
-! CHECK-SAME:  %[[VAL_0:.*]]: !fir.ref<!fir.complex<8>>{{.*}}, %[[VAL_1:.*]]: !fir.ref<f64>{{.*}}) {
+! CMPLX-LABEL: func @_QPabs_testzd(
+! CMPLX-SAME:  %[[VAL_0:.*]]: !fir.ref<!fir.complex<8>>{{.*}}, %[[VAL_1:.*]]: !fir.ref<f64>{{.*}}) {
 subroutine abs_testzd(a, b)
-! CHECK:  %[[VAL_2:.*]] = fir.load %[[VAL_0]] : !fir.ref<!fir.complex<8>>
-! CHECK:  %[[VAL_3:.*]] = fir.call @cabs(%[[VAL_2]]) : (!fir.complex<8>) -> f64
-! CHECK:  fir.store %[[VAL_3]] to %[[VAL_1]] : !fir.ref<f64>
-! CHECK:  return
+! CMPLX:  %[[VAL_2:.*]] = fir.load %[[VAL_0]] : !fir.ref<!fir.complex<8>>
+! CMPLX-FAST: %[[VAL_3:.*]] = fir.convert %[[VAL_2]] : (!fir.complex<8>) -> complex<f64>
+! CMPLX-FAST: %[[VAL_4:.*]] = complex.abs %[[VAL_3]] : complex<f64>
+! CMPLX-PRECISE:  %[[VAL_4:.*]] = fir.call @cabs(%[[VAL_2]]) : (!fir.complex<8>) -> f64
+! CMPLX:  fir.store %[[VAL_4]] to %[[VAL_1]] : !fir.ref<f64>
+! CMPLX:  return
   complex(kind=8) :: a
   real(kind=8) :: b
   b = abs(a)
 end subroutine abs_testzd

flang/test/Lower/Intrinsics/exp.f90

-! RUN: bbc -emit-fir -outline-intrinsics %s -o - | FileCheck %s
-! RUN: %flang_fc1 -emit-fir -mllvm -outline-intrinsics %s -o - | FileCheck %s
+! RUN: bbc -emit-fir -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CHECK,CMPLX,CMPLX-FAST"
+! RUN: bbc -emit-fir --math-runtime=precise -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CMPLX,CMPLX-PRECISE"
+! RUN: bbc -emit-fir --disable-mlir-complex -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CMPLX,CMPLX-PRECISE"
+! RUN: %flang_fc1 -emit-fir -mllvm -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CHECK,CMPLX,CMPLX-FAST"
+! RUN: %flang_fc1 -emit-fir -mllvm -outline-intrinsics -mllvm --math-runtime=precise %s -o - | FileCheck %s --check-prefixes="CMPLX,CMPLX-PRECISE"
+! RUN: %flang_fc1 -emit-fir -mllvm -outline-intrinsics -mllvm --disable-mlir-complex %s -o - | FileCheck %s --check-prefixes="CMPLX,CMPLX-PRECISE"
 
 ! CHECK-LABEL: exp_testr
 ! CHECK-SAME: (%[[AREF:.*]]: !fir.ref<f32> {{.*}}, %[[BREF:.*]]: !fir.ref<f32> {{.*}})
 subroutine exp_testr(a, b)
   real :: a, b
 ! CHECK:  %[[A:.*]] = fir.load %[[AREF:.*]] : !fir.ref<f32>
 ! CHECK:  %[[RES:.*]] = fir.call @fir.exp.f32.f32(%[[A]]) : (f32) -> f32
 ! CHECK:  fir.store %[[RES]] to %[[BREF]] : !fir.ref<f32>

 ! CHECK: %[[RESULT32_OUTLINE:.*]] = math.exp %[[ARG32_OUTLINE]] : f32
 ! CHECK: return %[[RESULT32_OUTLINE]] : f32
 
 ! CHECK-LABEL: private @fir.exp.f64.f64
 ! CHECK-SAME: (%[[ARG64_OUTLINE:.*]]: f64) -> f64
 ! CHECK: %[[RESULT64_OUTLINE:.*]] = math.exp %[[ARG64_OUTLINE]] : f64
 ! CHECK: return %[[RESULT64_OUTLINE]] : f64
 
-! CHECK-LABEL: private @fir.exp.z4.z4
-! CHECK-SAME: (%[[ARG32_OUTLINE]]: !fir.complex<4>) -> !fir.complex<4>
-! CHECK: %[[RESULT32_OUTLINE]] = fir.call @cexpf(%[[ARG32_OUTLINE]]) : (!fir.complex<4>) -> !fir.complex<4>
-! CHECK: return %[[RESULT32_OUTLINE]] : !fir.complex<4>
-
-! CHECK-LABEL: private @fir.exp.z8.z8
-! CHECK-SAME: (%[[ARG64_OUTLINE]]: !fir.complex<8>) -> !fir.complex<8>
-! CHECK: %[[RESULT64_OUTLINE]] = fir.call @cexp(%[[ARG64_OUTLINE]]) : (!fir.complex<8>) -> !fir.complex<8>
-! CHECK: return %[[RESULT64_OUTLINE]] : !fir.complex<8>
+! CMPLX-LABEL: private @fir.exp.z4.z4
+! CMPLX-SAME: (%[[ARG32_OUTLINE:.*]]: !fir.complex<4>) -> !fir.complex<4>
+! CMPLX-FAST: %[[C:.*]] = fir.convert %[[ARG32_OUTLINE]] : (!fir.complex<4>) -> complex<f32>
+! CMPLX-FAST: %[[E:.*]] = complex.exp %[[C]] : complex<f32>
+! CMPLX-FAST: %[[RESULT32_OUTLINE:.*]] = fir.convert %[[E]] : (complex<f32>) -> !fir.complex<4>
+! CMPLX-PRECISE: %[[RESULT32_OUTLINE:.*]] = fir.call @cexpf(%[[ARG32_OUTLINE]]) : (!fir.complex<4>) -> !fir.complex<4>
+! CMPLX: return %[[RESULT32_OUTLINE]] : !fir.complex<4>
+
+! CMPLX-LABEL: private @fir.exp.z8.z8
+! CMPLX-SAME: (%[[ARG64_OUTLINE:.*]]: !fir.complex<8>) -> !fir.complex<8>
+! CMPLX-FAST: %[[C:.*]] = fir.convert %[[ARG64_OUTLINE]] : (!fir.complex<8>) -> complex<f64>
+! CMPLX-FAST: %[[E:.*]] = complex.exp %[[C]] : complex<f64>
+! CMPLX-FAST: %[[RESULT64_OUTLINE:.*]] = fir.convert %[[E]] : (complex<f64>) -> !fir.complex<8>
+! CMPLX-PRECISE: %[[RESULT64_OUTLINE:.*]] = fir.call @cexp(%[[ARG64_OUTLINE]]) : (!fir.complex<8>) -> !fir.complex<8>
+! CMPLX: return %[[RESULT64_OUTLINE]] : !fir.complex<8>

flang/test/Lower/Intrinsics/log.f90

-! RUN: bbc -emit-fir -outline-intrinsics %s -o - | FileCheck %s
-! RUN: %flang_fc1 -emit-fir -mllvm -outline-intrinsics %s -o - | FileCheck %s
+! RUN: bbc -emit-fir -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CHECK,CMPLX,CMPLX-FAST"
+! RUN: bbc -emit-fir --math-runtime=precise -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CMPLX,CMPLX-PRECISE"
+! RUN: bbc -emit-fir --disable-mlir-complex -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CMPLX,CMPLX-PRECISE"
+! RUN: %flang_fc1 -emit-fir -mllvm -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CHECK,CMPLX,CMPLX-FAST"
+! RUN: %flang_fc1 -emit-fir -mllvm -outline-intrinsics -mllvm --math-runtime=precise %s -o - | FileCheck %s --check-prefixes="CMPLX,CMPLX-PRECISE"
+! RUN: %flang_fc1 -emit-fir -mllvm -outline-intrinsics -mllvm --disable-mlir-complex %s -o - | FileCheck %s --check-prefixes="CMPLX,CMPLX-PRECISE"
 
 ! CHECK-LABEL: log_testr
 ! CHECK-SAME: (%[[AREF:.*]]: !fir.ref<f32> {{.*}}, %[[BREF:.*]]: !fir.ref<f32> {{.*}})
 subroutine log_testr(a, b)
   real :: a, b
 ! CHECK:  %[[A:.*]] = fir.load %[[AREF:.*]] : !fir.ref<f32>
 ! CHECK:  %[[RES:.*]] = fir.call @fir.log.f32.f32(%[[A]]) : (f32) -> f32
 ! CHECK:  fir.store %[[RES]] to %[[BREF]] : !fir.ref<f32>

 ! CHECK: %[[RESULT32_OUTLINE:.*]] = math.log %[[ARG32_OUTLINE]] : f32
 ! CHECK: return %[[RESULT32_OUTLINE]] : f32
 
 ! CHECK-LABEL: private @fir.log.f64.f64
 ! CHECK-SAME: (%[[ARG64_OUTLINE:.*]]: f64) -> f64
 ! CHECK: %[[RESULT64_OUTLINE:.*]] = math.log %[[ARG64_OUTLINE]] : f64
 ! CHECK: return %[[RESULT64_OUTLINE]] : f64
 
-! CHECK-LABEL: private @fir.log.z4.z4
-! CHECK-SAME: (%[[ARG32_OUTLINE]]: !fir.complex<4>) -> !fir.complex<4>
-! CHECK: %[[RESULT32_OUTLINE]] = fir.call @clogf(%[[ARG32_OUTLINE]]) : (!fir.complex<4>) -> !fir.complex<4>
-! CHECK: return %[[RESULT32_OUTLINE]] : !fir.complex<4>
-
-! CHECK-LABEL: private @fir.log.z8.z8
-! CHECK-SAME: (%[[ARG64_OUTLINE]]: !fir.complex<8>) -> !fir.complex<8>
-! CHECK: %[[RESULT64_OUTLINE]] = fir.call @clog(%[[ARG64_OUTLINE]]) : (!fir.complex<8>) -> !fir.complex<8>
-! CHECK: return %[[RESULT64_OUTLINE]] : !fir.complex<8>
+! CMPLX-LABEL: private @fir.log.z4.z4
+! CMPLX-SAME: (%[[ARG32_OUTLINE:.*]]: !fir.complex<4>) -> !fir.complex<4>
+! CMPLX-FAST: %[[C:.*]] = fir.convert %[[ARG32_OUTLINE]] : (!fir.complex<4>) -> complex<f32>
+! CMPLX-FAST: %[[E:.*]] = complex.log %[[C]] : complex<f32>
+! CMPLX-FAST: %[[RESULT32_OUTLINE:.*]] = fir.convert %[[E]] : (complex<f32>) -> !fir.complex<4>
+! CMPLX-PRECISE: %[[RESULT32_OUTLINE:.*]] = fir.call @clogf(%[[ARG32_OUTLINE]]) : (!fir.complex<4>) -> !fir.complex<4>
+! CMPLX: return %[[RESULT32_OUTLINE]] : !fir.complex<4>
+
+! CMPLX-LABEL: private @fir.log.z8.z8
+! CMPLX-SAME: (%[[ARG64_OUTLINE:.*]]: !fir.complex<8>) -> !fir.complex<8>
+! CMPLX-FAST: %[[C:.*]] = fir.convert %[[ARG64_OUTLINE]] : (!fir.complex<8>) -> complex<f64>
+! CMPLX-FAST: %[[E:.*]] = complex.log %[[C]] : complex<f64>
+! CMPLX-FAST: %[[RESULT64_OUTLINE:.*]] = fir.convert %[[E]] : (complex<f64>) -> !fir.complex<8>
+! CMPLX-PRECISE: %[[RESULT64_OUTLINE:.*]] = fir.call @clog(%[[ARG64_OUTLINE]]) : (!fir.complex<8>) -> !fir.complex<8>
+! CMPLX: return %[[RESULT64_OUTLINE]] : !fir.complex<8>
 
 ! CHECK-LABEL: private @fir.log10.f32.f32
 ! CHECK-SAME: (%[[ARG32_OUTLINE:.*]]: f32) -> f32
 ! CHECK: %[[RESULT32_OUTLINE:.*]] = math.log10 %[[ARG32_OUTLINE]] : f32
 ! CHECK: return %[[RESULT32_OUTLINE]] : f32
 
 ! CHECK-LABEL: private @fir.log10.f64.f64
 ! CHECK-SAME: (%[[ARG64_OUTLINE:.*]]: f64) -> f64
 ! CHECK: %[[RESULT64_OUTLINE:.*]] = math.log10 %[[ARG64_OUTLINE]] : f64
 ! CHECK: return %[[RESULT64_OUTLINE]] : f64

flang/test/Lower/math-lowering.f90

 ! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @llvm.fabs.f128({{%[A-Za-z0-9._]+}}) : (f128) -> f128
 
 function test_complex4(c)
   complex(4) :: c, test_complex4
   test_complex4 = abs(c)
 end function
 
 ! ALL-LABEL: @_QPtest_complex4
-! ALL: {{%[A-Za-z0-9._]+}} = fir.call @cabsf({{%[A-Za-z0-9._]+}}) : (!fir.complex<4>) -> f32
+! FAST: {{%[A-Za-z0-9._]+}} = complex.abs {{%[A-Za-z0-9._]+}} : complex<f32>
+! RELAXED: {{%[A-Za-z0-9._]+}} = complex.abs {{%[A-Za-z0-9._]+}} : complex<f32>
+! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @cabsf({{%[A-Za-z0-9._]+}}) : (!fir.complex<4>) -> f32
 
 function test_complex8(c)
   complex(8) :: c, test_complex8
   test_complex8 = abs(c)
 end function
 
 ! ALL-LABEL: @_QPtest_complex8
-! ALL: {{%[A-Za-z0-9._]+}} = fir.call @cabs({{%[A-Za-z0-9._]+}}) : (!fir.complex<8>) -> f64
+! FAST: {{%[A-Za-z0-9._]+}} = complex.abs {{%[A-Za-z0-9._]+}} : complex<f64>
+! RELAXED: {{%[A-Za-z0-9._]+}} = complex.abs {{%[A-Za-z0-9._]+}} : complex<f64>
+! PRECISE: {{%[A-Za-z0-9._]+}} = fir.call @cabs({{%[A-Za-z0-9._]+}}) : (!fir.complex<8>) -> f64
 
 //--- aint.f90
 ! RUN: bbc -emit-fir %t/aint.f90 -o - --math-runtime=fast | FileCheck --check-prefixes=ALL %t/aint.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=fast %t/aint.f90 -o - | FileCheck --check-prefixes=ALL %t/aint.f90
 ! RUN: bbc -emit-fir %t/aint.f90 -o - --math-runtime=relaxed | FileCheck --check-prefixes=ALL %t/aint.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=relaxed %t/aint.f90 -o - | FileCheck --check-prefixes=ALL %t/aint.f90
 ! RUN: bbc -emit-fir %t/aint.f90 -o - --math-runtime=precise | FileCheck --check-prefixes=ALL %t/aint.f90
 ! RUN: %flang_fc1 -emit-fir -mllvm -math-runtime=precise %t/aint.f90 -o - | FileCheck --check-prefixes=ALL %t/aint.f90

flang/test/Lower/power-operator.f90

-! RUN: bbc -emit-fir %s -o - | FileCheck %s
+! RUN: bbc -emit-fir %s -o - | FileCheck %s --check-prefixes="CHECK,CMPLX-FAST"
+! RUN: bbc --math-runtime=precise -emit-fir %s -o - | FileCheck %s --check-prefixes="CMPLX-PRECISE"
+! RUN: bbc --disable-mlir-complex -emit-fir %s -o - | FileCheck %s --check-prefixes="CMPLX-PRECISE"
+! RUN: %flang_fc1 -emit-fir %s -o - | FileCheck %s --check-prefixes="CHECK,CMPLX-FAST"
+! RUN: %flang_fc1 -emit-fir -mllvm --math-runtime=precise %s -o - | FileCheck %s --check-prefixes="CMPLX-PRECISE"
+! RUN: %flang_fc1 -emit-fir -mllvm --disable-mlir-complex %s -o - | FileCheck %s --check-prefixes="CMPLX-PRECISE"
 
 ! Test power operation lowering
 
 ! CHECK-LABEL: pow_r4_i4
 subroutine pow_r4_i4(x, y, z)
   real :: x, z
   integer :: y
   z = x ** y
subroutine pow_c8_i8(x, y, z)
   z = x ** y
   ! CHECK: call @_FortranAzpowk
 end subroutine
 
 ! CHECK-LABEL: pow_c4_c4
 subroutine pow_c4_c4(x, y, z)
   complex :: x, y, z
   z = x ** y
-  ! CHECK: call @cpowf
+  ! CMPLX-FAST: complex.pow %{{.*}}, %{{.*}} : complex<f32>
+  ! CMPLX-PRECISE: call @cpowf
 end subroutine
 
 ! CHECK-LABEL: pow_c8_c8
 subroutine pow_c8_c8(x, y, z)
   complex(8) :: x, y, z
   z = x ** y
-  ! CHECK: call @cpow
+  ! CMPLX-FAST: complex.pow %{{.*}}, %{{.*}} : complex<f64>
+  ! CMPLX-PRECISE: call @cpow
 end subroutine
 

flang/test/Lower/sqrt.f90

-! RUN: bbc -emit-fir -outline-intrinsics %s -o - | FileCheck %s
-! RUN: %flang_fc1 -emit-fir -mllvm -outline-intrinsics %s -o - | FileCheck %s
+! RUN: bbc -emit-fir -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CHECK,CMPLX-FAST"
+! RUN: bbc --math-runtime=precise -emit-fir -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CMPLX-PRECISE"
+! RUN: bbc --disable-mlir-complex -emit-fir -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CMPLX-PRECISE"
+! RUN: %flang_fc1 -emit-fir -mllvm -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CHECK,CMPLX-FAST"
+! RUN: %flang_fc1 -emit-fir -mllvm --math-runtime=precise -mllvm -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CMPLX-PRECISE"
+! RUN: %flang_fc1 -emit-fir -mllvm --disable-mlir-complex -mllvm -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CMPLX-PRECISE"
 
 ! CHECK-LABEL: sqrt_testr
 subroutine sqrt_testr(a, b)
   real :: a, b
 ! CHECK: fir.call @fir.sqrt.f32.f32
   b = sqrt(a)
 end subroutine
 

 
 ! CHECK-LABEL: @fir.sqrt.f32.f32
 ! CHECK: math.sqrt %{{.*}} : f32
 
 ! CHECK-LABEL: @fir.sqrt.f64.f64
 ! CHECK: math.sqrt %{{.*}} : f64
 
 ! CHECK-LABEL: func private @fir.sqrt.z4.z4
-! CHECK: fir.call @csqrtf
+! CMPLX-FAST: complex.sqrt %{{.*}} : complex<f32>
+! CMPLX-PRECISE: fir.call @csqrtf
 
 ! CHECK-LABEL: @fir.sqrt.z8.z8
-! CHECK: fir.call @csqrt
+! CMPLX-FAST: complex.sqrt %{{.*}} : complex<f64>
+! CMPLX-PRECISE: fir.call @csqrt

flang/test/Lower/trigonometric-intrinsics.f90

-! RUN: bbc -emit-fir -outline-intrinsics %s -o - | FileCheck %s
+! RUN: bbc -emit-fir -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CHECK,CMPLX-FAST"
+! RUN: bbc --math-runtime=precise -emit-fir -outline-intrinsics %s -o - | FileCheck %s --check-prefixes="CMPLX-PRECISE"
 ! RUN: %flang_fc1 -emit-fir -mllvm -outline-intrinsics %s -o - | FileCheck %s
 
 ! CHECK-LABEL: tan_testr
 subroutine tan_testr(a, b)
   real :: a, b
 ! CHECK: fir.call @fir.tan.f32.f32
   b = tan(a)
 end subroutine

 
 ! CHECK-LABEL: @fir.tan.f32.f32
 ! CHECK: math.tan %{{.*}} : f32
 
 ! CHECK-LABEL: @fir.tan.f64.f64
 ! CHECK: math.tan %{{.*}} : f64
 
 ! CHECK-LABEL: @fir.tan.z4.z4
-! CHECK: fir.call @ctanf
+! CMPLX-FAST: complex.tan %{{.*}} : complex<f32>
+! CMPLX-PRECISE: fir.call @ctanf
 
 ! CHECK-LABEL: @fir.tan.z8.z8
-! CHECK: fir.call @ctan
+! CMPLX-FAST: complex.tan %{{.*}} : complex<f64>
+! CMPLX-PRECISE: fir.call @ctan
 
 ! CHECK-LABEL: @fir.atan.f32.f32
 ! CHECK: math.atan %{{.*}} : f32
 
 ! CHECK-LABEL: @fir.atan.f64.f64
 ! CHECK: math.atan %{{.*}} : f64
 
 ! CHECK-LABEL: @fir.atan.z4.z4
 ! CHECK: fir.call @catanf
 
 ! CHECK-LABEL: @fir.atan.z8.z8
 ! CHECK: fir.call @catan
 
 ! CHECK-LABEL: @fir.cos.f32.f32
 ! CHECK: math.cos %{{.*}} : f32
 
 ! CHECK-LABEL: @fir.cos.f64.f64
 ! CHECK: math.cos %{{.*}} : f64
 
 ! CHECK-LABEL: @fir.cos.z4.z4
-! CHECK: fir.call @ccosf
+! CMPLX-FAST: complex.cos %{{.*}} : complex<f32>
+! CMPLX-PRECISE: fir.call @ccosf
 
 ! CHECK-LABEL: @fir.cos.z8.z8
-! CHECK: fir.call @ccos
+! CMPLX-FAST: complex.cos %{{.*}} : complex<f64>
+! CMPLX-PRECISE: fir.call @ccos
 
 ! CHECK-LABEL: @fir.cosh.f32.f32
 ! CHECK: fir.call {{.*}}cosh
 
 ! CHECK-LABEL: @fir.cosh.f64.f64
 ! CHECK: fir.call {{.*}}cosh
 
 ! CHECK-LABEL: @fir.cosh.z4.z4
 ! CHECK: fir.call @ccoshf
 
 ! CHECK-LABEL: @fir.cosh.z8.z8
 ! CHECK: fir.call @ccosh
 
 ! CHECK-LABEL: @fir.sin.f32.f32
 ! CHECK: math.sin %{{.*}} : f32
 
 ! CHECK-LABEL: @fir.sin.f64.f64
 ! CHECK: math.sin %{{.*}} : f64
 
 ! CHECK-LABEL: @fir.sin.z4.z4
-! CHECK: fir.call @csinf
+! CMPLX-FAST: complex.sin %{{.*}} : complex<f32>
+! CMPLX-PRECISE: fir.call @csinf
 
 ! CHECK-LABEL: @fir.sin.z8.z8
-! CHECK: fir.call @csin
+! CMPLX-FAST: complex.sin %{{.*}} : complex<f64>
+! CMPLX-PRECISE: fir.call @csin
 
 ! CHECK-LABEL: @fir.sinh.f32.f32
 ! CHECK: fir.call {{.*}}sinh
 
 ! CHECK-LABEL: @fir.sinh.f64.f64
 ! CHECK: fir.call {{.*}}sinh
 
 ! CHECK-LABEL: @fir.sinh.z4.z4
 ! CHECK: fir.call @csinhf
 
 ! CHECK-LABEL: @fir.sinh.z8.z8
 ! CHECK: fir.call @csinh