[Flang] Change fir.divc to perform library call rather than generate inline operations.

[SBallantyne]

Summary

Currently fir.divc is always lowered to a sequence of llvm operations to perform complex division, however this causes issues for extreme values when the calculations overflow. While this behaviour would be fine at -Ofast, this is currently the default at all levels.

This patch changes fir.divc to lower to a library call instead, except for when KIND=3 as there is no appropriate library call for this case.

[SBallantyne]

Formatting

[SBallantyne]

Add test to check for generating library call or generating inline code

[vzakhari]

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3643

I guess these names might not be available on all targets. I think we'd better call Fortran runtime library implementation here.

[SBallantyne]

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3643

These functions are provided by compiler-rt, which as i understand is already a requirement for flang. I think it would be better to just reuse the implementation from there, rather than add it again in the fortran runtime library.

[clementval]

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3643

Where did you see it is a requirement?

[kiranchandramohan]

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3643

It is listed in the Readme (https://github.com/llvm/llvm-project/tree/main/flang#building-flang-in-tree). I believe @PeteSteinfeld discussed this before and added it.

[vzakhari]

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3643

I think flang driver does not link clang_rt.builtins (under -flang-experimental-exec), and currently the dependencies on these functions are satisfied by libgcc on Linux. It seems the driver needs to be changed, if we want to rely on compiler-rt implementation. Did I miss something?

[kiranchandramohan]

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3643

That is a good point. We can discuss this in the Wednesday Call.

The build instructions were updated in https://reviews.llvm.org/D116566 by @PeteSteinfeld. I am assuming at that time Pete was using a custom driver. And we probably missed this while adding the flang-experimental-exec flag. Also, not clear whether libgcc covers this on Linux. Either way, it looks like we have to agree and/or update the driver code or the Readme.

[vzakhari]

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3643

I think Pete's comment about the dependency on compiler-rt might be related to the compiler itself, e.g. you may see that -lrt in build/tools/flang/tools/flang-driver/CMakeFiles/flang-new.dir/link.txt

[vzakhari]

I wonder whether intermediate computations generated by ComplexToStandard conversion complex::DivOp honor the range of the element's data type. Maybe we can think of replacing the FIR operation with the one from the Complex dialect?

[kiranchandramohan]

In D145808#4190001, @vzakhari wrote:

I wonder whether intermediate computations generated by ComplexToStandard conversion complex::DivOp honor the range of the element's data type. Maybe we can think of replacing the FIR operation with the one from the Complex dialect?

I think @SBallantyne tried using the mlir::Complex:DivOp first and it did not work.

flang/lib/Optimizer/CodeGen/CodeGen.cpp

3643

Possibly.

[kiranchandramohan]

@jdoerfert Would using functions from compiler-rt cause issues with target offloading?
Context: We are thinking of using functions from compiler-rt for implementing the complex division operation. The existing implementations in Flang (an inline one) and the one on MLIR both have precision issues whereas the compiler-rt ones work fine.

[DavidTruby]

In D145808#4197028, @kiranchandramohan wrote:

@jdoerfert Would using functions from compiler-rt cause issues with target offloading?
Context: We are thinking of using functions from compiler-rt for implementing the complex division operation. The existing implementations in Flang (an inline one) and the one on MLIR both have precision issues whereas the compiler-rt ones work fine.

I think these functions exist for target offloading too, I've used complex numbers in omp regions in C/C++ before and it has worked, and afaict clang just calls these functions. At least for Nvidia there's clang/lib/Headers/__clang_cuda_complex_builtins.h which contains them.

[vzakhari]

In D145808#4199315, @DavidTruby wrote:

In D145808#4197028, @kiranchandramohan wrote:

@jdoerfert Would using functions from compiler-rt cause issues with target offloading?
Context: We are thinking of using functions from compiler-rt for implementing the complex division operation. The existing implementations in Flang (an inline one) and the one on MLIR both have precision issues whereas the compiler-rt ones work fine.

I think these functions exist for target offloading too, I've used complex numbers in omp regions in C/C++ before and it has worked, and afaict clang just calls these functions. At least for Nvidia there's clang/lib/Headers/__clang_cuda_complex_builtins.h which contains them.

Thank you, David. Since it is already the case with clang, I suppose, it should be okay for Flang too.

[SBallantyne]

Update to move half float to inline instead of library call as compiler-rt doesn't currently support divhc3. This may be added at some point as this currently causes errors for clang when libgcc is not available, see this issue: https://github.com/llvm/llvm-project/issues/61914

[PeteSteinfeld]

@SBallantyne, this patch causes problems with COMPLEX division. See the comments I added to the commit -- https://reviews.llvm.org/rGa7bb8e273f433cceeb547e87d04114178573496a

flang/lib/Optimizer/CodeGen/CodeGen.cpp

 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/Pass/Pass.h"
 #include "mlir/Pass/PassManager.h"
 #include "mlir/Target/LLVMIR/ModuleTranslation.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/TypeSwitch.h"
+#include <mlir/IR/ValueRange.h>
 
 namespace fir {
 #define GEN_PASS_DEF_FIRTOLLVMLOWERING
 #include "flang/Optimizer/CodeGen/CGPasses.h.inc"
 } // namespace fir
 
 #define DEBUG_TYPE "flang-codegen"
 
matchAndRewrite(fir::MulcOp mulc, OpAdaptor adaptor,
     auto ra = rewriter.create<mlir::LLVM::UndefOp>(loc, ty);
     auto r1 = rewriter.create<mlir::LLVM::InsertValueOp>(loc, ra, rr, 0);
     auto r0 = rewriter.create<mlir::LLVM::InsertValueOp>(loc, r1, ri, 1);
     rewriter.replaceOp(mulc, r0.getResult());
     return mlir::success();
   }
 };
 
-/// Inlined complex division
+static mlir::LogicalResult getDivc3(fir::DivcOp op,
+                                    mlir::ConversionPatternRewriter &rewriter,
+                                    std::string funcName, mlir::Type returnType,
+                                    llvm::SmallVector<mlir::Type> argType,
+                                    llvm::SmallVector<mlir::Value> args) {
+  auto module = op->getParentOfType<mlir::ModuleOp>();
+  auto loc = op.getLoc();
+  if (mlir::LLVM::LLVMFuncOp divideFunc =
+          module.lookupSymbol<mlir::LLVM::LLVMFuncOp>(funcName)) {
+    auto call = rewriter.create<mlir::LLVM::CallOp>(
+        loc, returnType, mlir::SymbolRefAttr::get(divideFunc), args);
+    rewriter.replaceOp(op, call->getResults());
+    return mlir::success();
+  }
+  mlir::OpBuilder moduleBuilder(
+      op->getParentOfType<mlir::ModuleOp>().getBodyRegion());
+  auto divideFunc = moduleBuilder.create<mlir::LLVM::LLVMFuncOp>(
+      rewriter.getUnknownLoc(), funcName,
+      mlir::LLVM::LLVMFunctionType::get(returnType, argType,
+                                        /*isVarArg=*/false));
+  auto call = rewriter.create<mlir::LLVM::CallOp>(
+      loc, returnType, mlir::SymbolRefAttr::get(divideFunc), args);
+  rewriter.replaceOp(op, call->getResults());
+  return mlir::success();
+}
+
+///  complex division
 struct DivcOpConversion : public FIROpConversion<fir::DivcOp> {
   using FIROpConversion::FIROpConversion;
 
   mlir::LogicalResult
   matchAndRewrite(fir::DivcOp divc, OpAdaptor adaptor,
                   mlir::ConversionPatternRewriter &rewriter) const override {
-    // TODO: Can we use a call to __divdc3 instead?
-    // Just generate inline code for now.
     // given: (x + iy) / (x' + iy')
     // result: ((xx'+yy')/d) + i((yx'-xy')/d) where d = x'x' + y'y'
     mlir::Value a = adaptor.getOperands()[0];
     mlir::Value b = adaptor.getOperands()[1];
     auto loc = divc.getLoc();
     mlir::Type eleTy = convertType(getComplexEleTy(divc.getType()));
-    mlir::Type ty = convertType(divc.getType());
+    llvm::SmallVector<mlir::Type> argTy = {eleTy, eleTy, eleTy, eleTy};
+    mlir::Type firReturnTy = divc.getType();
+    mlir::Type ty = convertType(firReturnTy);
     auto x0 = rewriter.create<mlir::LLVM::ExtractValueOp>(loc, a, 0);
     auto y0 = rewriter.create<mlir::LLVM::ExtractValueOp>(loc, a, 1);
     auto x1 = rewriter.create<mlir::LLVM::ExtractValueOp>(loc, b, 0);
     auto y1 = rewriter.create<mlir::LLVM::ExtractValueOp>(loc, b, 1);
+
+    fir::KindTy kind = (firReturnTy.dyn_cast<fir::ComplexType>()).getFKind();
+    mlir::SmallVector<mlir::Value> args = {x0, y0, x1, y1};
+    switch (kind) {
+    default:
+      llvm_unreachable("Unsupported complex type");
+    case 2:
+      return getDivc3(divc, rewriter, "__divhc3", ty, argTy, args);
+    case 4:
+      return getDivc3(divc, rewriter, "__divsc3", ty, argTy, args);
+    case 8:
+      return getDivc3(divc, rewriter, "__divdc3", ty, argTy, args);
+    case 10:
+      return getDivc3(divc, rewriter, "__divxc3", ty, argTy, args);
+    case 16:
+      return getDivc3(divc, rewriter, "__divtc3", ty, argTy, args);
+    case 3:
+      // No library function for bfloat - use inline version
       auto xx = rewriter.create<mlir::LLVM::FMulOp>(loc, eleTy, x0, x1);
       auto x1x1 = rewriter.create<mlir::LLVM::FMulOp>(loc, eleTy, x1, x1);
       auto yx = rewriter.create<mlir::LLVM::FMulOp>(loc, eleTy, y0, x1);
       auto xy = rewriter.create<mlir::LLVM::FMulOp>(loc, eleTy, x0, y1);
       auto yy = rewriter.create<mlir::LLVM::FMulOp>(loc, eleTy, y0, y1);
       auto y1y1 = rewriter.create<mlir::LLVM::FMulOp>(loc, eleTy, y1, y1);
       auto d = rewriter.create<mlir::LLVM::FAddOp>(loc, eleTy, x1x1, y1y1);
       auto rrn = rewriter.create<mlir::LLVM::FAddOp>(loc, eleTy, xx, yy);
       auto rin = rewriter.create<mlir::LLVM::FSubOp>(loc, eleTy, yx, xy);
       auto rr = rewriter.create<mlir::LLVM::FDivOp>(loc, eleTy, rrn, d);
       auto ri = rewriter.create<mlir::LLVM::FDivOp>(loc, eleTy, rin, d);
       auto ra = rewriter.create<mlir::LLVM::UndefOp>(loc, ty);
       auto r1 = rewriter.create<mlir::LLVM::InsertValueOp>(loc, ra, rr, 0);
       auto r0 = rewriter.create<mlir::LLVM::InsertValueOp>(loc, r1, ri, 1);
       rewriter.replaceOp(divc, r0.getResult());
       return mlir::success();
     }
+  }
 };
 
 /// Inlined complex negation
 struct NegcOpConversion : public FIROpConversion<fir::NegcOp> {
   using FIROpConversion::FIROpConversion;
 
   mlir::LogicalResult
   matchAndRewrite(fir::NegcOp neg, OpAdaptor adaptor,

flang/test/Fir/convert-to-llvm.fir

 
 // CHECK-LABEL: llvm.func @fir_complex_div(
 // CHECK-SAME:                             %[[ARG0:.*]]: !llvm.struct<(f128, f128)>,
 // CHECK-SAME:                             %[[ARG1:.*]]: !llvm.struct<(f128, f128)>) -> !llvm.struct<(f128, f128)> {
 // CHECK:         %[[X0:.*]] = llvm.extractvalue %[[ARG0]][0] : !llvm.struct<(f128, f128)>
 // CHECK:         %[[Y0:.*]] = llvm.extractvalue %[[ARG0]][1] : !llvm.struct<(f128, f128)>
 // CHECK:         %[[X1:.*]] = llvm.extractvalue %[[ARG1]][0] : !llvm.struct<(f128, f128)>
 // CHECK:         %[[Y1:.*]] = llvm.extractvalue %[[ARG1]][1] : !llvm.struct<(f128, f128)>
-// CHECK:         %[[MUL_X0_X1:.*]] = llvm.fmul %[[X0]], %[[X1]]  : f128
-// CHECK:         %[[MUL_X1_X1:.*]] = llvm.fmul %[[X1]], %[[X1]]  : f128
-// CHECK:         %[[MUL_Y0_X1:.*]] = llvm.fmul %[[Y0]], %[[X1]]  : f128
-// CHECK:         %[[MUL_X0_Y1:.*]] = llvm.fmul %[[X0]], %[[Y1]]  : f128
-// CHECK:         %[[MUL_Y0_Y1:.*]] = llvm.fmul %[[Y0]], %[[Y1]]  : f128
-// CHECK:         %[[MUL_Y1_Y1:.*]] = llvm.fmul %[[Y1]], %[[Y1]]  : f128
-// CHECK:         %[[ADD_X1X1_Y1Y1:.*]] = llvm.fadd %[[MUL_X1_X1]], %[[MUL_Y1_Y1]]  : f128
-// CHECK:         %[[ADD_X0X1_Y0Y1:.*]] = llvm.fadd %[[MUL_X0_X1]], %[[MUL_Y0_Y1]]  : f128
-// CHECK:         %[[SUB_Y0X1_X0Y1:.*]] = llvm.fsub %[[MUL_Y0_X1]], %[[MUL_X0_Y1]]  : f128
-// CHECK:         %[[DIV0:.*]] = llvm.fdiv %[[ADD_X0X1_Y0Y1]], %[[ADD_X1X1_Y1Y1]]  : f128
-// CHECK:         %[[DIV1:.*]] = llvm.fdiv %[[SUB_Y0X1_X0Y1]], %[[ADD_X1X1_Y1Y1]]  : f128
-// CHECK:         %{{.*}} = llvm.mlir.undef : !llvm.struct<(f128, f128)>
-// CHECK:         %{{.*}} = llvm.insertvalue %[[DIV0]], %{{.*}}[0] : !llvm.struct<(f128, f128)>
-// CHECK:         %{{.*}} = llvm.insertvalue %[[DIV1]], %{{.*}}[1] : !llvm.struct<(f128, f128)>
+// CHECK:         %[[CALL:.*]] = llvm.call @__divtc3(%[[X0]], %[[Y0]], %[[X1]], %[[Y1]]) : (f128, f128, f128, f128) -> !llvm.struct<(f128, f128)>
 // CHECK:         llvm.return %{{.*}} : !llvm.struct<(f128, f128)>
 
 // -----
 
+// Test FIR complex division inlines for KIND=
+
+func.func @fir_complex_div(%a: !fir.complex<3>, %b: !fir.complex<3>) -> !fir.complex<3> {
+  %c = fir.divc %a, %b : !fir.complex<3>
+  return %c : !fir.complex<3>
+}
+
+// CHECK-LABEL: llvm.func @fir_complex_div(
+// CHECK-SAME:                             %[[ARG0:.*]]: !llvm.struct<(bf16, bf16)>,
+// CHECK-SAME:                             %[[ARG1:.*]]: !llvm.struct<(bf16, bf16)>) -> !llvm.struct<(bf16, bf16)> {
+// CHECK:         %[[X0:.*]] = llvm.extractvalue %[[ARG0]][0] : !llvm.struct<(bf16, bf16)>
+// CHECK:         %[[Y0:.*]] = llvm.extractvalue %[[ARG0]][1] : !llvm.struct<(bf16, bf16)>
+// CHECK:         %[[X1:.*]] = llvm.extractvalue %[[ARG1]][0] : !llvm.struct<(bf16, bf16)>
+// CHECK:         %[[Y1:.*]] = llvm.extractvalue %[[ARG1]][1] : !llvm.struct<(bf16, bf16)>
+// CHECK:         %[[MUL_X0_X1:.*]] = llvm.fmul %[[X0]], %[[X1]]  : bf16
+// CHECK:         %[[MUL_X1_X1:.*]] = llvm.fmul %[[X1]], %[[X1]]  : bf16
+// CHECK:         %[[MUL_Y0_X1:.*]] = llvm.fmul %[[Y0]], %[[X1]]  : bf16
+// CHECK:         %[[MUL_X0_Y1:.*]] = llvm.fmul %[[X0]], %[[Y1]]  : bf16
+// CHECK:         %[[MUL_Y0_Y1:.*]] = llvm.fmul %[[Y0]], %[[Y1]]  : bf16
+// CHECK:         %[[MUL_Y1_Y1:.*]] = llvm.fmul %[[Y1]], %[[Y1]]  : bf16
+// CHECK:         %[[ADD_X1X1_Y1Y1:.*]] = llvm.fadd %[[MUL_X1_X1]], %[[MUL_Y1_Y1]]  : bf16
+// CHECK:         %[[ADD_X0X1_Y0Y1:.*]] = llvm.fadd %[[MUL_X0_X1]], %[[MUL_Y0_Y1]]  : bf16
+// CHECK:         %[[SUB_Y0X1_X0Y1:.*]] = llvm.fsub %[[MUL_Y0_X1]], %[[MUL_X0_Y1]]  : bf16
+// CHECK:         %[[DIV0:.*]] = llvm.fdiv %[[ADD_X0X1_Y0Y1]], %[[ADD_X1X1_Y1Y1]]  : bf16
+// CHECK:         %[[DIV1:.*]] = llvm.fdiv %[[SUB_Y0X1_X0Y1]], %[[ADD_X1X1_Y1Y1]]  : bf16
+// CHECK:         %{{.*}} = llvm.mlir.undef : !llvm.struct<(bf16, bf16)>
+// CHECK:         %{{.*}} = llvm.insertvalue %[[DIV0]], %{{.*}}[0] : !llvm.struct<(bf16, bf16)>
+// CHECK:         %{{.*}} = llvm.insertvalue %[[DIV1]], %{{.*}}[1] : !llvm.struct<(bf16, bf16)>
+// CHECK:         llvm.return %{{.*}} : !llvm.struct<(bf16, bf16)>
+
+
+// -----
+
 // Test FIR complex negation conversion
 // given: -(x + iy)
 // result: -x - iy
 
 func.func @fir_complex_neg(%a: !fir.complex<16>) -> !fir.complex<16> {
   %c = fir.negc %a : !fir.complex<16>
   return %c : !fir.complex<16>
 }