[AArch64TTI] Compute imm materialization cost for AArch64 intrinsics

[fhahn]

Summary

Currently, getIntImmCost returns TCC_Free for almost all intrinsics.
For most AArch64 specific intrinsics however, it looks like integer
constants cannot be folded into most of them (at least the ones I checked).

Unless we know that we can fold integer operands with the intrinsic, we
handle more cases correctly by returning the cost to materialize the
immediate than return TCC_Free.

[SjoerdMeijer]

llvm/test/Transforms/ConstantHoisting/AArch64/const-hoist-intrinsics.ll

2

I thought tests for instruction costs are checked with these kind of opt tests:

opt < %s  -cost-model -analyze -cost-kind=...

is that not a better/more direct way of testing this?

[fhahn]

Update the test to return the materialization cost for AArch64 intrinsics, not all intrinsics not explicitly matched, as intrinsics like lifetime markers or debug instructions should be free.

I am not sure what the best way is to match all AArch64 intrinsics.

[fhahn]

llvm/test/Transforms/ConstantHoisting/AArch64/const-hoist-intrinsics.ll

2

I think the various cost-kinds just focus just on the instruction, not the arguments. So there seems to be no way to get it to print the cost of materializing the immediate.

[merge_guards_bot]

Build result: pass - 60287 tests passed, 0 failed and 732 were skipped.

Log files: console-log.txt, CMakeCache.txt

[SjoerdMeijer]

I wanted to look into this to make a less hand wavy suggestion, but I got lost a little bit lost, which always happens when I look in that jungle that is TTI and TTIImpl etc., so will just ask a question instead.

But I was wondering why can't we look at the instruction, and also look at the (immediate) arguments, and depending on how we expect them to get materialized return a cost that is something like the cost_instruction + cost_immediate?

And then test for something like:

Cost Model: Found an estimated cost of 3 for instruction:   %bar.3 = call i32 @llvm.aarch64.stxr.p0i64(i64 -9223372036317904704, i64* %ptr.3)

[fhahn]

In D70669#1759879, @SjoerdMeijer wrote:

I wanted to look into this to make a less hand wavy suggestion, but I got lost a little bit lost, which always happens when I look in that jungle that is TTI and TTIImpl etc., so will just ask a question instead.

But I was wondering why can't we look at the instruction, and also look at the (immediate) arguments, and depending on how we expect them to get materialized return a cost that is something like the cost_instruction + cost_immediate?

I think currently we just look at the instruction and ignore the operands for the cost. IIRC some passes include the cost of the arguments in the cost they use, but I think in some cases that's quite pass specific. I think the problem with including the cost of the arguments is that in order to compute the cost of an argument that's an instruction, do we also compute the cost of that instruction? How far do we recurse? For constants that's no need to recurse, but I am not sure if there is a big benefit by including their cost?

Anyways, I agree that there's a lot of potential to improve TTI, but I am not sure it should be done as part of this patch :)

[ributzka]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

162

Please convert this to an early return instead.

[fhahn]

Use early exit for AArch64 intrinsics, thanks!

[SjoerdMeijer]

Thanks, and looks reasonable to me.

[merge_guards_bot]

Build result: pass - 60333 tests passed, 0 failed and 732 were skipped.

Log files: console-log.txt, CMakeCache.txt

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

int AArch64TTIImpl::getIntImmCost(Intrinsic::ID IID, unsigned Idx,
   assert(Ty->isIntegerTy());
 
   unsigned BitSize = Ty->getPrimitiveSizeInBits();
   // There is no cost model for constants with a bit size of 0. Return TCC_Free
   // here, so that constant hoisting will ignore this constant.
   if (BitSize == 0)
     return TTI::TCC_Free;
 
+  // Deal with non-AArch64 specific intrinsics depending on the specific
+  // intrinsic, as most of them are free, like debug or lifetime intrinsics. For
+  // AArch64 intrinsics, compute the materialization cost directly.
+  if (IID < Intrinsic::aarch64_addg || IID > Intrinsic::aarch64_udiv)
     switch (IID) {
     default:
       return TTI::TCC_Free;
     case Intrinsic::sadd_with_overflow:
     case Intrinsic::uadd_with_overflow:
     case Intrinsic::ssub_with_overflow:
     case Intrinsic::usub_with_overflow:
     case Intrinsic::smul_with_overflow:
     case Intrinsic::umul_with_overflow:
       if (Idx == 1) {
         int NumConstants = (BitSize + 63) / 64;
         int Cost = AArch64TTIImpl::getIntImmCost(Imm, Ty);
         return (Cost <= NumConstants * TTI::TCC_Basic)
                    ? static_cast<int>(TTI::TCC_Free)
                    : Cost;
       }
       break;
     case Intrinsic::experimental_stackmap:
-    if ((Idx < 2) || (Imm.getBitWidth() <= 64 && isInt<64>(Imm.getSExtValue())))
+      if ((Idx < 2) ||
+          (Imm.getBitWidth() <= 64 && isInt<64>(Imm.getSExtValue())))
         return TTI::TCC_Free;
       break;
     case Intrinsic::experimental_patchpoint_void:
     case Intrinsic::experimental_patchpoint_i64:
-    if ((Idx < 4) || (Imm.getBitWidth() <= 64 && isInt<64>(Imm.getSExtValue())))
+      if ((Idx < 4) ||
+          (Imm.getBitWidth() <= 64 && isInt<64>(Imm.getSExtValue())))
         return TTI::TCC_Free;
       break;
     }
   return AArch64TTIImpl::getIntImmCost(Imm, Ty);
 }
 
 TargetTransformInfo::PopcntSupportKind
 AArch64TTIImpl::getPopcntSupport(unsigned TyWidth) {
   assert(isPowerOf2_32(TyWidth) && "Ty width must be power of 2");
   if (TyWidth == 32 || TyWidth == 64)
     return TTI::PSK_FastHardware;

llvm/test/Transforms/ConstantHoisting/AArch64/const-hoist-intrinsics.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=arm64-darwin-unknown -S -consthoist < %s | FileCheck %s
+
+; Make sure we hoist constants out of intrinsics.
+
+define void @test_stxr(i64* %ptr) {
+; CHECK-LABEL: @test_stxr(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[CONST:%.*]] = bitcast i64 -9223372036317904832 to i64
+; CHECK-NEXT:    [[PTR_0:%.*]] = getelementptr i64, i64* [[PTR:%.*]], i64 0
+; CHECK-NEXT:    [[CONST_MAT:%.*]] = add i64 [[CONST]], -64
+; CHECK-NEXT:    [[BAR_0:%.*]] = call i32 @llvm.aarch64.stxr.p0i64(i64 [[CONST_MAT]], i64* [[PTR_0]])
+; CHECK-NEXT:    [[PTR_1:%.*]] = getelementptr i64, i64* [[PTR]], i64 1
+; CHECK-NEXT:    [[BAR_1:%.*]] = call i32 @llvm.aarch64.stxr.p0i64(i64 [[CONST]], i64* [[PTR_1]])
+; CHECK-NEXT:    [[PTR_2:%.*]] = getelementptr i64, i64* [[PTR]], i64 2
+; CHECK-NEXT:    [[CONST_MAT1:%.*]] = add i64 [[CONST]], 64
+; CHECK-NEXT:    [[BAR_2:%.*]] = call i32 @llvm.aarch64.stxr.p0i64(i64 [[CONST_MAT1]], i64* [[PTR_2]])
+; CHECK-NEXT:    [[PTR_3:%.*]] = getelementptr i64, i64* [[PTR]], i64 3
+; CHECK-NEXT:    [[CONST_MAT2:%.*]] = add i64 [[CONST]], 128
+; CHECK-NEXT:    [[BAR_3:%.*]] = call i32 @llvm.aarch64.stxr.p0i64(i64 [[CONST_MAT2]], i64* [[PTR_3]])
+; CHECK-NEXT:    ret void
+;
+entry:
+  %ptr.0 = getelementptr i64, i64* %ptr, i64 0
+  %bar.0 = call i32 @llvm.aarch64.stxr.p0i64(i64 -9223372036317904896, i64* %ptr.0)
+  %ptr.1 = getelementptr i64, i64* %ptr, i64 1
+  %bar.1 = call i32 @llvm.aarch64.stxr.p0i64(i64 -9223372036317904832,  i64* %ptr.1)
+  %ptr.2 = getelementptr i64, i64* %ptr, i64 2
+  %bar.2 = call i32 @llvm.aarch64.stxr.p0i64(i64 -9223372036317904768, i64* %ptr.2)
+  %ptr.3 = getelementptr i64, i64* %ptr, i64 3
+  %bar.3 = call i32 @llvm.aarch64.stxr.p0i64(i64 -9223372036317904704, i64* %ptr.3)
+  ret void
+}
+
+declare i32 @llvm.aarch64.stxr.p0i64(i64 , i64*)
+
+define i64 @test_udiv(i64 %x) {
+; CHECK-LABEL: @test_udiv(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[CONST:%.*]] = bitcast i64 -9223372036317904832 to i64
+; CHECK-NEXT:    [[CONST_MAT:%.*]] = add i64 [[CONST]], -64
+; CHECK-NEXT:    [[BAR_0:%.*]] = call i64 @llvm.aarch64.udiv.i64(i64 [[CONST_MAT]], i64 [[X:%.*]])
+; CHECK-NEXT:    [[BAR_1:%.*]] = call i64 @llvm.aarch64.udiv.i64(i64 [[CONST]], i64 [[X]])
+; CHECK-NEXT:    [[CONST_MAT1:%.*]] = add i64 [[CONST]], 64
+; CHECK-NEXT:    [[BAR_2:%.*]] = call i64 @llvm.aarch64.udiv.i64(i64 [[CONST_MAT1]], i64 [[X]])
+; CHECK-NEXT:    [[CONST_MAT2:%.*]] = add i64 [[CONST]], 128
+; CHECK-NEXT:    [[BAR_3:%.*]] = call i64 @llvm.aarch64.udiv.i64(i64 [[CONST_MAT2]], i64 [[X]])
+; CHECK-NEXT:    [[RES_1:%.*]] = add i64 [[BAR_0]], [[BAR_1]]
+; CHECK-NEXT:    [[RES_2:%.*]] = add i64 [[RES_1]], [[BAR_2]]
+; CHECK-NEXT:    [[RES_3:%.*]] = add i64 [[RES_2]], [[BAR_3]]
+; CHECK-NEXT:    ret i64 [[RES_3]]
+;
+entry:
+  %bar.0 = call i64 @llvm.aarch64.udiv.i64.i64(i64 -9223372036317904896, i64 %x)
+  %bar.1 = call i64 @llvm.aarch64.udiv.i64.i64(i64 -9223372036317904832,  i64 %x)
+  %bar.2 = call i64 @llvm.aarch64.udiv.i64.i64(i64 -9223372036317904768, i64 %x)
+  %bar.3 = call i64 @llvm.aarch64.udiv.i64.i64(i64 -9223372036317904704, i64 %x)
+  %res.1 = add i64 %bar.0, %bar.1
+  %res.2 = add i64 %res.1, %bar.2
+  %res.3 = add i64 %res.2, %bar.3
+  ret i64 %res.3
+}
+
+declare i64 @llvm.aarch64.udiv.i64.i64(i64, i64)
+
+define void @test_free_intrinsics(i64 %x, i8* %ptr) {
+; CHECK-LABEL: @test_free_intrinsics(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    call void @llvm.lifetime.start.p0i8(i64 100000000032, i8* [[PTR:%.*]])
+; CHECK-NEXT:    call void @llvm.lifetime.start.p0i8(i64 100000000064, i8* [[PTR]])
+; CHECK-NEXT:    call void @llvm.lifetime.end.p0i8(i64 100000000128, i8* [[PTR]])
+; CHECK-NEXT:    [[I:%.*]] = call {}* @llvm.invariant.start.p0i8(i64 100000000256, i8* [[PTR]])
+; CHECK-NEXT:    call void @llvm.invariant.end.p0i8({}* [[I]], i64 100000000256, i8* [[PTR]])
+; CHECK-NEXT:    ret void
+;
+entry:
+  call void @llvm.lifetime.start.p0i8(i64 100000000032, i8* %ptr)
+  call void @llvm.lifetime.start.p0i8(i64 100000000064, i8* %ptr)
+  call void @llvm.lifetime.end.p0i8(i64 100000000128, i8* %ptr)
+  %i = call {}* @llvm.invariant.start.p0i8(i64 100000000256, i8* %ptr)
+  call void @llvm.invariant.end.p0i8({}* %i, i64 100000000256, i8* %ptr)
+  ret void
+}
+
+declare void @llvm.lifetime.start.p0i8(i64, i8*)
+declare void @llvm.lifetime.end.p0i8(i64, i8*)
+
+declare {}* @llvm.invariant.start.p0i8(i64, i8* nocapture)
+declare void @llvm.invariant.end.p0i8({}*, i64, i8* nocapture)