[SVE][CostModel] Add instruction cost for operations on scalable vectors

[nasherm]

Summary

The following operations have no associated cost for them
when applied to scalable vectors, and as a consequence
can trigger a crash when a call is made to
AArch64TTIImpl::getCastInstrCost():

• fptrunc
• trunc
• fpext
• fpto(u,s)i

This patch adds costs for these operations and
relevant regression tests.

[sdesmalen]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

555

Please avoid changing the formatting of the table, because that makes this diff unnecessarily big.

llvm/test/CodeGen/AArch64/fptrunc-cost.ll

1 ↗

(On Diff #331808)

The test is in the wrong directory. SVE Cost-model tests should be added in llvm/test/Analysis/CostModel/AArch64

17 ↗

(On Diff #331808)

I think you may be able to write the test very similar to what was done in D97758, where you can invoke the cost-model directly on the fptrunc operation.
Or is there a specific reason that invoking the loop-vectorizer is required here?

[david-arm]

llvm/test/CodeGen/AArch64/fptrunc-cost.ll

17 ↗

(On Diff #331808)

I think that opt -cost-model -analyze doesn't crash because it calculates the cost in a different way. The loop vectoriser decides to do things "differently" by passing the fptrunc op to the getCastInstrCost function. :(

[nasherm]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

555

I believe this was a clang-format change.

llvm/test/CodeGen/AArch64/fptrunc-cost.ll

1 ↗

(On Diff #331808)

Ah my mistake, will fix!

[david-arm]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

555

Hi @nasherm, yeah unfortunately it does happen on my patches too. Typically we try to avoid reformatting large tables like this or large switch-case statements that have a pre-existing format.

[nasherm]

Refactored regression test. Removed clang-format changes

[nasherm]

Remove lint

[david-arm]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

470

nit: I think this should be:

// From nxvmf32 to nxvmf64

473

Are we missing other types/combinations here too? For example:

{ ISD::FP_ROUND, MVT::nxv2f16, MVT::nxv2f64, 1 },

and

{ ISD::FP_ROUND, MVT::nxv4f16, MVT::nxv4f32, 1 },

[nasherm]

Responded to David's comments to take into account half-precision
floats.

[david-arm]

Hi @nasherm, thanks for addressing the previous review comments - it looks much better now! I just have one more comment about the estimated costs in the table.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

473

I think any conversions that involve illegal types that are too large for a single register will be split up into multiple instructions. For example, nxv8f32 is twice the size of a normal SVE register, which means we actually need 2 instructions that convert nxv4f32 -> nxv4f16. Then, we need a final third instruction to interleave these two results together. I'd expect something a bit like:

{ ISD::FP_ROUND, MVT::nxv8f16, MVT::nxv8f32, 3 }, (2 converts + interleave)
...
{ ISD::FP_ROUND, MVT::nxv4f16, MVT::nxv4f64, 3 }, (2 converts + interleave)
...
{ ISD::FP_ROUND, MVT::nxv8f16, MVT::nxv8f64, 7 }, (4 converts + 3 interleaves)
...
{ ISD::FP_ROUND, MVT::nxv4f32, MVT::nxv4f64, 3 }, (2 converts + interleave)
...
{ ISD::FP_ROUND, MVT::nxv8f32, MVT::nxv8f64, 6 }, (4 converts + 2 interleaves)

It's worth pointing out that these costs are just estimates - the most important thing is the costs should be higher to reflect the increased complexity of the operation.

[nasherm]

Combined several patches which all deal with the same problem: instruction
costs for scalable vectors.

[david-arm]

Hi @nasherm, thanks for all the good fixes here! I think it looks very close now - just have a couple of minor comments. I'm fine with most of the costs as the most important thing for now is to avoid crashing and we can refine these later if necessary. However, I think the truncation costs probably do need fixing.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

382

I'm not sure that these instructions are 'free', i.e. a cost of 0. For SVE truncation usually involves a cmpXX instruction to generate a predicate, i.e.

cmpne p1.d, p0/g, z0.d, z1.d

I think we probably want at least a cost of 1 here.

502

I think for conversions from nxv8f32->nxv8i16 there are two instructions + interleaving required here - so maybe a cost of 3?

[nasherm]

Response to David's cost comments, minor changes.

[CarolineConcatto]

llvm/test/Analysis/CostModel/AArch64/sve-fpext.ll

10 ↗

(On Diff #332989)

I believe you can remove the space before ;
and I think you can use CHECK-NEXT

20 ↗

(On Diff #332989)

Also add space between the last CHECK and the first instruction.

[nasherm]

Implemented Carol's suggestions.

[david-arm]

LGTM with nit addressed!

llvm/test/Analysis/CostModel/AArch64/sve-fpext.ll

10 ↗

(On Diff #332989)

nit: Can you address @CarolineConcatto's comment before merging? For example, I think all these tests just a need a simple sed replace:

sed -i 's/  ;CHECK/; CHECK/g'

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

int AArch64TTIImpl::getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
 
   EVT SrcTy = TLI->getValueType(DL, Src);
   EVT DstTy = TLI->getValueType(DL, Dst);
 
   if (!SrcTy.isSimple() || !DstTy.isSimple())
     return AdjustCost(
         BaseT::getCastInstrCost(Opcode, Dst, Src, CCH, CostKind, I));
 
   static const TypeConversionCostTblEntry
   ConversionTbl[] = {
     { ISD::TRUNCATE, MVT::v4i16, MVT::v4i32,  1 },
     { ISD::TRUNCATE, MVT::v4i32, MVT::v4i64,  0 },
     { ISD::TRUNCATE, MVT::v8i8,  MVT::v8i32,  3 },
     { ISD::TRUNCATE, MVT::v16i8, MVT::v16i32, 6 },
 
     // The number of shll instructions for the extension.
     { ISD::SIGN_EXTEND, MVT::v4i64,  MVT::v4i16, 3 },
     { ISD::ZERO_EXTEND, MVT::v4i64,  MVT::v4i16, 3 },
     { ISD::SIGN_EXTEND, MVT::v4i64,  MVT::v4i32, 2 },
     { ISD::ZERO_EXTEND, MVT::v4i64,  MVT::v4i32, 2 },
     { ISD::SIGN_EXTEND, MVT::v8i32,  MVT::v8i8,  3 },
     { ISD::ZERO_EXTEND, MVT::v8i32,  MVT::v8i8,  3 },
     { ISD::SIGN_EXTEND, MVT::v8i32,  MVT::v8i16, 2 },
     { ISD::ZERO_EXTEND, MVT::v8i32,  MVT::v8i16, 2 },
     { ISD::SIGN_EXTEND, MVT::v8i64,  MVT::v8i8,  7 },
ConversionTbl[] = {
 
     // Complex, from v2f64: legal type is v2i32, 1 narrowing => ~2.
     { ISD::FP_TO_SINT, MVT::v2i32, MVT::v2f64, 2 },
     { ISD::FP_TO_SINT, MVT::v2i16, MVT::v2f64, 2 },
     { ISD::FP_TO_SINT, MVT::v2i8,  MVT::v2f64, 2 },
     { ISD::FP_TO_UINT, MVT::v2i32, MVT::v2f64, 2 },
     { ISD::FP_TO_UINT, MVT::v2i16, MVT::v2f64, 2 },
     { ISD::FP_TO_UINT, MVT::v2i8,  MVT::v2f64, 2 },
+
+    // Truncate from nxvmf32 to nxvmf16
+    { ISD::FP_ROUND, MVT::nxv2f16, MVT::nxv2f32, 1 },
+    { ISD::FP_ROUND, MVT::nxv4f16, MVT::nxv4f32, 1 },
+    { ISD::FP_ROUND, MVT::nxv8f16, MVT::nxv8f32, 1 },
+
+    // Truncate from nxvmf64 to nxvmf16
+    { ISD::FP_ROUND, MVT::nxv2f16, MVT::nxv2f64, 1 },
+    { ISD::FP_ROUND, MVT::nxv4f16, MVT::nxv4f64, 1 },
+    { ISD::FP_ROUND, MVT::nxv8f16, MVT::nxv8f64, 1 },
+
+    // Truncate from nxvmf64 to nxvmf32
+    { ISD::FP_ROUND, MVT::nxv2f32, MVT::nxv2f64, 1 },
+    { ISD::FP_ROUND, MVT::nxv4f32, MVT::nxv4f64, 1 },
+    { ISD::FP_ROUND, MVT::nxv8f32, MVT::nxv8f64, 1 },
   };
 
   if (const auto *Entry = ConvertCostTableLookup(ConversionTbl, ISD,
                                                  DstTy.getSimpleVT(),
                                                  SrcTy.getSimpleVT()))
     return AdjustCost(Entry->Cost);
 
   return AdjustCost(
       BaseT::getCastInstrCost(Opcode, Dst, Src, CCH, CostKind, I));
 }
 
 int AArch64TTIImpl::getExtractWithExtendCost(unsigned Opcode, Type *Dst,
                                              VectorType *VecTy,
                                              unsigned Index) {
 
   // Make sure we were given a valid extend opcode.
   assert((Opcode == Instruction::SExt || Opcode == Instruction::ZExt) &&
          "Invalid opcode");
 
   // We are extending an element we extract from a vector, so the source type
   // of the extend is the element type of the vector.
   auto *Src = VecTy->getElementType();
 
   // Sign- and zero-extends are for integer types only.
   assert(isa<IntegerType>(Dst) && isa<IntegerType>(Src) && "Invalid type");
 
   // Get the cost for the extract. We compute the cost (if any) for the extend
int AArch64TTIImpl::getExtractWithExtendCost(unsigned Opcode, Type *Dst,
   }
 
   // If we are unable to perform the extend for free, get the default cost.
   return Cost + getCastInstrCost(Opcode, Dst, Src, TTI::CastContextHint::None,
                                  CostKind);
 }
 
 unsigned AArch64TTIImpl::getCFInstrCost(unsigned Opcode,
                                         TTI::TargetCostKind CostKind) {
   if (CostKind != TTI::TCK_RecipThroughput)
     return Opcode == Instruction::PHI ? 0 : 1;
   assert(CostKind == TTI::TCK_RecipThroughput && "unexpected CostKind");
   // Branches are assumed to be predicted.
   return 0;
 }
 
 int AArch64TTIImpl::getVectorInstrCost(unsigned Opcode, Type *Val,

llvm/test/Analysis/CostModel/AArch64/sve-fptrunc.ll

+; RUN: opt -cost-model -analyze -mtriple aarch64-linux-gnu -mattr=+sve -S -o - < %s  2>&1| FileCheck %s
+
+target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64-unknown-linux-gnu"
+
+define void @sve_fptruncs() {
+  ; CHECK-LABEL: 'sve_fptruncs'
+  ; CHECK: Cost Model: Found an estimated cost of 1 for instruction:   %nxv2_f16_from_f32 = fptrunc <vscale x 2 x float> undef to <vscale x 2 x half>
+  ; CHECK: Cost Model: Found an estimated cost of 1 for instruction:   %nxv4_f16_from_f32 = fptrunc <vscale x 4 x float> undef to <vscale x 4 x half>
+  ; CHECK: Cost Model: Found an estimated cost of 1 for instruction:   %nxv8_f16_from_f32 = fptrunc <vscale x 8 x float> undef to <vscale x 8 x half>
+  ; CHECK: Cost Model: Found an estimated cost of 1 for instruction:   %nxv2_f16_from_f64 = fptrunc <vscale x 2 x double> undef to <vscale x 2 x half>
+  ; CHECK: Cost Model: Found an estimated cost of 1 for instruction:   %nxv4_f16_from_f64 = fptrunc <vscale x 4 x double> undef to <vscale x 4 x half>
+  ; CHECK: Cost Model: Found an estimated cost of 1 for instruction:   %nxv8_f16_from_f64 = fptrunc <vscale x 8 x double> undef to <vscale x 8 x half>
+  ; CHECK: Cost Model: Found an estimated cost of 1 for instruction:   %nxv2_f32_from_f64 = fptrunc <vscale x 2 x double> undef to <vscale x 2 x float>
+  ; CHECK: Cost Model: Found an estimated cost of 1 for instruction:   %nxv4_f32_from_f64 = fptrunc <vscale x 4 x double> undef to <vscale x 4 x float>
+  ; CHECK: Cost Model: Found an estimated cost of 1 for instruction:   %nxv8_f32_from_f64 = fptrunc <vscale x 8 x double> undef to <vscale x 8 x float>
+  %nxv2_f16_from_f32 = fptrunc <vscale x 2 x float> undef to <vscale x 2 x half>
+  %nxv4_f16_from_f32 = fptrunc <vscale x 4 x float> undef to <vscale x 4 x half>
+  %nxv8_f16_from_f32 = fptrunc <vscale x 8 x float> undef to <vscale x 8 x half>
+
+  %nxv2_f16_from_f64 = fptrunc <vscale x 2 x double> undef to <vscale x 2 x half>
+  %nxv4_f16_from_f64 = fptrunc <vscale x 4 x double> undef to <vscale x 4 x half>
+  %nxv8_f16_from_f64 = fptrunc <vscale x 8 x double> undef to <vscale x 8 x half>
+
+  %nxv2_f32_from_f64 = fptrunc <vscale x 2 x double> undef to <vscale x 2 x float>
+  %nxv4_f32_from_f64 = fptrunc <vscale x 4 x double> undef to <vscale x 4 x float>
+  %nxv8_f32_from_f64 = fptrunc <vscale x 8 x double> undef to <vscale x 8 x float>
+
+  ret void
+}