[AArch64] Consider widening instructions in cast cost calculation

[mssimpso]

Summary

The AArch64 instruction set has a few "widening" instructions (e.g., uaddl, saddl, uaddw, etc.) that take one or more doubleword operands and produce quadword results. The operands are automatically sign- or zero-extended as appropriate. However, In LLVM IR, these extends are explicit. This patch updates TTI to consider these widening instructions when determining the cost of sign- and zero-extends. If an extend is a part of a widening operation, it will be eliminated during code generation, so the reported cost should be zero.

[kristof.beyls]

lib/Target/AArch64/AArch64TargetTransformInfo.cpp

195–196

If the idea is that this function only return true if the lengthening/widening is zero-cost; maybe the function name should be isLengtheningFree instead of isLengthening?
Or something similar indicating the expected cost of widening is zero?

lib/Target/AArch64/AArch64TargetTransformInfo.h

46–47

I think the term "widening" is used for this operation, both in the LLVM code base, as well as in the ARMARM.
So, probably best to use isWidening here too.

test/Analysis/CostModel/AArch64/lengthening-casts.ll

2–15

Next to tests that verify that a 0 cost is returned, maybe it'd also be useful to explicitly test that the widening instruction variant actually gets generated?

[mssimpso]

Thanks Kristof! Responses inline.

lib/Target/AArch64/AArch64TargetTransformInfo.cpp

195–196

Sounds good! I like isLengtheningFree.

lib/Target/AArch64/AArch64TargetTransformInfo.h

46–47

From what I've read, ARM uses "lengthening" to refer to the case where both operands are doublewords (e.g., uaddl) and "widening" to refer to the case when only one operand is a doubleword (e.g., uaddw). This patch only deals with the lengthening cases. We might handle the widening cases in the future, but they may not be as straightforward because the operand order matters. I can add a comment here explaining all of this.

test/Analysis/CostModel/AArch64/lengthening-casts.ll

2–15

We actually already have code generation tests for all of these cases over in test/CodeGen/AArch64 (see arm64-vadd.ll for the uaddl cases, for example). I just copied them over here to test the cost model. I'm fine adding an additional "llc" run-line to this test, though, that verifies the code generation property in this context. I checked and there are other cost model tests that do this (e.g., X86/scalarize.ll), so this wouldn't be that unusual.

[kristof.beyls]

lib/Target/AArch64/AArch64TargetTransformInfo.h

46–47

I was making my comment based on Table C3-67 which contains UADDL in the ARMARM copy at https://static.docs.arm.com/ddi0487/b/DDI0487B_a_armv8_arm.pdf.
The table header is "SIMD widening and narrowing arithmetic instructions".
The table entry for UADDL has "UADDL, UADDL2 Unsigned add long (vector form)".
Therefore, I am assuming the L in the mnemonic stands for "long" not "lengthening".

Maybe there is other documentation that specifically calls this specific case "lengthening"?

test/Analysis/CostModel/AArch64/lengthening-casts.ll

2–15

Oh, we already have the tests to check the actual code generation. Than I think this is fine.
I think that you could argue both for and against the actual code generation test being in the same location as the cost model test.
I don't feel strongly enough either way to argue.

[mssimpso]

lib/Target/AArch64/AArch64TargetTransformInfo.h

46–47

I was looking at different documentation (possibly outdated): http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0024a/ch07s03.html. Yours looks better to me, so let's go with isWideningFree, like you suggested.

[mssimpso]

lib/Target/AArch64/AArch64TargetTransformInfo.h

46–47

Also, it looks like we already handle the W variants as well, where only one operand is extended. I was thinking the operand order in the IR might affect our ability to match these instructions, but it doesn't.

I'll update the patch to consider W variants (this will actually make it simpler since we don't need to care about the other operand of the binop), and add W tests.

[mssimpso]

Addressed Kristof's comments.

• Changed wording to refer to "widening" instructions rather than "lengthening" instructions.
• Changed function name to is isWideningFree.
• Updated the logic to consider the W instruction variants in addition to the L variants.
• Added code generation tests along side the cost test cases.
• Added additional tests for the W variants.

[kristof.beyls]

In D32706#743682, @mssimpso wrote:

Addressed Kristof's comments.

• Changed wording to refer to "widening" instructions rather than "lengthening" instructions.
• Changed function name to is isWideningFree.
• Updated the logic to consider the W instruction variants in addition to the L variants.
• Added code generation tests along side the cost test cases.
• Added additional tests for the W variants.

Thanks Matt.
I'm not familiar enough with this code to feel comfortable approving, so I think someone else should do that.
Other than that, I only have one more remark, now that I've looked at the patch again: maybe there should also be a "negative" test, i.e. verifying that a non-zero cost is computed when a widening instruction cannot be generated, e.g. when the type operated on cannot map to a native instruction?

[mssimpso]

In D32706#744317, @kristof.beyls wrote:

Thanks Matt.
I'm not familiar enough with this code to feel comfortable approving, so I think someone else should do that.
Other than that, I only have one more remark, now that I've looked at the patch again: maybe there should also be a "negative" test, i.e. verifying that a non-zero cost is computed when a widening instruction cannot be generated, e.g. when the type operated on cannot map to a native instruction?

Thanks Kristof! I'll add some negative tests and wait for more reviews.

[mssimpso]

Added some negative tests.

[evandro]

It shouldn't be assumed that such widening instructions are always free. Their latency may be longer than the equivalent instruction sequences. Preferably, the cost of both choices should weighed after consulting the backend.

[mssimpso]

In D32706#745289, @evandro wrote:

It shouldn't be assumed that such widening instructions are always free. Their latency may be longer than the equivalent instruction sequences. Preferably, the cost of both choices should weighed after consulting the backend.

Hi Evandro,

Thanks for the comment.

We mean "free" in the sense that the extends considered in this patch don't exist in the generated code. They're only there in the IR to make it well-typed (i.e., an IR "add" can't operate on <2 x i32> vectors and produce a <2 x i64> result). This is consistent with the way we use "free" in other places in the cost model, when applied to extends.

If you're suggesting, for example, that an ADD and a UADDL might in theory have different max latencies, I suppose this could be true for some hardware, but in a quick look through the sub-target scheduling models, none of them seem make a distinction that I saw. I think we generally prefer to omit sub-target details from the cost-model unless necessary.

I'm not sure what you mean by "both choices" since there's only once code sequence that is generated. Do you mean that the backed should decide whether to generate, for example, an extend plus ADD vs. UADDL based on cost? I don't think that's something we can do today,

[evandro]

In D32706#745329, @mssimpso wrote:

We mean "free" in the sense that the extends considered in this patch don't exist in the generated code. They're only there in the IR to make it well-typed (i.e., an IR "add" can't operate on <2 x i32> vectors and produce a <2 x i64> result). This is consistent with the way we use "free" in other places in the cost model, when applied to extends.

I see.

If you're suggesting, for example, that an ADD and a UADDL might in theory have different max latencies, I suppose this could be true for some hardware, but in a quick look through the sub-target scheduling models, none of them seem make a distinction that I saw. I think we generally prefer to omit sub-target details from the cost-model unless necessary.

Exynos M1-2 do: 1 cycle for ASIMD ADD and 3 cycles for ASIMD SADDL. I tried this patch on Exynos and noticed minor regressions and no improvements in a handful of workloads.

I'm not sure what you mean by "both choices" since there's only once code sequence that is generated. Do you mean that the backed should decide whether to generate, for example, an extend plus ADD vs. UADDL based on cost? I don't think that's something we can do today,

I'll have to think more about this.

[mssimpso]

In D32706#745348, @evandro wrote:

Exynos M1-2 do: 1 cycle for ASIMD ADD and 3 cycles for ASIMD SADDL. I tried this patch on Exynos and noticed minor regressions and no improvements in a handful of workloads.

OK, yes I see that now. I missed it on the first look! What do you think about adding a new sub-target feature for this then? Something like "FastWideningOps"? We could condition this TTI change on the feature to prevent it from hurting performance on Exynos. And regarding your last point, you could then later use it to guide instruction selection to not generate the widening ops if the non-widening versions (with added extends) are more profitable.

[evandro]

In D32706#746087, @mssimpso wrote:

OK, yes I see that now. I missed it on the first look! What do you think about adding a new sub-target feature for this then? Something like "FastWideningOps"? We could condition this TTI change on the feature to prevent it from hurting performance on Exynos. And regarding your last point, you could then later use it to guide instruction selection to not generate the widening ops if the non-widening versions (with added extends) are more profitable.

Please, not yet another feature. It should be possible, methinks, to getSubtarget().getSchedModel() to compare the relative costs.

Do you have a publicly available benchmark where this change is meaningful so that I can evaluate it better? For I haven't seen any change in our workloads after this change. Maybe, in the end, it doesn't matter; or maybe it's not that common.

[mssimpso]

In D32706#746122, @evandro wrote:

Please, not yet another feature. It should be possible, methinks, to getSubtarget().getSchedModel() to compare the relative costs.

I can't really comment on whether that would be possible or not, because I haven't looked at it. It seems to be somewhat tangential to this patch, though, since we currently always generate the widening instruction variants if possible, eliminating the extends. So I'm not sure how that would affect TTI. It seems like you're wanting to change instruction selection?

Do you have a publicly available benchmark where this change is meaningful so that I can evaluate it better? For I haven't seen any change in our workloads after this change. Maybe, in the end, it doesn't matter; or maybe it's not that common.

Yes, this change is primarily to encourage more aggressive SLP in spec2006/h264ref. It provides a ~17% improvement on Falkor.

[mssimpso]

Addressed Evandro's concerns.

Hey Evando,

I appreciate all the feedback. After discussing your concerns with @gberry, he suggested an alternative solution, which I've implemented in this updated version of the patch. The basic idea is to treat the widening instructions as single operations (rather than operations composed of two or three instructions), whose cost is attached to the arithmetic instruction. This design conceptually better matches the code that we generate.

I've rearranged the patch so that we (1) mark the extends as "free" and (2) apply a sub-target specified overhead to the arithmetic instructions that are widening. I've left the default overhead zero (so this version of the patch is functionally the same as the last one), but sub-targets will now have the flexibility to increase this overhead to better match the capabilities of their hardware. Thus, an "add" ultimately mapping to UADDL can have a different cost than an "add" mapping to ADD, for example. Does this make sense?

Please let me know what you think.

[evandro]

@mssimpso, yes, it makes a lot of sense! This is a more sensible approach, methinks.

[mssimpso]

Thanks Evandro!

lib/Target/AArch64/AArch64TargetTransformInfo.h

class AArch64TTIImpl : public BasicTTIImplBase<AArch64TTIImpl> {
   const AArch64TargetLowering *getTLI() const { return TLI; }
 
   enum MemIntrinsicType {
     VECTOR_LDST_TWO_ELEMENTS,
     VECTOR_LDST_THREE_ELEMENTS,
     VECTOR_LDST_FOUR_ELEMENTS
   };
 
+  bool isLengthening(Type *Dst, Type *Src, const Instruction *I);
+
 public:
   explicit AArch64TTIImpl(const AArch64TargetMachine *TM, const Function &F)
       : BaseT(TM, F.getParent()->getDataLayout()), ST(TM->getSubtargetImpl(F)),
         TLI(ST->getTargetLowering()) {}
 
   /// \name Scalar TTI Implementations
   /// @{
 

lib/Target/AArch64/AArch64TargetTransformInfo.cpp

 AArch64TTIImpl::getPopcntSupport(unsigned TyWidth) {
   assert(isPowerOf2_32(TyWidth) && "Ty width must be power of 2");
   if (TyWidth == 32 || TyWidth == 64)
     return TTI::PSK_FastHardware;
   // TODO: AArch64TargetLowering::LowerCTPOP() supports 128bit popcount.
   return TTI::PSK_Software;
 }
 
+bool AArch64TTIImpl::isLengthening(Type *Dst, Type *Src, const Instruction *I) {
+
+  // A helper that returns true if the given instruction is a sign- or
+  // zero-extend.
+  auto isExtend = [](const Instruction *I) {
+    return isa<SExtInst>(I) || isa<ZExtInst>(I);
+  };
+
+  // Exit early if the cast is not an extension, has more that one use, or is
+  // not a vector cast.
+  if (!isExtend(I) || !I->hasOneUse() || !Dst->isVectorTy())
+    return false;
+
+  // Determine if the single user of the cast is an operation having a
+  // lengthening variant.
+  //
+  // TODO: Add additional lengthening operations (e.g., mul, shl, etc.) once we
+  //       verify that the extends are eliminated during code generation.
+  auto *SingleUser = cast<Instruction>(*I->user_begin());
+  switch (SingleUser->getOpcode()) {
+  case Instruction::Add: // UADDL(2), SADDL(2)
+  case Instruction::Sub: // USUBL(2), SSUBL(2)
+    break;
+  default:
+    return false;
+  }
+
+  // Get the source scalar type. "SrcIRTy" can differ from "Src" since it's
+  // associated with an observable IR instruction, whereas "Src" may not be.
+  // For example, we may be computing the cost of vectorizing the instruction.
+  // In this case, "SrcIRTy" would be the original scalar type and "Src" would
+  // be the vector type for which we're computing the cost.
+  auto *SrcIRTy = cast<CastInst>(I)->getSrcTy();
+
+  // Lengthening instructions operate on doubleword operands and produce
+  // quadword results. Thus, both operands must be an extension of the same
+  // kind.
+  for (const Use &U : SingleUser->operands()) {
+    auto *Cast = dyn_cast<CastInst>(U.get());
+    if (!Cast || !isExtend(Cast) || (!I && Cast->getSrcTy() != SrcIRTy))
+      return false;
+  }
+
+  // Legalize the destination type and ensure it can be used in a lengthening
+  // operation (i.e., it must be 8h, 4s or 2d).
+  auto DstTy = TLI->getTypeLegalizationCost(DL, Dst);
+  unsigned DstElTySize = DstTy.second.getScalarSizeInBits();
+  if (!DstTy.second.is128BitVector() || DstElTySize < 16)
+    return false;
+
+  // Get the total number of vector elements in the legalized destination type.
+  unsigned NumDstEls = DstTy.first * DstTy.second.getVectorNumElements();
+
+  // Legalize the source type and ensure it is a vector.
+  auto SrcTy = TLI->getTypeLegalizationCost(DL, Src);
+  if (!SrcTy.second.isVector())
+    return false;
+
+  // Get the source vector element size and total number of elements in the
+  // legalized source type.
+  unsigned SrcElTySize = SrcTy.second.getScalarSizeInBits();
+  unsigned NumSrcEls = SrcTy.first * SrcTy.second.getVectorNumElements();
+
+  // Return true if the legalized source and destination types have the same
+  // number of vector elements and the destination element type size is twice
+  // that of the source type.
+  return NumDstEls == NumSrcEls && 2 * SrcElTySize == DstElTySize;
+}
+
 int AArch64TTIImpl::getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
                                      const Instruction *I) {
   int ISD = TLI->InstructionOpcodeToISD(Opcode);
   assert(ISD && "Invalid opcode");
 
+  // If the cast is observable, and it's used by an instruction having a
+  // lengthening variant (e.g., uaddl, saddl, etc.), it may be free.
+  if (I && isLengthening(Dst, Src, I))
+    return 0;
+
   EVT SrcTy = TLI->getValueType(DL, Src);
   EVT DstTy = TLI->getValueType(DL, Dst);
 
   if (!SrcTy.isSimple() || !DstTy.isSimple())
     return BaseT::getCastInstrCost(Opcode, Dst, Src);
 
   static const TypeConversionCostTblEntry
   ConversionTbl[] = {

test/Analysis/CostModel/AArch64/lengthening-casts.ll

+; RUN: opt < %s -cost-model -analyze | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64--linux-gnu"
+
+; CHECK-LABEL: uaddl_8h
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = zext <8 x i8> %a to <8 x i16>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = zext <8 x i8> %b to <8 x i16>
+define <8 x i16> @uaddl_8h(<8 x i8> %a, <8 x i8> %b) {
+  %tmp0 = zext <8 x i8> %a to <8 x i16>
+  %tmp1 = zext <8 x i8> %b to <8 x i16>
+  %tmp2 = add <8 x i16> %tmp0, %tmp1
+  ret <8 x i16> %tmp2
+}
+
+; CHECK-LABEL: uaddl_4s
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = zext <4 x i16> %a to <4 x i32>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = zext <4 x i16> %b to <4 x i32>
+define <4 x i32> @uaddl_4s(<4 x i16> %a, <4 x i16> %b) {
+  %tmp0 = zext <4 x i16> %a to <4 x i32>
+  %tmp1 = zext <4 x i16> %b to <4 x i32>
+  %tmp2 = add <4 x i32> %tmp0, %tmp1
+  ret <4 x i32> %tmp2
+}
+
+; CHECK-LABEL: uaddl_2d
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = zext <2 x i32> %a to <2 x i64>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = zext <2 x i32> %b to <2 x i64>
+define <2 x i64> @uaddl_2d(<2 x i32> %a, <2 x i32> %b) {
+  %tmp0 = zext <2 x i32> %a to <2 x i64>
+  %tmp1 = zext <2 x i32> %b to <2 x i64>
+  %tmp2 = add <2 x i64> %tmp0, %tmp1
+  ret <2 x i64> %tmp2
+}
+
+; CHECK-LABEL: uaddl2_8h
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = zext <16 x i8> %a to <16 x i16>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = zext <16 x i8> %b to <16 x i16>
+define <16 x i16> @uaddl2_8h(<16 x i8> %a, <16 x i8> %b) {
+  %tmp0 = zext <16 x i8> %a to <16 x i16>
+  %tmp1 = zext <16 x i8> %b to <16 x i16>
+  %tmp2 = add <16 x i16> %tmp0, %tmp1
+  ret <16 x i16> %tmp2
+}
+
+; CHECK-LABEL: uaddl2_4s
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = zext <8 x i16> %a to <8 x i32>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = zext <8 x i16> %b to <8 x i32>
+define <8 x i32> @uaddl2_4s(<8 x i16> %a, <8 x i16> %b) {
+  %tmp0 = zext <8 x i16> %a to <8 x i32>
+  %tmp1 = zext <8 x i16> %b to <8 x i32>
+  %tmp2 = add <8 x i32> %tmp0, %tmp1
+  ret <8 x i32> %tmp2
+}
+
+; CHECK-LABEL: uaddl2_2d
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = zext <4 x i32> %a to <4 x i64>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = zext <4 x i32> %b to <4 x i64>
+define <4 x i64> @uaddl2_2d(<4 x i32> %a, <4 x i32> %b) {
+  %tmp0 = zext <4 x i32> %a to <4 x i64>
+  %tmp1 = zext <4 x i32> %b to <4 x i64>
+  %tmp2 = add <4 x i64> %tmp0, %tmp1
+  ret <4 x i64> %tmp2
+}
+
+; CHECK-LABEL: saddl_8h
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = sext <8 x i8> %a to <8 x i16>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = sext <8 x i8> %b to <8 x i16>
+define <8 x i16> @saddl_8h(<8 x i8> %a, <8 x i8> %b) {
+  %tmp0 = sext <8 x i8> %a to <8 x i16>
+  %tmp1 = sext <8 x i8> %b to <8 x i16>
+  %tmp2 = add <8 x i16> %tmp0, %tmp1
+  ret <8 x i16> %tmp2
+}
+
+; CHECK-LABEL: saddl_4s
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = sext <4 x i16> %a to <4 x i32>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = sext <4 x i16> %b to <4 x i32>
+define <4 x i32> @saddl_4s(<4 x i16> %a, <4 x i16> %b) {
+  %tmp0 = sext <4 x i16> %a to <4 x i32>
+  %tmp1 = sext <4 x i16> %b to <4 x i32>
+  %tmp2 = add <4 x i32> %tmp0, %tmp1
+  ret <4 x i32> %tmp2
+}
+
+; CHECK-LABEL: saddl_2d
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = sext <2 x i32> %a to <2 x i64>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = sext <2 x i32> %b to <2 x i64>
+define <2 x i64> @saddl_2d(<2 x i32> %a, <2 x i32> %b) {
+  %tmp0 = sext <2 x i32> %a to <2 x i64>
+  %tmp1 = sext <2 x i32> %b to <2 x i64>
+  %tmp2 = add <2 x i64> %tmp0, %tmp1
+  ret <2 x i64> %tmp2
+}
+
+; CHECK-LABEL: saddl2_8h
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = sext <16 x i8> %a to <16 x i16>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = sext <16 x i8> %b to <16 x i16>
+define <16 x i16> @saddl2_8h(<16 x i8> %a, <16 x i8> %b) {
+  %tmp0 = sext <16 x i8> %a to <16 x i16>
+  %tmp1 = sext <16 x i8> %b to <16 x i16>
+  %tmp2 = add <16 x i16> %tmp0, %tmp1
+  ret <16 x i16> %tmp2
+}
+
+; CHECK-LABEL: saddl2_4s
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = sext <8 x i16> %a to <8 x i32>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = sext <8 x i16> %b to <8 x i32>
+define <8 x i32> @saddl2_4s(<8 x i16> %a, <8 x i16> %b) {
+  %tmp0 = sext <8 x i16> %a to <8 x i32>
+  %tmp1 = sext <8 x i16> %b to <8 x i32>
+  %tmp2 = add <8 x i32> %tmp0, %tmp1
+  ret <8 x i32> %tmp2
+}
+
+; CHECK-LABEL: saddl2_2d
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = sext <4 x i32> %a to <4 x i64>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = sext <4 x i32> %b to <4 x i64>
+define <4 x i64> @saddl2_2d(<4 x i32> %a, <4 x i32> %b) {
+  %tmp0 = sext <4 x i32> %a to <4 x i64>
+  %tmp1 = sext <4 x i32> %b to <4 x i64>
+  %tmp2 = add <4 x i64> %tmp0, %tmp1
+  ret <4 x i64> %tmp2
+}
+
+; CHECK-LABEL: usubl_8h
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = zext <8 x i8> %a to <8 x i16>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = zext <8 x i8> %b to <8 x i16>
+define <8 x i16> @usubl_8h(<8 x i8> %a, <8 x i8> %b) {
+  %tmp0 = zext <8 x i8> %a to <8 x i16>
+  %tmp1 = zext <8 x i8> %b to <8 x i16>
+  %tmp2 = sub <8 x i16> %tmp0, %tmp1
+  ret <8 x i16> %tmp2
+}
+
+; CHECK-LABEL: usubl_4s
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = zext <4 x i16> %a to <4 x i32>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = zext <4 x i16> %b to <4 x i32>
+define <4 x i32> @usubl_4s(<4 x i16> %a, <4 x i16> %b) {
+  %tmp0 = zext <4 x i16> %a to <4 x i32>
+  %tmp1 = zext <4 x i16> %b to <4 x i32>
+  %tmp2 = sub <4 x i32> %tmp0, %tmp1
+  ret <4 x i32> %tmp2
+}
+
+; CHECK-LABEL: usubl_2d
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = zext <2 x i32> %a to <2 x i64>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = zext <2 x i32> %b to <2 x i64>
+define <2 x i64> @usubl_2d(<2 x i32> %a, <2 x i32> %b) {
+  %tmp0 = zext <2 x i32> %a to <2 x i64>
+  %tmp1 = zext <2 x i32> %b to <2 x i64>
+  %tmp2 = sub <2 x i64> %tmp0, %tmp1
+  ret <2 x i64> %tmp2
+}
+
+; CHECK-LABEL: usubl2_8h
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = zext <16 x i8> %a to <16 x i16>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = zext <16 x i8> %b to <16 x i16>
+define <16 x i16> @usubl2_8h(<16 x i8> %a, <16 x i8> %b) {
+  %tmp0 = zext <16 x i8> %a to <16 x i16>
+  %tmp1 = zext <16 x i8> %b to <16 x i16>
+  %tmp2 = sub <16 x i16> %tmp0, %tmp1
+  ret <16 x i16> %tmp2
+}
+
+; CHECK-LABEL: usubl2_4s
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = zext <8 x i16> %a to <8 x i32>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = zext <8 x i16> %b to <8 x i32>
+define <8 x i32> @usubl2_4s(<8 x i16> %a, <8 x i16> %b) {
+  %tmp0 = zext <8 x i16> %a to <8 x i32>
+  %tmp1 = zext <8 x i16> %b to <8 x i32>
+  %tmp2 = sub <8 x i32> %tmp0, %tmp1
+  ret <8 x i32> %tmp2
+}
+
+; CHECK-LABEL: usubl2_2d
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = zext <4 x i32> %a to <4 x i64>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = zext <4 x i32> %b to <4 x i64>
+define <4 x i64> @usubl2_2d(<4 x i32> %a, <4 x i32> %b) {
+  %tmp0 = zext <4 x i32> %a to <4 x i64>
+  %tmp1 = zext <4 x i32> %b to <4 x i64>
+  %tmp2 = sub <4 x i64> %tmp0, %tmp1
+  ret <4 x i64> %tmp2
+}
+
+; CHECK-LABEL: ssubl_8h
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = sext <8 x i8> %a to <8 x i16>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = sext <8 x i8> %b to <8 x i16>
+define <8 x i16> @ssubl_8h(<8 x i8> %a, <8 x i8> %b) {
+  %tmp0 = sext <8 x i8> %a to <8 x i16>
+  %tmp1 = sext <8 x i8> %b to <8 x i16>
+  %tmp2 = sub <8 x i16> %tmp0, %tmp1
+  ret <8 x i16> %tmp2
+}
+
+; CHECK-LABEL: ssubl_4s
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = sext <4 x i16> %a to <4 x i32>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = sext <4 x i16> %b to <4 x i32>
+define <4 x i32> @ssubl_4s(<4 x i16> %a, <4 x i16> %b) {
+  %tmp0 = sext <4 x i16> %a to <4 x i32>
+  %tmp1 = sext <4 x i16> %b to <4 x i32>
+  %tmp2 = sub <4 x i32> %tmp0, %tmp1
+  ret <4 x i32> %tmp2
+}
+
+; CHECK-LABEL: ssubl_2d
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = sext <2 x i32> %a to <2 x i64>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = sext <2 x i32> %b to <2 x i64>
+define <2 x i64> @ssubl_2d(<2 x i32> %a, <2 x i32> %b) {
+  %tmp0 = sext <2 x i32> %a to <2 x i64>
+  %tmp1 = sext <2 x i32> %b to <2 x i64>
+  %tmp2 = sub <2 x i64> %tmp0, %tmp1
+  ret <2 x i64> %tmp2
+}
+
+; CHECK-LABEL: ssubl2_8h
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = sext <16 x i8> %a to <16 x i16>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = sext <16 x i8> %b to <16 x i16>
+define <16 x i16> @ssubl2_8h(<16 x i8> %a, <16 x i8> %b) {
+  %tmp0 = sext <16 x i8> %a to <16 x i16>
+  %tmp1 = sext <16 x i8> %b to <16 x i16>
+  %tmp2 = sub <16 x i16> %tmp0, %tmp1
+  ret <16 x i16> %tmp2
+}
+
+; CHECK-LABEL: ssubl2_4s
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = sext <8 x i16> %a to <8 x i32>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = sext <8 x i16> %b to <8 x i32>
+define <8 x i32> @ssubl2_4s(<8 x i16> %a, <8 x i16> %b) {
+  %tmp0 = sext <8 x i16> %a to <8 x i32>
+  %tmp1 = sext <8 x i16> %b to <8 x i32>
+  %tmp2 = sub <8 x i32> %tmp0, %tmp1
+  ret <8 x i32> %tmp2
+}
+
+; CHECK-LABEL: ssubl2_2d
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp0 = sext <4 x i32> %a to <4 x i64>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %tmp1 = sext <4 x i32> %b to <4 x i64>
+define <4 x i64> @ssubl2_2d(<4 x i32> %a, <4 x i32> %b) {
+  %tmp0 = sext <4 x i32> %a to <4 x i64>
+  %tmp1 = sext <4 x i32> %b to <4 x i64>
+  %tmp2 = sub <4 x i64> %tmp0, %tmp1
+  ret <4 x i64> %tmp2
+}