[AArch64][SelectionDAG] Optimize multiplication by constant

[Allen]

Summary

InstCombine will merge the x*5+x into x*6,
then decompose MUL By const into shift/add or shift/sub, eg:
Change the costmodel to lower a = b * C where C = (2^n + 1) * 2^m to

add     w0, w0, w0, lsl n
lsl     w0, w0, m

Fixes AArch64 part of https://github.com/llvm/llvm-project/issues/57255.

[craig.topper]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13602

Is this copied from RISC-V? That doesn't look like an AArch64 instruction name

[Allen]

update comment

[Allen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13602

yes, I updated comment, thanks @craig.topper

[hiraditya]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13599

What is the rationale behind checking Imm+1 etc?

[Allen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13599

I think it can use shl + sub to replace mul, such as
https://gcc.godbolt.org/z/szh7Eb9K8

BTW: some other target, such as RISCVTargetLowering::decomposeMulByConstant also has such logic

llvm/test/CodeGen/AArch64/mul_pow2.ll

226

will be fixed with D132325

[hiraditya]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13599

as per godbolt link, it is already happening in some cases?

[efriedma]

It's not obvious that the replacement sequences are consistently faster. At least on some cores, "add x8, x8, w0, sxtw #1" and "smull x0, w0, w8" have exactly the same throughput, so transforming from the smull to a two-instruction sequence involving the add isn't really profitable.

On a related note, many cores have optimizations for arithmetic with lsl #n, so we should prefer that over uxtw/sxtw.

llvm/test/CodeGen/AArch64/mul_pow2.ll

183

I think the suggestion misses a zero-extension. Should be able to ubfiz, though.

296

I think you can save an instruction here: instead of "-(x*4+x*2)", compute x*2-x*8.

518

This seems to be overriding our existing logic here to produce a worse result.

llvm/test/CodeGen/AArch64/sve-intrinsics-counting-elems-i32.ll

169

Probably need some logic to allow folding inch/dech, assuming there isn't some reason to avoid them.

[dmgreen]

Hi - I had been looking at mul vs add+shift recently, but had not had the time to get very far and had not got to the important part yet - exactly where and when should we be doing the transform, especially when you consider all the various cpus present.

AArch64 usually handles mul vs add/shift here: https://github.com/llvm/llvm-project/blob/6c6c4f6a9b3ef2d7db937cb78784245ea8a61418/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp#L14465
With a cost-model of sorts here: https://github.com/llvm/llvm-project/blob/6c6c4f6a9b3ef2d7db937cb78784245ea8a61418/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp#L14442

My understanding (but some of this may not be correct) is that:

• The existing cost model is conservative and could do with adjustment, but is there for a reason. It prevents the transform when there is an add/sub user (to create a madd) or if there is a zext/sext operand (to create a umull).
• A good cost model is hard to be precise about but add cost 1, shift costs 1, add+shift usually costs 2 but can be 1 for some cpus and constants. mul costs somewhere between 2 and 5, depending on the cpu, size (i32/i64) and the values in the registers. madd costs the same as mul (so the add is free). Newer cpus have a lower cost for mul, especially i64 mul.
• Replacing a mul with 2 instructions is probably good, a mul with 3 instructions is more iffy.
• I was using https://godbolt.org/z/cPTEMnP5x with different C to compare when we do the transform compared to gcc.
• The cost model is certainly worse where the operands lead into a load/store pointer operand, as the add can be folded into the addressing mode so it won't form a madd. Small shifts can sometimes be free again. I was probably planning to alter the existing profitability checks.

So whilst this may be an improvement on some targets over what we have already, a lot of the changes here either look like obvious regressions (cntd/decd/etc), are non-obvious whether they are regressions or not (madd vs add+lsl+add_lsl), or are not longer testing the point of the tests (machine-combiner-madd.ll).

[Allen]

Thanks for your information, and it seems not a easy work :)

llvm/test/CodeGen/AArch64/mul_pow2.ll

183

Yes, this has been addressed with D132325, thanks.

llvm/test/CodeGen/AArch64/sve-intrinsics-counting-elems-i32.ll

169

Thanks, I'll try this first

[hiraditya]

I agree with @eli.friedman that decomposing multiply into two separate instructions may not be profitable.
Even when latency of multiply is high, having two instructions in the pipeline adds complexity (dependencies, re-ordering, retiring etc).

Another thing to consider would be the number of ALUs and MACs(Multiply Accumulate units).
In case the ALU pressure is high MAC operations may be more efficient as there wont be stalls.

[Allen]

Add more restrict to disable these have negitate effort

[Allen]

rebase to fix new added case Transforms/SeparateConstOffsetFromGEP/AArch64/split-gep.ll

[dmgreen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13593

We can't just expect the first user of a const will be the mul we are interested in.

All the other handling of decomposing mul into add+shift is currently done in performMulCombine. Would it be better to just alter the code that is already there? It would make it easier to be more precise with the costmodel.

llvm/test/CodeGen/AArch64/mul_pow2.ll

148–151

I think that (considering the mov as free in terms of latency), in this case the madd would be worth 2-3, the lsl+add_lsl+add would cost 3-4. It would depend heavily on the exact cpu though.
For i64 muls the madd would have a higher cost (it was 4 on one cpu I tested, but newer cpus are better).

325

Is this the case you are interested in? Could we change the existing costmodel to be more precise with which sub operand it considers free?

// Conservatively do not lower to shift+add+shift if the mul might be
// folded into madd or msub.
if (N->hasOneUse() && (N->use_begin()->getOpcode() == ISD::ADD ||
                       N->use_begin()->getOpcode() == ISD::SUB))
  return SDValue();

[Allen]

refactor with costmodel

[Allen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13593

Apply your commend, and refactor it with costmodel, thanks

llvm/test/CodeGen/AArch64/mul_pow2.ll

325

Thanks for your idea, I can try this, but why the sub operand can be considered free ?

325

Done, thanks

[efriedma]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

14454

Do you want to check specifically for a constant that fits into the immediate operand of an add/sub? If it doesn't fit, it doesn't really matter if it's constant.

14500

Move computation of "ShiftedVal1" into the if statment?

llvm/test/CodeGen/AArch64/mul_pow2.ll

324

There is one other possible sequence here:

mov w8, #6
mov w9, #-1
madd w0, w0, w8, w9

This is obviously not that exciting in isolation, but it might make sense if we can hoist the "mov" instructions out of a loop.

[dmgreen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

14454

I don't think it matters it the AddSub has one use? (It might for some other optimizations, but that can be a separate patch).

It matters that for a sub the AddSub->operand(1) == N. If it is AddSub->operand(0) == N, then we cannot fold to a msub, so that shouldn't block the transform to add+shifts.
The isIntOrFPConstant check can check for isLegalAddImmediate, to be a little more precise where it will generate a mul+addri or subri.

[Allen]

llvm/test/CodeGen/AArch64/mul_pow2.ll

324

hi @efriedma, I try to generate the new sequence as you showed, but I don't know how to put a const into a register? as the operand of madd should not be const value.

For the const **AddSub->getOperand(1)**, even when I use a ISD::ADD , it still retrun a const value.

 SDValue Const = DAG.getNode(ISD::ADD, DL, VT, AddSub->getOperand(1), DAG.getConstant(0, DL, VT));

(gdb) p Const->dump()
t5: i32 = Constant<-1>
$10 = void
(gdb) p AddSub->getOperand(1)->dump()
 t5: i32 = Constant<-1>

[Allen]

address comment

[Allen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

14454

Thanks @dmgreen and @efriedma for detail suggestion

14500

Done ,thanks

llvm/test/CodeGen/AArch64/mul_pow2.ll

148–151

done, block the change as we can make use of madd

[efriedma]

llvm/test/CodeGen/AArch64/mul_pow2.ll

324

You don't have to do anything special to "put a const in a register". For example, if you write unsigned a(unsigned x) { return x * 0x33333333 + 0xF0F0F0F0; }, you get code like I suggested. Getting the same code with smaller immediates is just a matter of making sure we pick the pattern for madd, and not the pattern for sub-with-imm.

[Allen]

llvm/test/CodeGen/AArch64/mul_pow2.ll

324

I try to address the commit with D134336

[Allen]

abandoned as the accepted in D134934 and D134336

llvm/lib/Target/AArch64/AArch64ISelLowering.h

bool lowerInterleavedLoad(LoadInst *LI,
                             ArrayRef<unsigned> Indices,
                             unsigned Factor) const override;
   bool lowerInterleavedStore(StoreInst *SI, ShuffleVectorInst *SVI,
                              unsigned Factor) const override;
 
   bool isLegalAddImmediate(int64_t) const override;
   bool isLegalICmpImmediate(int64_t) const override;
 
+  bool decomposeMulByConstant(LLVMContext &Context, EVT VT,
+                              SDValue C) const override;
+
   bool isMulAddWithConstProfitable(SDValue AddNode,
                                    SDValue ConstNode) const override;
 
   bool shouldConsiderGEPOffsetSplit() const override;
 
   EVT getOptimalMemOpType(const MemOp &Op,
                           const AttributeList &FuncAttributes) const override;
 

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

bool AArch64TargetLowering::isLegalAddImmediate(int64_t Immed) const {
   Immed = std::abs(Immed);
   bool IsLegal = ((Immed >> 12) == 0 ||
                   ((Immed & 0xfff) == 0 && Immed >> 24 == 0));
   LLVM_DEBUG(dbgs() << "Is " << Immed
                     << " legal add imm: " << (IsLegal ? "yes" : "no") << "\n");
   return IsLegal;
 }
 
+// Return true if the MUL can be replaced with shift/sub/add with cheaper cost.
+bool AArch64TargetLowering::decomposeMulByConstant(LLVMContext &Context, EVT VT,
+                                                   SDValue C) const {
+  // Check integral scalar types.
+  if (!VT.isScalarInteger())
+    return false;
+  if (auto *ConstNode = dyn_cast<ConstantSDNode>(C.getNode())) {
+    if (!ConstNode->getAPIntValue().isSignedIntN(64))
+      return false;
+
+    const APInt &Imm = ConstNode->getAPIntValue();
+    if ((Imm + 1).isPowerOf2() || (Imm - 1).isPowerOf2() ||
+        (1 - Imm).isPowerOf2() || (-1 - Imm).isPowerOf2())
+      return true;
+
+    // Optimize the MUL to (SH*ADD x, (SLLI x, bits)).
+    if ((Imm - 2).isPowerOf2() || (Imm - 4).isPowerOf2() ||
+        (Imm - 8).isPowerOf2())
+      return true;
+  }
+
+  return false;
+}
+
 // Return false to prevent folding
 // (mul (add x, c1), c2) -> (add (mul x, c2), c2*c1) in DAGCombine,
 // if the folding leads to worse code.
 bool AArch64TargetLowering::isMulAddWithConstProfitable(
     SDValue AddNode, SDValue ConstNode) const {
   // Let the DAGCombiner decide for vector types and large types.
   const EVT VT = AddNode.getValueType();
   if (VT.isVector() || VT.getScalarSizeInBits() > 64)
if (N0->hasOneUse() && (isSignExtended(N0.getNode(), DAG) ||
       return SDValue();
     // Conservatively do not lower to shift+add+shift if the mul might be
     // folded into madd or msub.
     if (N->hasOneUse() && (N->use_begin()->getOpcode() == ISD::ADD ||
                            N->use_begin()->getOpcode() == ISD::SUB))
       return SDValue();
   }
   // Use ShiftedConstValue instead of ConstValue to support both shift+add/sub
   // and shift+add+shift.
   APInt ShiftedConstValue = ConstValue.ashr(TrailingZeroes);
 
   unsigned ShiftAmt;
   // Is the shifted value the LHS operand of the add/sub?
   bool ShiftValUseIsN0 = true;
   // Do we need to negate the result?
   bool NegateResult = false;
 
static SDValue performMulCombine(SDNode *N, SelectionDAG &DAG,
   }
 
   SDValue ShiftedVal = DAG.getNode(ISD::SHL, DL, VT, N0,
                                    DAG.getConstant(ShiftAmt, DL, MVT::i64));
 
   SDValue AddSubN0 = ShiftValUseIsN0 ? ShiftedVal : N0;
   SDValue AddSubN1 = ShiftValUseIsN0 ? N0 : ShiftedVal;
   SDValue Res = DAG.getNode(AddSubOpc, DL, VT, AddSubN0, AddSubN1);
   assert(!(NegateResult && TrailingZeroes) &&
          "NegateResult and TrailingZeroes cannot both be true for now.");
   // Negate the result.
   if (NegateResult)
     return DAG.getNode(ISD::SUB, DL, VT, DAG.getConstant(0, DL, VT), Res);
   // Shift the result.
   if (TrailingZeroes)
     return DAG.getNode(ISD::SHL, DL, VT, Res,
                        DAG.getConstant(TrailingZeroes, DL, MVT::i64));

llvm/test/CodeGen/AArch64/arm64-mul.ll

entry:
   %tmp2 = mul i64 %tmp1, 139968
   ret i64 %tmp2
 }
 
 ; Check 64-bit multiplication is used for constants > 32 bits.
 define i64 @t10(i32 %a) nounwind {
 ; CHECK-LABEL: t10:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    mov w8, #2
 ; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
-; CHECK-NEXT:    sxtw x9, w0
-; CHECK-NEXT:    movk w8, #32768, lsl #16
-; CHECK-NEXT:    mul x0, x9, x8
+; CHECK-NEXT:    sbfiz x8, x0, #31, #32
+; CHECK-NEXT:    add x0, x8, w0, sxtw #1
 ; CHECK-NEXT:    ret
 entry:
   %tmp1 = sext i32 %a to i64
   %tmp2 = mul i64 %tmp1, 2147483650 ; = 2^31 + 2
   ret i64 %tmp2
 }
 
 ; Check the sext_inreg case.

llvm/test/CodeGen/AArch64/arm64_32.ll

true:
   ret void
 
 false:
   ret void
 }
 
 define { [18 x i8] }* @test_gep_nonpow2({ [18 x i8] }* %a0, i32 %a1) {
 ; CHECK-LABEL: test_gep_nonpow2:
-; CHECK-OPT:      mov w[[SIZE:[0-9]+]], #18
-; CHECK-OPT-NEXT: smaddl x0, w1, w[[SIZE]], x0
+; CHECK-OPT:      sbfiz   x8, x1, #4, #32
+; CHECK-OPT-NEXT: add     x8, x8, w1, sxtw #1
+; CHECK-OPT-NEXT: add     x0, x0, x8
 ; CHECK-OPT-NEXT: ret
 
-; CHECK-FAST:      mov w[[SIZE:[0-9]+]], #18
-; CHECK-FAST-NEXT: smaddl [[TMP:x[0-9]+]], w1, w[[SIZE]], x0
-; CHECK-FAST-NEXT: and x0, [[TMP]], #0xffffffff
+; CHECK-FAST:      sbfiz   x8, x1, #4, #32
+; CHECK-FAST-NEXT: add     x8, x8, w1, sxtw #1
+; CHECK-FAST-NEXT: add     x8, x0, x8
+; CHECK-FAST-NEXT: and     x0, x8, #0xffffffff
 ; CHECK-FAST-NEXT: ret
   %tmp0 = getelementptr inbounds { [18 x i8] }, { [18 x i8] }* %a0, i32 %a1
   ret { [18 x i8] }* %tmp0
 }
 
 define void @test_memset(i64 %in, i8 %value)  {
 ; CHECK-LABEL: test_memset:
 ; CHECK-DAG: and x8, x0, #0xffffffff

llvm/test/CodeGen/AArch64/machine-combiner-madd.ll

 ; Test all AArch64 subarches with scheduling models.
 ; RUN: llc -mtriple=aarch64-linux-gnu -mcpu=a64fx      < %s | FileCheck %s
 ; RUN: llc -mtriple=aarch64-linux-gnu -mcpu=cortex-a57 < %s | FileCheck %s
 ; RUN: llc -mtriple=aarch64-linux-gnu -mcpu=cortex-a72 < %s | FileCheck %s
 ; RUN: llc -mtriple=aarch64-linux-gnu -mcpu=cortex-a73 < %s | FileCheck %s
 ; RUN: llc -mtriple=aarch64-linux-gnu -mcpu=cyclone    < %s | FileCheck %s
 ; RUN: llc -mtriple=aarch64-linux-gnu -mcpu=exynos-m3  < %s | FileCheck %s
 ; RUN: llc -mtriple=aarch64-linux-gnu -mcpu=kryo       < %s | FileCheck %s
 ; RUN: llc -mtriple=aarch64-linux-gnu -mcpu=thunderx2t99 < %s | FileCheck %s
 ; RUN: llc -mtriple=aarch64-linux-gnu -mcpu=thunderx3t110 < %s | FileCheck %s
 ; RUN: llc -mtriple=aarch64-linux-gnu -mcpu=tsv110 < %s | FileCheck %s
 
 ; Make sure that inst-combine fuses the multiply add in the addressing mode of
 ; the load.
 
-; CHECK-LABEL: fun:
-; CHECK-NOT: mul
-; CHECK:     madd
-; CHECK-NOT: mul
 
 %class.D = type { %class.basic_string.base, [4 x i8] }
 %class.basic_string.base = type <{ i64, i64, i32 }>
 @a = global %class.D* zeroinitializer, align 8
 declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1)
 define internal void @fun() section ".text.startup" {
+; CHECK-LABEL: fun:
+; CHECK-NOT: mul
+; CHECK-NOT: madd
+; CHECK-COUNT-2: lsl
 entry:
   %tmp.i.i = alloca %class.D, align 8
   %y = bitcast %class.D* %tmp.i.i to i8*
   br label %loop
 loop:
   %conv11.i.i = phi i64 [ 0, %entry ], [ %inc.i.i, %loop ]
   %i = phi i64 [ undef, %entry ], [ %inc.i.i, %loop ]
   %x = load %class.D*, %class.D** getelementptr inbounds (%class.D*, %class.D** @a, i64 0), align 8

llvm/test/CodeGen/AArch64/mul_pow2.ll

; GISEL-NEXT:    ret
 
   %mul = mul nsw i32 %x, 5
   ret i32 %mul
 }
 
 define i32 @test6_32b(i32 %x) {
 ; CHECK-LABEL: test6_32b:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    add w8, w0, w0, lsl #1
-; CHECK-NEXT:    lsl w0, w8, #1
+; CHECK-NEXT:    lsl w8, w0, #2
+; CHECK-NEXT:    add w0, w8, w0, lsl #1
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test6_32b:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    add w8, w0, w0, lsl #1
 ; GISEL-NEXT:    lsl w0, w8, #1
 ; GISEL-NEXT:    ret
 
   %mul = mul nsw i32 %x, 6
   ret i32 %mul
 }
 
 define i64 @test6_64b(i64 %x) {
 ; CHECK-LABEL: test6_64b:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    add x8, x0, x0, lsl #1
-; CHECK-NEXT:    lsl x0, x8, #1
+; CHECK-NEXT:    lsl x8, x0, #2
+; CHECK-NEXT:    add x0, x8, x0, lsl #1
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test6_64b:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    add x8, x0, x0, lsl #1
 ; GISEL-NEXT:    lsl x0, x8, #1
 ; GISEL-NEXT:    ret
 
   %mul = mul nsw i64 %x, 6
   ret i64 %mul
 }
 
 ; mul that appears together with add, sub, s(z)ext is not supported to be
 ; converted to the combination of lsl, add/sub yet.
 define i64 @test6_umull(i32 %x) {
 ; CHECK-LABEL: test6_umull:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #6
-; CHECK-NEXT:    umull x0, w0, w8
+; CHECK-NEXT:    mov w8, w0
+; CHECK-NEXT:    lsl x8, x8, #2
+; CHECK-NEXT:    add x0, x8, w0, uxtw #1
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test6_umull:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    mov w8, #6
 ; GISEL-NEXT:    umull x0, w0, w8
 ; GISEL-NEXT:    ret
 
   %ext = zext i32 %x to i64
   %mul = mul nsw i64 %ext, 6
   ret i64 %mul
 }
 
 define i64 @test6_smull(i32 %x) {
 ; CHECK-LABEL: test6_smull:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #6
-; CHECK-NEXT:    smull x0, w0, w8
+; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
+; CHECK-NEXT:    sbfiz x8, x0, #2, #32
+; CHECK-NEXT:    add x0, x8, w0, sxtw #1
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test6_smull:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    mov w8, #6
 ; GISEL-NEXT:    smull x0, w0, w8
 ; GISEL-NEXT:    ret
 
   %ext = sext i32 %x to i64
   %mul = mul nsw i64 %ext, 6
   ret i64 %mul
 }
 
 define i32 @test6_madd(i32 %x, i32 %y) {
 ; CHECK-LABEL: test6_madd:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #6
-; CHECK-NEXT:    madd w0, w0, w8, w1
+; CHECK-NEXT:    lsl w8, w0, #2
+; CHECK-NEXT:    add w8, w8, w0, lsl #1
+; CHECK-NEXT:    add w0, w8, w1
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test6_madd:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    mov w8, #6
 ; GISEL-NEXT:    madd w0, w0, w8, w1
 ; GISEL-NEXT:    ret
 
   %mul = mul nsw i32 %x, 6
   %add = add i32 %mul, %y
   ret i32 %add
 }
 
 define i32 @test6_msub(i32 %x, i32 %y) {
 ; CHECK-LABEL: test6_msub:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #6
-; CHECK-NEXT:    msub w0, w0, w8, w1
+; CHECK-NEXT:    lsl w8, w0, #2
+; CHECK-NEXT:    add w8, w8, w0, lsl #1
+; CHECK-NEXT:    sub w0, w1, w8
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test6_msub:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    mov w8, #6
 ; GISEL-NEXT:    msub w0, w0, w8, w1
 ; GISEL-NEXT:    ret
 
   %mul = mul nsw i32 %x, 6
   %sub = sub i32 %y, %mul
   ret i32 %sub
 }
 
+; TODO: mov w8, w0 + lsl x8, x8, #2 should combine into lsl x8, x0, #2
 define i64 @test6_umaddl(i32 %x, i64 %y) {
 ; CHECK-LABEL: test6_umaddl:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #6
-; CHECK-NEXT:    umaddl x0, w0, w8, x1
+; CHECK-NEXT:    mov w8, w0
+; CHECK-NEXT:    lsl x8, x8, #2
+; CHECK-NEXT:    add x8, x8, w0, uxtw #1
+; CHECK-NEXT:    add x0, x8, x1
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test6_umaddl:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    mov w8, #6
 ; GISEL-NEXT:    umaddl x0, w0, w8, x1
 ; GISEL-NEXT:    ret
 
   %ext = zext i32 %x to i64
   %mul = mul nsw i64 %ext, 6
   %add = add i64 %mul, %y
   ret i64 %add
 }
 
 define i64 @test6_smaddl(i32 %x, i64 %y) {
 ; CHECK-LABEL: test6_smaddl:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #6
-; CHECK-NEXT:    smaddl x0, w0, w8, x1
+; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
+; CHECK-NEXT:    sbfiz x8, x0, #2, #32
+; CHECK-NEXT:    add x8, x8, w0, sxtw #1
+; CHECK-NEXT:    add x0, x8, x1
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test6_smaddl:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    mov w8, #6
 ; GISEL-NEXT:    smaddl x0, w0, w8, x1
 ; GISEL-NEXT:    ret
 
   %ext = sext i32 %x to i64
   %mul = mul nsw i64 %ext, 6
   %add = add i64 %mul, %y
   ret i64 %add
 }
 
+; TODO: mov w8, w0 + lsl x8, x8, #2 should combine into lsl x8, x0, #2
 define i64 @test6_umsubl(i32 %x, i64 %y) {
 ; CHECK-LABEL: test6_umsubl:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #6
-; CHECK-NEXT:    umsubl x0, w0, w8, x1
+; CHECK-NEXT:    mov w8, w0
+; CHECK-NEXT:    lsl x8, x8, #2
+; CHECK-NEXT:    add x8, x8, w0, uxtw #1
+; CHECK-NEXT:    sub x0, x1, x8
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test6_umsubl:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    mov w8, #6
 ; GISEL-NEXT:    umsubl x0, w0, w8, x1
 ; GISEL-NEXT:    ret
 
   %ext = zext i32 %x to i64
   %mul = mul nsw i64 %ext, 6
   %sub = sub i64 %y, %mul
   ret i64 %sub
 }
 
 define i64 @test6_smsubl(i32 %x, i64 %y) {
 ; CHECK-LABEL: test6_smsubl:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #6
-; CHECK-NEXT:    smsubl x0, w0, w8, x1
+; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
+; CHECK-NEXT:    sbfiz x8, x0, #2, #32
+; CHECK-NEXT:    add x8, x8, w0, sxtw #1
+; CHECK-NEXT:    sub x0, x1, x8
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test6_smsubl:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    mov w8, #6
 ; GISEL-NEXT:    smsubl x0, w0, w8, x1
 ; GISEL-NEXT:    ret
 
   %ext = sext i32 %x to i64
   %mul = mul nsw i64 %ext, 6
   %sub = sub i64 %y, %mul
   ret i64 %sub
 }
 
 define i64 @test6_umnegl(i32 %x) {
 ; CHECK-LABEL: test6_umnegl:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #6
-; CHECK-NEXT:    umnegl x0, w0, w8
+; CHECK-NEXT:    mov w8, w0
+; CHECK-NEXT:    lsl x8, x8, #2
+; CHECK-NEXT:    add x8, x8, w0, uxtw #1
+; CHECK-NEXT:    neg x0, x8
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test6_umnegl:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    mov w8, #6
 ; GISEL-NEXT:    umnegl x0, w0, w8
 ; GISEL-NEXT:    ret
 
   %ext = zext i32 %x to i64
   %mul = mul nsw i64 %ext, 6
   %sub = sub i64 0, %mul
   ret i64 %sub
 }
 
 define i64 @test6_smnegl(i32 %x) {
 ; CHECK-LABEL: test6_smnegl:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #6
-; CHECK-NEXT:    smnegl x0, w0, w8
+; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
+; CHECK-NEXT:    sbfiz x8, x0, #2, #32
+; CHECK-NEXT:    add x8, x8, w0, sxtw #1
+; CHECK-NEXT:    neg x0, x8
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test6_smnegl:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    mov w8, #6
 ; GISEL-NEXT:    smnegl x0, w0, w8
 ; GISEL-NEXT:    ret
 
   %ext = sext i32 %x to i64
   %mul = mul nsw i64 %ext, 6
   %sub = sub i64 0, %mul
   ret i64 %sub
 }
 
+define i32 @mull6_sub(i32 %x) {
+; CHECK-LABEL: mull6_sub:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    lsl w8, w0, #2
+; CHECK-NEXT:    add w8, w8, w0, lsl #1
+; CHECK-NEXT:    sub w0, w8, #1
+; CHECK-NEXT:    ret
+;
+; GISEL-LABEL: mull6_sub:
+; GISEL:       // %bb.0:
+; GISEL-NEXT:    mov w8, #6
+; GISEL-NEXT:    mul w8, w0, w8
+; GISEL-NEXT:    sub w0, w8, #1
+; GISEL-NEXT:    ret
+  %mul = mul nsw i32 %x, 6
+  %sub = add nsw i32 %mul, -1
+  ret i32 %sub
+}
+
 define i32 @test7(i32 %x) {
 ; CHECK-LABEL: test7:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    lsl w8, w0, #3
 ; CHECK-NEXT:    sub w0, w8, w0
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test7:
; GISEL-NEXT:    ret
 
   %mul = mul nsw i32 %x, 9
   ret i32 %mul
 }
 
 define i32 @test10(i32 %x) {
 ; CHECK-LABEL: test10:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    add w8, w0, w0, lsl #2
-; CHECK-NEXT:    lsl w0, w8, #1
+; CHECK-NEXT:    lsl w8, w0, #3
+; CHECK-NEXT:    add w0, w8, w0, lsl #1
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test10:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    add w8, w0, w0, lsl #2
 ; GISEL-NEXT:    lsl w0, w8, #1
 ; GISEL-NEXT:    ret
 
; GISEL-NEXT:    ret
 
   %mul = mul nsw i32 %x, 11
   ret i32 %mul
 }
 
 define i32 @test12(i32 %x) {
 ; CHECK-LABEL: test12:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    add w8, w0, w0, lsl #1
-; CHECK-NEXT:    lsl w0, w8, #2
+; CHECK-NEXT:    lsl w8, w0, #3
+; CHECK-NEXT:    add w0, w8, w0, lsl #2
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test12:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    add w8, w0, w0, lsl #1
 ; GISEL-NEXT:    lsl w0, w8, #2
 ; GISEL-NEXT:    ret
 
; GISEL-NEXT:    ret
 
   %mul = mul nsw i32 %x, -2
   ret i32 %mul
 }
 
 define i32 @ntest3(i32 %x) {
 ; CHECK-LABEL: ntest3:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    sub w0, w0, w0, lsl #2
+; CHECK-NEXT:    add w8, w0, w0, lsl #1
+; CHECK-NEXT:    neg w0, w8
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: ntest3:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    sub w0, w0, w0, lsl #2
 ; GISEL-NEXT:    ret
 
   %mul = mul nsw i32 %x, -3

 ; CHECK-NEXT:    movi v2.4s, #1, msl #16
 ; CHECK-NEXT:    shl v0.4s, v0.4s, #6
 ; CHECK-NEXT:    sub v0.4s, v1.4s, v0.4s
 ; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: muladd_demand_commute:
 ; GISEL:       // %bb.0:
-; GISEL-NEXT:    adrp x8, .LCPI42_1
-; GISEL-NEXT:    ldr q2, [x8, :lo12:.LCPI42_1]
-; GISEL-NEXT:    adrp x8, .LCPI42_0
+; GISEL-NEXT:    adrp x8, .LCPI43_1
+; GISEL-NEXT:    ldr q2, [x8, :lo12:.LCPI43_1]
+; GISEL-NEXT:    adrp x8, .LCPI43_0
 ; GISEL-NEXT:    mla v1.4s, v0.4s, v2.4s
-; GISEL-NEXT:    ldr q0, [x8, :lo12:.LCPI42_0]
+; GISEL-NEXT:    ldr q0, [x8, :lo12:.LCPI43_0]
 ; GISEL-NEXT:    and v0.16b, v1.16b, v0.16b
 ; GISEL-NEXT:    ret
   %m = mul <4 x i32> %x, <i32 131008, i32 131008, i32 131008, i32 131008>
   %a = add <4 x i32> %m, %y
   %r = and <4 x i32> %a, <i32 131071, i32 131071, i32 131071, i32 131071>
   ret <4 x i32> %r
 }

llvm/test/CodeGen/AArch64/srem-seteq-illegal-types.ll

; CHECK-NEXT:    ret
   %srem = srem i29 %X, 99
   %cmp = icmp eq i29 %srem, 0
   ret i1 %cmp
 }
 
 define i1 @test_srem_even(i4 %X) nounwind {
 ; CHECK-LABEL: test_srem_even:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    sbfx w9, w0, #0, #4
-; CHECK-NEXT:    mov w8, #6
-; CHECK-NEXT:    add w9, w9, w9, lsl #1
-; CHECK-NEXT:    ubfx w10, w9, #7, #1
-; CHECK-NEXT:    add w9, w10, w9, lsr #4
-; CHECK-NEXT:    msub w8, w9, w8, w0
+; CHECK-NEXT:    sbfx w8, w0, #0, #4
+; CHECK-NEXT:    add w8, w8, w8, lsl #1
+; CHECK-NEXT:    ubfx w9, w8, #7, #1
+; CHECK-NEXT:    add w8, w9, w8, lsr #4
+; CHECK-NEXT:    lsl w9, w8, #2
+; CHECK-NEXT:    add w8, w9, w8, lsl #1
+; CHECK-NEXT:    sub w8, w0, w8
 ; CHECK-NEXT:    and w8, w8, #0xf
 ; CHECK-NEXT:    cmp w8, #1
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %srem = srem i4 %X, 6
   %cmp = icmp eq i4 %srem, 1
   ret i1 %cmp
 }

llvm/test/CodeGen/AArch64/sve-intrinsics-counting-elems-i32.ll

; CHECK-NEXT:    ret
   %conv = trunc i64 %cnt to i32
   %out = add i32 %conv, %a
   ret i32 %out
 }
 
 define i32 @inch_mul(i32 %a) {
 ; NO_SCALAR_INC-LABEL: inch_mul:
 ; NO_SCALAR_INC:       // %bb.0:
-; NO_SCALAR_INC-NEXT:    mov w8, #5
-; NO_SCALAR_INC-NEXT:    cnth x9, vl8
-; NO_SCALAR_INC-NEXT:    madd w0, w9, w8, w0
+; NO_SCALAR_INC-NEXT:    cnth x8, vl8
+; NO_SCALAR_INC-NEXT:    add w8, w8, w8, lsl #2
+; NO_SCALAR_INC-NEXT:    add w0, w8, w0
 ; NO_SCALAR_INC-NEXT:    ret
 ;
 ; CHECK-LABEL: inch_mul:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
-; CHECK-NEXT:    inch x0, vl8, mul #5
-; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    cnth x8, vl8
+; CHECK-NEXT:    // kill: def $w8 killed $w8 killed $x8 def $x8
+; CHECK-NEXT:    inch x8, vl8, mul #4
+; CHECK-NEXT:    add w0, w8, w0
 ; CHECK-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cnth(i32 8)
   %conv = trunc i64 %cnt to i32
   %mul = mul i32 %conv, 5
   %out = add i32 %mul, %a
   ret i32 %out
 }
 
; CHECK-NEXT:    ret
   %conv = trunc i64 %cnt to i32
   %out = sub i32 %a, %conv
   ret i32 %out
 }
 
 define i32 @dech_mul(i32 %a) {
 ; NO_SCALAR_INC-LABEL: dech_mul:
 ; NO_SCALAR_INC:       // %bb.0:
-; NO_SCALAR_INC-NEXT:    mov w8, #7
-; NO_SCALAR_INC-NEXT:    cnth x9, vl16
-; NO_SCALAR_INC-NEXT:    msub w0, w9, w8, w0
+; NO_SCALAR_INC-NEXT:    cnth x8, vl16
+; NO_SCALAR_INC-NEXT:    sub w8, w8, w8, lsl #3
+; NO_SCALAR_INC-NEXT:    add w0, w0, w8
 ; NO_SCALAR_INC-NEXT:    ret
 ;
 ; CHECK-LABEL: dech_mul:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
-; CHECK-NEXT:    dech x0, vl16, mul #7
-; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    cnth x8, vl16
+; CHECK-NEXT:    // kill: def $w8 killed $w8 killed $x8 def $x8
+; CHECK-NEXT:    dech x8, vl16, mul #8
+; CHECK-NEXT:    add w0, w0, w8
 ; CHECK-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cnth(i32 9)
   %conv = trunc i64 %cnt to i32
   %mul = mul i32 %conv, 7
   %out = sub i32 %a, %mul
   ret i32 %out
 }
 
; CHECK-NEXT:    ret
   %conv = trunc i64 %cnt to i32
   %out = add i32 %conv, %a
   ret i32 %out
 }
 
 define i32 @incw_mul(i32 %a) {
 ; NO_SCALAR_INC-LABEL: incw_mul:
 ; NO_SCALAR_INC:       // %bb.0:
-; NO_SCALAR_INC-NEXT:    mov w8, #12
-; NO_SCALAR_INC-NEXT:    cntw x9, vl32
-; NO_SCALAR_INC-NEXT:    madd w0, w9, w8, w0
+; NO_SCALAR_INC-NEXT:    cntw x8, vl32
+; NO_SCALAR_INC-NEXT:    lsl w9, w8, #3
+; NO_SCALAR_INC-NEXT:    add w8, w9, w8, lsl #2
+; NO_SCALAR_INC-NEXT:    add w0, w8, w0
 ; NO_SCALAR_INC-NEXT:    ret
 ;
 ; CHECK-LABEL: incw_mul:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
-; CHECK-NEXT:    incw x0, vl32, mul #12
-; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    cntw x8, vl32
+; CHECK-NEXT:    lsl w8, w8, #3
+; CHECK-NEXT:    incw x8, vl32, mul #4
+; CHECK-NEXT:    add w0, w8, w0
 ; CHECK-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cntw(i32 10)
   %conv = trunc i64 %cnt to i32
   %mul = mul i32 %conv, 12
   %out = add i32 %mul, %a
   ret i32 %out
 }
 
; CHECK-NEXT:    ret
   %conv = trunc i64 %cnt to i32
   %add = add i32 %base, %conv
   ret i32 %add
 }
 
 define i32 @incd_mul(i32 %base) {
 ; NO_SCALAR_INC-LABEL: incd_mul:
 ; NO_SCALAR_INC:       // %bb.0:
-; NO_SCALAR_INC-NEXT:    mov w8, #15
-; NO_SCALAR_INC-NEXT:    cntd x9, vl64
-; NO_SCALAR_INC-NEXT:    madd w0, w9, w8, w0
+; NO_SCALAR_INC-NEXT:    cntd x8, vl64
+; NO_SCALAR_INC-NEXT:    lsl w9, w8, #4
+; NO_SCALAR_INC-NEXT:    sub w8, w9, w8
+; NO_SCALAR_INC-NEXT:    add w0, w0, w8
 ; NO_SCALAR_INC-NEXT:    ret
 ;
 ; CHECK-LABEL: incd_mul:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
-; CHECK-NEXT:    incd x0, vl64, mul #15
-; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    cntd x8, vl64
+; CHECK-NEXT:    lsl w8, w8, #4
+; CHECK-NEXT:    decd x8, vl64
+; CHECK-NEXT:    add w0, w0, w8
 ; CHECK-NEXT:    ret
   %cnt = tail call i64 @llvm.aarch64.sve.cntd(i32 11)
   %conv = trunc i64 %cnt to i32
   %mul = mul i32 %conv, 15
   %add = add i32 %base, %mul
   ret i32 %add
 }
 
; CHECK-NEXT:    ret
   %conv = trunc i64 %cnt to i32
   %out = sub i32 %a, %conv
   ret i32 %out
 }
 
 define i32 @decd_mul(i32 %a) {
 ; NO_SCALAR_INC-LABEL: decd_mul:
 ; NO_SCALAR_INC:       // %bb.0:
-; NO_SCALAR_INC-NEXT:    mov w8, #9
-; NO_SCALAR_INC-NEXT:    cntd x9, vl2
-; NO_SCALAR_INC-NEXT:    msub w0, w9, w8, w0
+; NO_SCALAR_INC-NEXT:    cntd x8, vl2
+; NO_SCALAR_INC-NEXT:    add w8, w8, w8, lsl #3
+; NO_SCALAR_INC-NEXT:    sub w0, w0, w8
 ; NO_SCALAR_INC-NEXT:    ret
 ;
 ; CHECK-LABEL: decd_mul:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
-; CHECK-NEXT:    decd x0, vl2, mul #9
-; CHECK-NEXT:    // kill: def $w0 killed $w0 killed $x0
+; CHECK-NEXT:    cntd x8, vl2
+; CHECK-NEXT:    // kill: def $w8 killed $w8 killed $x8 def $x8
+; CHECK-NEXT:    incd x8, vl2, mul #8
+; CHECK-NEXT:    sub w0, w0, w8
 ; CHECK-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cntd(i32 2)
   %conv = trunc i64 %cnt to i32
   %mul = mul i32 %conv, 9
   %out = sub i32 %a, %mul
   ret i32 %out
 }
 
 declare i64 @llvm.aarch64.sve.cntb(i32 %pattern)
 declare i64 @llvm.aarch64.sve.cnth(i32 %pattern)
 declare i64 @llvm.aarch64.sve.cntw(i32 %pattern)
 declare i64 @llvm.aarch64.sve.cntd(i32 %pattern)

llvm/test/CodeGen/AArch64/sve-intrinsics-counting-elems.ll

 ; USE_SCALAR_INC-NEXT:    ret
   %out = call i64 @llvm.aarch64.sve.cntb(i32 2)
   ret i64 %out
 }
 
 define i64 @cntb_mul3() {
 ; CHECK-LABEL: cntb_mul3:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cntb x0, vl6, mul #3
+; CHECK-NEXT:    cntb x8, vl6
+; CHECK-NEXT:    add x0, x8, x8, lsl #1
 ; CHECK-NEXT:    ret
 ;
 ; USE_SCALAR_INC-LABEL: cntb_mul3:
 ; USE_SCALAR_INC:       // %bb.0:
-; USE_SCALAR_INC-NEXT:    cntb x0, vl6, mul #3
+; USE_SCALAR_INC-NEXT:    cntb x0, vl6
+; USE_SCALAR_INC-NEXT:    incb x0, vl6, mul #2
 ; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cntb(i32 6)
   %out = mul i64 %cnt, 3
   ret i64 %out
 }
 
 define i64 @cntb_mul4() {
 ; CHECK-LABEL: cntb_mul4:

 ; USE_SCALAR_INC-NEXT:    ret
   %out = call i64 @llvm.aarch64.sve.cnth(i32 3)
   ret i64 %out
 }
 
 define i64 @cnth_mul5() {
 ; CHECK-LABEL: cnth_mul5:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cnth x0, vl7, mul #5
+; CHECK-NEXT:    cnth x8, vl7
+; CHECK-NEXT:    add x0, x8, x8, lsl #2
 ; CHECK-NEXT:    ret
 ;
 ; USE_SCALAR_INC-LABEL: cnth_mul5:
 ; USE_SCALAR_INC:       // %bb.0:
-; USE_SCALAR_INC-NEXT:    cnth x0, vl7, mul #5
+; USE_SCALAR_INC-NEXT:    cnth x0, vl7
+; USE_SCALAR_INC-NEXT:    inch x0, vl7, mul #4
 ; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cnth(i32 7)
   %out = mul i64 %cnt, 5
   ret i64 %out
 }
 
 define i64 @cnth_mul8() {
 ; CHECK-LABEL: cnth_mul8:

 ; USE_SCALAR_INC-NEXT:    ret
   %out = call i64 @llvm.aarch64.sve.cntd(i32 5)
   ret i64 %out
 }
 
 define i64 @cntd_mul15() {
 ; CHECK-LABEL: cntd_mul15:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cntd x0, vl16, mul #15
+; CHECK-NEXT:    cntd x8, vl16
+; CHECK-NEXT:    cntd x9, vl16, mul #16
+; CHECK-NEXT:    sub x0, x9, x8
 ; CHECK-NEXT:    ret
 ;
 ; USE_SCALAR_INC-LABEL: cntd_mul15:
 ; USE_SCALAR_INC:       // %bb.0:
-; USE_SCALAR_INC-NEXT:    cntd x0, vl16, mul #15
+; USE_SCALAR_INC-NEXT:    cntd x0, vl16, mul #16
+; USE_SCALAR_INC-NEXT:    decd x0, vl16
 ; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cntd(i32 9)
   %out = mul i64 %cnt, 15
   ret i64 %out
 }
 
 define i64 @cntd_mul16() {
 ; CHECK-LABEL: cntd_mul16:
; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cnth(i32 4)
   %out = add i64 %cnt, %a
   ret i64 %out
 }
 
 define i64 @inch_mul(i64 %a) {
 ; CHECK-LABEL: inch_mul:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cnth x8, vl8, mul #5
+; CHECK-NEXT:    cnth x8, vl8
+; CHECK-NEXT:    add x8, x8, x8, lsl #2
 ; CHECK-NEXT:    add x0, x8, x0
 ; CHECK-NEXT:    ret
 ;
 ; USE_SCALAR_INC-LABEL: inch_mul:
 ; USE_SCALAR_INC:       // %bb.0:
-; USE_SCALAR_INC-NEXT:    inch x0, vl8, mul #5
+; USE_SCALAR_INC-NEXT:    cnth x8, vl8
+; USE_SCALAR_INC-NEXT:    inch x8, vl8, mul #4
+; USE_SCALAR_INC-NEXT:    add x0, x8, x0
 ; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cnth(i32 8)
   %mul = mul i64 %cnt, 5
   %out = add i64 %mul, %a
   ret i64 %out
 }
 
 ;
; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cnth(i32 1)
   %out = sub i64 %a, %cnt
   ret i64 %out
 }
 
 define i64 @dech_mul(i64 %a) {
 ; CHECK-LABEL: dech_mul:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cnth x8, vl16, mul #7
-; CHECK-NEXT:    sub x0, x0, x8
+; CHECK-NEXT:    cnth x8, vl16
+; CHECK-NEXT:    sub x8, x8, x8, lsl #3
+; CHECK-NEXT:    add x0, x0, x8
 ; CHECK-NEXT:    ret
 ;
 ; USE_SCALAR_INC-LABEL: dech_mul:
 ; USE_SCALAR_INC:       // %bb.0:
-; USE_SCALAR_INC-NEXT:    dech x0, vl16, mul #7
+; USE_SCALAR_INC-NEXT:    cnth x8, vl16
+; USE_SCALAR_INC-NEXT:    dech x8, vl16, mul #8
+; USE_SCALAR_INC-NEXT:    add x0, x0, x8
 ; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cnth(i32 9)
   %mul = mul i64 %cnt, 7
   %out = sub i64 %a, %mul
   ret i64 %out
 }
 
 ;
; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cntw(i32 16)
   %out = add i64 %cnt, %a
   ret i64 %out
 }
 
 define i64 @incw_mul(i64 %a) {
 ; CHECK-LABEL: incw_mul:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cntw x8, vl32, mul #12
+; CHECK-NEXT:    cntw x8, vl32
+; CHECK-NEXT:    cntw x9, vl32, mul #8
+; CHECK-NEXT:    add x8, x9, x8, lsl #2
 ; CHECK-NEXT:    add x0, x8, x0
 ; CHECK-NEXT:    ret
 ;
 ; USE_SCALAR_INC-LABEL: incw_mul:
 ; USE_SCALAR_INC:       // %bb.0:
-; USE_SCALAR_INC-NEXT:    incw x0, vl32, mul #12
+; USE_SCALAR_INC-NEXT:    cntw x8, vl32, mul #8
+; USE_SCALAR_INC-NEXT:    incw x8, vl32, mul #4
+; USE_SCALAR_INC-NEXT:    add x0, x8, x0
 ; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cntw(i32 10)
   %mul = mul i64 %cnt, 12
   %out = add i64 %mul, %a
   ret i64 %out
 }
 
 ;
; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cntd(i32 8)
   %out = add i64 %cnt, %a
   ret i64 %out
 }
 
 define i64 @incd_mul(i64 %a) {
 ; CHECK-LABEL: incd_mul:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cntd x8, all, mul #15
+; CHECK-NEXT:    cntd x8
+; CHECK-NEXT:    cntd x9, all, mul #16
+; CHECK-NEXT:    sub x8, x9, x8
 ; CHECK-NEXT:    add x0, x8, x0
 ; CHECK-NEXT:    ret
 ;
 ; USE_SCALAR_INC-LABEL: incd_mul:
 ; USE_SCALAR_INC:       // %bb.0:
-; USE_SCALAR_INC-NEXT:    incd x0, all, mul #15
+; USE_SCALAR_INC-NEXT:    cntd x8, all, mul #16
+; USE_SCALAR_INC-NEXT:    decd x8
+; USE_SCALAR_INC-NEXT:    add x0, x8, x0
 ; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cntd(i32 31)
   %mul = mul i64 %cnt, 15
   %out = add i64 %mul, %a
   ret i64 %out
 }
 
 ;
; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cntd(i32 16)
   %out = sub i64 %a, %cnt
   ret i64 %out
 }
 
 define i64 @decd_mul(i64 %a) {
 ; CHECK-LABEL: decd_mul:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cntd x8, vl2, mul #9
+; CHECK-NEXT:    cntd x8, vl2
+; CHECK-NEXT:    add x8, x8, x8, lsl #3
 ; CHECK-NEXT:    sub x0, x0, x8
 ; CHECK-NEXT:    ret
 ;
 ; USE_SCALAR_INC-LABEL: decd_mul:
 ; USE_SCALAR_INC:       // %bb.0:
-; USE_SCALAR_INC-NEXT:    decd x0, vl2, mul #9
+; USE_SCALAR_INC-NEXT:    cntd x8, vl2
+; USE_SCALAR_INC-NEXT:    incd x8, vl2, mul #8
+; USE_SCALAR_INC-NEXT:    sub x0, x0, x8
 ; USE_SCALAR_INC-NEXT:    ret
   %cnt = call i64 @llvm.aarch64.sve.cntd(i32 2)
   %mul = mul i64 %cnt, 9
   %out = sub i64 %a, %mul
   ret i64 %out
 }
 
 declare i64 @llvm.aarch64.sve.cntb(i32 %pattern)
 declare i64 @llvm.aarch64.sve.cnth(i32 %pattern)
 declare i64 @llvm.aarch64.sve.cntw(i32 %pattern)
 declare i64 @llvm.aarch64.sve.cntd(i32 %pattern)
 
 declare i64 @llvm.aarch64.sve.cntp.nxv16i1(<vscale x 16 x i1>, <vscale x 16 x i1>)
 declare i64 @llvm.aarch64.sve.cntp.nxv8i1(<vscale x 8 x i1>, <vscale x 8 x i1>)
 declare i64 @llvm.aarch64.sve.cntp.nxv4i1(<vscale x 4 x i1>, <vscale x 4 x i1>)
 declare i64 @llvm.aarch64.sve.cntp.nxv2i1(<vscale x 2 x i1>, <vscale x 2 x i1>)

llvm/test/CodeGen/AArch64/typepromotion-phisret.ll

 
 exit:                                             ; preds = %if.end
   ret i16 %unrelated
 }
 
 define i16 @promote_arg_return(i16 zeroext %arg1, i16 zeroext %arg2, i8* %res) {
 ; CHECK-LABEL: promote_arg_return:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    add w8, w0, w0, lsl #1
-; CHECK-NEXT:    add w8, w8, #45
+; CHECK-NEXT:    add w8, w0, #15
+; CHECK-NEXT:    add w8, w8, w8, lsl #1
 ; CHECK-NEXT:    cmp w8, w1
 ; CHECK-NEXT:    cset w8, lo
 ; CHECK-NEXT:    strb w8, [x2]
 ; CHECK-NEXT:    ret
   %add = add nuw i16 %arg1, 15
   %mul = mul nuw nsw i16 %add, 3
   %cmp = icmp ult i16 %mul, %arg2
   %conv = zext i1 %cmp to i8

llvm/test/CodeGen/AArch64/urem-seteq-illegal-types.ll

; CHECK-NEXT:    ret
   %urem = urem i27 %X, 14
   %cmp = icmp eq i27 %urem, 0
   ret i1 %cmp
 }
 
 define i1 @test_urem_odd_setne(i4 %X) nounwind {
 ; CHECK-LABEL: test_urem_odd_setne:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #13
-; CHECK-NEXT:    mul w8, w0, w8
+; CHECK-NEXT:    add w8, w0, w0, lsl #1
+; CHECK-NEXT:    neg w8, w8
 ; CHECK-NEXT:    and w8, w8, #0xf
 ; CHECK-NEXT:    cmp w8, #3
 ; CHECK-NEXT:    cset w0, hi
 ; CHECK-NEXT:    ret
   %urem = urem i4 %X, 5
   %cmp = icmp ne i4 %urem, 0
   ret i1 %cmp
 }

llvm/test/Transforms/SeparateConstOffsetFromGEP/AArch64/split-gep.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -O3 -mtriple=aarch64-linux-gnu | FileCheck %s
 
 %struct = type { i32, i32, i32 }
 
 define i32 @test1(%struct* %ptr, i64 %idx) {
 ; CHECK-LABEL: test1:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #12
-; CHECK-NEXT:    madd x8, x1, x8, x0
+; CHECK-NEXT:    lsl x8, x1, #3
+; CHECK-NEXT:    add x8, x8, x1, lsl #2
+; CHECK-NEXT:    add x8, x0, x8
 ; CHECK-NEXT:    ldr w9, [x8, #4]
 ; CHECK-NEXT:    tbnz w9, #31, .LBB0_2
 ; CHECK-NEXT:  // %bb.1:
 ; CHECK-NEXT:    mov w0, wzr
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB0_2: // %then
 ; CHECK-NEXT:    ldr w8, [x8, #8]
 ; CHECK-NEXT:    add w0, w9, w8