[AArch64] Lower multiplication by a constant int to madd

[Allen]

Summary

Lower a = b * C -1 into madd

a) instcombine change b * C -1 --> b * C + (-1)
b) machine-combine change b * C + (-1) --> madd

Assembler will transform the neg immedate of sub to add, see https://gcc.godbolt.org/z/cTcxePPf4
Fixes AArch64 part of https://github.com/llvm/llvm-project/issues/57255.

[efriedma]

That's mysterious... we had this code for years, but only used it for immediate we can synthesize using ORR. Huh.

Do we want to make the same change for MULADDWI_OP1 ?

[Allen]

Address the MULADDWI_OP1 according comment

[efriedma]

Code changes seem fine.

This sort of change might have some unexpected performance effects... do you have a benchmarking environment to see if there's an impact?

llvm/test/CodeGen/AArch64/machine-outliner-throw.ll

1

I think this NOTE was intentionally dropped from the test because some of the CHECK lines are manually written?

[Allen]

In D134336#3809859, @efriedma wrote:

Code changes seem fine.

This sort of change might have some unexpected performance effects... do you have a benchmarking environment to see if there's an impact?

Thanks for your suggestion, and I will try a performance test based on spec2017 in the near future.

[Allen]

Drop the Note in file machine-outliner-throw.ll

[Allen]

hi, @efriedma

• base on commit 7328ff75ba, I add this patch and test the performace change, there is no notable regression case base on tsv110 target. (I think there are still some fluctuations in performance and this combination does not significantly affect performance is expected.)
• ps: the 548.exchange2_r compile fail as it is a fortran source code, see https://www.spec.org/cpu2017/Docs/overview.html#benchmarks

llvm/test/CodeGen/AArch64/machine-outliner-throw.ll

1

Done, thanks

[Allen]

ping

[efriedma]

Perf seems fine, then.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5942

Sorry, missed an issue here the first time I looked at this...

The current complete set of opcodes that immediate expansion can produce is listed in AArch64ExpandPseudo::expandMOVImm . If the opcode is movz/movn, there are two immediate operands. But the opcode can also be ORRWri/ORRXri; in that case, there a register operand (wzr/xzr), and an immediate operand.

I think this comes up for something like int f(int a) { return a * 44 + 16773120; }.

[Allen]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5942

Despite the fact that movz/movn have two immediate operands, but it will generate a mov as alias, which only has one immediate operand.
see the following code frag in AArch64_IMM::expandMOVImm.

// Prefer MOVZ/MOVN over ORR because of the rules for the "mov" alias.
if ((BitSize / 16) - OneChunks <= 1 || (BitSize / 16) - ZeroChunks <= 1) {
  expandMOVImmSimple(Imm, BitSize, OneChunks, ZeroChunks, Insn);
  return;
}

[efriedma]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5942

The first instruction returned by expandMOVImm is always "mov", yes. "mov" is an alias for one of three instructions: movz, movn, and orr. (See section C3.3.4 in the ARM architecture reference.)

As far as I can tell, the operands you're adding are correct for movz and movn, but not orr.

[Allen]

revert to use ORR V, ZR, Imm if valid

[Allen]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5942

Done, revert to use orr if valid,thanks

[efriedma]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5942

Do we have test coverage for any constants like 16773120?

Instead of calling processLogicalImmediate like this, it would be more intuitive to explicitly check that MovI->Opcode is the opcode we expect. The current code works, but it's tightly tied to the exact implementation of expandMOVImm.

[Allen]

update to use MovI->Opcode check the opcode

[efriedma]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5799

Just make this say "mov", since that's always the alias.

5839

Maybe assert that the opcode is movn or movz here? For documentation.

[Allen]

add assert to check the opcode

[Allen]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5799

Done

5839

Apply your comment, thanks

[efriedma]

LGTM, but please clang-format before you merge.

[Allen]

update with clang-format

[Allen]

Update with clang-format

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

if (Pattern == MachineCombinerPattern::MULADDW_OP2) {
       RC = &AArch64::GPR64RegClass;
     }
     MUL = genFusedMultiply(MF, MRI, TII, Root, InsInstrs, 2, Opc, RC);
     break;
   case MachineCombinerPattern::MULADDWI_OP1:
   case MachineCombinerPattern::MULADDXI_OP1: {
     // MUL I=A,B,0
     // ADD R,I,Imm
-    // ==> ORR  V, ZR, Imm
+    // ==> MOV V, Imm
     // ==> MADD R,A,B,V
     // --- Create(MADD);
     const TargetRegisterClass *OrrRC;
     unsigned BitSize, OrrOpc, ZeroReg;
     if (Pattern == MachineCombinerPattern::MULADDWI_OP1) {
       OrrOpc = AArch64::ORRWri;
       OrrRC = &AArch64::GPR32spRegClass;
       BitSize = 32;
case MachineCombinerPattern::MULADDXI_OP1: {
     Register NewVR = MRI.createVirtualRegister(OrrRC);
     uint64_t Imm = Root.getOperand(2).getImm();
 
     if (Root.getOperand(3).isImm()) {
       unsigned Val = Root.getOperand(3).getImm();
       Imm = Imm << Val;
     }
     uint64_t UImm = SignExtend64(Imm, BitSize);
-    uint64_t Encoding;
-    if (!AArch64_AM::processLogicalImmediate(UImm, BitSize, Encoding))
+    // The immediate can be composed via a single instruction.
+    SmallVector<AArch64_IMM::ImmInsnModel, 4> Insn;
+    AArch64_IMM::expandMOVImm(UImm, BitSize, Insn);
+    if (Insn.size() != 1)
       return;
-    MachineInstrBuilder MIB1 =
-        BuildMI(MF, Root.getDebugLoc(), TII->get(OrrOpc), NewVR)
+    auto MovI = Insn.begin();
+    MachineInstrBuilder MIB1;
+    // MOV is an alias for one of three instructions: movz, movn, and orr.
+    if (MovI->Opcode == OrrOpc)
+      MIB1 = BuildMI(MF, Root.getDebugLoc(), TII->get(OrrOpc), NewVR)
                  .addReg(ZeroReg)
-            .addImm(Encoding);
+                 .addImm(MovI->Op2);
+    else {
+      if (BitSize == 32)
+        assert((MovI->Opcode == AArch64::MOVNWi ||
+                MovI->Opcode == AArch64::MOVZWi) &&
+               "Expected opcode");
+      else
+        assert((MovI->Opcode == AArch64::MOVNXi ||
+                MovI->Opcode == AArch64::MOVZXi) &&
+               "Expected opcode");
+      MIB1 = BuildMI(MF, Root.getDebugLoc(), TII->get(MovI->Opcode), NewVR)
+                 .addImm(MovI->Op1)
+                 .addImm(MovI->Op2);
+    }
     InsInstrs.push_back(MIB1);
     InstrIdxForVirtReg.insert(std::make_pair(NewVR, 0));
     MUL = genMaddR(MF, MRI, TII, Root, InsInstrs, 1, Opc, NewVR, RC);
     break;
   }
   case MachineCombinerPattern::MULSUBW_OP1:
   case MachineCombinerPattern::MULSUBX_OP1: {
     // MUL I=A,B,0
if (Pattern == MachineCombinerPattern::MULSUBW_OP2) {
       RC = &AArch64::GPR64RegClass;
     }
     MUL = genFusedMultiply(MF, MRI, TII, Root, InsInstrs, 2, Opc, RC);
     break;
   case MachineCombinerPattern::MULSUBWI_OP1:
   case MachineCombinerPattern::MULSUBXI_OP1: {
     // MUL I=A,B,0
     // SUB R,I, Imm
-    // ==> ORR  V, ZR, -Imm
+    // ==> MOV  V, -Imm
     // ==> MADD R,A,B,V // = -Imm + A*B
     // --- Create(MADD);
     const TargetRegisterClass *OrrRC;
     unsigned BitSize, OrrOpc, ZeroReg;
     if (Pattern == MachineCombinerPattern::MULSUBWI_OP1) {
       OrrOpc = AArch64::ORRWri;
       OrrRC = &AArch64::GPR32spRegClass;
       BitSize = 32;
case MachineCombinerPattern::MULSUBXI_OP1: {
     }
     Register NewVR = MRI.createVirtualRegister(OrrRC);
     uint64_t Imm = Root.getOperand(2).getImm();
     if (Root.getOperand(3).isImm()) {
       unsigned Val = Root.getOperand(3).getImm();
       Imm = Imm << Val;
     }
     uint64_t UImm = SignExtend64(-Imm, BitSize);
-    uint64_t Encoding;
-    if (!AArch64_AM::processLogicalImmediate(UImm, BitSize, Encoding))
+    // The immediate can be composed via a single instruction.
+    SmallVector<AArch64_IMM::ImmInsnModel, 4> Insn;
+    AArch64_IMM::expandMOVImm(UImm, BitSize, Insn);
+    if (Insn.size() != 1)
       return;
-    MachineInstrBuilder MIB1 =
-        BuildMI(MF, Root.getDebugLoc(), TII->get(OrrOpc), NewVR)
+    auto MovI = Insn.begin();
+    MachineInstrBuilder MIB1;
+    // MOV is an alias for one of three instructions: movz, movn, and orr.
+    if (MovI->Opcode == OrrOpc)
+      MIB1 = BuildMI(MF, Root.getDebugLoc(), TII->get(OrrOpc), NewVR)
                  .addReg(ZeroReg)
-            .addImm(Encoding);
+                 .addImm(MovI->Op2);
+    else {
+      if (BitSize == 32)
+        assert((MovI->Opcode == AArch64::MOVNWi ||
+                MovI->Opcode == AArch64::MOVZWi) &&
+               "Expected opcode");
+      else
+        assert((MovI->Opcode == AArch64::MOVNXi ||
+                MovI->Opcode == AArch64::MOVZXi) &&
+               "Expected opcode");
+      MIB1 = BuildMI(MF, Root.getDebugLoc(), TII->get(MovI->Opcode), NewVR)
+                 .addImm(MovI->Op1)
+                 .addImm(MovI->Op2);
+    }
     InsInstrs.push_back(MIB1);
     InstrIdxForVirtReg.insert(std::make_pair(NewVR, 0));
     MUL = genMaddR(MF, MRI, TII, Root, InsInstrs, 1, Opc, NewVR, RC);
     break;
   }
 
   case MachineCombinerPattern::MULADDv8i8_OP1:
     Opc = AArch64::MLAv8i8;

llvm/test/CodeGen/AArch64/addimm-mulimm.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs < %s -mtriple=aarch64-linux-gnu | FileCheck %s
 
 define i64 @addimm_mulimm_accept_00(i64 %a) {
 ; CHECK-LABEL: addimm_mulimm_accept_00:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #37
-; CHECK-NEXT:    mul x8, x0, x8
-; CHECK-NEXT:    add x0, x8, #1147
+; CHECK-NEXT:    mov x9, #1147
+; CHECK-NEXT:    madd x0, x0, x8, x9
 ; CHECK-NEXT:    ret
   %tmp0 = add i64 %a, 31
   %tmp1 = mul i64 %tmp0, 37
   ret i64 %tmp1
 }
 
 define i64 @addimm_mulimm_accept_01(i64 %a) {
 ; CHECK-LABEL: addimm_mulimm_accept_01:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #37
-; CHECK-NEXT:    mul x8, x0, x8
-; CHECK-NEXT:    sub x0, x8, #1147
+; CHECK-NEXT:    mov x9, #-1147
+; CHECK-NEXT:    madd x0, x0, x8, x9
 ; CHECK-NEXT:    ret
   %tmp0 = add i64 %a, -31
   %tmp1 = mul i64 %tmp0, 37
   ret i64 %tmp1
 }
 
 define signext i32 @addimm_mulimm_accept_02(i32 signext %a) {
 ; CHECK-LABEL: addimm_mulimm_accept_02:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #37
-; CHECK-NEXT:    mul w8, w0, w8
-; CHECK-NEXT:    add w0, w8, #1147
+; CHECK-NEXT:    mov w9, #1147
+; CHECK-NEXT:    madd w0, w0, w8, w9
 ; CHECK-NEXT:    ret
   %tmp0 = add i32 %a, 31
   %tmp1 = mul i32 %tmp0, 37
   ret i32 %tmp1
 }
 
 define signext i32 @addimm_mulimm_accept_03(i32 signext %a) {
 ; CHECK-LABEL: addimm_mulimm_accept_03:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #37
-; CHECK-NEXT:    mul w8, w0, w8
-; CHECK-NEXT:    sub w0, w8, #1147
+; CHECK-NEXT:    mov w9, #-1147
+; CHECK-NEXT:    madd w0, w0, w8, w9
 ; CHECK-NEXT:    ret
   %tmp0 = add i32 %a, -31
   %tmp1 = mul i32 %tmp0, 37
   ret i32 %tmp1
 }
 
 define i64 @addimm_mulimm_accept_10(i64 %a) {
 ; CHECK-LABEL: addimm_mulimm_accept_10:

llvm/test/CodeGen/AArch64/machine-outliner-throw.ll

 ; RUN: llc -verify-machineinstrs -enable-machine-outliner -mtriple=aarch64-arm-none-eabi < %s | FileCheck %s
 ; RUN: llc -verify-machineinstrs -enable-machine-outliner -mtriple=aarch64-arm-none-eabi -stop-after=machine-outliner < %s | FileCheck %s -check-prefix=TARGET_FEATURES
 
 ; Make sure that we haven't added nouwind.
 ; TARGET_FEATURES: define internal void @OUTLINED_FUNCTION_0()
 ; TARGET_FEATURES-SAME: #[[ATTR_NUM:[0-9]+]]
 ; TARGET_FEATURES: attributes #[[ATTR_NUM]] = { minsize optsize }
 
 define dso_local i32 @_Z5func1i(i32 %x) #0 {
 ; CHECK-LABEL: _Z5func1i:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset w19, -8
 ; CHECK-NEXT:    .cfi_offset w30, -16
-; CHECK-NEXT:    orr w8, wzr, #0x1
+; CHECK-NEXT:    mov w8, #1
 ; CHECK-NEXT:    madd w19, w0, w0, w8
 ; CHECK-NEXT:    mov w0, #4
 ; CHECK-NEXT:    bl __cxa_allocate_exception
 ; CHECK-NEXT:    bl OUTLINED_FUNCTION_0
 entry:
   %mul = mul nsw i32 %x, %x
   %add = add nuw nsw i32 %mul, 1
   %exception = tail call i8* @__cxa_allocate_exception(i64 4) #1
   %0 = bitcast i8* %exception to i32*
   store i32 %add, i32* %0, align 16
   tail call void @__cxa_throw(i8* %exception, i8* bitcast (i8** @_ZTIi to i8*), i8* null) #2
   unreachable
 }
 
 define dso_local i32 @_Z5func2c(i8 %x) #0 {
 ; CHECK-LABEL: _Z5func2c:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset w19, -8
 ; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    and w8, w0, #0xff
 ; CHECK-NEXT:    mov w0, #4
-; CHECK-NEXT:    orr w9, wzr, #0x1
+; CHECK-NEXT:    mov w9, #1
 ; CHECK-NEXT:    madd w19, w8, w8, w9
 ; CHECK-NEXT:    bl __cxa_allocate_exception
 ; CHECK-NEXT:    bl OUTLINED_FUNCTION_0
 entry:
   %conv = zext i8 %x to i32
   %mul = mul nuw nsw i32 %conv, %conv
   %add = add nuw nsw i32 %mul, 1
   %exception = tail call i8* @__cxa_allocate_exception(i64 4) #1

llvm/test/CodeGen/AArch64/madd-combiner.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-apple-darwin            -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,CHECK-ISEL
 ; RUN: llc -mtriple=aarch64-apple-darwin -fast-isel -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,CHECK-FAST
 
 ; Test that we use the correct register class.
 define i32 @mul_add_imm(i32 %a, i32 %b) {
 ; CHECK-LABEL: mul_add_imm:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    orr w8, wzr, #0x4
+; CHECK-NEXT:    mov w8, #4
 ; CHECK-NEXT:    madd w0, w0, w1, w8
 ; CHECK-NEXT:    ret
   %1 = mul i32 %a, %b
   %2 = add i32 %1, 4
   ret i32 %2
 }
 
 define i32 @mul_sub_imm1(i32 %a, i32 %b) {

 ; CHECK-ISEL-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-ISEL-NEXT:    cbnz w8, LBB2_1
 ; CHECK-ISEL-NEXT:  ; %bb.2: ; %for.end20
 ; CHECK-ISEL-NEXT:    ret
 ;
 ; CHECK-FAST-LABEL: mul_add_imm2:
 ; CHECK-FAST:       ; %bb.0: ; %entry
 ; CHECK-FAST-NEXT:    mov x8, #-3
-; CHECK-FAST-NEXT:    orr x9, xzr, #0xfffffffffffffffd
+; CHECK-FAST-NEXT:    mov x9, #-3
 ; CHECK-FAST-NEXT:    madd x8, x8, x8, x9
 ; CHECK-FAST-NEXT:    mov x9, #45968
 ; CHECK-FAST-NEXT:    movk x9, #48484, lsl #16
 ; CHECK-FAST-NEXT:    movk x9, #323, lsl #32
 ; CHECK-FAST-NEXT:  LBB2_1: ; %for.body8
 ; CHECK-FAST-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-FAST-NEXT:    cmp x8, x9
 ; CHECK-FAST-NEXT:    b.lt LBB2_1

llvm/test/CodeGen/AArch64/mul_pow2.ll

 ; GISEL-NEXT:    ret
 
   %ext = sext i32 %x to i64
   %mul = mul nsw i64 %ext, 6
   %sub = sub i64 0, %mul
   ret i64 %sub
 }
 
+; We may hoist the "mov" instructions out of a loop
+define i32 @mull6_sub(i32 %x) {
+; CHECK-LABEL: mull6_sub:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, #6
+; CHECK-NEXT:    mov w9, #-1
+; CHECK-NEXT:    madd w0, w0, w8, w9
+; CHECK-NEXT:    ret
+;
+; GISEL-LABEL: mull6_sub:
+; GISEL:       // %bb.0:
+; GISEL-NEXT:    mov w8, #6
+; GISEL-NEXT:    mov w9, #-1
+; GISEL-NEXT:    madd w0, w0, w8, w9
+; GISEL-NEXT:    ret
+  %mul = mul nsw i32 %x, 6
+  %sub = add nsw i32 %mul, -1
+  ret i32 %sub
+}
+
+define i64 @mull6_sub_orr(i64 %x) {
+; CHECK-LABEL: mull6_sub_orr:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, #6
+; CHECK-NEXT:    mov x9, #16773120
+; CHECK-NEXT:    madd x0, x0, x8, x9
+; CHECK-NEXT:    ret
+;
+; GISEL-LABEL: mull6_sub_orr:
+; GISEL:       // %bb.0:
+; GISEL-NEXT:    mov w8, #6
+; GISEL-NEXT:    mov x9, #16773120
+; GISEL-NEXT:    madd x0, x0, x8, x9
+; GISEL-NEXT:    ret
+  %mul = mul nsw i64 %x, 6
+  %sub = add nsw i64 %mul, 16773120
+  ret i64 %sub
+}
+
 define i32 @test7(i32 %x) {
 ; CHECK-LABEL: test7:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    lsl w8, w0, #3
 ; CHECK-NEXT:    sub w0, w8, w0
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test7:

 ; CHECK-NEXT:    movi v2.4s, #1, msl #16
 ; CHECK-NEXT:    shl v0.4s, v0.4s, #6
 ; CHECK-NEXT:    sub v0.4s, v1.4s, v0.4s
 ; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: muladd_demand_commute:
 ; GISEL:       // %bb.0:
-; GISEL-NEXT:    adrp x8, .LCPI42_1
-; GISEL-NEXT:    ldr q2, [x8, :lo12:.LCPI42_1]
-; GISEL-NEXT:    adrp x8, .LCPI42_0
+; GISEL-NEXT:    adrp x8, .LCPI44_1
+; GISEL-NEXT:    ldr q2, [x8, :lo12:.LCPI44_1]
+; GISEL-NEXT:    adrp x8, .LCPI44_0
 ; GISEL-NEXT:    mla v1.4s, v0.4s, v2.4s
-; GISEL-NEXT:    ldr q0, [x8, :lo12:.LCPI42_0]
+; GISEL-NEXT:    ldr q0, [x8, :lo12:.LCPI44_0]
 ; GISEL-NEXT:    and v0.16b, v1.16b, v0.16b
 ; GISEL-NEXT:    ret
   %m = mul <4 x i32> %x, <i32 131008, i32 131008, i32 131008, i32 131008>
   %a = add <4 x i32> %m, %y
   %r = and <4 x i32> %a, <i32 131071, i32 131071, i32 131071, i32 131071>
   ret <4 x i32> %r
 }

llvm/test/CodeGen/AArch64/srem-seteq.ll

; CHECK-NEXT:    ret
   ret i32 %ret
 }
 
 ; This is like test_srem_odd, except the divisor has bit 30 set.
 define i32 @test_srem_odd_bit30(i32 %X) nounwind {
 ; CHECK-LABEL: test_srem_odd_bit30:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #43691
-; CHECK-NEXT:    orr w9, wzr, #0x1
+; CHECK-NEXT:    mov w9, #1
 ; CHECK-NEXT:    movk w8, #27306, lsl #16
 ; CHECK-NEXT:    madd w8, w0, w8, w9
 ; CHECK-NEXT:    cmp w8, #3
 ; CHECK-NEXT:    cset w0, lo
 ; CHECK-NEXT:    ret
   %srem = srem i32 %X, 1073741827
   %cmp = icmp eq i32 %srem, 0
   %ret = zext i1 %cmp to i32
   ret i32 %ret
 }
 
 ; This is like test_srem_odd, except the divisor has bit 31 set.
 define i32 @test_srem_odd_bit31(i32 %X) nounwind {
 ; CHECK-LABEL: test_srem_odd_bit31:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #21845
-; CHECK-NEXT:    orr w9, wzr, #0x1
+; CHECK-NEXT:    mov w9, #1
 ; CHECK-NEXT:    movk w8, #54613, lsl #16
 ; CHECK-NEXT:    madd w8, w0, w8, w9
 ; CHECK-NEXT:    cmp w8, #3
 ; CHECK-NEXT:    cset w0, lo
 ; CHECK-NEXT:    ret
   %srem = srem i32 %X, 2147483651
   %cmp = icmp eq i32 %srem, 0
   %ret = zext i1 %cmp to i32
; CHECK-NEXT:    ret
   ret i32 %ret
 }
 
 ; This is like test_srem_even, except the divisor has bit 30 set.
 define i32 @test_srem_even_bit30(i32 %X) nounwind {
 ; CHECK-LABEL: test_srem_even_bit30:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #20165
-; CHECK-NEXT:    orr w9, wzr, #0x8
+; CHECK-NEXT:    mov w9, #8
 ; CHECK-NEXT:    movk w8, #64748, lsl #16
 ; CHECK-NEXT:    madd w8, w0, w8, w9
 ; CHECK-NEXT:    ror w8, w8, #3
 ; CHECK-NEXT:    cmp w8, #3
 ; CHECK-NEXT:    cset w0, lo
 ; CHECK-NEXT:    ret
   %srem = srem i32 %X, 1073741928
   %cmp = icmp eq i32 %srem, 0
   %ret = zext i1 %cmp to i32
   ret i32 %ret
 }
 
 ; This is like test_srem_odd, except the divisor has bit 31 set.
 define i32 @test_srem_even_bit31(i32 %X) nounwind {
 ; CHECK-LABEL: test_srem_even_bit31:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #1285
-; CHECK-NEXT:    orr w9, wzr, #0x2
+; CHECK-NEXT:    mov w9, #2
 ; CHECK-NEXT:    movk w8, #50437, lsl #16
 ; CHECK-NEXT:    madd w8, w0, w8, w9
 ; CHECK-NEXT:    ror w8, w8, #1
 ; CHECK-NEXT:    cmp w8, #3
 ; CHECK-NEXT:    cset w0, lo
 ; CHECK-NEXT:    ret
   %srem = srem i32 %X, 2147483750
   %cmp = icmp eq i32 %srem, 0

llvm/test/CodeGen/AArch64/urem-seteq-nonzero.ll

; CHECK-NEXT:    ret
   %cmp = icmp eq i32 %urem, 2
   ret i1 %cmp
 }
 
 define i1 @t32_6_3(i32 %X) nounwind {
 ; CHECK-LABEL: t32_6_3:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #43691
-; CHECK-NEXT:    mov w9, #43691
+; CHECK-NEXT:    mov w9, #-1
 ; CHECK-NEXT:    movk w8, #43690, lsl #16
+; CHECK-NEXT:    madd w8, w0, w8, w9
+; CHECK-NEXT:    mov w9, #43691
 ; CHECK-NEXT:    movk w9, #10922, lsl #16
-; CHECK-NEXT:    mul w8, w0, w8
-; CHECK-NEXT:    sub w8, w8, #1
 ; CHECK-NEXT:    ror w8, w8, #1
 ; CHECK-NEXT:    cmp w8, w9
 ; CHECK-NEXT:    cset w0, lo
 ; CHECK-NEXT:    ret
   %urem = urem i32 %X, 6
   %cmp = icmp eq i32 %urem, 3
   ret i1 %cmp
 }
; CHECK-NEXT:    ret
   %cmp = icmp eq i16 %urem, 2
   ret i1 %cmp
 }
 
 define i1 @t8_3_2(i8 %X) nounwind {
 ; CHECK-LABEL: t8_3_2:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #-85
-; CHECK-NEXT:    mul w8, w0, w8
-; CHECK-NEXT:    sub w8, w8, #86
+; CHECK-NEXT:    mov w9, #-86
+; CHECK-NEXT:    madd w8, w0, w8, w9
 ; CHECK-NEXT:    and w8, w8, #0xff
 ; CHECK-NEXT:    cmp w8, #85
 ; CHECK-NEXT:    cset w0, lo
 ; CHECK-NEXT:    ret
   %urem = urem i8 %X, 3
   %cmp = icmp eq i8 %urem, 2
   ret i1 %cmp
 }