[x86] try harder to form LEA from ADD to avoid flag conflicts (PR40483)

[spatel]

Summary

LEA doesn't affect flags, so use it more liberally to replace an ADD when we know that the ADD operands affect flags.

In the motivating example from PR40483:
https://bugs.llvm.org/show_bug.cgi?id=40483
...this lets us avoid duplicating a math op just to avoid flag conflict.

[grandinj]

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

2487

Should this not be an || condition ?

[spatel]

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

2487

It could be || (and I tried that initially), but I saw possible regressions (missed load folding) with that logic.

Given that this is a heuristic, it's hard to determine statically when it will be profitable.

Ok if I add a TODO comment about extending this in a follow-up patch? If we can recover the load folding, I agree that using || would be better.

[grandinj]

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

2487

Sounds reasonable (noting that I am just a bystander here, not a real reviewer)

[RKSimon]

LGTM - @craig.topper any comments?

[craig.topper]

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

2482

Should we make sure the flag output has a user? ADC/SBB could definitely only have a live flag input. Not sure about the others.

[spatel]

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

2482

Yes, that seems safer, although we do try to reduce back to regular opcodes if the flag wasn't needed. I don't see any test diffs with a draft of that change.

[spatel]

Patch updated:

1. Check flag uses to avoid unintended transform.
2. Add TODO comment about && vs. ||.

[craig.topper]

Are the multiply test changes due to the flags being used by seto? But seto usage should never be in danger of creating the instruction duplication we're seeing in the motivating case. It does look like we're getting an improvement on those tests, but not for the reason we're selecting LEA.

[spatel]

In D64707#1586664, @craig.topper wrote:

Are the multiply test changes due to the flags being used by seto? But seto usage should never be in danger of creating the instruction duplication we're seeing in the motivating case. It does look like we're getting an improvement on those tests, but not for the reason we're selecting LEA.

Yes, I think you're right - we have something like this:

t95: i32,i32 = X86ISD::ADD t73:1, t93
t102: i32,i32 = X86ISD::UMUL t36, t38
t106: i32,i32 = X86ISD::UMUL t40, t33
  t93: i32 = add t106, t102
        t107: i8 = X86ISD::SETCC Constant:i8<0>, t106:1
      t104: i8 = X86ISD::SETCC Constant:i8<0>, t102:1

So the use of LEA is presumably just making things easier for register allocation.
Not sure how to handle this:

1. Remove UMUL/SMUL from the cases in this patch and forego those improvements.
2. Add a comment expanding on our reasoning for the transform and keep the improvements.
3. Refine the flags check somehow (and again forego those improvements in this patch)?

[RKSimon]

If we focussed just on PR40483 for now - do we just need X86ISD ADD + SUB (ADC + SBB) ?

[spatel]

In D64707#1591179, @RKSimon wrote:

If we focussed just on PR40483 for now - do we just need X86ISD ADD + SUB (ADC + SBB) ?

Yes - mul and logic don't need to be in the list to change the motivating bug. I'll comment those opcodes out, and we can make this patch independent of them.

[spatel]

Patch updated:
Just the add/sub opcodes for now.

[RKSimon]

LGTM - cheers. Ideally we'd get this into the 9.00 release branch.

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

bool X86DAGToDAGISel::selectLEAAddr(SDValue N,
   if (AM.hasSymbolicDisplacement()) {
     // For X86-64, always use LEA to materialize RIP-relative addresses.
     if (Subtarget->is64Bit())
       Complexity = 4;
     else
       Complexity += 2;
   }
 
+  // Heuristic: try harder to form an LEA from ADD if the operands set flags.
+  // Unlike ADD, LEA does not affect flags, so we will be less likely to require
+  // duplicating flag-producing instructions later in the pipeline.
+  if (N.getOpcode() == ISD::ADD) {
+    auto isMathWithFlags = [](SDValue V) {
+      switch (V.getOpcode()) {
+      case X86ISD::ADD:
+      case X86ISD::SUB:
+      case X86ISD::ADC:
+      case X86ISD::SBB:
+      case X86ISD::SMUL:
+      case X86ISD::UMUL:
+      case X86ISD::OR:
+      case X86ISD::XOR:
+      case X86ISD::AND:
+        // Value 1 is the flag output of the node - verify it's not dead.
+        return !SDValue(V.getNode(), 1).use_empty();
+      default:
+        return false;
+      }
+    };
+    // TODO: This could be an 'or' rather than 'and' to make the transform more
+    //       likely to happen. We might want to factor in whether there's a
+    //       load folding opportunity for the math op that disappears with LEA.
+    if (isMathWithFlags(N.getOperand(0)) && isMathWithFlags(N.getOperand(1)))
+      Complexity++;
+  }
+
   if (AM.Disp)
     Complexity++;
 
   // If it isn't worth using an LEA, reject it.
   if (Complexity <= 2)
     return false;
 
   getAddressOperands(AM, DL, VT, Base, Scale, Index, Disp, Segment);

llvm/test/CodeGen/X86/combine-sbb.ll

; X64-NEXT:    retq
   %9 = add i32 %8, %5
   %10 = select i1 %7, i32 %9, i32 0
   ret i32 %10
 }
 
 define i32 @PR40483_sub6(i32*, i32) nounwind {
 ; X86-LABEL: PR40483_sub6:
 ; X86:       # %bb.0:
-; X86-NEXT:    pushl %edi
-; X86-NEXT:    pushl %esi
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    movl (%edx), %esi
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT:    movl %esi, %ecx
-; X86-NEXT:    subl %edi, %ecx
+; X86-NEXT:    movl (%edx), %ecx
 ; X86-NEXT:    xorl %eax, %eax
-; X86-NEXT:    subl %edi, %esi
-; X86-NEXT:    movl %esi, (%edx)
+; X86-NEXT:    subl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl %ecx, (%edx)
 ; X86-NEXT:    jae .LBB8_2
 ; X86-NEXT:  # %bb.1:
-; X86-NEXT:    addl %ecx, %ecx
-; X86-NEXT:    movl %ecx, %eax
+; X86-NEXT:    leal (%ecx,%ecx), %eax
 ; X86-NEXT:  .LBB8_2:
-; X86-NEXT:    popl %esi
-; X86-NEXT:    popl %edi
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: PR40483_sub6:
 ; X64:       # %bb.0:
-; X64-NEXT:    movl (%rdi), %ecx
-; X64-NEXT:    movl %ecx, %edx
-; X64-NEXT:    subl %esi, %edx
-; X64-NEXT:    addl %edx, %edx
-; X64-NEXT:    xorl %eax, %eax
-; X64-NEXT:    subl %esi, %ecx
-; X64-NEXT:    movl %ecx, (%rdi)
-; X64-NEXT:    cmovbl %edx, %eax
+; X64-NEXT:    movl (%rdi), %eax
+; X64-NEXT:    xorl %ecx, %ecx
+; X64-NEXT:    subl %esi, %eax
+; X64-NEXT:    movl %eax, (%rdi)
+; X64-NEXT:    leal (%rax,%rax), %eax
+; X64-NEXT:    cmovael %ecx, %eax
 ; X64-NEXT:    retq
   %3 = load i32, i32* %0, align 8
   %4 = tail call { i8, i32 } @llvm.x86.subborrow.32(i8 0, i32 %3, i32 %1)
   %5 = extractvalue { i8, i32 } %4, 1
   store i32 %5, i32* %0, align 8
   %6 = extractvalue { i8, i32 } %4, 0
   %7 = icmp eq i8 %6, 0
   %8 = sub i32 %3, %1
   %9 = add i32 %8, %5
   %10 = select i1 %7, i32 0, i32 %9
   ret i32 %10
 }
 
 declare { i8, i32 } @llvm.x86.subborrow.32(i8, i32, i32)

llvm/test/CodeGen/X86/umulo-128-legalisation-lowering.ll

 ; X64-NEXT:    testq %rsi, %rsi
 ; X64-NEXT:    setne %r9b
 ; X64-NEXT:    andb %dl, %r9b
 ; X64-NEXT:    mulq %r8
 ; X64-NEXT:    movq %rax, %rsi
 ; X64-NEXT:    seto %r10b
 ; X64-NEXT:    movq %rcx, %rax
 ; X64-NEXT:    mulq %rdi
-; X64-NEXT:    movq %rax, %rcx
 ; X64-NEXT:    seto %r11b
 ; X64-NEXT:    orb %r10b, %r11b
-; X64-NEXT:    addq %rsi, %rcx
+; X64-NEXT:    leaq (%rsi,%rax), %rcx
 ; X64-NEXT:    movq %rdi, %rax
 ; X64-NEXT:    mulq %r8
 ; X64-NEXT:    addq %rcx, %rdx
 ; X64-NEXT:    setb %cl
 ; X64-NEXT:    orb %r11b, %cl
 ; X64-NEXT:    orb %r9b, %cl
 ; X64-NEXT:    retq
 ;

 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X86-NEXT:    mull %ebx
 ; X86-NEXT:    movl %eax, %ecx
 ; X86-NEXT:    seto {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill
 ; X86-NEXT:    movl %esi, %eax
 ; X86-NEXT:    mull %edi
-; X86-NEXT:    movl %eax, %esi
+; X86-NEXT:    leal (%ecx,%eax), %esi
 ; X86-NEXT:    seto {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill
-; X86-NEXT:    addl %ecx, %esi
 ; X86-NEXT:    movl %edi, %eax
 ; X86-NEXT:    mull %ebx
-; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    movl %eax, %ebp
 ; X86-NEXT:    movl %edx, %ecx
 ; X86-NEXT:    addl %esi, %ecx
 ; X86-NEXT:    setb {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebp
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X86-NEXT:    mull %ebp
-; X86-NEXT:    movl %eax, %edi
-; X86-NEXT:    seto {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
-; X86-NEXT:    movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    mull %ebx
 ; X86-NEXT:    movl %eax, %esi
 ; X86-NEXT:    seto {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill
-; X86-NEXT:    addl %edi, %esi
-; X86-NEXT:    movl %ebx, %eax
-; X86-NEXT:    mull %ebp
-; X86-NEXT:    movl %ebp, %ebx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
+; X86-NEXT:    movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    mull %edi
+; X86-NEXT:    leal (%esi,%eax), %esi
+; X86-NEXT:    seto {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill
+; X86-NEXT:    movl %edi, %eax
+; X86-NEXT:    mull %ebx
 ; X86-NEXT:    movl %edx, %edi
 ; X86-NEXT:    addl %esi, %edi
 ; X86-NEXT:    setb {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill
-; X86-NEXT:    addl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Folded Reload
+; X86-NEXT:    addl %ebp, %eax
 ; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    adcl %ecx, %edi
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebp
-; X86-NEXT:    movl %ebp, %eax
-; X86-NEXT:    movl %ebx, %ecx
-; X86-NEXT:    mull %ebx
-; X86-NEXT:    movl %edx, %ebx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
+; X86-NEXT:    movl %ebx, %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    mull %ecx
+; X86-NEXT:    movl %edx, %esi
 ; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    mull %ecx
-; X86-NEXT:    movl %edx, %esi
+; X86-NEXT:    movl %edx, %ebp
 ; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    addl %ebx, %ecx
-; X86-NEXT:    adcl $0, %esi
-; X86-NEXT:    movl %ebp, %eax
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebp
-; X86-NEXT:    mull %ebp
-; X86-NEXT:    movl %edx, %ebx
+; X86-NEXT:    addl %esi, %ecx
+; X86-NEXT:    adcl $0, %ebp
+; X86-NEXT:    movl %ebx, %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    mull %edx
+; X86-NEXT:    movl %edx, %esi
 ; X86-NEXT:    addl %ecx, %eax
 ; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT:    adcl %esi, %ebx
-; X86-NEXT:    setb {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill
+; X86-NEXT:    adcl %ebp, %esi
+; X86-NEXT:    setb %bl
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X86-NEXT:    mull %ebp
+; X86-NEXT:    mull {{[0-9]+}}(%esp)
 ; X86-NEXT:    movl %edx, %ecx
-; X86-NEXT:    movl %eax, %esi
-; X86-NEXT:    addl %ebx, %esi
-; X86-NEXT:    movzbl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 1-byte Folded Reload
+; X86-NEXT:    movl %eax, %ebp
+; X86-NEXT:    addl %esi, %ebp
+; X86-NEXT:    movzbl %bl, %eax
 ; X86-NEXT:    adcl %eax, %ecx
 ; X86-NEXT:    xorl %edx, %edx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    mull %edx
-; X86-NEXT:    movl %edx, %ebp
-; X86-NEXT:    movl %eax, %ebx
+; X86-NEXT:    movl %edx, %ebx
+; X86-NEXT:    movl %eax, %esi
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    xorl %edx, %edx
 ; X86-NEXT:    mull %edx
-; X86-NEXT:    addl %ebx, %eax
-; X86-NEXT:    adcl %ebp, %edx
 ; X86-NEXT:    addl %esi, %eax
+; X86-NEXT:    adcl %ebx, %edx
+; X86-NEXT:    addl %ebp, %eax
 ; X86-NEXT:    adcl %ecx, %edx
 ; X86-NEXT:    addl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Folded Reload
 ; X86-NEXT:    adcl %edi, %edx
 ; X86-NEXT:    setb {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Spill
 ; X86-NEXT:    cmpl $0, {{[0-9]+}}(%esp)
 ; X86-NEXT:    setne %cl
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X86-NEXT:    testl %esi, %esi

llvm/test/CodeGen/X86/umulo-64-legalisation-lowering.ll

 ; X86-NEXT:    testl %eax, %eax
 ; X86-NEXT:    setne %bl
 ; X86-NEXT:    andb %dl, %bl
 ; X86-NEXT:    mull {{[0-9]+}}(%esp)
 ; X86-NEXT:    movl %eax, %edi
 ; X86-NEXT:    seto %cl
 ; X86-NEXT:    movl %esi, %eax
 ; X86-NEXT:    mull %ebp
-; X86-NEXT:    movl %eax, %esi
 ; X86-NEXT:    seto %ch
 ; X86-NEXT:    orb %cl, %ch
-; X86-NEXT:    addl %edi, %esi
+; X86-NEXT:    leal (%edi,%eax), %esi
 ; X86-NEXT:    movl %ebp, %eax
 ; X86-NEXT:    mull {{[0-9]+}}(%esp)
 ; X86-NEXT:    addl %esi, %edx
 ; X86-NEXT:    setb %cl
 ; X86-NEXT:    orb %ch, %cl
 ; X86-NEXT:    orb %bl, %cl
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    .cfi_def_cfa_offset 16

llvm/test/CodeGen/X86/vec_umulo.ll

 }
 
 define <2 x i32> @umulo_v2i128(<2 x i128> %a0, <2 x i128> %a1, <2 x i128>* %p2) nounwind {
 ; SSE2-LABEL: umulo_v2i128:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    pushq %rbp
 ; SSE2-NEXT:    pushq %r15
 ; SSE2-NEXT:    pushq %r14
-; SSE2-NEXT:    pushq %r13
 ; SSE2-NEXT:    pushq %r12
 ; SSE2-NEXT:    pushq %rbx
 ; SSE2-NEXT:    movq %rcx, %rax
 ; SSE2-NEXT:    movq %rdx, %r12
 ; SSE2-NEXT:    movq %rdi, %r11
 ; SSE2-NEXT:    movq {{[0-9]+}}(%rsp), %r14
 ; SSE2-NEXT:    movq {{[0-9]+}}(%rsp), %r15
 ; SSE2-NEXT:    movq {{[0-9]+}}(%rsp), %r10
 ; SSE2-NEXT:    testq %r10, %r10
 ; SSE2-NEXT:    setne %dl
 ; SSE2-NEXT:    testq %rcx, %rcx
-; SSE2-NEXT:    setne %r13b
-; SSE2-NEXT:    andb %dl, %r13b
+; SSE2-NEXT:    setne %bl
+; SSE2-NEXT:    andb %dl, %bl
 ; SSE2-NEXT:    mulq %r15
 ; SSE2-NEXT:    movq %rax, %rdi
 ; SSE2-NEXT:    seto %bpl
 ; SSE2-NEXT:    movq %r10, %rax
 ; SSE2-NEXT:    mulq %r12
-; SSE2-NEXT:    movq %rax, %rbx
 ; SSE2-NEXT:    seto %cl
 ; SSE2-NEXT:    orb %bpl, %cl
-; SSE2-NEXT:    addq %rdi, %rbx
+; SSE2-NEXT:    leaq (%rdi,%rax), %rbp
 ; SSE2-NEXT:    movq %r12, %rax
 ; SSE2-NEXT:    mulq %r15
 ; SSE2-NEXT:    movq %rax, %r10
-; SSE2-NEXT:    movq %rdx, %r15
-; SSE2-NEXT:    addq %rbx, %r15
+; SSE2-NEXT:    movq %rdx, %rdi
+; SSE2-NEXT:    addq %rbp, %rdi
 ; SSE2-NEXT:    setb %al
 ; SSE2-NEXT:    orb %cl, %al
-; SSE2-NEXT:    orb %r13b, %al
+; SSE2-NEXT:    orb %bl, %al
 ; SSE2-NEXT:    movzbl %al, %ebp
 ; SSE2-NEXT:    testq %r9, %r9
 ; SSE2-NEXT:    setne %al
 ; SSE2-NEXT:    testq %rsi, %rsi
-; SSE2-NEXT:    setne %r13b
-; SSE2-NEXT:    andb %al, %r13b
+; SSE2-NEXT:    setne %r12b
+; SSE2-NEXT:    andb %al, %r12b
 ; SSE2-NEXT:    movq %rsi, %rax
 ; SSE2-NEXT:    mulq %r8
 ; SSE2-NEXT:    movq %rax, %rsi
-; SSE2-NEXT:    seto %r12b
+; SSE2-NEXT:    seto %r15b
 ; SSE2-NEXT:    movq %r9, %rax
 ; SSE2-NEXT:    mulq %r11
-; SSE2-NEXT:    movq %rax, %rdi
-; SSE2-NEXT:    seto %bl
-; SSE2-NEXT:    orb %r12b, %bl
-; SSE2-NEXT:    addq %rsi, %rdi
+; SSE2-NEXT:    seto %cl
+; SSE2-NEXT:    orb %r15b, %cl
+; SSE2-NEXT:    addq %rax, %rsi
 ; SSE2-NEXT:    movq %r11, %rax
 ; SSE2-NEXT:    mulq %r8
-; SSE2-NEXT:    addq %rdi, %rdx
-; SSE2-NEXT:    setb %cl
-; SSE2-NEXT:    orb %bl, %cl
-; SSE2-NEXT:    orb %r13b, %cl
-; SSE2-NEXT:    movzbl %cl, %ecx
+; SSE2-NEXT:    addq %rsi, %rdx
+; SSE2-NEXT:    setb %bl
+; SSE2-NEXT:    orb %cl, %bl
+; SSE2-NEXT:    orb %r12b, %bl
+; SSE2-NEXT:    movzbl %bl, %ecx
 ; SSE2-NEXT:    movd %ecx, %xmm0
 ; SSE2-NEXT:    pinsrw $4, %ebp, %xmm0
 ; SSE2-NEXT:    movq %r10, 16(%r14)
 ; SSE2-NEXT:    movq %rax, (%r14)
-; SSE2-NEXT:    movq %r15, 24(%r14)
+; SSE2-NEXT:    movq %rdi, 24(%r14)
 ; SSE2-NEXT:    movq %rdx, 8(%r14)
 ; SSE2-NEXT:    psllq $63, %xmm0
 ; SSE2-NEXT:    psrad $31, %xmm0
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,3,3]
 ; SSE2-NEXT:    popq %rbx
 ; SSE2-NEXT:    popq %r12
-; SSE2-NEXT:    popq %r13
 ; SSE2-NEXT:    popq %r14
 ; SSE2-NEXT:    popq %r15
 ; SSE2-NEXT:    popq %rbp
 ; SSE2-NEXT:    retq
 ;
 ; SSSE3-LABEL: umulo_v2i128:
 ; SSSE3:       # %bb.0:
 ; SSSE3-NEXT:    pushq %rbp
 ; SSSE3-NEXT:    pushq %r15
 ; SSSE3-NEXT:    pushq %r14
-; SSSE3-NEXT:    pushq %r13
 ; SSSE3-NEXT:    pushq %r12
 ; SSSE3-NEXT:    pushq %rbx
 ; SSSE3-NEXT:    movq %rcx, %rax
 ; SSSE3-NEXT:    movq %rdx, %r12
 ; SSSE3-NEXT:    movq %rdi, %r11
 ; SSSE3-NEXT:    movq {{[0-9]+}}(%rsp), %r14
 ; SSSE3-NEXT:    movq {{[0-9]+}}(%rsp), %r15
 ; SSSE3-NEXT:    movq {{[0-9]+}}(%rsp), %r10
 ; SSSE3-NEXT:    testq %r10, %r10
 ; SSSE3-NEXT:    setne %dl
 ; SSSE3-NEXT:    testq %rcx, %rcx
-; SSSE3-NEXT:    setne %r13b
-; SSSE3-NEXT:    andb %dl, %r13b
+; SSSE3-NEXT:    setne %bl
+; SSSE3-NEXT:    andb %dl, %bl
 ; SSSE3-NEXT:    mulq %r15
 ; SSSE3-NEXT:    movq %rax, %rdi
 ; SSSE3-NEXT:    seto %bpl
 ; SSSE3-NEXT:    movq %r10, %rax
 ; SSSE3-NEXT:    mulq %r12
-; SSSE3-NEXT:    movq %rax, %rbx
 ; SSSE3-NEXT:    seto %cl
 ; SSSE3-NEXT:    orb %bpl, %cl
-; SSSE3-NEXT:    addq %rdi, %rbx
+; SSSE3-NEXT:    leaq (%rdi,%rax), %rbp
 ; SSSE3-NEXT:    movq %r12, %rax
 ; SSSE3-NEXT:    mulq %r15
 ; SSSE3-NEXT:    movq %rax, %r10
-; SSSE3-NEXT:    movq %rdx, %r15
-; SSSE3-NEXT:    addq %rbx, %r15
+; SSSE3-NEXT:    movq %rdx, %rdi
+; SSSE3-NEXT:    addq %rbp, %rdi
 ; SSSE3-NEXT:    setb %al
 ; SSSE3-NEXT:    orb %cl, %al
-; SSSE3-NEXT:    orb %r13b, %al
+; SSSE3-NEXT:    orb %bl, %al
 ; SSSE3-NEXT:    movzbl %al, %ebp
 ; SSSE3-NEXT:    testq %r9, %r9
 ; SSSE3-NEXT:    setne %al
 ; SSSE3-NEXT:    testq %rsi, %rsi
-; SSSE3-NEXT:    setne %r13b
-; SSSE3-NEXT:    andb %al, %r13b
+; SSSE3-NEXT:    setne %r12b
+; SSSE3-NEXT:    andb %al, %r12b
 ; SSSE3-NEXT:    movq %rsi, %rax
 ; SSSE3-NEXT:    mulq %r8
 ; SSSE3-NEXT:    movq %rax, %rsi
-; SSSE3-NEXT:    seto %r12b
+; SSSE3-NEXT:    seto %r15b
 ; SSSE3-NEXT:    movq %r9, %rax
 ; SSSE3-NEXT:    mulq %r11
-; SSSE3-NEXT:    movq %rax, %rdi
-; SSSE3-NEXT:    seto %bl
-; SSSE3-NEXT:    orb %r12b, %bl
-; SSSE3-NEXT:    addq %rsi, %rdi
+; SSSE3-NEXT:    seto %cl
+; SSSE3-NEXT:    orb %r15b, %cl
+; SSSE3-NEXT:    addq %rax, %rsi
 ; SSSE3-NEXT:    movq %r11, %rax
 ; SSSE3-NEXT:    mulq %r8
-; SSSE3-NEXT:    addq %rdi, %rdx
-; SSSE3-NEXT:    setb %cl
-; SSSE3-NEXT:    orb %bl, %cl
-; SSSE3-NEXT:    orb %r13b, %cl
-; SSSE3-NEXT:    movzbl %cl, %ecx
+; SSSE3-NEXT:    addq %rsi, %rdx
+; SSSE3-NEXT:    setb %bl
+; SSSE3-NEXT:    orb %cl, %bl
+; SSSE3-NEXT:    orb %r12b, %bl
+; SSSE3-NEXT:    movzbl %bl, %ecx
 ; SSSE3-NEXT:    movd %ecx, %xmm0
 ; SSSE3-NEXT:    pinsrw $4, %ebp, %xmm0
 ; SSSE3-NEXT:    movq %r10, 16(%r14)
 ; SSSE3-NEXT:    movq %rax, (%r14)
-; SSSE3-NEXT:    movq %r15, 24(%r14)
+; SSSE3-NEXT:    movq %rdi, 24(%r14)
 ; SSSE3-NEXT:    movq %rdx, 8(%r14)
 ; SSSE3-NEXT:    psllq $63, %xmm0
 ; SSSE3-NEXT:    psrad $31, %xmm0
 ; SSSE3-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,3,3]
 ; SSSE3-NEXT:    popq %rbx
 ; SSSE3-NEXT:    popq %r12
-; SSSE3-NEXT:    popq %r13
 ; SSSE3-NEXT:    popq %r14
 ; SSSE3-NEXT:    popq %r15
 ; SSSE3-NEXT:    popq %rbp
 ; SSSE3-NEXT:    retq
 ;
 ; SSE41-LABEL: umulo_v2i128:
 ; SSE41:       # %bb.0:
 ; SSE41-NEXT:    pushq %rbp
 ; SSE41-NEXT:    pushq %r15
 ; SSE41-NEXT:    pushq %r14
-; SSE41-NEXT:    pushq %r13
 ; SSE41-NEXT:    pushq %r12
 ; SSE41-NEXT:    pushq %rbx
 ; SSE41-NEXT:    movq %rcx, %rax
 ; SSE41-NEXT:    movq %rdx, %r12
 ; SSE41-NEXT:    movq %rdi, %r11
 ; SSE41-NEXT:    movq {{[0-9]+}}(%rsp), %r14
 ; SSE41-NEXT:    movq {{[0-9]+}}(%rsp), %r15
 ; SSE41-NEXT:    movq {{[0-9]+}}(%rsp), %r10
 ; SSE41-NEXT:    testq %r10, %r10
 ; SSE41-NEXT:    setne %dl
 ; SSE41-NEXT:    testq %rcx, %rcx
-; SSE41-NEXT:    setne %r13b
-; SSE41-NEXT:    andb %dl, %r13b
+; SSE41-NEXT:    setne %bl
+; SSE41-NEXT:    andb %dl, %bl
 ; SSE41-NEXT:    mulq %r15
 ; SSE41-NEXT:    movq %rax, %rdi
 ; SSE41-NEXT:    seto %bpl
 ; SSE41-NEXT:    movq %r10, %rax
 ; SSE41-NEXT:    mulq %r12
-; SSE41-NEXT:    movq %rax, %rbx
 ; SSE41-NEXT:    seto %cl
 ; SSE41-NEXT:    orb %bpl, %cl
-; SSE41-NEXT:    addq %rdi, %rbx
+; SSE41-NEXT:    leaq (%rdi,%rax), %rbp
 ; SSE41-NEXT:    movq %r12, %rax
 ; SSE41-NEXT:    mulq %r15
 ; SSE41-NEXT:    movq %rax, %r10
-; SSE41-NEXT:    movq %rdx, %r15
-; SSE41-NEXT:    addq %rbx, %r15
+; SSE41-NEXT:    movq %rdx, %rdi
+; SSE41-NEXT:    addq %rbp, %rdi
 ; SSE41-NEXT:    setb %al
 ; SSE41-NEXT:    orb %cl, %al
-; SSE41-NEXT:    orb %r13b, %al
+; SSE41-NEXT:    orb %bl, %al
 ; SSE41-NEXT:    movzbl %al, %ebp
 ; SSE41-NEXT:    testq %r9, %r9
 ; SSE41-NEXT:    setne %al
 ; SSE41-NEXT:    testq %rsi, %rsi
-; SSE41-NEXT:    setne %r13b
-; SSE41-NEXT:    andb %al, %r13b
+; SSE41-NEXT:    setne %r12b
+; SSE41-NEXT:    andb %al, %r12b
 ; SSE41-NEXT:    movq %rsi, %rax
 ; SSE41-NEXT:    mulq %r8
 ; SSE41-NEXT:    movq %rax, %rsi
-; SSE41-NEXT:    seto %r12b
+; SSE41-NEXT:    seto %r15b
 ; SSE41-NEXT:    movq %r9, %rax
 ; SSE41-NEXT:    mulq %r11
-; SSE41-NEXT:    movq %rax, %rdi
-; SSE41-NEXT:    seto %bl
-; SSE41-NEXT:    orb %r12b, %bl
-; SSE41-NEXT:    addq %rsi, %rdi
+; SSE41-NEXT:    seto %cl
+; SSE41-NEXT:    orb %r15b, %cl
+; SSE41-NEXT:    addq %rax, %rsi
 ; SSE41-NEXT:    movq %r11, %rax
 ; SSE41-NEXT:    mulq %r8
-; SSE41-NEXT:    addq %rdi, %rdx
-; SSE41-NEXT:    setb %cl
-; SSE41-NEXT:    orb %bl, %cl
-; SSE41-NEXT:    orb %r13b, %cl
-; SSE41-NEXT:    movzbl %cl, %ecx
+; SSE41-NEXT:    addq %rsi, %rdx
+; SSE41-NEXT:    setb %bl
+; SSE41-NEXT:    orb %cl, %bl
+; SSE41-NEXT:    orb %r12b, %bl
+; SSE41-NEXT:    movzbl %bl, %ecx
 ; SSE41-NEXT:    movd %ecx, %xmm0
 ; SSE41-NEXT:    pinsrb $8, %ebp, %xmm0
 ; SSE41-NEXT:    movq %r10, 16(%r14)
 ; SSE41-NEXT:    movq %rax, (%r14)
-; SSE41-NEXT:    movq %r15, 24(%r14)
+; SSE41-NEXT:    movq %rdi, 24(%r14)
 ; SSE41-NEXT:    movq %rdx, 8(%r14)
 ; SSE41-NEXT:    psllq $63, %xmm0
 ; SSE41-NEXT:    psrad $31, %xmm0
 ; SSE41-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,3,3]
 ; SSE41-NEXT:    popq %rbx
 ; SSE41-NEXT:    popq %r12
-; SSE41-NEXT:    popq %r13
 ; SSE41-NEXT:    popq %r14
 ; SSE41-NEXT:    popq %r15
 ; SSE41-NEXT:    popq %rbp
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: umulo_v2i128:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    pushq %rbp
 ; AVX1-NEXT:    pushq %r15
 ; AVX1-NEXT:    pushq %r14
-; AVX1-NEXT:    pushq %r13
 ; AVX1-NEXT:    pushq %r12
 ; AVX1-NEXT:    pushq %rbx
 ; AVX1-NEXT:    movq %rcx, %rax
 ; AVX1-NEXT:    movq %rdx, %r12
 ; AVX1-NEXT:    movq %rdi, %r11
 ; AVX1-NEXT:    movq {{[0-9]+}}(%rsp), %r14
 ; AVX1-NEXT:    movq {{[0-9]+}}(%rsp), %r15
 ; AVX1-NEXT:    movq {{[0-9]+}}(%rsp), %r10
 ; AVX1-NEXT:    testq %r10, %r10
 ; AVX1-NEXT:    setne %dl
 ; AVX1-NEXT:    testq %rcx, %rcx
-; AVX1-NEXT:    setne %r13b
-; AVX1-NEXT:    andb %dl, %r13b
+; AVX1-NEXT:    setne %bl
+; AVX1-NEXT:    andb %dl, %bl
 ; AVX1-NEXT:    mulq %r15
 ; AVX1-NEXT:    movq %rax, %rdi
 ; AVX1-NEXT:    seto %bpl
 ; AVX1-NEXT:    movq %r10, %rax
 ; AVX1-NEXT:    mulq %r12
-; AVX1-NEXT:    movq %rax, %rbx
 ; AVX1-NEXT:    seto %cl
 ; AVX1-NEXT:    orb %bpl, %cl
-; AVX1-NEXT:    addq %rdi, %rbx
+; AVX1-NEXT:    leaq (%rdi,%rax), %rbp
 ; AVX1-NEXT:    movq %r12, %rax
 ; AVX1-NEXT:    mulq %r15
 ; AVX1-NEXT:    movq %rax, %r10
-; AVX1-NEXT:    movq %rdx, %r15
-; AVX1-NEXT:    addq %rbx, %r15
+; AVX1-NEXT:    movq %rdx, %rdi
+; AVX1-NEXT:    addq %rbp, %rdi
 ; AVX1-NEXT:    setb %al
 ; AVX1-NEXT:    orb %cl, %al
-; AVX1-NEXT:    orb %r13b, %al
+; AVX1-NEXT:    orb %bl, %al
 ; AVX1-NEXT:    movzbl %al, %ebp
 ; AVX1-NEXT:    testq %r9, %r9
 ; AVX1-NEXT:    setne %al
 ; AVX1-NEXT:    testq %rsi, %rsi
-; AVX1-NEXT:    setne %r13b
-; AVX1-NEXT:    andb %al, %r13b
+; AVX1-NEXT:    setne %r12b
+; AVX1-NEXT:    andb %al, %r12b
 ; AVX1-NEXT:    movq %rsi, %rax
 ; AVX1-NEXT:    mulq %r8
 ; AVX1-NEXT:    movq %rax, %rsi
-; AVX1-NEXT:    seto %r12b
+; AVX1-NEXT:    seto %r15b
 ; AVX1-NEXT:    movq %r9, %rax
 ; AVX1-NEXT:    mulq %r11
-; AVX1-NEXT:    movq %rax, %rdi
 ; AVX1-NEXT:    seto %cl
-; AVX1-NEXT:    orb %r12b, %cl
-; AVX1-NEXT:    addq %rsi, %rdi
+; AVX1-NEXT:    orb %r15b, %cl
+; AVX1-NEXT:    addq %rax, %rsi
 ; AVX1-NEXT:    movq %r11, %rax
 ; AVX1-NEXT:    mulq %r8
-; AVX1-NEXT:    addq %rdi, %rdx
+; AVX1-NEXT:    addq %rsi, %rdx
 ; AVX1-NEXT:    setb %bl
 ; AVX1-NEXT:    orb %cl, %bl
-; AVX1-NEXT:    orb %r13b, %bl
+; AVX1-NEXT:    orb %r12b, %bl
 ; AVX1-NEXT:    movzbl %bl, %ecx
 ; AVX1-NEXT:    vmovd %ecx, %xmm0
 ; AVX1-NEXT:    vpinsrb $8, %ebp, %xmm0, %xmm0
 ; AVX1-NEXT:    movq %r10, 16(%r14)
 ; AVX1-NEXT:    movq %rax, (%r14)
-; AVX1-NEXT:    movq %r15, 24(%r14)
+; AVX1-NEXT:    movq %rdi, 24(%r14)
 ; AVX1-NEXT:    movq %rdx, 8(%r14)
 ; AVX1-NEXT:    vpsllq $63, %xmm0, %xmm0
 ; AVX1-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; AVX1-NEXT:    vpcmpgtq %xmm0, %xmm1, %xmm0
 ; AVX1-NEXT:    popq %rbx
 ; AVX1-NEXT:    popq %r12
-; AVX1-NEXT:    popq %r13
 ; AVX1-NEXT:    popq %r14
 ; AVX1-NEXT:    popq %r15
 ; AVX1-NEXT:    popq %rbp
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: umulo_v2i128:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    pushq %rbp
 ; AVX2-NEXT:    pushq %r15
 ; AVX2-NEXT:    pushq %r14
-; AVX2-NEXT:    pushq %r13
 ; AVX2-NEXT:    pushq %r12
 ; AVX2-NEXT:    pushq %rbx
 ; AVX2-NEXT:    movq %rcx, %rax
 ; AVX2-NEXT:    movq %rdx, %r12
 ; AVX2-NEXT:    movq %rdi, %r11
 ; AVX2-NEXT:    movq {{[0-9]+}}(%rsp), %r14
 ; AVX2-NEXT:    movq {{[0-9]+}}(%rsp), %r15
 ; AVX2-NEXT:    movq {{[0-9]+}}(%rsp), %r10
 ; AVX2-NEXT:    testq %r10, %r10
 ; AVX2-NEXT:    setne %dl
 ; AVX2-NEXT:    testq %rcx, %rcx
-; AVX2-NEXT:    setne %r13b
-; AVX2-NEXT:    andb %dl, %r13b
+; AVX2-NEXT:    setne %bl
+; AVX2-NEXT:    andb %dl, %bl
 ; AVX2-NEXT:    mulq %r15
 ; AVX2-NEXT:    movq %rax, %rdi
 ; AVX2-NEXT:    seto %bpl
 ; AVX2-NEXT:    movq %r10, %rax
 ; AVX2-NEXT:    mulq %r12
-; AVX2-NEXT:    movq %rax, %rbx
 ; AVX2-NEXT:    seto %cl
 ; AVX2-NEXT:    orb %bpl, %cl
-; AVX2-NEXT:    addq %rdi, %rbx
+; AVX2-NEXT:    leaq (%rdi,%rax), %rbp
 ; AVX2-NEXT:    movq %r12, %rax
 ; AVX2-NEXT:    mulq %r15
 ; AVX2-NEXT:    movq %rax, %r10
-; AVX2-NEXT:    movq %rdx, %r15
-; AVX2-NEXT:    addq %rbx, %r15
+; AVX2-NEXT:    movq %rdx, %rdi
+; AVX2-NEXT:    addq %rbp, %rdi
 ; AVX2-NEXT:    setb %al
 ; AVX2-NEXT:    orb %cl, %al
-; AVX2-NEXT:    orb %r13b, %al
+; AVX2-NEXT:    orb %bl, %al
 ; AVX2-NEXT:    movzbl %al, %ebp
 ; AVX2-NEXT:    testq %r9, %r9
 ; AVX2-NEXT:    setne %al
 ; AVX2-NEXT:    testq %rsi, %rsi
-; AVX2-NEXT:    setne %r13b
-; AVX2-NEXT:    andb %al, %r13b
+; AVX2-NEXT:    setne %r12b
+; AVX2-NEXT:    andb %al, %r12b
 ; AVX2-NEXT:    movq %rsi, %rax
 ; AVX2-NEXT:    mulq %r8
 ; AVX2-NEXT:    movq %rax, %rsi
-; AVX2-NEXT:    seto %r12b
+; AVX2-NEXT:    seto %r15b
 ; AVX2-NEXT:    movq %r9, %rax
 ; AVX2-NEXT:    mulq %r11
-; AVX2-NEXT:    movq %rax, %rdi
 ; AVX2-NEXT:    seto %cl
-; AVX2-NEXT:    orb %r12b, %cl
-; AVX2-NEXT:    addq %rsi, %rdi
+; AVX2-NEXT:    orb %r15b, %cl
+; AVX2-NEXT:    addq %rax, %rsi
 ; AVX2-NEXT:    movq %r11, %rax
 ; AVX2-NEXT:    mulq %r8
-; AVX2-NEXT:    addq %rdi, %rdx
+; AVX2-NEXT:    addq %rsi, %rdx
 ; AVX2-NEXT:    setb %bl
 ; AVX2-NEXT:    orb %cl, %bl
-; AVX2-NEXT:    orb %r13b, %bl
+; AVX2-NEXT:    orb %r12b, %bl
 ; AVX2-NEXT:    movzbl %bl, %ecx
 ; AVX2-NEXT:    vmovd %ecx, %xmm0
 ; AVX2-NEXT:    vpinsrb $8, %ebp, %xmm0, %xmm0
 ; AVX2-NEXT:    movq %r10, 16(%r14)
 ; AVX2-NEXT:    movq %rax, (%r14)
-; AVX2-NEXT:    movq %r15, 24(%r14)
+; AVX2-NEXT:    movq %rdi, 24(%r14)
 ; AVX2-NEXT:    movq %rdx, 8(%r14)
 ; AVX2-NEXT:    vpsllq $63, %xmm0, %xmm0
 ; AVX2-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; AVX2-NEXT:    vpcmpgtq %xmm0, %xmm1, %xmm0
 ; AVX2-NEXT:    popq %rbx
 ; AVX2-NEXT:    popq %r12
-; AVX2-NEXT:    popq %r13
 ; AVX2-NEXT:    popq %r14
 ; AVX2-NEXT:    popq %r15
 ; AVX2-NEXT:    popq %rbp
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: umulo_v2i128:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    pushq %r15
 ; AVX512-NEXT:    pushq %r14
-; AVX512-NEXT:    pushq %r13
 ; AVX512-NEXT:    pushq %r12
 ; AVX512-NEXT:    pushq %rbx
 ; AVX512-NEXT:    movq %rcx, %rax
 ; AVX512-NEXT:    movq %rdx, %r12
 ; AVX512-NEXT:    movq %rdi, %r11
 ; AVX512-NEXT:    movq {{[0-9]+}}(%rsp), %r14
 ; AVX512-NEXT:    movq {{[0-9]+}}(%rsp), %r15
 ; AVX512-NEXT:    movq {{[0-9]+}}(%rsp), %r10
 ; AVX512-NEXT:    testq %r10, %r10
 ; AVX512-NEXT:    setne %dl
 ; AVX512-NEXT:    testq %rcx, %rcx
-; AVX512-NEXT:    setne %r13b
-; AVX512-NEXT:    andb %dl, %r13b
+; AVX512-NEXT:    setne %bl
+; AVX512-NEXT:    andb %dl, %bl
 ; AVX512-NEXT:    mulq %r15
 ; AVX512-NEXT:    movq %rax, %rdi
 ; AVX512-NEXT:    seto %bpl
 ; AVX512-NEXT:    movq %r10, %rax
 ; AVX512-NEXT:    mulq %r12
-; AVX512-NEXT:    movq %rax, %rbx
 ; AVX512-NEXT:    seto %cl
 ; AVX512-NEXT:    orb %bpl, %cl
-; AVX512-NEXT:    addq %rdi, %rbx
+; AVX512-NEXT:    leaq (%rdi,%rax), %rbp
 ; AVX512-NEXT:    movq %r12, %rax
 ; AVX512-NEXT:    mulq %r15
 ; AVX512-NEXT:    movq %rax, %r10
-; AVX512-NEXT:    movq %rdx, %r15
-; AVX512-NEXT:    addq %rbx, %r15
+; AVX512-NEXT:    movq %rdx, %rdi
+; AVX512-NEXT:    addq %rbp, %rdi
 ; AVX512-NEXT:    setb %al
 ; AVX512-NEXT:    orb %cl, %al
-; AVX512-NEXT:    orb %r13b, %al
+; AVX512-NEXT:    orb %bl, %al
 ; AVX512-NEXT:    kmovd %eax, %k0
 ; AVX512-NEXT:    kshiftlw $1, %k0, %k0
 ; AVX512-NEXT:    testq %r9, %r9
 ; AVX512-NEXT:    setne %al
 ; AVX512-NEXT:    testq %rsi, %rsi
 ; AVX512-NEXT:    setne %cl
 ; AVX512-NEXT:    andb %al, %cl
 ; AVX512-NEXT:    movq %rsi, %rax
 ; AVX512-NEXT:    mulq %r8
 ; AVX512-NEXT:    movq %rax, %rsi
 ; AVX512-NEXT:    seto %bpl
 ; AVX512-NEXT:    movq %r9, %rax
 ; AVX512-NEXT:    mulq %r11
-; AVX512-NEXT:    movq %rax, %rdi
 ; AVX512-NEXT:    seto %bl
 ; AVX512-NEXT:    orb %bpl, %bl
-; AVX512-NEXT:    addq %rsi, %rdi
+; AVX512-NEXT:    addq %rax, %rsi
 ; AVX512-NEXT:    movq %r11, %rax
 ; AVX512-NEXT:    mulq %r8
-; AVX512-NEXT:    addq %rdi, %rdx
+; AVX512-NEXT:    addq %rsi, %rdx
 ; AVX512-NEXT:    setb %sil
 ; AVX512-NEXT:    orb %bl, %sil
 ; AVX512-NEXT:    orb %cl, %sil
 ; AVX512-NEXT:    andl $1, %esi
 ; AVX512-NEXT:    kmovw %esi, %k1
 ; AVX512-NEXT:    korw %k0, %k1, %k1
 ; AVX512-NEXT:    movq %r10, 16(%r14)
 ; AVX512-NEXT:    movq %rax, (%r14)
-; AVX512-NEXT:    movq %r15, 24(%r14)
+; AVX512-NEXT:    movq %rdi, 24(%r14)
 ; AVX512-NEXT:    movq %rdx, 8(%r14)
 ; AVX512-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
 ; AVX512-NEXT:    vmovdqa64 %xmm0, %xmm0 {%k1} {z}
 ; AVX512-NEXT:    popq %rbx
 ; AVX512-NEXT:    popq %r12
-; AVX512-NEXT:    popq %r13
 ; AVX512-NEXT:    popq %r14
 ; AVX512-NEXT:    popq %r15
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %t = call {<2 x i128>, <2 x i1>} @llvm.umul.with.overflow.v2i128(<2 x i128> %a0, <2 x i128> %a1)
   %val = extractvalue {<2 x i128>, <2 x i1>} %t, 0
   %obit = extractvalue {<2 x i128>, <2 x i1>} %t, 1
   %res = sext <2 x i1> %obit to <2 x i32>
   store <2 x i128> %val, <2 x i128>* %p2
   ret <2 x i32> %res
 }