[X86][FastISel] Remove with.overflow + select fold (PR54369)

[nikic]

Summary

This is an alternative to D122825. Instead of trying to account for possible EFLAGS clobber by constant materialization, remove the with.overflow + select fold entirely, on the premise that non-trivial/fragile folds like this don't belong in FastISel. I've kept the with.overflow + br fold for now, as we don't have any known issues for that one.

Fixes https://github.com/llvm/llvm-project/issues/54369.

[RKSimon]

Please can we see the effect of removing the with.overflow + br case and the foldX86XALUIntrinsic method entirely?

[nikic]

@RKSimon Here's how dropping it entirely looks like: https://gist.github.com/nikic/9e59e8e7b635a54f0d75aad89b2632d6

[RKSimon]

In D123014#3429542, @nikic wrote:

@RKSimon Here's how dropping it entirely looks like: https://gist.github.com/nikic/9e59e8e7b635a54f0d75aad89b2632d6

I think I prefer getting rid of it entirely, but I'd accept just this patch if others want to keep some eflags folding behaviour.

@pengfei @craig.topper any thoughts?

[pengfei]

In D123014#3435488, @RKSimon wrote:

In D123014#3429542, @nikic wrote:

@RKSimon Here's how dropping it entirely looks like: https://gist.github.com/nikic/9e59e8e7b635a54f0d75aad89b2632d6

I think I prefer getting rid of it entirely, but I'd accept just this patch if others want to keep some eflags folding behaviour.

@pengfei @craig.topper any thoughts?

If I read the patches correct, this patch and further removing the whole folding are just for theoretical potential issues but we cannot think out for now, right?
Given we don't do much optimizations with FastISel (which means it's rare we meet another fail in future), I prefer to D122825 (maybe with comments for future information). Furthermore, I think we can even check if the constant is 0.

[RKSimon]

OK, I've no strong objections to D122825 instead if there's a motivation to retain fast-isel optimizations

[craig.topper]

In D123014#3435763, @pengfei wrote:

In D123014#3435488, @RKSimon wrote:

In D123014#3429542, @nikic wrote:

@RKSimon Here's how dropping it entirely looks like: https://gist.github.com/nikic/9e59e8e7b635a54f0d75aad89b2632d6

I think I prefer getting rid of it entirely, but I'd accept just this patch if others want to keep some eflags folding behaviour.

@pengfei @craig.topper any thoughts?

If I read the patches correct, this patch and further removing the whole folding are just for theoretical potential issues but we cannot think out for now, right?
Given we don't do much optimizations with FastISel (which means it's rare we meet another fail in future), I prefer to D122825 (maybe with comments for future information). Furthermore, I think we can even check if the constant is 0.

You can't check if the constant is 0. It could be a ConstantExpr that includes a use of a 0. The ConstantExpr will get expanded to individual instructions and that might create a mov of 0. It could also expand to an expression that includes an add or other instruction which will also clobber the flags. For example, this test case is broken today because it creates an add instruction between the overflow add and the cmov.

@a = global i32 0, align 4

define i64 @adder(i64 %lhs, i64 %rhs) {
        %res = call { i64, i1 } @llvm.sadd.with.overflow.i64(i64 %lhs, i64 %rhs)
        %errorbit = extractvalue { i64, i1 } %res, 1
        %errorval = select i1 %errorbit, i64 148, i64 add (i64 ptrtoint (i32* @a to i64), i64 5)
        ret i64 %errorval
}

declare { i64, i1 } @llvm.sadd.with.overflow.i64(i64 %a, i64 %b)

[pengfei]

In D123014#3436522, @craig.topper wrote:

In D123014#3435763, @pengfei wrote:

In D123014#3435488, @RKSimon wrote:

In D123014#3429542, @nikic wrote:

@RKSimon Here's how dropping it entirely looks like: https://gist.github.com/nikic/9e59e8e7b635a54f0d75aad89b2632d6

I think I prefer getting rid of it entirely, but I'd accept just this patch if others want to keep some eflags folding behaviour.

@pengfei @craig.topper any thoughts?

If I read the patches correct, this patch and further removing the whole folding are just for theoretical potential issues but we cannot think out for now, right?
Given we don't do much optimizations with FastISel (which means it's rare we meet another fail in future), I prefer to D122825 (maybe with comments for future information). Furthermore, I think we can even check if the constant is 0.

You can't check if the constant is 0. It could be a ConstantExpr that includes a use of a 0. The ConstantExpr will get expanded to individual instructions and that might create a mov of 0. It could also expand to an expression that includes an add or other instruction which will also clobber the flags. For example, this test case is broken today because it creates an add instruction between the overflow add and the cmov.

@a = global i32 0, align 4

define i64 @adder(i64 %lhs, i64 %rhs) {
        %res = call { i64, i1 } @llvm.sadd.with.overflow.i64(i64 %lhs, i64 %rhs)
        %errorbit = extractvalue { i64, i1 } %res, 1
        %errorval = select i1 %errorbit, i64 148, i64 add (i64 ptrtoint (i32* @a to i64), i64 5)
        ret i64 %errorval
}

declare { i64, i1 } @llvm.sadd.with.overflow.i64(i64 %a, i64 %b)

Thanks Craig! Good to know we are allowing such IR.

[nikic]

In D123014#3436522, @craig.topper wrote:

In D123014#3435763, @pengfei wrote:

In D123014#3435488, @RKSimon wrote:

In D123014#3429542, @nikic wrote:

@RKSimon Here's how dropping it entirely looks like: https://gist.github.com/nikic/9e59e8e7b635a54f0d75aad89b2632d6

I think I prefer getting rid of it entirely, but I'd accept just this patch if others want to keep some eflags folding behaviour.

@pengfei @craig.topper any thoughts?

If I read the patches correct, this patch and further removing the whole folding are just for theoretical potential issues but we cannot think out for now, right?
Given we don't do much optimizations with FastISel (which means it's rare we meet another fail in future), I prefer to D122825 (maybe with comments for future information). Furthermore, I think we can even check if the constant is 0.

You can't check if the constant is 0. It could be a ConstantExpr that includes a use of a 0. The ConstantExpr will get expanded to individual instructions and that might create a mov of 0. It could also expand to an expression that includes an add or other instruction which will also clobber the flags. For example, this test case is broken today because it creates an add instruction between the overflow add and the cmov.

@a = global i32 0, align 4

define i64 @adder(i64 %lhs, i64 %rhs) {
        %res = call { i64, i1 } @llvm.sadd.with.overflow.i64(i64 %lhs, i64 %rhs)
        %errorbit = extractvalue { i64, i1 } %res, 1
        %errorval = select i1 %errorbit, i64 148, i64 add (i64 ptrtoint (i32* @a to i64), i64 5)
        ret i64 %errorval
}

declare { i64, i1 } @llvm.sadd.with.overflow.i64(i64 %a, i64 %b)

Thanks, this is a nice additional test case. I've committed it in https://github.com/llvm/llvm-project/commit/8ae33cb300262738d9798edb612c7431612f56f6. (Either approach would fix that one as well.)

[pengfei]

In D123014#3438410, @nikic wrote:

In D123014#3436522, @craig.topper wrote:

In D123014#3435763, @pengfei wrote:

In D123014#3435488, @RKSimon wrote:

In D123014#3429542, @nikic wrote:

@RKSimon Here's how dropping it entirely looks like: https://gist.github.com/nikic/9e59e8e7b635a54f0d75aad89b2632d6

I think I prefer getting rid of it entirely, but I'd accept just this patch if others want to keep some eflags folding behaviour.

@pengfei @craig.topper any thoughts?

If I read the patches correct, this patch and further removing the whole folding are just for theoretical potential issues but we cannot think out for now, right?
Given we don't do much optimizations with FastISel (which means it's rare we meet another fail in future), I prefer to D122825 (maybe with comments for future information). Furthermore, I think we can even check if the constant is 0.

You can't check if the constant is 0. It could be a ConstantExpr that includes a use of a 0. The ConstantExpr will get expanded to individual instructions and that might create a mov of 0. It could also expand to an expression that includes an add or other instruction which will also clobber the flags. For example, this test case is broken today because it creates an add instruction between the overflow add and the cmov.

@a = global i32 0, align 4

define i64 @adder(i64 %lhs, i64 %rhs) {
        %res = call { i64, i1 } @llvm.sadd.with.overflow.i64(i64 %lhs, i64 %rhs)
        %errorbit = extractvalue { i64, i1 } %res, 1
        %errorval = select i1 %errorbit, i64 148, i64 add (i64 ptrtoint (i32* @a to i64), i64 5)
        ret i64 %errorval
}

declare { i64, i1 } @llvm.sadd.with.overflow.i64(i64 %a, i64 %b)

Thanks, this is a nice additional test case. I've committed it in https://github.com/llvm/llvm-project/commit/8ae33cb300262738d9798edb612c7431612f56f6. (Either approach would fix that one as well.)

Does D122825 still work if we don't have any constant operand? https://godbolt.org/z/5r6TvGxaP

[nikic]

In D123014#3438560, @pengfei wrote:

Does D122825 still work if we don't have any constant operand? https://godbolt.org/z/5r6TvGxaP

Yes: Constant expressions are still Constant, just not ConstantInt.

[pengfei]

In D123014#3438645, @nikic wrote:

In D123014#3438560, @pengfei wrote:

Does D122825 still work if we don't have any constant operand? https://godbolt.org/z/5r6TvGxaP

Yes: Constant expressions are still Constant, just not ConstantInt.

Got it. Thanks!

[nikic]

Abandoning in favor of D122825.

llvm/lib/Target/X86/X86FastISel.cpp

if (SETFOpc) {
         BuildMI(*FuncInfo.MBB, FuncInfo.InsertPt, DbgLoc, II, TmpReg)
           .addReg(FlagReg2).addReg(FlagReg1);
       } else {
         BuildMI(*FuncInfo.MBB, FuncInfo.InsertPt, DbgLoc, II)
           .addReg(FlagReg2).addReg(FlagReg1);
       }
     }
     NeedTest = false;
-  } else if (foldX86XALUIntrinsic(CC, I, Cond)) {
-    // Fake request the condition, otherwise the intrinsic might be completely
-    // optimized away.
-    Register TmpReg = getRegForValue(Cond);
-    if (TmpReg == 0)
-      return false;
-
-    NeedTest = false;
   }
 
   if (NeedTest) {
     // Selects operate on i1, however, CondReg is 8 bits width and may contain
     // garbage. Indeed, only the less significant bit is supposed to be
     // accurate. If we read more than the lsb, we may see non-zero values
     // whereas lsb is zero. Therefore, we have to truncate Op0Reg to i1 for
     // the select. This is achieved by performing TEST against 1.

llvm/test/CodeGen/X86/pr54369.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=x86_64-- -O0 < %s | FileCheck %s
 
-; FIXME: This is currently miscompiled due to an eflags clobber.
 define i64 @adder(i64 %lhs, i64 %rhs) {
 ; CHECK-LABEL: adder:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    addq %rsi, %rdi
-; CHECK-NEXT:    seto %al
+; CHECK-NEXT:    seto %dl
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    # kill: def $rax killed $eax
 ; CHECK-NEXT:    movl $148, %ecx
-; CHECK-NEXT:    cmovoq %rcx, %rax
+; CHECK-NEXT:    testb $1, %dl
+; CHECK-NEXT:    cmovneq %rcx, %rax
 ; CHECK-NEXT:    retq
   %res = call { i64, i1 } @llvm.sadd.with.overflow.i64(i64 %lhs, i64 %rhs)
   %errorbit = extractvalue { i64, i1 } %res, 1
   %errorval = select i1 %errorbit, i64 148, i64 0
   ret i64 %errorval
 }
 
 declare { i64, i1 } @llvm.sadd.with.overflow.i64(i64 %a, i64 %b)

llvm/test/CodeGen/X86/xaluo.ll

 ; SDAG-NEXT:    cmovol %edi, %eax
 ; SDAG-NEXT:    retq
 ;
 ; FAST-LABEL: saddoselecti32:
 ; FAST:       ## %bb.0:
 ; FAST-NEXT:    movl %esi, %eax
 ; FAST-NEXT:    movl %edi, %ecx
 ; FAST-NEXT:    addl %esi, %ecx
-; FAST-NEXT:    cmovol %edi, %eax
+; FAST-NEXT:    seto %cl
+; FAST-NEXT:    testb $1, %cl
+; FAST-NEXT:    cmovnel %edi, %eax
 ; FAST-NEXT:    retq
   %t = call {i32, i1} @llvm.sadd.with.overflow.i32(i32 %v1, i32 %v2)
   %obit = extractvalue {i32, i1} %t, 1
   %ret = select i1 %obit, i32 %v1, i32 %v2
   ret i32 %ret
 }
 
 define i64 @saddoselecti64(i64 %v1, i64 %v2) {
 ; SDAG-LABEL: saddoselecti64:
 ; SDAG:       ## %bb.0:
 ; SDAG-NEXT:    movq %rsi, %rax
 ; SDAG-NEXT:    movq %rdi, %rcx
 ; SDAG-NEXT:    addq %rsi, %rcx
 ; SDAG-NEXT:    cmovoq %rdi, %rax
 ; SDAG-NEXT:    retq
 ;
 ; FAST-LABEL: saddoselecti64:
 ; FAST:       ## %bb.0:
 ; FAST-NEXT:    movq %rsi, %rax
 ; FAST-NEXT:    movq %rdi, %rcx
 ; FAST-NEXT:    addq %rsi, %rcx
-; FAST-NEXT:    cmovoq %rdi, %rax
+; FAST-NEXT:    seto %cl
+; FAST-NEXT:    testb $1, %cl
+; FAST-NEXT:    cmovneq %rdi, %rax
 ; FAST-NEXT:    retq
   %t = call {i64, i1} @llvm.sadd.with.overflow.i64(i64 %v1, i64 %v2)
   %obit = extractvalue {i64, i1} %t, 1
   %ret = select i1 %obit, i64 %v1, i64 %v2
   ret i64 %ret
 }
 
 define i32 @uaddoselecti32(i32 %v1, i32 %v2) {
 ; SDAG-LABEL: uaddoselecti32:
 ; SDAG:       ## %bb.0:
 ; SDAG-NEXT:    movl %esi, %eax
 ; SDAG-NEXT:    movl %edi, %ecx
 ; SDAG-NEXT:    addl %esi, %ecx
 ; SDAG-NEXT:    cmovbl %edi, %eax
 ; SDAG-NEXT:    retq
 ;
 ; FAST-LABEL: uaddoselecti32:
 ; FAST:       ## %bb.0:
 ; FAST-NEXT:    movl %esi, %eax
 ; FAST-NEXT:    movl %edi, %ecx
 ; FAST-NEXT:    addl %esi, %ecx
-; FAST-NEXT:    cmovbl %edi, %eax
+; FAST-NEXT:    setb %cl
+; FAST-NEXT:    testb $1, %cl
+; FAST-NEXT:    cmovnel %edi, %eax
 ; FAST-NEXT:    retq
   %t = call {i32, i1} @llvm.uadd.with.overflow.i32(i32 %v1, i32 %v2)
   %obit = extractvalue {i32, i1} %t, 1
   %ret = select i1 %obit, i32 %v1, i32 %v2
   ret i32 %ret
 }
 
 define i64 @uaddoselecti64(i64 %v1, i64 %v2) {
 ; SDAG-LABEL: uaddoselecti64:
 ; SDAG:       ## %bb.0:
 ; SDAG-NEXT:    movq %rsi, %rax
 ; SDAG-NEXT:    movq %rdi, %rcx
 ; SDAG-NEXT:    addq %rsi, %rcx
 ; SDAG-NEXT:    cmovbq %rdi, %rax
 ; SDAG-NEXT:    retq
 ;
 ; FAST-LABEL: uaddoselecti64:
 ; FAST:       ## %bb.0:
 ; FAST-NEXT:    movq %rsi, %rax
 ; FAST-NEXT:    movq %rdi, %rcx
 ; FAST-NEXT:    addq %rsi, %rcx
-; FAST-NEXT:    cmovbq %rdi, %rax
+; FAST-NEXT:    setb %cl
+; FAST-NEXT:    testb $1, %cl
+; FAST-NEXT:    cmovneq %rdi, %rax
 ; FAST-NEXT:    retq
   %t = call {i64, i1} @llvm.uadd.with.overflow.i64(i64 %v1, i64 %v2)
   %obit = extractvalue {i64, i1} %t, 1
   %ret = select i1 %obit, i64 %v1, i64 %v2
   ret i64 %ret
 }
 
 define i32 @ssuboselecti32(i32 %v1, i32 %v2) {
 ; SDAG-LABEL: ssuboselecti32:
 ; SDAG:       ## %bb.0:
 ; SDAG-NEXT:    movl %esi, %eax
 ; SDAG-NEXT:    cmpl %esi, %edi
 ; SDAG-NEXT:    cmovol %edi, %eax
 ; SDAG-NEXT:    retq
 ;
 ; FAST-LABEL: ssuboselecti32:
 ; FAST:       ## %bb.0:
 ; FAST-NEXT:    movl %esi, %eax
 ; FAST-NEXT:    cmpl %esi, %edi
-; FAST-NEXT:    cmovol %edi, %eax
+; FAST-NEXT:    seto %cl
+; FAST-NEXT:    testb $1, %cl
+; FAST-NEXT:    cmovnel %edi, %eax
 ; FAST-NEXT:    retq
   %t = call {i32, i1} @llvm.ssub.with.overflow.i32(i32 %v1, i32 %v2)
   %obit = extractvalue {i32, i1} %t, 1
   %ret = select i1 %obit, i32 %v1, i32 %v2
   ret i32 %ret
 }
 
 define i64 @ssuboselecti64(i64 %v1, i64 %v2) {
 ; SDAG-LABEL: ssuboselecti64:
 ; SDAG:       ## %bb.0:
 ; SDAG-NEXT:    movq %rsi, %rax
 ; SDAG-NEXT:    cmpq %rsi, %rdi
 ; SDAG-NEXT:    cmovoq %rdi, %rax
 ; SDAG-NEXT:    retq
 ;
 ; FAST-LABEL: ssuboselecti64:
 ; FAST:       ## %bb.0:
 ; FAST-NEXT:    movq %rsi, %rax
 ; FAST-NEXT:    cmpq %rsi, %rdi
-; FAST-NEXT:    cmovoq %rdi, %rax
+; FAST-NEXT:    seto %cl
+; FAST-NEXT:    testb $1, %cl
+; FAST-NEXT:    cmovneq %rdi, %rax
 ; FAST-NEXT:    retq
   %t = call {i64, i1} @llvm.ssub.with.overflow.i64(i64 %v1, i64 %v2)
   %obit = extractvalue {i64, i1} %t, 1
   %ret = select i1 %obit, i64 %v1, i64 %v2
   ret i64 %ret
 }
 
 define i32 @usuboselecti32(i32 %v1, i32 %v2) {
 ; SDAG-LABEL: usuboselecti32:
 ; SDAG:       ## %bb.0:
 ; SDAG-NEXT:    movl %esi, %eax
 ; SDAG-NEXT:    cmpl %esi, %edi
 ; SDAG-NEXT:    cmovbl %edi, %eax
 ; SDAG-NEXT:    retq
 ;
 ; FAST-LABEL: usuboselecti32:
 ; FAST:       ## %bb.0:
 ; FAST-NEXT:    movl %esi, %eax
 ; FAST-NEXT:    cmpl %esi, %edi
-; FAST-NEXT:    cmovbl %edi, %eax
+; FAST-NEXT:    setb %cl
+; FAST-NEXT:    testb $1, %cl
+; FAST-NEXT:    cmovnel %edi, %eax
 ; FAST-NEXT:    retq
   %t = call {i32, i1} @llvm.usub.with.overflow.i32(i32 %v1, i32 %v2)
   %obit = extractvalue {i32, i1} %t, 1
   %ret = select i1 %obit, i32 %v1, i32 %v2
   ret i32 %ret
 }
 
 define i64 @usuboselecti64(i64 %v1, i64 %v2) {
 ; SDAG-LABEL: usuboselecti64:
 ; SDAG:       ## %bb.0:
 ; SDAG-NEXT:    movq %rsi, %rax
 ; SDAG-NEXT:    cmpq %rsi, %rdi
 ; SDAG-NEXT:    cmovbq %rdi, %rax
 ; SDAG-NEXT:    retq
 ;
 ; FAST-LABEL: usuboselecti64:
 ; FAST:       ## %bb.0:
 ; FAST-NEXT:    movq %rsi, %rax
 ; FAST-NEXT:    cmpq %rsi, %rdi
-; FAST-NEXT:    cmovbq %rdi, %rax
+; FAST-NEXT:    setb %cl
+; FAST-NEXT:    testb $1, %cl
+; FAST-NEXT:    cmovneq %rdi, %rax
 ; FAST-NEXT:    retq
   %t = call {i64, i1} @llvm.usub.with.overflow.i64(i64 %v1, i64 %v2)
   %obit = extractvalue {i64, i1} %t, 1
   %ret = select i1 %obit, i64 %v1, i64 %v2
   ret i64 %ret
 }
 
 ;

 ; SDAG-NEXT:    incl %ecx
 ; SDAG-NEXT:    cmovol %edi, %eax
 ; SDAG-NEXT:    movl %ecx, (%rdx)
 ; SDAG-NEXT:    retq
 ;
 ; FAST-LABEL: incovfselectstore:
 ; FAST:       ## %bb.0:
 ; FAST-NEXT:    movl %esi, %eax
-; FAST-NEXT:    movl %edi, %ecx
-; FAST-NEXT:    incl %ecx
-; FAST-NEXT:    cmovol %edi, %eax
-; FAST-NEXT:    movl %ecx, (%rdx)
+; FAST-NEXT:    movl %edi, %esi
+; FAST-NEXT:    incl %esi
+; FAST-NEXT:    seto %cl
+; FAST-NEXT:    testb $1, %cl
+; FAST-NEXT:    cmovnel %edi, %eax
+; FAST-NEXT:    movl %esi, (%rdx)
 ; FAST-NEXT:    retq
   %t = call {i32, i1} @llvm.sadd.with.overflow.i32(i32 %v1, i32 1)
   %obit = extractvalue {i32, i1} %t, 1
   %ret = select i1 %obit, i32 %v1, i32 %v2
   %val = extractvalue {i32, i1} %t, 0
   store i32 %val, i32* %x
   ret i32 %ret
 }
 
 ; Make sure we select a DEC for both the data use and the flag use.
 define i32 @decovfselectstore(i32 %v1, i32 %v2, i32* %x) {
 ; GENERIC-LABEL: decovfselectstore:
 ; GENERIC:       ## %bb.0:
 ; GENERIC-NEXT:    movl %esi, %eax
 ; GENERIC-NEXT:    movl %edi, %ecx
 ; GENERIC-NEXT:    decl %ecx
 ; GENERIC-NEXT:    cmovol %edi, %eax
 ; GENERIC-NEXT:    movl %ecx, (%rdx)
 ; GENERIC-NEXT:    retq
 ;
 ; FAST-LABEL: decovfselectstore:
 ; FAST:       ## %bb.0:
 ; FAST-NEXT:    movl %esi, %eax
-; FAST-NEXT:    movl %edi, %ecx
-; FAST-NEXT:    decl %ecx
-; FAST-NEXT:    cmovol %edi, %eax
-; FAST-NEXT:    movl %ecx, (%rdx)
+; FAST-NEXT:    movl %edi, %esi
+; FAST-NEXT:    decl %esi
+; FAST-NEXT:    seto %cl
+; FAST-NEXT:    testb $1, %cl
+; FAST-NEXT:    cmovnel %edi, %eax
+; FAST-NEXT:    movl %esi, (%rdx)
 ; FAST-NEXT:    retq
 ;
 ; KNL-LABEL: decovfselectstore:
 ; KNL:       ## %bb.0:
 ; KNL-NEXT:    movl %esi, %eax
 ; KNL-NEXT:    movl %edi, %ecx
 ; KNL-NEXT:    addl $-1, %ecx
 ; KNL-NEXT:    cmovol %edi, %eax

llvm/test/CodeGen/X86/xmulo.ll

; WIN32-NEXT:    retl
   store i64 %val, i64* %res
   ret i1 %obit
 }
 
 ;
 ; Check the use of the overflow bit in combination with a select instruction.
 ;
 define i32 @smuloselecti32(i32 %v1, i32 %v2) {
-; LINUX-LABEL: smuloselecti32:
-; LINUX:       # %bb.0:
-; LINUX-NEXT:    movl %esi, %eax
-; LINUX-NEXT:    movl %edi, %ecx
-; LINUX-NEXT:    imull %esi, %ecx
-; LINUX-NEXT:    cmovol %edi, %eax
-; LINUX-NEXT:    retq
+; SDAG-LABEL: smuloselecti32:
+; SDAG:       # %bb.0:
+; SDAG-NEXT:    movl %esi, %eax
+; SDAG-NEXT:    movl %edi, %ecx
+; SDAG-NEXT:    imull %esi, %ecx
+; SDAG-NEXT:    cmovol %edi, %eax
+; SDAG-NEXT:    retq
+;
+; FAST-LABEL: smuloselecti32:
+; FAST:       # %bb.0:
+; FAST-NEXT:    movl %esi, %eax
+; FAST-NEXT:    movl %edi, %ecx
+; FAST-NEXT:    imull %esi, %ecx
+; FAST-NEXT:    seto %cl
+; FAST-NEXT:    testb $1, %cl
+; FAST-NEXT:    cmovnel %edi, %eax
+; FAST-NEXT:    retq
 ;
 ; WIN64-LABEL: smuloselecti32:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    movl %edx, %eax
 ; WIN64-NEXT:    movl %ecx, %edx
 ; WIN64-NEXT:    imull %eax, %edx
 ; WIN64-NEXT:    cmovol %ecx, %eax
 ; WIN64-NEXT:    retq

 ; WIN32-NEXT:    retl
   %t = call {i32, i1} @llvm.smul.with.overflow.i32(i32 %v1, i32 %v2)
   %obit = extractvalue {i32, i1} %t, 1
   %ret = select i1 %obit, i32 %v1, i32 %v2
   ret i32 %ret
 }
 
 define i64 @smuloselecti64(i64 %v1, i64 %v2) {
-; LINUX-LABEL: smuloselecti64:
-; LINUX:       # %bb.0:
-; LINUX-NEXT:    movq %rsi, %rax
-; LINUX-NEXT:    movq %rdi, %rcx
-; LINUX-NEXT:    imulq %rsi, %rcx
-; LINUX-NEXT:    cmovoq %rdi, %rax
-; LINUX-NEXT:    retq
+; SDAG-LABEL: smuloselecti64:
+; SDAG:       # %bb.0:
+; SDAG-NEXT:    movq %rsi, %rax
+; SDAG-NEXT:    movq %rdi, %rcx
+; SDAG-NEXT:    imulq %rsi, %rcx
+; SDAG-NEXT:    cmovoq %rdi, %rax
+; SDAG-NEXT:    retq
+;
+; FAST-LABEL: smuloselecti64:
+; FAST:       # %bb.0:
+; FAST-NEXT:    movq %rsi, %rax
+; FAST-NEXT:    movq %rdi, %rcx
+; FAST-NEXT:    imulq %rsi, %rcx
+; FAST-NEXT:    seto %cl
+; FAST-NEXT:    testb $1, %cl
+; FAST-NEXT:    cmovneq %rdi, %rax
+; FAST-NEXT:    retq
 ;
 ; WIN64-LABEL: smuloselecti64:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    movq %rdx, %rax
 ; WIN64-NEXT:    movq %rcx, %rdx
 ; WIN64-NEXT:    imulq %rax, %rdx
 ; WIN64-NEXT:    cmovoq %rcx, %rax
 ; WIN64-NEXT:    retq

 ; WIN32-NEXT:    retl
   %t = call {i64, i1} @llvm.smul.with.overflow.i64(i64 %v1, i64 %v2)
   %obit = extractvalue {i64, i1} %t, 1
   %ret = select i1 %obit, i64 %v1, i64 %v2
   ret i64 %ret
 }
 
 define i32 @umuloselecti32(i32 %v1, i32 %v2) {
-; LINUX-LABEL: umuloselecti32:
-; LINUX:       # %bb.0:
-; LINUX-NEXT:    movl %edi, %eax
-; LINUX-NEXT:    mull %esi
-; LINUX-NEXT:    cmovol %edi, %esi
-; LINUX-NEXT:    movl %esi, %eax
-; LINUX-NEXT:    retq
+; SDAG-LABEL: umuloselecti32:
+; SDAG:       # %bb.0:
+; SDAG-NEXT:    movl %edi, %eax
+; SDAG-NEXT:    mull %esi
+; SDAG-NEXT:    cmovol %edi, %esi
+; SDAG-NEXT:    movl %esi, %eax
+; SDAG-NEXT:    retq
+;
+; FAST-LABEL: umuloselecti32:
+; FAST:       # %bb.0:
+; FAST-NEXT:    movl %edi, %eax
+; FAST-NEXT:    mull %esi
+; FAST-NEXT:    seto %al
+; FAST-NEXT:    testb $1, %al
+; FAST-NEXT:    cmovnel %edi, %esi
+; FAST-NEXT:    movl %esi, %eax
+; FAST-NEXT:    retq
 ;
 ; WIN64-LABEL: umuloselecti32:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    movl %edx, %r8d
 ; WIN64-NEXT:    movl %ecx, %eax
 ; WIN64-NEXT:    mull %edx
 ; WIN64-NEXT:    cmovol %ecx, %r8d
 ; WIN64-NEXT:    movl %r8d, %eax

 ; WIN32-NEXT:    retl
   %t = call {i32, i1} @llvm.umul.with.overflow.i32(i32 %v1, i32 %v2)
   %obit = extractvalue {i32, i1} %t, 1
   %ret = select i1 %obit, i32 %v1, i32 %v2
   ret i32 %ret
 }
 
 define i64 @umuloselecti64(i64 %v1, i64 %v2) {
-; LINUX-LABEL: umuloselecti64:
-; LINUX:       # %bb.0:
-; LINUX-NEXT:    movq %rdi, %rax
-; LINUX-NEXT:    mulq %rsi
-; LINUX-NEXT:    cmovoq %rdi, %rsi
-; LINUX-NEXT:    movq %rsi, %rax
-; LINUX-NEXT:    retq
+; SDAG-LABEL: umuloselecti64:
+; SDAG:       # %bb.0:
+; SDAG-NEXT:    movq %rdi, %rax
+; SDAG-NEXT:    mulq %rsi
+; SDAG-NEXT:    cmovoq %rdi, %rsi
+; SDAG-NEXT:    movq %rsi, %rax
+; SDAG-NEXT:    retq
+;
+; FAST-LABEL: umuloselecti64:
+; FAST:       # %bb.0:
+; FAST-NEXT:    movq %rdi, %rax
+; FAST-NEXT:    mulq %rsi
+; FAST-NEXT:    seto %al
+; FAST-NEXT:    testb $1, %al
+; FAST-NEXT:    cmovneq %rdi, %rsi
+; FAST-NEXT:    movq %rsi, %rax
+; FAST-NEXT:    retq
 ;
 ; WIN64-LABEL: umuloselecti64:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    movq %rdx, %r8
 ; WIN64-NEXT:    movq %rcx, %rax
 ; WIN64-NEXT:    mulq %rdx
 ; WIN64-NEXT:    cmovoq %rcx, %r8
 ; WIN64-NEXT:    movq %r8, %rax