Fix PR47960 - Incorrect transformation of fabs with nnan flag

[Krishnakariya]

Summary

Bug Fix for PR: https://bugs.llvm.org/show_bug.cgi?id=47960

This patch makes sure that the fast math flag used in the 'select' instruction is the same as the 'fabs' instruction after the transformation.

[spatel]

The current code was added with rG8b6d9f6 and a reaction/partial fix for:
https://llvm.org/PR38086

I was afraid that relying on the select for FMF could hinder further optimizations because propagation of FMF to select was (and still is) incomplete.
But I don't have any clear evidence to show the effect either way, so I think it's fine to give this a try. Just be on the lookout for regressions.

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

2868–2869

Remove the FIXME code comment.

llvm/test/Transforms/InstCombine/fabs.ll

268

Add a test comment here to make it explicit that nnan knowledge is lost with the transform.

[nlopes]

FYI just added support for select w/ fast-math to Alive2 so we can check these tests.

[Krishnakariya]

Added description for the test case.

[spatel]

In D101727#2746020, @nlopes wrote:

FYI just added support for select w/ fast-math to Alive2 so we can check these tests.

Great! Is it available on the online instance yet?
Checking my understanding on this example:
https://alive2.llvm.org/ce/z/M2Fe49

define double @src(double %x) {
%0:
  %lezero = fcmp ole double %x, 0.000000
  %negx = fsub nnan double 0.000000, %x
  %fabs = select i1 %lezero, double %negx, double %x
  ret double %fabs
}
=>
define double @tgt(double %x) {
%0:
  %fabs = fabs nnan double %x
  ret double %fabs
}
Transformation doesn't verify!

ERROR: Target is more poisonous than source

Example:
double %x = NaN

Source:
i1 %lezero = #x0 (0)
double %negx = poison
double %fabs = NaN

Target:
double %fabs = poison
Source value: NaN
Target value: poison

In the source function, if we have "nnan" on the select, and we choose %x (NaN), then the return value must be poison? The "nnan" on the select was dropped in the output pane though?

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

2883

What about this one and the fneg transforms below?
If the FMF propagation logic is identical, we should change all of these simultaneously to keep things consistent.
We already have regression tests for all 4 variants (but you might want to add extra tests to make sure we're not losing coverage).

[nlopes]

In D101727#2759617, @spatel wrote:

In D101727#2746020, @nlopes wrote:

FYI just added support for select w/ fast-math to Alive2 so we can check these tests.

Great! Is it available on the online instance yet?
Checking my understanding on this example:
https://alive2.llvm.org/ce/z/M2Fe49

define double @src(double %x) {
%0:
  %lezero = fcmp ole double %x, 0.000000
  %negx = fsub nnan double 0.000000, %x
  %fabs = select i1 %lezero, double %negx, double %x
  ret double %fabs
}
=>
define double @tgt(double %x) {
%0:
  %fabs = fabs nnan double %x
  ret double %fabs
}
Transformation doesn't verify!

ERROR: Target is more poisonous than source

Example:
double %x = NaN

Source:
i1 %lezero = #x0 (0)
double %negx = poison
double %fabs = NaN

Target:
double %fabs = poison
Source value: NaN
Target value: poison

In the source function, if we have "nnan" on the select, and we choose %x (NaN), then the return value must be poison? The "nnan" on the select was dropped in the output pane though?

Your example is correct, yes! Thanks for checking.
There are actually two issues:

• There was a typo in the select fast-math implementation, which I've just fixed.
• The online version hasn't been updated in a while. 'nnan' should appear in the output, so the online version is before the fast-math changes. Pinging @regehr so he updates the online Alive2 when he has time.

[Krishnakariya]

I have made changes in FMF propagation logic.
FMF like nnan and ninf should be propagated only if present in select instruction, while for other flags like arcp, reassoc, it doesn't really depend on select instruction.

Another thing that I want to point out is that after dropping nnan flag, the below test encounters an error.

Example link: https://alive2.llvm.org/ce/z/YEcHxG

define double @select_fcmp_nnan_nsz_olt_zero(double %x) {
%0:
  %ltzero = fcmp olt double %x, 0.000000
  %negx = fsub nnan nsz double -0.000000, %x
  %fabs = select i1 %ltzero, double %negx, double %x
  ret double %fabs
}
=>
define double @select_fcmp_nnan_nsz_olt_zero(double %x) {
%0:
  %1 = fabs nsz double %x
  ret double %1
}
Transformation doesn't verify!
ERROR: Value mismatch

Example:
double %x = #x8000000000000000 (-0.0)

Source:
i1 %ltzero = #x0 (0)
double %negx = #x0000000000000000 (+0.0)
double %fabs = #x8000000000000000 (-0.0)

Target:
double %1 = #x0000000000000000 (+0.0)
Source value: #x8000000000000000 (-0.0)
Target value: #x0000000000000000 (+0.0)

[Krishnakariya]

This patch is complete and is ready for review.

Another thing that I want to point out is that after dropping nnan flag, the below test encounters an error.

Example link: https://alive2.llvm.org/ce/z/YEcHxG

This example is a new bug detected by Alive 2 after introducing and testing this patch.

[spatel]

I'm confused. This patch started by just switching the FMF propagation source from the fneg/fsub to the select. Now, we are back to taking flags from fneg and then intersecting some flags from the select.

But this approach has known bugs, as you have shown in the new Alive2 example. Why is this version of the patch better than just propagating all flags from the select?

[Krishnakariya]

In D101727#2832971, @spatel wrote:

I'm confused. This patch started by just switching the FMF propagation source from the fneg/fsub to the select. Now, we are back to taking flags from fneg and then intersecting some flags from the select.

But this approach has known bugs, as you have shown in the new Alive2 example. Why is this version of the patch better than just propagating all flags from the select?

I am sorry for the confusion. This version has few issues which were detected by Alive2. So, I have made new changes as follows:

1. flags are propagated if present in select instruction.
2. FSub logic: Removed the check for nnan's flag presence in fsub instruction for doing the optimization. Example: https://alive2.llvm.org/ce/z/tYeZub (fsub).
3. FNeg logic: Instead of checking the presence of nnan and nsz flag in fneg instruction, added a check for nsz instruction in select instruction. Example: https://alive2.llvm.org/ce/z/TbuLnk (fneg), https://alive2.llvm.org/ce/z/yBDCZ_ (fsub).

[spatel]

1. Please upload patch with full context:

https://llvm.org/docs/Phabricator.html#requesting-a-review-via-the-web-interface

2. I prefer that we add more tests rather than alter the FMF on the existing tests. We want to be able to verify that all of these flag permutations are behaving as expected. We can commit the baseline tests as a preliminary step, so we just see the diffs in this review. If you do not have commit access, let me know so I can commit for you.

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

2867–2868

If we are dropping the nnan requirement, this comment should be removed. Can we make that change independently of the change that propagates flags from the select without breaking anything? If so, let's do that first as its own patch before this one.

[Krishnakariya]

Updates:

1. Added check for nnan (earlier removed) and added new tests.
2. Check for nsz in select for fneg logic is required. So, I didn't remove this check. Please see this example: https://alive2.llvm.org/ce/z/sEfTYx.

[Krishnakariya]

Ping @spatel, a gentle reminder. Could you please review the updates on this patch?

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

2867–2868

If we are dropping the nnan requirement, this comment should be removed. Can we make that change independently of the change that propagates flags from the select without breaking anything? If so, let's do that first as its own patch before this one.

[spatel]

In D101727#2879492, @Krishnakariya wrote:

Ping @spatel, a gentle reminder. Could you please review the updates on this patch?

I think you missed my earlier comment:
I prefer that we add more tests rather than alter the FMF on the existing tests. We want to be able to verify that all of these flag permutations are behaving as expected. We can commit the baseline tests as a preliminary step, so we just see the diffs in this review. If you do not have commit access, let me know so I can commit for you.

I'd like to have a final patch with no changes to the IR in the original tests; that is, we should still have test coverage for the existing patterns before this patch (and they will show that the transform does not fire any more in several cases). Does that make sense?

[Krishnakariya]

Hi @spatel,
I am sorry for my delayed responses to this patch. I wanted to ask for some clarification on your previous comments.
I have created few new tests similar to the original ones, with the only change that the FMF flag present in the fsub/fneg is now present on the select instruction. Original tests are left untouched.
Is this what you were expecting in your last comment?

[spatel]

In D101727#2896447, @Krishnakariya wrote:

Hi @spatel,
I am sorry for my delayed responses to this patch. I wanted to ask for some clarification on your previous comments.
I have created few new tests similar to the original ones, with the only change that the FMF flag present in the fsub/fneg is now present on the select instruction. Original tests are left untouched.
Is this what you were expecting in your last comment?

Yes, that looks like better coverage. We'll need to adjust some test comments too.
Please commit the extra tests with current CHECK line output (no code changes) to main. Then update this patch on top of that, so we will see the diffs.

[spatel]

LGTM - I think this is safe now and seems to fix the bugs noted by Alive2 , but we really should follow-up by cleaning this up: we don't need FMF at all in some cases, and we don't need to worry about preserving sign of NaN in the default FP environment.

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

2873

We could ease this condition, right?
If either of the fsub or the select has nnan, then the result can have nnan. It seems we don't need any FMF at all on some of these patterns:
https://alive2.llvm.org/ce/z/3bfTSb

Let's not complicate this patch any more though. If that's correct, then let's make changes as a follow-up patch (with more tests to confirm).

[Krishnakariya]

Yes, we may remove FMF checks as below. I will make a new patch for this.
FSub logic: Remove check for nnan flag presence in fsub instruction. Example: https://alive2.llvm.org/ce/z/tYeZub (fsub).
FNeg logic: Remove check for the presence of nnan and nsz flag in fneg instruction. Example: https://alive2.llvm.org/ce/z/TbuLnk (fneg), https://alive2.llvm.org/ce/z/yBDCZ_ (fsub).

Could you please commit this patch on my behalf? I do not have commit access yet.

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

if ((Cmp0 == TrueVal && Cmp1 == FalseVal) ||
       }
 
       // NOTE: if we wanted to, this is where to detect MIN/MAX
     }
   }
 
   // Canonicalize select with fcmp to fabs(). -0.0 makes this tricky. We need
   // fast-math-flags (nsz) or fsub with +0.0 (not fneg) for this to work. We
   // also require nnan because we do not want to unintentionally change the
   // sign of a NaN value.
-  // FIXME: These folds should test/propagate FMF from the select, not the
-  //        fsub or fneg.
   // (X <= +/-0.0) ? (0.0 - X) : X --> fabs(X)
   Instruction *FSub;
   if (match(CondVal, m_FCmp(Pred, m_Specific(FalseVal), m_AnyZeroFP())) &&
       match(TrueVal, m_FSub(m_PosZeroFP(), m_Specific(FalseVal))) &&
       match(TrueVal, m_Instruction(FSub)) && FSub->hasNoNaNs() &&
       (Pred == FCmpInst::FCMP_OLE || Pred == FCmpInst::FCMP_ULE)) {
-    Value *Fabs = Builder.CreateUnaryIntrinsic(Intrinsic::fabs, FalseVal, FSub);
+    Value *Fabs = Builder.CreateUnaryIntrinsic(Intrinsic::fabs, FalseVal, &SI);
     return replaceInstUsesWith(SI, Fabs);
   }
   // (X >  +/-0.0) ? X : (0.0 - X) --> fabs(X)
   if (match(CondVal, m_FCmp(Pred, m_Specific(TrueVal), m_AnyZeroFP())) &&
       match(FalseVal, m_FSub(m_PosZeroFP(), m_Specific(TrueVal))) &&
       match(FalseVal, m_Instruction(FSub)) && FSub->hasNoNaNs() &&
       (Pred == FCmpInst::FCMP_OGT || Pred == FCmpInst::FCMP_UGT)) {
-    Value *Fabs = Builder.CreateUnaryIntrinsic(Intrinsic::fabs, TrueVal, FSub);
+    Value *Fabs = Builder.CreateUnaryIntrinsic(Intrinsic::fabs, TrueVal, &SI);
     return replaceInstUsesWith(SI, Fabs);
   }
   // With nnan and nsz:
   // (X <  +/-0.0) ? -X : X --> fabs(X)
   // (X <= +/-0.0) ? -X : X --> fabs(X)
   Instruction *FNeg;
   if (match(CondVal, m_FCmp(Pred, m_Specific(FalseVal), m_AnyZeroFP())) &&
       match(TrueVal, m_FNeg(m_Specific(FalseVal))) &&
-      match(TrueVal, m_Instruction(FNeg)) &&
-      FNeg->hasNoNaNs() && FNeg->hasNoSignedZeros() &&
+      match(TrueVal, m_Instruction(FNeg)) && FNeg->hasNoNaNs() &&
+      FNeg->hasNoSignedZeros() && SI.hasNoSignedZeros() &&
       (Pred == FCmpInst::FCMP_OLT || Pred == FCmpInst::FCMP_OLE ||
        Pred == FCmpInst::FCMP_ULT || Pred == FCmpInst::FCMP_ULE)) {
-    Value *Fabs = Builder.CreateUnaryIntrinsic(Intrinsic::fabs, FalseVal, FNeg);
+    Value *Fabs = Builder.CreateUnaryIntrinsic(Intrinsic::fabs, FalseVal, &SI);
     return replaceInstUsesWith(SI, Fabs);
   }
   // With nnan and nsz:
   // (X >  +/-0.0) ? X : -X --> fabs(X)
   // (X >= +/-0.0) ? X : -X --> fabs(X)
   if (match(CondVal, m_FCmp(Pred, m_Specific(TrueVal), m_AnyZeroFP())) &&
       match(FalseVal, m_FNeg(m_Specific(TrueVal))) &&
-      match(FalseVal, m_Instruction(FNeg)) &&
-      FNeg->hasNoNaNs() && FNeg->hasNoSignedZeros() &&
+      match(FalseVal, m_Instruction(FNeg)) && FNeg->hasNoNaNs() &&
+      FNeg->hasNoSignedZeros() && SI.hasNoSignedZeros() &&
       (Pred == FCmpInst::FCMP_OGT || Pred == FCmpInst::FCMP_OGE ||
        Pred == FCmpInst::FCMP_UGT || Pred == FCmpInst::FCMP_UGE)) {
-    Value *Fabs = Builder.CreateUnaryIntrinsic(Intrinsic::fabs, TrueVal, FNeg);
+    Value *Fabs = Builder.CreateUnaryIntrinsic(Intrinsic::fabs, TrueVal, &SI);
     return replaceInstUsesWith(SI, Fabs);
   }
 
   // See if we are selecting two values based on a comparison of the two values.
   if (ICmpInst *ICI = dyn_cast<ICmpInst>(CondVal))
     if (Instruction *Result = foldSelectInstWithICmp(SI, ICI))
       return Result;
 

llvm/test/Transforms/InstCombine/fabs.ll

 ; CHECK-NEXT:    ret double [[FABS]]
 ;
   %lezero = fcmp ole double %x, 0.0
   %negx = fsub double 0.0, %x
   %fabs = select i1 %lezero, double %negx, double %x
   ret double %fabs
 }
 
 ; X <= 0.0 ? (0.0 - X) : X --> fabs(X)
 
 define double @select_fcmp_nnan_ole_zero(double %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_ole_zero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan double @llvm.fabs.f64(double [[X:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call double @llvm.fabs.f64(double [[X:%.*]])
 ; CHECK-NEXT:    ret double [[TMP1]]
 ;
   %lezero = fcmp ole double %x, 0.0
   %negx = fsub nnan double 0.0, %x
   %fabs = select i1 %lezero, double %negx, double %x
   ret double %fabs
 }
 
 define double @select_nnan_fcmp_nnan_ole_zero(double %x) {
 ; CHECK-LABEL: @select_nnan_fcmp_nnan_ole_zero(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call nnan double @llvm.fabs.f64(double [[X:%.*]])
 ; CHECK-NEXT:    ret double [[TMP1]]
 ;
   %lezero = fcmp ole double %x, 0.0
   %negx = fsub nnan double 0.0, %x
   %fabs = select nnan i1 %lezero, double %negx, double %x
   ret double %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define double @select_fcmp_nnan_ule_zero(double %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_ule_zero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan double @llvm.fabs.f64(double [[X:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call double @llvm.fabs.f64(double [[X:%.*]])
 ; CHECK-NEXT:    ret double [[TMP1]]
 ;
   %lezero = fcmp ule double %x, 0.0
   %negx = fsub nnan double 0.0, %x
   %fabs = select i1 %lezero, double %negx, double %x
   ret double %fabs
 }
 
;
   %fabs = select i1 %lezero, double %negx, double %x
   ret double %fabs
 }
 
 ; X <= -0.0 ? (0.0 - X) : X --> fabs(X)
 
 define <2 x float> @select_fcmp_nnan_ole_negzero(<2 x float> %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_ole_negzero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan <2 x float> @llvm.fabs.v2f32(<2 x float> [[X:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x float> @llvm.fabs.v2f32(<2 x float> [[X:%.*]])
 ; CHECK-NEXT:    ret <2 x float> [[TMP1]]
 ;
   %lezero = fcmp ole <2 x float> %x, <float -0.0, float -0.0>
   %negx = fsub nnan <2 x float> <float 0.0, float undef>, %x
   %fabs = select <2 x i1> %lezero, <2 x float> %negx, <2 x float> %x
   ret <2 x float> %fabs
 }
 
 define <2 x float> @select_nnan_fcmp_nnan_ole_negzero(<2 x float> %x) {
 ; CHECK-LABEL: @select_nnan_fcmp_nnan_ole_negzero(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call nnan <2 x float> @llvm.fabs.v2f32(<2 x float> [[X:%.*]])
 ; CHECK-NEXT:    ret <2 x float> [[TMP1]]
 ;
   %lezero = fcmp ole <2 x float> %x, <float -0.0, float -0.0>
   %negx = fsub nnan <2 x float> <float 0.0, float undef>, %x
   %fabs = select nnan <2 x i1> %lezero, <2 x float> %negx, <2 x float> %x
   ret <2 x float> %fabs
 }
 
 ; X > 0.0 ? X : (0.0 - X) --> fabs(X)
 
 define fp128 @select_fcmp_nnan_ogt_zero(fp128 %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_ogt_zero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan fp128 @llvm.fabs.f128(fp128 [[X:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call fp128 @llvm.fabs.f128(fp128 [[X:%.*]])
 ; CHECK-NEXT:    ret fp128 [[TMP1]]
 ;
   %gtzero = fcmp ogt fp128 %x, zeroinitializer
   %negx = fsub nnan fp128 zeroinitializer, %x
   %fabs = select i1 %gtzero, fp128 %x, fp128 %negx
   ret fp128 %fabs
 }
 
 define fp128 @select_nnan_fcmp_nnan_ogt_zero(fp128 %x) {
 ; CHECK-LABEL: @select_nnan_fcmp_nnan_ogt_zero(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call nnan fp128 @llvm.fabs.f128(fp128 [[X:%.*]])
 ; CHECK-NEXT:    ret fp128 [[TMP1]]
 ;
   %gtzero = fcmp ogt fp128 %x, zeroinitializer
   %negx = fsub nnan fp128 zeroinitializer, %x
   %fabs = select nnan i1 %gtzero, fp128 %x, fp128 %negx
   ret fp128 %fabs
 }
 
 ; X > -0.0 ? X : (0.0 - X) --> fabs(X)
 
 define half @select_fcmp_nnan_ogt_negzero(half %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_ogt_negzero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan half @llvm.fabs.f16(half [[X:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
 ; CHECK-NEXT:    ret half [[TMP1]]
 ;
   %gtzero = fcmp ogt half %x, -0.0
   %negx = fsub nnan half 0.0, %x
   %fabs = select i1 %gtzero, half %x, half %negx
   ret half %fabs
 }
 
 define half @select_nnan_fcmp_nnan_ogt_negzero(half %x) {
 ; CHECK-LABEL: @select_nnan_fcmp_nnan_ogt_negzero(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call nnan half @llvm.fabs.f16(half [[X:%.*]])
 ; CHECK-NEXT:    ret half [[TMP1]]
 ;
   %gtzero = fcmp ogt half %x, -0.0
   %negx = fsub nnan half 0.0, %x
   %fabs = select nnan i1 %gtzero, half %x, half %negx
   ret half %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define half @select_fcmp_nnan_ugt_negzero(half %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_ugt_negzero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan half @llvm.fabs.f16(half [[X:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call half @llvm.fabs.f16(half [[X:%.*]])
 ; CHECK-NEXT:    ret half [[TMP1]]
 ;
   %gtzero = fcmp ugt half %x, -0.0
   %negx = fsub nnan half 0.0, %x
   %fabs = select i1 %gtzero, half %x, half %negx
   ret half %fabs
 }
 
;
   %fabs = select i1 %gtzero, half %x, half %negx
   ret half %fabs
 }
 
 ; X < 0.0 ? -X : X --> fabs(X)
 
 define double @select_fcmp_nnan_nsz_olt_zero(double %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_olt_zero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz double @llvm.fabs.f64(double [[X:%.*]])
-; CHECK-NEXT:    ret double [[TMP1]]
+; CHECK-NEXT:    [[LTZERO:%.*]] = fcmp olt double [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz double [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LTZERO]], double [[NEGX]], double [[X]]
+; CHECK-NEXT:    ret double [[FABS]]
 ;
   %ltzero = fcmp olt double %x, 0.0
   %negx = fsub nnan nsz double -0.0, %x
   %fabs = select i1 %ltzero, double %negx, double %x
   ret double %fabs
 }
 
 define double @select_nnan_nsz_fcmp_nnan_nsz_olt_zero(double %x) {
 ; CHECK-LABEL: @select_nnan_nsz_fcmp_nnan_nsz_olt_zero(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz double @llvm.fabs.f64(double [[X:%.*]])
 ; CHECK-NEXT:    ret double [[TMP1]]
 ;
   %ltzero = fcmp olt double %x, 0.0
   %negx = fsub nnan nsz double -0.0, %x
   %fabs = select nnan nsz i1 %ltzero, double %negx, double %x
   ret double %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define double @select_fcmp_nnan_nsz_ult_zero(double %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ult_zero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz double @llvm.fabs.f64(double [[X:%.*]])
-; CHECK-NEXT:    ret double [[TMP1]]
+; CHECK-NEXT:    [[LTZERO:%.*]] = fcmp ult double [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz double [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LTZERO]], double [[NEGX]], double [[X]]
+; CHECK-NEXT:    ret double [[FABS]]
 ;
   %ltzero = fcmp ult double %x, 0.0
   %negx = fsub nnan nsz double -0.0, %x
   %fabs = select i1 %ltzero, double %negx, double %x
   ret double %fabs
 }
 
 define double @select_fcmp_nnan_nsz_olt_zero_unary_fneg(double %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_olt_zero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz double @llvm.fabs.f64(double [[X:%.*]])
-; CHECK-NEXT:    ret double [[TMP1]]
+; CHECK-NEXT:    [[LTZERO:%.*]] = fcmp olt double [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz double [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LTZERO]], double [[NEGX]], double [[X]]
+; CHECK-NEXT:    ret double [[FABS]]
 ;
   %ltzero = fcmp olt double %x, 0.0
   %negx = fneg nnan nsz double %x
   %fabs = select i1 %ltzero, double %negx, double %x
   ret double %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define double @select_fcmp_nnan_nsz_ult_zero_unary_fneg(double %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ult_zero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz double @llvm.fabs.f64(double [[X:%.*]])
-; CHECK-NEXT:    ret double [[TMP1]]
+; CHECK-NEXT:    [[LTZERO:%.*]] = fcmp ult double [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz double [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LTZERO]], double [[NEGX]], double [[X]]
+; CHECK-NEXT:    ret double [[FABS]]
 ;
   %ltzero = fcmp ult double %x, 0.0
   %negx = fneg nnan nsz double %x
   %fabs = select i1 %ltzero, double %negx, double %x
   ret double %fabs
 }
 
 ; X < -0.0 ? -X : X --> fabs(X)
 
 define float @select_fcmp_nnan_nsz_olt_negzero(float %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_olt_negzero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan ninf nsz float @llvm.fabs.f32(float [[X:%.*]])
-; CHECK-NEXT:    ret float [[TMP1]]
+; CHECK-NEXT:    [[LTZERO:%.*]] = fcmp olt float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan ninf nsz float [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LTZERO]], float [[NEGX]], float [[X]]
+; CHECK-NEXT:    ret float [[FABS]]
 ;
   %ltzero = fcmp olt float %x, -0.0
   %negx = fsub nnan ninf nsz float -0.0, %x
   %fabs = select i1 %ltzero, float %negx, float %x
   ret float %fabs
 }
 
 define float @select_nnan_ninf_nsz_fcmp_nnan_nsz_olt_negzero(float %x) {
 ; CHECK-LABEL: @select_nnan_ninf_nsz_fcmp_nnan_nsz_olt_negzero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz float @llvm.fabs.f32(float [[X:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call nnan ninf nsz float @llvm.fabs.f32(float [[X:%.*]])
 ; CHECK-NEXT:    ret float [[TMP1]]
 ;
   %ltzero = fcmp olt float %x, -0.0
   %negx = fsub nnan nsz float -0.0, %x
   %fabs = select nnan ninf nsz i1 %ltzero, float %negx, float %x
   ret float %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define float @select_fcmp_nnan_nsz_ult_negzero(float %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ult_negzero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan ninf nsz float @llvm.fabs.f32(float [[X:%.*]])
-; CHECK-NEXT:    ret float [[TMP1]]
+; CHECK-NEXT:    [[LTZERO:%.*]] = fcmp ult float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan ninf nsz float [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LTZERO]], float [[NEGX]], float [[X]]
+; CHECK-NEXT:    ret float [[FABS]]
 ;
   %ltzero = fcmp ult float %x, -0.0
   %negx = fsub nnan ninf nsz float -0.0, %x
   %fabs = select i1 %ltzero, float %negx, float %x
   ret float %fabs
 }
 
 define float @select_fcmp_nnan_nsz_olt_negzero_unary_fneg(float %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_olt_negzero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan ninf nsz float @llvm.fabs.f32(float [[X:%.*]])
-; CHECK-NEXT:    ret float [[TMP1]]
+; CHECK-NEXT:    [[LTZERO:%.*]] = fcmp olt float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan ninf nsz float [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LTZERO]], float [[NEGX]], float [[X]]
+; CHECK-NEXT:    ret float [[FABS]]
 ;
   %ltzero = fcmp olt float %x, -0.0
   %negx = fneg nnan ninf nsz float %x
   %fabs = select i1 %ltzero, float %negx, float %x
   ret float %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define float @select_fcmp_nnan_nsz_ult_negzero_unary_fneg(float %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ult_negzero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan ninf nsz float @llvm.fabs.f32(float [[X:%.*]])
-; CHECK-NEXT:    ret float [[TMP1]]
+; CHECK-NEXT:    [[LTZERO:%.*]] = fcmp ult float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan ninf nsz float [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LTZERO]], float [[NEGX]], float [[X]]
+; CHECK-NEXT:    ret float [[FABS]]
 ;
   %ltzero = fcmp ult float %x, -0.0
   %negx = fneg nnan ninf nsz float %x
   %fabs = select i1 %ltzero, float %negx, float %x
   ret float %fabs
 }
 
 ; X <= 0.0 ? -X : X --> fabs(X)
 
 define double @select_fcmp_nnan_nsz_ole_zero(double %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ole_zero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call fast double @llvm.fabs.f64(double [[X:%.*]])
-; CHECK-NEXT:    ret double [[TMP1]]
+; CHECK-NEXT:    [[LEZERO:%.*]] = fcmp ole double [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg fast double [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LEZERO]], double [[NEGX]], double [[X]]
+; CHECK-NEXT:    ret double [[FABS]]
 ;
   %lezero = fcmp ole double %x, 0.0
   %negx = fsub fast double -0.0, %x
   %fabs = select i1 %lezero, double %negx, double %x
   ret double %fabs
 }
 
 define double @select_fast_fcmp_nnan_nsz_ole_zero(double %x) {
 ; CHECK-LABEL: @select_fast_fcmp_nnan_nsz_ole_zero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz double @llvm.fabs.f64(double [[X:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call fast double @llvm.fabs.f64(double [[X:%.*]])
 ; CHECK-NEXT:    ret double [[TMP1]]
 ;
   %lezero = fcmp ole double %x, 0.0
   %negx = fsub nnan nsz double -0.0, %x
   %fabs = select fast i1 %lezero, double %negx, double %x
   ret double %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define double @select_fcmp_nnan_nsz_ule_zero(double %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ule_zero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call fast double @llvm.fabs.f64(double [[X:%.*]])
-; CHECK-NEXT:    ret double [[TMP1]]
+; CHECK-NEXT:    [[LEZERO:%.*]] = fcmp ule double [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg fast double [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LEZERO]], double [[NEGX]], double [[X]]
+; CHECK-NEXT:    ret double [[FABS]]
 ;
   %lezero = fcmp ule double %x, 0.0
   %negx = fsub fast double -0.0, %x
   %fabs = select i1 %lezero, double %negx, double %x
   ret double %fabs
 }
 
 define double @select_fcmp_nnan_nsz_ole_zero_unary_fneg(double %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ole_zero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call fast double @llvm.fabs.f64(double [[X:%.*]])
-; CHECK-NEXT:    ret double [[TMP1]]
+; CHECK-NEXT:    [[LEZERO:%.*]] = fcmp ole double [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg fast double [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LEZERO]], double [[NEGX]], double [[X]]
+; CHECK-NEXT:    ret double [[FABS]]
 ;
   %lezero = fcmp ole double %x, 0.0
   %negx = fneg fast double %x
   %fabs = select i1 %lezero, double %negx, double %x
   ret double %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define double @select_fcmp_nnan_nsz_ule_zero_unary_fneg(double %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ule_zero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call fast double @llvm.fabs.f64(double [[X:%.*]])
-; CHECK-NEXT:    ret double [[TMP1]]
+; CHECK-NEXT:    [[LEZERO:%.*]] = fcmp ule double [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg fast double [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LEZERO]], double [[NEGX]], double [[X]]
+; CHECK-NEXT:    ret double [[FABS]]
 ;
   %lezero = fcmp ule double %x, 0.0
   %negx = fneg fast double %x
   %fabs = select i1 %lezero, double %negx, double %x
   ret double %fabs
 }
 
 ; X <= -0.0 ? -X : X --> fabs(X)
 
 define float @select_fcmp_nnan_nsz_ole_negzero(float %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ole_negzero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz float @llvm.fabs.f32(float [[X:%.*]])
-; CHECK-NEXT:    ret float [[TMP1]]
+; CHECK-NEXT:    [[LEZERO:%.*]] = fcmp ole float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz float [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LEZERO]], float [[NEGX]], float [[X]]
+; CHECK-NEXT:    ret float [[FABS]]
 ;
   %lezero = fcmp ole float %x, -0.0
   %negx = fsub nnan nsz float -0.0, %x
   %fabs = select i1 %lezero, float %negx, float %x
   ret float %fabs
 }
 
 define float @select_nnan_nsz_fcmp_nnan_nsz_ole_negzero(float %x) {
 ; CHECK-LABEL: @select_nnan_nsz_fcmp_nnan_nsz_ole_negzero(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz float @llvm.fabs.f32(float [[X:%.*]])
 ; CHECK-NEXT:    ret float [[TMP1]]
 ;
   %lezero = fcmp ole float %x, -0.0
   %negx = fsub nnan nsz float -0.0, %x
   %fabs = select nnan nsz i1 %lezero, float %negx, float %x
   ret float %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define float @select_fcmp_nnan_nsz_ule_negzero(float %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ule_negzero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz float @llvm.fabs.f32(float [[X:%.*]])
-; CHECK-NEXT:    ret float [[TMP1]]
+; CHECK-NEXT:    [[LEZERO:%.*]] = fcmp ule float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz float [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LEZERO]], float [[NEGX]], float [[X]]
+; CHECK-NEXT:    ret float [[FABS]]
 ;
   %lezero = fcmp ule float %x, -0.0
   %negx = fsub nnan nsz float -0.0, %x
   %fabs = select i1 %lezero, float %negx, float %x
   ret float %fabs
 }
 
 define float @select_fcmp_nnan_nsz_ole_negzero_unary_fneg(float %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ole_negzero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz float @llvm.fabs.f32(float [[X:%.*]])
-; CHECK-NEXT:    ret float [[TMP1]]
+; CHECK-NEXT:    [[LEZERO:%.*]] = fcmp ole float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz float [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LEZERO]], float [[NEGX]], float [[X]]
+; CHECK-NEXT:    ret float [[FABS]]
 ;
   %lezero = fcmp ole float %x, -0.0
   %negx = fneg nnan nsz float %x
   %fabs = select i1 %lezero, float %negx, float %x
   ret float %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define float @select_fcmp_nnan_nsz_ule_negzero_unary_fneg(float %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ule_negzero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz float @llvm.fabs.f32(float [[X:%.*]])
-; CHECK-NEXT:    ret float [[TMP1]]
+; CHECK-NEXT:    [[LEZERO:%.*]] = fcmp ule float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz float [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[LEZERO]], float [[NEGX]], float [[X]]
+; CHECK-NEXT:    ret float [[FABS]]
 ;
   %lezero = fcmp ule float %x, -0.0
   %negx = fneg nnan nsz float %x
   %fabs = select i1 %lezero, float %negx, float %x
   ret float %fabs
 }
 
 ; X > 0.0 ? X : (0.0 - X) --> fabs(X)
 
 define <2 x float> @select_fcmp_nnan_nsz_ogt_zero(<2 x float> %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ogt_zero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz arcp <2 x float> @llvm.fabs.v2f32(<2 x float> [[X:%.*]])
-; CHECK-NEXT:    ret <2 x float> [[TMP1]]
+; CHECK-NEXT:    [[GTZERO:%.*]] = fcmp ogt <2 x float> [[X:%.*]], zeroinitializer
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz arcp <2 x float> [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select <2 x i1> [[GTZERO]], <2 x float> [[X]], <2 x float> [[NEGX]]
+; CHECK-NEXT:    ret <2 x float> [[FABS]]
 ;
   %gtzero = fcmp ogt <2 x float> %x, zeroinitializer
   %negx = fsub nnan nsz arcp <2 x float> <float -0.0, float -0.0>, %x
   %fabs = select <2 x i1> %gtzero, <2 x float> %x, <2 x float> %negx
   ret <2 x float> %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define <2 x float> @select_fcmp_nnan_nsz_ugt_zero(<2 x float> %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ugt_zero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz arcp <2 x float> @llvm.fabs.v2f32(<2 x float> [[X:%.*]])
-; CHECK-NEXT:    ret <2 x float> [[TMP1]]
+; CHECK-NEXT:    [[GTZERO:%.*]] = fcmp ugt <2 x float> [[X:%.*]], zeroinitializer
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz arcp <2 x float> [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select <2 x i1> [[GTZERO]], <2 x float> [[X]], <2 x float> [[NEGX]]
+; CHECK-NEXT:    ret <2 x float> [[FABS]]
 ;
   %gtzero = fcmp ugt <2 x float> %x, zeroinitializer
   %negx = fsub nnan nsz arcp <2 x float> <float -0.0, float -0.0>, %x
   %fabs = select <2 x i1> %gtzero, <2 x float> %x, <2 x float> %negx
   ret <2 x float> %fabs
 }
 
 define <2 x float> @select_fcmp_nnan_nsz_ogt_zero_unary_fneg(<2 x float> %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ogt_zero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz arcp <2 x float> @llvm.fabs.v2f32(<2 x float> [[X:%.*]])
-; CHECK-NEXT:    ret <2 x float> [[TMP1]]
+; CHECK-NEXT:    [[GTZERO:%.*]] = fcmp ogt <2 x float> [[X:%.*]], zeroinitializer
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz arcp <2 x float> [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select <2 x i1> [[GTZERO]], <2 x float> [[X]], <2 x float> [[NEGX]]
+; CHECK-NEXT:    ret <2 x float> [[FABS]]
 ;
   %gtzero = fcmp ogt <2 x float> %x, zeroinitializer
   %negx = fneg nnan nsz arcp <2 x float> %x
   %fabs = select <2 x i1> %gtzero, <2 x float> %x, <2 x float> %negx
   ret <2 x float> %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define <2 x float> @select_fcmp_nnan_nsz_ugt_zero_unary_fneg(<2 x float> %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ugt_zero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz arcp <2 x float> @llvm.fabs.v2f32(<2 x float> [[X:%.*]])
-; CHECK-NEXT:    ret <2 x float> [[TMP1]]
+; CHECK-NEXT:    [[GTZERO:%.*]] = fcmp ugt <2 x float> [[X:%.*]], zeroinitializer
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz arcp <2 x float> [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select <2 x i1> [[GTZERO]], <2 x float> [[X]], <2 x float> [[NEGX]]
+; CHECK-NEXT:    ret <2 x float> [[FABS]]
 ;
   %gtzero = fcmp ugt <2 x float> %x, zeroinitializer
   %negx = fneg nnan nsz arcp <2 x float> %x
   %fabs = select <2 x i1> %gtzero, <2 x float> %x, <2 x float> %negx
   ret <2 x float> %fabs
 }
 
 ; X > -0.0 ? X : (0.0 - X) --> fabs(X)
 
 define half @select_fcmp_nnan_nsz_ogt_negzero(half %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ogt_negzero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call fast half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    ret half [[TMP1]]
+; CHECK-NEXT:    [[GTZERO:%.*]] = fcmp ogt half [[X:%.*]], 0xH0000
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg fast half [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[GTZERO]], half [[X]], half [[NEGX]]
+; CHECK-NEXT:    ret half [[FABS]]
 ;
   %gtzero = fcmp ogt half %x, -0.0
   %negx = fsub fast half 0.0, %x
   %fabs = select i1 %gtzero, half %x, half %negx
   ret half %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define half @select_fcmp_nnan_nsz_ugt_negzero(half %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_ugt_negzero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call fast half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    ret half [[TMP1]]
+; CHECK-NEXT:    [[GTZERO:%.*]] = fcmp ugt half [[X:%.*]], 0xH0000
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg fast half [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[GTZERO]], half [[X]], half [[NEGX]]
+; CHECK-NEXT:    ret half [[FABS]]
 ;
   %gtzero = fcmp ugt half %x, -0.0
   %negx = fsub fast half 0.0, %x
   %fabs = select i1 %gtzero, half %x, half %negx
   ret half %fabs
 }
 
 ; X > 0.0 ? X : (0.0 - X) --> fabs(X)
 
 define <2 x double> @select_fcmp_nnan_nsz_oge_zero(<2 x double> %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_oge_zero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call reassoc nnan nsz <2 x double> @llvm.fabs.v2f64(<2 x double> [[X:%.*]])
-; CHECK-NEXT:    ret <2 x double> [[TMP1]]
+; CHECK-NEXT:    [[GEZERO:%.*]] = fcmp oge <2 x double> [[X:%.*]], zeroinitializer
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg reassoc nnan nsz <2 x double> [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select <2 x i1> [[GEZERO]], <2 x double> [[X]], <2 x double> [[NEGX]]
+; CHECK-NEXT:    ret <2 x double> [[FABS]]
 ;
   %gezero = fcmp oge <2 x double> %x, zeroinitializer
   %negx = fsub nnan nsz reassoc <2 x double> <double -0.0, double -0.0>, %x
   %fabs = select <2 x i1> %gezero, <2 x double> %x, <2 x double> %negx
   ret <2 x double> %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define <2 x double> @select_fcmp_nnan_nsz_uge_zero(<2 x double> %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_uge_zero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call reassoc nnan nsz <2 x double> @llvm.fabs.v2f64(<2 x double> [[X:%.*]])
-; CHECK-NEXT:    ret <2 x double> [[TMP1]]
+; CHECK-NEXT:    [[GEZERO:%.*]] = fcmp uge <2 x double> [[X:%.*]], zeroinitializer
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg reassoc nnan nsz <2 x double> [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select <2 x i1> [[GEZERO]], <2 x double> [[X]], <2 x double> [[NEGX]]
+; CHECK-NEXT:    ret <2 x double> [[FABS]]
 ;
   %gezero = fcmp uge <2 x double> %x, zeroinitializer
   %negx = fsub nnan nsz reassoc <2 x double> <double -0.0, double -0.0>, %x
   %fabs = select <2 x i1> %gezero, <2 x double> %x, <2 x double> %negx
   ret <2 x double> %fabs
 }
 
 define <2 x double> @select_fcmp_nnan_nsz_oge_zero_unary_fneg(<2 x double> %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_oge_zero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call reassoc nnan nsz <2 x double> @llvm.fabs.v2f64(<2 x double> [[X:%.*]])
-; CHECK-NEXT:    ret <2 x double> [[TMP1]]
+; CHECK-NEXT:    [[GEZERO:%.*]] = fcmp oge <2 x double> [[X:%.*]], zeroinitializer
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg reassoc nnan nsz <2 x double> [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select <2 x i1> [[GEZERO]], <2 x double> [[X]], <2 x double> [[NEGX]]
+; CHECK-NEXT:    ret <2 x double> [[FABS]]
 ;
   %gezero = fcmp oge <2 x double> %x, zeroinitializer
   %negx = fneg nnan nsz reassoc <2 x double> %x
   %fabs = select <2 x i1> %gezero, <2 x double> %x, <2 x double> %negx
   ret <2 x double> %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define <2 x double> @select_fcmp_nnan_nsz_uge_zero_unary_fneg(<2 x double> %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_uge_zero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call reassoc nnan nsz <2 x double> @llvm.fabs.v2f64(<2 x double> [[X:%.*]])
-; CHECK-NEXT:    ret <2 x double> [[TMP1]]
+; CHECK-NEXT:    [[GEZERO:%.*]] = fcmp uge <2 x double> [[X:%.*]], zeroinitializer
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg reassoc nnan nsz <2 x double> [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select <2 x i1> [[GEZERO]], <2 x double> [[X]], <2 x double> [[NEGX]]
+; CHECK-NEXT:    ret <2 x double> [[FABS]]
 ;
   %gezero = fcmp uge <2 x double> %x, zeroinitializer
   %negx = fneg nnan nsz reassoc <2 x double> %x
   %fabs = select <2 x i1> %gezero, <2 x double> %x, <2 x double> %negx
   ret <2 x double> %fabs
 }
 
 ; X > -0.0 ? X : (0.0 - X) --> fabs(X)
 
 define half @select_fcmp_nnan_nsz_oge_negzero(half %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_oge_negzero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    ret half [[TMP1]]
+; CHECK-NEXT:    [[GEZERO:%.*]] = fcmp oge half [[X:%.*]], 0xH0000
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz half [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[GEZERO]], half [[X]], half [[NEGX]]
+; CHECK-NEXT:    ret half [[FABS]]
 ;
   %gezero = fcmp oge half %x, -0.0
   %negx = fsub nnan nsz half -0.0, %x
   %fabs = select i1 %gezero, half %x, half %negx
   ret half %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define half @select_fcmp_nnan_nsz_uge_negzero(half %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_uge_negzero(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    ret half [[TMP1]]
+; CHECK-NEXT:    [[GEZERO:%.*]] = fcmp uge half [[X:%.*]], 0xH0000
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz half [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[GEZERO]], half [[X]], half [[NEGX]]
+; CHECK-NEXT:    ret half [[FABS]]
 ;
   %gezero = fcmp uge half %x, -0.0
   %negx = fsub nnan nsz half -0.0, %x
   %fabs = select i1 %gezero, half %x, half %negx
   ret half %fabs
 }
 
 define half @select_fcmp_nnan_nsz_oge_negzero_unary_fneg(half %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_oge_negzero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    ret half [[TMP1]]
+; CHECK-NEXT:    [[GEZERO:%.*]] = fcmp oge half [[X:%.*]], 0xH0000
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz half [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[GEZERO]], half [[X]], half [[NEGX]]
+; CHECK-NEXT:    ret half [[FABS]]
 ;
   %gezero = fcmp oge half %x, -0.0
   %negx = fneg nnan nsz half %x
   %fabs = select i1 %gezero, half %x, half %negx
   ret half %fabs
 }
 
 ; Repeat with unordered predicate - nnan allows us to treat ordered/unordered identically.
 
 define half @select_fcmp_nnan_nsz_uge_negzero_unary_fneg(half %x) {
 ; CHECK-LABEL: @select_fcmp_nnan_nsz_uge_negzero_unary_fneg(
-; CHECK-NEXT:    [[TMP1:%.*]] = call nnan nsz half @llvm.fabs.f16(half [[X:%.*]])
-; CHECK-NEXT:    ret half [[TMP1]]
+; CHECK-NEXT:    [[GEZERO:%.*]] = fcmp uge half [[X:%.*]], 0xH0000
+; CHECK-NEXT:    [[NEGX:%.*]] = fneg nnan nsz half [[X]]
+; CHECK-NEXT:    [[FABS:%.*]] = select i1 [[GEZERO]], half [[X]], half [[NEGX]]
+; CHECK-NEXT:    ret half [[FABS]]
 ;
   %gezero = fcmp uge half %x, -0.0
   %negx = fneg nnan nsz half %x
   %fabs = select i1 %gezero, half %x, half %negx
   ret half %fabs
 }
 
 define float @select_fneg(i1 %c, float %x) {