[AArch64] Default to zero-cycle-zeroing FP registers.

[SjoerdMeijer]

Summary

It is generally beneficial to prefer "movi d0, #0" over "fmov s0, wzr" as this is most efficient across all cores. For newer cores, fmov instructions are also eliminated early and there is no difference with movi, but this is not true for other/older cores. Thus this standardises on using movi.

[paquette]

LGTM

[dmgreen]

What CPU is this expected to be better for? I don't buy the "int -> fp register transfer". I'm not going to pretend to know how cpus work internally, but there is no real register value it is transferring.

[dmgreen]

OK. I think I see what's wrong. According to the A55 software optimization guide, the dual issue for a movi is a little more restrictive than fmov, which can lead to slower code. We would probably want to prefer the fmov there. Which probably applies to other inorder cpus.

I don't have great visibility on other cpus. I just happen to have some very low noise A55 tests that can show whether this kind of small change is actually beneficial.

It looks from the other optimization guides like the two instructions should be treated the same, performance wise. I would be surprised if a fmov s0, wzr wasn't really treated like a form of "FP move, immed", although I have no evidence one way or the other which way it works.

[SjoerdMeijer]

Fair enough, let's refrain from micro-architectural details. But the point is that zero-cost zeroing idioms are supported on integer operations, which is why this is preferred. This should always gives the same or better performance, but it looks like you found a bit of corner case with dual issuing, which is a bit surprising but perhaps makes some sense for smaller in-order cores. I will add FeatureNoZCZeroingFP to the A55's description.

[SjoerdMeijer]

This sets FeatureNoZCZeroingFP for some older cores.

I have considered not making this the default, but just setting ZCZeroingFP for each core. But I think the cores not supporting this are an exception, the trend is that this supported by newer cores, and GCC also defaults to this, see https://godbolt.org/z/Mvxvze48M. Thus making ZCZeroingFP the default still makes sense I think.

[SjoerdMeijer]

After some more discussions, it turns out the original revision was doing the right thing. Except that we should be using the .2s variant as that may be more efficient on some cores.

[SjoerdMeijer]

Adjust this to D99710, that uses movi d0 that zeros 64 bits and not 128 bits, which enables this as a default for all cores.

[david-arm]

llvm/test/CodeGen/AArch64/arm64-fp-contract-zero.ll

10

This looks like a regression here I think

[dmgreen]

Thanks. My tests agreed, LGTM

llvm/test/CodeGen/AArch64/arm64-fp-contract-zero.ll

10

It's just regenerating the tests with the test script now, by the look of it.

[SjoerdMeijer]

llvm/test/CodeGen/AArch64/arm64-fp-contract-zero.ll

10

Yep, exactly that. I wanted to see all of the codegen for this example, so just used the script.

Thanks for reviewing!

[david-arm]

LGTM too!

llvm/lib/Target/AArch64/AArch64.td

def FeatureSVE2BitPerm : SubtargetFeature<"sve2-bitperm", "HasSVE2BitPerm", "true",
   "Enable bit permutation SVE2 instructions", [FeatureSVE2]>;
 
 def FeatureZCRegMove : SubtargetFeature<"zcm", "HasZeroCycleRegMove", "true",
                                         "Has zero-cycle register moves">;
 
 def FeatureZCZeroingGP : SubtargetFeature<"zcz-gp", "HasZeroCycleZeroingGP", "true",
                                         "Has zero-cycle zeroing instructions for generic registers">;
 
-def FeatureZCZeroingFP : SubtargetFeature<"zcz-fp", "HasZeroCycleZeroingFP", "true",
-                                        "Has zero-cycle zeroing instructions for FP registers">;
+def FeatureNoZCZeroingFP : SubtargetFeature<"no-zcz-fp", "HasZeroCycleZeroingFP", "false",
+                                        "Has no zero-cycle zeroing instructions for FP registers">;
 
 def FeatureZCZeroing : SubtargetFeature<"zcz", "HasZeroCycleZeroing", "true",
                                         "Has zero-cycle zeroing instructions",
-                                        [FeatureZCZeroingGP, FeatureZCZeroingFP]>;
+                                        [FeatureZCZeroingGP]>;
 
 /// ... but the floating-point version doesn't quite work in rare cases on older
 /// CPUs.
 def FeatureZCZeroingFPWorkaround : SubtargetFeature<"zcz-fp-workaround",
     "HasZeroCycleZeroingFPWorkaround", "true",
     "The zero-cycle floating-point zeroing instruction has a bug">;
 
 def FeatureStrictAlign : SubtargetFeature<"strict-align",
def ProcExynosM3 : SubtargetFeature<"exynosm3", "ARMProcFamily", "ExynosM3",
                                      FeatureForce32BitJumpTables,
                                      FeatureFuseAddress,
                                      FeatureFuseAES,
                                      FeatureFuseCCSelect,
                                      FeatureFuseLiterals,
                                      FeatureLSLFast,
                                      FeaturePerfMon,
                                      FeaturePostRAScheduler,
-                                     FeaturePredictableSelectIsExpensive,
-                                     FeatureZCZeroingFP]>;
+                                     FeaturePredictableSelectIsExpensive]>;
 
 def ProcExynosM4 : SubtargetFeature<"exynosm4", "ARMProcFamily", "ExynosM3",
                                     "Samsung Exynos-M4 processors",
                                     [HasV8_2aOps,
                                      FeatureArithmeticBccFusion,
                                      FeatureArithmeticCbzFusion,
                                      FeatureCrypto,
                                      FeatureDotProd,

llvm/lib/Target/AArch64/AArch64Subtarget.h

protected:
   bool HasSPE_EEF = false;
 
   // HasZeroCycleRegMove - Has zero-cycle register mov instructions.
   bool HasZeroCycleRegMove = false;
 
   // HasZeroCycleZeroing - Has zero-cycle zeroing instructions.
   bool HasZeroCycleZeroing = false;
   bool HasZeroCycleZeroingGP = false;
-  bool HasZeroCycleZeroingFP = false;
   bool HasZeroCycleZeroingFPWorkaround = false;
 
+  // It is generally beneficial to rewrite "fmov s0, wzr" to "movi d0, #0".
+  // as movi is more efficient across all cores. Newer cores can eliminate
+  // fmovs early and there is no difference with movi, but this not true for
+  // all implementations.
+  bool HasZeroCycleZeroingFP = true;
+
   // StrictAlign - Disallow unaligned memory accesses.
   bool StrictAlign = false;
 
   // NegativeImmediates - transform instructions with negative immediates
   bool NegativeImmediates = true;
 
   // Enable 64-bit vectorization in SLP.
   unsigned MinVectorRegisterBitWidth = 64;

llvm/test/CodeGen/AArch64/arm64-fast-isel-materialize.ll

 ; GISEL:       fmov s0, #1.25000000
   ret float 1.250000e+00
 }
 
 define float @fmov_float2() {
 ; CHECK-LABEL: fmov_float2
 ; CHECK:       fmov s0, wzr
 ; GISEL-LABEL: fmov_float2
-; GISEL:       fmov s0, wzr
+; GISEL:       movi d0, #0000000000000000
   ret float 0.0e+00
 }
 
 define double @fmov_double1() {
 ; CHECK-LABEL: fmov_double1
 ; CHECK:       fmov d0, #1.25000000
 ; GISEL-LABEL: fmov_double1
 ; GISEL:       fmov d0, #1.25000000
   ret double 1.250000e+00
 }
 
 define double @fmov_double2() {
 ; CHECK-LABEL: fmov_double2
 ; CHECK:       fmov d0, xzr
 ; GISEL-LABEL: fmov_double2
-; GISEL:       fmov d0, xzr
+; GISEL:       movi d0, #0000000000000000
   ret double 0.0e+00
 }
 
 ; Materialize from constant pool
 define float @cp_float() {
 ; CHECK-LABEL: cp_float
 ; CHECK:       adrp [[REG:x[0-9]+]], {{lCPI[0-9]+_0}}@PAGE
 ; CHECK-NEXT:  ldr s0, {{\[}}[[REG]], {{lCPI[0-9]+_0}}@PAGEOFF{{\]}}

llvm/test/CodeGen/AArch64/arm64-fp-contract-zero.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=arm64 -fp-contract=fast -o - %s | FileCheck %s
 
 
 ; Make sure we don't try to fold an fneg into +0.0, creating an illegal constant
 ; -0.0. It's also good, though not essential, that we don't resort to a litpool.
 define double @test_fms_fold(double %a, double %b) {
 ; CHECK-LABEL: test_fms_fold:
-; CHECK: fmov {{d[0-9]+}}, xzr
-; CHECK: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    movi d2, #0000000000000000
+; CHECK-NEXT:    fmul d1, d1, d2
+; CHECK-NEXT:    fnmsub d0, d0, d2, d1
+; CHECK-NEXT:    ret
   %mul = fmul double %a, 0.000000e+00
   %mul1 = fmul double %b, 0.000000e+00
   %sub = fsub double %mul, %mul1
   ret double %sub
 }

llvm/test/CodeGen/AArch64/arm64-rev.ll

 ; CHECK-NEXT:    dup.4s v1, v1[0]
 ; CHECK-NEXT:    ext.16b v0, v0, v1, #12
 ; CHECK-NEXT:    rev64.4s v0, v0
 ; CHECK-NEXT:    str q0, [x1, #176]
 ; CHECK-NEXT:    ret
 ;
 ; FALLBACK-LABEL: float_vrev64:
 ; FALLBACK:       // %bb.0: // %entry
-; FALLBACK-NEXT:    fmov s0, wzr
+; FALLBACK-NEXT:    movi  d0, #0000000000000000
 ; FALLBACK-NEXT:    mov.s v0[1], v0[0]
 ; FALLBACK-NEXT:    mov.s v0[2], v0[0]
 ; FALLBACK-NEXT:    adrp x8, .LCPI28_0
 ; FALLBACK-NEXT:    mov.s v0[3], v0[0]
 ; FALLBACK-NEXT:    ldr q1, [x0]
 ; FALLBACK-NEXT:    ldr q2, [x8, :lo12:.LCPI28_0]
 ; FALLBACK-NEXT:    tbl.16b v0, { v0, v1 }, v2
 ; FALLBACK-NEXT:    str q0, [x1, #176]

llvm/test/CodeGen/AArch64/arm64-zero-cycle-zeroing.ll

-; RUN: llc < %s -mtriple=aarch64-linux-gnu -mattr=-zcz                    | FileCheck %s -check-prefixes=ALL,NONEGP,NONEFP
+; RUN: llc < %s -mtriple=aarch64-linux-gnu -mattr=-zcz-gp,+no-zcz-fp      | FileCheck %s -check-prefixes=ALL,NONEGP,NONEFP
 ; RUN: llc < %s -mtriple=aarch64-linux-gnu -mattr=+zcz                    | FileCheck %s -check-prefixes=ALL,ZEROGP,ZEROFP
 ; RUN: llc < %s -mtriple=aarch64-linux-gnu -mattr=+zcz -mattr=+fullfp16   | FileCheck %s -check-prefixes=ALL,ZEROGP,ZERO16
-; RUN: llc < %s -mtriple=aarch64-linux-gnu -mattr=+zcz-gp                 | FileCheck %s -check-prefixes=ALL,ZEROGP,NONEFP
-; RUN: llc < %s -mtriple=aarch64-linux-gnu -mattr=+zcz-fp                 | FileCheck %s -check-prefixes=ALL,NONEGP,ZEROFP
+; RUN: llc < %s -mtriple=aarch64-linux-gnu -mattr=+zcz-gp,+no-zcz-fp      | FileCheck %s -check-prefixes=ALL,ZEROGP,NONEFP
+; RUN: llc < %s -mtriple=aarch64-linux-gnu                                | FileCheck %s -check-prefixes=ALL,NONEGP,ZEROFP
 ; RUN: llc < %s -mtriple=arm64-apple-ios   -mcpu=cyclone                  | FileCheck %s -check-prefixes=ALL,ZEROGP,NONEFP
 ; RUN: llc < %s -mtriple=arm64-linux-gnu   -mcpu=apple-a10                | FileCheck %s -check-prefixes=ALL,ZEROGP,ZEROFP
 ; RUN: llc < %s -mtriple=arm64-apple-ios   -mcpu=cyclone -mattr=+fullfp16 | FileCheck %s -check-prefixes=ALL,ZEROGP,NONE16
 ; RUN: llc < %s -mtriple=aarch64-linux-gnu -mcpu=exynos-m3                | FileCheck %s -check-prefixes=ALL,NONEGP,ZEROFP
 ; RUN: llc < %s -mtriple=aarch64-linux-gnu -mcpu=kryo                     | FileCheck %s -check-prefixes=ALL,ZEROGP,ZEROFP
-; RUN: llc < %s -mtriple=aarch64-linux-gnu -mcpu=falkor                   | FileCheck %s -check-prefixes=ALL,ZEROGP,ZEROFP
+; UN: llc < %s -mtriple=aarch64-linux-gnu -mcpu=falkor                   | FileCheck %s -check-prefixes=ALL,ZEROGP,ZEROFP
 
 declare void @bar(half, float, double, <2 x double>)
 declare void @bari(i32, i32)
 declare void @barl(i64, i64)
 declare void @barf(float, float)
 
 define void @t1() nounwind ssp {
 entry:

llvm/test/CodeGen/AArch64/f16-imm.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=aarch64-none-eabi -mattr=+fullfp16 | FileCheck %s --check-prefixes=CHECK,CHECK-FP16,CHECK-NOZCZ
+; RUN: llc < %s -mtriple=aarch64-none-eabi -mattr=+fullfp16,+no-zcz-fp | FileCheck %s --check-prefixes=CHECK,CHECK-FP16,CHECK-NOZCZ
 ; RUN: llc < %s -mtriple=aarch64-none-eabi -mattr=+fullfp16,+zcz | FileCheck %s --check-prefixes=CHECK,CHECK-FP16,CHECK-ZCZ
 ; RUN: llc < %s -mtriple=aarch64-none-eabi -mattr=-fullfp16 | FileCheck %s --check-prefixes=CHECK,CHECK-NOFP16
 
 define half @Const0() {
 ; CHECK-NOZCZ-LABEL: Const0:
 ; CHECK-NOZCZ:       // %bb.0: // %entry
 ; CHECK-NOZCZ-NEXT:    fmov h0, wzr
 ; CHECK-NOZCZ-NEXT:    ret

llvm/test/CodeGen/AArch64/fptosi-sat-scalar.ll

 declare  i64 @llvm.fptosi.sat.i64.f32 (float)
 declare i100 @llvm.fptosi.sat.i100.f32(float)
 declare i128 @llvm.fptosi.sat.i128.f32(float)
 
 define i1 @test_signed_i1_f32(float %f) nounwind {
 ; CHECK-LABEL: test_signed_i1_f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fmov s1, #-1.00000000
-; CHECK-NEXT:    fmov s2, wzr
+; CHECK-NEXT:    movi d2, #0000000000000000
 ; CHECK-NEXT:    fmaxnm s1, s0, s1
 ; CHECK-NEXT:    fminnm s1, s1, s2
 ; CHECK-NEXT:    fcvtzs w8, s1
 ; CHECK-NEXT:    fcmp s0, s0
 ; CHECK-NEXT:    csel w8, wzr, w8, vs
 ; CHECK-NEXT:    and w0, w8, #0x1
 ; CHECK-NEXT:    ret
     %x = call i1 @llvm.fptosi.sat.i1.f32(float %f)

 declare  i64 @llvm.fptosi.sat.i64.f64 (double)
 declare i100 @llvm.fptosi.sat.i100.f64(double)
 declare i128 @llvm.fptosi.sat.i128.f64(double)
 
 define i1 @test_signed_i1_f64(double %f) nounwind {
 ; CHECK-LABEL: test_signed_i1_f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fmov d1, #-1.00000000
-; CHECK-NEXT:    fmov d2, xzr
+; CHECK-NEXT:    movi d2, #0000000000000000
 ; CHECK-NEXT:    fmaxnm d1, d0, d1
 ; CHECK-NEXT:    fminnm d1, d1, d2
 ; CHECK-NEXT:    fcvtzs w8, d1
 ; CHECK-NEXT:    fcmp d0, d0
 ; CHECK-NEXT:    csel w8, wzr, w8, vs
 ; CHECK-NEXT:    and w0, w8, #0x1
 ; CHECK-NEXT:    ret
     %x = call i1 @llvm.fptosi.sat.i1.f64(double %f)

 declare i100 @llvm.fptosi.sat.i100.f16(half)
 declare i128 @llvm.fptosi.sat.i128.f16(half)
 
 define i1 @test_signed_i1_f16(half %f) nounwind {
 ; CHECK-LABEL: test_signed_i1_f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fcvt s0, h0
 ; CHECK-NEXT:    fmov s1, #-1.00000000
-; CHECK-NEXT:    fmov s2, wzr
+; CHECK-NEXT:    movi d2, #0000000000000000
 ; CHECK-NEXT:    fmaxnm s1, s0, s1
 ; CHECK-NEXT:    fminnm s1, s1, s2
 ; CHECK-NEXT:    fcvtzs w8, s1
 ; CHECK-NEXT:    fcmp s0, s0
 ; CHECK-NEXT:    csel w8, wzr, w8, vs
 ; CHECK-NEXT:    and w0, w8, #0x1
 ; CHECK-NEXT:    ret
     %x = call i1 @llvm.fptosi.sat.i1.f16(half %f)

llvm/test/CodeGen/AArch64/fptosi-sat-vector.ll

 declare <2 x i128> @llvm.fptosi.sat.v2f32.v2i128(<2 x float>)
 
 define <2 x i1> @test_signed_v2f32_v2i1(<2 x float> %f) {
 ; CHECK-LABEL: test_signed_v2f32_v2i1:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    mov s1, v0.s[1]
 ; CHECK-NEXT:    fmov s2, #-1.00000000
-; CHECK-NEXT:    fmov s3, wzr
+; CHECK-NEXT:    movi d3, #0000000000000000
 ; CHECK-NEXT:    fmaxnm s4, s1, s2
 ; CHECK-NEXT:    fcmp s1, s1
 ; CHECK-NEXT:    fmaxnm s1, s0, s2
 ; CHECK-NEXT:    fminnm s2, s4, s3
 ; CHECK-NEXT:    fminnm s1, s1, s3
 ; CHECK-NEXT:    fcvtzs w8, s2
 ; CHECK-NEXT:    fcvtzs w9, s1
 ; CHECK-NEXT:    csel w8, wzr, w8, vs

 declare <2 x i100> @llvm.fptosi.sat.v2f64.v2i100(<2 x double>)
 declare <2 x i128> @llvm.fptosi.sat.v2f64.v2i128(<2 x double>)
 
 define <2 x i1> @test_signed_v2f64_v2i1(<2 x double> %f) {
 ; CHECK-LABEL: test_signed_v2f64_v2i1:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov d1, v0.d[1]
 ; CHECK-NEXT:    fmov d2, #-1.00000000
-; CHECK-NEXT:    fmov d3, xzr
+; CHECK-NEXT:    movi d3, #0000000000000000
 ; CHECK-NEXT:    fmaxnm d4, d1, d2
 ; CHECK-NEXT:    fcmp d1, d1
 ; CHECK-NEXT:    fmaxnm d1, d0, d2
 ; CHECK-NEXT:    fminnm d2, d4, d3
 ; CHECK-NEXT:    fminnm d1, d1, d3
 ; CHECK-NEXT:    fcvtzs w8, d2
 ; CHECK-NEXT:    fcvtzs w9, d1
 ; CHECK-NEXT:    csel w8, wzr, w8, vs

 declare <4 x i128> @llvm.fptosi.sat.v4f16.v4i128(<4 x half>)
 
 define <4 x i1> @test_signed_v4f16_v4i1(<4 x half> %f) {
 ; CHECK-LABEL: test_signed_v4f16_v4i1:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    fmov s2, #-1.00000000
 ; CHECK-NEXT:    fcvt s4, h0
-; CHECK-NEXT:    fmov s3, wzr
+; CHECK-NEXT:    movi d3, #0000000000000000
 ; CHECK-NEXT:    fmaxnm s5, s4, s2
 ; CHECK-NEXT:    mov h1, v0.h[1]
 ; CHECK-NEXT:    fminnm s5, s5, s3
 ; CHECK-NEXT:    fcvtzs w8, s5
 ; CHECK-NEXT:    mov h5, v0.h[2]
 ; CHECK-NEXT:    mov h0, v0.h[3]
 ; CHECK-NEXT:    fcvt s1, h1
 ; CHECK-NEXT:    fcvt s6, h0

llvm/test/CodeGen/AArch64/fptoui-sat-scalar.ll

 declare  i50 @llvm.fptoui.sat.i50.f32 (float)
 declare  i64 @llvm.fptoui.sat.i64.f32 (float)
 declare i100 @llvm.fptoui.sat.i100.f32(float)
 declare i128 @llvm.fptoui.sat.i128.f32(float)
 
 define i1 @test_unsigned_i1_f32(float %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i1_f32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s1, #1.00000000
 ; CHECK-NEXT:    fminnm s0, s0, s1
 ; CHECK-NEXT:    fcvtzu w8, s0
 ; CHECK-NEXT:    and w0, w8, #0x1
 ; CHECK-NEXT:    ret
     %x = call i1 @llvm.fptoui.sat.i1.f32(float %f)
     ret i1 %x
 }
 
 define i8 @test_unsigned_i8_f32(float %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i8_f32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    mov w8, #1132396544
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s1, w8
 ; CHECK-NEXT:    fminnm s0, s0, s1
 ; CHECK-NEXT:    fcvtzu w0, s0
 ; CHECK-NEXT:    ret
     %x = call i8 @llvm.fptoui.sat.i8.f32(float %f)
     ret i8 %x
 }
 
 define i13 @test_unsigned_i13_f32(float %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i13_f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #63488
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk w8, #17919, lsl #16
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s1, w8
 ; CHECK-NEXT:    fminnm s0, s0, s1
 ; CHECK-NEXT:    fcvtzu w0, s0
 ; CHECK-NEXT:    ret
     %x = call i13 @llvm.fptoui.sat.i13.f32(float %f)
     ret i13 %x
 }
 
 define i16 @test_unsigned_i16_f32(float %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i16_f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #65280
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk w8, #18303, lsl #16
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s1, w8
 ; CHECK-NEXT:    fminnm s0, s0, s1
 ; CHECK-NEXT:    fcvtzu w0, s0
 ; CHECK-NEXT:    ret
     %x = call i16 @llvm.fptoui.sat.i16.f32(float %f)
     ret i16 %x
 }
 
 define i19 @test_unsigned_i19_f32(float %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i19_f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #65504
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk w8, #18687, lsl #16
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s1, w8
 ; CHECK-NEXT:    fminnm s0, s0, s1
 ; CHECK-NEXT:    fcvtzu w0, s0
 ; CHECK-NEXT:    ret
     %x = call i19 @llvm.fptoui.sat.i19.f32(float %f)
     ret i19 %x

 declare  i50 @llvm.fptoui.sat.i50.f64 (double)
 declare  i64 @llvm.fptoui.sat.i64.f64 (double)
 declare i100 @llvm.fptoui.sat.i100.f64(double)
 declare i128 @llvm.fptoui.sat.i128.f64(double)
 
 define i1 @test_unsigned_i1_f64(double %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i1_f64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d1, #1.00000000
 ; CHECK-NEXT:    fminnm d0, d0, d1
 ; CHECK-NEXT:    fcvtzu w8, d0
 ; CHECK-NEXT:    and w0, w8, #0x1
 ; CHECK-NEXT:    ret
     %x = call i1 @llvm.fptoui.sat.i1.f64(double %f)
     ret i1 %x
 }
 
 define i8 @test_unsigned_i8_f64(double %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i8_f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #246290604621824
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16495, lsl #48
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d1, x8
 ; CHECK-NEXT:    fminnm d0, d0, d1
 ; CHECK-NEXT:    fcvtzu w0, d0
 ; CHECK-NEXT:    ret
     %x = call i8 @llvm.fptoui.sat.i8.f64(double %f)
     ret i8 %x
 }
 
 define i13 @test_unsigned_i13_f64(double %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i13_f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #280375465082880
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16575, lsl #48
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d1, x8
 ; CHECK-NEXT:    fminnm d0, d0, d1
 ; CHECK-NEXT:    fcvtzu w0, d0
 ; CHECK-NEXT:    ret
     %x = call i13 @llvm.fptoui.sat.i13.f64(double %f)
     ret i13 %x
 }
 
 define i16 @test_unsigned_i16_f64(double %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i16_f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #281337537757184
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16623, lsl #48
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d1, x8
 ; CHECK-NEXT:    fminnm d0, d0, d1
 ; CHECK-NEXT:    fcvtzu w0, d0
 ; CHECK-NEXT:    ret
     %x = call i16 @llvm.fptoui.sat.i16.f64(double %f)
     ret i16 %x
 }
 
 define i19 @test_unsigned_i19_f64(double %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i19_f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #281457796841472
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16671, lsl #48
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d1, x8
 ; CHECK-NEXT:    fminnm d0, d0, d1
 ; CHECK-NEXT:    fcvtzu w0, d0
 ; CHECK-NEXT:    ret
     %x = call i19 @llvm.fptoui.sat.i19.f64(double %f)
     ret i19 %x
 }
 
 define i32 @test_unsigned_i32_f64(double %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i32_f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #281474974613504
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16879, lsl #48
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d1, x8
 ; CHECK-NEXT:    fminnm d0, d0, d1
 ; CHECK-NEXT:    fcvtzu w0, d0
 ; CHECK-NEXT:    ret
     %x = call i32 @llvm.fptoui.sat.i32.f64(double %f)
     ret i32 %x
 }
 
 define i50 @test_unsigned_i50_f64(double %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i50_f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #-8
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #17167, lsl #48
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d1, x8
 ; CHECK-NEXT:    fminnm d0, d0, d1
 ; CHECK-NEXT:    fcvtzu x0, d0
 ; CHECK-NEXT:    ret
     %x = call i50 @llvm.fptoui.sat.i50.f64(double %f)
     ret i50 %x

 declare  i64 @llvm.fptoui.sat.i64.f16 (half)
 declare i100 @llvm.fptoui.sat.i100.f16(half)
 declare i128 @llvm.fptoui.sat.i128.f16(half)
 
 define i1 @test_unsigned_i1_f16(half %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i1_f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fcvt s0, h0
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s1, #1.00000000
 ; CHECK-NEXT:    fminnm s0, s0, s1
 ; CHECK-NEXT:    fcvtzu w8, s0
 ; CHECK-NEXT:    and w0, w8, #0x1
 ; CHECK-NEXT:    ret
     %x = call i1 @llvm.fptoui.sat.i1.f16(half %f)
     ret i1 %x
 }
 
 define i8 @test_unsigned_i8_f16(half %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i8_f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fcvt s0, h0
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    mov w8, #1132396544
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s1, w8
 ; CHECK-NEXT:    fminnm s0, s0, s1
 ; CHECK-NEXT:    fcvtzu w0, s0
 ; CHECK-NEXT:    ret
     %x = call i8 @llvm.fptoui.sat.i8.f16(half %f)
     ret i8 %x
 }
 
 define i13 @test_unsigned_i13_f16(half %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i13_f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #63488
 ; CHECK-NEXT:    fcvt s0, h0
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk w8, #17919, lsl #16
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s1, w8
 ; CHECK-NEXT:    fminnm s0, s0, s1
 ; CHECK-NEXT:    fcvtzu w0, s0
 ; CHECK-NEXT:    ret
     %x = call i13 @llvm.fptoui.sat.i13.f16(half %f)
     ret i13 %x
 }
 
 define i16 @test_unsigned_i16_f16(half %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i16_f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #65280
 ; CHECK-NEXT:    fcvt s0, h0
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk w8, #18303, lsl #16
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s1, w8
 ; CHECK-NEXT:    fminnm s0, s0, s1
 ; CHECK-NEXT:    fcvtzu w0, s0
 ; CHECK-NEXT:    ret
     %x = call i16 @llvm.fptoui.sat.i16.f16(half %f)
     ret i16 %x
 }
 
 define i19 @test_unsigned_i19_f16(half %f) nounwind {
 ; CHECK-LABEL: test_unsigned_i19_f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #65504
 ; CHECK-NEXT:    fcvt s0, h0
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk w8, #18687, lsl #16
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s1, w8
 ; CHECK-NEXT:    fminnm s0, s0, s1
 ; CHECK-NEXT:    fcvtzu w0, s0
 ; CHECK-NEXT:    ret
     %x = call i19 @llvm.fptoui.sat.i19.f16(half %f)
     ret i19 %x

llvm/test/CodeGen/AArch64/fptoui-sat-vector.ll

 declare <4 x i32> @llvm.fptoui.sat.v4f64.v4i32 (<4 x double>)
 declare <5 x i32> @llvm.fptoui.sat.v5f64.v5i32 (<5 x double>)
 declare <6 x i32> @llvm.fptoui.sat.v6f64.v6i32 (<6 x double>)
 
 define <1 x i32> @test_unsigned_v1f64_v1i32(<1 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v1f64_v1i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #281474974613504
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16879, lsl #48
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d1, x8
 ; CHECK-NEXT:    fminnm d0, d0, d1
 ; CHECK-NEXT:    fcvtzu w8, d0
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    ret
     %x = call <1 x i32> @llvm.fptoui.sat.v1f64.v1i32(<1 x double> %f)
     ret <1 x i32> %x
 }
 
 define <2 x i32> @test_unsigned_v2f64_v2i32(<2 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v2f64_v2i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #281474974613504
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16879, lsl #48
 ; CHECK-NEXT:    mov d2, v0.d[1]
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d3, x8
 ; CHECK-NEXT:    fmaxnm d1, d2, d1
 ; CHECK-NEXT:    fminnm d0, d0, d3
 ; CHECK-NEXT:    fcvtzu w8, d0
 ; CHECK-NEXT:    fminnm d1, d1, d3
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    fcvtzu w8, d1
 ; CHECK-NEXT:    mov v0.s[1], w8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
     %x = call <2 x i32> @llvm.fptoui.sat.v2f64.v2i32(<2 x double> %f)
     ret <2 x i32> %x
 }
 
 define <3 x i32> @test_unsigned_v3f64_v3i32(<3 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v3f64_v3i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #281474974613504
-; CHECK-NEXT:    fmov d3, xzr
+; CHECK-NEXT:    movi d3, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16879, lsl #48
 ; CHECK-NEXT:    fmaxnm d0, d0, d3
 ; CHECK-NEXT:    fmov d4, x8
 ; CHECK-NEXT:    fmaxnm d1, d1, d3
 ; CHECK-NEXT:    fmaxnm d2, d2, d3
 ; CHECK-NEXT:    fmaxnm d3, d3, d0
 ; CHECK-NEXT:    fminnm d0, d0, d4
 ; CHECK-NEXT:    fminnm d1, d1, d4
; CHECK-NEXT:    ret
     %x = call <3 x i32> @llvm.fptoui.sat.v3f64.v3i32(<3 x double> %f)
     ret <3 x i32> %x
 }
 
 define <4 x i32> @test_unsigned_v4f64_v4i32(<4 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v4f64_v4i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #281474974613504
-; CHECK-NEXT:    fmov d2, xzr
+; CHECK-NEXT:    movi d2, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16879, lsl #48
 ; CHECK-NEXT:    mov d3, v0.d[1]
 ; CHECK-NEXT:    mov d4, v1.d[1]
 ; CHECK-NEXT:    fmaxnm d0, d0, d2
 ; CHECK-NEXT:    fmaxnm d3, d3, d2
 ; CHECK-NEXT:    fmaxnm d1, d1, d2
 ; CHECK-NEXT:    fmaxnm d2, d4, d2
 ; CHECK-NEXT:    fmov d4, x8
; CHECK-NEXT:    ret
     %x = call <4 x i32> @llvm.fptoui.sat.v4f64.v4i32(<4 x double> %f)
     ret <4 x i32> %x
 }
 
 define <5 x i32> @test_unsigned_v5f64_v5i32(<5 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v5f64_v5i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #281474974613504
-; CHECK-NEXT:    fmov d5, xzr
+; CHECK-NEXT:    movi d5, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16879, lsl #48
 ; CHECK-NEXT:    fmaxnm d0, d0, d5
 ; CHECK-NEXT:    fmov d6, x8
 ; CHECK-NEXT:    fmaxnm d1, d1, d5
 ; CHECK-NEXT:    fmaxnm d2, d2, d5
 ; CHECK-NEXT:    fmaxnm d3, d3, d5
 ; CHECK-NEXT:    fmaxnm d4, d4, d5
 ; CHECK-NEXT:    fminnm d0, d0, d6
; CHECK-NEXT:    ret
     %x = call <5 x i32> @llvm.fptoui.sat.v5f64.v5i32(<5 x double> %f)
     ret <5 x i32> %x
 }
 
 define <6 x i32> @test_unsigned_v6f64_v6i32(<6 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v6f64_v6i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #281474974613504
-; CHECK-NEXT:    fmov d6, xzr
+; CHECK-NEXT:    movi d6, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16879, lsl #48
 ; CHECK-NEXT:    fmaxnm d0, d0, d6
 ; CHECK-NEXT:    fmov d7, x8
 ; CHECK-NEXT:    fmaxnm d1, d1, d6
 ; CHECK-NEXT:    fmaxnm d2, d2, d6
 ; CHECK-NEXT:    fmaxnm d3, d3, d6
 ; CHECK-NEXT:    fmaxnm d4, d4, d6
 ; CHECK-NEXT:    fmaxnm d5, d5, d6

 declare <2 x  i64> @llvm.fptoui.sat.v2f32.v2i64 (<2 x float>)
 declare <2 x i100> @llvm.fptoui.sat.v2f32.v2i100(<2 x float>)
 declare <2 x i128> @llvm.fptoui.sat.v2f32.v2i128(<2 x float>)
 
 define <2 x i1> @test_unsigned_v2f32_v2i1(<2 x float> %f) {
 ; CHECK-LABEL: test_unsigned_v2f32_v2i1:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    fmov s2, #1.00000000
 ; CHECK-NEXT:    mov s3, v0.s[1]
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmaxnm s1, s3, s1
 ; CHECK-NEXT:    fminnm s0, s0, s2
 ; CHECK-NEXT:    fcvtzu w8, s0
 ; CHECK-NEXT:    fminnm s1, s1, s2
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    fcvtzu w8, s1
 ; CHECK-NEXT:    mov v0.s[1], w8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
     %x = call <2 x i1> @llvm.fptoui.sat.v2f32.v2i1(<2 x float> %f)
     ret <2 x i1> %x
 }
 
 define <2 x i8> @test_unsigned_v2f32_v2i8(<2 x float> %f) {
 ; CHECK-LABEL: test_unsigned_v2f32_v2i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    mov w8, #1132396544
 ; CHECK-NEXT:    mov s2, v0.s[1]
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s3, w8
 ; CHECK-NEXT:    fmaxnm s1, s2, s1
 ; CHECK-NEXT:    fminnm s0, s0, s3
 ; CHECK-NEXT:    fcvtzu w8, s0
 ; CHECK-NEXT:    fminnm s1, s1, s3
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    fcvtzu w8, s1
 ; CHECK-NEXT:    mov v0.s[1], w8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
     %x = call <2 x i8> @llvm.fptoui.sat.v2f32.v2i8(<2 x float> %f)
     ret <2 x i8> %x
 }
 
 define <2 x i13> @test_unsigned_v2f32_v2i13(<2 x float> %f) {
 ; CHECK-LABEL: test_unsigned_v2f32_v2i13:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #63488
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk w8, #17919, lsl #16
 ; CHECK-NEXT:    mov s2, v0.s[1]
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s3, w8
 ; CHECK-NEXT:    fmaxnm s1, s2, s1
 ; CHECK-NEXT:    fminnm s0, s0, s3
 ; CHECK-NEXT:    fcvtzu w8, s0
 ; CHECK-NEXT:    fminnm s1, s1, s3
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    fcvtzu w8, s1
 ; CHECK-NEXT:    mov v0.s[1], w8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
     %x = call <2 x i13> @llvm.fptoui.sat.v2f32.v2i13(<2 x float> %f)
     ret <2 x i13> %x
 }
 
 define <2 x i16> @test_unsigned_v2f32_v2i16(<2 x float> %f) {
 ; CHECK-LABEL: test_unsigned_v2f32_v2i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #65280
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk w8, #18303, lsl #16
 ; CHECK-NEXT:    mov s2, v0.s[1]
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s3, w8
 ; CHECK-NEXT:    fmaxnm s1, s2, s1
 ; CHECK-NEXT:    fminnm s0, s0, s3
 ; CHECK-NEXT:    fcvtzu w8, s0
 ; CHECK-NEXT:    fminnm s1, s1, s3
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    fcvtzu w8, s1
 ; CHECK-NEXT:    mov v0.s[1], w8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
     %x = call <2 x i16> @llvm.fptoui.sat.v2f32.v2i16(<2 x float> %f)
     ret <2 x i16> %x
 }
 
 define <2 x i19> @test_unsigned_v2f32_v2i19(<2 x float> %f) {
 ; CHECK-LABEL: test_unsigned_v2f32_v2i19:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #65504
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    fmov s1, wzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk w8, #18687, lsl #16
 ; CHECK-NEXT:    mov s2, v0.s[1]
 ; CHECK-NEXT:    fmaxnm s0, s0, s1
 ; CHECK-NEXT:    fmov s3, w8
 ; CHECK-NEXT:    fmaxnm s1, s2, s1
 ; CHECK-NEXT:    fminnm s0, s0, s3
 ; CHECK-NEXT:    fcvtzu w8, s0
 ; CHECK-NEXT:    fminnm s1, s1, s3

 declare <2 x  i50> @llvm.fptoui.sat.v2f64.v2i50 (<2 x double>)
 declare <2 x  i64> @llvm.fptoui.sat.v2f64.v2i64 (<2 x double>)
 declare <2 x i100> @llvm.fptoui.sat.v2f64.v2i100(<2 x double>)
 declare <2 x i128> @llvm.fptoui.sat.v2f64.v2i128(<2 x double>)
 
 define <2 x i1> @test_unsigned_v2f64_v2i1(<2 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v2f64_v2i1:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    fmov d2, #1.00000000
 ; CHECK-NEXT:    mov d3, v0.d[1]
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmaxnm d1, d3, d1
 ; CHECK-NEXT:    fminnm d0, d0, d2
 ; CHECK-NEXT:    fcvtzu w8, d0
 ; CHECK-NEXT:    fminnm d1, d1, d2
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    fcvtzu w8, d1
 ; CHECK-NEXT:    mov v0.s[1], w8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
     %x = call <2 x i1> @llvm.fptoui.sat.v2f64.v2i1(<2 x double> %f)
     ret <2 x i1> %x
 }
 
 define <2 x i8> @test_unsigned_v2f64_v2i8(<2 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v2f64_v2i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #246290604621824
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16495, lsl #48
 ; CHECK-NEXT:    mov d2, v0.d[1]
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d3, x8
 ; CHECK-NEXT:    fmaxnm d1, d2, d1
 ; CHECK-NEXT:    fminnm d0, d0, d3
 ; CHECK-NEXT:    fcvtzu w8, d0
 ; CHECK-NEXT:    fminnm d1, d1, d3
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    fcvtzu w8, d1
 ; CHECK-NEXT:    mov v0.s[1], w8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
     %x = call <2 x i8> @llvm.fptoui.sat.v2f64.v2i8(<2 x double> %f)
     ret <2 x i8> %x
 }
 
 define <2 x i13> @test_unsigned_v2f64_v2i13(<2 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v2f64_v2i13:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #280375465082880
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16575, lsl #48
 ; CHECK-NEXT:    mov d2, v0.d[1]
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d3, x8
 ; CHECK-NEXT:    fmaxnm d1, d2, d1
 ; CHECK-NEXT:    fminnm d0, d0, d3
 ; CHECK-NEXT:    fcvtzu w8, d0
 ; CHECK-NEXT:    fminnm d1, d1, d3
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    fcvtzu w8, d1
 ; CHECK-NEXT:    mov v0.s[1], w8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
     %x = call <2 x i13> @llvm.fptoui.sat.v2f64.v2i13(<2 x double> %f)
     ret <2 x i13> %x
 }
 
 define <2 x i16> @test_unsigned_v2f64_v2i16(<2 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v2f64_v2i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #281337537757184
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16623, lsl #48
 ; CHECK-NEXT:    mov d2, v0.d[1]
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d3, x8
 ; CHECK-NEXT:    fmaxnm d1, d2, d1
 ; CHECK-NEXT:    fminnm d0, d0, d3
 ; CHECK-NEXT:    fcvtzu w8, d0
 ; CHECK-NEXT:    fminnm d1, d1, d3
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    fcvtzu w8, d1
 ; CHECK-NEXT:    mov v0.s[1], w8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
     %x = call <2 x i16> @llvm.fptoui.sat.v2f64.v2i16(<2 x double> %f)
     ret <2 x i16> %x
 }
 
 define <2 x i19> @test_unsigned_v2f64_v2i19(<2 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v2f64_v2i19:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #281457796841472
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16671, lsl #48
 ; CHECK-NEXT:    mov d2, v0.d[1]
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d3, x8
 ; CHECK-NEXT:    fmaxnm d1, d2, d1
 ; CHECK-NEXT:    fminnm d0, d0, d3
 ; CHECK-NEXT:    fcvtzu w8, d0
 ; CHECK-NEXT:    fminnm d1, d1, d3
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    fcvtzu w8, d1
 ; CHECK-NEXT:    mov v0.s[1], w8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
     %x = call <2 x i19> @llvm.fptoui.sat.v2f64.v2i19(<2 x double> %f)
     ret <2 x i19> %x
 }
 
 define <2 x i32> @test_unsigned_v2f64_v2i32_duplicate(<2 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v2f64_v2i32_duplicate:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #281474974613504
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #16879, lsl #48
 ; CHECK-NEXT:    mov d2, v0.d[1]
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d3, x8
 ; CHECK-NEXT:    fmaxnm d1, d2, d1
 ; CHECK-NEXT:    fminnm d0, d0, d3
 ; CHECK-NEXT:    fcvtzu w8, d0
 ; CHECK-NEXT:    fminnm d1, d1, d3
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    fcvtzu w8, d1
 ; CHECK-NEXT:    mov v0.s[1], w8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
     %x = call <2 x i32> @llvm.fptoui.sat.v2f64.v2i32(<2 x double> %f)
     ret <2 x i32> %x
 }
 
 define <2 x i50> @test_unsigned_v2f64_v2i50(<2 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v2f64_v2i50:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #-8
-; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    movi d1, #0000000000000000
 ; CHECK-NEXT:    movk x8, #17167, lsl #48
 ; CHECK-NEXT:    mov d2, v0.d[1]
 ; CHECK-NEXT:    fmaxnm d0, d0, d1
 ; CHECK-NEXT:    fmov d3, x8
 ; CHECK-NEXT:    fmaxnm d1, d2, d1
 ; CHECK-NEXT:    fminnm d0, d0, d3
 ; CHECK-NEXT:    fcvtzu x8, d0
 ; CHECK-NEXT:    fminnm d1, d1, d3

 define <4 x i1> @test_unsigned_v4f16_v4i1(<4 x half> %f) {
 ; CHECK-LABEL: test_unsigned_v4f16_v4i1:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    fcvt s1, h0
 ; CHECK-NEXT:    mov h3, v0.h[1]
 ; CHECK-NEXT:    mov h4, v0.h[2]
 ; CHECK-NEXT:    mov h0, v0.h[3]
-; CHECK-NEXT:    fmov s2, wzr
+; CHECK-NEXT:    movi d2, #0000000000000000
 ; CHECK-NEXT:    fcvt s3, h3
 ; CHECK-NEXT:    fcvt s4, h4
 ; CHECK-NEXT:    fcvt s0, h0
 ; CHECK-NEXT:    fmaxnm s1, s1, s2
 ; CHECK-NEXT:    fmaxnm s3, s3, s2
 ; CHECK-NEXT:    fmaxnm s4, s4, s2
 ; CHECK-NEXT:    fmaxnm s0, s0, s2
 ; CHECK-NEXT:    fmov s2, #1.00000000

 define <4 x i8> @test_unsigned_v4f16_v4i8(<4 x half> %f) {
 ; CHECK-LABEL: test_unsigned_v4f16_v4i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    fcvt s1, h0
 ; CHECK-NEXT:    mov h3, v0.h[1]
 ; CHECK-NEXT:    mov h4, v0.h[2]
 ; CHECK-NEXT:    mov h0, v0.h[3]
-; CHECK-NEXT:    fmov s2, wzr
+; CHECK-NEXT:    movi d2, #0000000000000000
 ; CHECK-NEXT:    mov w8, #1132396544
 ; CHECK-NEXT:    fcvt s3, h3
 ; CHECK-NEXT:    fcvt s4, h4
 ; CHECK-NEXT:    fcvt s0, h0
 ; CHECK-NEXT:    fmaxnm s1, s1, s2
 ; CHECK-NEXT:    fmaxnm s3, s3, s2
 ; CHECK-NEXT:    fmaxnm s4, s4, s2
 ; CHECK-NEXT:    fmaxnm s0, s0, s2

 ; CHECK-LABEL: test_unsigned_v4f16_v4i13:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    fcvt s1, h0
 ; CHECK-NEXT:    mov w8, #63488
 ; CHECK-NEXT:    mov h3, v0.h[1]
 ; CHECK-NEXT:    mov h4, v0.h[2]
 ; CHECK-NEXT:    mov h0, v0.h[3]
-; CHECK-NEXT:    fmov s2, wzr
+; CHECK-NEXT:    movi d2, #0000000000000000
 ; CHECK-NEXT:    movk w8, #17919, lsl #16
 ; CHECK-NEXT:    fcvt s3, h3
 ; CHECK-NEXT:    fcvt s4, h4
 ; CHECK-NEXT:    fcvt s0, h0
 ; CHECK-NEXT:    fmaxnm s1, s1, s2
 ; CHECK-NEXT:    fmaxnm s3, s3, s2
 ; CHECK-NEXT:    fmaxnm s4, s4, s2
 ; CHECK-NEXT:    fmaxnm s0, s0, s2

 ; CHECK-LABEL: test_unsigned_v4f16_v4i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    fcvt s1, h0
 ; CHECK-NEXT:    mov w8, #65280
 ; CHECK-NEXT:    mov h3, v0.h[1]
 ; CHECK-NEXT:    mov h4, v0.h[2]
 ; CHECK-NEXT:    mov h0, v0.h[3]
-; CHECK-NEXT:    fmov s2, wzr
+; CHECK-NEXT:    movi d2, #0000000000000000
 ; CHECK-NEXT:    movk w8, #18303, lsl #16
 ; CHECK-NEXT:    fcvt s3, h3
 ; CHECK-NEXT:    fcvt s4, h4
 ; CHECK-NEXT:    fcvt s0, h0
 ; CHECK-NEXT:    fmaxnm s1, s1, s2
 ; CHECK-NEXT:    fmaxnm s3, s3, s2
 ; CHECK-NEXT:    fmaxnm s4, s4, s2
 ; CHECK-NEXT:    fmaxnm s0, s0, s2

 ; CHECK-LABEL: test_unsigned_v4f16_v4i19:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    fcvt s1, h0
 ; CHECK-NEXT:    mov w8, #65504
 ; CHECK-NEXT:    mov h3, v0.h[1]
 ; CHECK-NEXT:    mov h4, v0.h[2]
 ; CHECK-NEXT:    mov h0, v0.h[3]
-; CHECK-NEXT:    fmov s2, wzr
+; CHECK-NEXT:    movi d2, #0000000000000000
 ; CHECK-NEXT:    movk w8, #18687, lsl #16
 ; CHECK-NEXT:    fcvt s3, h3
 ; CHECK-NEXT:    fcvt s4, h4
 ; CHECK-NEXT:    fcvt s0, h0
 ; CHECK-NEXT:    fmaxnm s1, s1, s2
 ; CHECK-NEXT:    fmaxnm s3, s3, s2
 ; CHECK-NEXT:    fmaxnm s4, s4, s2
 ; CHECK-NEXT:    fmaxnm s0, s0, s2

llvm/test/CodeGen/AArch64/remat-float0.ll

 ; RUN: llc < %s -mtriple=aarch64-none-linux-gnu -verify-machineinstrs | FileCheck %s
 
-; Check that float 0 gets rematerialized with an fmov of zero reg instead
+; Check that float 0 gets rematerialized with an "movi zero" instead
 ; of spilled/filled.
 
 declare void @bar(float)
 
 define void @foo() {
 ; CHECK-LABEL: foo:
-; CHECK: fmov s0, wzr
+; CHECK: movi    d0, #0000000000000000
 ; CHECK: bl bar
-; CHECK: fmov s0, wzr
+; CHECK: movi    d0, #0000000000000000
 ; CHECK: bl bar
   call void @bar(float 0.000000e+00)
   call void asm sideeffect "", "~{s0},~{s1},~{s2},~{s3},~{s4},~{s5},~{s6},~{s7},~{s8},~{s9},~{s10},~{s11},~{s12},~{s13},~{s14},~{s15},~{s16},~{s17},~{s18},~{s19},~{s20},~{s21},~{s22},~{s23},~{s24},~{s25},~{s26},~{s27},~{s28},~{s29},~{s30},~{s31}"()
   call void @bar(float 0.000000e+00)
   ret void
 }