[X86] Replacing X86-specific floor and ceil vector intrinsics with generic LLVM intrinsics

[mike.dvoretsky]

Summary

Currently, X86 floor and ceil intrinsics for vectors are implemented as target-specific intrinsics that use the generic rounding instruction of the corresponding vector processing feature (ROUND* or VRNDSCALE*). This patch replaces those specific cases with calls to target-independent @llvm.floor.* and @llvm.ceil.* intrinsics. This doesn't affect the resulting machine code, as those intrinsics are lowered to the same instructions, but exposes these specific rounding cases to generic optimizations.

This patch also has an LLVM part, D45203. An alternative InstCombine-based implementation is proposed in D48067.

[craig.topper]

include/clang/Basic/BuiltinsX86.def

951 ↗

(On Diff #140745)

I'd prefer CGBuiltin to detect the specific immediates on the rndscale value. Primarily because we should be able to optimize _mm512_roundscale_pd when the ceil/floor immediate is used.

[mike.dvoretsky]

On suggestion from @craig.topper moved all lowering to CGBuiltin.cpp with no new builtins added. Instead the existing builtins are lowered if their immediate values correspond to generic ceil and floor operations. D45203 is now required to enable transformations.

[craig.topper]

What about rndscaless/rndscalesd?

clang/lib/CodeGen/CGBuiltin.cpp

8477

I'm not sure we should even try to emit a mask for the legacy scalar intrinsics. Does this get removed well by the middle or backend?

8490

Why Int32? That's not the right mask width for the legacy intrinsics.

[mike.dvoretsky]

clang/lib/CodeGen/CGBuiltin.cpp

8477

The masking is done to represent all operations handled here in a uniform way. D45203 removes it in the backend.

[craig.topper]

But it’s not really consistent because the mask is being removed early for the packed intrinsics, but late for the scalar intrinsics. Doesn’t it also introduce extra code for fast isel?

[craig.topper]

There's a similar patch for sqrt here https://reviews.llvm.org/D41168 and it uses a scalar sqrt and insert element for the scalar case. I think we need a consistent direction here.

[mike.dvoretsky]

Changed the scalar intrinsic lowering to work via extract-insert. D45203 contains tests for folding the resulting IR patterns.

[craig.topper]

I'm not sure whether we should be doing this here or in InstCombine. @spatel, what do you think?

[spatel]

In D45202#1126616, @craig.topper wrote:

I'm not sure whether we should be doing this here or in InstCombine. @spatel, what do you think?

It's been a while since I looked at these. Last memory I have is for the conversion from x86 masked ops to the generic LLVM intrinsics, and we did that in InstCombineCalls. I don't know if there was any sound reasoning for that though. If it makes no functional difference, I'd continue with that structure just so we don't become scattered in the transform.

[mike.dvoretsky]

Abandoning this due to D48067 being accepted instead.

clang/lib/CodeGen/CGBuiltin.cpp

static Value *EmitX86MaskedCompare(CodeGenFunction &CGF, unsigned CC,
   return EmitX86MaskedCompareResult(CGF, Cmp, NumElts, MaskIn);
 }
 
 static Value *EmitX86ConvertToMask(CodeGenFunction &CGF, Value *In) {
   Value *Zero = Constant::getNullValue(In->getType());
   return EmitX86MaskedCompare(CGF, 1, true, { In, Zero });
 }
 
+static Value *EmitX86Round(CodeGenFunction &CGF, ArrayRef<Value *> Ops,
+                           unsigned BuiltinID) {
+  int RoundControl;
+  if (BuiltinID == clang::X86::BI__builtin_ia32_roundss ||
+      BuiltinID == clang::X86::BI__builtin_ia32_roundsd)
+    RoundControl = cast<ConstantInt>(Ops[2])->getSExtValue();
+  else if (BuiltinID == clang::X86::BI__builtin_ia32_rndscalesd_round_mask ||
+           BuiltinID == clang::X86::BI__builtin_ia32_rndscaless_round_mask)
+    RoundControl = cast<ConstantInt>(Ops[4])->getSExtValue();
+  else
+    RoundControl = cast<ConstantInt>(Ops[1])->getSExtValue();
+
+  int SAE;
+  if (BuiltinID == clang::X86::BI__builtin_ia32_rndscaleps_mask ||
+      BuiltinID == clang::X86::BI__builtin_ia32_rndscalepd_mask)
+    SAE = cast<ConstantInt>(Ops[4])->getSExtValue();
+  else if (BuiltinID == clang::X86::BI__builtin_ia32_rndscalesd_round_mask ||
+           BuiltinID == clang::X86::BI__builtin_ia32_rndscaless_round_mask)
+    SAE = cast<ConstantInt>(Ops[5])->getSExtValue();
+  else
+    SAE = 4;
+
+  if (SAE != 4 || (RoundControl != 2 /*ceil*/ && RoundControl != 1 /*floor*/)) {
+    Intrinsic::ID ID;
+    switch (BuiltinID) {
+    default: llvm_unreachable("Unsupported intrinsic!");
+    case clang::X86::BI__builtin_ia32_roundps:
+      ID = Intrinsic::x86_sse41_round_ps;
+      break;
+    case clang::X86::BI__builtin_ia32_roundss:
+      ID = Intrinsic::x86_sse41_round_ss;
+      break;
+    case clang::X86::BI__builtin_ia32_roundsd:
+      ID = Intrinsic::x86_sse41_round_sd;
+      break;
+    case clang::X86::BI__builtin_ia32_roundpd:
+      ID = Intrinsic::x86_sse41_round_pd;
+      break;
+    case clang::X86::BI__builtin_ia32_roundpd256:
+      ID = Intrinsic::x86_avx_round_pd_256;
+      break;
+    case clang::X86::BI__builtin_ia32_roundps256:
+      ID = Intrinsic::x86_avx_round_ps_256;
+      break;
+    case clang::X86::BI__builtin_ia32_rndscaleps_mask:
+      ID = Intrinsic::x86_avx512_mask_rndscale_ps_512;
+      break;
+    case clang::X86::BI__builtin_ia32_rndscalepd_mask:
+      ID = Intrinsic::x86_avx512_mask_rndscale_pd_512;
+      break;
+    case clang::X86::BI__builtin_ia32_rndscalepd_128_mask:
+      ID = Intrinsic::x86_avx512_mask_rndscale_pd_128;
+      break;
+    case clang::X86::BI__builtin_ia32_rndscalepd_256_mask:
+      ID = Intrinsic::x86_avx512_mask_rndscale_pd_256;
+      break;
+    case clang::X86::BI__builtin_ia32_rndscaleps_128_mask:
+      ID = Intrinsic::x86_avx512_mask_rndscale_ps_128;
+      break;
+    case clang::X86::BI__builtin_ia32_rndscaleps_256_mask:
+      ID = Intrinsic::x86_avx512_mask_rndscale_ps_256;
+      break;
+    case clang::X86::BI__builtin_ia32_rndscalesd_round_mask:
+      ID = Intrinsic::x86_avx512_mask_rndscale_sd;
+      break;
+    case clang::X86::BI__builtin_ia32_rndscaless_round_mask:
+      ID = Intrinsic::x86_avx512_mask_rndscale_ss;
+      break;
+    }
+    llvm::Function *F = CGF.CGM.getIntrinsic(ID);
+    return CGF.Builder.CreateCall(F, Ops);
+  }
+
+  Value *Src, *Dst, *Mask;
+  bool IsScalar = false;
+  if (BuiltinID == clang::X86::BI__builtin_ia32_roundss ||
+      BuiltinID == clang::X86::BI__builtin_ia32_roundsd ||
+      BuiltinID == clang::X86::BI__builtin_ia32_rndscalesd_round_mask ||
+      BuiltinID == clang::X86::BI__builtin_ia32_rndscaless_round_mask) {
+    IsScalar = true;
+    if (BuiltinID == clang::X86::BI__builtin_ia32_rndscalesd_round_mask ||
+        BuiltinID == clang::X86::BI__builtin_ia32_rndscaless_round_mask) {
+      llvm::Type *MaskTy = Ops[3]->getType();
+      llvm::Type *I32Ty = CGF.Builder.getInt32Ty();
+      Value *One = llvm::ConstantInt::get(I32Ty, 1);
+      Value *Zero = llvm::Constant::getNullValue(I32Ty);
+      Mask = (MaskTy == I32Ty) ? Ops[3] : CGF.Builder.CreateZExt(Ops[3], I32Ty);
+      Mask = CGF.Builder.CreateAnd(Mask, One);
+      Mask = CGF.Builder.CreateICmp(ICmpInst::ICMP_NE, Mask, Zero);
+      Dst = Ops[2];
+    }
+    else
+      Dst = Ops[0];
+    Src = CGF.Builder.CreateExtractElement(Ops[1], (uint64_t)0);
+  } else {
+    Src = Ops[0];
+    if (BuiltinID == clang::X86::BI__builtin_ia32_rndscaleps_mask ||
+        BuiltinID == clang::X86::BI__builtin_ia32_rndscalepd_mask ||
+        BuiltinID == clang::X86::BI__builtin_ia32_rndscalepd_128_mask ||
+        BuiltinID == clang::X86::BI__builtin_ia32_rndscalepd_256_mask ||
+        BuiltinID == clang::X86::BI__builtin_ia32_rndscaleps_128_mask ||
+        BuiltinID == clang::X86::BI__builtin_ia32_rndscaleps_256_mask) {
+      Dst = Ops[2];
+      Mask = Ops[3];
+    } else {
+      Dst = Src;
+      Mask = llvm::ConstantInt::getAllOnesValue(CGF.Builder.getIntNTy(Src->getType()->getVectorNumElements()));
+    }
+  }
+
+  Intrinsic::ID ID = (RoundControl == 2) ? Intrinsic::ceil : Intrinsic::floor;
+  Value *F = CGF.CGM.getIntrinsic(ID, Src->getType());
+  Value *Res = CGF.Builder.CreateCall(F, {Src});
+  if (!IsScalar)
+    return EmitX86Select(CGF, Mask, Res, Dst);
+  if (BuiltinID == clang::X86::BI__builtin_ia32_rndscalesd_round_mask ||
+      BuiltinID == clang::X86::BI__builtin_ia32_rndscaless_round_mask) {
+    Dst = CGF.Builder.CreateExtractElement(Dst, (uint64_t)0);
+    Res = CGF.Builder.CreateSelect(Mask, Res, Dst);
+    Dst = Ops[0];
+  }
+  return CGF.Builder.CreateInsertElement(Dst, Res, (uint64_t)0);
+}
+
 static Value *EmitX86Abs(CodeGenFunction &CGF, ArrayRef<Value *> Ops) {
 
   llvm::Type *Ty = Ops[0]->getType();
   Value *Zero = llvm::Constant::getNullValue(Ty);
   Value *Sub = CGF.Builder.CreateSub(Zero, Ops[0]);
   Value *Cmp = CGF.Builder.CreateICmp(ICmpInst::ICMP_SGT, Ops[0], Zero);
   Value *Res = CGF.Builder.CreateSelect(Cmp, Ops[0], Sub);
   return Res;
RHS = Builder.CreateShuffleVector(RHS, RHS,
                                       makeArrayRef(Indices, NumElts / 2));
     // Concat the vectors.
     // NOTE: Operands are swapped to match the intrinsic definition.
     Value *Res = Builder.CreateShuffleVector(RHS, LHS,
                                              makeArrayRef(Indices, NumElts));
     return Builder.CreateBitCast(Res, Ops[0]->getType());
   }
 
+  case X86::BI__builtin_ia32_roundps:
+  case X86::BI__builtin_ia32_roundss:
+  case X86::BI__builtin_ia32_roundsd:
+  case X86::BI__builtin_ia32_roundpd:
+  case X86::BI__builtin_ia32_roundpd256:
+  case X86::BI__builtin_ia32_roundps256:
+  case X86::BI__builtin_ia32_rndscaleps_mask:
+  case X86::BI__builtin_ia32_rndscalepd_mask:
+  case X86::BI__builtin_ia32_rndscalepd_128_mask:
+  case X86::BI__builtin_ia32_rndscalepd_256_mask:
+  case X86::BI__builtin_ia32_rndscaleps_128_mask:
+  case X86::BI__builtin_ia32_rndscaleps_256_mask:
+  case X86::BI__builtin_ia32_rndscalesd_round_mask:
+  case X86::BI__builtin_ia32_rndscaless_round_mask:
+    return EmitX86Round(*this, Ops, BuiltinID);
+
   case X86::BI__builtin_ia32_vplzcntd_128:
   case X86::BI__builtin_ia32_vplzcntd_256:
   case X86::BI__builtin_ia32_vplzcntd_512:
   case X86::BI__builtin_ia32_vplzcntq_128:
   case X86::BI__builtin_ia32_vplzcntq_256:
   case X86::BI__builtin_ia32_vplzcntq_512: {
     Function *F = CGM.getIntrinsic(Intrinsic::ctlz, Ops[0]->getType());
     return Builder.CreateCall(F, {Ops[0],Builder.getInt1(false)});

clang/test/CodeGen/avx-builtins.c

 __m128i test_mm256_castsi256_si128(__m256i A) {
   // CHECK-LABEL: test_mm256_castsi256_si128
   // CHECK: shufflevector <4 x i64> %{{.*}}, <4 x i64> %{{.*}}, <2 x i32> <i32 0, i32 1>
   return _mm256_castsi256_si128(A);
 }
 
 __m256d test_mm256_ceil_pd(__m256d x) {
   // CHECK-LABEL: test_mm256_ceil_pd
-  // CHECK: call <4 x double> @llvm.x86.avx.round.pd.256(<4 x double> %{{.*}}, i32 2)
+  // CHECK: @llvm.ceil.v4f64
+  // CHECK-NOT: select
   return _mm256_ceil_pd(x);
 }
 
 __m256 test_mm_ceil_ps(__m256 x) {
   // CHECK-LABEL: test_mm_ceil_ps
-  // CHECK: call <8 x float> @llvm.x86.avx.round.ps.256(<8 x float> %{{.*}}, i32 2)
+  // CHECK: @llvm.ceil.v8f32
+  // CHECK-NOT: select
   return _mm256_ceil_ps(x);
 }
 
 __m128d test_mm_cmp_pd(__m128d A, __m128d B) {
   // CHECK-LABEL: test_mm_cmp_pd
   // CHECK: call <2 x double> @llvm.x86.sse2.cmp.pd(<2 x double> %{{.*}}, <2 x double> %{{.*}}, i8 13)
   return _mm_cmp_pd(A, B, _CMP_GE_OS);
 }

 __m128i test_mm256_extractf128_si256(__m256i A) {
   // CHECK-LABEL: test_mm256_extractf128_si256
   // CHECK: shufflevector <4 x i64> %{{.*}}, <4 x i64> zeroinitializer, <2 x i32> <i32 2, i32 3>
   return _mm256_extractf128_si256(A, 1);
 }
 
 __m256d test_mm256_floor_pd(__m256d x) {
   // CHECK-LABEL: test_mm256_floor_pd
-  // CHECK: call <4 x double> @llvm.x86.avx.round.pd.256(<4 x double> %{{.*}}, i32 1)
+  // CHECK: @llvm.floor.v4f64
+  // CHECK-NOT: select
   return _mm256_floor_pd(x);
 }
 
 __m256 test_mm_floor_ps(__m256 x) {
   // CHECK-LABEL: test_mm_floor_ps
-  // CHECK: call <8 x float> @llvm.x86.avx.round.ps.256(<8 x float> %{{.*}}, i32 1)
+  // CHECK: @llvm.floor.v8f32
+  // CHECK-NOT: select
   return _mm256_floor_ps(x);
 }
 
 __m256d test_mm256_hadd_pd(__m256d A, __m256d B) {
   // CHECK-LABEL: test_mm256_hadd_pd
   // CHECK: call <4 x double> @llvm.x86.avx.hadd.pd.256(<4 x double> %{{.*}}, <4 x double> %{{.*}})
   return _mm256_hadd_pd(A, B);
 }

clang/test/CodeGen/avx512f-builtins.c

 
 __m512 test_mm512_min_round_ps(__m512 __A,__m512 __B)
 {
   // CHECK-LABEL: @test_mm512_min_round_ps
   // CHECK: @llvm.x86.avx512.mask.min.ps.512
   return _mm512_min_round_ps(__A,__B,_MM_FROUND_CUR_DIRECTION);
 }
 
+__m512 test_mm512_floor_ps(__m512 __A)
+{
+  // CHECK-LABEL: @test_mm512_floor_ps
+  // CHECK: @llvm.floor.v16f32
+  // CHECK-NOT: select
+  return _mm512_floor_ps(__A);
+}
+
+__m512d test_mm512_floor_pd(__m512d __A)
+{
+  // CHECK-LABEL: @test_mm512_floor_pd
+  // CHECK: @llvm.floor.v8f64
+  // CHECK-NOT: select
+  return _mm512_floor_pd(__A);
+}
+
 __m512 test_mm512_mask_floor_ps (__m512 __W, __mmask16 __U, __m512 __A)
 {
-  // CHECK-LABEL: @test_mm512_mask_floor_ps 
-  // CHECK: @llvm.x86.avx512.mask.rndscale.ps.512
+  // CHECK-LABEL: @test_mm512_mask_floor_ps
+  // CHECK: @llvm.floor.v16f32
+  // CHECK: select <16 x i1> %{{.*}}, <16 x float> %{{.*}}, <16 x float> %{{.*}}
   return _mm512_mask_floor_ps (__W,__U,__A);
 }
 
 __m512d test_mm512_mask_floor_pd (__m512d __W, __mmask8 __U, __m512d __A)
 {
-  // CHECK-LABEL: @test_mm512_mask_floor_pd 
-  // CHECK: @llvm.x86.avx512.mask.rndscale.pd.512
+  // CHECK-LABEL: @test_mm512_mask_floor_pd
+  // CHECK: @llvm.floor.v8f64
+  // CHECK: select <8 x i1> %{{.*}}, <8 x double> %{{.*}}, <8 x double> %{{.*}}
   return _mm512_mask_floor_pd (__W,__U,__A);
 }
 
+__m512 test_mm512_ceil_ps(__m512 __A)
+{
+  // CHECK-LABEL: @test_mm512_ceil_ps
+  // CHECK: @llvm.ceil.v16f32
+  // CHECK-NOT: select
+  return _mm512_ceil_ps(__A);
+}
+
+__m512d test_mm512_ceil_pd(__m512d __A)
+{
+  // CHECK-LABEL: @test_mm512_ceil_pd
+  // CHECK: @llvm.ceil.v8f64
+  // CHECK-NOT: select
+  return _mm512_ceil_pd(__A);
+}
+
 __m512 test_mm512_mask_ceil_ps (__m512 __W, __mmask16 __U, __m512 __A)
 {
-  // CHECK-LABEL: @test_mm512_mask_ceil_ps 
-  // CHECK: @llvm.x86.avx512.mask.rndscale.ps.512
+  // CHECK-LABEL: @test_mm512_mask_ceil_ps
+  // CHECK: @llvm.ceil.v16f32
+  // CHECK: select <16 x i1> %{{.*}}, <16 x float> %{{.*}}, <16 x float> %{{.*}}
   return _mm512_mask_ceil_ps (__W,__U,__A);
 }
 
 __m512d test_mm512_mask_ceil_pd (__m512d __W, __mmask8 __U, __m512d __A)
 {
-  // CHECK-LABEL: @test_mm512_mask_ceil_pd 
-  // CHECK: @llvm.x86.avx512.mask.rndscale.pd.512
+  // CHECK-LABEL: @test_mm512_mask_ceil_pd
+  // CHECK: @llvm.ceil.v8f64
+  // CHECK: select <8 x i1> %{{.*}}, <8 x double> %{{.*}}, <8 x double> %{{.*}}
   return _mm512_mask_ceil_pd (__W,__U,__A);
 }
 
 __m512 test_mm512_mask_roundscale_ps(__m512 __W, __mmask16 __U, __m512 __A) 
 {
   // CHECK-LABEL: @test_mm512_mask_roundscale_ps
   // CHECK: @llvm.x86.avx512.mask.rndscale.ps.512
+  return _mm512_mask_roundscale_ps(__W,__U,__A, 3);
+}
+
+__m512 test_mm512_mask_roundscale_floor_ps(__m512 __W, __mmask16 __U, __m512 __A) 
+{
+  // CHECK-LABEL: @test_mm512_mask_roundscale_floor_ps
+  // CHECK: @llvm.floor.v16f32
+  // CHECK: select <16 x i1> %{{.*}}, <16 x float> %{{.*}}, <16 x float> %{{.*}}
   return _mm512_mask_roundscale_ps(__W,__U,__A, 1);
 }
 
+__m512 test_mm512_mask_roundscale_ceil_ps(__m512 __W, __mmask16 __U, __m512 __A) 
+{
+  // CHECK-LABEL: @test_mm512_mask_roundscale_ceil_ps
+  // CHECK: @llvm.ceil.v16f32
+  // CHECK: select <16 x i1> %{{.*}}, <16 x float> %{{.*}}, <16 x float> %{{.*}}
+  return _mm512_mask_roundscale_ps(__W,__U,__A, 2);
+}
+
 __m512 test_mm512_maskz_roundscale_ps(__mmask16 __U, __m512 __A) 
 {
   // CHECK-LABEL: @test_mm512_maskz_roundscale_ps
   // CHECK: @llvm.x86.avx512.mask.rndscale.ps.512
-  return _mm512_maskz_roundscale_ps(__U,__A, 1);
+  return _mm512_maskz_roundscale_ps(__U,__A, 3);
 }
 
 __m512 test_mm512_mask_roundscale_round_ps(__m512 __A,__mmask16 __U,__m512 __C)
 {
   // CHECK-LABEL: @test_mm512_mask_roundscale_round_ps
   // CHECK: @llvm.x86.avx512.mask.rndscale.ps.512
   return _mm512_mask_roundscale_round_ps(__A,__U,__C,3,_MM_FROUND_CUR_DIRECTION);
 }
__m512 test_mm512_roundscale_round_ps(__m512 __A)
   // CHECK: @llvm.x86.avx512.mask.rndscale.ps.512
   return _mm512_roundscale_round_ps(__A,3,_MM_FROUND_CUR_DIRECTION);
 }
 
 __m512d test_mm512_mask_roundscale_pd(__m512d __W, __mmask8 __U, __m512d __A) 
 {
   // CHECK-LABEL: @test_mm512_mask_roundscale_pd
   // CHECK: @llvm.x86.avx512.mask.rndscale.pd.512
-  return _mm512_mask_roundscale_pd(__W,__U,__A, 1);
+  return _mm512_mask_roundscale_pd(__W,__U,__A, 3);
 }
 
 __m512d test_mm512_maskz_roundscale_pd(__mmask8 __U, __m512d __A) 
 {
   // CHECK-LABEL: @test_mm512_maskz_roundscale_pd
   // CHECK: @llvm.x86.avx512.mask.rndscale.pd.512
-  return _mm512_maskz_roundscale_pd(__U,__A, 1);
+  return _mm512_maskz_roundscale_pd(__U,__A, 3);
 }
 
 __m512d test_mm512_mask_roundscale_round_pd(__m512d __A,__mmask8 __U,__m512d __C)
 {
   // CHECK-LABEL: @test_mm512_mask_roundscale_round_pd
   // CHECK: @llvm.x86.avx512.mask.rndscale.pd.512
   return _mm512_mask_roundscale_round_pd(__A,__U,__C,3,_MM_FROUND_CUR_DIRECTION);
 }

clang/test/CodeGen/sse41-builtins.c

 __m128 test_mm_blendv_ps(__m128 V1, __m128 V2, __m128 V3) {
   // CHECK-LABEL: test_mm_blendv_ps
   // CHECK: call <4 x float> @llvm.x86.sse41.blendvps(<4 x float> %{{.*}}, <4 x float> %{{.*}}, <4 x float> %{{.*}})
   return _mm_blendv_ps(V1, V2, V3);
 }
 
 __m128d test_mm_ceil_pd(__m128d x) {
   // CHECK-LABEL: test_mm_ceil_pd
-  // CHECK: call <2 x double> @llvm.x86.sse41.round.pd(<2 x double> %{{.*}}, i32 2)
+  // CHECK: @llvm.ceil.v2f64
+  // CHECK-NOT: select
   return _mm_ceil_pd(x);
 }
 
 __m128 test_mm_ceil_ps(__m128 x) {
   // CHECK-LABEL: test_mm_ceil_ps
-  // CHECK: call <4 x float> @llvm.x86.sse41.round.ps(<4 x float> %{{.*}}, i32 2)
+  // CHECK: @llvm.ceil.v4f32
+  // CHECK-NOT: select
   return _mm_ceil_ps(x);
 }
 
 __m128d test_mm_ceil_sd(__m128d x, __m128d y) {
   // CHECK-LABEL: test_mm_ceil_sd
-  // CHECK: call <2 x double> @llvm.x86.sse41.round.sd(<2 x double> %{{.*}}, <2 x double> %{{.*}}, i32 2)
+  // CHECK: extractelement
+  // CHECK: @llvm.ceil.f64
+  // CHECK: insertelement
   return _mm_ceil_sd(x, y);
 }
 
 __m128 test_mm_ceil_ss(__m128 x, __m128 y) {
   // CHECK-LABEL: test_mm_ceil_ss
-  // CHECK: call <4 x float> @llvm.x86.sse41.round.ss(<4 x float> %{{.*}}, <4 x float> %{{.*}}, i32 2)
+  // CHECK: extractelement
+  // CHECK: @llvm.ceil.f32
+  // CHECK: insertelement
   return _mm_ceil_ss(x, y);
 }
 
 __m128i test_mm_cmpeq_epi64(__m128i A, __m128i B) {
   // CHECK-LABEL: test_mm_cmpeq_epi64
   // CHECK: icmp eq <2 x i64>
   // CHECK: sext <2 x i1> %{{.*}} to <2 x i64>
   return _mm_cmpeq_epi64(A, B);

 int test_mm_extract_ps(__m128 x) {
   // CHECK-LABEL: test_mm_extract_ps
   // CHECK: extractelement <4 x float> %{{.*}}, i32 1
   return _mm_extract_ps(x, 1);
 }
 
 __m128d test_mm_floor_pd(__m128d x) {
   // CHECK-LABEL: test_mm_floor_pd
-  // CHECK: call <2 x double> @llvm.x86.sse41.round.pd(<2 x double> %{{.*}}, i32 1)
+  // CHECK: @llvm.floor.v2f64
+  // CHECK-NOT: select
   return _mm_floor_pd(x);
 }
 
 __m128 test_mm_floor_ps(__m128 x) {
   // CHECK-LABEL: test_mm_floor_ps
-  // CHECK: call <4 x float> @llvm.x86.sse41.round.ps(<4 x float> %{{.*}}, i32 1)
+  // CHECK: @llvm.floor.v4f32
+  // CHECK-NOT: select
   return _mm_floor_ps(x);
 }
 
 __m128d test_mm_floor_sd(__m128d x, __m128d y) {
   // CHECK-LABEL: test_mm_floor_sd
-  // CHECK: call <2 x double> @llvm.x86.sse41.round.sd(<2 x double> %{{.*}}, <2 x double> %{{.*}}, i32 1)
+  // CHECK: extractelement
+  // CHECK: @llvm.floor.f64
+  // CHECK: insertelement
   return _mm_floor_sd(x, y);
 }
 
 __m128 test_mm_floor_ss(__m128 x, __m128 y) {
   // CHECK-LABEL: test_mm_floor_ss
-  // CHECK: call <4 x float> @llvm.x86.sse41.round.ss(<4 x float> %{{.*}}, <4 x float> %{{.*}}, i32 1)
+  // CHECK: extractelement
+  // CHECK: @llvm.floor.f32
+  // CHECK: insertelement
   return _mm_floor_ss(x, y);
 }
 
 __m128i test_mm_insert_epi8(__m128i x, char b) {
   // CHECK-LABEL: test_mm_insert_epi8
   // CHECK: insertelement <16 x i8> %{{.*}}, i8 %{{.*}}, i32 0
   return _mm_insert_epi8(x, b, 16);
 }