[clang][NewPM] Add new pass manager RUN lines to avx512f-builtins.c

[craig.topper]

Summary

This is split from D63174 to see what it takes to get both pass managers to do the same thing

I've disable -O0 optnone and ran instsimplify on both outputs. This seems to get us converged with some test updates. The only thing I don't like is the tests of true and false comparison predicates with masking. The only IR we ended up with is just loads and stores. The cmp and AND instruction are folded out by instsimplify.

I'll do more tests from D63174 if we think this is a decent direction

[leonardchan]

Any updates on this? I'm thinking that in the meantime maybe we could commit D63174 and work on this while that lands. If so, we could get an upstream new PM buildbot that can catch any new PM regressions.

[craig.topper]

@chandlerc ping

[spatel]

I skimmed D63174 but haven't applied either of these patches to test locally, so I may not have the full picture.

IMO, we do not want clang regression tests running -instcombine/-instsimplify. That can cause clang tests to break when an underlying LLVM change is made. Forcing LLVM devs to depend on clang and fix the resulting breakage is backwards and unexpected extra work. This has happened to me several times.

As a compromise to the -O0 IR explosion, we do have precedent for running the optimizer's -mem2reg pass since that doesn't change frequently at this point.

And I haven't tried this, but we do have utils/update_cc_test_checks.py - this is supposed to take the manual labor out of generating assertions in the same way that we do in the optimizer and codegen regression tests with utils/update_test_checks.py and utils/update_llc_test_checks.py. Can you start with that and remove the irrelevant CHECK lines, so only the common/important lines remain? Or just use independent FileCheck '--check-prefixes'?

[lebedev.ri]

In D63638#1560846, @spatel wrote:

I skimmed D63174 but haven't applied either of these patches to test locally, so I may not have the full picture.

IMO, we do not want clang regression tests running -instcombine/-instsimplify. That can cause clang tests to break when an underlying LLVM change is made. Forcing LLVM devs to depend on clang and fix the resulting breakage is backwards and unexpected extra work.

+1

This has happened to me several times.

As a compromise to the -O0 IR explosion, we do have precedent for running the optimizer's -mem2reg pass since that doesn't change frequently at this point.

And I haven't tried this, but we do have utils/update_cc_test_checks.py - this is supposed to take the manual labor out of generating assertions in the same way that we do in the optimizer and codegen regression tests with utils/update_test_checks.py and utils/update_llc_test_checks.py. Can you start with that and remove the irrelevant CHECK lines, so only the common/important lines remain? Or just use independent FileCheck '--check-prefixes'?

That script has bitrot and is unusable last time i checked; everyone preferred to manually write broken checklines here :)

[craig.topper]

In D63638#1560846, @spatel wrote:

I skimmed D63174 but haven't applied either of these patches to test locally, so I may not have the full picture.

IMO, we do not want clang regression tests running -instcombine/-instsimplify. That can cause clang tests to break when an underlying LLVM change is made. Forcing LLVM devs to depend on clang and fix the resulting breakage is backwards and unexpected extra work. This has happened to me several times.

As a compromise to the -O0 IR explosion, we do have precedent for running the optimizer's -mem2reg pass since that doesn't change frequently at this point.

And I haven't tried this, but we do have utils/update_cc_test_checks.py - this is supposed to take the manual labor out of generating assertions in the same way that we do in the optimizer and codegen regression tests with utils/update_test_checks.py and utils/update_llc_test_checks.py. Can you start with that and remove the irrelevant CHECK lines, so only the common/important lines remain? Or just use independent FileCheck '--check-prefixes'?

I definitely agree running -instcombine would be bad since it can replace squences with other sequences. -instsimplify is a little less scary because our intrinsic tests shouldn't really have a lot of things that are trivially reducible. Though that may not be as true as I want it to be. The main issue we seemed to need -instsimplify for with the new pass manager is to merge redundant bitcasts. The inliner in the old pass manager seemed to do that itself, but the new pass manager's always inliner doesn't.

[leonardchan]

In D63638#1560991, @craig.topper wrote:

In D63638#1560846, @spatel wrote:

I skimmed D63174 but haven't applied either of these patches to test locally, so I may not have the full picture.

IMO, we do not want clang regression tests running -instcombine/-instsimplify. That can cause clang tests to break when an underlying LLVM change is made. Forcing LLVM devs to depend on clang and fix the resulting breakage is backwards and unexpected extra work. This has happened to me several times.

As a compromise to the -O0 IR explosion, we do have precedent for running the optimizer's -mem2reg pass since that doesn't change frequently at this point.

And I haven't tried this, but we do have utils/update_cc_test_checks.py - this is supposed to take the manual labor out of generating assertions in the same way that we do in the optimizer and codegen regression tests with utils/update_test_checks.py and utils/update_llc_test_checks.py. Can you start with that and remove the irrelevant CHECK lines, so only the common/important lines remain? Or just use independent FileCheck '--check-prefixes'?

I definitely agree running -instcombine would be bad since it can replace squences with other sequences. -instsimplify is a little less scary because our intrinsic tests shouldn't really have a lot of things that are trivially reducible. Though that may not be as true as I want it to be. The main issue we seemed to need -instsimplify for with the new pass manager is to merge redundant bitcasts. The inliner in the old pass manager seemed to do that itself, but the new pass manager's always inliner doesn't.

I think it could be that the new PM Inliner isn't added to the pipeline at -O0. It only seems to be added during optimized runs. @chandlerc might know if this was intentional or not. If so, perhaps these bitcasts are intended and the new PM is still doing its job in this case.

[craig.topper]

There's some inliner running because the intrinsics are implemented as always_inline functions and they are clearly being inlined in -O0. In a previous post, Chandler said the new PM has a special inliner for always_inline in -O0 and the old pass manager just used the normal inliner.

[leonardchan]

In D63638#1574373, @craig.topper wrote:

There's some inliner running because the intrinsics are implemented as always_inline functions and they are clearly being inlined in -O0. In a previous post, Chandler said the new PM has a special inliner for always_inline in -O0 and the old pass manager just used the normal inliner.

Oh I forgot that these were marked always_inline. Yes, this special inliner is the AlwaysInliner which is purposefully designed differently than the normal inliner in the legacy PM according to D23299. I'm proposing that we could perhaps just edit the tests to ignore the bitcasts since the different behavior is intended (the AlwaysInliner isn't doing extra work like combining these bitcasts). This way we can still check for the various intrinsics emitted without their IR instruction mappings getting optimized out, and we won't need to use instsimplify to make sure the IR matches.

Taking an example from my other patch, we'd have something like:

diff --git a/clang/test/CodeGen/avx512f-builtins.c b/clang/test/CodeGen/avx512f-builtins.c
index 15571b639b6..4ad63d73235 100644
--- a/clang/test/CodeGen/avx512f-builtins.c
+++ b/clang/test/CodeGen/avx512f-builtins.c
@@ -10479,7 +10479,7 @@ __m512i test_mm512_maskz_abs_epi64 (__mmask8 __U, __m512i __A)
   // CHECK: [[SUB:%.*]] = sub <8 x i64> zeroinitializer, [[A:%.*]]
   // CHECK: [[CMP:%.*]] = icmp sgt <8 x i64> [[A]], zeroinitializer
   // CHECK: [[SEL:%.*]] = select <8 x i1> [[CMP]], <8 x i64> [[A]], <8 x i64> [[SUB]]
-  // CHECK: select <8 x i1> %{{.*}}, <8 x i64> [[SEL]], <8 x i64> %{{.*}}
+  // CHECK: select <8 x i1> %{{.*}}, <8 x i64> {{.*}}, <8 x i64> %{{.*}}  // Ignore the output of the redundant bitcasts
   return _mm512_maskz_abs_epi64 (__U,__A);
 }

It also seems like for some of these tests that some bitcasts are already ignored.

[craig.topper]

In D63638#1574740, @leonardchan wrote:

In D63638#1574373, @craig.topper wrote:

There's some inliner running because the intrinsics are implemented as always_inline functions and they are clearly being inlined in -O0. In a previous post, Chandler said the new PM has a special inliner for always_inline in -O0 and the old pass manager just used the normal inliner.

Oh I forgot that these were marked always_inline. Yes, this special inliner is the AlwaysInliner which is purposefully designed differently than the normal inliner in the legacy PM according to D23299. I'm proposing that we could perhaps just edit the tests to ignore the bitcasts since the different behavior is intended (the AlwaysInliner isn't doing extra work like combining these bitcasts). This way we can still check for the various intrinsics emitted without their IR instruction mappings getting optimized out, and we won't need to use instsimplify to make sure the IR matches.

Taking an example from my other patch, we'd have something like:

diff --git a/clang/test/CodeGen/avx512f-builtins.c b/clang/test/CodeGen/avx512f-builtins.c
index 15571b639b6..4ad63d73235 100644
--- a/clang/test/CodeGen/avx512f-builtins.c
+++ b/clang/test/CodeGen/avx512f-builtins.c
@@ -10479,7 +10479,7 @@ __m512i test_mm512_maskz_abs_epi64 (__mmask8 __U, __m512i __A)
   // CHECK: [[SUB:%.*]] = sub <8 x i64> zeroinitializer, [[A:%.*]]
   // CHECK: [[CMP:%.*]] = icmp sgt <8 x i64> [[A]], zeroinitializer
   // CHECK: [[SEL:%.*]] = select <8 x i1> [[CMP]], <8 x i64> [[A]], <8 x i64> [[SUB]]
-  // CHECK: select <8 x i1> %{{.*}}, <8 x i64> [[SEL]], <8 x i64> %{{.*}}
+  // CHECK: select <8 x i1> %{{.*}}, <8 x i64> {{.*}}, <8 x i64> %{{.*}}  // Ignore the output of the redundant bitcasts
   return _mm512_maskz_abs_epi64 (__U,__A);
 }

It also seems like for some of these tests that some bitcasts are already ignored.

That would allow the select operands to be completely reversed silently. I'll admit that the intrinsics tests probably already have cases where the checks are weak, but we shouldn't lower the quality of the ones that do better checking already.

[craig.topper]

What if we just only check the output from the new pass manager. I don't think I care about the differences between the two.

[leonardchan]

In D63638#1574927, @craig.topper wrote:

What if we just only check the output from the new pass manager. I don't think I care about the differences between the two.

+1

[leonardchan]

*ping* Is it ok to proceed with only checking the new PM output for these tests? If so I could just edit my previous patch to remove the legacy PM run lines since they already include the bitcasts from the new PM.

[chandlerc]

Just to make sure we're on the same page (and sorry I didn't jump in sooner)...

With the old PM, *anything* that is always_inline *gets* instsimplify run on it, even at -O0, even if you didn't want that. So using -instsimplify explicitly is, IMO, not any more scary of a reliance on LLVM's behavior than the old PM already subjected us to...

That said, if the x86 maintainers are comfortable with *only* using the new PM (because it has an always inliner that literally does nothing else and thus has an absolute minimum amount of LLVM transformations applied), I certainly don't have any objections. =D

[craig.topper]

In D63638#1581973, @chandlerc wrote:

Just to make sure we're on the same page (and sorry I didn't jump in sooner)...

With the old PM, *anything* that is always_inline *gets* instsimplify run on it, even at -O0, even if you didn't want that. So using -instsimplify explicitly is, IMO, not any more scary of a reliance on LLVM's behavior than the old PM already subjected us to...

That said, if the x86 maintainers are comfortable with *only* using the new PM (because it has an always inliner that literally does nothing else and thus has an absolute minimum amount of LLVM transformations applied), I certainly don't have any objections. =D

My assumption is that eventually there will only be the "new PM". So eventually we'll only be testing that PM. So I don't have any issue testing only it now.

[leonardchan]

I created D65110 if we're ok with just using the new PM.

[RKSimon]

In D63638#1596313, @leonardchan wrote:

I created D65110 if we're ok with just using the new PM.

@craig.topper Is this patch still relevant?

[craig.topper]

I don't think so.

clang/test/CodeGen/avx512f-builtins.c

-// RUN: %clang_cc1 -ffreestanding %s -triple=x86_64-apple-darwin -target-feature +avx512f -emit-llvm -o - -Wall -Werror | FileCheck %s
-// RUN: %clang_cc1 -fms-extensions -fms-compatibility -ffreestanding %s -triple=x86_64-windows-msvc -target-feature +avx512f -emit-llvm -o - -Wall -Werror | FileCheck %s
+// RUN: %clang_cc1 -fno-experimental-new-pass-manager -ffreestanding %s -triple=x86_64-apple-darwin -target-feature +avx512f -emit-llvm -o - -Wall -Werror -disable-O0-optnone | opt -instsimplify -S | FileCheck %s
+// RUN: %clang_cc1 -fno-experimental-new-pass-manager -fms-extensions -fms-compatibility -ffreestanding %s -triple=x86_64-windows-msvc -target-feature +avx512f -emit-llvm -o - -Wall -Werror -disable-O0-optnone | opt -instsimplify -S | FileCheck %s
+
+// RUN: %clang_cc1 -fexperimental-new-pass-manager -ffreestanding %s -triple=x86_64-apple-darwin -target-feature +avx512f -emit-llvm -o - -Wall -Werror -disable-O0-optnone | opt -instsimplify -S | FileCheck %s
+// RUN: %clang_cc1 -fexperimental-new-pass-manager -fms-extensions -fms-compatibility -ffreestanding %s -triple=x86_64-windows-msvc -target-feature +avx512f -emit-llvm -o - -Wall -Werror -disable-O0-optnone | opt -instsimplify -S | FileCheck %s
 
 #include <immintrin.h>
 
 __m512d test_mm512_sqrt_pd(__m512d a)
 {
   // CHECK-LABEL: @test_mm512_sqrt_pd
   // CHECK: call <8 x double> @llvm.sqrt.v8f64(<8 x double> %{{.*}})
   return _mm512_sqrt_pd(a);

 __mmask16 test_mm512_cmp_ps_mask_ngt_us(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_ngt_us
   // CHECK: fcmp ule <16 x float> %{{.*}}, %{{.*}}
   return _mm512_cmp_ps_mask(a, b, _CMP_NGT_US);
 }
 
 __mmask16 test_mm512_cmp_ps_mask_false_oq(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_false_oq
-  // CHECK: fcmp false <16 x float> %{{.*}}, %{{.*}}
+  // CHECK: ret i16 0
   return _mm512_cmp_ps_mask(a, b, _CMP_FALSE_OQ);
 }
 
 __mmask16 test_mm512_cmp_ps_mask_neq_oq(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_neq_oq
   // CHECK: fcmp one <16 x float> %{{.*}}, %{{.*}}
   return _mm512_cmp_ps_mask(a, b, _CMP_NEQ_OQ);
 }
 
 __mmask16 test_mm512_cmp_ps_mask_ge_os(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_ge_os
   // CHECK: fcmp oge <16 x float> %{{.*}}, %{{.*}}
   return _mm512_cmp_ps_mask(a, b, _CMP_GE_OS);
 }
 
 __mmask16 test_mm512_cmp_ps_mask_gt_os(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_gt_os
   // CHECK: fcmp ogt <16 x float> %{{.*}}, %{{.*}}
   return _mm512_cmp_ps_mask(a, b, _CMP_GT_OS);
 }
 
 __mmask16 test_mm512_cmp_ps_mask_true_uq(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_true_uq
-  // CHECK: fcmp true <16 x float> %{{.*}}, %{{.*}}
+  // CHECK: ret i16 -1
   return _mm512_cmp_ps_mask(a, b, _CMP_TRUE_UQ);
 }
 
 __mmask16 test_mm512_cmp_ps_mask_eq_os(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_eq_os
   // CHECK: fcmp oeq <16 x float> %{{.*}}, %{{.*}}
   return _mm512_cmp_ps_mask(a, b, _CMP_EQ_OS);
 }

 __mmask16 test_mm512_cmp_ps_mask_ngt_uq(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_ngt_uq
   // CHECK: fcmp ule <16 x float> %{{.*}}, %{{.*}}
   return _mm512_cmp_ps_mask(a, b, _CMP_NGT_UQ);
 }
 
 __mmask16 test_mm512_cmp_ps_mask_false_os(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_false_os
-  // CHECK: fcmp false <16 x float> %{{.*}}, %{{.*}}
+  // CHECK: ret i16 0
   return _mm512_cmp_ps_mask(a, b, _CMP_FALSE_OS);
 }
 
 __mmask16 test_mm512_cmp_ps_mask_neq_os(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_neq_os
   // CHECK: fcmp one <16 x float> %{{.*}}, %{{.*}}
   return _mm512_cmp_ps_mask(a, b, _CMP_NEQ_OS);
 }
 
 __mmask16 test_mm512_cmp_ps_mask_ge_oq(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_ge_oq
   // CHECK: fcmp oge <16 x float> %{{.*}}, %{{.*}}
   return _mm512_cmp_ps_mask(a, b, _CMP_GE_OQ);
 }
 
 __mmask16 test_mm512_cmp_ps_mask_gt_oq(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_gt_oq
   // CHECK: fcmp ogt <16 x float> %{{.*}}, %{{.*}}
   return _mm512_cmp_ps_mask(a, b, _CMP_GT_OQ);
 }
 
 __mmask16 test_mm512_cmp_ps_mask_true_us(__m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_cmp_ps_mask_true_us
-  // CHECK: fcmp true <16 x float> %{{.*}}, %{{.*}}
+  // CHECK: ret i16 -1
   return _mm512_cmp_ps_mask(a, b, _CMP_TRUE_US);
 }
 
 __mmask16 test_mm512_mask_cmp_ps_mask_eq_oq(__mmask16 m, __m512 a, __m512 b) {
   // CHECK-LABEL: @test_mm512_mask_cmp_ps_mask_eq_oq
   // CHECK: [[CMP:%.*]] = fcmp oeq <16 x float> %{{.*}}, %{{.*}}
   // CHECK: and <16 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_ps_mask(m, a, b, _CMP_EQ_OQ);
__mmask16 test_mm512_mask_cmp_ps_mask_ngt_us(__mmask16 m, __m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_mask_cmp_ps_mask_ngt_us
   // CHECK: [[CMP:%.*]] = fcmp ule <16 x float> %{{.*}}, %{{.*}}
   // CHECK: and <16 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_ps_mask(m, a, b, _CMP_NGT_US);
 }
 
 __mmask16 test_mm512_mask_cmp_ps_mask_false_oq(__mmask16 m, __m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_mask_cmp_ps_mask_false_oq
-  // CHECK: [[CMP:%.*]] = fcmp false <16 x float> %{{.*}}, %{{.*}}
-  // CHECK: and <16 x i1> [[CMP]], {{.*}}
+  // FIXME: How to test?
   return _mm512_mask_cmp_ps_mask(m, a, b, _CMP_FALSE_OQ);
 }
 
 __mmask16 test_mm512_mask_cmp_ps_mask_neq_oq(__mmask16 m, __m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_mask_cmp_ps_mask_neq_oq
   // CHECK: [[CMP:%.*]] = fcmp one <16 x float> %{{.*}}, %{{.*}}
   // CHECK: and <16 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_ps_mask(m, a, b, _CMP_NEQ_OQ);
__mmask16 test_mm512_mask_cmp_ps_mask_gt_os(__mmask16 m, __m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_mask_cmp_ps_mask_gt_os
   // CHECK: [[CMP:%.*]] = fcmp ogt <16 x float> %{{.*}}, %{{.*}}
   // CHECK: and <16 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_ps_mask(m, a, b, _CMP_GT_OS);
 }
 
 __mmask16 test_mm512_mask_cmp_ps_mask_true_uq(__mmask16 m, __m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_mask_cmp_ps_mask_true_uq
-  // CHECK: [[CMP:%.*]] = fcmp true <16 x float> %{{.*}}, %{{.*}}
-  // CHECK: and <16 x i1> [[CMP]], {{.*}}
+  // FIXME: How to test?
   return _mm512_mask_cmp_ps_mask(m, a, b, _CMP_TRUE_UQ);
 }
 
 __mmask16 test_mm512_mask_cmp_ps_mask_eq_os(__mmask16 m, __m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_mask_cmp_ps_mask_eq_os
   // CHECK: [[CMP:%.*]] = fcmp oeq <16 x float> %{{.*}}, %{{.*}}
   // CHECK: and <16 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_ps_mask(m, a, b, _CMP_EQ_OS);
__mmask16 test_mm512_mask_cmp_ps_mask_ngt_uq(__mmask16 m, __m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_mask_cmp_ps_mask_ngt_uq
   // CHECK: [[CMP:%.*]] = fcmp ule <16 x float> %{{.*}}, %{{.*}}
   // CHECK: and <16 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_ps_mask(m, a, b, _CMP_NGT_UQ);
 }
 
 __mmask16 test_mm512_mask_cmp_ps_mask_false_os(__mmask16 m, __m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_mask_cmp_ps_mask_false_os
-  // CHECK: [[CMP:%.*]] = fcmp false <16 x float> %{{.*}}, %{{.*}}
-  // CHECK: and <16 x i1> [[CMP]], {{.*}}
+  // FIXME: How to test?
   return _mm512_mask_cmp_ps_mask(m, a, b, _CMP_FALSE_OS);
 }
 
 __mmask16 test_mm512_mask_cmp_ps_mask_neq_os(__mmask16 m, __m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_mask_cmp_ps_mask_neq_os
   // CHECK: [[CMP:%.*]] = fcmp one <16 x float> %{{.*}}, %{{.*}}
   // CHECK: and <16 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_ps_mask(m, a, b, _CMP_NEQ_OS);
__mmask16 test_mm512_mask_cmp_ps_mask_gt_oq(__mmask16 m, __m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_mask_cmp_ps_mask_gt_oq
   // CHECK: [[CMP:%.*]] = fcmp ogt <16 x float> %{{.*}}, %{{.*}}
   // CHECK: and <16 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_ps_mask(m, a, b, _CMP_GT_OQ);
 }
 
 __mmask16 test_mm512_mask_cmp_ps_mask_true_us(__mmask16 m, __m512 a, __m512 b) {
   // CHECK-LABEL: test_mm512_mask_cmp_ps_mask_true_us
-  // CHECK: [[CMP:%.*]] = fcmp true <16 x float> %{{.*}}, %{{.*}}
-  // CHECK: and <16 x i1> [[CMP]], {{.*}}
+  // FIXME: How to test?
   return _mm512_mask_cmp_ps_mask(m, a, b, _CMP_TRUE_US);
 }
 
 __mmask8 test_mm512_cmp_round_pd_mask(__m512d a, __m512d b) {
   // CHECK-LABEL: @test_mm512_cmp_round_pd_mask
   // CHECK: [[CMP:%.*]] = fcmp oeq <8 x double> %{{.*}}, %{{.*}}
   return _mm512_cmp_round_pd_mask(a, b, _CMP_EQ_OQ, _MM_FROUND_NO_EXC);
 }

 __mmask8 test_mm512_cmp_pd_mask_ngt_us(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_ngt_us
   // CHECK: fcmp ule <8 x double> %{{.*}}, %{{.*}}
   return _mm512_cmp_pd_mask(a, b, _CMP_NGT_US);
 }
 
 __mmask8 test_mm512_cmp_pd_mask_false_oq(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_false_oq
-  // CHECK: fcmp false <8 x double> %{{.*}}, %{{.*}}
+  // CHECK: ret i8 0
   return _mm512_cmp_pd_mask(a, b, _CMP_FALSE_OQ);
 }
 
 __mmask8 test_mm512_cmp_pd_mask_neq_oq(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_neq_oq
   // CHECK: fcmp one <8 x double> %{{.*}}, %{{.*}}
   return _mm512_cmp_pd_mask(a, b, _CMP_NEQ_OQ);
 }
 
 __mmask8 test_mm512_cmp_pd_mask_ge_os(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_ge_os
   // CHECK: fcmp oge <8 x double> %{{.*}}, %{{.*}}
   return _mm512_cmp_pd_mask(a, b, _CMP_GE_OS);
 }
 
 __mmask8 test_mm512_cmp_pd_mask_gt_os(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_gt_os
   // CHECK: fcmp ogt <8 x double> %{{.*}}, %{{.*}}
   return _mm512_cmp_pd_mask(a, b, _CMP_GT_OS);
 }
 
 __mmask8 test_mm512_cmp_pd_mask_true_uq(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_true_uq
-  // CHECK: fcmp true <8 x double> %{{.*}}, %{{.*}}
+  // CHECK: ret i8 -1
   return _mm512_cmp_pd_mask(a, b, _CMP_TRUE_UQ);
 }
 
 __mmask8 test_mm512_cmp_pd_mask_eq_os(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_eq_os
   // CHECK: fcmp oeq <8 x double> %{{.*}}, %{{.*}}
   return _mm512_cmp_pd_mask(a, b, _CMP_EQ_OS);
 }

 __mmask8 test_mm512_cmp_pd_mask_ngt_uq(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_ngt_uq
   // CHECK: fcmp ule <8 x double> %{{.*}}, %{{.*}}
   return _mm512_cmp_pd_mask(a, b, _CMP_NGT_UQ);
 }
 
 __mmask8 test_mm512_cmp_pd_mask_false_os(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_false_os
-  // CHECK: fcmp false <8 x double> %{{.*}}, %{{.*}}
+  // CHECK: ret i8 0
   return _mm512_cmp_pd_mask(a, b, _CMP_FALSE_OS);
 }
 
 __mmask8 test_mm512_cmp_pd_mask_neq_os(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_neq_os
   // CHECK: fcmp one <8 x double> %{{.*}}, %{{.*}}
   return _mm512_cmp_pd_mask(a, b, _CMP_NEQ_OS);
 }
 
 __mmask8 test_mm512_cmp_pd_mask_ge_oq(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_ge_oq
   // CHECK: fcmp oge <8 x double> %{{.*}}, %{{.*}}
   return _mm512_cmp_pd_mask(a, b, _CMP_GE_OQ);
 }
 
 __mmask8 test_mm512_cmp_pd_mask_gt_oq(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_gt_oq
   // CHECK: fcmp ogt <8 x double> %{{.*}}, %{{.*}}
   return _mm512_cmp_pd_mask(a, b, _CMP_GT_OQ);
 }
 
 __mmask8 test_mm512_cmp_pd_mask_true_us(__m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_cmp_pd_mask_true_us
-  // CHECK: fcmp true <8 x double> %{{.*}}, %{{.*}}
+  // CHECK: ret i8 -1
   return _mm512_cmp_pd_mask(a, b, _CMP_TRUE_US);
 }
 
 __mmask8 test_mm512_mask_cmp_pd_mask_eq_oq(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: @test_mm512_mask_cmp_pd_mask_eq_oq
   // CHECK: [[CMP:%.*]] = fcmp oeq <8 x double> %{{.*}}, %{{.*}}
   // CHECK: and <8 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_pd_mask(m, a, b, _CMP_EQ_OQ);
__mmask8 test_mm512_mask_cmp_pd_mask_ngt_us(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_mask_cmp_pd_mask_ngt_us
   // CHECK: [[CMP:%.*]] = fcmp ule <8 x double> %{{.*}}, %{{.*}}
   // CHECK: and <8 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_pd_mask(m, a, b, _CMP_NGT_US);
 }
 
 __mmask8 test_mm512_mask_cmp_pd_mask_false_oq(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_mask_cmp_pd_mask_false_oq
-  // CHECK: [[CMP:%.*]] = fcmp false <8 x double> %{{.*}}, %{{.*}}
-  // CHECK: and <8 x i1> [[CMP]], {{.*}}
+  // FIXME: How to test?
   return _mm512_mask_cmp_pd_mask(m, a, b, _CMP_FALSE_OQ);
 }
 
 __mmask8 test_mm512_mask_cmp_pd_mask_neq_oq(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_mask_cmp_pd_mask_neq_oq
   // CHECK: [[CMP:%.*]] = fcmp one <8 x double> %{{.*}}, %{{.*}}
   // CHECK: and <8 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_pd_mask(m, a, b, _CMP_NEQ_OQ);
__mmask8 test_mm512_mask_cmp_pd_mask_gt_os(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_mask_cmp_pd_mask_gt_os
   // CHECK: [[CMP:%.*]] = fcmp ogt <8 x double> %{{.*}}, %{{.*}}
   // CHECK: and <8 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_pd_mask(m, a, b, _CMP_GT_OS);
 }
 
 __mmask8 test_mm512_mask_cmp_pd_mask_true_uq(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_mask_cmp_pd_mask_true_uq
-  // CHECK: [[CMP:%.*]] = fcmp true <8 x double> %{{.*}}, %{{.*}}
-  // CHECK: and <8 x i1> [[CMP]], {{.*}}
+  // FIXME: How to test?
   return _mm512_mask_cmp_pd_mask(m, a, b, _CMP_TRUE_UQ);
 }
 
 __mmask8 test_mm512_mask_cmp_pd_mask_eq_os(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_mask_cmp_pd_mask_eq_os
   // CHECK: [[CMP:%.*]] = fcmp oeq <8 x double> %{{.*}}, %{{.*}}
   // CHECK: and <8 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_pd_mask(m, a, b, _CMP_EQ_OS);
__mmask8 test_mm512_mask_cmp_pd_mask_ngt_uq(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_mask_cmp_pd_mask_ngt_uq
   // CHECK: [[CMP:%.*]] = fcmp ule <8 x double> %{{.*}}, %{{.*}}
   // CHECK: and <8 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_pd_mask(m, a, b, _CMP_NGT_UQ);
 }
 
 __mmask8 test_mm512_mask_cmp_pd_mask_false_os(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_mask_cmp_pd_mask_false_os
-  // CHECK: [[CMP:%.*]] = fcmp false <8 x double> %{{.*}}, %{{.*}}
-  // CHECK: and <8 x i1> [[CMP]], {{.*}}
+  // FIXME: How to test?
   return _mm512_mask_cmp_pd_mask(m, a, b, _CMP_FALSE_OS);
 }
 
 __mmask8 test_mm512_mask_cmp_pd_mask_neq_os(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_mask_cmp_pd_mask_neq_os
   // CHECK: [[CMP:%.*]] = fcmp one <8 x double> %{{.*}}, %{{.*}}
   // CHECK: and <8 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_pd_mask(m, a, b, _CMP_NEQ_OS);
__mmask8 test_mm512_mask_cmp_pd_mask_gt_oq(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_mask_cmp_pd_mask_gt_oq
   // CHECK: [[CMP:%.*]] = fcmp ogt <8 x double> %{{.*}}, %{{.*}}
   // CHECK: and <8 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_pd_mask(m, a, b, _CMP_GT_OQ);
 }
 
 __mmask8 test_mm512_mask_cmp_pd_mask_true_us(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: test_mm512_mask_cmp_pd_mask_true_us
-  // CHECK: [[CMP:%.*]] = fcmp true <8 x double> %{{.*}}, %{{.*}}
-  // CHECK: and <8 x i1> [[CMP]], {{.*}}
+  // FIXME: How to test?
   return _mm512_mask_cmp_pd_mask(m, a, b, _CMP_TRUE_US);
 }
 
 __mmask8 test_mm512_mask_cmp_pd_mask(__mmask8 m, __m512d a, __m512d b) {
   // CHECK-LABEL: @test_mm512_mask_cmp_pd_mask
   // CHECK: [[CMP:%.*]] = fcmp oeq <8 x double> %{{.*}}, %{{.*}}
   // CHECK: and <8 x i1> [[CMP]], {{.*}}
   return _mm512_mask_cmp_pd_mask(m, a, b, 0);
__m512i test_mm512_mask_permutexvar_epi32(__m512i __W, __mmask16 __M, __m512i __X, __m512i __Y) {
   return _mm512_mask_permutexvar_epi32(__W, __M, __X, __Y); 
 }
 
 __mmask16 test_mm512_kand(__m512i __A, __m512i __B, __m512i __C, __m512i __D, __m512i __E, __m512i __F) {
   // CHECK-LABEL: @test_mm512_kand
   // CHECK: [[LHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RES:%.*]] = and <16 x i1> [[LHS]], [[RHS]]
-  // CHECK: bitcast <16 x i1> [[RES]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[RES]]
+  // CHECK: bitcast <16 x i1> [[AND]] to i16
   return _mm512_mask_cmpneq_epu32_mask(_mm512_kand(_mm512_cmpneq_epu32_mask(__A, __B),
                                                    _mm512_cmpneq_epu32_mask(__C, __D)),
                                                    __E, __F);
 }
 
 __mmask16 test_mm512_kandn(__m512i __A, __m512i __B, __m512i __C, __m512i __D, __m512i __E, __m512i __F) {
   // CHECK-LABEL: @test_mm512_kandn
   // CHECK: [[LHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[NOT:%.*]] = xor <16 x i1> [[LHS]], <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>
   // CHECK: [[RES:%.*]] = and <16 x i1> [[NOT]], [[RHS]]
-  // CHECK: bitcast <16 x i1> [[RES]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[RES]]
+  // CHECK: bitcast <16 x i1> [[AND]] to i16
   return _mm512_mask_cmpneq_epu32_mask(_mm512_kandn(_mm512_cmpneq_epu32_mask(__A, __B),
                                                     _mm512_cmpneq_epu32_mask(__C, __D)),
                                                     __E, __F);
 }
 
 __mmask16 test_mm512_kor(__m512i __A, __m512i __B, __m512i __C, __m512i __D, __m512i __E, __m512i __F) {
   // CHECK-LABEL: @test_mm512_kor
   // CHECK: [[LHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RES:%.*]] = or <16 x i1> [[LHS]], [[RHS]]
-  // CHECK: bitcast <16 x i1> [[RES]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[RES]]
+  // CHECK: bitcast <16 x i1> [[AND]] to i16
   return _mm512_mask_cmpneq_epu32_mask(_mm512_kor(_mm512_cmpneq_epu32_mask(__A, __B),
                                                   _mm512_cmpneq_epu32_mask(__C, __D)),
                                                   __E, __F);
 }
 
 int test_mm512_kortestc(__m512i __A, __m512i __B, __m512i __C, __m512i __D) {
   // CHECK-LABEL: @test_mm512_kortestc
   // CHECK: [[LHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>

 
 __mmask16 test_mm512_kunpackb(__m512i __A, __m512i __B, __m512i __C, __m512i __D, __m512i __E, __m512i __F) {
   // CHECK-LABEL: @test_mm512_kunpackb
   // CHECK: [[LHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[LHS2:%.*]] = shufflevector <16 x i1> [[LHS]], <16 x i1> [[LHS]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   // CHECK: [[RHS2:%.*]] = shufflevector <16 x i1> [[RHS]], <16 x i1> [[RHS]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   // CHECK: [[CONCAT:%.*]] = shufflevector <8 x i1> [[RHS2]], <8 x i1> [[LHS2]], <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
-  // CHECK: bitcast <16 x i1> [[CONCAT]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[CONCAT]]
+  // CHECK: bitcast <16 x i1> [[AND]] to i16
   return _mm512_mask_cmpneq_epu32_mask(_mm512_kunpackb(_mm512_cmpneq_epu32_mask(__A, __B),
                                                        _mm512_cmpneq_epu32_mask(__C, __D)),
                                                        __E, __F);
 }
 
 __mmask16 test_mm512_kxnor(__m512i __A, __m512i __B, __m512i __C, __m512i __D, __m512i __E, __m512i __F) {
   // CHECK-LABEL: @test_mm512_kxnor
   // CHECK: [[LHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[NOT:%.*]] = xor <16 x i1> [[LHS]], <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>
   // CHECK: [[RES:%.*]] = xor <16 x i1> [[NOT]], [[RHS]]
-  // CHECK: bitcast <16 x i1> [[RES]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[RES]]
+  // CHECK: bitcast <16 x i1> [[AND]] to i16
   return _mm512_mask_cmpneq_epu32_mask(_mm512_kxnor(_mm512_cmpneq_epu32_mask(__A, __B),
                                                     _mm512_cmpneq_epu32_mask(__C, __D)),
                                                     __E, __F);
 }
 
 __mmask16 test_mm512_kxor(__m512i __A, __m512i __B, __m512i __C, __m512i __D, __m512i __E, __m512i __F) {
   // CHECK-LABEL: @test_mm512_kxor
   // CHECK: [[LHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RES:%.*]] = xor <16 x i1> [[LHS]], [[RHS]]
-  // CHECK: bitcast <16 x i1> [[RES]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[RES]]
+  // CHECK: bitcast <16 x i1> [[AND]] to i16
   return _mm512_mask_cmpneq_epu32_mask(_mm512_kxor(_mm512_cmpneq_epu32_mask(__A, __B),
                                                    _mm512_cmpneq_epu32_mask(__C, __D)),
                                                    __E, __F);
 }
 
 __mmask16 test_knot_mask16(__mmask16 a) {
   // CHECK-LABEL: @test_knot_mask16
   // CHECK: [[IN:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[NOT:%.*]] = xor <16 x i1> [[IN]], <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>
   // CHECK: bitcast <16 x i1> [[NOT]] to i16
   return _knot_mask16(a);
 }
 
 __mmask16 test_kand_mask16(__m512i __A, __m512i __B, __m512i __C, __m512i __D, __m512i __E, __m512i __F) {
   // CHECK-LABEL: @test_kand_mask16
   // CHECK: [[LHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RES:%.*]] = and <16 x i1> [[LHS]], [[RHS]]
-  // CHECK: bitcast <16 x i1> [[RES]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[RES]]
+  // CHECK: bitcast <16 x i1> [[AND]] to i16
   return _mm512_mask_cmpneq_epu32_mask(_kand_mask16(_mm512_cmpneq_epu32_mask(__A, __B),
                                                     _mm512_cmpneq_epu32_mask(__C, __D)),
                                                     __E, __F);
 }
 
 __mmask16 test_kandn_mask16(__m512i __A, __m512i __B, __m512i __C, __m512i __D, __m512i __E, __m512i __F) {
   // CHECK-LABEL: @test_kandn_mask16
   // CHECK: [[LHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[NOT:%.*]] = xor <16 x i1> [[LHS]], <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>
   // CHECK: [[RES:%.*]] = and <16 x i1> [[NOT]], [[RHS]]
-  // CHECK: bitcast <16 x i1> [[RES]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[RES]]
+  // CHECK: bitcast <16 x i1> [[AND]] to i16
   return _mm512_mask_cmpneq_epu32_mask(_kandn_mask16(_mm512_cmpneq_epu32_mask(__A, __B),
                                                      _mm512_cmpneq_epu32_mask(__C, __D)),
                                                      __E, __F);
 }
 
 __mmask16 test_kor_mask16(__m512i __A, __m512i __B, __m512i __C, __m512i __D, __m512i __E, __m512i __F) {
   // CHECK-LABEL: @test_kor_mask16
   // CHECK: [[LHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RES:%.*]] = or <16 x i1> [[LHS]], [[RHS]]
-  // CHECK: bitcast <16 x i1> [[RES]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[RES]]
+  // CHECK: bitcast <16 x i1> [[AND]] to i16
   return _mm512_mask_cmpneq_epu32_mask(_kor_mask16(_mm512_cmpneq_epu32_mask(__A, __B),
                                                    _mm512_cmpneq_epu32_mask(__C, __D)),
                                                    __E, __F);
 }
 
 __mmask16 test_kxnor_mask16(__m512i __A, __m512i __B, __m512i __C, __m512i __D, __m512i __E, __m512i __F) {
   // CHECK-LABEL: @test_kxnor_mask16
   // CHECK: [[LHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[NOT:%.*]] = xor <16 x i1> [[LHS]], <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>
   // CHECK: [[RES:%.*]] = xor <16 x i1> [[NOT]], [[RHS]]
-  // CHECK: bitcast <16 x i1> [[RES]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[RES]]
+  // CHECK: bitcast <16 x i1> [[AND]] to i16
   return _mm512_mask_cmpneq_epu32_mask(_kxnor_mask16(_mm512_cmpneq_epu32_mask(__A, __B),
                                                      _mm512_cmpneq_epu32_mask(__C, __D)),
                                                      __E, __F);
 }
 
 __mmask16 test_kxor_mask16(__m512i __A, __m512i __B, __m512i __C, __m512i __D, __m512i __E, __m512i __F) {
   // CHECK-LABEL: @test_kxor_mask16
   // CHECK: [[LHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RHS:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
   // CHECK: [[RES:%.*]] = xor <16 x i1> [[LHS]], [[RHS]]
-  // CHECK: bitcast <16 x i1> [[RES]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %1, %3
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[RES]]
+  // CHECK: bitcast <16 x i1> [[AND]] to i16
   return _mm512_mask_cmpneq_epu32_mask(_kxor_mask16(_mm512_cmpneq_epu32_mask(__A, __B),
                                                     _mm512_cmpneq_epu32_mask(__C, __D)),
                                                     __E, __F);
 }
 
 __mmask16 test_kshiftli_mask16(__m512i A, __m512i B, __m512i C, __m512i D) {
   // CHECK-LABEL: @test_kshiftli_mask16
-  // CHECK: [[VAL:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
+  // CHECK: [[VAL:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
   // CHECK: [[RES:%.*]] = shufflevector <16 x i1> zeroinitializer, <16 x i1> [[VAL]], <16 x i32> <i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30>
-  // CHECK: bitcast <16 x i1> [[RES]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[RES]]
+  // CHECK: bitcast <16 x i1> [[AND]]
   return _mm512_mask_cmpneq_epu32_mask(_kshiftli_mask16(_mm512_cmpneq_epu32_mask(A, B), 1), C, D);
 }
 
 __mmask16 test_kshiftri_mask16(__m512i A, __m512i B, __m512i C, __m512i D) {
   // CHECK-LABEL: @test_kshiftri_mask16
-  // CHECK: [[VAL:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
+  // CHECK: [[VAL:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
   // CHECK: [[RES:%.*]] = shufflevector <16 x i1> [[VAL]], <16 x i1> zeroinitializer, <16 x i32> <i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16>
-  // CHECK: bitcast <16 x i1> [[RES]] to i16
+  // CHECK: [[MASK:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK: [[AND:%.*]] = and <16 x i1> [[MASK]], [[RES]]
+  // CHECK: bitcast <16 x i1> [[AND]]
   return _mm512_mask_cmpneq_epu32_mask(_kshiftri_mask16(_mm512_cmpneq_epu32_mask(A, B), 1), C, D);
 }
 
 unsigned int test_cvtmask16_u32(__m512i A, __m512i B) {
   // CHECK-LABEL: @test_cvtmask16_u32
-  // CHECK: bitcast <16 x i1> %{{.*}} to i16
-  // CHECK: bitcast i16 %{{.*}} to <16 x i1>
+  // CHECK [[CMP:%.*]] = icmp ne <16 x i32> %{{.*}}, %{{.*}}
+  // CHECK [[CAST:%.*]] = bitcast <16 x i1> [[CMP]] to i16
   // CHECK: zext i16 %{{.*}} to i32
   return _cvtmask16_u32(_mm512_cmpneq_epu32_mask(A, B));
 }
 
 __mmask16 test_cvtu32_mask16(__m512i A, __m512i B, unsigned int C) {
   // CHECK-LABEL: @test_cvtu32_mask16
   // CHECK: trunc i32 %{{.*}} to i16
   // CHECK: bitcast i16 %{{.*}} to <16 x i1>