[AArch64] Sink operands to allow for bitselect instructions

[pranavk]

Summary

The pattern Or(And(A, MaskValue), And(B, ~MaskValue)), where ~MaskValue = Xor(MaskValue, -1) gets lowered to bitselect instruction when NEON is available. However, when this pattern is in a loop and MaskValue lives outside of the immediate basic block, instruction selection isn't able to choose bitselect and we end up with sequence of ORs and ANDs. This patch sinks such MaskValue into the basic block to allow backend to select bit select instructions.

This will solve performance bugs mentioned in this comment: https://github.com/llvm/llvm-project/issues/49305#issuecomment-1440828393

VBSL intrinsics can be found here: https://developer.arm.com/architectures/instruction-sets/intrinsics/#q=vbsl

[efriedma]

The primary tradeoff here is that existing optimizations won't understand the intrinsic... for example, we can't constant-fold, or automatically invert the mask. But making the intrinsics more predictably produce efficient sequences is probably worthwhile.

(Any other opinions here?)

[efriedma]

I guess I should note both the examples in https://github.com/llvm/llvm-project/issues/49305 could probably be fixed in other ways... we have heuristics to, for example, sink logic ops into loops when it's profitable. But that requires someone to notice the specific issues, and take the time to diagnose/fix them.

[dmgreen]

My preference would be for fixing the code we have, not introducing new intrinsics. Intrinsics act as black-boxes for the optimizer, and I'm pretty sure I've heard of cases in the past of the compiler optimizing the or/and/xor's to nicer sequences of instructions. It would be a shame to lose that.

The number of instructions in bsl is quite high compared to many intrinsics, they probably have a high chance of going wrong. But if we know of ways to optimize them (through shouldSinkOperands), then that would be my preference. It can end up helping all cases, not just the intrinsics.

[pranavk]

Change shouldSinkOperand to allow backend to generate bitselect instructions

[pranavk]

I agree. I changed the implementation to not introduce the intrinsic. I will need another change in InstCombine to handle case #1 mentioned on github bug report. I will have separate patch for it changing InstCombine. Thanks

[pranavk]

[AArch64][InstCombine] Bail out for bitselect instructions

[pranavk]

[AArch64] Change shouldSinkOperand to allow bitselect instructions

[dmgreen]

Thanks for working on this. I noticed there was another instance of vbsl being reported recently in https://github.com/llvm/llvm-project/issues/62642. Hopefully it can be addresses via extra optimizations too.

Can you add a testcase for the issues in https://github.com/llvm/llvm-project/issues/49305? And look into the existing tests.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

14343 ↗

(On Diff #521115)

That sounds like it might be a bug that happens if it tries to sink too many operands? From what I remember the order they are put into Ops might matter. And if it is sinking to the Or it might need to add both the And as well as the Not.

14346 ↗

(On Diff #521115)

I->hasOneUse()

14348 ↗

(On Diff #521115)

I->user_back();

14364 ↗

(On Diff #521115)

Can this be simplified with a m_Not matcher? In general instructions will be canonicalized so that constants are on the RHS.

If we are sinking the Not, I feel this should want to test that the pattern makes up a bsl, and if it does then sink the operand of I. i.e. something like checking that OI is m_c_Or(m_c_And(m_Value(A),m_Value(B)),m_specific(I)), and then checking that I is m_c_And(m_Not(m_Specific(A)),m_Value(D)) or the other way around.

[pranavk]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

14343 ↗

(On Diff #521115)

When I tried to sink Or, I didn't add both Ands in the Vector. So it was sinking it just before Or even though it was used by And one instruction before the location where it sunk it. So, I don't think it's a bug. I just forgot to add Ands to the vector. I tried adding Ands to the vector and it works. So I have changed my implementation to switch-case under Instruction::Or as I think that makes it easier to read this code.

14364 ↗

(On Diff #521115)

Absolutely. I didn't look close enough it seems in PatternMatcher.h file. After discovering whole bunch of pattern matchers, I have shortened/simplified this implementation using m_not, etc. Please look at the latest patch.

I have additionally added more guards/checks to prevent this from happening when one of the And, or its operands are not available in same basic block.

[pranavk]

Address reviewer comments

[pranavk]

tests coming

[pranavk]

add test

[pranavk]

More concise pattern matching

[dmgreen]

Thanks. LGTM with a few extra suggestions.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

14357 ↗

(On Diff #521545)

I'm not sure if this is necessary, so long as some of the operands can be sunk, but it is probably OK for the moment to keep as-is.

14361–14363 ↗

(On Diff #521545)

I think this can avoid the loop if we just use
Ops.push_back(&MainAnd->getOperandUse(MainAnd->getOperand(0) == IA ? 1 : 0));

llvm/test/CodeGen/AArch64/aarch64-bit-gen.ll

148 ↗

(On Diff #521545)

Can you run utils/update_llc_test_checks.py on the file, to generate the runtime checks? There will be more of them but that should be OK in this case. It doesn't looks too large.

[pranavk]

address reviewer comments

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

14361–14363 ↗

(On Diff #521545)

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

[pranavk]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

14361–14363 ↗

(On Diff #521545)

Done. this was left as part of my final refactoring. Uploaded new patch.

[dmgreen]

Thanks. LGTM

[pranavk]

I noticed there was another instance of vbsl being reported recently in https://github.com/llvm/llvm-project/issues/62642. Hopefully it can be addresses via extra optimizations too.

This is another InstCombine problem -- as soon as it sees constant, InstCombine runs demanded bits pass/analysis and tries to do clever tricks with sequence of and/and/or. We need to teach InstCombine here to not touch the sequence as we expect it to be vectorized in instruction lowering. I will try to fix this when looking at the other InstCombine problem (https://reviews.llvm.org/D150316) we are talking about.

[pranavk]

This comment was removed by pranavk.

[pranavk]

I forgot to update the differential link in the commit but this patch was merged as part of (https://github.com/llvm/llvm-project/commit/726785b1594c6b567c5c8ddd59075aee726590c6)

clang/lib/CodeGen/CGBuiltin.cpp

static const ARMVectorIntrinsicInfo AArch64SIMDIntrinsicMap[] = {
   NEONMAP2(vbcaxq_u32, aarch64_crypto_bcaxu, aarch64_crypto_bcaxs, Add1ArgType | UnsignedAlts),
   NEONMAP2(vbcaxq_u64, aarch64_crypto_bcaxu, aarch64_crypto_bcaxs, Add1ArgType | UnsignedAlts),
   NEONMAP2(vbcaxq_u8, aarch64_crypto_bcaxu, aarch64_crypto_bcaxs, Add1ArgType | UnsignedAlts),
   NEONMAP1(vbfdot_f32, aarch64_neon_bfdot, 0),
   NEONMAP1(vbfdotq_f32, aarch64_neon_bfdot, 0),
   NEONMAP1(vbfmlalbq_f32, aarch64_neon_bfmlalb, 0),
   NEONMAP1(vbfmlaltq_f32, aarch64_neon_bfmlalt, 0),
   NEONMAP1(vbfmmlaq_f32, aarch64_neon_bfmmla, 0),
+  NEONMAP1(vbsl_v, aarch64_neon_bsl, 0),
+  NEONMAP1(vbslq_v, aarch64_neon_bsl, 0),
   NEONMAP1(vcadd_rot270_f16, aarch64_neon_vcadd_rot270, Add1ArgType),
   NEONMAP1(vcadd_rot270_f32, aarch64_neon_vcadd_rot270, Add1ArgType),
   NEONMAP1(vcadd_rot90_f16, aarch64_neon_vcadd_rot90, Add1ArgType),
   NEONMAP1(vcadd_rot90_f32, aarch64_neon_vcadd_rot90, Add1ArgType),
   NEONMAP1(vcaddq_rot270_f16, aarch64_neon_vcadd_rot270, Add1ArgType),
   NEONMAP1(vcaddq_rot270_f32, aarch64_neon_vcadd_rot270, Add1ArgType),
   NEONMAP1(vcaddq_rot270_f64, aarch64_neon_vcadd_rot270, Add1ArgType),
   NEONMAP1(vcaddq_rot90_f16, aarch64_neon_vcadd_rot90, Add1ArgType),
case NEON::BI__builtin_neon_vbfdotq_f32: {
     llvm::Type *Tys[2] = { Ty, InputTy };
     return EmitNeonCall(CGM.getIntrinsic(Int, Tys), Ops, "vbfdot");
   }
   case NEON::BI__builtin_neon___a32_vcvt_bf16_f32: {
     llvm::Type *Tys[1] = { Ty };
     Function *F = CGM.getIntrinsic(Int, Tys);
     return EmitNeonCall(F, Ops, "vcvtfp2bf");
   }
-
+  case NEON::BI__builtin_neon_vbsl_v:
+  case NEON::BI__builtin_neon_vbslq_v: {
+    // Even though LLVM IR intrinsic defines vbsl with llvm_anyvector_ty, only
+    // v8i8 and v16i8 are valid candidates for codegen here.
+    auto *InputTy =
+        llvm::FixedVectorType::get(Int8Ty, Ty->getPrimitiveSizeInBits() / 8);
+    llvm::Type *Tys[1] = { InputTy };
+    Function *F = CGM.getIntrinsic(Int, Tys);
+    return EmitNeonCall(F, Ops, "vbsl");
+   }
   }
 
   assert(Int && "Expected valid intrinsic number");
 
   // Determine the type(s) of this overloaded AArch64 intrinsic.
   Function *F = LookupNeonLLVMIntrinsic(Int, Modifier, Ty, E);
 
   Value *Result = EmitNeonCall(F, Ops, NameHint);

clang/test/CodeGen/aarch64-neon-intrinsics.c

 // CHECK:   [[TMP1:%.*]] = bitcast <2 x double> %v2 to <16 x i8>
 // CHECK:   [[VABD2_I:%.*]] = call <2 x double> @llvm.aarch64.neon.fabd.v2f64(<2 x double> %v1, <2 x double> %v2)
 // CHECK:   ret <2 x double> [[VABD2_I]]
 float64x2_t test_vabdq_f64(float64x2_t v1, float64x2_t v2) {
   return vabdq_f64(v1, v2);
 }
 
 // CHECK-LABEL: @test_vbsl_s8(
-// CHECK:   [[VBSL_I:%.*]] = and <8 x i8> %v1, %v2
-// CHECK:   [[TMP0:%.*]] = xor <8 x i8> %v1, <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
-// CHECK:   [[VBSL1_I:%.*]] = and <8 x i8> [[TMP0]], %v3
-// CHECK:   [[VBSL2_I:%.*]] = or <8 x i8> [[VBSL_I]], [[VBSL1_I]]
-// CHECK:   ret <8 x i8> [[VBSL2_I]]
+// CHECK:   [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3)
+// CHECK:   ret <8 x i8> [[VBSL_I]]
 int8x8_t test_vbsl_s8(uint8x8_t v1, int8x8_t v2, int8x8_t v3) {
   return vbsl_s8(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbsl_s16(
-// CHECK:   [[TMP0:%.*]] = bitcast <4 x i16> %v1 to <8 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <4 x i16> %v2 to <8 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <4 x i16> %v3 to <8 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <4 x i16> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <4 x i16> %v1, <i16 -1, i16 -1, i16 -1, i16 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <4 x i16> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <4 x i16> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   [[TMP4:%.*]] = bitcast <4 x i16> [[VBSL5_I]] to <8 x i8>
-// CHECK:   ret <8 x i8> [[TMP4]]
+// CHECK:    [[TMP0:%.*]] = bitcast <4 x i16> %v1 to <8 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <4 x i16> %v2 to <8 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <4 x i16> %v3 to <8 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> [[TMP0]], <8 x i8> [[TMP1]], <8 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <8 x i8> [[VBSL_I]] to <4 x i16>
+// CHECK:    [[TMP4:%.*]] = bitcast <4 x i16> [[TMP3]] to <8 x i8>
+// CHECK:    ret <8 x i8> [[TMP4]]
+//
 int8x8_t test_vbsl_s16(uint16x4_t v1, int16x4_t v2, int16x4_t v3) {
   return (int8x8_t)vbsl_s16(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbsl_s32(
-// CHECK:   [[TMP0:%.*]] = bitcast <2 x i32> %v1 to <8 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <2 x i32> %v2 to <8 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <2 x i32> %v3 to <8 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <2 x i32> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <2 x i32> %v1, <i32 -1, i32 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <2 x i32> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <2 x i32> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <2 x i32> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <2 x i32> %v1 to <8 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <2 x i32> %v2 to <8 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <2 x i32> %v3 to <8 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> [[TMP0]], <8 x i8> [[TMP1]], <8 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <8 x i8> [[VBSL_I]] to <2 x i32>
+// CHECK:    ret <2 x i32> [[TMP3]]
+//
 int32x2_t test_vbsl_s32(uint32x2_t v1, int32x2_t v2, int32x2_t v3) {
   return vbsl_s32(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbsl_s64(
-// CHECK:   [[TMP0:%.*]] = bitcast <1 x i64> %v1 to <8 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <1 x i64> %v2 to <8 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <1 x i64> %v3 to <8 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <1 x i64> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <1 x i64> %v1, <i64 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <1 x i64> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <1 x i64> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <1 x i64> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <1 x i64> %v1 to <8 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <1 x i64> %v2 to <8 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <1 x i64> %v3 to <8 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> [[TMP0]], <8 x i8> [[TMP1]], <8 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <8 x i8> [[VBSL_I]] to <1 x i64>
+// CHECK:    ret <1 x i64> [[TMP3]]
+//
 int64x1_t test_vbsl_s64(uint64x1_t v1, int64x1_t v2, int64x1_t v3) {
   return vbsl_s64(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbsl_u8(
-// CHECK:   [[VBSL_I:%.*]] = and <8 x i8> %v1, %v2
-// CHECK:   [[TMP0:%.*]] = xor <8 x i8> %v1, <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
-// CHECK:   [[VBSL1_I:%.*]] = and <8 x i8> [[TMP0]], %v3
-// CHECK:   [[VBSL2_I:%.*]] = or <8 x i8> [[VBSL_I]], [[VBSL1_I]]
-// CHECK:   ret <8 x i8> [[VBSL2_I]]
+// CHECK:    [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3)
+// CHECK:    ret <8 x i8> [[VBSL_I]]
+//
 uint8x8_t test_vbsl_u8(uint8x8_t v1, uint8x8_t v2, uint8x8_t v3) {
   return vbsl_u8(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbsl_u16(
-// CHECK:   [[TMP0:%.*]] = bitcast <4 x i16> %v1 to <8 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <4 x i16> %v2 to <8 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <4 x i16> %v3 to <8 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <4 x i16> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <4 x i16> %v1, <i16 -1, i16 -1, i16 -1, i16 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <4 x i16> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <4 x i16> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <4 x i16> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <4 x i16> %v1 to <8 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <4 x i16> %v2 to <8 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <4 x i16> %v3 to <8 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> [[TMP0]], <8 x i8> [[TMP1]], <8 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <8 x i8> [[VBSL_I]] to <4 x i16>
+// CHECK:    ret <4 x i16> [[TMP3]]
+//
 uint16x4_t test_vbsl_u16(uint16x4_t v1, uint16x4_t v2, uint16x4_t v3) {
   return vbsl_u16(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbsl_u32(
-// CHECK:   [[TMP0:%.*]] = bitcast <2 x i32> %v1 to <8 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <2 x i32> %v2 to <8 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <2 x i32> %v3 to <8 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <2 x i32> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <2 x i32> %v1, <i32 -1, i32 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <2 x i32> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <2 x i32> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <2 x i32> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <2 x i32> %v1 to <8 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <2 x i32> %v2 to <8 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <2 x i32> %v3 to <8 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> [[TMP0]], <8 x i8> [[TMP1]], <8 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <8 x i8> [[VBSL_I]] to <2 x i32>
+// CHECK:    ret <2 x i32> [[TMP3]]
+//
 uint32x2_t test_vbsl_u32(uint32x2_t v1, uint32x2_t v2, uint32x2_t v3) {
   return vbsl_u32(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbsl_u64(
-// CHECK:   [[TMP0:%.*]] = bitcast <1 x i64> %v1 to <8 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <1 x i64> %v2 to <8 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <1 x i64> %v3 to <8 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <1 x i64> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <1 x i64> %v1, <i64 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <1 x i64> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <1 x i64> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <1 x i64> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <1 x i64> %v1 to <8 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <1 x i64> %v2 to <8 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <1 x i64> %v3 to <8 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> [[TMP0]], <8 x i8> [[TMP1]], <8 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <8 x i8> [[VBSL_I]] to <1 x i64>
+// CHECK:    ret <1 x i64> [[TMP3]]
+//
 uint64x1_t test_vbsl_u64(uint64x1_t v1, uint64x1_t v2, uint64x1_t v3) {
   return vbsl_u64(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbsl_f32(
-// CHECK:   [[TMP1:%.*]] = bitcast <2 x i32> %v1 to <8 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <2 x float> %v2 to <8 x i8>
-// CHECK:   [[TMP3:%.*]] = bitcast <2 x float> %v3 to <8 x i8>
-// CHECK:   [[VBSL1_I:%.*]] = bitcast <8 x i8> [[TMP2]] to <2 x i32>
-// CHECK:   [[VBSL2_I:%.*]] = bitcast <8 x i8> [[TMP3]] to <2 x i32>
-// CHECK:   [[VBSL3_I:%.*]] = and <2 x i32> %v1, [[VBSL1_I]]
-// CHECK:   [[TMP4:%.*]] = xor <2 x i32> %v1, <i32 -1, i32 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <2 x i32> [[TMP4]], [[VBSL2_I]]
-// CHECK:   [[VBSL5_I:%.*]] = or <2 x i32> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   [[TMP5:%.*]] = bitcast <2 x i32> [[VBSL5_I]] to <2 x float>
-// CHECK:   ret <2 x float> [[TMP5]]
+// CHECK:    [[TMP0:%.*]] = bitcast <2 x i32> %v1 to <8 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <2 x float> %v2 to <8 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <2 x float> %v3 to <8 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> [[TMP0]], <8 x i8> [[TMP1]], <8 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <8 x i8> [[VBSL_I]] to <2 x float>
+// CHECK:    ret <2 x float> [[TMP3]]
+//
 float32x2_t test_vbsl_f32(uint32x2_t v1, float32x2_t v2, float32x2_t v3) {
   return vbsl_f32(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbsl_f64(
-// CHECK:   [[TMP0:%.*]] = bitcast <1 x i64> %v1 to <8 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <1 x double> %v2 to <8 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <1 x double> %v3 to <8 x i8>
-// CHECK:   [[VBSL1_I:%.*]] = bitcast <8 x i8> [[TMP1]] to <1 x i64>
-// CHECK:   [[VBSL2_I:%.*]] = bitcast <8 x i8> [[TMP2]] to <1 x i64>
-// CHECK:   [[VBSL3_I:%.*]] = and <1 x i64> %v1, [[VBSL1_I]]
-// CHECK:   [[TMP3:%.*]] = xor <1 x i64> %v1, <i64 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <1 x i64> [[TMP3]], [[VBSL2_I]]
-// CHECK:   [[VBSL5_I:%.*]] = or <1 x i64> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   [[TMP4:%.*]] = bitcast <1 x i64> [[VBSL5_I]] to <1 x double>
-// CHECK:   ret <1 x double> [[TMP4]]
+// CHECK:    [[TMP0:%.*]] = bitcast <1 x i64> %v1 to <8 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <1 x double> %v2 to <8 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <1 x double> %v3 to <8 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> [[TMP0]], <8 x i8> [[TMP1]], <8 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <8 x i8> [[VBSL_I]] to <1 x double>
+// CHECK:    ret <1 x double> [[TMP3]]
+//
 float64x1_t test_vbsl_f64(uint64x1_t v1, float64x1_t v2, float64x1_t v3) {
   return vbsl_f64(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbsl_p8(
-// CHECK:   [[VBSL_I:%.*]] = and <8 x i8> %v1, %v2
-// CHECK:   [[TMP0:%.*]] = xor <8 x i8> %v1, <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
-// CHECK:   [[VBSL1_I:%.*]] = and <8 x i8> [[TMP0]], %v3
-// CHECK:   [[VBSL2_I:%.*]] = or <8 x i8> [[VBSL_I]], [[VBSL1_I]]
-// CHECK:   ret <8 x i8> [[VBSL2_I]]
+// CHECK:    [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> %v1, <8 x i8> %v2, <8 x i8> %v3)
+// CHECK:    ret <8 x i8> [[VBSL_I]]
+//
 poly8x8_t test_vbsl_p8(uint8x8_t v1, poly8x8_t v2, poly8x8_t v3) {
   return vbsl_p8(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbsl_p16(
-// CHECK:   [[TMP0:%.*]] = bitcast <4 x i16> %v1 to <8 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <4 x i16> %v2 to <8 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <4 x i16> %v3 to <8 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <4 x i16> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <4 x i16> %v1, <i16 -1, i16 -1, i16 -1, i16 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <4 x i16> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <4 x i16> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <4 x i16> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <4 x i16> %v1 to <8 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <4 x i16> %v2 to <8 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <4 x i16> %v3 to <8 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> [[TMP0]], <8 x i8> [[TMP1]], <8 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <8 x i8> [[VBSL_I]] to <4 x i16>
+// CHECK:    ret <4 x i16> [[TMP3]]
+//
 poly16x4_t test_vbsl_p16(uint16x4_t v1, poly16x4_t v2, poly16x4_t v3) {
   return vbsl_p16(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbslq_s8(
-// CHECK:   [[VBSL_I:%.*]] = and <16 x i8> %v1, %v2
-// CHECK:   [[TMP0:%.*]] = xor <16 x i8> %v1, <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
-// CHECK:   [[VBSL1_I:%.*]] = and <16 x i8> [[TMP0]], %v3
-// CHECK:   [[VBSL2_I:%.*]] = or <16 x i8> [[VBSL_I]], [[VBSL1_I]]
-// CHECK:   ret <16 x i8> [[VBSL2_I]]
+// CHECK:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> %v1, <16 x i8> %v2, <16 x i8> %v3)
+// CHECK:    ret <16 x i8> [[VBSL_I]]
+//
 int8x16_t test_vbslq_s8(uint8x16_t v1, int8x16_t v2, int8x16_t v3) {
   return vbslq_s8(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbslq_s16(
-// CHECK:   [[TMP0:%.*]] = bitcast <8 x i16> %v1 to <16 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <8 x i16> %v2 to <16 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <8 x i16> %v3 to <16 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <8 x i16> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <8 x i16> %v1, <i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <8 x i16> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <8 x i16> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <8 x i16> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <8 x i16> %v1 to <16 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <8 x i16> %v2 to <16 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <8 x i16> %v3 to <16 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> [[TMP0]], <16 x i8> [[TMP1]], <16 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <16 x i8> [[VBSL_I]] to <8 x i16>
+// CHECK:    ret <8 x i16> [[TMP3]]
+//
 int16x8_t test_vbslq_s16(uint16x8_t v1, int16x8_t v2, int16x8_t v3) {
   return vbslq_s16(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbslq_s32(
-// CHECK:   [[TMP0:%.*]] = bitcast <4 x i32> %v1 to <16 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <4 x i32> %v2 to <16 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <4 x i32> %v3 to <16 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <4 x i32> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <4 x i32> %v1, <i32 -1, i32 -1, i32 -1, i32 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <4 x i32> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <4 x i32> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <4 x i32> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <4 x i32> %v1 to <16 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <4 x i32> %v2 to <16 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <4 x i32> %v3 to <16 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> [[TMP0]], <16 x i8> [[TMP1]], <16 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <16 x i8> [[VBSL_I]] to <4 x i32>
+// CHECK:    ret <4 x i32> [[TMP3]]
+//
 int32x4_t test_vbslq_s32(uint32x4_t v1, int32x4_t v2, int32x4_t v3) {
   return vbslq_s32(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbslq_s64(
-// CHECK:   [[TMP0:%.*]] = bitcast <2 x i64> %v1 to <16 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <2 x i64> %v2 to <16 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <2 x i64> %v3 to <16 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <2 x i64> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <2 x i64> %v1, <i64 -1, i64 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <2 x i64> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <2 x i64> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <2 x i64> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <2 x i64> %v1 to <16 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <2 x i64> %v2 to <16 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <2 x i64> %v3 to <16 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> [[TMP0]], <16 x i8> [[TMP1]], <16 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <16 x i8> [[VBSL_I]] to <2 x i64>
+// CHECK:    ret <2 x i64> [[TMP3]]
+//
 int64x2_t test_vbslq_s64(uint64x2_t v1, int64x2_t v2, int64x2_t v3) {
   return vbslq_s64(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbslq_u8(
-// CHECK:   [[VBSL_I:%.*]] = and <16 x i8> %v1, %v2
-// CHECK:   [[TMP0:%.*]] = xor <16 x i8> %v1, <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
-// CHECK:   [[VBSL1_I:%.*]] = and <16 x i8> [[TMP0]], %v3
-// CHECK:   [[VBSL2_I:%.*]] = or <16 x i8> [[VBSL_I]], [[VBSL1_I]]
-// CHECK:   ret <16 x i8> [[VBSL2_I]]
+// CHECK:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> %v1, <16 x i8> %v2, <16 x i8> %v3)
+// CHECK:    ret <16 x i8> [[VBSL_I]]
+//
 uint8x16_t test_vbslq_u8(uint8x16_t v1, uint8x16_t v2, uint8x16_t v3) {
   return vbslq_u8(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbslq_u16(
-// CHECK:   [[TMP0:%.*]] = bitcast <8 x i16> %v1 to <16 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <8 x i16> %v2 to <16 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <8 x i16> %v3 to <16 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <8 x i16> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <8 x i16> %v1, <i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <8 x i16> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <8 x i16> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <8 x i16> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <8 x i16> %v1 to <16 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <8 x i16> %v2 to <16 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <8 x i16> %v3 to <16 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> [[TMP0]], <16 x i8> [[TMP1]], <16 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <16 x i8> [[VBSL_I]] to <8 x i16>
+// CHECK:    ret <8 x i16> [[TMP3]]
+//
 uint16x8_t test_vbslq_u16(uint16x8_t v1, uint16x8_t v2, uint16x8_t v3) {
   return vbslq_u16(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbslq_u32(
-// CHECK:   [[TMP0:%.*]] = bitcast <4 x i32> %v1 to <16 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <4 x i32> %v2 to <16 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <4 x i32> %v3 to <16 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <4 x i32> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <4 x i32> %v1, <i32 -1, i32 -1, i32 -1, i32 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <4 x i32> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <4 x i32> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <4 x i32> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <4 x i32> %v1 to <16 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <4 x i32> %v2 to <16 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <4 x i32> %v3 to <16 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> [[TMP0]], <16 x i8> [[TMP1]], <16 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <16 x i8> [[VBSL_I]] to <4 x i32>
+// CHECK:    ret <4 x i32> [[TMP3]]
+//
 int32x4_t test_vbslq_u32(uint32x4_t v1, int32x4_t v2, int32x4_t v3) {
   return vbslq_s32(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbslq_u64(
-// CHECK:   [[TMP0:%.*]] = bitcast <2 x i64> %v1 to <16 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <2 x i64> %v2 to <16 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <2 x i64> %v3 to <16 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <2 x i64> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <2 x i64> %v1, <i64 -1, i64 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <2 x i64> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <2 x i64> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <2 x i64> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <2 x i64> %v1 to <16 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <2 x i64> %v2 to <16 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <2 x i64> %v3 to <16 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> [[TMP0]], <16 x i8> [[TMP1]], <16 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <16 x i8> [[VBSL_I]] to <2 x i64>
+// CHECK:    ret <2 x i64> [[TMP3]]
+//
 uint64x2_t test_vbslq_u64(uint64x2_t v1, uint64x2_t v2, uint64x2_t v3) {
   return vbslq_u64(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbslq_f32(
-// CHECK:   [[TMP0:%.*]] = bitcast <4 x i32> %v1 to <16 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <4 x float> %v2 to <16 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <4 x float> %v3 to <16 x i8>
-// CHECK:   [[VBSL1_I:%.*]] = bitcast <16 x i8> [[TMP1]] to <4 x i32>
-// CHECK:   [[VBSL2_I:%.*]] = bitcast <16 x i8> [[TMP2]] to <4 x i32>
-// CHECK:   [[VBSL3_I:%.*]] = and <4 x i32> %v1, [[VBSL1_I]]
-// CHECK:   [[TMP3:%.*]] = xor <4 x i32> %v1, <i32 -1, i32 -1, i32 -1, i32 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <4 x i32> [[TMP3]], [[VBSL2_I]]
-// CHECK:   [[VBSL5_I:%.*]] = or <4 x i32> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   [[TMP4:%.*]] = bitcast <4 x i32> [[VBSL5_I]] to <4 x float>
-// CHECK:   ret <4 x float> [[TMP4]]
+// CHECK:    [[TMP0:%.*]] = bitcast <4 x i32> %v1 to <16 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <4 x float> %v2 to <16 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <4 x float> %v3 to <16 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> [[TMP0]], <16 x i8> [[TMP1]], <16 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <16 x i8> [[VBSL_I]] to <4 x float>
+// CHECK:    ret <4 x float> [[TMP3]]
+//
 float32x4_t test_vbslq_f32(uint32x4_t v1, float32x4_t v2, float32x4_t v3) {
   return vbslq_f32(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbslq_p8(
-// CHECK:   [[VBSL_I:%.*]] = and <16 x i8> %v1, %v2
-// CHECK:   [[TMP0:%.*]] = xor <16 x i8> %v1, <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
-// CHECK:   [[VBSL1_I:%.*]] = and <16 x i8> [[TMP0]], %v3
-// CHECK:   [[VBSL2_I:%.*]] = or <16 x i8> [[VBSL_I]], [[VBSL1_I]]
-// CHECK:   ret <16 x i8> [[VBSL2_I]]
+// CHECK:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> %v1, <16 x i8> %v2, <16 x i8> %v3)
+// CHECK:    ret <16 x i8> [[VBSL_I]]
+//
 poly8x16_t test_vbslq_p8(uint8x16_t v1, poly8x16_t v2, poly8x16_t v3) {
   return vbslq_p8(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbslq_p16(
-// CHECK:   [[TMP0:%.*]] = bitcast <8 x i16> %v1 to <16 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <8 x i16> %v2 to <16 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <8 x i16> %v3 to <16 x i8>
-// CHECK:   [[VBSL3_I:%.*]] = and <8 x i16> %v1, %v2
-// CHECK:   [[TMP3:%.*]] = xor <8 x i16> %v1, <i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <8 x i16> [[TMP3]], %v3
-// CHECK:   [[VBSL5_I:%.*]] = or <8 x i16> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <8 x i16> [[VBSL5_I]]
+// CHECK:    [[TMP0:%.*]] = bitcast <8 x i16> %v1 to <16 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <8 x i16> %v2 to <16 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <8 x i16> %v3 to <16 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> [[TMP0]], <16 x i8> [[TMP1]], <16 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <16 x i8> [[VBSL_I]] to <8 x i16>
+// CHECK:    ret <8 x i16> [[TMP3]]
+//
 poly16x8_t test_vbslq_p16(uint16x8_t v1, poly16x8_t v2, poly16x8_t v3) {
   return vbslq_p16(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vbslq_f64(
-// CHECK:   [[TMP0:%.*]] = bitcast <2 x i64> %v1 to <16 x i8>
-// CHECK:   [[TMP1:%.*]] = bitcast <2 x double> %v2 to <16 x i8>
-// CHECK:   [[TMP2:%.*]] = bitcast <2 x double> %v3 to <16 x i8>
-// CHECK:   [[VBSL1_I:%.*]] = bitcast <16 x i8> [[TMP1]] to <2 x i64>
-// CHECK:   [[VBSL2_I:%.*]] = bitcast <16 x i8> [[TMP2]] to <2 x i64>
-// CHECK:   [[VBSL3_I:%.*]] = and <2 x i64> %v1, [[VBSL1_I]]
-// CHECK:   [[TMP3:%.*]] = xor <2 x i64> %v1, <i64 -1, i64 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <2 x i64> [[TMP3]], [[VBSL2_I]]
-// CHECK:   [[VBSL5_I:%.*]] = or <2 x i64> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   [[TMP4:%.*]] = bitcast <2 x i64> [[VBSL5_I]] to <2 x double>
-// CHECK:   ret <2 x double> [[TMP4]]
+// CHECK:    [[TMP0:%.*]] = bitcast <2 x i64> %v1 to <16 x i8>
+// CHECK:    [[TMP1:%.*]] = bitcast <2 x double> %v2 to <16 x i8>
+// CHECK:    [[TMP2:%.*]] = bitcast <2 x double> %v3 to <16 x i8>
+// CHECK:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> [[TMP0]], <16 x i8> [[TMP1]], <16 x i8> [[TMP2]])
+// CHECK:    [[TMP3:%.*]] = bitcast <16 x i8> [[VBSL_I]] to <2 x double>
+// CHECK:    ret <2 x double> [[TMP3]]
+//
 float64x2_t test_vbslq_f64(uint64x2_t v1, float64x2_t v2, float64x2_t v3) {
   return vbslq_f64(v1, v2, v3);
 }
 
 // CHECK-LABEL: @test_vrecps_f32(
 // CHECK:   [[TMP0:%.*]] = bitcast <2 x float> %v1 to <8 x i8>
 // CHECK:   [[TMP1:%.*]] = bitcast <2 x float> %v2 to <8 x i8>
 // CHECK:   [[VRECPS_V2_I:%.*]] = call <2 x float> @llvm.aarch64.neon.frecps.v2f32(<2 x float> %v1, <2 x float> %v2)

clang/test/CodeGen/aarch64-poly64.c

 // CHECK:   [[TMP5:%.*]] = icmp ne <2 x i64> [[TMP4]], zeroinitializer
 // CHECK:   [[VTST_I:%.*]] = sext <2 x i1> [[TMP5]] to <2 x i64>
 // CHECK:   ret <2 x i64> [[VTST_I]]
 uint64x2_t test_vtstq_p64(poly64x2_t a, poly64x2_t b) {
   return vtstq_p64(a, b);
 }
 
 // CHECK-LABEL: define{{.*}} <1 x i64> @test_vbsl_p64(<1 x i64> noundef %a, <1 x i64> noundef %b, <1 x i64> noundef %c) #0 {
-// CHECK:   [[VBSL3_I:%.*]] = and <1 x i64> %a, %b
-// CHECK:   [[TMP3:%.*]] = xor <1 x i64> %a, <i64 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <1 x i64> [[TMP3]], %c
-// CHECK:   [[VBSL5_I:%.*]] = or <1 x i64> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <1 x i64> [[VBSL5_I]]
+// CHECK:   [[TMP0:%.*]] = bitcast <1 x i64> %a to <8 x i8>
+// CHECK:   [[TMP1:%.*]] = bitcast <1 x i64> %b to <8 x i8>
+// CHECK:   [[TMP2:%.*]] = bitcast <1 x i64> %c to <8 x i8>
+// CHECK:   [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> [[TMP0]], <8 x i8> [[TMP1]], <8 x i8> [[TMP2]])
+// CHECK:   [[TMP3:%.*]]  = bitcast <8 x i8> [[VBSL_I]] to <1 x i64>
+// CHECK:   ret <1 x i64> [[TMP3]]
 poly64x1_t test_vbsl_p64(poly64x1_t a, poly64x1_t b, poly64x1_t c) {
   return vbsl_p64(a, b, c);
 }
 
 // CHECK-LABEL: define{{.*}} <2 x i64> @test_vbslq_p64(<2 x i64> noundef %a, <2 x i64> noundef %b, <2 x i64> noundef %c) #0 {
-// CHECK:   [[VBSL3_I:%.*]] = and <2 x i64> %a, %b
-// CHECK:   [[TMP3:%.*]] = xor <2 x i64> %a, <i64 -1, i64 -1>
-// CHECK:   [[VBSL4_I:%.*]] = and <2 x i64> [[TMP3]], %c
-// CHECK:   [[VBSL5_I:%.*]] = or <2 x i64> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK:   ret <2 x i64> [[VBSL5_I]]
+// CHECK:   [[TMP0:%.*]] = bitcast <2 x i64> %a to <16 x i8>
+// CHECK:   [[TMP1:%.*]] = bitcast <2 x i64> %b to <16 x i8>
+// CHECK:   [[TMP2:%.*]] = bitcast <2 x i64> %c to <16 x i8>
+// CHECK:   [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> [[TMP0]], <16 x i8> [[TMP1]], <16 x i8> [[TMP2]])
+// CHECK:   [[TMP3:%.*]] = bitcast <16 x i8> [[VBSL_I]] to <2 x i64>
+// CHECK:   ret <2 x i64> [[TMP3]]
 poly64x2_t test_vbslq_p64(poly64x2_t a, poly64x2_t b, poly64x2_t c) {
   return vbslq_p64(a, b, c);
 }
 
 // CHECK-LABEL: define{{.*}} i64 @test_vget_lane_p64(<1 x i64> noundef %v) #0 {
 // CHECK:   [[VGET_LANE:%.*]] = extractelement <1 x i64> %v, i32 0
 // CHECK:   ret i64 [[VGET_LANE]]
 poly64_t test_vget_lane_p64(poly64x1_t v) {

clang/test/CodeGen/aarch64-v8.2a-neon-intrinsics.c

 }
 
 // CHECK-LABEL: define {{[^@]+}}@test_vbsl_f16
 // CHECK-SAME: (<4 x i16> noundef [[A:%.*]], <4 x half> noundef [[B:%.*]], <4 x half> noundef [[C:%.*]]) #[[ATTR0]] {
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = bitcast <4 x i16> [[A]] to <8 x i8>
 // CHECK-NEXT:    [[TMP1:%.*]] = bitcast <4 x half> [[B]] to <8 x i8>
 // CHECK-NEXT:    [[TMP2:%.*]] = bitcast <4 x half> [[C]] to <8 x i8>
-// CHECK-NEXT:    [[VBSL1_I:%.*]] = bitcast <8 x i8> [[TMP1]] to <4 x i16>
-// CHECK-NEXT:    [[VBSL2_I:%.*]] = bitcast <8 x i8> [[TMP2]] to <4 x i16>
-// CHECK-NEXT:    [[VBSL3_I:%.*]] = and <4 x i16> [[A]], [[VBSL1_I]]
-// CHECK-NEXT:    [[TMP3:%.*]] = xor <4 x i16> [[A]], <i16 -1, i16 -1, i16 -1, i16 -1>
-// CHECK-NEXT:    [[VBSL4_I:%.*]] = and <4 x i16> [[TMP3]], [[VBSL2_I]]
-// CHECK-NEXT:    [[VBSL5_I:%.*]] = or <4 x i16> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK-NEXT:    [[TMP4:%.*]] = bitcast <4 x i16> [[VBSL5_I]] to <4 x half>
-// CHECK-NEXT:    ret <4 x half> [[TMP4]]
+// CHECK-NEXT:    [[VBSL_I:%.*]] = call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> [[TMP0]], <8 x i8> [[TMP1]], <8 x i8> [[TMP2]])
+// CHECK-NEXT:    [[TMP3:%.*]] = bitcast <8 x i8> [[VBSL_I]] to <4 x half>
 //
 float16x4_t test_vbsl_f16(uint16x4_t a, float16x4_t b, float16x4_t c) {
   return vbsl_f16(a, b, c);
 }
 
 // CHECK-LABEL: define {{[^@]+}}@test_vbslq_f16
 // CHECK-SAME: (<8 x i16> noundef [[A:%.*]], <8 x half> noundef [[B:%.*]], <8 x half> noundef [[C:%.*]]) #[[ATTR1]] {
 // CHECK-NEXT:  entry:
 // CHECK-NEXT:    [[TMP0:%.*]] = bitcast <8 x i16> [[A]] to <16 x i8>
 // CHECK-NEXT:    [[TMP1:%.*]] = bitcast <8 x half> [[B]] to <16 x i8>
 // CHECK-NEXT:    [[TMP2:%.*]] = bitcast <8 x half> [[C]] to <16 x i8>
-// CHECK-NEXT:    [[VBSL1_I:%.*]] = bitcast <16 x i8> [[TMP1]] to <8 x i16>
-// CHECK-NEXT:    [[VBSL2_I:%.*]] = bitcast <16 x i8> [[TMP2]] to <8 x i16>
-// CHECK-NEXT:    [[VBSL3_I:%.*]] = and <8 x i16> [[A]], [[VBSL1_I]]
-// CHECK-NEXT:    [[TMP3:%.*]] = xor <8 x i16> [[A]], <i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1>
-// CHECK-NEXT:    [[VBSL4_I:%.*]] = and <8 x i16> [[TMP3]], [[VBSL2_I]]
-// CHECK-NEXT:    [[VBSL5_I:%.*]] = or <8 x i16> [[VBSL3_I]], [[VBSL4_I]]
-// CHECK-NEXT:    [[TMP4:%.*]] = bitcast <8 x i16> [[VBSL5_I]] to <8 x half>
-// CHECK-NEXT:    ret <8 x half> [[TMP4]]
+// CHECK-NEXT:    [[VBSL_I:%.*]] = call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> %0, <16 x i8> %1, <16 x i8> %2)
+// CHECK-NEXT:    [[TMP3:%.*]] = bitcast <16 x i8> [[VBSL_I]] to <8 x half>
+// CHECK-NEXT:    ret <8 x half> [[TMP3]]
 //
 float16x8_t test_vbslq_f16(uint16x8_t a, float16x8_t b, float16x8_t c) {
   return vbslq_f16(a, b, c);
 }
 
 // CHECK-LABEL: define {{[^@]+}}@test_vzip_f16
 // CHECK-SAME: (<4 x half> noundef [[A:%.*]], <4 x half> noundef [[B:%.*]]) #[[ATTR0]] {
 // CHECK-NEXT:  entry:

llvm/include/llvm/IR/IntrinsicsAArch64.td

let TargetPrefix = "aarch64", IntrProperties = [IntrNoMem] in {
   def int_aarch64_neon_fcvtzu : AdvSIMD_FPToIntRounding_Intrinsic;
 
   // v8.5-A Vector FP Rounding
   def int_aarch64_neon_frint32x : AdvSIMD_1FloatArg_Intrinsic;
   def int_aarch64_neon_frint32z : AdvSIMD_1FloatArg_Intrinsic;
   def int_aarch64_neon_frint64x : AdvSIMD_1FloatArg_Intrinsic;
   def int_aarch64_neon_frint64z : AdvSIMD_1FloatArg_Intrinsic;
 
+  def int_aarch64_neon_bsl : AdvSIMD_3VectorArg_Intrinsic;
+
   // Scalar FP->Int conversions
 
   // Vector FP Inexact Narrowing
   def int_aarch64_neon_fcvtxn : AdvSIMD_1VectorArg_Expand_Intrinsic;
 
   // Scalar FP Inexact Narrowing
   def int_aarch64_sisd_fcvtxn : DefaultAttrsIntrinsic<[llvm_float_ty], [llvm_double_ty],
                                         [IntrNoMem]>;

llvm/lib/Target/AArch64/AArch64InstrInfo.td

 defm ORN : SIMDLogicalThreeVector<0, 0b11, "orn",
                                   BinOpFrag<(or node:$LHS, (vnot node:$RHS))> >;
 defm ORR : SIMDLogicalThreeVector<0, 0b10, "orr", or>;
 
 // Pseudo bitwise select pattern BSP.
 // It is expanded into BSL/BIT/BIF after register allocation.
 defm BSP : SIMDLogicalThreeVectorPseudo<TriOpFrag<(or (and node:$LHS, node:$MHS),
                                                       (and (vnot node:$LHS), node:$RHS))>>;
-defm BSL : SIMDLogicalThreeVectorTied<1, 0b01, "bsl">;
+defm BSL : SIMDLogicalThreeVectorTied<1, 0b01, "bsl", int_aarch64_neon_bsl>;
 defm BIT : SIMDLogicalThreeVectorTied<1, 0b10, "bit", AArch64bit>;
 defm BIF : SIMDLogicalThreeVectorTied<1, 0b11, "bif">;
 
 def : Pat<(AArch64bsp (v8i8 V64:$Rd), V64:$Rn, V64:$Rm),
           (BSPv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
 def : Pat<(AArch64bsp (v4i16 V64:$Rd), V64:$Rn, V64:$Rm),
           (BSPv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
 def : Pat<(AArch64bsp (v2i32 V64:$Rd), V64:$Rn, V64:$Rm),

llvm/test/CodeGen/AArch64/aarch64-vbsl.ll

+; RUN: llc -mtriple=aarch64-none-linux-gnu -mattr=+neon < %s -o -| FileCheck %s
+
+define <8 x i8> @vbsl.v8i8(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2) {
+entry:
+; CHECK-LABEL: vbsl.v8i8
+; CHECK: bsl   v0.8b, v1.8b, v2.8b
+  %vbsl.i = tail call <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8> %v0, <8 x i8> %v1, <8 x i8> %v2)
+  ret <8 x i8> %vbsl.i
+}
+
+define <16 x i8> @vbslq.v8i8(<16 x i8> %v0, <16 x i8> %v1, <16 x i8> %v2) {
+entry:
+; CHECK-LABEL: vbslq.v8i8
+; CHECK: bsl   v0.16b, v1.16b, v2.16b
+  %vbsl.i = tail call <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8> %v0, <16 x i8> %v1, <16 x i8> %v2)
+  ret <16 x i8> %vbsl.i
+}
+
+declare <8 x i8> @llvm.aarch64.neon.bsl.v8i8(<8 x i8>, <8 x i8>, <8 x i8>) #2
+declare <16 x i8> @llvm.aarch64.neon.bsl.v16i8(<16 x i8>, <16 x i8>, <16 x i8>) #2
+