[AArch64] Add BIT/BIF support.

[ilinpv]

Summary

This patch added generation of SIMD bitwise insert BIT/BIF instructions.
In the absence of GCC-like functionality for optimal constraints satisfaction during register allocation
the bitwise insert and select patterns are matched by pseudo bitwise select BSP instruction with not tied def.
It is expanded later after register allocation with def tied to BSL/BIT/BIF depending on operands registers.
This allows to get rid of redundant moves.

[samparker]

Would you mind re-uploading with some more context in the diff please?

[ilinpv]

More diff context added

[dmgreen]

Can you update with all the context? -U999999. It helps to be able to see the entire file here in phabricator, which would not otherwise be available.

For these tests that have changed/been added, can you run the update_llc_test_checks scripts on them all and commit that as a separate initial (NFC) patch? That way the differences here are more obvious. It looks like a nice improvement, but it's hard to tell when the test just says "CHECK: bsl"

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

1346

Is this ever generated from anything?

AArch64ISD::BSL and AArch64ISD::BIT seem to be created from ISelLowering. Would it make sense to convert them to use BSLP as well? We could then presumably remove a lot of the explicit patterns for BSL, BIT and BIF.

llvm/lib/Target/AArch64/AArch64InstrFormats.td

5213

We should make sure that the schedules have info on BSLP, if they previously had explicit info for BSP/BIT/BIF.

[ilinpv]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

1346

It is not generated, just a manual target node name. BSL and BIF can be converted to use BSLP. However using BSLP for BIT in ISelLowering can lead to additional move generation on expansion, see vector-fcopysign as example.

llvm/lib/Target/AArch64/AArch64InstrFormats.td

5213

Schedules using explicit regex "^(BIF|BIT|BSL)". As I understand it schedules passes happen after pseudo expansion, so at schedule stage we have all BSLP expanded to BIF/BIT/BSL.

[ilinpv]

Diff amended in accordance with comments.

[dmgreen]

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

464–469

Do you know if there are any test cases that test this "extra move" path?

llvm/lib/Target/AArch64/AArch64InstrFormats.td

5213

I think we run scheduling both pre-ra and post-ra. But that might depend on the target. So the pre-ra scheduling will still see the BSLP's. (I think you are right about post-ra though, those will see the "real" instruction.)

llvm/lib/Target/AArch64/AArch64InstrInfo.td

3968

AArch64bslp can be null_frag here and in BSL.

3988–4040

Most of these patterns can be removed I think. We need patterns for
TriOpFrag<(or (and ... -> BLSP (which are handled in the SIMDLogicalThreeVectorPseudo definnition)
AArch64bslp -> BSLP (for which we need these extra patterns).
AArch64bit -> BIT, if we are keeping that one around (again handled inside SIMDLogicalThreeVectorTied)

The other shouldn't be needed unless I'm missing something.

[ilinpv]

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

464–469

Yes, CodeGen/AArch64/arm64-promote-const.ll for example

[ilinpv]

Obsolete patterns removed, schedulers patched.

[dmgreen]

Thanks for the scheduling changes. LGTM, with a couple of minor nitpicks.

llvm/lib/Target/AArch64/AArch64InstrInfo.td

3965

This line it quite long. Maybe try and split it up a little like the BIC above?

3988–4004

I believe that these patterns will already be included inside the SIMDLogicalThreeVectorTied BIT class, so aren't needed again here?

[ilinpv]

llvm/lib/Target/AArch64/AArch64InstrInfo.td

3965

Sure

3988–4004

Yes, accidentally remained here.

[vvereschaka]

Hello @ilinpv,

these changes break the tests on the expensive check builder
http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-ubuntu/builds/2884

*** Bad machine code: Explicit definition marked as use ***
- function:    test5
- basic block: %bb.0 entry (0x558983ebb6b8)
- instruction: ORRv16i8 $q0, killed renamable $q1, killed renamable $q1
- operand 0:   $q0

*** Bad machine code: Using an undefined physical register ***
- function:    test5
- basic block: %bb.0 entry (0x558983ebb6b8)
- instruction: ORRv16i8 $q0, killed renamable $q1, killed renamable $q1
- operand 0:   $q0

*** Bad machine code: Using an undefined physical register ***
- function:    test5
- basic block: %bb.0 entry (0x558983ebb6b8)
- instruction: renamable $q0 = BSLv16i8 $q0(tied-def 0), killed renamable $q2, killed renamable $q3
- operand 1:   $q0(tied-def 0)
LLVM ERROR: Found 3 machine code errors.

[dmgreen]

Ah. Kill Flags. Should hopefully be fixed up in da147ef0a5c6d31c21d31a52b97235a629830c15

Thanks for the report.

[ilinpv]

Fixed in dc0b8159890134a59fdf34b20e8b2052d9456441. All AArch64 expensive check tests passed.

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

 /// do the expansion and return true.  Otherwise return false.
 bool AArch64ExpandPseudo::expandMI(MachineBasicBlock &MBB,
                                    MachineBasicBlock::iterator MBBI,
                                    MachineBasicBlock::iterator &NextMBBI) {
   MachineInstr &MI = *MBBI;
   unsigned Opcode = MI.getOpcode();
   switch (Opcode) {
   default:
     break;
 
+  case AArch64::BSLPv8i8:
+  case AArch64::BSLPv16i8: {
+    Register DstReg = MI.getOperand(0).getReg();
+    if (DstReg == MI.getOperand(3).getReg()) {
+      // Expand to BIT
+      BuildMI(MBB, MBBI, MI.getDebugLoc(),
+              TII->get(Opcode == AArch64::BSLPv8i8 ? AArch64::BITv8i8
+                                                   : AArch64::BITv16i8))
+          .add(MI.getOperand(0))
+          .add(MI.getOperand(3))
+          .add(MI.getOperand(2))
+          .add(MI.getOperand(1));
+    } else if (DstReg == MI.getOperand(2).getReg()) {
+      // Expand to BIF
+      BuildMI(MBB, MBBI, MI.getDebugLoc(),
+              TII->get(Opcode == AArch64::BSLPv8i8 ? AArch64::BIFv8i8
+                                                   : AArch64::BIFv16i8))
+          .add(MI.getOperand(0))
+          .add(MI.getOperand(2))
+          .add(MI.getOperand(3))
+          .add(MI.getOperand(1));
+    } else {
+      // Expand to BSL, use additional move if required
+      if (DstReg == MI.getOperand(1).getReg()) {
+        BuildMI(MBB, MBBI, MI.getDebugLoc(),
+                TII->get(Opcode == AArch64::BSLPv8i8 ? AArch64::BSLv8i8
+                                                     : AArch64::BSLv16i8))
+            .add(MI.getOperand(0))
+            .add(MI.getOperand(1))
+            .add(MI.getOperand(2))
+            .add(MI.getOperand(3));
+      } else {
+        BuildMI(MBB, MBBI, MI.getDebugLoc(),
+                TII->get(Opcode == AArch64::BSLPv8i8 ? AArch64::ORRv8i8
+                                                     : AArch64::ORRv16i8))
+            .addReg(DstReg)
+            .add(MI.getOperand(1))
+            .add(MI.getOperand(1));
+        BuildMI(MBB, MBBI, MI.getDebugLoc(),
+                TII->get(Opcode == AArch64::BSLPv8i8 ? AArch64::BSLv8i8
+                                                     : AArch64::BSLv16i8))
+            .add(MI.getOperand(0))
+            .addReg(DstReg)
+            .add(MI.getOperand(2))
+            .add(MI.getOperand(3));
+      }
+    }
+    MI.eraseFromParent();
+    return true;
+  }
+
   case AArch64::ADDWrr:
   case AArch64::SUBWrr:
   case AArch64::ADDXrr:
   case AArch64::SUBXrr:
   case AArch64::ADDSWrr:
   case AArch64::SUBSWrr:
   case AArch64::ADDSXrr:
   case AArch64::SUBSXrr:
   case AArch64::ANDWrr:
   case AArch64::ANDXrr:

llvm/lib/Target/AArch64/AArch64ISelLowering.h

   MVNIshift,
   MVNImsl,
 
   // Vector immediate ops
   BICi,
   ORRi,
 
   // Vector bit select: similar to ISD::VSELECT but not all bits within an
   // element must be identical.
   BSL,
+  BSLP,
 
   // Vector arithmetic negation
   NEG,
 
   // Vector shuffles
   ZIP1,
   ZIP2,
   UZP1,
   UZP2,
   TRN1,

   UMINV_PRED,
   ORV_PRED,
   EORV_PRED,
   ANDV_PRED,
 
   // Vector bitwise negation
   NOT,
 
   // Vector bitwise selection
   BIT,
+  BIF,
 
   // Compare-and-branch
   CBZ,
   CBNZ,
   TBZ,
   TBNZ,
 
   // Tail calls
   TC_RETURN,
 

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

   case AArch64ISD::MOVI:              return "AArch64ISD::MOVI";
   case AArch64ISD::MOVIshift:         return "AArch64ISD::MOVIshift";
   case AArch64ISD::MOVIedit:          return "AArch64ISD::MOVIedit";
   case AArch64ISD::MOVImsl:           return "AArch64ISD::MOVImsl";
   case AArch64ISD::FMOV:              return "AArch64ISD::FMOV";
   case AArch64ISD::MVNIshift:         return "AArch64ISD::MVNIshift";
   case AArch64ISD::MVNImsl:           return "AArch64ISD::MVNImsl";
   case AArch64ISD::BICi:              return "AArch64ISD::BICi";
   case AArch64ISD::ORRi:              return "AArch64ISD::ORRi";
   case AArch64ISD::BSL:               return "AArch64ISD::BSL";
+  case AArch64ISD::BSLP:              return "AArch64ISD::BSLP";
   case AArch64ISD::NEG:               return "AArch64ISD::NEG";
   case AArch64ISD::EXTR:              return "AArch64ISD::EXTR";
   case AArch64ISD::ZIP1:              return "AArch64ISD::ZIP1";
   case AArch64ISD::ZIP2:              return "AArch64ISD::ZIP2";
   case AArch64ISD::UZP1:              return "AArch64ISD::UZP1";
   case AArch64ISD::UZP2:              return "AArch64ISD::UZP2";
   case AArch64ISD::TRN1:              return "AArch64ISD::TRN1";
   case AArch64ISD::TRN2:              return "AArch64ISD::TRN2";
   case AArch64ISD::REV16:             return "AArch64ISD::REV16";
   case AArch64ISD::REV32:             return "AArch64ISD::REV32";

   case AArch64ISD::EORV_PRED:         return "AArch64ISD::EORV_PRED";
   case AArch64ISD::ANDV_PRED:         return "AArch64ISD::ANDV_PRED";
   case AArch64ISD::CLASTA_N:          return "AArch64ISD::CLASTA_N";
   case AArch64ISD::CLASTB_N:          return "AArch64ISD::CLASTB_N";
   case AArch64ISD::LASTA:             return "AArch64ISD::LASTA";
   case AArch64ISD::LASTB:             return "AArch64ISD::LASTB";
   case AArch64ISD::REV:               return "AArch64ISD::REV";
   case AArch64ISD::TBL:               return "AArch64ISD::TBL";
   case AArch64ISD::NOT:               return "AArch64ISD::NOT";
   case AArch64ISD::BIT:               return "AArch64ISD::BIT";
+  case AArch64ISD::BIF:               return "AArch64ISD::BIF";
   case AArch64ISD::CBZ:               return "AArch64ISD::CBZ";
   case AArch64ISD::CBNZ:              return "AArch64ISD::CBNZ";
   case AArch64ISD::TBZ:               return "AArch64ISD::TBZ";
   case AArch64ISD::TBNZ:              return "AArch64ISD::TBNZ";
   case AArch64ISD::TC_RETURN:         return "AArch64ISD::TC_RETURN";
   case AArch64ISD::PREFETCH:          return "AArch64ISD::PREFETCH";
   case AArch64ISD::SITOF:             return "AArch64ISD::SITOF";
   case AArch64ISD::UITOF:             return "AArch64ISD::UITOF";
   case AArch64ISD::NVCAST:            return "AArch64ISD::NVCAST";
   case AArch64ISD::SQSHL_I:           return "AArch64ISD::SQSHL_I";

llvm/lib/Target/AArch64/AArch64InstrFormats.td

   let Inst{29}    = U;
   let Inst{28-24} = 0b01110;
   let Inst{23-21} = size;
   let Inst{20-16} = Rm;
   let Inst{15-11} = opcode;
   let Inst{10}    = 1;
   let Inst{9-5}   = Rn;
   let Inst{4-0}   = Rd;
 }
 
+let mayLoad = 0, mayStore = 0, hasSideEffects = 0 in
+class BaseSIMDThreeSameVectorPseudo<RegisterOperand regtype, list<dag> pattern>
+  : Pseudo<(outs regtype:$dst), (ins regtype:$Rd, regtype:$Rn, regtype:$Rm), pattern>,
+    Sched<[WriteV]>;
+
+multiclass SIMDLogicalThreeVectorPseudo<SDPatternOperator OpNode> {
+  def v8i8  : BaseSIMDThreeSameVectorPseudo<V64,
+             [(set (v8i8 V64:$dst),
+                   (OpNode (v8i8 V64:$Rd), (v8i8 V64:$Rn), (v8i8 V64:$Rm)))]>;
+  def v16i8  : BaseSIMDThreeSameVectorPseudo<V128,
+             [(set (v16i8 V128:$dst),
+                   (OpNode (v16i8 V128:$Rd), (v16i8 V128:$Rn),
+                           (v16i8 V128:$Rm)))]>;
+
+  def : Pat<(v4i16 (OpNode (v4i16 V64:$LHS), (v4i16 V64:$MHS),
+                           (v4i16 V64:$RHS))),
+          (!cast<Instruction>(NAME#"v8i8")
+            V64:$LHS, V64:$MHS, V64:$RHS)>;
+  def : Pat<(v2i32 (OpNode (v2i32 V64:$LHS), (v2i32 V64:$MHS),
+                           (v2i32 V64:$RHS))),
+          (!cast<Instruction>(NAME#"v8i8")
+            V64:$LHS, V64:$MHS, V64:$RHS)>;
+  def : Pat<(v1i64 (OpNode (v1i64 V64:$LHS), (v1i64 V64:$MHS),
+                           (v1i64 V64:$RHS))),
+          (!cast<Instruction>(NAME#"v8i8")
+            V64:$LHS, V64:$MHS, V64:$RHS)>;
+
+  def : Pat<(v8i16 (OpNode (v8i16 V128:$LHS), (v8i16 V128:$MHS),
+                           (v8i16 V128:$RHS))),
+      (!cast<Instruction>(NAME#"v16i8")
+        V128:$LHS, V128:$MHS, V128:$RHS)>;
+  def : Pat<(v4i32 (OpNode (v4i32 V128:$LHS), (v4i32 V128:$MHS),
+                           (v4i32 V128:$RHS))),
+      (!cast<Instruction>(NAME#"v16i8")
+        V128:$LHS, V128:$MHS, V128:$RHS)>;
+  def : Pat<(v2i64 (OpNode (v2i64 V128:$LHS), (v2i64 V128:$MHS),
+                           (v2i64 V128:$RHS))),
+      (!cast<Instruction>(NAME#"v16i8")
+        V128:$LHS, V128:$MHS, V128:$RHS)>;
+}
+
 // All operand sizes distinguished in the encoding.
 multiclass SIMDThreeSameVector<bit U, bits<5> opc, string asm,
                                SDPatternOperator OpNode> {
   def v8i8  : BaseSIMDThreeSameVector<0, U, 0b001, opc, V64,
                                       asm, ".8b",
          [(set (v8i8 V64:$Rd), (OpNode (v8i8 V64:$Rn), (v8i8 V64:$Rm)))]>;
   def v16i8 : BaseSIMDThreeSameVector<1, U, 0b001, opc, V128,
                                       asm, ".16b",
          [(set (v16i8 V128:$Rd), (OpNode (v16i8 V128:$Rn), (v16i8 V128:$Rm)))]>;
   def v4i16 : BaseSIMDThreeSameVector<0, U, 0b011, opc, V64,

llvm/lib/Target/AArch64/AArch64InstrInfo.td

 def AArch64vlshr : SDNode<"AArch64ISD::VLSHR", SDT_AArch64vshift>;
 def AArch64vshl : SDNode<"AArch64ISD::VSHL", SDT_AArch64vshift>;
 def AArch64sqshli : SDNode<"AArch64ISD::SQSHL_I", SDT_AArch64vshift>;
 def AArch64uqshli : SDNode<"AArch64ISD::UQSHL_I", SDT_AArch64vshift>;
 def AArch64sqshlui : SDNode<"AArch64ISD::SQSHLU_I", SDT_AArch64vshift>;
 def AArch64srshri : SDNode<"AArch64ISD::SRSHR_I", SDT_AArch64vshift>;
 def AArch64urshri : SDNode<"AArch64ISD::URSHR_I", SDT_AArch64vshift>;
 
 def AArch64not: SDNode<"AArch64ISD::NOT", SDT_AArch64unvec>;
 def AArch64bit: SDNode<"AArch64ISD::BIT", SDT_AArch64trivec>;
+def AArch64bif: SDNode<"AArch64ISD::BIF", SDT_AArch64trivec>;
 def AArch64bsl: SDNode<"AArch64ISD::BSL", SDT_AArch64trivec>;
+def AArch64bslp: SDNode<"AArch64ISD::BSLP", SDT_AArch64trivec>;
 
 def AArch64cmeq: SDNode<"AArch64ISD::CMEQ", SDT_AArch64binvec>;
 def AArch64cmge: SDNode<"AArch64ISD::CMGE", SDT_AArch64binvec>;
 def AArch64cmgt: SDNode<"AArch64ISD::CMGT", SDT_AArch64binvec>;
 def AArch64cmhi: SDNode<"AArch64ISD::CMHI", SDT_AArch64binvec>;
 def AArch64cmhs: SDNode<"AArch64ISD::CMHS", SDT_AArch64binvec>;
 
 def AArch64fcmeq: SDNode<"AArch64ISD::FCMEQ", SDT_AArch64fcmp>;
 def AArch64fcmge: SDNode<"AArch64ISD::FCMGE", SDT_AArch64fcmp>;
 def AArch64fcmgt: SDNode<"AArch64ISD::FCMGT", SDT_AArch64fcmp>;

 
 // Extra saturate patterns, other than the intrinsics matches above
 defm : SIMDThreeSameVectorExtraPatterns<"SQADD", saddsat>;
 defm : SIMDThreeSameVectorExtraPatterns<"UQADD", uaddsat>;
 defm : SIMDThreeSameVectorExtraPatterns<"SQSUB", ssubsat>;
 defm : SIMDThreeSameVectorExtraPatterns<"UQSUB", usubsat>;
 
 defm AND : SIMDLogicalThreeVector<0, 0b00, "and", and>;
 defm BIC : SIMDLogicalThreeVector<0, 0b01, "bic",
                                   BinOpFrag<(and node:$LHS, (vnot node:$RHS))> >;
-defm BIF : SIMDLogicalThreeVector<1, 0b11, "bif">;
-defm BIT : SIMDLogicalThreeVectorTied<1, 0b10, "bit", AArch64bit>;
-defm BSL : SIMDLogicalThreeVectorTied<1, 0b01, "bsl",
-    TriOpFrag<(or (and node:$LHS, node:$MHS), (and (vnot node:$LHS), node:$RHS))>>;
 defm EOR : SIMDLogicalThreeVector<1, 0b00, "eor", xor>;
 defm ORN : SIMDLogicalThreeVector<0, 0b11, "orn",
                                   BinOpFrag<(or node:$LHS, (vnot node:$RHS))> >;
 defm ORR : SIMDLogicalThreeVector<0, 0b10, "orr", or>;
 
+// Pseudo bitwise bsl-like pattern BSLP. 
+// It is expanded into BSL/BIT/BIF after register allocation. 
+defm BSLP : SIMDLogicalThreeVectorPseudo<TriOpFrag<(or (and node:$LHS, node:$MHS), (and (vnot node:$LHS), node:$RHS))>>;
+defm BSL : SIMDLogicalThreeVectorTied<1, 0b01, "bsl", AArch64bsl>;
+defm BIT : SIMDLogicalThreeVectorTied<1, 0b10, "bit", AArch64bit>;
+defm BIF : SIMDLogicalThreeVectorTied<1, 0b11, "bif", AArch64bif>;
+
+def : Pat<(AArch64bslp (v8i8 V64:$Rd), V64:$Rn, V64:$Rm),
+          (BSLPv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
+def : Pat<(AArch64bslp (v4i16 V64:$Rd), V64:$Rn, V64:$Rm),
+          (BSLPv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
+def : Pat<(AArch64bslp (v2i32 V64:$Rd), V64:$Rn, V64:$Rm),
+          (BSLPv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
+def : Pat<(AArch64bslp (v1i64 V64:$Rd), V64:$Rn, V64:$Rm),
+          (BSLPv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
+
+def : Pat<(AArch64bslp (v16i8 V128:$Rd), V128:$Rn, V128:$Rm),
+          (BSLPv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
+def : Pat<(AArch64bslp (v8i16 V128:$Rd), V128:$Rn, V128:$Rm),
+          (BSLPv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
+def : Pat<(AArch64bslp (v4i32 V128:$Rd), V128:$Rn, V128:$Rm),
+          (BSLPv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
+def : Pat<(AArch64bslp (v2i64 V128:$Rd), V128:$Rn, V128:$Rm),
+          (BSLPv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
 
 def : Pat<(AArch64bsl (v8i8 V64:$Rd), V64:$Rn, V64:$Rm),
           (BSLv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
 def : Pat<(AArch64bsl (v4i16 V64:$Rd), V64:$Rn, V64:$Rm),
           (BSLv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
 def : Pat<(AArch64bsl (v2i32 V64:$Rd), V64:$Rn, V64:$Rm),
           (BSLv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
 def : Pat<(AArch64bsl (v1i64 V64:$Rd), V64:$Rn, V64:$Rm),
           (BSLv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
 
 def : Pat<(AArch64bsl (v16i8 V128:$Rd), V128:$Rn, V128:$Rm),
           (BSLv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
 def : Pat<(AArch64bsl (v8i16 V128:$Rd), V128:$Rn, V128:$Rm),
           (BSLv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
 def : Pat<(AArch64bsl (v4i32 V128:$Rd), V128:$Rn, V128:$Rm),
           (BSLv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
 def : Pat<(AArch64bsl (v2i64 V128:$Rd), V128:$Rn, V128:$Rm),
           (BSLv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
 
+def : Pat<(AArch64bit (v8i8 V64:$Rd), V64:$Rn, V64:$Rm),
+          (BITv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
+def : Pat<(AArch64bit (v4i16 V64:$Rd), V64:$Rn, V64:$Rm),
+          (BITv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
+def : Pat<(AArch64bit (v2i32 V64:$Rd), V64:$Rn, V64:$Rm),
+          (BITv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
+def : Pat<(AArch64bit (v1i64 V64:$Rd), V64:$Rn, V64:$Rm),
+          (BITv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
+
+def : Pat<(AArch64bit (v16i8 V128:$Rd), V128:$Rn, V128:$Rm),
+          (BITv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
+def : Pat<(AArch64bit (v8i16 V128:$Rd), V128:$Rn, V128:$Rm),
+          (BITv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
+def : Pat<(AArch64bit (v4i32 V128:$Rd), V128:$Rn, V128:$Rm),
+          (BITv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
+def : Pat<(AArch64bit (v2i64 V128:$Rd), V128:$Rn, V128:$Rm),
+          (BITv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
+
+def : Pat<(AArch64bif (v8i8 V64:$Rd), V64:$Rn, V64:$Rm),
+          (BIFv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
+def : Pat<(AArch64bif (v4i16 V64:$Rd), V64:$Rn, V64:$Rm),
+          (BIFv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
+def : Pat<(AArch64bif (v2i32 V64:$Rd), V64:$Rn, V64:$Rm),
+          (BIFv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
+def : Pat<(AArch64bif (v1i64 V64:$Rd), V64:$Rn, V64:$Rm),
+          (BIFv8i8 V64:$Rd, V64:$Rn, V64:$Rm)>;
+
+def : Pat<(AArch64bif (v16i8 V128:$Rd), V128:$Rn, V128:$Rm),
+          (BIFv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
+def : Pat<(AArch64bif (v8i16 V128:$Rd), V128:$Rn, V128:$Rm),
+          (BIFv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
+def : Pat<(AArch64bif (v4i32 V128:$Rd), V128:$Rn, V128:$Rm),
+          (BIFv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
+def : Pat<(AArch64bif (v2i64 V128:$Rd), V128:$Rn, V128:$Rm),
+          (BIFv16i8 V128:$Rd, V128:$Rn, V128:$Rm)>;
+
 def : InstAlias<"mov{\t$dst.16b, $src.16b|.16b\t$dst, $src}",
                 (ORRv16i8 V128:$dst, V128:$src, V128:$src), 1>;
 def : InstAlias<"mov{\t$dst.8h, $src.8h|.8h\t$dst, $src}",
                 (ORRv16i8 V128:$dst, V128:$src, V128:$src), 0>;
 def : InstAlias<"mov{\t$dst.4s, $src.4s|.4s\t$dst, $src}",
                 (ORRv16i8 V128:$dst, V128:$src, V128:$src), 0>;
 def : InstAlias<"mov{\t$dst.2d, $src.2d|.2d\t$dst, $src}",
                 (ORRv16i8 V128:$dst, V128:$src, V128:$src), 0>;
 
 def : InstAlias<"mov{\t$dst.8b, $src.8b|.8b\t$dst, $src}",

llvm/test/CodeGen/AArch64/aarch64-bif-gen.ll

+; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
+
+; BIF Bitwise Insert if False
+;
+; 8-bit vectors tests
+
+define <1 x i8> @test_bitf_v1i8(<1 x i8> %A, <1 x i8> %B, <1 x i8> %C) {
+; CHECK-LABEL: test_bitf_v1i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bif  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %neg = xor <1 x i8> %C, <i8 -1>
+  %and = and <1 x i8> %neg, %B 
+  %and1 = and <1 x i8> %C, %A
+  %or = or <1 x i8> %and, %and1
+  ret <1 x i8> %or
+}
+
+; 16-bit vectors tests
+
+define <1 x i16> @test_bitf_v1i16(<1 x i16> %A, <1 x i16> %B, <1 x i16> %C) {
+; CHECK-LABEL: test_bitf_v1i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bif  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %neg = xor <1 x i16> %C, <i16 -1>
+  %and = and <1 x i16> %neg, %B 
+  %and1 = and <1 x i16> %C, %A
+  %or = or <1 x i16> %and, %and1
+  ret <1 x i16> %or
+}
+
+; 32-bit vectors tests
+
+define <1 x i32> @test_bitf_v1i32(<1 x i32> %A, <1 x i32> %B, <1 x i32> %C) {
+; CHECK-LABEL: test_bitf_v1i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bif  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %neg = xor <1 x i32> %C, <i32 -1>
+  %and = and <1 x i32> %neg, %B 
+  %and1 = and <1 x i32> %C, %A
+  %or = or <1 x i32> %and, %and1
+  ret <1 x i32> %or
+}
+
+; 64-bit vectors tests
+
+define <1 x i64> @test_bitf_v1i64(<1 x i64> %A, <1 x i64> %B, <1 x i64> %C) {
+; CHECK-LABEL: test_bitf_v1i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bif  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %neg = xor <1 x i64> %C, <i64 -1>
+  %and = and <1 x i64> %neg, %B 
+  %and1 = and <1 x i64> %C, %A
+  %or = or <1 x i64> %and, %and1
+  ret <1 x i64> %or
+}
+
+define <2 x i32> @test_bitf_v2i32(<2 x i32> %A, <2 x i32> %B, <2 x i32> %C) {
+; CHECK-LABEL: test_bitf_v2i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bif  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %neg = xor <2 x i32> %C, <i32 -1, i32 -1>
+  %and = and <2 x i32> %neg, %B 
+  %and1 = and <2 x i32> %C, %A
+  %or = or <2 x i32> %and, %and1
+  ret <2 x i32> %or
+}
+
+define <4 x i16> @test_bitf_v4i16(<4 x i16> %A, <4 x i16> %B, <4 x i16> %C) {
+; CHECK-LABEL: test_bitf_v4i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bif  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %neg = xor <4 x i16> %C, <i16 -1, i16 -1, i16 -1, i16 -1>
+  %and = and <4 x i16> %neg, %B 
+  %and1 = and <4 x i16> %C, %A
+  %or = or <4 x i16> %and, %and1
+  ret <4 x i16> %or
+}
+
+define <8 x i8> @test_bitf_v8i8(<8 x i8> %A, <8 x i8> %B, <8 x i8> %C) {
+; CHECK-LABEL: test_bitf_v8i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bif  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %neg = xor <8 x i8> %C, <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
+  %and = and <8 x i8> %neg, %B 
+  %and1 = and <8 x i8> %C, %A
+  %or = or <8 x i8> %and, %and1
+  ret <8 x i8> %or
+}
+
+; 128-bit vectors tests
+
+define <2 x i64> @test_bitf_v2i64(<2 x i64> %A, <2 x i64> %B, <2 x i64> %C) {
+; CHECK-LABEL: test_bitf_v2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bif  v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    ret
+  %neg = xor <2 x i64> %C, <i64 -1, i64 -1>
+  %and = and <2 x i64> %neg, %B 
+  %and1 = and <2 x i64> %C, %A
+  %or = or <2 x i64> %and, %and1
+  ret <2 x i64> %or
+}
+
+define <4 x i32> @test_bitf_v4i32(<4 x i32> %A, <4 x i32> %B, <4 x i32> %C) {
+; CHECK-LABEL: test_bitf_v4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bif  v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    ret
+  %neg = xor <4 x i32> %C, <i32 -1, i32 -1, i32 -1, i32 -1>
+  %and = and <4 x i32> %neg, %B 
+  %and1 = and <4 x i32> %C, %A
+  %or = or <4 x i32> %and, %and1
+  ret <4 x i32> %or
+}
+
+define <8 x i16> @test_bitf_v8i16(<8 x i16> %A, <8 x i16> %B, <8 x i16> %C) {
+; CHECK-LABEL: test_bitf_v8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bif  v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    ret
+  %neg = xor <8 x i16> %C, <i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1>
+  %and = and <8 x i16> %neg, %B 
+  %and1 = and <8 x i16> %C, %A
+  %or = or <8 x i16> %and, %and1
+  ret <8 x i16> %or
+}
+
+define <16 x i8> @test_bitf_v16i8(<16 x i8> %A, <16 x i8> %B, <16 x i8> %C) {
+; CHECK-LABEL: test_bitf_v16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bif  v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    ret
+  %neg = xor <16 x i8> %C, <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
+  %and = and <16 x i8> %neg, %B 
+  %and1 = and <16 x i8> %C, %A
+  %or = or <16 x i8> %and, %and1
+  ret <16 x i8> %or
+}

llvm/test/CodeGen/AArch64/aarch64-bit-gen.ll

+; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
+
+; BIT Bitwise Insert if True
+;
+; 8-bit vectors tests
+
+define <1 x i8> @test_bit_v1i8(<1 x i8> %A, <1 x i8> %B, <1 x i8> %C) {
+; CHECK-LABEL: test_bit_v1i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bit  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %and = and <1 x i8> %C, %B 
+  %neg = xor <1 x i8> %C, <i8 -1>
+  %and1 = and <1 x i8> %neg, %A
+  %or = or <1 x i8> %and, %and1
+  ret <1 x i8> %or
+}
+
+; 16-bit vectors tests
+
+define <1 x i16> @test_bit_v1i16(<1 x i16> %A, <1 x i16> %B, <1 x i16> %C) {
+; CHECK-LABEL: test_bit_v1i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bit  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %and = and <1 x i16> %C, %B 
+  %neg = xor <1 x i16> %C, <i16 -1>
+  %and1 = and <1 x i16> %neg, %A
+  %or = or <1 x i16> %and, %and1
+  ret <1 x i16> %or
+}
+
+; 32-bit vectors tests
+
+define <1 x i32> @test_bit_v1i32(<1 x i32> %A, <1 x i32> %B, <1 x i32> %C) {
+; CHECK-LABEL: test_bit_v1i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bit  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %and = and <1 x i32> %C, %B 
+  %neg = xor <1 x i32> %C, <i32 -1>
+  %and1 = and <1 x i32> %neg, %A
+  %or = or <1 x i32> %and, %and1
+  ret <1 x i32> %or
+}
+
+; 64-bit vectors tests
+
+define <1 x i64> @test_bit_v1i64(<1 x i64> %A, <1 x i64> %B, <1 x i64> %C) {
+; CHECK-LABEL: test_bit_v1i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bit  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %and = and <1 x i64> %C, %B 
+  %neg = xor <1 x i64> %C, <i64 -1>
+  %and1 = and <1 x i64> %neg, %A
+  %or = or <1 x i64> %and, %and1
+  ret <1 x i64> %or
+}
+
+define <2 x i32> @test_bit_v2i32(<2 x i32> %A, <2 x i32> %B, <2 x i32> %C) {
+; CHECK-LABEL: test_bit_v2i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bit  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %and = and <2 x i32> %C, %B 
+  %neg = xor <2 x i32> %C, <i32 -1, i32 -1>
+  %and1 = and <2 x i32> %neg, %A
+  %or = or <2 x i32> %and, %and1
+  ret <2 x i32> %or
+}
+
+define <4 x i16> @test_bit_v4i16(<4 x i16> %A, <4 x i16> %B, <4 x i16> %C) {
+; CHECK-LABEL: test_bit_v4i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bit  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %and = and <4 x i16> %C, %B 
+  %neg = xor <4 x i16> %C, <i16 -1, i16 -1, i16 -1, i16 -1>
+  %and1 = and <4 x i16> %neg, %A
+  %or = or <4 x i16> %and, %and1
+  ret <4 x i16> %or
+}
+
+define <8 x i8> @test_bit_v8i8(<8 x i8> %A, <8 x i8> %B, <8 x i8> %C) {
+; CHECK-LABEL: test_bit_v8i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bit  v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    ret
+  %and = and <8 x i8> %C, %B 
+  %neg = xor <8 x i8> %C, <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
+  %and1 = and <8 x i8> %neg, %A
+  %or = or <8 x i8> %and, %and1
+  ret <8 x i8> %or
+}
+
+; 128-bit vectors tests
+
+define <2 x i64> @test_bit_v2i64(<2 x i64> %A, <2 x i64> %B, <2 x i64> %C) {
+; CHECK-LABEL: test_bit_v2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bit  v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    ret
+  %and = and <2 x i64> %C, %B 
+  %neg = xor <2 x i64> %C, <i64 -1, i64 -1>
+  %and1 = and <2 x i64> %neg, %A
+  %or = or <2 x i64> %and, %and1
+  ret <2 x i64> %or
+}
+
+define <4 x i32> @test_bit_v4i32(<4 x i32> %A, <4 x i32> %B, <4 x i32> %C) {
+; CHECK-LABEL: test_bit_v4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bit  v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    ret
+  %and = and <4 x i32> %C, %B 
+  %neg = xor <4 x i32> %C, <i32 -1, i32 -1, i32 -1, i32 -1>
+  %and1 = and <4 x i32> %neg, %A
+  %or = or <4 x i32> %and, %and1
+  ret <4 x i32> %or
+}
+
+define <8 x i16> @test_bit_v8i16(<8 x i16> %A, <8 x i16> %B, <8 x i16> %C) {
+; CHECK-LABEL: test_bit_v8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bit  v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    ret
+  %and = and <8 x i16> %C, %B 
+  %neg = xor <8 x i16> %C, <i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1>
+  %and1 = and <8 x i16> %neg, %A
+  %or = or <8 x i16> %and, %and1
+  ret <8 x i16> %or
+}
+
+define <16 x i8> @test_bit_v16i8(<16 x i8> %A, <16 x i8> %B, <16 x i8> %C) {
+; CHECK-LABEL: test_bit_v16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    bit  v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    ret
+  %and = and <16 x i8> %C, %B 
+  %neg = xor <16 x i8> %C, <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
+  %and1 = and <16 x i8> %neg, %A
+  %or = or <16 x i8> %and, %and1
+  ret <16 x i8> %or
+}

llvm/test/CodeGen/AArch64/arm64-neon-select_cc.ll

 ; RUN: llc  -mtriple=arm64-none-linux-gnu -mattr=+neon -fp-contract=fast \
 ; RUN:        < %s -verify-machineinstrs -asm-verbose=false | FileCheck %s
 
 define <8x i8> @test_select_cc_v8i8_i8(i8 %a, i8 %b, <8x i8> %c, <8x i8> %d ) {
 ; CHECK-LABEL: test_select_cc_v8i8_i8:
 ; CHECK-DAG: fmov s[[LHS:[0-9]+]], w0
 ; CHECK-DAG: fmov s[[RHS:[0-9]+]], w1
 ; CHECK: cmeq [[MASK:v[0-9]+]].8b, v[[LHS]].8b, v[[RHS]].8b
 ; CHECK: dup [[DUPMASK:v[0-9]+]].8b, [[MASK]].b[0]
-; CHECK: bsl [[DUPMASK]].8b, v0.8b, v1.8b
+; CHECK: bif v0.8b, v1.8b, [[DUPMASK]].8b 
   %cmp31 = icmp eq i8 %a, %b
   %e = select i1 %cmp31, <8x i8> %c, <8x i8> %d
   ret <8x i8> %e
 }
 
 define <8x i8> @test_select_cc_v8i8_f32(float %a, float %b, <8x i8> %c, <8x i8> %d ) {
 ; CHECK-LABEL: test_select_cc_v8i8_f32:
 ; CHECK: fcmeq [[MASK:v[0-9]+]].2s, v0.2s, v1.2s
 ; CHECK-NEXT: dup [[DUPMASK:v[0-9]+]].2s, [[MASK]].s[0]
 ; CHECK-NEXT: bsl [[DUPMASK]].8b, v2.8b, v3.8b

   %e = select i1 %cmp31, <8x i8> %c, <8x i8> %d
   ret <8x i8> %e
 }
 
 define <16x i8> @test_select_cc_v16i8_i8(i8 %a, i8 %b, <16x i8> %c, <16x i8> %d ) {
 ; CHECK-LABEL: test_select_cc_v16i8_i8:
 ; CHECK-DAG: fmov s[[LHS:[0-9]+]], w0
 ; CHECK-DAG: fmov s[[RHS:[0-9]+]], w1
 ; CHECK: cmeq [[MASK:v[0-9]+]].16b, v[[LHS]].16b, v[[RHS]].16b
 ; CHECK: dup [[DUPMASK:v[0-9]+]].16b, [[MASK]].b[0]
-; CHECK: bsl [[DUPMASK]].16b, v0.16b, v1.16b
+; CHECK: bif v0.16b, v1.16b, [[DUPMASK]].16b
   %cmp31 = icmp eq i8 %a, %b
   %e = select i1 %cmp31, <16x i8> %c, <16x i8> %d
   ret <16x i8> %e
 }
 
 define <16x i8> @test_select_cc_v16i8_f32(float %a, float %b, <16x i8> %c, <16x i8> %d ) {
 ; CHECK-LABEL: test_select_cc_v16i8_f32:
 ; CHECK: fcmeq [[MASK:v[0-9]+]].4s, v0.4s, v1.4s
 ; CHECK-NEXT: dup [[DUPMASK:v[0-9]+]].4s, [[MASK]].s[0]
 ; CHECK-NEXT: bsl [[DUPMASK]].16b, v2.16b, v3.16b

   %e = select i1 %cmp31, <16x i8> %c, <16x i8> %d
   ret <16x i8> %e
 }
 
 define <4x i16> @test_select_cc_v4i16(i16 %a, i16 %b, <4x i16> %c, <4x i16> %d ) {
 ; CHECK-LABEL: test_select_cc_v4i16:
 ; CHECK-DAG: fmov s[[LHS:[0-9]+]], w0
 ; CHECK-DAG: fmov s[[RHS:[0-9]+]], w1
 ; CHECK: cmeq [[MASK:v[0-9]+]].4h, v[[LHS]].4h, v[[RHS]].4h
 ; CHECK: dup [[DUPMASK:v[0-9]+]].4h, [[MASK]].h[0]
-; CHECK: bsl [[DUPMASK]].8b, v0.8b, v1.8b
+; CHECK: bif v0.8b, v1.8b, [[DUPMASK]].8b 
   %cmp31 = icmp eq i16 %a, %b
   %e = select i1 %cmp31, <4x i16> %c, <4x i16> %d
   ret <4x i16> %e
 }
 
 define <8x i16> @test_select_cc_v8i16(i16 %a, i16 %b, <8x i16> %c, <8x i16> %d ) {
 ; CHECK-LABEL: test_select_cc_v8i16:
 ; CHECK-DAG: fmov s[[LHS:[0-9]+]], w0
 ; CHECK-DAG: fmov s[[RHS:[0-9]+]], w1
 ; CHECK: cmeq [[MASK:v[0-9]+]].8h, v[[LHS]].8h, v[[RHS]].8h
 ; CHECK: dup [[DUPMASK:v[0-9]+]].8h, [[MASK]].h[0]
-; CHECK: bsl [[DUPMASK]].16b, v0.16b, v1.16b
+; CHECK: bif v0.16b, v1.16b, [[DUPMASK]].16b 
   %cmp31 = icmp eq i16 %a, %b
   %e = select i1 %cmp31, <8x i16> %c, <8x i16> %d
   ret <8x i16> %e
 }
 
 define <2x i32> @test_select_cc_v2i32(i32 %a, i32 %b, <2x i32> %c, <2x i32> %d ) {
 ; CHECK-LABEL: test_select_cc_v2i32:
 ; CHECK-DAG: fmov s[[LHS:[0-9]+]], w0
 ; CHECK-DAG: fmov s[[RHS:[0-9]+]], w1
 ; CHECK: cmeq [[MASK:v[0-9]+]].2s, v[[LHS]].2s, v[[RHS]].2s
 ; CHECK: dup [[DUPMASK:v[0-9]+]].2s, [[MASK]].s[0]
-; CHECK: bsl [[DUPMASK]].8b, v0.8b, v1.8b
+; CHECK: bif v0.8b, v1.8b, [[DUPMASK]].8b 
   %cmp31 = icmp eq i32 %a, %b
   %e = select i1 %cmp31, <2x i32> %c, <2x i32> %d
   ret <2x i32> %e
 }
 
 define <4x i32> @test_select_cc_v4i32(i32 %a, i32 %b, <4x i32> %c, <4x i32> %d ) {
 ; CHECK-LABEL: test_select_cc_v4i32:
 ; CHECK-DAG: fmov s[[LHS:[0-9]+]], w0
 ; CHECK-DAG: fmov s[[RHS:[0-9]+]], w1
 ; CHECK: cmeq [[MASK:v[0-9]+]].4s, v[[LHS]].4s, v[[RHS]].4s
 ; CHECK: dup [[DUPMASK:v[0-9]+]].4s, [[MASK]].s[0]
-; CHECK: bsl [[DUPMASK]].16b, v0.16b, v1.16b
+; CHECK: bif v0.16b, v1.16b, [[DUPMASK]].16b 
   %cmp31 = icmp eq i32 %a, %b
   %e = select i1 %cmp31, <4x i32> %c, <4x i32> %d
   ret <4x i32> %e
 }
 
 define <1x i64> @test_select_cc_v1i64(i64 %a, i64 %b, <1x i64> %c, <1x i64> %d ) {
 ; CHECK-LABEL: test_select_cc_v1i64:
 ; CHECK-DAG: fmov d[[LHS:[0-9]+]], x0
 ; CHECK-DAG: fmov d[[RHS:[0-9]+]], x1
 ; CHECK: cmeq d[[MASK:[0-9]+]], d[[LHS]], d[[RHS]]
-; CHECK: bsl v[[MASK]].8b, v0.8b, v1.8b
+; CHECK: bif v0.8b, v1.8b, v[[MASK]].8b 
   %cmp31 = icmp eq i64 %a, %b
   %e = select i1 %cmp31, <1x i64> %c, <1x i64> %d
   ret <1x i64> %e
 }
 
 define <2x i64> @test_select_cc_v2i64(i64 %a, i64 %b, <2x i64> %c, <2x i64> %d ) {
 ; CHECK-LABEL: test_select_cc_v2i64:
 ; CHECK-DAG: fmov d[[LHS:[0-9]+]], x0
 ; CHECK-DAG: fmov d[[RHS:[0-9]+]], x1
 ; CHECK: cmeq [[MASK:v[0-9]+]].2d, v[[LHS]].2d, v[[RHS]].2d
 ; CHECK: dup [[DUPMASK:v[0-9]+]].2d, [[MASK]].d[0]
-; CHECK: bsl [[DUPMASK]].16b, v0.16b, v1.16b
+; CHECK: bif v0.16b, v1.16b, [[DUPMASK]].16b 
   %cmp31 = icmp eq i64 %a, %b
   %e = select i1 %cmp31, <2x i64> %c, <2x i64> %d
   ret <2x i64> %e
 }
 
 define <1 x float> @test_select_cc_v1f32(float %a, float %b, <1 x float> %c, <1 x float> %d ) {
 ; CHECK-LABEL: test_select_cc_v1f32:
 ; CHECK: fcmeq [[MASK:v[0-9]+]].2s, v0.2s, v1.2s
 ; CHECK-NEXT: bsl [[MASK]].8b, v2.8b, v3.8b
   %cmp31 = fcmp oeq float %a, %b

   %e = select i1 %cmp31, <4x float> %c, <4x float> %d
   ret <4x float> %e
 }
 
 define <4x float> @test_select_cc_v4f32_icmp(i32 %a, i32 %b, <4x float> %c, <4x float> %d ) {
 ; CHECK-LABEL: test_select_cc_v4f32_icmp:
 ; CHECK-DAG: fmov s[[LHS:[0-9]+]], w0
 ; CHECK-DAG: fmov s[[RHS:[0-9]+]], w1
 ; CHECK: cmeq [[MASK:v[0-9]+]].4s, v[[LHS]].4s, v[[RHS]].4s
 ; CHECK: dup [[DUPMASK:v[0-9]+]].4s, [[MASK]].s[0]
-; CHECK: bsl [[DUPMASK]].16b, v0.16b, v1.16b
+; CHECK: bif v0.16b, v1.16b, [[DUPMASK]].16b 
   %cmp31 = icmp eq i32 %a, %b
   %e = select i1 %cmp31, <4x float> %c, <4x float> %d
   ret <4x float> %e
 }
 
 define <1 x double> @test_select_cc_v1f64(double %a, double %b, <1 x double> %c, <1 x double> %d ) {
 ; CHECK-LABEL: test_select_cc_v1f64:
 ; CHECK: fcmeq d[[MASK:[0-9]+]], d0, d1
 ; CHECK: bsl v[[MASK]].8b, v2.8b, v3.8b
   %cmp31 = fcmp oeq double %a, %b
   %e = select i1 %cmp31, <1 x double> %c, <1 x double> %d
   ret <1 x double> %e
 }
 
 define <1 x double> @test_select_cc_v1f64_icmp(i64 %a, i64 %b, <1 x double> %c, <1 x double> %d ) {
 ; CHECK-LABEL: test_select_cc_v1f64_icmp:
 ; CHECK-DAG: fmov [[LHS:d[0-9]+]], x0
 ; CHECK-DAG: fmov [[RHS:d[0-9]+]], x1
 ; CHECK: cmeq d[[MASK:[0-9]+]], [[LHS]], [[RHS]]
-; CHECK: bsl v[[MASK]].8b, v0.8b, v1.8b
+; CHECK: bif v0.8b, v1.8b, v[[MASK]].8b
   %cmp31 = icmp eq i64 %a, %b
   %e = select i1 %cmp31, <1 x double> %c, <1 x double> %d
   ret <1 x double> %e
 }
 
 define <2 x double> @test_select_cc_v2f64(double %a, double %b, <2 x double> %c, <2 x double> %d ) {
 ; CHECK-LABEL: test_select_cc_v2f64:
 ; CHECK: fcmeq [[MASK:v[0-9]+]].2d, v0.2d, v1.2d
 ; CHECK: dup [[DUPMASK:v[0-9]+]].2d, [[MASK]].d[0]
 ; CHECK: bsl [[DUPMASK]].16b, v2.16b, v3.16b

 }
 
 ; Special case: when the select condition is an icmp with i1 operands, don't
 ; do the comparison on vectors.
 ; Part of PR21549.
 define <2 x i32> @test_select_cc_v2i32_icmpi1(i1 %cc, <2 x i32> %a, <2 x i32> %b) {
 ; CHECK-LABEL: test_select_cc_v2i32_icmpi1:
 ; CHECK: tst   w0, #0x1
 ; CHECK: csetm [[MASK:w[0-9]+]], ne
 ; CHECK: dup   [[DUPMASK:v[0-9]+]].2s, [[MASK]]
-; CHECK: bsl   [[DUPMASK]].8b, v0.8b, v1.8b
-; CHECK: mov   v0.16b, [[DUPMASK]].16b
+; CHECK: bif   v0.8b, v1.8b, [[DUPMASK]].8b 
   %cmp = icmp ne i1 %cc, 0
   %e = select i1 %cmp, <2 x i32> %a, <2 x i32> %b
   ret <2 x i32> %e
 }
 
 ; Also make sure we support irregular/non-power-of-2 types such as v3f32.
 define <3 x float> @test_select_cc_v3f32_fcmp_f32(<3 x float> %a, <3 x float> %b, float %c1, float %c2) #0 {
 ; CHECK-LABEL: test_select_cc_v3f32_fcmp_f32:
 ; CHECK-NEXT: fcmeq [[MASK:v[0-9]+]].4s, v2.4s, v3.4s
 ; CHECK-NEXT: dup [[DUPMASK:v[0-9]+]].4s, [[MASK]].s[0]
-; CHECK-NEXT: bsl [[DUPMASK:v[0-9]+]].16b, v0.16b, v1.16b
-; CHECK-NEXT: mov v0.16b, [[DUPMASK]].16b
+; CHECK-NEXT: bif v0.16b, v1.16b, [[DUPMASK:v[0-9]+]].16b 
 ; CHECK-NEXT: ret
   %cc = fcmp oeq float %c1, %c2
   %r = select i1 %cc, <3 x float> %a, <3 x float> %b
   ret <3 x float> %r
 }
 
 define <3 x float> @test_select_cc_v3f32_fcmp_f64(<3 x float> %a, <3 x float> %b, double %c1, double %c2) #0 {
 ; CHECK-LABEL: test_select_cc_v3f32_fcmp_f64:
 ; CHECK-NEXT: fcmeq [[MASK:v[0-9]+]].2d, v2.2d, v3.2d
 ; CHECK-NEXT: dup [[DUPMASK:v[0-9]+]].2d, [[MASK]].d[0]
-; CHECK-NEXT: bsl [[DUPMASK:v[0-9]+]].16b, v0.16b, v1.16b
-; CHECK-NEXT: mov v0.16b, [[DUPMASK]].16b
+; CHECK-NEXT: bif v0.16b, v1.16b, [[DUPMASK:v[0-9]+]].16b 
 ; CHECK-NEXT: ret
   %cc = fcmp oeq double %c1, %c2
   %r = select i1 %cc, <3 x float> %a, <3 x float> %b
   ret <3 x float> %r
 }
 
 attributes #0 = { nounwind}

llvm/test/CodeGen/AArch64/fp16-vector-shuffle.ll

 ; RUN: llc < %s -mtriple=aarch64-none-eabi | FileCheck %s
 
 ; float16x4_t select_64(float16x4_t a, float16x4_t b, uint16x4_t c) { return vbsl_u16(c, a, b); }
 define <4 x half> @select_64(<4 x half> %a, <4 x half> %b, <4 x i16> %c) #0 {
 ; CHECK-LABEL: select_64:
-; CHECK: bsl
+; CHECK: bif
 entry:
   %0 = bitcast <4 x half> %a to <4 x i16>
   %1 = bitcast <4 x half> %b to <4 x i16>
   %vbsl3.i = and <4 x i16> %0, %c
   %2 = xor <4 x i16> %c, <i16 -1, i16 -1, i16 -1, i16 -1>
   %vbsl4.i = and <4 x i16> %1, %2
   %vbsl5.i = or <4 x i16> %vbsl3.i, %vbsl4.i
   %3 = bitcast <4 x i16> %vbsl5.i to <4 x half>
   ret <4 x half> %3
 }
 
 ; float16x8_t select_128(float16x8_t a, float16x8_t b, uint16x8_t c) { return vbslq_u16(c, a, b); }
 define <8 x half> @select_128(<8 x half> %a, <8 x half> %b, <8 x i16> %c) #0 {
 ; CHECK-LABEL: select_128:
-; CHECK: bsl
+; CHECK: bif
 entry:
   %0 = bitcast <8 x half> %a to <8 x i16>
   %1 = bitcast <8 x half> %b to <8 x i16>
   %vbsl3.i = and <8 x i16> %0, %c
   %2 = xor <8 x i16> %c, <i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1>
   %vbsl4.i = and <8 x i16> %1, %2
   %vbsl5.i = or <8 x i16> %vbsl3.i, %vbsl4.i
   %3 = bitcast <8 x i16> %vbsl5.i to <8 x half>
   ret <8 x half> %3
 }

llvm/test/CodeGen/AArch64/sat-add.ll

   ret <4 x i32> %r
 }
 
 define <2 x i64> @unsigned_sat_constant_v2i64_using_min(<2 x i64> %x) {
 ; CHECK-LABEL: unsigned_sat_constant_v2i64_using_min:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov x8, #-43
 ; CHECK-NEXT:    dup v1.2d, x8
 ; CHECK-NEXT:    mov w9, #42
 ; CHECK-NEXT:    cmhi v2.2d, v1.2d, v0.2d
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    dup v0.2d, x9
-; CHECK-NEXT:    add v0.2d, v2.2d, v0.2d
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    dup v1.2d, x9
+; CHECK-NEXT:    add v0.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    ret
   %c = icmp ult <2 x i64> %x, <i64 -43, i64 -43>
   %s = select <2 x i1> %c, <2 x i64> %x, <2 x i64> <i64 -43, i64 -43>
   %r = add <2 x i64> %s, <i64 42, i64 42>
   ret <2 x i64> %r
 }
 
 define <2 x i64> @unsigned_sat_constant_v2i64_using_cmp_sum(<2 x i64> %x) {
 ; CHECK-LABEL: unsigned_sat_constant_v2i64_using_cmp_sum:
 ; CHECK:       // %bb.0:

   %c = icmp ugt <4 x i32> %x, %noty
   %r = select <4 x i1> %c, <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, <4 x i32> %a
   ret <4 x i32> %r
 }
 
 define <2 x i64> @unsigned_sat_variable_v2i64_using_min(<2 x i64> %x, <2 x i64> %y) {
 ; CHECK-LABEL: unsigned_sat_variable_v2i64_using_min:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mvn v2.16b, v1.16b
 ; CHECK-NEXT:    cmhi v3.2d, v2.2d, v0.2d
-; CHECK-NEXT:    bsl v3.16b, v0.16b, v2.16b
-; CHECK-NEXT:    add v0.2d, v3.2d, v1.2d
+; CHECK-NEXT:    bif v0.16b, v2.16b, v3.16b
+; CHECK-NEXT:    add v0.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    ret
   %noty = xor <2 x i64> %y, <i64 -1, i64 -1>
   %c = icmp ult <2 x i64> %x, %noty
   %s = select <2 x i1> %c, <2 x i64> %x, <2 x i64> %noty
   %r = add <2 x i64> %s, %y
   ret <2 x i64> %r
 }
 
 define <2 x i64> @unsigned_sat_variable_v2i64_using_cmp_sum(<2 x i64> %x, <2 x i64> %y) {
 ; CHECK-LABEL: unsigned_sat_variable_v2i64_using_cmp_sum:

llvm/test/CodeGen/AArch64/sqrt-fastmath.ll

 ;
 ; CHECK-LABEL: f2sqrt:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    frsqrte v1.2s, v0.2s
 ; CHECK-NEXT:    fmul v2.2s, v1.2s, v1.2s
 ; CHECK-NEXT:    frsqrts v2.2s, v0.2s, v2.2s
 ; CHECK-NEXT:    fmul v1.2s, v1.2s, v2.2s
 ; CHECK-NEXT:    fmul v2.2s, v1.2s, v1.2s
 ; CHECK-NEXT:    frsqrts v2.2s, v0.2s, v2.2s
 ; CHECK-NEXT:    fmul v2.2s, v2.2s, v0.2s
-; CHECK-NEXT:    fmul v2.2s, v1.2s, v2.2s
-; CHECK-NEXT:    fcmeq v1.2s, v0.2s, #0.0
-; CHECK-NEXT:    bsl v1.8b, v0.8b, v2.8b
-; CHECK-NEXT:    mov v0.16b, v1.16b
+; CHECK-NEXT:    fmul v1.2s, v1.2s, v2.2s
+; CHECK-NEXT:    fcmeq v2.2s, v0.2s, #0.0
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %1 = tail call fast <2 x float> @llvm.sqrt.v2f32(<2 x float> %a)
   ret <2 x float> %1
 }
 
 define <4 x float> @f4sqrt(<4 x float> %a) #0 {
 ; FAULT-LABEL: f4sqrt:
 ; FAULT:       // %bb.0:
 ; FAULT-NEXT:    fsqrt v0.4s, v0.4s
 ; FAULT-NEXT:    ret
 ;
 ; CHECK-LABEL: f4sqrt:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    frsqrte v1.4s, v0.4s
 ; CHECK-NEXT:    fmul v2.4s, v1.4s, v1.4s
 ; CHECK-NEXT:    frsqrts v2.4s, v0.4s, v2.4s
 ; CHECK-NEXT:    fmul v1.4s, v1.4s, v2.4s
 ; CHECK-NEXT:    fmul v2.4s, v1.4s, v1.4s
 ; CHECK-NEXT:    frsqrts v2.4s, v0.4s, v2.4s
 ; CHECK-NEXT:    fmul v2.4s, v2.4s, v0.4s
-; CHECK-NEXT:    fmul v2.4s, v1.4s, v2.4s
-; CHECK-NEXT:    fcmeq v1.4s, v0.4s, #0.0
-; CHECK-NEXT:    bsl v1.16b, v0.16b, v2.16b
-; CHECK-NEXT:    mov v0.16b, v1.16b
+; CHECK-NEXT:    fmul v1.4s, v1.4s, v2.4s
+; CHECK-NEXT:    fcmeq v2.4s, v0.4s, #0.0
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %1 = tail call fast <4 x float> @llvm.sqrt.v4f32(<4 x float> %a)
   ret <4 x float> %1
 }
 
 define <8 x float> @f8sqrt(<8 x float> %a) #0 {
 ; FAULT-LABEL: f8sqrt:
 ; FAULT:       // %bb.0:
 ; FAULT-NEXT:    fsqrt v0.4s, v0.4s
 ; FAULT-NEXT:    fsqrt v1.4s, v1.4s

 ;
 ; CHECK-LABEL: f8sqrt:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    frsqrte v2.4s, v0.4s
 ; CHECK-NEXT:    fmul v3.4s, v2.4s, v2.4s
 ; CHECK-NEXT:    frsqrts v3.4s, v0.4s, v3.4s
 ; CHECK-NEXT:    fmul v2.4s, v2.4s, v3.4s
 ; CHECK-NEXT:    fmul v3.4s, v2.4s, v2.4s
 ; CHECK-NEXT:    frsqrts v3.4s, v0.4s, v3.4s
 ; CHECK-NEXT:    fmul v3.4s, v3.4s, v0.4s
-; CHECK-NEXT:    fmul v3.4s, v2.4s, v3.4s
-; CHECK-NEXT:    fcmeq v2.4s, v0.4s, #0.0
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v3.16b
-; CHECK-NEXT:    frsqrte v0.4s, v1.4s
-; CHECK-NEXT:    fmul v3.4s, v0.4s, v0.4s
+; CHECK-NEXT:    fmul v2.4s, v2.4s, v3.4s
+; CHECK-NEXT:    fcmeq v3.4s, v0.4s, #0.0
+; CHECK-NEXT:    bif v0.16b, v2.16b, v3.16b
+; CHECK-NEXT:    frsqrte v2.4s, v1.4s
+; CHECK-NEXT:    fmul v3.4s, v2.4s, v2.4s
 ; CHECK-NEXT:    frsqrts v3.4s, v1.4s, v3.4s
-; CHECK-NEXT:    fmul v0.4s, v0.4s, v3.4s
-; CHECK-NEXT:    fmul v3.4s, v0.4s, v0.4s
+; CHECK-NEXT:    fmul v2.4s, v2.4s, v3.4s
+; CHECK-NEXT:    fmul v3.4s, v2.4s, v2.4s
 ; CHECK-NEXT:    frsqrts v3.4s, v1.4s, v3.4s
 ; CHECK-NEXT:    fmul v3.4s, v3.4s, v1.4s
-; CHECK-NEXT:    fmul v0.4s, v0.4s, v3.4s
+; CHECK-NEXT:    fmul v2.4s, v2.4s, v3.4s
 ; CHECK-NEXT:    fcmeq v3.4s, v1.4s, #0.0
-; CHECK-NEXT:    bsl v3.16b, v1.16b, v0.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
-; CHECK-NEXT:    mov v1.16b, v3.16b
+; CHECK-NEXT:    bif v1.16b, v2.16b, v3.16b
 ; CHECK-NEXT:    ret
   %1 = tail call fast <8 x float> @llvm.sqrt.v8f32(<8 x float> %a)
   ret <8 x float> %1
 }
 
 define double @dsqrt(double %a) #0 {
 ; FAULT-LABEL: dsqrt:
 ; FAULT:       // %bb.0:
 ; FAULT-NEXT:    fsqrt d0, d0
 ; FAULT-NEXT:    ret

 ; CHECK-NEXT:    frsqrte v1.2d, v0.2d
 ; CHECK-NEXT:    fmul v2.2d, v1.2d, v1.2d
 ; CHECK-NEXT:    frsqrts v2.2d, v0.2d, v2.2d
 ; CHECK-NEXT:    fmul v1.2d, v1.2d, v2.2d
 ; CHECK-NEXT:    fmul v2.2d, v1.2d, v1.2d
 ; CHECK-NEXT:    frsqrts v2.2d, v0.2d, v2.2d
 ; CHECK-NEXT:    fmul v1.2d, v1.2d, v2.2d
 ; CHECK-NEXT:    fmul v2.2d, v1.2d, v1.2d
 ; CHECK-NEXT:    frsqrts v2.2d, v0.2d, v2.2d
 ; CHECK-NEXT:    fmul v2.2d, v2.2d, v0.2d
-; CHECK-NEXT:    fmul v2.2d, v1.2d, v2.2d
-; CHECK-NEXT:    fcmeq v1.2d, v0.2d, #0.0
-; CHECK-NEXT:    bsl v1.16b, v0.16b, v2.16b
-; CHECK-NEXT:    mov v0.16b, v1.16b
+; CHECK-NEXT:    fmul v1.2d, v1.2d, v2.2d
+; CHECK-NEXT:    fcmeq v2.2d, v0.2d, #0.0
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %1 = tail call fast <2 x double> @llvm.sqrt.v2f64(<2 x double> %a)
   ret <2 x double> %1
 }
 
 define <4 x double> @d4sqrt(<4 x double> %a) #0 {
 ; FAULT-LABEL: d4sqrt:
 ; FAULT:       // %bb.0:
 ; FAULT-NEXT:    fsqrt v0.2d, v0.2d
 ; FAULT-NEXT:    fsqrt v1.2d, v1.2d

 ; CHECK-NEXT:    frsqrte v2.2d, v0.2d
 ; CHECK-NEXT:    fmul v3.2d, v2.2d, v2.2d
 ; CHECK-NEXT:    frsqrts v3.2d, v0.2d, v3.2d
 ; CHECK-NEXT:    fmul v2.2d, v2.2d, v3.2d
 ; CHECK-NEXT:    fmul v3.2d, v2.2d, v2.2d
 ; CHECK-NEXT:    frsqrts v3.2d, v0.2d, v3.2d
 ; CHECK-NEXT:    fmul v2.2d, v2.2d, v3.2d
 ; CHECK-NEXT:    fmul v3.2d, v2.2d, v2.2d
 ; CHECK-NEXT:    frsqrts v3.2d, v0.2d, v3.2d
 ; CHECK-NEXT:    fmul v3.2d, v3.2d, v0.2d
-; CHECK-NEXT:    fmul v3.2d, v2.2d, v3.2d
-; CHECK-NEXT:    fcmeq v2.2d, v0.2d, #0.0
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v3.16b
-; CHECK-NEXT:    frsqrte v0.2d, v1.2d
-; CHECK-NEXT:    fmul v3.2d, v0.2d, v0.2d
+; CHECK-NEXT:    fmul v2.2d, v2.2d, v3.2d
+; CHECK-NEXT:    fcmeq v3.2d, v0.2d, #0.0
+; CHECK-NEXT:    bif v0.16b, v2.16b, v3.16b
+; CHECK-NEXT:    frsqrte v2.2d, v1.2d
+; CHECK-NEXT:    fmul v3.2d, v2.2d, v2.2d
 ; CHECK-NEXT:    frsqrts v3.2d, v1.2d, v3.2d
-; CHECK-NEXT:    fmul v0.2d, v0.2d, v3.2d
-; CHECK-NEXT:    fmul v3.2d, v0.2d, v0.2d
+; CHECK-NEXT:    fmul v2.2d, v2.2d, v3.2d
+; CHECK-NEXT:    fmul v3.2d, v2.2d, v2.2d
 ; CHECK-NEXT:    frsqrts v3.2d, v1.2d, v3.2d
-; CHECK-NEXT:    fmul v0.2d, v0.2d, v3.2d
-; CHECK-NEXT:    fmul v3.2d, v0.2d, v0.2d
+; CHECK-NEXT:    fmul v2.2d, v2.2d, v3.2d
+; CHECK-NEXT:    fmul v3.2d, v2.2d, v2.2d
 ; CHECK-NEXT:    frsqrts v3.2d, v1.2d, v3.2d
 ; CHECK-NEXT:    fmul v3.2d, v3.2d, v1.2d
-; CHECK-NEXT:    fmul v0.2d, v0.2d, v3.2d
+; CHECK-NEXT:    fmul v2.2d, v2.2d, v3.2d
 ; CHECK-NEXT:    fcmeq v3.2d, v1.2d, #0.0
-; CHECK-NEXT:    bsl v3.16b, v1.16b, v0.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
-; CHECK-NEXT:    mov v1.16b, v3.16b
+; CHECK-NEXT:    bif v1.16b, v2.16b, v3.16b
 ; CHECK-NEXT:    ret
   %1 = tail call fast <4 x double> @llvm.sqrt.v4f64(<4 x double> %a)
   ret <4 x double> %1
 }
 
 define float @frsqrt(float %a) #0 {
 ; FAULT-LABEL: frsqrt:
 ; FAULT:       // %bb.0:
 ; FAULT-NEXT:    fsqrt s0, s0
 ; FAULT-NEXT:    fmov s1, #1.00000000

llvm/test/CodeGen/AArch64/unfold-masked-merge-vector-variablemask-const.ll

   %n0 = xor <4 x i32> %x, <i32 -1, i32 -1, i32 -1, i32 -1> ; %x
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, <i32 -1, i32 -1, i32 -1, i32 -1>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @out_constant_varx_42(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: out_constant_varx_42:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v1.4s, #42
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %mask, %x
   %my = and <4 x i32> %notmask, <i32 42, i32 42, i32 42, i32 42>
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @in_constant_varx_42(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: in_constant_varx_42:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v1.4s, #42
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i32> %x, <i32 42, i32 42, i32 42, i32 42> ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, <i32 42, i32 42, i32 42, i32 42>
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @out_constant_varx_42_invmask(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: out_constant_varx_42_invmask:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v1.4s, #42
-; CHECK-NEXT:    bsl v2.16b, v1.16b, v0.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bit v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %notmask, %x
   %my = and <4 x i32> %mask, <i32 42, i32 42, i32 42, i32 42>
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @in_constant_varx_42_invmask(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: in_constant_varx_42_invmask:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v1.4s, #42
-; CHECK-NEXT:    bsl v2.16b, v1.16b, v0.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bit v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %n0 = xor <4 x i32> %x, <i32 42, i32 42, i32 42, i32 42> ; %x
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, <i32 42, i32 42, i32 42, i32 42>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @out_constant_mone_vary(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: out_constant_mone_vary:

   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %n0 = xor <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, %y ; %x
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
 }
 
 define <4 x i32> @out_constant_42_vary(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: out_constant_42_vary:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov v0.16b, v2.16b
-; CHECK-NEXT:    movi v2.4s, #42
-; CHECK-NEXT:    bsl v0.16b, v2.16b, v1.16b
+; CHECK-NEXT:    movi v0.4s, #42
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %mask, <i32 42, i32 42, i32 42, i32 42>
   %my = and <4 x i32> %notmask, %y
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @in_constant_42_vary(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: in_constant_42_vary:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov v0.16b, v2.16b
-; CHECK-NEXT:    movi v2.4s, #42
-; CHECK-NEXT:    bsl v0.16b, v2.16b, v1.16b
+; CHECK-NEXT:    movi v0.4s, #42
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i32> <i32 42, i32 42, i32 42, i32 42>, %y ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @out_constant_42_vary_invmask(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: out_constant_42_vary_invmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov v0.16b, v2.16b
-; CHECK-NEXT:    movi v2.4s, #42
-; CHECK-NEXT:    bsl v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    movi v0.4s, #42
+; CHECK-NEXT:    bit v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %notmask, <i32 42, i32 42, i32 42, i32 42>
   %my = and <4 x i32> %mask, %y
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @in_constant_42_vary_invmask(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: in_constant_42_vary_invmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov v0.16b, v2.16b
-; CHECK-NEXT:    movi v2.4s, #42
-; CHECK-NEXT:    bsl v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    movi v0.4s, #42
+; CHECK-NEXT:    bit v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %n0 = xor <4 x i32> <i32 42, i32 42, i32 42, i32 42>, %y ; %x
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
 }

llvm/test/CodeGen/AArch64/unfold-masked-merge-vector-variablemask.ll

 ; All the advanced stuff (negative tests, commutativity) is handled in the
 ; scalar version of the test only.
 
 ; ============================================================================ ;
 ; 8-bit vector width
 ; ============================================================================ ;
 
 define <1 x i8> @out_v1i8(<1 x i8> %x, <1 x i8> %y, <1 x i8> %mask) nounwind {
 ; CHECK-LABEL: out_v1i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %mx = and <1 x i8> %x, %mask
   %notmask = xor <1 x i8> %mask, <i8 -1>
   %my = and <1 x i8> %y, %notmask
   %r = or <1 x i8> %mx, %my
   ret <1 x i8> %r
 }
 
 ; ============================================================================ ;
 ; 16-bit vector width

   %mx = and <2 x i8> %x, %mask
   %notmask = xor <2 x i8> %mask, <i8 -1, i8 -1>
   %my = and <2 x i8> %y, %notmask
   %r = or <2 x i8> %mx, %my
   ret <2 x i8> %r
 }
 
 define <1 x i16> @out_v1i16(<1 x i16> %x, <1 x i16> %y, <1 x i16> %mask) nounwind {
 ; CHECK-LABEL: out_v1i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %mx = and <1 x i16> %x, %mask
   %notmask = xor <1 x i16> %mask, <i16 -1>
   %my = and <1 x i16> %y, %notmask
   %r = or <1 x i16> %mx, %my
   ret <1 x i16> %r
 }
 
 ; ============================================================================ ;
 ; 32-bit vector width

   %mx = and <2 x i16> %x, %mask
   %notmask = xor <2 x i16> %mask, <i16 -1, i16 -1>
   %my = and <2 x i16> %y, %notmask
   %r = or <2 x i16> %mx, %my
   ret <2 x i16> %r
 }
 
 define <1 x i32> @out_v1i32(<1 x i32> %x, <1 x i32> %y, <1 x i32> %mask) nounwind {
 ; CHECK-LABEL: out_v1i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %mx = and <1 x i32> %x, %mask
   %notmask = xor <1 x i32> %mask, <i32 -1>
   %my = and <1 x i32> %y, %notmask
   %r = or <1 x i32> %mx, %my
   ret <1 x i32> %r
 }
 
 ; ============================================================================ ;
 ; 64-bit vector width
 ; ============================================================================ ;
 
 define <8 x i8> @out_v8i8(<8 x i8> %x, <8 x i8> %y, <8 x i8> %mask) nounwind {
 ; CHECK-LABEL: out_v8i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %mx = and <8 x i8> %x, %mask
   %notmask = xor <8 x i8> %mask, <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
   %my = and <8 x i8> %y, %notmask
   %r = or <8 x i8> %mx, %my
   ret <8 x i8> %r
 }
 
 define <4 x i16> @out_v4i16(<4 x i16> %x, <4 x i16> %y, <4 x i16> %mask) nounwind {
 ; CHECK-LABEL: out_v4i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %mx = and <4 x i16> %x, %mask
   %notmask = xor <4 x i16> %mask, <i16 -1, i16 -1, i16 -1, i16 -1>
   %my = and <4 x i16> %y, %notmask
   %r = or <4 x i16> %mx, %my
   ret <4 x i16> %r
 }
 
 define <4 x i16> @out_v4i16_undef(<4 x i16> %x, <4 x i16> %y, <4 x i16> %mask) nounwind {
 ; CHECK-LABEL: out_v4i16_undef:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %mx = and <4 x i16> %x, %mask
   %notmask = xor <4 x i16> %mask, <i16 -1, i16 -1, i16 undef, i16 -1>
   %my = and <4 x i16> %y, %notmask
   %r = or <4 x i16> %mx, %my
   ret <4 x i16> %r
 }
 
 define <2 x i32> @out_v2i32(<2 x i32> %x, <2 x i32> %y, <2 x i32> %mask) nounwind {
 ; CHECK-LABEL: out_v2i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %mx = and <2 x i32> %x, %mask
   %notmask = xor <2 x i32> %mask, <i32 -1, i32 -1>
   %my = and <2 x i32> %y, %notmask
   %r = or <2 x i32> %mx, %my
   ret <2 x i32> %r
 }
 
 define <1 x i64> @out_v1i64(<1 x i64> %x, <1 x i64> %y, <1 x i64> %mask) nounwind {
 ; CHECK-LABEL: out_v1i64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %mx = and <1 x i64> %x, %mask
   %notmask = xor <1 x i64> %mask, <i64 -1>
   %my = and <1 x i64> %y, %notmask
   %r = or <1 x i64> %mx, %my
   ret <1 x i64> %r
 }
 
 ; ============================================================================ ;
 ; 128-bit vector width
 ; ============================================================================ ;
 
 define <16 x i8> @out_v16i8(<16 x i8> %x, <16 x i8> %y, <16 x i8> %mask) nounwind {
 ; CHECK-LABEL: out_v16i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %mx = and <16 x i8> %x, %mask
   %notmask = xor <16 x i8> %mask, <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
   %my = and <16 x i8> %y, %notmask
   %r = or <16 x i8> %mx, %my
   ret <16 x i8> %r
 }
 
 define <8 x i16> @out_v8i16(<8 x i16> %x, <8 x i16> %y, <8 x i16> %mask) nounwind {
 ; CHECK-LABEL: out_v8i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %mx = and <8 x i16> %x, %mask
   %notmask = xor <8 x i16> %mask, <i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1>
   %my = and <8 x i16> %y, %notmask
   %r = or <8 x i16> %mx, %my
   ret <8 x i16> %r
 }
 
 define <4 x i32> @out_v4i32(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) nounwind {
 ; CHECK-LABEL: out_v4i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %mx = and <4 x i32> %x, %mask
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %my = and <4 x i32> %y, %notmask
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @out_v4i32_undef(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) nounwind {
 ; CHECK-LABEL: out_v4i32_undef:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %mx = and <4 x i32> %x, %mask
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 undef, i32 -1>
   %my = and <4 x i32> %y, %notmask
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <2 x i64> @out_v2i64(<2 x i64> %x, <2 x i64> %y, <2 x i64> %mask) nounwind {
 ; CHECK-LABEL: out_v2i64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %mx = and <2 x i64> %x, %mask
   %notmask = xor <2 x i64> %mask, <i64 -1, i64 -1>
   %my = and <2 x i64> %y, %notmask
   %r = or <2 x i64> %mx, %my
   ret <2 x i64> %r
 }
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ; Should be the same as the previous one.
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
 ; ============================================================================ ;
 ; 8-bit vector width
 ; ============================================================================ ;
 
 define <1 x i8> @in_v1i8(<1 x i8> %x, <1 x i8> %y, <1 x i8> %mask) nounwind {
 ; CHECK-LABEL: in_v1i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %n0 = xor <1 x i8> %x, %y
   %n1 = and <1 x i8> %n0, %mask
   %r = xor <1 x i8> %n1, %y
   ret <1 x i8> %r
 }
 
 ; ============================================================================ ;
 ; 16-bit vector width
 ; ============================================================================ ;
 
 define <2 x i8> @in_v2i8(<2 x i8> %x, <2 x i8> %y, <2 x i8> %mask) nounwind {
 ; CHECK-LABEL: in_v2i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %n0 = xor <2 x i8> %x, %y
   %n1 = and <2 x i8> %n0, %mask
   %r = xor <2 x i8> %n1, %y
   ret <2 x i8> %r
 }
 
 define <1 x i16> @in_v1i16(<1 x i16> %x, <1 x i16> %y, <1 x i16> %mask) nounwind {
 ; CHECK-LABEL: in_v1i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %n0 = xor <1 x i16> %x, %y
   %n1 = and <1 x i16> %n0, %mask
   %r = xor <1 x i16> %n1, %y
   ret <1 x i16> %r
 }
 
 ; ============================================================================ ;
 ; 32-bit vector width
 ; ============================================================================ ;
 
 define <4 x i8> @in_v4i8(<4 x i8> %x, <4 x i8> %y, <4 x i8> %mask) nounwind {
 ; CHECK-LABEL: in_v4i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i8> %x, %y
   %n1 = and <4 x i8> %n0, %mask
   %r = xor <4 x i8> %n1, %y
   ret <4 x i8> %r
 }
 
 define <2 x i16> @in_v2i16(<2 x i16> %x, <2 x i16> %y, <2 x i16> %mask) nounwind {
 ; CHECK-LABEL: in_v2i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %n0 = xor <2 x i16> %x, %y
   %n1 = and <2 x i16> %n0, %mask
   %r = xor <2 x i16> %n1, %y
   ret <2 x i16> %r
 }
 
 define <1 x i32> @in_v1i32(<1 x i32> %x, <1 x i32> %y, <1 x i32> %mask) nounwind {
 ; CHECK-LABEL: in_v1i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %n0 = xor <1 x i32> %x, %y
   %n1 = and <1 x i32> %n0, %mask
   %r = xor <1 x i32> %n1, %y
   ret <1 x i32> %r
 }
 
 ; ============================================================================ ;
 ; 64-bit vector width
 ; ============================================================================ ;
 
 define <8 x i8> @in_v8i8(<8 x i8> %x, <8 x i8> %y, <8 x i8> %mask) nounwind {
 ; CHECK-LABEL: in_v8i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %n0 = xor <8 x i8> %x, %y
   %n1 = and <8 x i8> %n0, %mask
   %r = xor <8 x i8> %n1, %y
   ret <8 x i8> %r
 }
 
 define <4 x i16> @in_v4i16(<4 x i16> %x, <4 x i16> %y, <4 x i16> %mask) nounwind {
 ; CHECK-LABEL: in_v4i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i16> %x, %y
   %n1 = and <4 x i16> %n0, %mask
   %r = xor <4 x i16> %n1, %y
   ret <4 x i16> %r
 }
 
 define <2 x i32> @in_v2i32(<2 x i32> %x, <2 x i32> %y, <2 x i32> %mask) nounwind {
 ; CHECK-LABEL: in_v2i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %n0 = xor <2 x i32> %x, %y
   %n1 = and <2 x i32> %n0, %mask
   %r = xor <2 x i32> %n1, %y
   ret <2 x i32> %r
 }
 
 define <1 x i64> @in_v1i64(<1 x i64> %x, <1 x i64> %y, <1 x i64> %mask) nounwind {
 ; CHECK-LABEL: in_v1i64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %n0 = xor <1 x i64> %x, %y
   %n1 = and <1 x i64> %n0, %mask
   %r = xor <1 x i64> %n1, %y
   ret <1 x i64> %r
 }
 
 ; ============================================================================ ;
 ; 128-bit vector width
 ; ============================================================================ ;
 
 define <16 x i8> @in_v16i8(<16 x i8> %x, <16 x i8> %y, <16 x i8> %mask) nounwind {
 ; CHECK-LABEL: in_v16i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <16 x i8> %x, %y
   %n1 = and <16 x i8> %n0, %mask
   %r = xor <16 x i8> %n1, %y
   ret <16 x i8> %r
 }
 
 define <8 x i16> @in_v8i16(<8 x i16> %x, <8 x i16> %y, <8 x i16> %mask) nounwind {
 ; CHECK-LABEL: in_v8i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <8 x i16> %x, %y
   %n1 = and <8 x i16> %n0, %mask
   %r = xor <8 x i16> %n1, %y
   ret <8 x i16> %r
 }
 
 define <4 x i32> @in_v4i32(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) nounwind {
 ; CHECK-LABEL: in_v4i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i32> %x, %y
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
 }
 
 define <2 x i64> @in_v2i64(<2 x i64> %x, <2 x i64> %y, <2 x i64> %mask) nounwind {
 ; CHECK-LABEL: in_v2i64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <2 x i64> %x, %y
   %n1 = and <2 x i64> %n0, %mask
   %r = xor <2 x i64> %n1, %y
   ret <2 x i64> %r
 }