[CodeGen][AArch64] Ensure isSExtCheaperThanZExt returns true for negative constants

[david-arm]

Summary

When we know the value we're extending is a negative constant then it
makes sense to use SIGN_EXTEND because this may improve code quality in
some cases, particularly when doing a constant splat of an unpacked vector
type. For example, for SVE when splatting the value -1 into all elements
of a vector of type <vscale x 2 x i32> the element type will get promoted
from i32 -> i64. In this case we want the splat value to sign-extend from
(i32 -1) -> (i64 -1), whereas currently it zero-extends from
(i32 -1) -> (i64 0xFFFFFFFF). Sign-extending the constant means we can use
a single mov immediate instruction.

New tests added here:

CodeGen/AArch64/sve-vector-splat.ll

I believe we see some code quality improvements in these existing
tests too:

CodeGen/AArch64/dag-numsignbits.ll
CodeGen/AArch64/reduce-and.ll
CodeGen/AArch64/unfold-masked-merge-vector-variablemask.ll

The apparent regressions in CodeGen/AArch64/fast-isel-cmp-vec.ll only
occur because the test disables codegen prepare and branch folding.

[peterwaller-arm]

Couple of discussion points. I can see the rationale, but I wonder about some of the test changes and whether this could be revealing a latent bug.

llvm/test/CodeGen/X86/vector-fshl-512.ll

1107 ↗

(On Diff #388865)

Can anyone comment if the sign extensions in these constants are NFC?

llvm/test/CodeGen/X86/vector-shift-shl-512.ll

334 ↗

(On Diff #388865)

Can anyone comment if these sll -> mul changes are expected & harmless?

[lebedev.ri]

llvm/test/CodeGen/X86/vector-shift-shl-512.ll

334 ↗

(On Diff #388865)

These look like regressions to me.

[david-arm]

llvm/test/CodeGen/X86/vector-shift-shl-512.ll

334 ↗

(On Diff #388865)

Hi @lebedev.ri, thanks for providing input here. Do you know if this means that there is a latent bug in the code where we should be explicitly using zero-extend here? I'm worried about cases where code is relying upon getAnyExtOrTrunc zero-extending.

[RKSimon]

llvm/test/CodeGen/X86/vector-shift-shl-512.ll

334 ↗

(On Diff #388865)

I'm not certain, but vXi8 shl by constant will fold to vXi8 multiplies by (pow2) constant, and will then be extended to vXi16 to make use of PMULLW - the upper bits aren't demanded so were any-extended, which was treated as a zero-extension during constant folding which preserved the pow2 nature which allowed it to be lowered back to a vXi16 shl. My guess is that now that they are sign-extended, the pow2 isn't seen any more.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

1319

There's no guarantee that the caller would use getAnyExtOrTrunc. I think this should be in the constant folding for getNode(ISD::ANY_EXTEND)

[craig.topper]

llvm/test/CodeGen/AArch64/sve-vector-splat.ll

119

Pre-commit the new tests so we can see the change?

[david-arm]

• Rewrote the patch to make use of isSExtCheaperThanZExt instead so that this becomes a AArch64-specific change.

[craig.topper]

llvm/lib/Target/AArch64/AArch64ISelLowering.h

1143

I think you can write if (!V)

[david-arm]

• Use if (!V) when checking for null SDValue objects.

[RKSimon]

llvm/include/llvm/CodeGen/TargetLowering.h

2643

Please can you update the documentation to explain the V arg (is it the src/dst?) and that it can be SDValue() if unknown.

[david-arm]

• Updated comments above isSExtCheaperThanZExt.

[RKSimon]

LGTM with one minor

llvm/test/CodeGen/AArch64/fast-isel-cmp-vec.ll

2

Regenerate + pre-commit before committing the patch so the patch just shows the codegen diff.

[mstorsjo]

I've bisected a miscompilation to this file.

To reproduce:

$ git clone git://source.ffmpeg.org/ffmpeg
$ cd ffmpeg
$ ./configure --cc=clang --samples=$(pwd)/../samples
$ make fate-rsync
$ make -j$(nproc)
$ make fate-dpcm-interplay

The breakage happens in the libavformat/ipmovie.c file. (I also saw a couple other broken tests, so I think there are other source files affected too, but I didn't bisect and pinpoint those failures.)

The issue can be observed with https://martin.st/temp/ipmovie-preproc.c, with clang -target aarch64-linux-gnu -O3 -o - ipmovie-preproc.c. The generated code contains differences like this:

--- old.s       2022-01-18 10:30:24.726016244 +0200
+++ new.s       2022-01-18 10:30:01.650536299 +0200
@@ -506,7 +506,7 @@
        mov     w1, #56
        bl      av_log
        add     x9, x19, #1104
-       mov     w21, #65535
+       mov     w21, #-1
 .LBB3_9:                                // %while.end
        ldr     x0, [x19]
        ldr     w8, [x0, #44]

[david-arm]

In D114357#3250506, @mstorsjo wrote:

I've bisected a miscompilation to this file.

To reproduce:

$ git clone git://source.ffmpeg.org/ffmpeg
$ cd ffmpeg
$ ./configure --cc=clang --samples=$(pwd)/../samples
$ make fate-rsync
$ make -j$(nproc)
$ make fate-dpcm-interplay

The breakage happens in the libavformat/ipmovie.c file. (I also saw a couple other broken tests, so I think there are other source files affected too, but I didn't bisect and pinpoint those failures.)

The issue can be observed with https://martin.st/temp/ipmovie-preproc.c, with clang -target aarch64-linux-gnu -O3 -o - ipmovie-preproc.c. The generated code contains differences like this:

--- old.s       2022-01-18 10:30:24.726016244 +0200
+++ new.s       2022-01-18 10:30:01.650536299 +0200
@@ -506,7 +506,7 @@
        mov     w1, #56
        bl      av_log
        add     x9, x19, #1104
-       mov     w21, #65535
+       mov     w21, #-1
 .LBB3_9:                                // %while.end
        ldr     x0, [x19]
        ldr     w8, [x0, #44]

Hi @mstorsjo, thanks for the info. I'll revert the patch again for now. It's strange because ANY_EXTEND should really mean "any", i.e. you don't care if it's sign-extend or zero-extend! I suspect there is a bug in codegen somewhere that either relies upon ANY_EXTEND actually being ZERO_EXTEND, or relies upon ANY_EXTEND being consistently the same. I'm worried because the isSExtCheaperThanZExt interface definitely allows for the possibility of sometimes choosing one over the other depending upon the types.

[mstorsjo]

In D114357#3250516, @david-arm wrote:

Hi @mstorsjo, thanks for the info. I'll revert the patch again for now. It's strange because ANY_EXTEND should really mean "any", i.e. you don't care if it's sign-extend or zero-extend! I suspect there is a bug in codegen somewhere that either relies upon ANY_EXTEND actually being ZERO_EXTEND, or relies upon ANY_EXTEND being consistently the same. I'm worried because the isSExtCheaperThanZExt interface definitely allows for the possibility of sometimes choosing one over the other depending upon the types.

Thanks, and that does indeed seem worrying.

For the sake of finding other possible similar related cases, with the same instructions, but with make -j$(nproc) fate it runs all tests - which currently triggered 24 failed tests (of which I believe there's maybe a 3-6 actual individual breakages), if you want to rerun more tests when you think you've pinpointed the root cause.

[craig.topper]

In D114357#3250516, @david-arm wrote:

In D114357#3250506, @mstorsjo wrote:

I've bisected a miscompilation to this file.

To reproduce:

$ git clone git://source.ffmpeg.org/ffmpeg
$ cd ffmpeg
$ ./configure --cc=clang --samples=$(pwd)/../samples
$ make fate-rsync
$ make -j$(nproc)
$ make fate-dpcm-interplay

The breakage happens in the libavformat/ipmovie.c file. (I also saw a couple other broken tests, so I think there are other source files affected too, but I didn't bisect and pinpoint those failures.)

The issue can be observed with https://martin.st/temp/ipmovie-preproc.c, with clang -target aarch64-linux-gnu -O3 -o - ipmovie-preproc.c. The generated code contains differences like this:

--- old.s       2022-01-18 10:30:24.726016244 +0200
+++ new.s       2022-01-18 10:30:01.650536299 +0200
@@ -506,7 +506,7 @@
        mov     w1, #56
        bl      av_log
        add     x9, x19, #1104
-       mov     w21, #65535
+       mov     w21, #-1
 .LBB3_9:                                // %while.end
        ldr     x0, [x19]
        ldr     w8, [x0, #44]

Hi @mstorsjo, thanks for the info. I'll revert the patch again for now. It's strange because ANY_EXTEND should really mean "any", i.e. you don't care if it's sign-extend or zero-extend! I suspect there is a bug in codegen somewhere that either relies upon ANY_EXTEND actually being ZERO_EXTEND, or relies upon ANY_EXTEND being consistently the same. I'm worried because the isSExtCheaperThanZExt interface definitely allows for the possibility of sometimes choosing one over the other depending upon the types.

I believe the culprit is FunctionLoweringInfo::ComputePHILiveOutRegInfo which assumes that ConstantInt inputs to phis will be zero extended.

llvm/include/llvm/CodeGen/TargetLowering.h

public:
   virtual bool isZExtFree(EVT FromTy, EVT ToTy) const { return false; }
   virtual bool isZExtFree(LLT FromTy, LLT ToTy, const DataLayout &DL,
                           LLVMContext &Ctx) const {
     return isZExtFree(getApproximateEVTForLLT(FromTy, DL, Ctx),
                       getApproximateEVTForLLT(ToTy, DL, Ctx));
   }
 
   /// Return true if sign-extension from FromTy to ToTy is cheaper than
   /// zero-extension.
-  virtual bool isSExtCheaperThanZExt(EVT FromTy, EVT ToTy) const {
+  virtual bool isSExtCheaperThanZExt(EVT FromTy, EVT ToTy, SDValue V) const {
     return false;
   }
 
   /// Return true if sinking I's operands to the same basic block as I is
   /// profitable, e.g. because the operands can be folded into a target
   /// instruction during instruction selection. After calling the function
   /// \p Ops contains the Uses to sink ordered by dominance (dominating users
   /// come first).

llvm/lib/CodeGen/CodeGenPrepare.cpp

bool CodeGenPrepare::optimizeSwitchInst(SwitchInst *SI) {
   auto *NewType = Type::getIntNTy(Context, RegWidth);
 
   // Extend the switch condition and case constants using the target preferred
   // extend unless the switch condition is a function argument with an extend
   // attribute. In that case, we can avoid an unnecessary mask/extension by
   // matching the argument extension instead.
   Instruction::CastOps ExtType = Instruction::ZExt;
   // Some targets prefer SExt over ZExt.
-  if (TLI->isSExtCheaperThanZExt(OldVT, RegType))
+  if (TLI->isSExtCheaperThanZExt(OldVT, RegType, SDValue()))
     ExtType = Instruction::SExt;
 
   if (auto *Arg = dyn_cast<Argument>(Cond)) {
     if (Arg->hasSExtAttr())
       ExtType = Instruction::SExt;
     if (Arg->hasZExtAttr())
       ExtType = Instruction::ZExt;
   }

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

void DAGTypeLegalizer::PromoteSetCCOperands(SDValue &LHS, SDValue &RHS,
   }
 
   assert((ISD::isUnsignedIntSetCC(CCCode) || ISD::isIntEqualitySetCC(CCCode)) &&
          "Unknown integer comparison!");
 
   SDValue OpL = GetPromotedInteger(LHS);
   SDValue OpR = GetPromotedInteger(RHS);
 
-  if (TLI.isSExtCheaperThanZExt(LHS.getValueType(), OpL.getValueType())) {
+  if (TLI.isSExtCheaperThanZExt(LHS.getValueType(), OpL.getValueType(), LHS)) {
     // The target would prefer to promote the comparison operand with sign
     // extension. Honor that unless the promoted values are already zero
     // extended.
     unsigned OpLEffectiveBits =
         DAG.computeKnownBits(OpL).countMaxActiveBits();
     unsigned OpREffectiveBits =
         DAG.computeKnownBits(OpR).countMaxActiveBits();
     if (OpLEffectiveBits <= LHS.getScalarValueSizeInBits() &&

llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h

private:
   // Get a promoted operand and sign or zero extend it to the final size
   // (depending on TargetLoweringInfo::isSExtCheaperThanZExt). For a given
   // subtarget and type, the choice of sign or zero-extension will be
   // consistent.
   SDValue SExtOrZExtPromotedInteger(SDValue Op) {
     EVT OldVT = Op.getValueType();
     SDLoc DL(Op);
     Op = GetPromotedInteger(Op);
-    if (TLI.isSExtCheaperThanZExt(OldVT, Op.getValueType()))
+    if (TLI.isSExtCheaperThanZExt(OldVT, Op.getValueType(), Op))
       return DAG.getNode(ISD::SIGN_EXTEND_INREG, DL, Op.getValueType(), Op,
                          DAG.getValueType(OldVT));
     return DAG.getZeroExtendInReg(Op, DL, OldVT);
   }
 
   // Promote the given operand V (vector or scalar) according to N's specific
   // reduction kind. N must be an integer VECREDUCE_* or VP_REDUCE_*. Returns
   // the nominal extension opcode (ISD::(ANY|ZERO|SIGN)_EXTEND) and the

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

SDValue Res =
                     {Chain, Op, getIntPtrConstant(0, DL)});
 
   return std::pair<SDValue, SDValue>(Res, SDValue(Res.getNode(), 1));
 }
 
 SDValue SelectionDAG::getAnyExtOrTrunc(SDValue Op, const SDLoc &DL, EVT VT) {
   return VT.bitsGT(Op.getValueType()) ?
     getNode(ISD::ANY_EXTEND, DL, VT, Op) :
     getNode(ISD::TRUNCATE, DL, VT, Op);
 }
 
 SDValue SelectionDAG::getSExtOrTrunc(SDValue Op, const SDLoc &DL, EVT VT) {
   return VT.bitsGT(Op.getValueType()) ?
     getNode(ISD::SIGN_EXTEND, DL, VT, Op) :
     getNode(ISD::TRUNCATE, DL, VT, Op);
 }
 
case ISD::TRUNCATE:
       if (C->isOpaque())
         break;
       LLVM_FALLTHROUGH;
     case ISD::ZERO_EXTEND:
       return getConstant(Val.zextOrTrunc(VT.getSizeInBits()), DL, VT,
                          C->isTargetOpcode(), C->isOpaque());
     case ISD::ANY_EXTEND:
       // Some targets like RISCV prefer to sign extend some types.
-      if (TLI->isSExtCheaperThanZExt(Operand.getValueType(), VT))
+      if (TLI->isSExtCheaperThanZExt(Operand.getValueType(), VT, Operand))
         return getConstant(Val.sextOrTrunc(VT.getSizeInBits()), DL, VT,
                            C->isTargetOpcode(), C->isOpaque());
       return getConstant(Val.zextOrTrunc(VT.getSizeInBits()), DL, VT,
                          C->isTargetOpcode(), C->isOpaque());
     case ISD::UINT_TO_FP:
     case ISD::SINT_TO_FP: {
       APFloat apf(EVTToAPFloatSemantics(VT),
                   APInt::getZero(VT.getSizeInBits()));

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

if (N0.getOpcode() == ISD::ZERO_EXTEND) {
         break;
       }
       default:
         break; // todo, be more careful with signed comparisons
       }
     } else if (N0.getOpcode() == ISD::SIGN_EXTEND_INREG &&
                (Cond == ISD::SETEQ || Cond == ISD::SETNE) &&
                !isSExtCheaperThanZExt(cast<VTSDNode>(N0.getOperand(1))->getVT(),
-                                      OpVT)) {
+                                      OpVT, N0.getOperand(1))) {
       EVT ExtSrcTy = cast<VTSDNode>(N0.getOperand(1))->getVT();
       unsigned ExtSrcTyBits = ExtSrcTy.getSizeInBits();
       EVT ExtDstTy = N0.getValueType();
       unsigned ExtDstTyBits = ExtDstTy.getSizeInBits();
 
       // If the constant doesn't fit into the number of bits for the source of
       // the sign extension, it is impossible for both sides to be equal.
       if (C1.getMinSignedBits() > ExtSrcTyBits)

llvm/lib/Target/AArch64/AArch64ISelLowering.h

private:
   //     cannot be used to convert between unpacked and packed types.
   // These can make "bitcasting" a multiphase process. REINTERPRET_CAST is used
   // to transition between unpacked and packed types of the same element type,
   // with BITCAST used otherwise.
   SDValue getSVESafeBitCast(EVT VT, SDValue Op, SelectionDAG &DAG) const;
 
   bool isConstantUnsignedBitfieldExtactLegal(unsigned Opc, LLT Ty1,
                                              LLT Ty2) const override;
+
+  bool isSExtCheaperThanZExt(EVT SrcVT, EVT DstVT, SDValue V) const override {
+    if (V == SDValue())
+      return false;
+    if (ConstantSDNode *C = isConstOrConstSplat(V))
+      return C->getAPIntValue().isNegative();
+    return false;
+  }
 };
 
 namespace AArch64 {
 FastISel *createFastISel(FunctionLoweringInfo &funcInfo,
                          const TargetLibraryInfo *libInfo);
 } // end namespace AArch64
 
 } // end namespace llvm
 
 #endif

llvm/lib/Target/RISCV/RISCVISelLowering.h

public:
   bool isLegalAddressingMode(const DataLayout &DL, const AddrMode &AM, Type *Ty,
                              unsigned AS,
                              Instruction *I = nullptr) const override;
   bool isLegalICmpImmediate(int64_t Imm) const override;
   bool isLegalAddImmediate(int64_t Imm) const override;
   bool isTruncateFree(Type *SrcTy, Type *DstTy) const override;
   bool isTruncateFree(EVT SrcVT, EVT DstVT) const override;
   bool isZExtFree(SDValue Val, EVT VT2) const override;
-  bool isSExtCheaperThanZExt(EVT SrcVT, EVT DstVT) const override;
+  bool isSExtCheaperThanZExt(EVT SrcVT, EVT DstVT, SDValue V) const override;
   bool isCheapToSpeculateCttz() const override;
   bool isCheapToSpeculateCtlz() const override;
   bool hasAndNotCompare(SDValue Y) const override;
   bool shouldSinkOperands(Instruction *I,
                           SmallVectorImpl<Use *> &Ops) const override;
   bool isFPImmLegal(const APFloat &Imm, EVT VT,
                     bool ForCodeSize) const override;
 

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

if ((MemVT == MVT::i8 || MemVT == MVT::i16 ||
         (LD->getExtensionType() == ISD::NON_EXTLOAD ||
          LD->getExtensionType() == ISD::ZEXTLOAD))
       return true;
   }
 
   return TargetLowering::isZExtFree(Val, VT2);
 }
 
-bool RISCVTargetLowering::isSExtCheaperThanZExt(EVT SrcVT, EVT DstVT) const {
+bool RISCVTargetLowering::isSExtCheaperThanZExt(EVT SrcVT, EVT DstVT,
+                                                SDValue V) const {
   return Subtarget.is64Bit() && SrcVT == MVT::i32 && DstVT == MVT::i64;
 }
 
 bool RISCVTargetLowering::isCheapToSpeculateCttz() const {
   return Subtarget.hasStdExtZbb();
 }
 
 bool RISCVTargetLowering::isCheapToSpeculateCtlz() const {

llvm/test/CodeGen/AArch64/arm64-vshuffle.ll

 ; RUN: llc < %s -mtriple=arm64-apple-ios7.0 -mcpu=cyclone | FileCheck %s
 
 
 ; CHECK: test1
 ; CHECK: movi.16b v[[REG0:[0-9]+]], #0
 define <8 x i1> @test1() {
 entry:
   %Shuff = shufflevector <8 x i1> <i1 0, i1 1, i1 2, i1 3, i1 4, i1 5, i1 6,
                                    i1 7>,
                          <8 x i1> <i1 0, i1 1, i1 2, i1 3, i1 4, i1 5, i1 6,
                                    i1 7>,
                          <8 x i32> <i32 2, i32 undef, i32 6, i32 undef, i32 10,
                                     i32 12, i32 14, i32 0>
   ret <8 x i1> %Shuff
 }
 
-; CHECK: lCPI1_0:
-; CHECK:          .byte   0                       ; 0x0
-; CHECK:          .byte   0                       ; 0x0
-; CHECK:          .byte   0                       ; 0x0
-; CHECK:          .byte   0                       ; 0x0
-; CHECK:          .byte   1                       ; 0x1
-; CHECK:          .byte   0                       ; 0x0
-; CHECK:          .byte   0                       ; 0x0
-; CHECK:          .byte   0                       ; 0x0
 ; CHECK: test2
-; CHECK: adrp    x[[REG2:[0-9]+]], lCPI1_0@PAGE
-; CHECK: ldr     d[[REG1:[0-9]+]], [x[[REG2]], lCPI1_0@PAGEOFF]
+; CHECK: movi    d{{[0-9]+}}, #0x0000ff00000000
 define <8 x i1>@test2() {
 bb:
   %Shuff = shufflevector <8 x i1> zeroinitializer,
      <8 x i1> <i1 0, i1 1, i1 1, i1 0, i1 0, i1 1, i1 0, i1 0>,
      <8 x i32> <i32 2, i32 undef, i32 6, i32 undef, i32 10, i32 12, i32 14,
                 i32 0>
   ret <8 x i1> %Shuff
 }
 
 ; CHECK: test3
-; CHECK: movi.4s v{{[0-9]+}}, #1
+; CHECK: movi.2d v{{[0-9]+}}, #0x0000ff000000ff
 define <16 x i1> @test3(i1* %ptr, i32 %v) {
 bb:
   %Shuff = shufflevector <16 x i1> <i1 0, i1 1, i1 1, i1 0, i1 0, i1 1, i1 0, i1 0, i1 0, i1 1, i1 1, i1 0, i1 0, i1 1, i1 0, i1 0>, <16 x i1> undef,
      <16 x i32> <i32 2, i32 undef, i32 6, i32 undef, i32 10, i32 12, i32 14,
                  i32 0, i32 2, i32 undef, i32 6, i32 undef, i32 10, i32 12,
                  i32 14, i32 0>
   ret <16 x i1> %Shuff
 }
+
+
 ; CHECK: lCPI3_0:
 ; CHECK:         .byte   0                       ; 0x0
 ; CHECK:         .byte   0                       ; 0x0
 ; CHECK:         .byte   0                       ; 0x0
-; CHECK:         .byte   1                       ; 0x1
+; CHECK:         .byte   255                     ; 0xff
 ; CHECK:         .byte   0                       ; 0x0
 ; CHECK:         .byte   0                       ; 0x0
 ; CHECK:         .byte   0                       ; 0x0
 ; CHECK:         .byte   0                       ; 0x0
 ; CHECK:         .byte   0                       ; 0x0
 ; CHECK:         .byte   0                       ; 0x0
 ; CHECK:         .byte   0                       ; 0x0
 ; CHECK:         .byte   0                       ; 0x0

llvm/test/CodeGen/AArch64/arm64_32-atomics.ll

 ; CHECK-LABEL: test_cmpxchg_ptr:
 ; CHECK: [[LOOP:LBB[0-9]+_[0-9]+]]:
 ; CHECK:     ldaxr [[OLD:w[0-9]+]], [x0]
 ; CHECK:     cmp [[OLD]], w1
 ; CHECK:     b.ne [[DONE:LBB[0-9]+_[0-9]+]]
 ; CHECK:     stlxr [[SUCCESS:w[0-9]+]], w2, [x0]
 ; CHECK:     cbnz [[SUCCESS]], [[LOOP]]
 
-; CHECK:     mov w1, #1
+; CHECK:     mov w1, #-1
 ; CHECK:     mov w0, [[OLD]]
 ; CHECK:     ret
 
 ; CHECK: [[DONE]]:
 ; CHECK:     mov w1, wzr
 ; CHECK:     mov w0, [[OLD]]
 ; CHECK:     clrex
 ; CHECK:     ret
   %res = cmpxchg i8** %addr, i8* %cmp, i8* %new acq_rel acquire
   ret {i8*, i1} %res
 }

llvm/test/CodeGen/AArch64/cmpxchg-idioms.ll

 ; CHECK-NEXT:    ldaxr w8, [x0]
 ; CHECK-NEXT:    cmp w8, w1
 ; CHECK-NEXT:    b.ne LBB0_4
 ; CHECK-NEXT:  ; %bb.2: ; %cmpxchg.trystore
 ; CHECK-NEXT:    ; in Loop: Header=BB0_1 Depth=1
 ; CHECK-NEXT:    stlxr w8, w2, [x0]
 ; CHECK-NEXT:    cbnz w8, LBB0_1
 ; CHECK-NEXT:  ; %bb.3:
-; CHECK-NEXT:    mov w0, #1
+; CHECK-NEXT:    mov w0, #-1
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  LBB0_4: ; %cmpxchg.nostore
 ; CHECK-NEXT:    mov w0, wzr
 ; CHECK-NEXT:    clrex
 ; CHECK-NEXT:    ret
 ;
 ; OUTLINE-ATOMICS-LABEL: test_return:
 ; OUTLINE-ATOMICS:       ; %bb.0:

 ; CHECK-NEXT:    ldaxrb w9, [x0]
 ; CHECK-NEXT:    cmp w9, w8
 ; CHECK-NEXT:    b.ne LBB1_4
 ; CHECK-NEXT:  ; %bb.2: ; %cmpxchg.trystore
 ; CHECK-NEXT:    ; in Loop: Header=BB1_1 Depth=1
 ; CHECK-NEXT:    stlxrb w9, w2, [x0]
 ; CHECK-NEXT:    cbnz w9, LBB1_1
 ; CHECK-NEXT:  ; %bb.3:
-; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    mov w8, #-1
 ; CHECK-NEXT:    eor w0, w8, #0x1
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  LBB1_4: ; %cmpxchg.nostore
 ; CHECK-NEXT:    eor w0, wzr, #0x1
 ; CHECK-NEXT:    clrex
 ; CHECK-NEXT:    ret
 ;
 ; OUTLINE-ATOMICS-LABEL: test_return_bool:

 ; CHECK-NEXT:    ldaxr w8, [x19]
 ; CHECK-NEXT:    cmp w8, w21
 ; CHECK-NEXT:    b.ne LBB3_4
 ; CHECK-NEXT:  ; %bb.2: ; %cmpxchg.trystore
 ; CHECK-NEXT:    ; in Loop: Header=BB3_1 Depth=1
 ; CHECK-NEXT:    stlxr w8, w20, [x19]
 ; CHECK-NEXT:    cbnz w8, LBB3_1
 ; CHECK-NEXT:  ; %bb.3:
-; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    mov w8, #-1
 ; CHECK-NEXT:    b LBB3_5
 ; CHECK-NEXT:  LBB3_4: ; %cmpxchg.nostore
 ; CHECK-NEXT:    mov w8, wzr
 ; CHECK-NEXT:    clrex
 ; CHECK-NEXT:  LBB3_5: ; %for.cond.preheader
 ; CHECK-NEXT:    mov w22, #2
 ; CHECK-NEXT:  LBB3_6: ; %for.cond
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1

llvm/test/CodeGen/AArch64/dag-numsignbits.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=aarch64-unknown | FileCheck %s
 
 ; PR32273
 
 define void @signbits_vXi1(<4 x i16> %a1) {
 ; CHECK-LABEL: signbits_vXi1:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    adrp x8, .LCPI0_0
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    mov w1, wzr
+; CHECK-NEXT:    movi v2.4h, #1
 ; CHECK-NEXT:    dup v0.4h, v0.h[0]
+; CHECK-NEXT:    mov w1, wzr
 ; CHECK-NEXT:    mov w2, wzr
 ; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI0_0]
-; CHECK-NEXT:    adrp x8, .LCPI0_1
 ; CHECK-NEXT:    add v0.4h, v0.4h, v1.4h
-; CHECK-NEXT:    movi v1.4h, #1
-; CHECK-NEXT:    cmgt v0.4h, v1.4h, v0.4h
-; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI0_1]
-; CHECK-NEXT:    and v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    shl v0.4h, v0.4h, #15
-; CHECK-NEXT:    cmlt v0.4h, v0.4h, #0
+; CHECK-NEXT:    cmgt v0.4h, v2.4h, v0.4h
 ; CHECK-NEXT:    umov w0, v0.h[0]
 ; CHECK-NEXT:    umov w3, v0.h[3]
 ; CHECK-NEXT:    b foo
   %tmp3 = shufflevector <4 x i16> %a1, <4 x i16> undef, <4 x i32> zeroinitializer
   %tmp5 = add <4 x i16> %tmp3, <i16 18249, i16 6701, i16 -18744, i16 -25086>
   %tmp6 = icmp slt <4 x i16> %tmp5, <i16 1, i16 1, i16 1, i16 1>
   %tmp7 = and <4 x i1> %tmp6, <i1 true, i1 false, i1 false, i1 true>
   %tmp8 = sext <4 x i1> %tmp7 to <4 x i16>

llvm/test/CodeGen/AArch64/fast-isel-cmp-vec.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-apple-darwin -fast-isel -verify-machineinstrs \
 ; RUN:   -aarch64-enable-atomic-cfg-tidy=0 -disable-cgp -disable-branch-fold \
 ; RUN:   < %s | FileCheck %s
 
 ;
 ; Verify that we don't mess up vector comparisons in fast-isel.
 ;
 
 define <2 x i32> @icmp_v2i32(<2 x i32> %a) {
 ; CHECK-LABEL: icmp_v2i32:
-; CHECK:      ; %bb.0:
-; CHECK-NEXT:  cmeq.2s [[CMP:v[0-9]+]], v0, #0
-; CHECK-NEXT: ; %bb.1:
-; CHECK-NEXT:  movi.2s [[MASK:v[0-9]+]], #1
-; CHECK-NEXT:  and.8b v0, [[CMP]], [[MASK]]
-; CHECK-NEXT:  ret
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    cmeq.2s v0, v0, #0
+; CHECK-NEXT:  ; %bb.1: ; %bb2
+; CHECK-NEXT:    movi.2s v1, #1
+; CHECK-NEXT:    and.8b v0, v0, v1
+; CHECK-NEXT:    ret
   %c = icmp eq <2 x i32> %a, zeroinitializer
   br label %bb2
 bb2:
   %z = zext <2 x i1> %c to <2 x i32>
   ret <2 x i32> %z
 }
 
 define <2 x i32> @icmp_constfold_v2i32(<2 x i32> %a) {
 ; CHECK-LABEL: icmp_constfold_v2i32:
-; CHECK:      ; %bb.0:
-; CHECK-NEXT:  movi.2s [[MASK:v[0-9]+]], #1
-; CHECK-NEXT:  and.8b v0, [[MASK]], [[MASK]]
-; CHECK-NEXT:  ret
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    movi.2d v0, #0xffffffffffffffff
+; CHECK-NEXT:  ; %bb.1: ; %bb2
+; CHECK-NEXT:    movi.2s v1, #1
+; CHECK-NEXT:    and.8b v0, v0, v1
+; CHECK-NEXT:    ret
   %1 = icmp eq <2 x i32> %a, %a
   br label %bb2
 bb2:
   %2 = zext <2 x i1> %1 to <2 x i32>
   ret <2 x i32> %2
 }
 
 define <4 x i32> @icmp_v4i32(<4 x i32> %a) {
 ; CHECK-LABEL: icmp_v4i32:
-; CHECK:      ; %bb.0:
-; CHECK-NEXT:  cmeq.4s [[CMP:v[0-9]+]], v0, #0
-; CHECK-NEXT:  xtn.4h [[CMPV4I16:v[0-9]+]], [[CMP]]
-; CHECK-NEXT: ; %bb.1:
-; CHECK-NEXT:  movi.4h [[MASK:v[0-9]+]], #1
-; CHECK-NEXT:  and.8b [[ZEXT:v[0-9]+]], [[CMPV4I16]], [[MASK]]
-; CHECK-NEXT:  ushll.4s v0, [[ZEXT]], #0
-; CHECK-NEXT:  ret
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    cmeq.4s v0, v0, #0
+; CHECK-NEXT:    xtn.4h v0, v0
+; CHECK-NEXT:  ; %bb.1: ; %bb2
+; CHECK-NEXT:    movi.4h v1, #1
+; CHECK-NEXT:    and.8b v0, v0, v1
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    ret
   %c = icmp eq <4 x i32> %a, zeroinitializer
   br label %bb2
 bb2:
   %z = zext <4 x i1> %c to <4 x i32>
   ret <4 x i32> %z
 }
 
 define <4 x i32> @icmp_constfold_v4i32(<4 x i32> %a) {
 ; CHECK-LABEL: icmp_constfold_v4i32:
-; CHECK:      ; %bb.0:
-; CHECK-NEXT:  movi.4h [[MASK:v[0-9]+]], #1
-; CHECK-NEXT: ; %bb.1:
-; CHECK-NEXT:  and.8b [[ZEXT:v[0-9]+]], [[MASK]], [[MASK]]
-; CHECK-NEXT:  ushll.4s v0, [[ZEXT]], #0
-; CHECK-NEXT:  ret
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    movi.2d v0, #0xffffffffffffffff
+; CHECK-NEXT:  ; %bb.1: ; %bb2
+; CHECK-NEXT:    movi.4h v1, #1
+; CHECK-NEXT:    and.8b v0, v0, v1
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    ret
   %1 = icmp eq <4 x i32> %a, %a
   br label %bb2
 bb2:
   %2 = zext <4 x i1> %1 to <4 x i32>
   ret <4 x i32> %2
 }
 
 define <16 x i8> @icmp_v16i8(<16 x i8> %a) {
 ; CHECK-LABEL: icmp_v16i8:
-; CHECK:      ; %bb.0:
-; CHECK-NEXT:  cmeq.16b [[CMP:v[0-9]+]], v0, #0
-; CHECK-NEXT: ; %bb.1:
-; CHECK-NEXT:  movi.16b [[MASK:v[0-9]+]], #1
-; CHECK-NEXT:  and.16b v0, [[CMP]], [[MASK]]
-; CHECK-NEXT:  ret
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    cmeq.16b v0, v0, #0
+; CHECK-NEXT:  ; %bb.1: ; %bb2
+; CHECK-NEXT:    movi.16b v1, #1
+; CHECK-NEXT:    and.16b v0, v0, v1
+; CHECK-NEXT:    ret
   %c = icmp eq <16 x i8> %a, zeroinitializer
   br label %bb2
 bb2:
   %z = zext <16 x i1> %c to <16 x i8>
   ret <16 x i8> %z
 }
 
 define <16 x i8> @icmp_constfold_v16i8(<16 x i8> %a) {
 ; CHECK-LABEL: icmp_constfold_v16i8:
-; CHECK:      ; %bb.0:
-; CHECK-NEXT:  movi.16b [[MASK:v[0-9]+]], #1
-; CHECK-NEXT:  and.16b v0, [[MASK]], [[MASK]]
-; CHECK-NEXT:  ret
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    movi.2d v0, #0xffffffffffffffff
+; CHECK-NEXT:  ; %bb.1: ; %bb2
+; CHECK-NEXT:    movi.16b v1, #1
+; CHECK-NEXT:    and.16b v0, v0, v1
+; CHECK-NEXT:    ret
   %1 = icmp eq <16 x i8> %a, %a
   br label %bb2
 bb2:
   %2 = zext <16 x i1> %1 to <16 x i8>
   ret <16 x i8> %2
 }

llvm/test/CodeGen/AArch64/funnel-shift.ll

 }
 
 ; extract(concat(0b1110000, 0b1111111) << 2) = 0b1000011
 
 declare i7 @llvm.fshl.i7(i7, i7, i7)
 define i7 @fshl_i7_const_fold() {
 ; CHECK-LABEL: fshl_i7_const_fold:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w0, #67
+; CHECK-NEXT:    mov w0, #-61
 ; CHECK-NEXT:    ret
   %f = call i7 @llvm.fshl.i7(i7 112, i7 127, i7 2)
   ret i7 %f
 }
 
 define i8 @fshl_i8_const_fold_overshift_1() {
 ; CHECK-LABEL: fshl_i8_const_fold_overshift_1:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w0, #128
+; CHECK-NEXT:    mov w0, #-128
 ; CHECK-NEXT:    ret
   %f = call i8 @llvm.fshl.i8(i8 255, i8 0, i8 15)
   ret i8 %f
 }
 
 define i8 @fshl_i8_const_fold_overshift_2() {
 ; CHECK-LABEL: fshl_i8_const_fold_overshift_2:
 ; CHECK:       // %bb.0:
; CHECK-NEXT:    ret
   ret i64 %f
 }
 
 ; This should work without any node-specific logic.
 
 define i8 @fshl_i8_const_fold() {
 ; CHECK-LABEL: fshl_i8_const_fold:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w0, #128
+; CHECK-NEXT:    mov w0, #-128
 ; CHECK-NEXT:    ret
   %f = call i8 @llvm.fshl.i8(i8 255, i8 0, i8 7)
   ret i8 %f
 }
 
 ; Repeat everything for funnel shift right.
 
 ; General case - all operands can be variables.

 ; CHECK-NEXT:    ret
   %f = call i7 @llvm.fshr.i7(i7 112, i7 127, i7 2)
   ret i7 %f
 }
 
 define i8 @fshr_i8_const_fold_overshift_1() {
 ; CHECK-LABEL: fshr_i8_const_fold_overshift_1:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w0, #254
+; CHECK-NEXT:    mov w0, #-2
 ; CHECK-NEXT:    ret
   %f = call i8 @llvm.fshr.i8(i8 255, i8 0, i8 15)
   ret i8 %f
 }
 
 define i8 @fshr_i8_const_fold_overshift_2() {
 ; CHECK-LABEL: fshr_i8_const_fold_overshift_2:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w0, #225
+; CHECK-NEXT:    mov w0, #-31
 ; CHECK-NEXT:    ret
   %f = call i8 @llvm.fshr.i8(i8 15, i8 15, i8 11)
   ret i8 %f
 }
 
 define i8 @fshr_i8_const_fold_overshift_3() {
 ; CHECK-LABEL: fshr_i8_const_fold_overshift_3:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w0, #255
+; CHECK-NEXT:    mov w0, #-1
 ; CHECK-NEXT:    ret
   %f = call i8 @llvm.fshr.i8(i8 0, i8 255, i8 8)
   ret i8 %f
 }
 
 ; With constant shift amount, this is 'extr'.
 
 define i32 @fshr_i32_const_shift(i32 %x, i32 %y) {
; CHECK-NEXT:    ret
   ret i64 %f
 }
 
 ; This should work without any node-specific logic.
 
 define i8 @fshr_i8_const_fold() {
 ; CHECK-LABEL: fshr_i8_const_fold:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w0, #254
+; CHECK-NEXT:    mov w0, #-2
 ; CHECK-NEXT:    ret
   %f = call i8 @llvm.fshr.i8(i8 255, i8 0, i8 7)
   ret i8 %f
 }
 
 define i32 @fshl_i32_shift_by_bitwidth(i32 %x, i32 %y) {
 ; CHECK-LABEL: fshl_i32_shift_by_bitwidth:
 ; CHECK:       // %bb.0:

llvm/test/CodeGen/AArch64/reduce-and.ll

; GISEL-NEXT:    ret
   %and_result = call i8 @llvm.vector.reduce.and.v1i8(<1 x i8> %a)
   ret i8 %and_result
 }
 
 define i8 @test_redand_v3i8(<3 x i8> %a) {
 ; CHECK-LABEL: test_redand_v3i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    and w8, w0, w1
-; CHECK-NEXT:    and w8, w8, w2
-; CHECK-NEXT:    and w0, w8, #0xff
+; CHECK-NEXT:    and w0, w8, w2
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test_redand_v3i8:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    and w8, w0, w1
 ; GISEL-NEXT:    and w0, w8, w2
 ; GISEL-NEXT:    ret
   %and_result = call i8 @llvm.vector.reduce.and.v3i8(<3 x i8> %a)

llvm/test/CodeGen/AArch64/redundant-copy-elim-empty-mbb.ll

 ; RUN: llc < %s | FileCheck %s
 ; Make sure we don't crash in AArch64RedundantCopyElimination when a
 ; MachineBasicBlock is empty.  PR29035.
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64-unknown-linux-gnu"
 
 declare i8* @bar()
 
 ; CHECK-LABEL: foo:
 ; CHECK: tbz
-; CHECK: mov{{.*}}, #1
+; CHECK: mov{{.*}}, #-1
 ; CHECK: ret
 ; CHECK: bl bar
 ; CHECK: cbnz
 ; CHECK: ret
 define i1 @foo(i1 %start) {
 entry:
   br i1 %start, label %cleanup, label %if.end
 

llvm/test/CodeGen/AArch64/statepoint-call-lowering.ll

 ; CHECK-NEXT:  .Ltmp8:
 ; CHECK-NEXT:    tbz w20, #0, .LBB8_2
 ; CHECK-NEXT:  // %bb.1: // %left
 ; CHECK-NEXT:    mov w19, w0
 ; CHECK-NEXT:    ldr x0, [sp, #8]
 ; CHECK-NEXT:    bl consume
 ; CHECK-NEXT:    b .LBB8_3
 ; CHECK-NEXT:  .LBB8_2:
-; CHECK-NEXT:    mov w19, #1
+; CHECK-NEXT:    mov w19, #-1
 ; CHECK-NEXT:  .LBB8_3: // %common.ret
 ; CHECK-NEXT:    and w0, w19, #0x1
 ; CHECK-NEXT:    ldp x20, x19, [sp, #16] // 16-byte Folded Reload
 ; CHECK-NEXT:    ldr x30, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
 entry:
   %safepoint_token = tail call token (i64, i32, i1 ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_i1f(i64 0, i32 0, i1 ()* @return_i1, i32 0, i32 0, i32 0, i32 0) ["gc-live" (i32 addrspace(1)* %a)]
   br i1 %external_cond, label %left, label %right

llvm/test/CodeGen/AArch64/sve-vector-splat.ll

 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov z0.h, w0
 ; CHECK-NEXT:    ret
   %ins = insertelement <vscale x 8 x i8> undef, i8 %val, i32 0
   %splat = shufflevector <vscale x 8 x i8> %ins, <vscale x 8 x i8> undef, <vscale x 8 x i32> zeroinitializer
   ret <vscale x 8 x i8> %splat
 }
 
 define <vscale x 8 x i8> @sve_splat_8xi8_imm() {
 ; CHECK-LABEL: sve_splat_8xi8_imm:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #255
-; CHECK-NEXT:    mov z0.h, w8
+; CHECK-NEXT:    mov z0.h, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    ret
   %ins = insertelement <vscale x 8 x i8> undef, i8 -1, i32 0
   %splat = shufflevector <vscale x 8 x i8> %ins, <vscale x 8 x i8> undef, <vscale x 8 x i32> zeroinitializer
   ret <vscale x 8 x i8> %splat
 }
 
 define <vscale x 2 x i16> @sve_splat_2xi16(i16 %val) {
 ; CHECK-LABEL: sve_splat_2xi16:
; CHECK-NEXT:    ret
   %ins = insertelement <vscale x 4 x i16> undef, i16 %val, i32 0
   %splat = shufflevector <vscale x 4 x i16> %ins, <vscale x 4 x i16> undef, <vscale x 4 x i32> zeroinitializer
   ret <vscale x 4 x i16> %splat
 }
 
 define <vscale x 4 x i16> @sve_splat_4xi16_imm() {
 ; CHECK-LABEL: sve_splat_4xi16_imm:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #65535
-; CHECK-NEXT:    mov z0.s, w8
+; CHECK-NEXT:    mov z0.s, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    ret
   %ins = insertelement <vscale x 4 x i16> undef, i16 -1, i32 0
   %splat = shufflevector <vscale x 4 x i16> %ins, <vscale x 4 x i16> undef, <vscale x 4 x i32> zeroinitializer
   ret <vscale x 4 x i16> %splat
 }
 
 define <vscale x 2 x i32> @sve_splat_2xi32(i32 %val) {
 ; CHECK-LABEL: sve_splat_2xi32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
 ; CHECK-NEXT:    mov z0.d, x0
 ; CHECK-NEXT:    ret
   %ins = insertelement <vscale x 2 x i32> undef, i32 %val, i32 0
   %splat = shufflevector <vscale x 2 x i32> %ins, <vscale x 2 x i32> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x i32> %splat
 }
 
 define <vscale x 2 x i32> @sve_splat_2xi32_imm() {
 ; CHECK-LABEL: sve_splat_2xi32_imm:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #-1
-; CHECK-NEXT:    mov z0.d, x8
+; CHECK-NEXT:    mov z0.d, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    ret
   %ins = insertelement <vscale x 2 x i32> undef, i32 -1, i32 0
   %splat = shufflevector <vscale x 2 x i32> %ins, <vscale x 2 x i32> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x i32> %splat
 }
 
 ;; Widen/split splats of wide vector types.
 

llvm/test/CodeGen/AArch64/unfold-masked-merge-vector-variablemask.ll

 
 ; ============================================================================ ;
 ; 16-bit vector width
 ; ============================================================================ ;
 
 define <2 x i8> @out_v2i8(<2 x i8> %x, <2 x i8> %y, <2 x i8> %mask) nounwind {
 ; CHECK-LABEL: out_v2i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    movi d3, #0x0000ff000000ff
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v2.8b, v2.8b, v3.8b
-; CHECK-NEXT:    and v1.8b, v1.8b, v2.8b
-; CHECK-NEXT:    orr v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %mx = and <2 x i8> %x, %mask
   %notmask = xor <2 x i8> %mask, <i8 -1, i8 -1>
   %my = and <2 x i8> %y, %notmask
   %r = or <2 x i8> %mx, %my
   ret <2 x i8> %r
 }
 

 
 ; ============================================================================ ;
 ; 32-bit vector width
 ; ============================================================================ ;
 
 define <4 x i8> @out_v4i8(<4 x i8> %x, <4 x i8> %y, <4 x i8> %mask) nounwind {
 ; CHECK-LABEL: out_v4i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    movi d3, #0xff00ff00ff00ff
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v2.8b, v2.8b, v3.8b
-; CHECK-NEXT:    and v1.8b, v1.8b, v2.8b
-; CHECK-NEXT:    orr v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %mx = and <4 x i8> %x, %mask
   %notmask = xor <4 x i8> %mask, <i8 -1, i8 -1, i8 -1, i8 -1>
   %my = and <4 x i8> %y, %notmask
   %r = or <4 x i8> %mx, %my
   ret <4 x i8> %r
 }
 
 define <4 x i8> @out_v4i8_undef(<4 x i8> %x, <4 x i8> %y, <4 x i8> %mask) nounwind {
 ; CHECK-LABEL: out_v4i8_undef:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    movi d3, #0xff00ff00ff00ff
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v2.8b, v2.8b, v3.8b
-; CHECK-NEXT:    and v1.8b, v1.8b, v2.8b
-; CHECK-NEXT:    orr v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %mx = and <4 x i8> %x, %mask
   %notmask = xor <4 x i8> %mask, <i8 -1, i8 -1, i8 undef, i8 -1>
   %my = and <4 x i8> %y, %notmask
   %r = or <4 x i8> %mx, %my
   ret <4 x i8> %r
 }
 
 define <2 x i16> @out_v2i16(<2 x i16> %x, <2 x i16> %y, <2 x i16> %mask) nounwind {
 ; CHECK-LABEL: out_v2i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    movi d3, #0x00ffff0000ffff
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v2.8b, v2.8b, v3.8b
-; CHECK-NEXT:    and v1.8b, v1.8b, v2.8b
-; CHECK-NEXT:    orr v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
 ; CHECK-NEXT:    ret
   %mx = and <2 x i16> %x, %mask
   %notmask = xor <2 x i16> %mask, <i16 -1, i16 -1>
   %my = and <2 x i16> %y, %notmask
   %r = or <2 x i16> %mx, %my
   ret <2 x i16> %r
 }
 

llvm/test/CodeGen/AArch64/vecreduce-and-legalization.ll

; CHECK-NEXT:    ret
   %b = call i128 @llvm.vector.reduce.and.v1i128(<1 x i128> %a)
   ret i128 %b
 }
 
 define i8 @test_v3i8(<3 x i8> %a) nounwind {
 ; CHECK-LABEL: test_v3i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    and w8, w0, w1
-; CHECK-NEXT:    and w8, w8, w2
-; CHECK-NEXT:    and w0, w8, #0xff
+; CHECK-NEXT:    and w0, w8, w2
 ; CHECK-NEXT:    ret
   %b = call i8 @llvm.vector.reduce.and.v3i8(<3 x i8> %a)
   ret i8 %b
 }
 
 define i8 @test_v9i8(<9 x i8> %a) nounwind {
 ; CHECK-LABEL: test_v9i8:
 ; CHECK:       // %bb.0: