[AArch64] Peephole rule to remove redundant cmp after cset.

[ilinpv]

Summary

Comparisons to zero or one after cset instructions can be safely
removed in examples like:

cset w9, eq          cset w9, eq
cmp  w9, #1   --->   <removed>
b.ne    .L1          b.ne    .L1

cset w9, eq          cset w9, eq
cmp  w9, #0   --->   <removed>
b.ne    .L1          b.eq    .L1

Peephole optimization to detect suitable cases and get rid of that
comparisons added.

[dmgreen]

I see that this is an extension to the existing peephole optimizations, but is there any reason not to do this during ISel?

[ilinpv]

In D98564#2624774, @dmgreen wrote:

I see that this is an extension to the existing peephole optimizations, but is there any reason not to do this during ISel?

This optimization relies on liveness analysis to check that condition flags are not used in successors (see areCFlagsAliveInSuccessors) and I guess it would not fit ISel.

[paquette]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1536

• The name here is a little misleading. This is intended only for Bcc + select instructions. Maybe findCondCodeUseOperandIdxForBranchOrSelect?

• This could use a Doxygen comment describing what it does.

1642

This could use a doxygen comment documenting what the function is for.

It'd also be nice to document the parameters in the Doxygen comment as well.

1642

If you return an Optional here you can still treat it like a bool, but also avoid passing UsedNZCV as an out-parameter.

1643

I don't think these should ever be nullptr?

1645

I think I would prefer an Optional<ArrayRef<MachineInstr *>> here if possible. I think that if someone were to use this and accidentally pass nullptr, it would be a mistake. Using an Optional better communicates that this is an optional parameter.

1655

Pull CmpInstr->getParent() out into its own variable?

1669

I don't think this actually does anything? It's writing to a local variable.

1868

I think it'd be good to break this sentence down a little.

\returns True if \p CmpInstr can be removed.

\p IsInvertCC is true if, after removing \p CmpInstr, uses of the condition code must be inverted.

1871

I don't think it's necessary to enumerate every case in this comment. It may fall out of sync if someone updates the code.

1889

These should never be nullptr according to the asserts, so they might as well be passed by reference.

1891

Use ArrayRef?

1919

Pull isSUBSRegImm out into a variable and use it in both of the if statements?

1973

Since MRI should never be nullptr, I think it makes sense for it to be a reference.

1990

This assert could use a string explaining it.

1993

Nit: static_cast is easier to grep for

llvm/test/CodeGen/AArch64/csinc-cmp-removal.mir

1

Nit: You can use update_mir_test_checks.py to generate the check lines. This makes it easier to update the test if necessary.

[dmgreen]

I imagine that the cset; cmp from the fp16 tests is just a missing optimization in ISel, and there might be good reason to do so there if it helps simplify the graph.

But like you said, this does fit in with the other peepholing code. And if it helps both ISels, that is good.

[ilinpv]

Work on comments.

[ilinpv]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1645

I am not sure about ArrayRef and const in examineCFlagsUse. The number of CCUseInstrs is defined in the analysis, instructions from CCUseInstrs can be later modified inverting condition code, CmpInstr can be removed.

1669

Goog catch! Lost * after code refactoring. Test for that added.

1871

I was inspired by canInstrSubstituteCmpInstr doxygen comments. I guess it should be clear what function is doing. Does it make sense to leave general doxygen comment for canCmpInstrBeRemoved, but put detailed ordinary comments inside function plus doxygen comments for examineCFlagsUse?

[paquette]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1536

Minor nit: Doxygenification

1594

We may want to assert that this is an immediate.

1871

I think that would be good.

[ilinpv]

Doxygen & comments update.

[ilinpv]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1594

getImm() has an assert inside.

[ilinpv]

Just a friendly ping. Jessica @paquette, do you have any further suggestions?

[dmgreen]

I think I managed to convince myself that this is correct. But there is a lot that could go wrong and subtly forgotten, this code has been a bit error prone in the past. Hopefully now that it's gained some infrastructure that's less likely.

LGTM, but please wait a couple of days in case @paquette has more comments.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1752

Add a message for the assert?

1855

I would leave brackets on statements like this, if the inner part is fairly big.

[nikic]

I'm seeing a miscompile due to this change, see https://bugs.llvm.org/show_bug.cgi?id=51476. I suspect this is because analyzeCompare() only distinguishes comparisons against zero (CmpValue==0) and non-zero (CmpValue==1), but not precisely which non-zero value is used, while this code assumes that CmpValue==1 is a comparison against exactly 1 and not some other non-zero value.

llvm/lib/Target/AArch64/AArch64InstrInfo.h

private:
   ///
   /// \param MBB A \p MachineBasicBlock in an outlined function.
   void fixupPostOutline(MachineBasicBlock &MBB) const;
 
   void instantiateCondBranch(MachineBasicBlock &MBB, const DebugLoc &DL,
                              MachineBasicBlock *TBB,
                              ArrayRef<MachineOperand> Cond) const;
   bool substituteCmpToZero(MachineInstr &CmpInstr, unsigned SrcReg,
-                           const MachineRegisterInfo *MRI) const;
+                           const MachineRegisterInfo &MRI) const;
+  bool removeCmpToZeroOrOne(MachineInstr &CmpInstr, unsigned SrcReg,
+                            int CmpValue, const MachineRegisterInfo &MRI) const;
 
   /// Returns an unused general-purpose register which can be used for
   /// constructing an outlined call if one exists. Returns 0 otherwise.
   unsigned findRegisterToSaveLRTo(const outliner::Candidate &C) const;
 
   /// Remove a ptest of a predicate-generating operation that already sets, or
   /// can be made to set, the condition codes in an identical manner
   bool optimizePTestInstr(MachineInstr *PTest, unsigned MaskReg,

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

bool AArch64InstrInfo::optimizeCompareInstr(
 
   if (CmpInstr.getOpcode() == AArch64::PTEST_PP)
     return optimizePTestInstr(&CmpInstr, SrcReg, SrcReg2, MRI);
 
   // Continue only if we have a "ri" where immediate is zero.
   // FIXME:CmpValue has already been converted to 0 or 1 in analyzeCompare
   // function.
   assert((CmpValue == 0 || CmpValue == 1) && "CmpValue must be 0 or 1!");
-  if (CmpValue != 0 || SrcReg2 != 0)
+  if (SrcReg2 != 0)
     return false;
 
   // CmpInstr is a Compare instruction if destination register is not used.
   if (!MRI->use_nodbg_empty(CmpInstr.getOperand(0).getReg()))
     return false;
 
-  return substituteCmpToZero(CmpInstr, SrcReg, MRI);
+  if (!CmpValue && substituteCmpToZero(CmpInstr, SrcReg, *MRI))
+    return true;
+  return removeCmpToZeroOrOne(CmpInstr, SrcReg, CmpValue, *MRI);
 }
 
 /// Get opcode of S version of Instr.
 /// If Instr is S version its opcode is returned.
 /// AArch64::INSTRUCTION_LIST_END is returned if Instr does not have S version
 /// or we are not interested in it.
 static unsigned sForm(MachineInstr &Instr) {
   switch (Instr.getOpcode()) {
static unsigned sForm(MachineInstr &Instr) {
   case AArch64::ANDWri:
     return AArch64::ANDSWri;
   case AArch64::ANDXri:
     return AArch64::ANDSXri;
   }
 }
 
 /// Check if AArch64::NZCV should be alive in successors of MBB.
-static bool areCFlagsAliveInSuccessors(MachineBasicBlock *MBB) {
+static bool areCFlagsAliveInSuccessors(const MachineBasicBlock *MBB) {
   for (auto *BB : MBB->successors())
     if (BB->isLiveIn(AArch64::NZCV))
       return true;
   return false;
 }
 
+/// \returns The condition code operand index for \p Instr if it is a branch
+/// or select and -1 otherwise.
+static int
+findCondCodeUseOperandIdxForBranchOrSelect(const MachineInstr &Instr) {
+  switch (Instr.getOpcode()) {
+  default:
+    return -1;
+
+  case AArch64::Bcc: {
+    int Idx = Instr.findRegisterUseOperandIdx(AArch64::NZCV);
+    assert(Idx >= 2);
+    return Idx - 2;
+  }
+
+  case AArch64::CSINVWr:
+  case AArch64::CSINVXr:
+  case AArch64::CSINCWr:
+  case AArch64::CSINCXr:
+  case AArch64::CSELWr:
+  case AArch64::CSELXr:
+  case AArch64::CSNEGWr:
+  case AArch64::CSNEGXr:
+  case AArch64::FCSELSrrr:
+  case AArch64::FCSELDrrr: {
+    int Idx = Instr.findRegisterUseOperandIdx(AArch64::NZCV);
+    assert(Idx >= 1);
+    return Idx - 1;
+  }
+  }
+}
+
 namespace {
 
 struct UsedNZCV {
   bool N = false;
   bool Z = false;
   bool C = false;
   bool V = false;
 

 };
 
 } // end anonymous namespace
 
 /// Find a condition code used by the instruction.
 /// Returns AArch64CC::Invalid if either the instruction does not use condition
 /// codes or we don't optimize CmpInstr in the presence of such instructions.
 static AArch64CC::CondCode findCondCodeUsedByInstr(const MachineInstr &Instr) {
-  switch (Instr.getOpcode()) {
-  default:
-    return AArch64CC::Invalid;
-
-  case AArch64::Bcc: {
-    int Idx = Instr.findRegisterUseOperandIdx(AArch64::NZCV);
-    assert(Idx >= 2);
-    return static_cast<AArch64CC::CondCode>(Instr.getOperand(Idx - 2).getImm());
-  }
-
-  case AArch64::CSINVWr:
-  case AArch64::CSINVXr:
-  case AArch64::CSINCWr:
-  case AArch64::CSINCXr:
-  case AArch64::CSELWr:
-  case AArch64::CSELXr:
-  case AArch64::CSNEGWr:
-  case AArch64::CSNEGXr:
-  case AArch64::FCSELSrrr:
-  case AArch64::FCSELDrrr: {
-    int Idx = Instr.findRegisterUseOperandIdx(AArch64::NZCV);
-    assert(Idx >= 1);
-    return static_cast<AArch64CC::CondCode>(Instr.getOperand(Idx - 1).getImm());
-  }
-  }
+  int CCIdx = findCondCodeUseOperandIdxForBranchOrSelect(Instr);
+  return CCIdx >= 0 ? static_cast<AArch64CC::CondCode>(
+                          Instr.getOperand(CCIdx).getImm())
+                    : AArch64CC::Invalid;
 }
 
 static UsedNZCV getUsedNZCV(AArch64CC::CondCode CC) {
   assert(CC != AArch64CC::Invalid);
   UsedNZCV UsedFlags;
   switch (CC) {
   default:
     break;
static UsedNZCV getUsedNZCV(AArch64CC::CondCode CC) {
   case AArch64CC::LT: // N and V differ
     UsedFlags.N = true;
     UsedFlags.V = true;
     break;
   }
   return UsedFlags;
 }
 
+/// \returns Conditions flags used after \p CmpInstr in its MachineBB if they
+/// are not containing C or V flags and NZCV flags are not alive in successors
+/// of the same \p CmpInstr and \p MI parent. \returns None otherwise.
+///
+/// Collect instructions using that flags in \p CCUseInstrs if provided.
+static Optional<UsedNZCV>
+examineCFlagsUse(MachineInstr &MI, MachineInstr &CmpInstr,
+                 const TargetRegisterInfo &TRI,
+                 SmallVectorImpl<MachineInstr *> *CCUseInstrs = nullptr) {
+  MachineBasicBlock *CmpParent = CmpInstr.getParent();
+  if (MI.getParent() != CmpParent)
+    return None;
+
+  if (areCFlagsAliveInSuccessors(CmpParent))
+    return None;
+
+  UsedNZCV NZCVUsedAfterCmp;
+  for (MachineInstr &Instr : instructionsWithoutDebug(
+           std::next(CmpInstr.getIterator()), CmpParent->instr_end())) {
+    if (Instr.readsRegister(AArch64::NZCV, &TRI)) {
+      AArch64CC::CondCode CC = findCondCodeUsedByInstr(Instr);
+      if (CC == AArch64CC::Invalid) // Unsupported conditional instruction
+        return None;
+      NZCVUsedAfterCmp |= getUsedNZCV(CC);
+      if (CCUseInstrs)
+        CCUseInstrs->push_back(&Instr);
+    }
+    if (Instr.modifiesRegister(AArch64::NZCV, &TRI))
+      break;
+  }
+  if (NZCVUsedAfterCmp.C || NZCVUsedAfterCmp.V)
+    return None;
+  return NZCVUsedAfterCmp;
+}
+
 static bool isADDSRegImm(unsigned Opcode) {
   return Opcode == AArch64::ADDSWri || Opcode == AArch64::ADDSXri;
 }
 
 static bool isSUBSRegImm(unsigned Opcode) {
   return Opcode == AArch64::SUBSWri || Opcode == AArch64::SUBSXri;
 }
 
 /// Check if CmpInstr can be substituted by MI.
 ///
 /// CmpInstr can be substituted:
 /// - CmpInstr is either 'ADDS %vreg, 0' or 'SUBS %vreg, 0'
 /// - and, MI and CmpInstr are from the same MachineBB
 /// - and, condition flags are not alive in successors of the CmpInstr parent
 /// - and, if MI opcode is the S form there must be no defs of flags between
 ///        MI and CmpInstr
 ///        or if MI opcode is not the S form there must be neither defs of flags
 ///        nor uses of flags between MI and CmpInstr.
 /// - and  C/V flags are not used after CmpInstr
-static bool canInstrSubstituteCmpInstr(MachineInstr *MI, MachineInstr *CmpInstr,
-                                       const TargetRegisterInfo *TRI) {
-  assert(MI);
-  assert(sForm(*MI) != AArch64::INSTRUCTION_LIST_END);
-  assert(CmpInstr);
+static bool canInstrSubstituteCmpInstr(MachineInstr &MI, MachineInstr &CmpInstr,
+                                       const TargetRegisterInfo &TRI) {
+  assert(sForm(MI) != AArch64::INSTRUCTION_LIST_END);
 
-  const unsigned CmpOpcode = CmpInstr->getOpcode();
+  const unsigned CmpOpcode = CmpInstr.getOpcode();
   if (!isADDSRegImm(CmpOpcode) && !isSUBSRegImm(CmpOpcode))
     return false;
 
-  if (MI->getParent() != CmpInstr->getParent())
-    return false;
-
-  if (areCFlagsAliveInSuccessors(CmpInstr->getParent()))
+  if (!examineCFlagsUse(MI, CmpInstr, TRI))
     return false;
 
   AccessKind AccessToCheck = AK_Write;
-  if (sForm(*MI) != MI->getOpcode())
+  if (sForm(MI) != MI.getOpcode())
     AccessToCheck = AK_All;
-  if (areCFlagsAccessedBetweenInstrs(MI, CmpInstr, TRI, AccessToCheck))
-    return false;
-
-  UsedNZCV NZCVUsedAfterCmp;
-  for (const MachineInstr &Instr :
-       instructionsWithoutDebug(std::next(CmpInstr->getIterator()),
-                                CmpInstr->getParent()->instr_end())) {
-    if (Instr.readsRegister(AArch64::NZCV, TRI)) {
-      AArch64CC::CondCode CC = findCondCodeUsedByInstr(Instr);
-      if (CC == AArch64CC::Invalid) // Unsupported conditional instruction
-        return false;
-      NZCVUsedAfterCmp |= getUsedNZCV(CC);
-    }
-
-    if (Instr.modifiesRegister(AArch64::NZCV, TRI))
-      break;
-  }
-
-  return !NZCVUsedAfterCmp.C && !NZCVUsedAfterCmp.V;
+  return !areCFlagsAccessedBetweenInstrs(&MI, &CmpInstr, &TRI, AccessToCheck);
 }
 
 /// Substitute an instruction comparing to zero with another instruction
 /// which produces needed condition flags.
 ///
 /// Return true on success.
 bool AArch64InstrInfo::substituteCmpToZero(
     MachineInstr &CmpInstr, unsigned SrcReg,
-    const MachineRegisterInfo *MRI) const {
-  assert(MRI);
+    const MachineRegisterInfo &MRI) const {
   // Get the unique definition of SrcReg.
-  MachineInstr *MI = MRI->getUniqueVRegDef(SrcReg);
+  MachineInstr *MI = MRI.getUniqueVRegDef(SrcReg);
   if (!MI)
     return false;
 
-  const TargetRegisterInfo *TRI = &getRegisterInfo();
+  const TargetRegisterInfo &TRI = getRegisterInfo();
 
   unsigned NewOpc = sForm(*MI);
   if (NewOpc == AArch64::INSTRUCTION_LIST_END)
     return false;
 
-  if (!canInstrSubstituteCmpInstr(MI, &CmpInstr, TRI))
+  if (!canInstrSubstituteCmpInstr(*MI, CmpInstr, TRI))
     return false;
 
   // Update the instruction to set NZCV.
   MI->setDesc(get(NewOpc));
   CmpInstr.eraseFromParent();
   bool succeeded = UpdateOperandRegClass(*MI);
   (void)succeeded;
   assert(succeeded && "Some operands reg class are incompatible!");
-  MI->addRegisterDefined(AArch64::NZCV, TRI);
+  MI->addRegisterDefined(AArch64::NZCV, &TRI);
+  return true;
+}
+
+/// \returns True if \p CmpInstr can be removed.
+///
+/// \p IsInvertCC is true if, after removing \p CmpInstr, condition
+/// codes used in \p CCUseInstrs must be inverted.
+static bool canCmpInstrBeRemoved(MachineInstr &MI, MachineInstr &CmpInstr,
+                                 int CmpValue, const TargetRegisterInfo &TRI,
+                                 SmallVectorImpl<MachineInstr *> &CCUseInstrs,
+                                 bool &IsInvertCC) {
+  assert(CmpValue == 0 || CmpValue == 1);
+
+  // MI is 'CSINCWr %vreg, wzr, wzr, <cc>' or 'CSINCXr %vreg, xzr, xzr, <cc>'
+  unsigned MIOpc = MI.getOpcode();
+  if (MIOpc == AArch64::CSINCWr) {
+    if (MI.getOperand(1).getReg() != AArch64::WZR ||
+        MI.getOperand(2).getReg() != AArch64::WZR)
+      return false;
+  } else if (MIOpc == AArch64::CSINCXr) {
+    if (MI.getOperand(1).getReg() != AArch64::XZR ||
+        MI.getOperand(2).getReg() != AArch64::XZR)
+      return false;
+  } else {
+    return false;
+  }
+  AArch64CC::CondCode MICC = findCondCodeUsedByInstr(MI);
+  if (MICC == AArch64CC::Invalid)
+    return false;
+
+  // NZCV needs to be defined
+  if (MI.findRegisterDefOperandIdx(AArch64::NZCV, true) != -1)
+    return false;
+
+  // CmpInstr is 'ADDS %vreg, 0' or 'SUBS %vreg, 0' or 'SUBS %vreg, 1'
+  const unsigned CmpOpcode = CmpInstr.getOpcode();
+  bool IsSubsRegImm = isSUBSRegImm(CmpOpcode);
+  if (CmpValue && !IsSubsRegImm)
+    return false;
+  if (!CmpValue && !IsSubsRegImm && !isADDSRegImm(CmpOpcode))
+    return false;
+
+  // MI conditions allowed: eq, ne, mi, pl
+  UsedNZCV MIUsedNZCV = getUsedNZCV(MICC);
+  if (MIUsedNZCV.C || MIUsedNZCV.V)
+    return false;
+
+  Optional<UsedNZCV> NZCVUsedAfterCmp =
+      examineCFlagsUse(MI, CmpInstr, TRI, &CCUseInstrs);
+  // Condition flags are not used in CmpInstr basic block successors and only
+  // Z or N flags allowed to be used after CmpInstr within its basic block
+  if (!NZCVUsedAfterCmp)
+    return false;
+  // Z or N flag used after CmpInstr must correspond to the flag used in MI
+  if ((MIUsedNZCV.Z && NZCVUsedAfterCmp->N) ||
+      (MIUsedNZCV.N && NZCVUsedAfterCmp->Z))
+    return false;
+  // If CmpInstr is comparison to zero MI conditions are limited to eq, ne
+  if (MIUsedNZCV.N && !CmpValue)
+    return false;
+
+  // There must be no defs of flags between MI and CmpInstr
+  if (areCFlagsAccessedBetweenInstrs(&MI, &CmpInstr, &TRI, AK_Write))
+    return false;
+
+  // Condition code is inverted in the following cases:
+  // 1. MI condition is ne; CmpInstr is 'ADDS %vreg, 0' or 'SUBS %vreg, 0'
+  // 2. MI condition is eq, pl; CmpInstr is 'SUBS %vreg, 1'
+  IsInvertCC = (CmpValue && (MICC == AArch64CC::EQ || MICC == AArch64CC::PL)) ||
+               (!CmpValue && MICC == AArch64CC::NE);
+  return true;
+}
+
+/// Remove comparision in csinc-cmp sequence
+///
+/// Examples:
+/// 1. \code
+///   csinc w9, wzr, wzr, ne
+///   cmp   w9, #0
+///   b.eq
+///    \endcode
+/// to
+///    \code
+///   csinc w9, wzr, wzr, ne
+///   b.ne
+///    \endcode
+///
+/// 2. \code
+///   csinc x2, xzr, xzr, mi
+///   cmp   x2, #1
+///   b.pl
+///    \endcode
+/// to
+///    \code
+///   csinc x2, xzr, xzr, mi
+///   b.pl
+///    \endcode
+///
+/// \param  CmpInstr comparison instruction
+/// \return True when comparison removed
+bool AArch64InstrInfo::removeCmpToZeroOrOne(
+    MachineInstr &CmpInstr, unsigned SrcReg, int CmpValue,
+    const MachineRegisterInfo &MRI) const {
+  MachineInstr *MI = MRI.getUniqueVRegDef(SrcReg);
+  if (!MI)
+    return false;
+  const TargetRegisterInfo &TRI = getRegisterInfo();
+  SmallVector<MachineInstr *, 4> CCUseInstrs;
+  bool IsInvertCC = false;
+  if (!canCmpInstrBeRemoved(*MI, CmpInstr, CmpValue, TRI, CCUseInstrs,
+                            IsInvertCC))
+    return false;
+  // Make transformation
+  CmpInstr.eraseFromParent();
+  if (IsInvertCC)
+    // Invert condition codes in CmpInstr CC users
+    for (MachineInstr *CCUseInstr : CCUseInstrs) {
+      int Idx = findCondCodeUseOperandIdxForBranchOrSelect(*CCUseInstr);
+      assert(Idx >= 0 && "Unexpected instruction using CC.");
+      MachineOperand &CCOperand = CCUseInstr->getOperand(Idx);
+      AArch64CC::CondCode CCUse = AArch64CC::getInvertedCondCode(
+          static_cast<AArch64CC::CondCode>(CCOperand.getImm()));
+      CCOperand.setImm(CCUse);
+    }
   return true;
 }
 
 bool AArch64InstrInfo::expandPostRAPseudo(MachineInstr &MI) const {
   if (MI.getOpcode() != TargetOpcode::LOAD_STACK_GUARD &&
       MI.getOpcode() != AArch64::CATCHRET)
     return false;
 
   MachineBasicBlock &MBB = *MI.getParent();
   auto &Subtarget = MBB.getParent()->getSubtarget<AArch64Subtarget>();
   auto TRI = Subtarget.getRegisterInfo();
   DebugLoc DL = MI.getDebugLoc();
 
   if (MI.getOpcode() == AArch64::CATCHRET) {
     // Skip to the first instruction before the epilog.
     const TargetInstrInfo *TII =
       MBB.getParent()->getSubtarget().getInstrInfo();
     MachineBasicBlock *TargetMBB = MI.getOperand(0).getMBB();
     auto MBBI = MachineBasicBlock::iterator(MI);
     MachineBasicBlock::iterator FirstEpilogSEH = std::prev(MBBI);
     while (FirstEpilogSEH->getFlag(MachineInstr::FrameDestroy) &&
            FirstEpilogSEH != MBB.begin())
       FirstEpilogSEH = std::prev(FirstEpilogSEH);
     if (FirstEpilogSEH != MBB.begin())
       FirstEpilogSEH = std::next(FirstEpilogSEH);
     BuildMI(MBB, FirstEpilogSEH, DL, TII->get(AArch64::ADRP))
         .addReg(AArch64::X0, RegState::Define)
         .addMBB(TargetMBB);
     BuildMI(MBB, FirstEpilogSEH, DL, TII->get(AArch64::ADDXri))
         .addReg(AArch64::X0, RegState::Define)
         .addReg(AArch64::X0)
         .addMBB(TargetMBB)
         .addImm(0);
     return true;
   }
if ((OpFlags & AArch64II::MO_GOT) != 0) {
     if (Subtarget.isTargetILP32()) {
       unsigned Reg32 = TRI->getSubReg(Reg, AArch64::sub_32);
       BuildMI(MBB, MI, DL, get(AArch64::LDRWui))
           .addDef(Reg32, RegState::Dead)
           .addUse(Reg, RegState::Kill)
           .addImm(0)
           .addMemOperand(*MI.memoperands_begin())
           .addDef(Reg, RegState::Implicit);
     } else {
       BuildMI(MBB, MI, DL, get(AArch64::LDRXui), Reg)
           .addReg(Reg, RegState::Kill)
           .addImm(0)
           .addMemOperand(*MI.memoperands_begin());
     }
   } else if (TM.getCodeModel() == CodeModel::Large) {
     assert(!Subtarget.isTargetILP32() && "how can large exist in ILP32?");
     BuildMI(MBB, MI, DL, get(AArch64::MOVZXi), Reg)
if (Subtarget.isTargetILP32()) {
           .addMemOperand(*MI.memoperands_begin());
     }
   }
 
   MBB.erase(MI);
 
   return true;
 }
 
 // Return true if this instruction simply sets its single destination register
 // to zero. This is equivalent to a register rename of the zero-register.
 bool AArch64InstrInfo::isGPRZero(const MachineInstr &MI) {
   switch (MI.getOpcode()) {
   default:
     break;
   case AArch64::MOVZWi:
   case AArch64::MOVZXi: // movz Rd, #0 (LSL #0)
     if (MI.getOperand(1).isImm() && MI.getOperand(1).getImm() == 0) {
       assert(MI.getDesc().getNumOperands() == 3 &&
              MI.getOperand(2).getImm() == 0 && "invalid MOVZi operands");
       return true;
     }
     break;
   case AArch64::ANDWri: // and Rd, Rzr, #imm
     return MI.getOperand(1).getReg() == AArch64::WZR;
   case AArch64::ANDXri:
     return MI.getOperand(1).getReg() == AArch64::XZR;
   case TargetOpcode::COPY:
     return MI.getOperand(1).getReg() == AArch64::WZR;
   }
   return false;
 }
 
 // Return true if this instruction simply renames a general register without
 // modifying bits.
 bool AArch64InstrInfo::isGPRCopy(const MachineInstr &MI) {
   switch (MI.getOpcode()) {

llvm/test/CodeGen/AArch64/csinc-cmp-removal.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -mtriple=aarch64 -run-pass=peephole-opt -verify-machineinstrs %s -o - | FileCheck %s
 ---
 name:            remove_subswr_after_csincwr
 tracksRegLiveness: true
 body:             |
   ; CHECK-LABEL: name: remove_subswr_after_csincwr
   ; CHECK: bb.0:
   ; CHECK:   successors: %bb.1(0x40000000), %bb.2(0x40000000)
   ; CHECK:   liveins: $x1
   ; CHECK:   [[COPY:%[0-9]+]]:gpr64common = COPY $x1
   ; CHECK:   [[DEF:%[0-9]+]]:gpr64 = IMPLICIT_DEF
   ; CHECK:   [[SUBSXrr:%[0-9]+]]:gpr64 = SUBSXrr killed [[DEF]], [[COPY]], implicit-def $nzcv
   ; CHECK:   [[CSINCWr:%[0-9]+]]:gpr32common = CSINCWr $wzr, $wzr, 1, implicit $nzcv
-  ; CHECK:   [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri killed [[CSINCWr]], 1, 0, implicit-def $nzcv
   ; CHECK:   Bcc 1, %bb.2, implicit $nzcv
   ; CHECK:   B %bb.1
   ; CHECK: bb.1:
   ; CHECK:   successors: %bb.2(0x80000000)
   ; CHECK:   B %bb.2
   ; CHECK: bb.2:
   ; CHECK:   RET_ReallyLR
   bb.0:
body:             |
   ; CHECK-LABEL: name: remove_subsxr_after_csincxr_invertcc
   ; CHECK: bb.0:
   ; CHECK:   successors: %bb.1(0x40000000), %bb.2(0x40000000)
   ; CHECK:   liveins: $x1
   ; CHECK:   [[COPY:%[0-9]+]]:gpr64common = COPY $x1
   ; CHECK:   [[DEF:%[0-9]+]]:gpr64 = IMPLICIT_DEF
   ; CHECK:   [[SUBSXrr:%[0-9]+]]:gpr64 = SUBSXrr killed [[DEF]], [[COPY]], implicit-def $nzcv
   ; CHECK:   [[CSINCXr:%[0-9]+]]:gpr64common = CSINCXr $xzr, $xzr, 1, implicit $nzcv
-  ; CHECK:   [[SUBSXri:%[0-9]+]]:gpr64 = SUBSXri killed [[CSINCXr]], 0, 0, implicit-def $nzcv
-  ; CHECK:   Bcc 0, %bb.2, implicit $nzcv
+  ; CHECK:   Bcc 1, %bb.2, implicit $nzcv
   ; CHECK:   B %bb.1
   ; CHECK: bb.1:
   ; CHECK:   successors: %bb.2(0x80000000)
   ; CHECK:   B %bb.2
   ; CHECK: bb.2:
   ; CHECK:   RET_ReallyLR
   bb.0:
     liveins: $x1

 name:            remove_addswr_after_csincwr_invertcc
 tracksRegLiveness: true
 body:             |
   ; CHECK-LABEL: name: remove_addswr_after_csincwr_invertcc
   ; CHECK: bb.0:
   ; CHECK:   successors: %bb.1(0x40000000), %bb.2(0x40000000)
   ; CHECK:   liveins: $nzcv
   ; CHECK:   [[CSINCWr:%[0-9]+]]:gpr32common = CSINCWr $wzr, $wzr, 1, implicit $nzcv
-  ; CHECK:   [[ADDSWri:%[0-9]+]]:gpr32 = ADDSWri killed [[CSINCWr]], 0, 0, implicit-def $nzcv
-  ; CHECK:   Bcc 1, %bb.2, implicit $nzcv
+  ; CHECK:   Bcc 0, %bb.2, implicit $nzcv
   ; CHECK:   B %bb.1
   ; CHECK: bb.1:
   ; CHECK:   successors: %bb.2(0x80000000)
   ; CHECK:   B %bb.2
   ; CHECK: bb.2:
   ; CHECK:   RET_ReallyLR
   bb.0:
     liveins: $nzcv

 name:            remove_subs1_after_csincN_invertcc
 tracksRegLiveness: true
 body:             |
   ; CHECK-LABEL: name: remove_subs1_after_csincN_invertcc
   ; CHECK: bb.0:
   ; CHECK:   successors: %bb.1(0x40000000), %bb.2(0x40000000)
   ; CHECK:   liveins: $nzcv
   ; CHECK:   [[CSINCWr:%[0-9]+]]:gpr32common = CSINCWr $wzr, $wzr, 5, implicit $nzcv
-  ; CHECK:   [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri killed [[CSINCWr]], 1, 0, implicit-def $nzcv
-  ; CHECK:   Bcc 4, %bb.2, implicit $nzcv
+  ; CHECK:   Bcc 5, %bb.2, implicit $nzcv
   ; CHECK:   B %bb.1
   ; CHECK: bb.1:
   ; CHECK:   successors: %bb.2(0x80000000)
   ; CHECK:   B %bb.2
   ; CHECK: bb.2:
   ; CHECK:   RET_ReallyLR
   bb.0:
     liveins: $nzcv

llvm/test/CodeGen/AArch64/f16-instructions.ll

 }
 
 ; CHECK-CVT-LABEL: test_select_cc:
 ; CHECK-CVT-DAG: fcvt s3, h3
 ; CHECK-CVT-DAG: fcvt s2, h2
 ; CHECK-CVT-DAG: fcvt s1, h1
 ; CHECK-CVT-DAG: fcvt s0, h0
 ; CHECK-CVT-DAG: fcmp s2, s3
-; CHECK-CVT-DAG: cset [[CC:w[0-9]+]], ne
-; CHECK-CVT-DAG: cmp [[CC]], #0
 ; CHECK-CVT-NEXT: fcsel s0, s0, s1, ne
 ; CHECK-CVT-NEXT: fcvt h0, s0
 ; CHECK-CVT-NEXT: ret
 
 ; CHECK-FP16-LABEL: test_select_cc:
 ; CHECK-FP16-NEXT: fcmp h2, h3
 ; CHECK-FP16-NEXT: fcsel h0, h0, h1, ne
 ; CHECK-FP16-NEXT: ret
define float @test_select_cc_f32_f16(float %a, float %b, half %c, half %d) #0 {
   %r = select i1 %cc, float %a, float %b
   ret float %r
 }
 
 ; CHECK-CVT-LABEL: test_select_cc_f16_f32:
 ; CHECK-CVT-DAG:  fcvt s0, h0
 ; CHECK-CVT-DAG:  fcvt s1, h1
 ; CHECK-CVT-DAG:  fcmp s2, s3
-; CHECK-CVT-DAG:  cset w8, ne
-; CHECK-CVT-NEXT: cmp w8, #0
 ; CHECK-CVT-NEXT: fcsel s0, s0, s1, ne
 ; CHECK-CVT-NEXT: fcvt h0, s0
 ; CHECK-CVT-NEXT: ret
 
 ; CHECK-FP16-LABEL: test_select_cc_f16_f32:
 ; CHECK-FP16-NEXT: fcmp s2, s3
 ; CHECK-FP16-NEXT: fcsel h0, h0, h1, ne
 ; CHECK-FP16-NEXT: ret