[AArch64][GlobalISel] Optimize conditional branches followed by unconditional branches

[aemerson]

Summary

If we have an icmp->brcond->br sequence where the brcond just branches to the next block jumping over the br, while the br takes the false edge, then we can modify the conditional branch to jump to the br's target while inverting the condition of the incoming icmp. This means we can eliminate the br as an unconditional branch to the fallthrough block.

[arsenm]

Why is this done in the target selector instead of in a generic combine?

[arsenm]

In D64354#1573842, @arsenm wrote:

Why is this done in the target selector instead of in a generic combine?

I'm pretty sure SelectionDAGBuilder does this today

[aditya_nandakumar]

I haven't read the patch yet, but just from reading the description I remembered that the DAG does this while building up the DAG. Maybe this needs to be done in the IRTranslator to keep this behavior? (This helps while comparing codegen with sdag/gisel).

[qcolombet]

llvm/lib/Target/AArch64/AArch64InstructionSelector.cpp

1171

Taking advantage of fall-through sounds like something beneficial for all targets.
Could we turn this into a combiner helper?

(Then you can call this specific helper in here.)

[arsenm]

FWIW I would like some control over what boolean sources are acceptable to invert. SelectionDAG inserting a condition invert between a control flow intrinsic and the branch required work to undo

[aemerson]

This seemed a little presumptive to do for all targets for me. I can try to move it earlier into a Combiner helper, since I don't want to put optimizations into the IRTranslator unless we can't do it in other places.

In D64354#1573876, @arsenm wrote:

FWIW I would like some control over what boolean sources are acceptable to invert. SelectionDAG inserting a condition invert between a control flow intrinsic and the branch required work to undo

In this case I'm just inverting the condition code, we shouldn't need to insert extra instructions.

[paquette]

Combiner helper makes sense here IMO.

llvm/lib/Target/AArch64/AArch64InstructionSelector.cpp

1186

Would BrIt->isTerminator() work here?

1197

Is trunc the only thing we should look through here? Can we also handle copies, for example?

[aemerson]

llvm/lib/Target/AArch64/AArch64InstructionSelector.cpp

1186

Branch is a terminator by nature, so it doesn't check the same thing, although the usefulness of this assert is debatable.

1197

For arm64 it seems to be the only thing in between compares and branches in practice due to our legalization rules. This code will be removed anyway since I'm going to put this before the legalizer so we shouldn't see any artifacts in between the ops,

[aemerson]

New patch to do the optimization in the CombinerHelper and add it to the prelegalizer combiner for AArch64.

This doesn't get all the code size improvements of the last patch, I'll make that a separate patch. On it's own, this change gives a 2.5% geomean improvement on CTMark -O0.

[aemerson]

Add missing test.

[aditya_nandakumar]

Unless others have comments, this looks good to go in to me.

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

392 ↗

(On Diff #208510)

Orthogonal to this patch - I've found that MachineOperand::getPredicate returning an unsigned and users having to explicitly cast to Predicate is a little annoying. What do you think of getPredicate returning Predicate type (in a separate patch)? In my use cases I've almost never used the unsigned value without casting to Predicate.

[aemerson]

Marking as accepted.

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

392 ↗

(On Diff #208510)

That sounds reasonable to me.

llvm/include/llvm/CodeGen/GlobalISel/InstructionSelector.h

 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_GLOBALISEL_INSTRUCTIONSELECTOR_H
 #define LLVM_CODEGEN_GLOBALISEL_INSTRUCTIONSELECTOR_H
 
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/Optional.h"
+#include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Support/CodeGenCoverage.h"
 #include "llvm/Support/LowLevelTypeImpl.h"
 #include <bitset>
 #include <cstddef>
 #include <cstdint>
 #include <functional>
 #include <initializer_list>
public:
   /// generic pre-isel instructions.
   ///
   /// \returns whether selection succeeded.
   /// \pre  I.getParent() && I.getParent()->getParent()
   /// \post
   ///   if returns true:
   ///     for I in all mutated/inserted instructions:
   ///       !isPreISelGenericOpcode(I.getOpcode())
-  virtual bool select(MachineInstr &I, CodeGenCoverage &CoverageInfo) const = 0;
+  virtual bool select(MachineInstr &I, CodeGenCoverage &CoverageInfo) = 0;
+
+  /// Initialize selector state.
+  void init() {
+    InstsMarkedForDeletion.clear();
+  }
+
+  /// Returns true if the given generic instruction has been marked for deletion
+  /// and therefore shouldn't be selected.
+  bool isInstMarkedForDeletion(MachineInstr *MI) const {
+    return InstsMarkedForDeletion.count(MI);
+  }
 
 protected:
   using ComplexRendererFns =
       Optional<SmallVector<std::function<void(MachineInstrBuilder &)>, 4>>;
   using RecordedMIVector = SmallVector<MachineInstr *, 4>;
   using NewMIVector = SmallVector<MachineInstrBuilder, 4>;
 
   struct MatcherState {
     std::vector<ComplexRendererFns::value_type> Renderers;
     RecordedMIVector MIs;
     DenseMap<unsigned, unsigned> TempRegisters;
 
     MatcherState(unsigned MaxRenderers);
   };
 
+  // Set of generic instructions which, during selection, have been marked for
+  // deletion. They're not immediately deleted during selection because the
+  // InstructionSelect pass has iterators which can't handle being invalidated.
+  SmallPtrSet<MachineInstr *, 32> InstsMarkedForDeletion;
+
+  void markInstForDeletion(MachineInstr *MI) {
+    InstsMarkedForDeletion.insert(MI);
+  }
 public:
   template <class PredicateBitset, class ComplexMatcherMemFn,
             class CustomRendererFn>
   struct ISelInfoTy {
     ISelInfoTy(const LLT *TypeObjects, size_t NumTypeObjects,
                const PredicateBitset *FeatureBitsets,
                const ComplexMatcherMemFn *ComplexPredicates,
                const CustomRendererFn *CustomRenderers)

llvm/include/llvm/CodeGen/MachineOperand.h

public:
   /// operand does not take ownership of the memory referenced by Mask, it must
   /// remain valid for the lifetime of the operand. See CreateRegMask().
   /// Any physreg with a 0 bit in the mask is clobbered by the instruction.
   void setRegMask(const uint32_t *RegMaskPtr) {
     assert(isRegMask() && "Wrong MachineOperand mutator");
     Contents.RegMask = RegMaskPtr;
   }
 
+  void setPredicate(unsigned Predicate) {
+    assert(isPredicate() && "Wrong MachineOperand mutator");
+    Contents.Pred = Predicate;
+  }
+
   //===--------------------------------------------------------------------===//
   // Other methods.
   //===--------------------------------------------------------------------===//
 
   /// Returns true if this operand is identical to the specified operand except
   /// for liveness related flags (isKill, isUndef and isDead). Note that this
   /// should stay in sync with the hash_value overload below.
   bool isIdenticalTo(const MachineOperand &Other) const;

llvm/include/llvm/CodeGen/TargetSubtargetInfo.h

virtual const SelectionDAGTargetInfo *getSelectionDAGInfo() const {
     return nullptr;
   }
   virtual const CallLowering *getCallLowering() const { return nullptr; }
 
   // FIXME: This lets targets specialize the selector by subtarget (which lets
   // us do things like a dedicated avx512 selector).  However, we might want
   // to also specialize selectors by MachineFunction, which would let us be
   // aware of optsize/optnone and such.
-  virtual const InstructionSelector *getInstructionSelector() const {
+  virtual InstructionSelector *getInstructionSelector() const {
     return nullptr;
   }
 
   virtual unsigned getHwMode() const { return 0; }
 
   /// Target can subclass this hook to select a different DAG scheduler.
   virtual RegisterScheduler::FunctionPassCtor
       getDAGScheduler(CodeGenOpt::Level) const {

llvm/lib/CodeGen/GlobalISel/InstructionSelect.cpp

bool InstructionSelect::runOnMachineFunction(MachineFunction &MF) {
   // If the ISel pipeline failed, do not bother running that pass.
   if (MF.getProperties().hasProperty(
           MachineFunctionProperties::Property::FailedISel))
     return false;
 
   LLVM_DEBUG(dbgs() << "Selecting function: " << MF.getName() << '\n');
 
   const TargetPassConfig &TPC = getAnalysis<TargetPassConfig>();
-  const InstructionSelector *ISel = MF.getSubtarget().getInstructionSelector();
+  InstructionSelector *ISel = MF.getSubtarget().getInstructionSelector();
+  ISel->init();
   CodeGenCoverage CoverageInfo;
   assert(ISel && "Cannot work without InstructionSelector");
 
   // An optimization remark emitter. Used to report failures.
   MachineOptimizationRemarkEmitter MORE(MF, /*MBFI=*/nullptr);
 
   // FIXME: There are many other MF/MFI fields we need to initialize.
 
#endif
       // We could have folded this instruction away already, making it dead.
       // If so, erase it.
       if (isTriviallyDead(MI, MRI)) {
         LLVM_DEBUG(dbgs() << "Is dead; erasing.\n");
         MI.eraseFromParentAndMarkDBGValuesForRemoval();
         continue;
       }
 
+      if (ISel->isInstMarkedForDeletion(&MI)) {
+        LLVM_DEBUG(dbgs() << "marked for deletion. Erasing...\n");
+        MI.eraseFromParentAndMarkDBGValuesForRemoval();
+        continue;
+      }
+
       if (!ISel->select(MI, CoverageInfo)) {
         // FIXME: It would be nice to dump all inserted instructions.  It's
         // not obvious how, esp. considering select() can insert after MI.
         reportGISelFailure(MF, TPC, MORE, "gisel-select", "cannot select", MI);
         return false;
       }
 
       // Dump the range of instructions that MI expanded into.

llvm/lib/Target/AArch64/AArch64InstructionSelector.cpp

 #undef GET_GLOBALISEL_PREDICATE_BITSET
 
 class AArch64InstructionSelector : public InstructionSelector {
 public:
   AArch64InstructionSelector(const AArch64TargetMachine &TM,
                              const AArch64Subtarget &STI,
                              const AArch64RegisterBankInfo &RBI);
 
-  bool select(MachineInstr &I, CodeGenCoverage &CoverageInfo) const override;
+  bool select(MachineInstr &I, CodeGenCoverage &CoverageInfo) override;
   static const char *getName() { return DEBUG_TYPE; }
 
 private:
   /// tblgen-erated 'select' implementation, used as the initial selector for
   /// the patterns that don't require complex C++.
   bool selectImpl(MachineInstr &I, CodeGenCoverage &CoverageInfo) const;
 
   // A lowering phase that runs before any selection attempts.
 
   void preISelLower(MachineInstr &I) const;
 
   // An early selection function that runs before the selectImpl() call.
-  bool earlySelect(MachineInstr &I) const;
+  bool earlySelect(MachineInstr &I);
 
   bool earlySelectSHL(MachineInstr &I, MachineRegisterInfo &MRI) const;
+  bool earlySelectBr(MachineInstr &I, MachineRegisterInfo &MRI);
 
   bool selectVaStartAAPCS(MachineInstr &I, MachineFunction &MF,
                           MachineRegisterInfo &MRI) const;
   bool selectVaStartDarwin(MachineInstr &I, MachineFunction &MF,
                            MachineRegisterInfo &MRI) const;
 
-  bool selectCompareBranch(MachineInstr &I, MachineFunction &MF,
-                           MachineRegisterInfo &MRI) const;
+  bool matchCompareBranch(MachineInstr &I, MachineRegisterInfo &MRI,
+                          unsigned &CBOpc, Register &Val) const;
+  bool selectCompareBranch(MachineInstr &I, MachineRegisterInfo &MRI) const;
 
   bool selectVectorASHR(MachineInstr &I, MachineRegisterInfo &MRI) const;
   bool selectVectorSHL(MachineInstr &I, MachineRegisterInfo &MRI) const;
 
   // Helper to generate an equivalent of scalar_to_vector into a new register,
   // returned via 'Dst'.
   MachineInstr *emitScalarToVector(unsigned EltSize,
                                    const TargetRegisterClass *DstRC,
private:
   MachineInstr *emitLoadFromConstantPool(Constant *CPVal,
                                          MachineIRBuilder &MIRBuilder) const;
 
   // Emit a vector concat operation.
   MachineInstr *emitVectorConcat(Optional<Register> Dst, Register Op1,
                                  Register Op2,
                                  MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitIntegerCompare(MachineOperand &LHS, MachineOperand &RHS,
+                                   MachineIRBuilder &MIRBuilder) const;
+  MachineInstr *emitIntegerCompareOpt(MachineOperand &LHS, MachineOperand &RHS,
                                       MachineOperand &Predicate,
                                       MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitCMN(MachineOperand &LHS, MachineOperand &RHS,
                         MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitExtractVectorElt(Optional<Register> DstReg,
                                      const RegisterBank &DstRB, LLT ScalarTy,
                                      Register VecReg, unsigned LaneIdx,
                                      MachineIRBuilder &MIRBuilder) const;
 
   /// Helper function for selecting G_FCONSTANT. If the G_FCONSTANT can be
case CmpInst::FCMP_ULE:
     CondCode = AArch64CC::LE;
     break;
   case CmpInst::FCMP_UNE:
     CondCode = AArch64CC::NE;
     break;
   }
 }
 
-bool AArch64InstructionSelector::selectCompareBranch(
-    MachineInstr &I, MachineFunction &MF, MachineRegisterInfo &MRI) const {
-
+bool AArch64InstructionSelector::matchCompareBranch(MachineInstr &I,
+                                                    MachineRegisterInfo &MRI,
+                                                    unsigned &CBOpc,
+                                                    Register &Val) const {
   const Register CondReg = I.getOperand(0).getReg();
-  MachineBasicBlock *DestMBB = I.getOperand(1).getMBB();
   MachineInstr *CCMI = MRI.getVRegDef(CondReg);
   if (CCMI->getOpcode() == TargetOpcode::G_TRUNC)
     CCMI = MRI.getVRegDef(CCMI->getOperand(1).getReg());
   if (CCMI->getOpcode() != TargetOpcode::G_ICMP)
     return false;
 
   Register LHS = CCMI->getOperand(2).getReg();
   Register RHS = CCMI->getOperand(3).getReg();
   if (!getConstantVRegVal(RHS, MRI))
     std::swap(RHS, LHS);
 
   const auto RHSImm = getConstantVRegVal(RHS, MRI);
   if (!RHSImm || *RHSImm != 0)
     return false;
 
   const RegisterBank &RB = *RBI.getRegBank(LHS, MRI, TRI);
   if (RB.getID() != AArch64::GPRRegBankID)
     return false;
 
   const auto Pred = (CmpInst::Predicate)CCMI->getOperand(1).getPredicate();
   if (Pred != CmpInst::ICMP_NE && Pred != CmpInst::ICMP_EQ)
     return false;
 
   const unsigned CmpWidth = MRI.getType(LHS).getSizeInBits();
-  unsigned CBOpc = 0;
+  CBOpc = 0;
   if (CmpWidth <= 32)
     CBOpc = (Pred == CmpInst::ICMP_EQ ? AArch64::CBZW : AArch64::CBNZW);
   else if (CmpWidth == 64)
     CBOpc = (Pred == CmpInst::ICMP_EQ ? AArch64::CBZX : AArch64::CBNZX);
   else
     return false;
+  Val = LHS;
+  return true;
+}
+
+bool AArch64InstructionSelector::selectCompareBranch(
+    MachineInstr &I, MachineRegisterInfo &MRI) const {
+  MachineBasicBlock *DestMBB = I.getOperand(1).getMBB();
+  unsigned CBOpc;
+  Register Val;
+  if (!matchCompareBranch(I, MRI, CBOpc, Val))
+    return false;
 
   BuildMI(*I.getParent(), I, I.getDebugLoc(), TII.get(CBOpc))
-      .addUse(LHS)
+      .addUse(Val)
       .addMBB(DestMBB)
       .constrainAllUses(TII, TRI, RBI);
 
   I.eraseFromParent();
   return true;
 }
 
 bool AArch64InstructionSelector::selectVectorSHL(
for (auto &RenderFn : *Imm1Fn)
     RenderFn(NewI);
   for (auto &RenderFn : *Imm2Fn)
     RenderFn(NewI);
 
   I.eraseFromParent();
   return constrainSelectedInstRegOperands(*NewI, TII, TRI, RBI);
 }
 
-bool AArch64InstructionSelector::earlySelect(MachineInstr &I) const {
+bool AArch64InstructionSelector::earlySelectBr(MachineInstr &I,
+                                               MachineRegisterInfo &MRI) {
+  assert(I.getOpcode() == TargetOpcode::G_BR && "Expected a G_BR");
+
+  // The importer will select a standard unconditional branch, but we first
+  // try to match the following:
+  // bb1:
+  //   %c(s32) = G_ICMP pred, %a, %b
+  //   %c1(s1) = G_TRUNC %c(s32)
+  //   G_BRCOND %c1, %bb2
+  //   G_BR %bb3
+  // bb2:
+  // ...
+  // bb3:
+
+  // The above pattern does not have a fall through to the successor bb2, always
+  // resulting in a branch no matter which path is taken. Here we try to find
+  // and replace that pattern with conditional branch to bb3 and otherwise
+  // fallthrough to bb2.
+
+  MachineBasicBlock *MBB = I.getParent();
+  MachineBasicBlock *BrTarget = I.getOperand(0).getMBB();
+  MachineBasicBlock::iterator BrIt(I);
+  if (BrIt == MBB->begin())
+    return false;
+  assert(std::next(BrIt) == MBB->end() &&
+         "expected G_BR to be a terminator");
+
+  MachineInstr *BrCond = &*std::prev(BrIt);
+  if (BrCond->getOpcode() != TargetOpcode::G_BRCOND)
+    return false;
+
+  // Check that the next block is the conditional branch target.
+  if (!MBB->isLayoutSuccessor(BrCond->getOperand(1).getMBB()))
+    return false;
+
+  // Look through the trunc to get the icmp.
+  MachineInstr *TruncMI = MRI.getVRegDef(BrCond->getOperand(0).getReg());
+  if (!TruncMI || TruncMI->getOpcode() != TargetOpcode::G_TRUNC)
+    return false;
+  MachineInstr *CmpMI = MRI.getVRegDef(TruncMI->getOperand(1).getReg());
+  if (!CmpMI || CmpMI->getOpcode() != TargetOpcode::G_ICMP)
+    return false;
+
+  CmpInst::Predicate InversePred = CmpInst::getInversePredicate(
+      (CmpInst::Predicate)CmpMI->getOperand(1).getPredicate());
+  const AArch64CC::CondCode InverseCC =
+      changeICMPPredToAArch64CC((CmpInst::Predicate)InversePred);
+
+  MachineIRBuilder MIB(I);
+
+  auto *MF = MBB->getParent();
+  // Before we generate our replacement code, first check to see if the brcond
+  // will be selected to a cbz/cbnz later. If so, modify icmp predicate and
+  // swap the branch targets, leaving the brcond to be selected later.
+  if (!MF->getFunction().hasFnAttribute(Attribute::SpeculativeLoadHardening)) {
+    unsigned CBOpc;
+    Register Val;
+    if (matchCompareBranch(*BrCond, MRI, CBOpc, Val)) {
+      // Need to invert our G_ICMP predicate since we're going to swap the
+      // branch targets around. First we have to check that there are no other
+      // users of the icmp/trunc except for the condbr.
+      if (!MRI.hasOneUse(TruncMI->getOperand(0).getReg()) ||
+          !MRI.hasOneUse(CmpMI->getOperand(0).getReg()))
+        return false;
+      CmpMI->getOperand(1).setPredicate(InversePred);
+      BrCond->getOperand(1).setMBB(BrTarget);
+      I.eraseFromParent();
+      return true;
+    }
+  }
+
+  // Generate a normal compare.
+  MachineInstr *Cmp =
+      emitIntegerCompare(CmpMI->getOperand(2), CmpMI->getOperand(3), MIB);
+  if (!Cmp)
+    return false;
+
+  // Create the conditional branch.
+  MIB.buildInstr(AArch64::Bcc, {}, {}).addImm(InverseCC).addMBB(BrTarget);
+
+  // We can't rely on the simple DCE to erase the brcond so we need to tell the
+  // select pass to delete it. We don't do it here to avoid invalidating
+  // iterators already held by callers.
+  markInstForDeletion(BrCond);
+
+  // Now delete old br, this block will now fall through to the successor.
+  I.eraseFromParent();
+  return true;
+}
+
+bool AArch64InstructionSelector::earlySelect(MachineInstr &I) {
   assert(I.getParent() && "Instruction should be in a basic block!");
   assert(I.getParent()->getParent() && "Instruction should be in a function!");
 
   MachineBasicBlock &MBB = *I.getParent();
   MachineFunction &MF = *MBB.getParent();
   MachineRegisterInfo &MRI = MF.getRegInfo();
 
   switch (I.getOpcode()) {
   case TargetOpcode::G_SHL:
     return earlySelectSHL(I, MRI);
+  case TargetOpcode::G_BR:
+    return earlySelectBr(I, MRI);
   default:
     return false;
   }
 }
 
 bool AArch64InstructionSelector::select(MachineInstr &I,
-                                        CodeGenCoverage &CoverageInfo) const {
+                                        CodeGenCoverage &CoverageInfo) {
   assert(I.getParent() && "Instruction should be in a basic block!");
   assert(I.getParent()->getParent() && "Instruction should be in a function!");
 
   MachineBasicBlock &MBB = *I.getParent();
   MachineFunction &MF = *MBB.getParent();
   MachineRegisterInfo &MRI = MF.getRegInfo();
 
   unsigned Opcode = I.getOpcode();
case TargetOpcode::G_BRCOND: {
     const Register CondReg = I.getOperand(0).getReg();
     MachineBasicBlock *DestMBB = I.getOperand(1).getMBB();
 
     // Speculation tracking/SLH assumes that optimized TB(N)Z/CB(N)Z
     // instructions will not be produced, as they are conditional branch
     // instructions that do not set flags.
     bool ProduceNonFlagSettingCondBr =
         !MF.getFunction().hasFnAttribute(Attribute::SpeculativeLoadHardening);
-    if (ProduceNonFlagSettingCondBr && selectCompareBranch(I, MF, MRI))
+    if (ProduceNonFlagSettingCondBr && selectCompareBranch(I, MRI))
       return true;
 
     if (ProduceNonFlagSettingCondBr) {
       auto MIB = BuildMI(MBB, I, I.getDebugLoc(), TII.get(AArch64::TBNZW))
                      .addUse(CondReg)
                      .addImm(/*bit offset=*/0)
                      .addMBB(DestMBB);
 
case TargetOpcode::G_ICMP: {
 
     if (Ty != LLT::scalar(32)) {
       LLVM_DEBUG(dbgs() << "G_ICMP result has type: " << Ty
                         << ", expected: " << LLT::scalar(32) << '\n');
       return false;
     }
 
     MachineIRBuilder MIRBuilder(I);
-    if (!emitIntegerCompare(I.getOperand(2), I.getOperand(3), I.getOperand(1),
-                            MIRBuilder))
+    if (!emitIntegerCompareOpt(I.getOperand(2), I.getOperand(3),
+                               I.getOperand(1), MIRBuilder))
       return false;
     emitCSetForICMP(I.getOperand(0).getReg(), I.getOperand(1).getPredicate(),
                     MIRBuilder);
     I.eraseFromParent();
     return true;
   }
 
   case TargetOpcode::G_FCMP: {
if (ImmFns) {
     CmpMI.addUse(RHS.getReg());
   }
 
   constrainSelectedInstRegOperands(*CmpMI, TII, TRI, RBI);
   return &*CmpMI;
 }
 
 MachineInstr *AArch64InstructionSelector::emitIntegerCompare(
-    MachineOperand &LHS, MachineOperand &RHS, MachineOperand &Predicate,
+    MachineOperand &LHS, MachineOperand &RHS,
     MachineIRBuilder &MIRBuilder) const {
   assert(LHS.isReg() && RHS.isReg() && "Expected LHS and RHS to be registers!");
   MachineRegisterInfo &MRI = MIRBuilder.getMF().getRegInfo();
-
-  // Fold the compare into a CMN if possible.
-  MachineInstr *Cmn = tryOptCMN(LHS, RHS, Predicate, MIRBuilder);
-  if (Cmn)
-    return Cmn;
-
-  // Can't fold into a CMN. Just emit a normal compare.
   unsigned CmpOpc = 0;
   Register ZReg;
 
   LLT CmpTy = MRI.getType(LHS.getReg());
   assert((CmpTy.isScalar() || CmpTy.isPointer()) &&
          "Expected scalar or pointer");
   if (CmpTy == LLT::scalar(32)) {
     CmpOpc = AArch64::SUBSWrr;
if (ImmFns) {
     CmpMI.addUse(RHS.getReg());
   }
 
   // Make sure that we can constrain the compare that we emitted.
   constrainSelectedInstRegOperands(*CmpMI, TII, TRI, RBI);
   return &*CmpMI;
 }
 
+MachineInstr *AArch64InstructionSelector::emitIntegerCompareOpt(
+    MachineOperand &LHS, MachineOperand &RHS, MachineOperand &Predicate,
+    MachineIRBuilder &MIRBuilder) const {
+  // Fold the compare into a CMN if possible.
+  MachineInstr *Cmn = tryOptCMN(LHS, RHS, Predicate, MIRBuilder);
+  if (Cmn)
+    return Cmn;
+  // Can't fold into a CMN. Just emit a normal compare.
+  return emitIntegerCompare(LHS, RHS, MIRBuilder);
+}
+
 MachineInstr *AArch64InstructionSelector::emitVectorConcat(
     Optional<Register> Dst, Register Op1, Register Op2,
     MachineIRBuilder &MIRBuilder) const {
   // We implement a vector concat by:
   // 1. Use scalar_to_vector to insert the lower vector into the larger dest
   // 2. Insert the upper vector into the destination's upper element
   // TODO: some of this code is common with G_BUILD_VECTOR handling.
   MachineRegisterInfo &MRI = MIRBuilder.getMF().getRegInfo();
bool AArch64InstructionSelector::tryOptSelect(MachineInstr &I) const {
   unsigned CondOpc = CondDef->getOpcode();
   if (CondOpc != TargetOpcode::G_ICMP && CondOpc != TargetOpcode::G_FCMP)
     return false;
 
   AArch64CC::CondCode CondCode;
   if (CondOpc == TargetOpcode::G_ICMP) {
     CondCode = changeICMPPredToAArch64CC(
         (CmpInst::Predicate)CondDef->getOperand(1).getPredicate());
-    if (!emitIntegerCompare(CondDef->getOperand(2), CondDef->getOperand(3),
+    if (!emitIntegerCompareOpt(CondDef->getOperand(2), CondDef->getOperand(3),
                                CondDef->getOperand(1), MIB)) {
       LLVM_DEBUG(dbgs() << "Couldn't emit compare for select!\n");
       return false;
     }
   } else {
     // Get the condition code for the select.
     AArch64CC::CondCode CondCode2;
     changeFCMPPredToAArch64CC(
         (CmpInst::Predicate)CondDef->getOperand(1).getPredicate(), CondCode,

llvm/lib/Target/AArch64/AArch64Subtarget.h

public:
   const AArch64TargetLowering *getTargetLowering() const override {
     return &TLInfo;
   }
   const AArch64InstrInfo *getInstrInfo() const override { return &InstrInfo; }
   const AArch64RegisterInfo *getRegisterInfo() const override {
     return &getInstrInfo()->getRegisterInfo();
   }
   const CallLowering *getCallLowering() const override;
-  const InstructionSelector *getInstructionSelector() const override;
+  InstructionSelector *getInstructionSelector() const override;
   const LegalizerInfo *getLegalizerInfo() const override;
   const RegisterBankInfo *getRegBankInfo() const override;
   const Triple &getTargetTriple() const { return TargetTriple; }
   bool enableMachineScheduler() const override { return true; }
   bool enablePostRAScheduler() const override {
     return UsePostRAScheduler;
   }
 

llvm/lib/Target/AArch64/AArch64Subtarget.cpp

AArch64Subtarget::AArch64Subtarget(const Triple &TT, const std::string &CPU,
 
   RegBankInfo.reset(RBI);
 }
 
 const CallLowering *AArch64Subtarget::getCallLowering() const {
   return CallLoweringInfo.get();
 }
 
-const InstructionSelector *AArch64Subtarget::getInstructionSelector() const {
+InstructionSelector *AArch64Subtarget::getInstructionSelector() const {
   return InstSelector.get();
 }
 
 const LegalizerInfo *AArch64Subtarget::getLegalizerInfo() const {
   return Legalizer.get();
 }
 
 const RegisterBankInfo *AArch64Subtarget::getRegBankInfo() const {

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h

 class SIRegisterInfo;
 
 class AMDGPUInstructionSelector : public InstructionSelector {
 public:
   AMDGPUInstructionSelector(const GCNSubtarget &STI,
                             const AMDGPURegisterBankInfo &RBI,
                             const AMDGPUTargetMachine &TM);
 
-  bool select(MachineInstr &I, CodeGenCoverage &CoverageInfo) const override;
+  bool select(MachineInstr &I, CodeGenCoverage &CoverageInfo) override;
   static const char *getName();
 
 private:
   struct GEPInfo {
     const MachineInstr &GEP;
     SmallVector<unsigned, 2> SgprParts;
     SmallVector<unsigned, 2> VgprParts;
     int64_t Imm;

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

bool AMDGPUInstructionSelector::selectG_FRAME_INDEX(MachineInstr &I) const {
   if (IsVGPR)
     I.addOperand(*MF, MachineOperand::CreateReg(AMDGPU::EXEC, false, true));
 
   return RBI.constrainGenericRegister(
     DstReg, IsVGPR ? AMDGPU::VGPR_32RegClass : AMDGPU::SReg_32RegClass, MRI);
 }
 
 bool AMDGPUInstructionSelector::select(MachineInstr &I,
-                                       CodeGenCoverage &CoverageInfo) const {
+                                       CodeGenCoverage &CoverageInfo) {
   if (I.isPHI())
     return selectPHI(I);
 
   if (!isPreISelGenericOpcode(I.getOpcode())) {
     if (I.isCopy())
       return selectCOPY(I);
     return true;
   }

llvm/lib/Target/AMDGPU/AMDGPUSubtarget.h

public:
   const SIRegisterInfo *getRegisterInfo() const override {
     return &InstrInfo.getRegisterInfo();
   }
 
   const CallLowering *getCallLowering() const override {
     return CallLoweringInfo.get();
   }
 
-  const InstructionSelector *getInstructionSelector() const override {
+  InstructionSelector *getInstructionSelector() const override {
     return InstSelector.get();
   }
 
   const LegalizerInfo *getLegalizerInfo() const override {
     return Legalizer.get();
   }
 
   const RegisterBankInfo *getRegBankInfo() const override {

llvm/lib/Target/ARM/ARMInstructionSelector.cpp

 #include "ARMGenGlobalISel.inc"
 #undef GET_GLOBALISEL_PREDICATE_BITSET
 
 class ARMInstructionSelector : public InstructionSelector {
 public:
   ARMInstructionSelector(const ARMBaseTargetMachine &TM, const ARMSubtarget &STI,
                          const ARMRegisterBankInfo &RBI);
 
-  bool select(MachineInstr &I, CodeGenCoverage &CoverageInfo) const override;
+  bool select(MachineInstr &I, CodeGenCoverage &CoverageInfo) override;
   static const char *getName() { return DEBUG_TYPE; }
 
 private:
   bool selectImpl(MachineInstr &I, CodeGenCoverage &CoverageInfo) const;
 
   struct CmpConstants;
   struct InsertInfo;
 
void ARMInstructionSelector::renderVFPF64Imm(
   APFloat FPImmValue = OldInst.getOperand(1).getFPImm()->getValueAPF();
   int FPImmEncoding = ARM_AM::getFP64Imm(FPImmValue);
   assert(FPImmEncoding != -1 && "Invalid immediate value");
 
   NewInstBuilder.addImm(FPImmEncoding);
 }
 
 bool ARMInstructionSelector::select(MachineInstr &I,
-                                    CodeGenCoverage &CoverageInfo) const {
+                                    CodeGenCoverage &CoverageInfo) {
   assert(I.getParent() && "Instruction should be in a basic block!");
   assert(I.getParent()->getParent() && "Instruction should be in a function!");
 
   auto &MBB = *I.getParent();
   auto &MF = *MBB.getParent();
   auto &MRI = MF.getRegInfo();
 
   if (!isPreISelGenericOpcode(I.getOpcode())) {

llvm/lib/Target/ARM/ARMSubtarget.h

const ARMFrameLowering *getFrameLowering() const override {
     return FrameLowering.get();
   }
 
   const ARMBaseRegisterInfo *getRegisterInfo() const override {
     return &InstrInfo->getRegisterInfo();
   }
 
   const CallLowering *getCallLowering() const override;
-  const InstructionSelector *getInstructionSelector() const override;
+  InstructionSelector *getInstructionSelector() const override;
   const LegalizerInfo *getLegalizerInfo() const override;
   const RegisterBankInfo *getRegBankInfo() const override;
 
 private:
   ARMSelectionDAGInfo TSInfo;
   // Either Thumb1FrameLowering or ARMFrameLowering.
   std::unique_ptr<ARMFrameLowering> FrameLowering;
   // Either Thumb1InstrInfo or Thumb2InstrInfo.

llvm/lib/Target/ARM/ARMSubtarget.cpp

ARMSubtarget::ARMSubtarget(const Triple &TT, const std::string &CPU,
 
   RegBankInfo.reset(RBI);
 }
 
 const CallLowering *ARMSubtarget::getCallLowering() const {
   return CallLoweringInfo.get();
 }
 
-const InstructionSelector *ARMSubtarget::getInstructionSelector() const {
+InstructionSelector *ARMSubtarget::getInstructionSelector() const {
   return InstSelector.get();
 }
 
 const LegalizerInfo *ARMSubtarget::getLegalizerInfo() const {
   return Legalizer.get();
 }
 
 const RegisterBankInfo *ARMSubtarget::getRegBankInfo() const {

llvm/lib/Target/Mips/MipsInstructionSelector.cpp

 #include "MipsGenGlobalISel.inc"
 #undef GET_GLOBALISEL_PREDICATE_BITSET
 
 class MipsInstructionSelector : public InstructionSelector {
 public:
   MipsInstructionSelector(const MipsTargetMachine &TM, const MipsSubtarget &STI,
                           const MipsRegisterBankInfo &RBI);
 
-  bool select(MachineInstr &I, CodeGenCoverage &CoverageInfo) const override;
+  bool select(MachineInstr &I, CodeGenCoverage &CoverageInfo) override;
   static const char *getName() { return DEBUG_TYPE; }
 
 private:
   bool selectImpl(MachineInstr &I, CodeGenCoverage &CoverageInfo) const;
   bool materialize32BitImm(Register DestReg, APInt Imm,
                            MachineIRBuilder &B) const;
   bool selectCopy(MachineInstr &I, MachineRegisterInfo &MRI) const;
 
else
     case 1:
       return Opc == TargetOpcode::G_SEXTLOAD ? Mips::LB : Mips::LBu;
     default:
       return Opc;
     }
 }
 
 bool MipsInstructionSelector::select(MachineInstr &I,
-                                     CodeGenCoverage &CoverageInfo) const {
+                                     CodeGenCoverage &CoverageInfo) {
 
   MachineBasicBlock &MBB = *I.getParent();
   MachineFunction &MF = *MBB.getParent();
   MachineRegisterInfo &MRI = MF.getRegInfo();
 
   if (!isPreISelGenericOpcode(I.getOpcode())) {
     if (I.isCopy())
       return selectCopy(I, MRI);

llvm/lib/Target/Mips/MipsSubtarget.h

protected:
   std::unique_ptr<LegalizerInfo> Legalizer;
   std::unique_ptr<RegisterBankInfo> RegBankInfo;
   std::unique_ptr<InstructionSelector> InstSelector;
 
 public:
   const CallLowering *getCallLowering() const override;
   const LegalizerInfo *getLegalizerInfo() const override;
   const RegisterBankInfo *getRegBankInfo() const override;
-  const InstructionSelector *getInstructionSelector() const override;
+  InstructionSelector *getInstructionSelector() const override;
 };
 } // End llvm namespace
 
 #endif

llvm/lib/Target/Mips/MipsSubtarget.cpp

 const LegalizerInfo *MipsSubtarget::getLegalizerInfo() const {
   return Legalizer.get();
 }
 
 const RegisterBankInfo *MipsSubtarget::getRegBankInfo() const {
   return RegBankInfo.get();
 }
 
-const InstructionSelector *MipsSubtarget::getInstructionSelector() const {
+InstructionSelector *MipsSubtarget::getInstructionSelector() const {
   return InstSelector.get();
 }

llvm/lib/Target/X86/X86InstructionSelector.cpp

 #include "X86GenGlobalISel.inc"
 #undef GET_GLOBALISEL_PREDICATE_BITSET
 
 class X86InstructionSelector : public InstructionSelector {
 public:
   X86InstructionSelector(const X86TargetMachine &TM, const X86Subtarget &STI,
                          const X86RegisterBankInfo &RBI);
 
-  bool select(MachineInstr &I, CodeGenCoverage &CoverageInfo) const override;
+  bool select(MachineInstr &I, CodeGenCoverage &CoverageInfo) override;
   static const char *getName() { return DEBUG_TYPE; }
 
 private:
   /// tblgen-erated 'select' implementation, used as the initial selector for
   /// the patterns that don't require complex C++.
   bool selectImpl(MachineInstr &I, CodeGenCoverage &CoverageInfo) const;
 
   // TODO: remove after supported by Tablegen-erated instruction selection.
private:
   bool selectCmp(MachineInstr &I, MachineRegisterInfo &MRI,
                  MachineFunction &MF) const;
   bool selectFCmp(MachineInstr &I, MachineRegisterInfo &MRI,
                   MachineFunction &MF) const;
   bool selectUadde(MachineInstr &I, MachineRegisterInfo &MRI,
                    MachineFunction &MF) const;
   bool selectCopy(MachineInstr &I, MachineRegisterInfo &MRI) const;
   bool selectUnmergeValues(MachineInstr &I, MachineRegisterInfo &MRI,
-                           MachineFunction &MF,
-                           CodeGenCoverage &CoverageInfo) const;
+                           MachineFunction &MF, CodeGenCoverage &CoverageInfo);
   bool selectMergeValues(MachineInstr &I, MachineRegisterInfo &MRI,
-                         MachineFunction &MF,
-                         CodeGenCoverage &CoverageInfo) const;
+                         MachineFunction &MF, CodeGenCoverage &CoverageInfo);
   bool selectInsert(MachineInstr &I, MachineRegisterInfo &MRI,
                     MachineFunction &MF) const;
   bool selectExtract(MachineInstr &I, MachineRegisterInfo &MRI,
                      MachineFunction &MF) const;
   bool selectCondBranch(MachineInstr &I, MachineRegisterInfo &MRI,
                         MachineFunction &MF) const;
   bool selectTurnIntoCOPY(MachineInstr &I, MachineRegisterInfo &MRI,
                           const unsigned DstReg,
if (!RBI.constrainGenericRegister(DstReg, *DstRC, MRI)) {
       return false;
     }
   }
   I.setDesc(TII.get(X86::COPY));
   return true;
 }
 
 bool X86InstructionSelector::select(MachineInstr &I,
-                                    CodeGenCoverage &CoverageInfo) const {
+                                    CodeGenCoverage &CoverageInfo) {
   assert(I.getParent() && "Instruction should be in a basic block!");
   assert(I.getParent()->getParent() && "Instruction should be in a function!");
 
   MachineBasicBlock &MBB = *I.getParent();
   MachineFunction &MF = *MBB.getParent();
   MachineRegisterInfo &MRI = MF.getRegInfo();
 
   unsigned Opcode = I.getOpcode();
bool X86InstructionSelector::selectInsert(MachineInstr &I,
 
   I.getOperand(3).setImm(Index);
 
   return constrainSelectedInstRegOperands(I, TII, TRI, RBI);
 }
 
 bool X86InstructionSelector::selectUnmergeValues(
     MachineInstr &I, MachineRegisterInfo &MRI, MachineFunction &MF,
-    CodeGenCoverage &CoverageInfo) const {
+    CodeGenCoverage &CoverageInfo) {
   assert((I.getOpcode() == TargetOpcode::G_UNMERGE_VALUES) &&
          "unexpected instruction");
 
   // Split to extracts.
   unsigned NumDefs = I.getNumOperands() - 1;
   unsigned SrcReg = I.getOperand(NumDefs).getReg();
   unsigned DefSize = MRI.getType(I.getOperand(0).getReg()).getSizeInBits();
 
bool X86InstructionSelector::selectUnmergeValues(
   }
 
   I.eraseFromParent();
   return true;
 }
 
 bool X86InstructionSelector::selectMergeValues(
     MachineInstr &I, MachineRegisterInfo &MRI, MachineFunction &MF,
-    CodeGenCoverage &CoverageInfo) const {
+    CodeGenCoverage &CoverageInfo) {
   assert((I.getOpcode() == TargetOpcode::G_MERGE_VALUES ||
           I.getOpcode() == TargetOpcode::G_CONCAT_VECTORS) &&
          "unexpected instruction");
 
   // Split to inserts.
   unsigned DstReg = I.getOperand(0).getReg();
   unsigned SrcReg0 = I.getOperand(1).getReg();
 

llvm/lib/Target/X86/X86Subtarget.h

public:
   unsigned getMaxInlineSizeThreshold() const { return MaxInlineSizeThreshold; }
 
   /// ParseSubtargetFeatures - Parses features string setting specified
   /// subtarget options.  Definition of function is auto generated by tblgen.
   void ParseSubtargetFeatures(StringRef CPU, StringRef FS);
 
   /// Methods used by Global ISel
   const CallLowering *getCallLowering() const override;
-  const InstructionSelector *getInstructionSelector() const override;
+  InstructionSelector *getInstructionSelector() const override;
   const LegalizerInfo *getLegalizerInfo() const override;
   const RegisterBankInfo *getRegBankInfo() const override;
 
 private:
   /// Initialize the full set of dependencies so we can use an initializer
   /// list for X86Subtarget.
   X86Subtarget &initializeSubtargetDependencies(StringRef CPU, StringRef FS);
   void initSubtargetFeatures(StringRef CPU, StringRef FS);

llvm/lib/Target/X86/X86Subtarget.cpp

X86Subtarget::X86Subtarget(const Triple &TT, StringRef CPU, StringRef FS,
   RegBankInfo.reset(RBI);
   InstSelector.reset(createX86InstructionSelector(TM, *this, *RBI));
 }
 
 const CallLowering *X86Subtarget::getCallLowering() const {
   return CallLoweringInfo.get();
 }
 
-const InstructionSelector *X86Subtarget::getInstructionSelector() const {
+InstructionSelector *X86Subtarget::getInstructionSelector() const {
   return InstSelector.get();
 }
 
 const LegalizerInfo *X86Subtarget::getLegalizerInfo() const {
   return Legalizer.get();
 }
 
 const RegisterBankInfo *X86Subtarget::getRegBankInfo() const {

llvm/test/CodeGen/AArch64/GlobalISel/opt-br-brcond.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple aarch64-unknown-unknown -run-pass=instruction-select -verify-machineinstrs %s -o - | FileCheck %s
+# This test checks that we correctly optimize a brcond->br sequence to fallthrough when available
+# and eliminate unnecessary csets/unconditional branches.
+--- |
+  target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
+  target triple = "arm64-apple-ios5.0.0"
+
+  define i32 @foo(i32 %a, i32 %b) {
+  entry:
+    %cmp = icmp sgt i32 %a, %b
+    br i1 %cmp, label %if.then, label %if.end
+
+  if.then:                                          ; preds = %entry
+    %add = add nsw i32 %b, %a
+    %call = tail call i32 @bar(i32 %add)
+    %add1 = add nsw i32 %call, %b
+    br label %return
+
+  if.end:                                           ; preds = %entry
+    %mul = mul nsw i32 %b, %b
+    %add2 = add nuw nsw i32 %mul, 2
+    br label %return
+
+  return:                                           ; preds = %if.end, %if.then
+    %retval.0 = phi i32 [ %add1, %if.then ], [ %add2, %if.end ]
+    ret i32 %retval.0
+  }
+
+  declare i32 @bar(i32)
+
+  ; Function Attrs: nounwind
+  declare void @llvm.stackprotector(i8*, i8**) #0
+
+  attributes #0 = { nounwind }
+
+...
+---
+name:            foo
+alignment:       2
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+frameInfo:
+  hasCalls:        true
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: foo
+  ; CHECK: bb.0.entry:
+  ; CHECK:   successors: %bb.1(0x40000000), %bb.2(0x40000000)
+  ; CHECK:   liveins: $w0, $w1
+  ; CHECK:   [[COPY:%[0-9]+]]:gpr32 = COPY $w0
+  ; CHECK:   [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+  ; CHECK:   $wzr = SUBSWrr [[COPY]], [[COPY1]], implicit-def $nzcv
+  ; CHECK:   Bcc 13, %bb.2, implicit $nzcv
+  ; CHECK: bb.1.if.then:
+  ; CHECK:   successors: %bb.3(0x80000000)
+  ; CHECK:   %5:gpr32 = nsw ADDWrr [[COPY1]], [[COPY]]
+  ; CHECK:   ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
+  ; CHECK:   $w0 = COPY %5
+  ; CHECK:   BL @bar, csr_aarch64_aapcs, implicit-def $lr, implicit $sp, implicit $w0, implicit-def $w0
+  ; CHECK:   [[COPY2:%[0-9]+]]:gpr32 = COPY $w0
+  ; CHECK:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
+  ; CHECK:   %7:gpr32 = nsw ADDWrr [[COPY2]], [[COPY1]]
+  ; CHECK:   B %bb.3
+  ; CHECK: bb.2.if.end:
+  ; CHECK:   successors: %bb.3(0x80000000)
+  ; CHECK:   [[MADDWrrr:%[0-9]+]]:gpr32common = MADDWrrr [[COPY1]], [[COPY1]], $wzr
+  ; CHECK:   [[ADDWri:%[0-9]+]]:gpr32sp = ADDWri [[MADDWrrr]], 2, 0
+  ; CHECK: bb.3.return:
+  ; CHECK:   [[PHI:%[0-9]+]]:gpr32 = PHI %7, %bb.1, [[ADDWri]], %bb.2
+  ; CHECK:   $w0 = COPY [[PHI]]
+  ; CHECK:   RET_ReallyLR implicit $w0
+  bb.1.entry:
+    liveins: $w0, $w1
+
+    %0:gpr(s32) = COPY $w0
+    %1:gpr(s32) = COPY $w1
+    %4:gpr(s32) = G_CONSTANT i32 2
+    %10:gpr(s32) = G_ICMP intpred(sgt), %0(s32), %1
+    %2:gpr(s1) = G_TRUNC %10(s32)
+    G_BRCOND %2(s1), %bb.2
+    G_BR %bb.3
+
+  bb.2.if.then:
+    %6:gpr(s32) = nsw G_ADD %1, %0
+    ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
+    $w0 = COPY %6(s32)
+    BL @bar, csr_aarch64_aapcs, implicit-def $lr, implicit $sp, implicit $w0, implicit-def $w0
+    %7:gpr(s32) = COPY $w0
+    ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
+    %8:gpr(s32) = nsw G_ADD %7, %1
+    G_BR %bb.4
+
+  bb.3.if.end:
+    %3:gpr(s32) = nsw G_MUL %1, %1
+    %5:gpr(s32) = nuw nsw G_ADD %3, %4
+
+  bb.4.return:
+    %9:gpr(s32) = G_PHI %8(s32), %bb.2, %5(s32), %bb.3
+    $w0 = COPY %9(s32)
+    RET_ReallyLR implicit $w0
+
+...

llvm/test/CodeGen/AArch64/GlobalISel/select-cbz.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -mtriple=aarch64-- -run-pass=instruction-select -verify-machineinstrs %s -o - | FileCheck %s
 
 --- |
-  define void @cbz_s32() { ret void }
-  define void @cbz_s64() { ret void }
   define void @cbnz_s32() { ret void }
   define void @cbnz_s64() { ret void }
+  define void @cbz_s32() { ret void }
+  define void @cbz_s64() { ret void }
 ...
 
+# These tests check for CBZ/CBNZ codegen, but they look inverted w.r.t the
+# constant in the icmp. Another optimization inverts the predicate so that the
+# block can fallthrough without using an unconditional br.
+
 ---
-# CHECK-LABEL: name: cbz_s32
-name:            cbz_s32
+name:            cbnz_s32
 legalized:       true
 regBankSelected: true
 
-# CHECK:  body:
-# CHECK:   bb.0:
-# CHECK:    %0:gpr32 = COPY $w0
-# CHECK:    CBZW %0, %bb.1
-# CHECK:    B %bb.0
 body:             |
+  ; CHECK-LABEL: name: cbnz_s32
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.0(0x40000000), %bb.1(0x40000000)
+  ; CHECK:   [[COPY:%[0-9]+]]:gpr32 = COPY $w0
+  ; CHECK:   CBNZW [[COPY]], %bb.0
+  ; CHECK: bb.1:
   bb.0:
     liveins: $w0
     successors: %bb.0, %bb.1
 
     %0:gpr(s32) = COPY $w0
     %1:gpr(s32) = G_CONSTANT i32 0
     %2:gpr(s32) = G_ICMP intpred(eq), %0, %1
     %3:gpr(s1) = G_TRUNC %2(s32)
     G_BRCOND %3(s1), %bb.1
     G_BR %bb.0
 
   bb.1:
 ...
 
 ---
-# CHECK-LABEL: name: cbz_s64
-name:            cbz_s64
+name:            cbnz_s64
 legalized:       true
 regBankSelected: true
 
-# CHECK:  body:
-# CHECK:   bb.0:
-# CHECK:    %0:gpr64 = COPY $x0
-# CHECK:    CBZX %0, %bb.1
-# CHECK:    B %bb.0
 body:             |
+  ; CHECK-LABEL: name: cbnz_s64
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.0(0x40000000), %bb.1(0x40000000)
+  ; CHECK:   [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+  ; CHECK:   CBNZX [[COPY]], %bb.0
+  ; CHECK: bb.1:
   bb.0:
     liveins: $x0
     successors: %bb.0, %bb.1
 
     %0:gpr(s64) = COPY $x0
     %1:gpr(s64) = G_CONSTANT i64 0
     %2:gpr(s32) = G_ICMP intpred(eq), %0, %1
     %3:gpr(s1) = G_TRUNC %2(s32)
     G_BRCOND %3(s1), %bb.1
     G_BR %bb.0
 
   bb.1:
 ...
 
 ---
-# CHECK-LABEL: name: cbnz_s32
-name:            cbnz_s32
+name:            cbz_s32
 legalized:       true
 regBankSelected: true
 
-# CHECK:  body:
-# CHECK:   bb.0:
-# CHECK:    %0:gpr32 = COPY $w0
-# CHECK:    CBNZW %0, %bb.1
-# CHECK:    B %bb.0
 body:             |
+  ; CHECK-LABEL: name: cbz_s32
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.0(0x40000000), %bb.1(0x40000000)
+  ; CHECK:   [[COPY:%[0-9]+]]:gpr32 = COPY $w0
+  ; CHECK:   CBZW [[COPY]], %bb.0
+  ; CHECK: bb.1:
   bb.0:
     liveins: $w0
     successors: %bb.0, %bb.1
 
     %0:gpr(s32) = COPY $w0
     %1:gpr(s32) = G_CONSTANT i32 0
     %2:gpr(s32) = G_ICMP intpred(ne), %0, %1
     %3:gpr(s1) = G_TRUNC %2(s32)
     G_BRCOND %3(s1), %bb.1
     G_BR %bb.0
 
   bb.1:
 ...
 
 ---
-# CHECK-LABEL: name: cbnz_s64
-name:            cbnz_s64
+name:            cbz_s64
 legalized:       true
 regBankSelected: true
 
-# CHECK:  body:
-# CHECK:   bb.0:
-# CHECK:    %0:gpr64 = COPY $x0
-# CHECK:    CBNZX %0, %bb.1
-# CHECK:    B %bb.0
 body:             |
+  ; CHECK-LABEL: name: cbz_s64
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.0(0x40000000), %bb.1(0x40000000)
+  ; CHECK:   [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+  ; CHECK:   CBZX [[COPY]], %bb.0
+  ; CHECK: bb.1:
   bb.0:
     liveins: $x0
     successors: %bb.0, %bb.1
 
     %0:gpr(s64) = COPY $x0
     %1:gpr(s64) = G_CONSTANT i64 0
     %2:gpr(s32) = G_ICMP intpred(ne), %0, %1
     %3:gpr(s1) = G_TRUNC %2(s32)
     G_BRCOND %3(s1), %bb.1
     G_BR %bb.0
 
   bb.1:
 ...