[AArch64][GlobalISel] Select G_UADDE/G_SADDE/G_USUBE/G_SSUBE

[tobias-stadler]

Summary

This implements the remaining overflow generating instructions in the AArch64 GlobalISel selector.
Now wide add/sub operations do not fallback to SelectionDAG anymore.
We make use of PostSelectOptimize to cleanup the hereby generated flag-setting operations when the carry-out is unused.
Since we do not fallback anymore when selecting add/sub atomics on O0 some test changes were required there.

Fixes: https://github.com/llvm/llvm-project/issues/59407

[arsenm]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

4820

Don't need null check

4876

I didn't even know hasAtMostUserInstrs existed, just use use_nodbg_empty

[tschuett]

How did you noticed the fallback?
They were marked as legal, but there was no support in instruction selection?

[tschuett]

Could you please add https://github.com/llvm/llvm-project/issues/59407 to the summary?

[tobias-stadler]

Address comments. Thanks.

[tobias-stadler]

Update commit summary to include fixed issue. Thanks.

[tobias-stadler]

In D153164#4430170, @tschuett wrote:

How did you noticed the fallback?
They were marked as legal, but there was no support in instruction selection?

We noticed this, because our application (database query compilation) makes frequent use of 128bit add/sub overflow intrinsics, which produced a significant amount of fallbacks when using GlobalISel. With this patch, we don't fallback anymore (at least from what we have tested so far).
The G_*E operations were marked as legal by https://reviews.llvm.org/D95325, but there was no selector implementation provided at that time.

[aemerson]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

4566

MachineIRBuilder has getMRI().

4798–4799

Since we only have 2 mutually exclusive choices, I'd just use something like bool IsSub = false. Then simplify below to:

if (Opcode == TargetOpcode::G_SSUBE || Opcode == TargetOpcode::G_USUBE)
  IsSub = true;

NegateCarry then is also redundant since it's only set when IsSub = true

4848–4867

This code looks pretty verbose. How about we add some new helper classes to GenericMachineInstrs.h so we have something like GBinOpCarryOut, GBinOpCarryIn. We could then eliminate this switch and replace with something like:

assert(isa<GBinOpCarryOut>(I) || isa<GBinOpCarryIn>(I));

if (auto *InMI = dyn_cast<GBinOpCarryIn>(&I)) {
  // Set NZCV carry according to carry-in VReg
  emitCarryIn(I, InMI->getCarryReg());
}

this way we also avoid hard coding an operand index.

[tobias-stadler]

Address comments. Thanks for the suggestions. I think the ifs and switches cleaned up quite nicely using the new helper classes.
My original train of thought behind the switch in emitCarryIn was that if anyone wanted to add support for carry-in setting of other instructions they would have to explicitly add their instruction and think about if they needed a negated carry and could opt-in to the optimizations only if they wanted to. I can't think of any other usecase though, so emitting the carry as-is also seems like a sane default.

Minor cleanup in PostSelectOptimize and added missed *Wrs variants of instructions.

[barannikov88]

Register getCarryOutReg();

For subtraction this is actually a borrow-out. This naming can be confusing.
Has anyone though about deviating from SelectionDAG here and inverting the meaning of the flag for subtractions, so that it is actually a carry for both add/sub?
I understand that we can't just invert the meaning, because many clients (including downstream) rely on the current behavior.
So maybe via deprecation? UADDE/UADDO aren't good names anyway (I still can't figure out what do E and O letters mean).

(not related to this review)
I also had this idea about adding a variant of all ALU operations with status flags output (NZVC), and using this output in G_BRCOND_S/G_SELECT_S along with condition code to test (e.g. "overflow set, zero clear").
Carry-producing instructions could then be replaced by these more general instructions.
This allows eliminating G_ICMP in many cases (and actually make G_ICMP redundant because it could be replaced with G_SUB_S with status flags output).
For example if ((x & y) <= 0) could be translated into G_AND_S + G_BRCOND_S with "not positive" condition code (N == 0 || Z == 0), if the target has one.
I've implemented this as custom nodes in a downstream SelectionDAG as proof of concept. It works, but there are some subtleties, and it is somewhat machine dependent (e.g. RISC-V does not have status flags, as far as I know).

I'm not very familiar with GISel, nor I have time to try to implement this idea upstream, but maybe it will interest someone.

[tobias-stadler]

In D153164#4440902, @barannikov88 wrote:

Register getCarryOutReg();

For subtraction this is actually a borrow-out. This naming can be confusing.
Has anyone though about deviating from SelectionDAG here and inverting the meaning of the flag for subtractions, so that it is actually a carry for both add/sub?
I understand that we can't just invert the meaning, because many clients (including downstream) rely on the current behavior.
So maybe via deprecation? UADDE/UADDO aren't good names anyway (I still can't figure out what do E and O letters mean).

(not related to this review)
I also had this idea about adding a variant of all ALU operations with status flags output (NZVC), and using this output in G_BRCOND_S/G_SELECT_S along with condition code to test (e.g. "overflow set, zero clear").
Carry-producing instructions could then be replaced by these more general instructions.
This allows eliminating G_ICMP in many cases (and actually make G_ICMP redundant because it could be replaced with G_SUB_S with status flags output).
For example if ((x & y) <= 0) could be translated into G_AND_S + G_BRCOND_S with "not positive" condition code (N == 0 || Z == 0), if the target has one.
I've implemented this as custom nodes in a downstream SelectionDAG as proof of concept. It works, but there are some subtleties, and it is somewhat machine dependent (e.g. RISC-V does not have status flags, as far as I know).

I'm not very familiar with GISel, nor I have time to try to implement this idea upstream, but maybe it will interest someone.

As far as my patch is concerned: CarryIn/CarryOut is consistent with the terminology used in MachineIRBuilder, the docs and other places I have seen, so I'd prefer to keep it this way.

[barannikov88]

In D153164#4442119, @tobias-stadler wrote:

As far as my patch is concerned: CarryIn/CarryOut is consistent with the terminology used in MachineIRBuilder, the docs and other places I have seen, so I'd prefer to keep it this way.

Sure, I just wanted to identify the (bigger) issue.

[aemerson]

LGTM. Thanks for the nice improvements.

[tobias-stadler]

In D153164#4445206, @aemerson wrote:

LGTM. Thanks for the nice improvements.

Thanks for the review. This is my first patch, so I don't have commit access. Could you land this for me, please?
Looks like I accidentally created the commit using the github no-reply mail. Please use this author information:

Tobias Stadler <[e-mail]>

llvm/include/llvm/CodeGen/GlobalISel/GenericMachineInstrs.h

 /// Represent a G_FCMP.
 class GFCmp : public GAnyCmp {
 public:
   static bool classof(const MachineInstr *MI) {
     return MI->getOpcode() == TargetOpcode::G_FCMP;
   }
 };
 
+/// Represents overflowing binary operations.
+/// Only carry-out:
+/// G_UADDO, G_SADDO, G_USUBO, G_SSUBO, G_UMULO, G_SMULO
+/// Carry-in and carry-out:
+/// G_UADDE, G_SADDE, G_USUBE, G_SSUBE
+class GBinOpCarryOut : public GenericMachineInstr {
+public:
+  Register getDstReg() const { return getReg(0); }
+  Register getCarryOutReg() const { return getReg(1); }
+  MachineOperand &getLHS() { return getOperand(2); }
+  MachineOperand &getRHS() { return getOperand(3); }
+
+  static bool classof(const MachineInstr *MI) {
+    switch (MI->getOpcode()) {
+    case TargetOpcode::G_UADDO:
+    case TargetOpcode::G_SADDO:
+    case TargetOpcode::G_USUBO:
+    case TargetOpcode::G_SSUBO:
+    case TargetOpcode::G_UADDE:
+    case TargetOpcode::G_SADDE:
+    case TargetOpcode::G_USUBE:
+    case TargetOpcode::G_SSUBE:
+    case TargetOpcode::G_UMULO:
+    case TargetOpcode::G_SMULO:
+      return true;
+    default:
+      return false;
+    }
+  }
+};
+
+/// Represents overflowing add/sub operations.
+/// Only carry-out:
+/// G_UADDO, G_SADDO, G_USUBO, G_SSUBO
+/// Carry-in and carry-out:
+/// G_UADDE, G_SADDE, G_USUBE, G_SSUBE
+class GAddSubCarryOut : public GBinOpCarryOut {
+public:
+  bool isAdd() const {
+    switch (getOpcode()) {
+    case TargetOpcode::G_UADDO:
+    case TargetOpcode::G_SADDO:
+    case TargetOpcode::G_UADDE:
+    case TargetOpcode::G_SADDE:
+      return true;
+    default:
+      return false;
+    }
+  }
+  bool isSub() const { return !isAdd(); }
+
+  bool isSigned() const {
+    switch (getOpcode()) {
+    case TargetOpcode::G_SADDO:
+    case TargetOpcode::G_SSUBO:
+    case TargetOpcode::G_SADDE:
+    case TargetOpcode::G_SSUBE:
+      return true;
+    default:
+      return false;
+    }
+  }
+  bool isUnsigned() const { return !isSigned(); }
+
+  static bool classof(const MachineInstr *MI) {
+    switch (MI->getOpcode()) {
+    case TargetOpcode::G_UADDO:
+    case TargetOpcode::G_SADDO:
+    case TargetOpcode::G_USUBO:
+    case TargetOpcode::G_SSUBO:
+    case TargetOpcode::G_UADDE:
+    case TargetOpcode::G_SADDE:
+    case TargetOpcode::G_USUBE:
+    case TargetOpcode::G_SSUBE:
+      return true;
+    default:
+      return false;
+    }
+  }
+};
+
+/// Represents overflowing add/sub operations that also consume a carry-in.
+/// G_UADDE, G_SADDE, G_USUBE, G_SSUBE
+class GAddSubCarryInOut : public GAddSubCarryOut {
+public:
+  Register getCarryInReg() const { return getReg(4); }
+
+  static bool classof(const MachineInstr *MI) {
+    switch (MI->getOpcode()) {
+    case TargetOpcode::G_UADDE:
+    case TargetOpcode::G_SADDE:
+    case TargetOpcode::G_USUBE:
+    case TargetOpcode::G_SSUBE:
+      return true;
+    default:
+      return false;
+    }
+  }
+};
+
 } // namespace llvm
 
 #endif // LLVM_CODEGEN_GLOBALISEL_GENERICMACHINEINSTRS_H

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

MachineInstr *emitAddSub(
       MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitADD(Register DefReg, MachineOperand &LHS,
                         MachineOperand &RHS,
                         MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitADDS(Register Dst, MachineOperand &LHS, MachineOperand &RHS,
                          MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitSUBS(Register Dst, MachineOperand &LHS, MachineOperand &RHS,
                          MachineIRBuilder &MIRBuilder) const;
+  MachineInstr *emitADCS(Register Dst, MachineOperand &LHS, MachineOperand &RHS,
+                         MachineIRBuilder &MIRBuilder) const;
+  MachineInstr *emitSBCS(Register Dst, MachineOperand &LHS, MachineOperand &RHS,
+                         MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitCMN(MachineOperand &LHS, MachineOperand &RHS,
                         MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitTST(MachineOperand &LHS, MachineOperand &RHS,
                         MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitSelect(Register Dst, Register LHS, Register RHS,
                            AArch64CC::CondCode CC,
                            MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitExtractVectorElt(std::optional<Register> DstReg,
                                      const RegisterBank &DstRB, LLT ScalarTy,
                                      Register VecReg, unsigned LaneIdx,
                                      MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitCSINC(Register Dst, Register Src1, Register Src2,
                           AArch64CC::CondCode Pred,
                           MachineIRBuilder &MIRBuilder) const;
   /// Emit a CSet for a FP compare.
   ///
   /// \p Dst is expected to be a 32-bit scalar register.
   MachineInstr *emitCSetForFCmp(Register Dst, CmpInst::Predicate Pred,
                                 MachineIRBuilder &MIRBuilder) const;
 
+  /// Emit an instruction that sets NZCV to the carry-in expected by \p I.
+  /// Might elide the instruction if the previous instruction already sets NZCV
+  /// correctly.
+  MachineInstr *emitCarryIn(MachineInstr &I, Register CarryReg);
+
   /// Emit the overflow op for \p Opcode.
   ///
   /// \p Opcode is expected to be an overflow op's opcode, e.g. G_UADDO,
   /// G_USUBO, etc.
   std::pair<MachineInstr *, AArch64CC::CondCode>
   emitOverflowOp(unsigned Opcode, Register Dst, MachineOperand &LHS,
                  MachineOperand &RHS, MachineIRBuilder &MIRBuilder) const;
 
+  bool selectOverflowOp(MachineInstr &I, MachineRegisterInfo &MRI);
+
   /// Emit expression as a conjunction (a series of CCMP/CFCMP ops).
   /// In some cases this is even possible with OR operations in the expression.
   MachineInstr *emitConjunction(Register Val, AArch64CC::CondCode &OutCC,
                                 MachineIRBuilder &MIB) const;
   MachineInstr *emitConditionalComparison(Register LHS, Register RHS,
                                           CmpInst::Predicate CC,
                                           AArch64CC::CondCode Predicate,
                                           AArch64CC::CondCode OutCC,
case TargetOpcode::G_OR: {
     return constrainSelectedInstRegOperands(I, TII, TRI, RBI);
   }
 
   case TargetOpcode::G_PTR_ADD: {
     emitADD(I.getOperand(0).getReg(), I.getOperand(1), I.getOperand(2), MIB);
     I.eraseFromParent();
     return true;
   }
+
+  case TargetOpcode::G_SADDE:
+  case TargetOpcode::G_UADDE:
+  case TargetOpcode::G_SSUBE:
+  case TargetOpcode::G_USUBE:
   case TargetOpcode::G_SADDO:
   case TargetOpcode::G_UADDO:
   case TargetOpcode::G_SSUBO:
-  case TargetOpcode::G_USUBO: {
-    // Emit the operation and get the correct condition code.
-    auto OpAndCC = emitOverflowOp(Opcode, I.getOperand(0).getReg(),
-                                  I.getOperand(2), I.getOperand(3), MIB);
-
-    // Now, put the overflow result in the register given by the first operand
-    // to the overflow op. CSINC increments the result when the predicate is
-    // false, so to get the increment when it's true, we need to use the
-    // inverse. In this case, we want to increment when carry is set.
-    Register ZReg = AArch64::WZR;
-    emitCSINC(/*Dst=*/I.getOperand(1).getReg(), /*Src1=*/ZReg, /*Src2=*/ZReg,
-              getInvertedCondCode(OpAndCC.second), MIB);
-    I.eraseFromParent();
-    return true;
-  }
+  case TargetOpcode::G_USUBO:
+    return selectOverflowOp(I, MRI);
 
   case TargetOpcode::G_PTRMASK: {
     Register MaskReg = I.getOperand(2).getReg();
     std::optional<int64_t> MaskVal = getIConstantVRegSExtVal(MaskReg, MRI);
     // TODO: Implement arbitrary cases
     if (!MaskVal || !isShiftedMask_64(*MaskVal))
       return false;
 
const std::array<std::array<unsigned, 2>, 5> OpcTable{
        {AArch64::SUBSXrs, AArch64::SUBSWrs},
        {AArch64::SUBSXrr, AArch64::SUBSWrr},
        {AArch64::ADDSXri, AArch64::ADDSWri},
        {AArch64::SUBSXrx, AArch64::SUBSWrx}}};
   return emitAddSub(OpcTable, Dst, LHS, RHS, MIRBuilder);
 }
 
 MachineInstr *
+AArch64InstructionSelector::emitADCS(Register Dst, MachineOperand &LHS,
+                                     MachineOperand &RHS,
+                                     MachineIRBuilder &MIRBuilder) const {
+  assert(LHS.isReg() && RHS.isReg() && "Expected register operands?");
+  MachineRegisterInfo *MRI = MIRBuilder.getMRI();
+  bool Is32Bit = (MRI->getType(LHS.getReg()).getSizeInBits() == 32);
+  static const unsigned OpcTable[2] = {AArch64::ADCSXr, AArch64::ADCSWr};
+  return emitInstr(OpcTable[Is32Bit], {Dst}, {LHS, RHS}, MIRBuilder);
+}
+
+MachineInstr *
+AArch64InstructionSelector::emitSBCS(Register Dst, MachineOperand &LHS,
+                                     MachineOperand &RHS,
+                                     MachineIRBuilder &MIRBuilder) const {
+  assert(LHS.isReg() && RHS.isReg() && "Expected register operands?");
+  MachineRegisterInfo *MRI = MIRBuilder.getMRI();
+  bool Is32Bit = (MRI->getType(LHS.getReg()).getSizeInBits() == 32);
+  static const unsigned OpcTable[2] = {AArch64::SBCSXr, AArch64::SBCSWr};
+  return emitInstr(OpcTable[Is32Bit], {Dst}, {LHS, RHS}, MIRBuilder);
+}
+
+MachineInstr *
 AArch64InstructionSelector::emitCMN(MachineOperand &LHS, MachineOperand &RHS,
                                     MachineIRBuilder &MIRBuilder) const {
   MachineRegisterInfo &MRI = MIRBuilder.getMF().getRegInfo();
   bool Is32Bit = (MRI.getType(LHS.getReg()).getSizeInBits() == 32);
   auto RC = Is32Bit ? &AArch64::GPR32RegClass : &AArch64::GPR64RegClass;
   return emitADDS(MRI.createVirtualRegister(RC), LHS, RHS, MIRBuilder);
 }
 
AArch64InstructionSelector::emitCSINC(Register Dst, Register Src1,
   assert(Size <= 64 && "Expected 64 bits or less only!");
   static const unsigned OpcTable[2] = {AArch64::CSINCWr, AArch64::CSINCXr};
   unsigned Opc = OpcTable[Size == 64];
   auto CSINC = MIRBuilder.buildInstr(Opc, {Dst}, {Src1, Src2}).addImm(Pred);
   constrainSelectedInstRegOperands(*CSINC, TII, TRI, RBI);
   return &*CSINC;
 }
 
+MachineInstr *AArch64InstructionSelector::emitCarryIn(MachineInstr &I,
+                                                      Register CarryReg) {
+  MachineRegisterInfo *MRI = MIB.getMRI();
+  unsigned Opcode = I.getOpcode();
+
+  // If the instruction is a SUB, we need to negate the carry,
+  // because borrowing is indicated by carry-flag == 0.
+  bool NeedsNegatedCarry =
+      (Opcode == TargetOpcode::G_USUBE || Opcode == TargetOpcode::G_SSUBE);
+
+  // If the previous instruction will already produce the correct carry, do not
+  // emit a carry generating instruction. E.g. for G_UADDE/G_USUBE sequences
+  // generated during legalization of wide add/sub. This optimization depends on
+  // these sequences not being interrupted by other instructions.
+  MachineInstr *SrcMI = MRI->getVRegDef(CarryReg);
+  if (SrcMI == I.getPrevNode()) {
+    if (auto *CarrySrcMI = dyn_cast<GAddSubCarryOut>(SrcMI)) {
+      bool ProducesNegatedCarry = CarrySrcMI->isSub();
+      if (NeedsNegatedCarry == ProducesNegatedCarry && CarrySrcMI->isUnsigned())
+        return nullptr;
+    }
+  }
+
+  Register DeadReg = MRI->createVirtualRegister(&AArch64::GPR32RegClass);
+
+  if (NeedsNegatedCarry) {
+    // (0 - Carry) sets !C in NZCV when Carry == 1
+    Register ZReg = AArch64::WZR;
+    return emitInstr(AArch64::SUBSWrr, {DeadReg}, {ZReg, CarryReg}, MIB);
+  }
+
+  // (Carry - 1) sets !C in NZCV when Carry == 0
+  auto Fns = select12BitValueWithLeftShift(1);
+  return emitInstr(AArch64::SUBSWri, {DeadReg}, {CarryReg}, MIB, Fns);
+}
+
+bool AArch64InstructionSelector::selectOverflowOp(MachineInstr &I,
+                                                  MachineRegisterInfo &MRI) {
+  auto &CarryMI = cast<GAddSubCarryOut>(I);
+
+  if (auto *CarryInMI = dyn_cast<GAddSubCarryInOut>(&I)) {
+    // Set NZCV carry according to carry-in VReg
+    emitCarryIn(I, CarryInMI->getCarryInReg());
+  }
+
+  // Emit the operation and get the correct condition code.
+  auto OpAndCC = emitOverflowOp(I.getOpcode(), CarryMI.getDstReg(),
+                                CarryMI.getLHS(), CarryMI.getRHS(), MIB);
+
+  Register CarryOutReg = CarryMI.getCarryOutReg();
+
+  // Don't convert carry-out to VReg if it is never used
+  if (!MRI.use_nodbg_empty(CarryOutReg)) {
+    // Now, put the overflow result in the register given by the first operand
+    // to the overflow op. CSINC increments the result when the predicate is
+    // false, so to get the increment when it's true, we need to use the
+    // inverse. In this case, we want to increment when carry is set.
+    Register ZReg = AArch64::WZR;
+    emitCSINC(/*Dst=*/CarryOutReg, /*Src1=*/ZReg, /*Src2=*/ZReg,
+              getInvertedCondCode(OpAndCC.second), MIB);
+  }
+
+  I.eraseFromParent();
+  return true;
+}
+
 std::pair<MachineInstr *, AArch64CC::CondCode>
 AArch64InstructionSelector::emitOverflowOp(unsigned Opcode, Register Dst,
                                            MachineOperand &LHS,
                                            MachineOperand &RHS,
                                            MachineIRBuilder &MIRBuilder) const {
   switch (Opcode) {
   default:
     llvm_unreachable("Unexpected opcode!");
   case TargetOpcode::G_SADDO:
     return std::make_pair(emitADDS(Dst, LHS, RHS, MIRBuilder), AArch64CC::VS);
   case TargetOpcode::G_UADDO:
     return std::make_pair(emitADDS(Dst, LHS, RHS, MIRBuilder), AArch64CC::HS);
   case TargetOpcode::G_SSUBO:
     return std::make_pair(emitSUBS(Dst, LHS, RHS, MIRBuilder), AArch64CC::VS);
   case TargetOpcode::G_USUBO:
     return std::make_pair(emitSUBS(Dst, LHS, RHS, MIRBuilder), AArch64CC::LO);
+  case TargetOpcode::G_SADDE:
+    return std::make_pair(emitADCS(Dst, LHS, RHS, MIRBuilder), AArch64CC::VS);
+  case TargetOpcode::G_UADDE:
+    return std::make_pair(emitADCS(Dst, LHS, RHS, MIRBuilder), AArch64CC::HS);
+  case TargetOpcode::G_SSUBE:
+    return std::make_pair(emitSBCS(Dst, LHS, RHS, MIRBuilder), AArch64CC::VS);
+  case TargetOpcode::G_USUBE:
+    return std::make_pair(emitSBCS(Dst, LHS, RHS, MIRBuilder), AArch64CC::LO);
   }
 }
 
 /// Returns true if @p Val is a tree of AND/OR/CMP operations that can be
 /// expressed as a conjunction.
 /// \param CanNegate    Set to true if we can negate the whole sub-tree just by
 ///                     changing the conditions on the CMP tests.
 ///                     (this means we can call emitConjunctionRec() with

llvm/lib/Target/AArch64/GISel/AArch64PostSelectOptimize.cpp

unsigned getNonFlagSettingVariant(unsigned Opc) {
   default:
     return 0;
   case AArch64::SUBSXrr:
     return AArch64::SUBXrr;
   case AArch64::SUBSWrr:
     return AArch64::SUBWrr;
   case AArch64::SUBSXrs:
     return AArch64::SUBXrs;
+  case AArch64::SUBSWrs:
+    return AArch64::SUBWrs;
   case AArch64::SUBSXri:
     return AArch64::SUBXri;
   case AArch64::SUBSWri:
     return AArch64::SUBWri;
+  case AArch64::ADDSXrr:
+    return AArch64::ADDXrr;
+  case AArch64::ADDSWrr:
+    return AArch64::ADDWrr;
+  case AArch64::ADDSXrs:
+    return AArch64::ADDXrs;
+  case AArch64::ADDSWrs:
+    return AArch64::ADDWrs;
+  case AArch64::ADDSXri:
+    return AArch64::ADDXri;
+  case AArch64::ADDSWri:
+    return AArch64::ADDWri;
+  case AArch64::SBCSXr:
+    return AArch64::SBCXr;
+  case AArch64::SBCSWr:
+    return AArch64::SBCWr;
+  case AArch64::ADCSXr:
+    return AArch64::ADCXr;
+  case AArch64::ADCSWr:
+    return AArch64::ADCWr;
   }
 }
 
 bool AArch64PostSelectOptimize::doPeepholeOpts(MachineBasicBlock &MBB) {
   bool Changed = false;
   for (auto &MI : make_early_inc_range(make_range(MBB.begin(), MBB.end()))) {
     Changed |= foldSimpleCrossClassCopies(MI);
   }
bool AArch64PostSelectOptimize::foldSimpleCrossClassCopies(MachineInstr &MI) {
   }
 
   MRI.replaceRegWith(Dst, Src);
   MI.eraseFromParent();
   return true;
 }
 
 bool AArch64PostSelectOptimize::optimizeNZCVDefs(MachineBasicBlock &MBB) {
+  // If we find a dead NZCV implicit-def, we
+  // - try to convert the operation to a non-flag-setting equivalent
+  // - or mark the def as dead to aid later peephole optimizations.
+
+  // Use cases:
+  // 1)
   // Consider the following code:
   //  FCMPSrr %0, %1, implicit-def $nzcv
   //  %sel1:gpr32 = CSELWr %_, %_, 12, implicit $nzcv
   //  %sub:gpr32 = SUBSWrr %_, %_, implicit-def $nzcv
   //  FCMPSrr %0, %1, implicit-def $nzcv
   //  %sel2:gpr32 = CSELWr %_, %_, 12, implicit $nzcv
   // This kind of code where we have 2 FCMPs each feeding a CSEL can happen
   // when we have a single IR fcmp being used by two selects. During selection,
   // to ensure that there can be no clobbering of nzcv between the fcmp and the
   // csel, we have to generate an fcmp immediately before each csel is
   // selected.
   // However, often we can essentially CSE these together later in MachineCSE.
   // This doesn't work though if there are unrelated flag-setting instructions
   // in between the two FCMPs. In this case, the SUBS defines NZCV
   // but it doesn't have any users, being overwritten by the second FCMP.
   //
-  // Our solution here is to try to convert flag setting operations between
-  // a interval of identical FCMPs, so that CSE will be able to eliminate one.
+  // 2)
+  // The instruction selector always emits the flag-setting variant of ADC/SBC
+  // while selecting G_UADDE/G_SADDE/G_USUBE/G_SSUBE. If the carry-out of these
+  // instructions is never used, we can switch to the non-flag-setting variant.
+
   bool Changed = false;
   auto &MF = *MBB.getParent();
   auto &Subtarget = MF.getSubtarget();
   const auto &TII = Subtarget.getInstrInfo();
   auto TRI = Subtarget.getRegisterInfo();
   auto RBI = Subtarget.getRegBankInfo();
   auto &MRI = MF.getRegInfo();
 
-  // The first step is to find the first and last FCMPs. If we have found
-  // at least two, then set the limit of the bottom-up walk to the first FCMP
-  // found since we're only interested in dealing with instructions between
-  // them.
-  MachineInstr *FirstCmp = nullptr, *LastCmp = nullptr;
-  for (auto &MI : instructionsWithoutDebug(MBB.begin(), MBB.end())) {
-    if (MI.getOpcode() == AArch64::FCMPSrr ||
-        MI.getOpcode() == AArch64::FCMPDrr) {
-      if (!FirstCmp)
-        FirstCmp = &MI;
-      else
-        LastCmp = &MI;
-    }
-  }
-
-  // In addition to converting flag-setting ops in fcmp ranges into non-flag
-  // setting ops, across the whole basic block we also detect when nzcv
-  // implicit-defs are dead, and mark them as dead. Peephole optimizations need
-  // this information later.
-
   LiveRegUnits LRU(*MBB.getParent()->getSubtarget().getRegisterInfo());
   LRU.addLiveOuts(MBB);
-  bool NZCVDead = LRU.available(AArch64::NZCV);
-  bool InsideCmpRange = false;
-  for (auto &II : instructionsWithoutDebug(MBB.rbegin(), MBB.rend())) {
-    LRU.stepBackward(II);
-
-    if (LastCmp) { // There's a range present in this block.
-      // If we're inside an fcmp range, look for begin instruction.
-      if (InsideCmpRange && &II == FirstCmp)
-        InsideCmpRange = false;
-      else if (&II == LastCmp)
-        InsideCmpRange = true;
-    }
 
-    // Did this instruction define NZCV?
-    bool NZCVDeadAtCurrInstr = LRU.available(AArch64::NZCV);
-    if (NZCVDead && NZCVDeadAtCurrInstr && II.definesRegister(AArch64::NZCV)) {
-      // If we have a def and NZCV is dead, then we may convert this op.
+  for (auto &II : instructionsWithoutDebug(MBB.rbegin(), MBB.rend())) {
+    bool NZCVDead = LRU.available(AArch64::NZCV);
+    if (NZCVDead && II.definesRegister(AArch64::NZCV)) {
+      // The instruction defines NZCV, but NZCV is dead.
       unsigned NewOpc = getNonFlagSettingVariant(II.getOpcode());
       int DeadNZCVIdx = II.findRegisterDefOperandIdx(AArch64::NZCV);
       if (DeadNZCVIdx != -1) {
-        // If we're inside an fcmp range, then convert flag setting ops.
-        if (InsideCmpRange && NewOpc) {
+        if (NewOpc) {
+          // If there is an equivalent non-flag-setting op, we convert.
           LLVM_DEBUG(dbgs() << "Post-select optimizer: converting flag-setting "
-                               "op in fcmp range: "
+                               "op: "
                             << II);
           II.setDesc(TII->get(NewOpc));
           II.removeOperand(DeadNZCVIdx);
           // Changing the opcode can result in differing regclass requirements,
           // e.g. SUBSWri uses gpr32 for the dest, whereas SUBWri uses gpr32sp.
           // Constrain the regclasses, possibly introducing a copy.
           constrainOperandRegClass(MF, *TRI, MRI, *TII, *RBI, II, II.getDesc(),
                                    II.getOperand(0), 0);
           Changed |= true;
         } else {
           // Otherwise, we just set the nzcv imp-def operand to be dead, so the
           // peephole optimizations can optimize them further.
           II.getOperand(DeadNZCVIdx).setIsDead();
         }
       }
     }
-
-    NZCVDead = NZCVDeadAtCurrInstr;
+    LRU.stepBackward(II);
   }
   return Changed;
 }
 
 bool AArch64PostSelectOptimize::runOnMachineFunction(MachineFunction &MF) {
   if (MF.getProperties().hasProperty(
           MachineFunctionProperties::Property::FailedISel))
     return false;

llvm/test/CodeGen/AArch64/Atomics/aarch64-atomicrmw-lse2.ll

 ; -O1:    add x9, x0, x1
 ; -O1:    stlxr w10, x9, [x8]
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_monotonic:
-; -O0:    adds x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_monotonic:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acquire:
-; -O0:    adds x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acquire:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_release:
-; -O0:    adds x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_release:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acq_rel:
-; -O0:    adds x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acq_rel:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_seq_cst:
-; -O0:    adds x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_seq_cst:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 16
     ret i128 %r
 }

 ; -O1:    add x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_monotonic:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acquire:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_release:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

 ; -O1:    sub x9, x0, x1
 ; -O1:    stlxr w10, x9, [x8]
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_monotonic:
-; -O0:    subs x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_monotonic:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acquire:
-; -O0:    subs x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acquire:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_release:
-; -O0:    subs x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_release:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
-; -O0:    subs x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
-; -O0:    subs x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 16
     ret i128 %r
 }

 ; -O1:    sub x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acquire:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_release:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

llvm/test/CodeGen/AArch64/Atomics/aarch64-atomicrmw-lse2_lse128.ll

 ; CHECK-LABEL: atomicrmw_add_i64_aligned_seq_cst:
 ; CHECK:    ldaddal x1, x0, [x0]
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_monotonic:
-; -O0:    adds x2, x10, x12
+; -O0:    adds x2, x9, x11
+; -O0:    and w11, w9, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    casp x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_monotonic:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    adds x8, x4, x2
 ; -O1:    casp x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acquire:
-; -O0:    adds x2, x10, x12
+; -O0:    adds x2, x9, x11
+; -O0:    and w11, w9, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    caspa x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acquire:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    adds x8, x4, x2
 ; -O1:    caspa x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_release:
-; -O0:    adds x2, x10, x12
+; -O0:    adds x2, x9, x11
+; -O0:    and w11, w9, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    caspl x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_release:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    adds x8, x4, x2
 ; -O1:    caspl x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw add ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acq_rel:
-; -O0:    adds x2, x10, x12
+; -O0:    adds x2, x9, x11
+; -O0:    and w11, w9, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    caspal x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acq_rel:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    adds x8, x4, x2
 ; -O1:    caspal x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_seq_cst:
-; -O0:    adds x2, x10, x12
+; -O0:    adds x2, x9, x11
+; -O0:    and w11, w9, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    caspal x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_seq_cst:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    adds x8, x4, x2
 ; -O1:    caspal x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 16

 ; -O1:    add x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_monotonic:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acquire:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_release:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

 ; CHECK-LABEL: atomicrmw_sub_i64_aligned_seq_cst:
 ; CHECK:    ldaddal x8, x0, [x0]
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_monotonic:
-; -O0:    subs x2, x10, x12
+; -O0:    subs x2, x9, x11
+; -O0:    and w11, w9, #0x1
 ; -O0:    casp x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_monotonic:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    subs x8, x4, x2
 ; -O1:    casp x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acquire:
-; -O0:    subs x2, x10, x12
+; -O0:    subs x2, x9, x11
+; -O0:    and w11, w9, #0x1
 ; -O0:    caspa x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acquire:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    subs x8, x4, x2
 ; -O1:    caspa x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_release:
-; -O0:    subs x2, x10, x12
+; -O0:    subs x2, x9, x11
+; -O0:    and w11, w9, #0x1
 ; -O0:    caspl x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_release:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    subs x8, x4, x2
 ; -O1:    caspl x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
-; -O0:    subs x2, x10, x12
+; -O0:    subs x2, x9, x11
+; -O0:    and w11, w9, #0x1
 ; -O0:    caspal x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    subs x8, x4, x2
 ; -O1:    caspal x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
-; -O0:    subs x2, x10, x12
+; -O0:    subs x2, x9, x11
+; -O0:    and w11, w9, #0x1
 ; -O0:    caspal x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    subs x8, x4, x2
 ; -O1:    caspal x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 16

 ; -O1:    sub x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acquire:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_release:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

llvm/test/CodeGen/AArch64/Atomics/aarch64-atomicrmw-outline_atomics.ll

 ; CHECK-LABEL: atomicrmw_add_i64_aligned_seq_cst:
 ; CHECK:    bl __aarch64_ldadd8_acq_rel
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_monotonic:
-; -O0:    adds x2, x0, x9
-; -O0:    bl __aarch64_cas16_relax
-; -O0:    subs x10, x10, x11
-; -O0:    ccmp x8, x9, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_monotonic:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acquire:
-; -O0:    adds x2, x0, x9
-; -O0:    bl __aarch64_cas16_acq
-; -O0:    subs x10, x10, x11
-; -O0:    ccmp x8, x9, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acquire:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_release:
-; -O0:    adds x2, x0, x9
-; -O0:    bl __aarch64_cas16_rel
-; -O0:    subs x10, x10, x11
-; -O0:    ccmp x8, x9, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_release:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acq_rel:
-; -O0:    adds x2, x0, x9
-; -O0:    bl __aarch64_cas16_acq_rel
-; -O0:    subs x10, x10, x11
-; -O0:    ccmp x8, x9, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acq_rel:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_seq_cst:
-; -O0:    adds x2, x0, x9
-; -O0:    bl __aarch64_cas16_acq_rel
-; -O0:    subs x10, x10, x11
-; -O0:    ccmp x8, x9, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_seq_cst:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 16
     ret i128 %r
 }

 ; -O1:    add x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_monotonic:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acquire:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_release:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

 ; -O1-LABEL: atomicrmw_sub_i64_aligned_seq_cst:
 ; -O1:    bl __aarch64_ldadd8_acq_rel
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_monotonic:
-; -O0:    subs x2, x0, x9
-; -O0:    bl __aarch64_cas16_relax
-; -O0:    subs x10, x10, x11
-; -O0:    ccmp x8, x9, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_monotonic:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acquire:
-; -O0:    subs x2, x0, x9
-; -O0:    bl __aarch64_cas16_acq
-; -O0:    subs x10, x10, x11
-; -O0:    ccmp x8, x9, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acquire:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_release:
-; -O0:    subs x2, x0, x9
-; -O0:    bl __aarch64_cas16_rel
-; -O0:    subs x10, x10, x11
-; -O0:    ccmp x8, x9, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_release:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
-; -O0:    subs x2, x0, x9
-; -O0:    bl __aarch64_cas16_acq_rel
-; -O0:    subs x10, x10, x11
-; -O0:    ccmp x8, x9, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
-; -O0:    subs x2, x0, x9
-; -O0:    bl __aarch64_cas16_acq_rel
-; -O0:    subs x10, x10, x11
-; -O0:    ccmp x8, x9, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 16
     ret i128 %r
 }

 ; -O1:    sub x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acquire:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_release:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

llvm/test/CodeGen/AArch64/Atomics/aarch64-atomicrmw-rcpc.ll

 ; -O1:    add x9, x0, x1
 ; -O1:    stlxr w10, x9, [x8]
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_monotonic:
-; -O0:    adds x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_monotonic:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acquire:
-; -O0:    adds x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acquire:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_release:
-; -O0:    adds x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_release:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acq_rel:
-; -O0:    adds x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acq_rel:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_seq_cst:
-; -O0:    adds x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_seq_cst:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 16
     ret i128 %r
 }

 ; -O1:    add x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_monotonic:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acquire:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_release:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

 ; -O1:    sub x9, x0, x1
 ; -O1:    stlxr w10, x9, [x8]
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_monotonic:
-; -O0:    subs x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_monotonic:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acquire:
-; -O0:    subs x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acquire:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_release:
-; -O0:    subs x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_release:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
-; -O0:    subs x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
-; -O0:    subs x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 16
     ret i128 %r
 }

 ; -O1:    sub x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acquire:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_release:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

llvm/test/CodeGen/AArch64/Atomics/aarch64-atomicrmw-rcpc3.ll

 ; -O1:    add x9, x0, x1
 ; -O1:    stlxr w10, x9, [x8]
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_monotonic:
-; -O0:    adds x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_monotonic:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acquire:
-; -O0:    adds x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acquire:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_release:
-; -O0:    adds x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_release:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acq_rel:
-; -O0:    adds x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acq_rel:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_seq_cst:
-; -O0:    adds x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_seq_cst:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 16
     ret i128 %r
 }

 ; -O1:    add x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_monotonic:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acquire:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_release:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

 ; -O1:    sub x9, x0, x1
 ; -O1:    stlxr w10, x9, [x8]
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_monotonic:
-; -O0:    subs x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_monotonic:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acquire:
-; -O0:    subs x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acquire:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_release:
-; -O0:    subs x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_release:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
-; -O0:    subs x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
-; -O0:    subs x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 16
     ret i128 %r
 }

 ; -O1:    sub x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acquire:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_release:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

llvm/test/CodeGen/AArch64/Atomics/aarch64-atomicrmw-v8_1a.ll

 ; CHECK-LABEL: atomicrmw_add_i64_aligned_seq_cst:
 ; CHECK:    ldaddal x1, x0, [x0]
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_monotonic:
-; -O0:    adds x2, x10, x12
+; -O0:    adds x2, x9, x11
+; -O0:    and w11, w9, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    casp x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_monotonic:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    adds x8, x4, x2
 ; -O1:    casp x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acquire:
-; -O0:    adds x2, x10, x12
+; -O0:    adds x2, x9, x11
+; -O0:    and w11, w9, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    caspa x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acquire:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    adds x8, x4, x2
 ; -O1:    caspa x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_release:
-; -O0:    adds x2, x10, x12
+; -O0:    adds x2, x9, x11
+; -O0:    and w11, w9, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    caspl x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_release:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    adds x8, x4, x2
 ; -O1:    caspl x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw add ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acq_rel:
-; -O0:    adds x2, x10, x12
+; -O0:    adds x2, x9, x11
+; -O0:    and w11, w9, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    caspal x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acq_rel:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    adds x8, x4, x2
 ; -O1:    caspal x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_seq_cst:
-; -O0:    adds x2, x10, x12
+; -O0:    adds x2, x9, x11
+; -O0:    and w11, w9, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    caspal x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_seq_cst:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    adds x8, x4, x2
 ; -O1:    caspal x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 16

 ; -O1:    add x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_monotonic:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acquire:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_release:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

 ; CHECK-LABEL: atomicrmw_sub_i64_aligned_seq_cst:
 ; CHECK:    ldaddal x8, x0, [x0]
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_monotonic:
-; -O0:    subs x2, x10, x12
+; -O0:    subs x2, x9, x11
+; -O0:    and w11, w9, #0x1
 ; -O0:    casp x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_monotonic:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    subs x8, x4, x2
 ; -O1:    casp x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acquire:
-; -O0:    subs x2, x10, x12
+; -O0:    subs x2, x9, x11
+; -O0:    and w11, w9, #0x1
 ; -O0:    caspa x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acquire:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    subs x8, x4, x2
 ; -O1:    caspa x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_release:
-; -O0:    subs x2, x10, x12
+; -O0:    subs x2, x9, x11
+; -O0:    and w11, w9, #0x1
 ; -O0:    caspl x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_release:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    subs x8, x4, x2
 ; -O1:    caspl x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
-; -O0:    subs x2, x10, x12
+; -O0:    subs x2, x9, x11
+; -O0:    and w11, w9, #0x1
 ; -O0:    caspal x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    subs x8, x4, x2
 ; -O1:    caspal x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
-; -O0:    subs x2, x10, x12
+; -O0:    subs x2, x9, x11
+; -O0:    and w11, w9, #0x1
 ; -O0:    caspal x0, x1, x2, x3, [x8]
-; -O0:    subs x11, x8, x11
-; -O0:    ccmp x9, x10, #0, eq
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
 ; -O1:    ldp x4, x5, [x0]
 ; -O1:    subs x8, x4, x2
 ; -O1:    caspal x4, x5, x8, x9, [x0]
 ; -O1:    cmp x5, x7
 ; -O1:    ccmp x4, x6, #0, eq
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 16

 ; -O1:    sub x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acquire:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_release:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

llvm/test/CodeGen/AArch64/Atomics/aarch64-atomicrmw-v8a.ll

 ; -O1:    add x9, x0, x1
 ; -O1:    stlxr w10, x9, [x8]
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_monotonic:
-; -O0:    adds x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_monotonic:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acquire:
-; -O0:    adds x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acquire:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_release:
-; -O0:    adds x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_release:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_acq_rel:
-; -O0:    adds x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_acq_rel:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_aligned_seq_cst:
-; -O0:    adds x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    adds x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    subs w10, w10, #1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_add_i128_aligned_seq_cst:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    adds x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 16
     ret i128 %r
 }

 ; -O1:    add x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_monotonic:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acquire:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_release:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_add_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
-; -O0:    adds x8, x11, x8
+; -O0:    adds x9, x8, x9
+; -O0:    and w11, w8, #0x1
+; -O0:    subs w11, w11, #1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_add_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    adds x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw add ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

 ; -O1:    sub x9, x0, x1
 ; -O1:    stlxr w10, x9, [x8]
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 8
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_monotonic:
-; -O0:    subs x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_monotonic:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acquire:
-; -O0:    subs x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stxp w8, x14, x15, [x9]
-; -O0:    stxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stxp w8, x14, x15, [x11]
+; -O0:    stxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acquire:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_release:
-; -O0:    subs x14, x11, x10
-; -O0:    ldxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_release:
 ; -O1:    ldxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
-; -O0:    subs x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_acq_rel:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 16
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_aligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
-; -O0:    subs x14, x11, x10
-; -O0:    ldaxp x10, x12, [x9]
-; -O0:    cmp x10, x11
-; -O0:    cmp x12, x13
-; -O0:    stlxp w8, x14, x15, [x9]
-; -O0:    stlxp w8, x10, x12, [x9]
-; -O0:    subs x12, x12, x13
-; -O0:    ccmp x10, x11, #0, eq
+; -O0:    subs x14, x8, x10
+; -O0:    and w10, w8, #0x1
+; -O0:    ldaxp x10, x9, [x11]
+; -O0:    cmp x10, x12
+; -O0:    cmp x9, x13
+; -O0:    stlxp w8, x14, x15, [x11]
+; -O0:    stlxp w8, x10, x9, [x11]
+; -O0:    eor x8, x10, x8
+; -O0:    eor x11, x9, x11
+; -O0:    orr x8, x8, x11
+; -O0:    subs x8, x8, #0
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_aligned_seq_cst:
 ; -O1:    ldaxp x0, x1, [x8]
 ; -O1:    subs x9, x0, x2
 ; -O1:    stlxp w11, x9, x10, [x8]
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 16
     ret i128 %r
 }

 ; -O1:    sub x8, x0, x19
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i64 %value seq_cst, align 1
     ret i64 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_monotonic(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_monotonic:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value monotonic, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acquire(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acquire:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acquire:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acquire, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_release(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_release:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_release:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value release, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_acq_rel(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_acq_rel:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value acq_rel, align 1
     ret i128 %r
 }
 
 define dso_local i128 @atomicrmw_sub_i128_unaligned_seq_cst(ptr %ptr, i128 %value) {
 ; -O0-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
-; -O0:    subs x8, x11, x8
+; -O0:    subs x9, x8, x9
+; -O0:    and w11, w8, #0x1
 ; -O0:    bl __atomic_compare_exchange
 ;
 ; -O1-LABEL: atomicrmw_sub_i128_unaligned_seq_cst:
 ; -O1:    ldp x0, x1, [x0]
 ; -O1:    subs x8, x0, x20
 ; -O1:    bl __atomic_compare_exchange
     %r = atomicrmw sub ptr %ptr, i128 %value seq_cst, align 1
     ret i128 %r

llvm/test/CodeGen/AArch64/GlobalISel/postselectopt-dead-cc-defs-in-fcmp.mir

liveins:
   - { reg: '$x0' }
   - { reg: '$w1' }
 body:             |
   bb.1:
     liveins: $w1, $x0, $s0, $s1
 
     ; CHECK-LABEL: name: test_fcmp_dead_cc
     ; CHECK: liveins: $w1, $x0, $s0, $s1
-    ; CHECK: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
-    ; CHECK: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
-    ; CHECK: [[COPY2:%[0-9]+]]:fpr32 = COPY $s0
-    ; CHECK: [[COPY3:%[0-9]+]]:fpr32 = COPY $s1
-    ; CHECK: [[COPY4:%[0-9]+]]:gpr32 = COPY $wzr
-    ; CHECK: FCMPSrr [[COPY2]], [[COPY3]], implicit-def dead $nzcv, implicit $fpcr
-    ; CHECK: [[SUBWrr:%[0-9]+]]:gpr32 = SUBWrr [[COPY1]], [[COPY4]]
-    ; CHECK: FCMPSrr [[COPY2]], [[COPY3]], implicit-def $nzcv, implicit $fpcr
-    ; CHECK: [[UBFMWri:%[0-9]+]]:gpr32common = UBFMWri [[SUBWrr]], 1, 31
-    ; CHECK: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
-    ; CHECK: [[CSELWr:%[0-9]+]]:gpr32 = CSELWr [[UBFMWri]], [[MOVi32imm]], 8, implicit $nzcv
-    ; CHECK: $w0 = COPY [[CSELWr]]
-    ; CHECK: RET_ReallyLR implicit $w0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:fpr32 = COPY $s0
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:fpr32 = COPY $s1
+    ; CHECK-NEXT: [[COPY4:%[0-9]+]]:gpr32 = COPY $wzr
+    ; CHECK-NEXT: FCMPSrr [[COPY2]], [[COPY3]], implicit-def dead $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[SUBWrr:%[0-9]+]]:gpr32 = SUBWrr [[COPY1]], [[COPY4]]
+    ; CHECK-NEXT: FCMPSrr [[COPY2]], [[COPY3]], implicit-def $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[UBFMWri:%[0-9]+]]:gpr32common = UBFMWri [[SUBWrr]], 1, 31
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
+    ; CHECK-NEXT: [[CSELWr:%[0-9]+]]:gpr32 = CSELWr [[UBFMWri]], [[MOVi32imm]], 8, implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[CSELWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %1:gpr64 = COPY $x0
     %2:gpr32 = COPY $w1
     %3:fpr32 = COPY $s0
     %4:fpr32 = COPY $s1
     %26:gpr32 = COPY $wzr
     FCMPSrr %3, %4, implicit-def $nzcv, implicit $fpcr
     %12:gpr32 = SUBSWrr %2, %26, implicit-def $nzcv
     FCMPSrr %3, %4, implicit-def $nzcv, implicit $fpcr
liveins:
   - { reg: '$x0' }
   - { reg: '$w1' }
 body:             |
   bb.1:
     liveins: $w1, $x0, $d0, $d1
 
     ; CHECK-LABEL: name: test_fcmp_64_dead_cc
     ; CHECK: liveins: $w1, $x0, $d0, $d1
-    ; CHECK: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
-    ; CHECK: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
-    ; CHECK: [[COPY2:%[0-9]+]]:fpr64 = COPY $d0
-    ; CHECK: [[COPY3:%[0-9]+]]:fpr64 = COPY $d1
-    ; CHECK: [[COPY4:%[0-9]+]]:gpr32 = COPY $wzr
-    ; CHECK: FCMPDrr [[COPY2]], [[COPY3]], implicit-def dead $nzcv, implicit $fpcr
-    ; CHECK: [[SUBWrr:%[0-9]+]]:gpr32 = SUBWrr [[COPY1]], [[COPY4]]
-    ; CHECK: FCMPDrr [[COPY2]], [[COPY3]], implicit-def $nzcv, implicit $fpcr
-    ; CHECK: [[UBFMWri:%[0-9]+]]:gpr32common = UBFMWri [[SUBWrr]], 1, 31
-    ; CHECK: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
-    ; CHECK: [[CSELWr:%[0-9]+]]:gpr32 = CSELWr [[UBFMWri]], [[MOVi32imm]], 8, implicit $nzcv
-    ; CHECK: $w0 = COPY [[CSELWr]]
-    ; CHECK: RET_ReallyLR implicit $w0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:fpr64 = COPY $d0
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:fpr64 = COPY $d1
+    ; CHECK-NEXT: [[COPY4:%[0-9]+]]:gpr32 = COPY $wzr
+    ; CHECK-NEXT: FCMPDrr [[COPY2]], [[COPY3]], implicit-def dead $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[SUBWrr:%[0-9]+]]:gpr32 = SUBWrr [[COPY1]], [[COPY4]]
+    ; CHECK-NEXT: FCMPDrr [[COPY2]], [[COPY3]], implicit-def $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[UBFMWri:%[0-9]+]]:gpr32common = UBFMWri [[SUBWrr]], 1, 31
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
+    ; CHECK-NEXT: [[CSELWr:%[0-9]+]]:gpr32 = CSELWr [[UBFMWri]], [[MOVi32imm]], 8, implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[CSELWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %1:gpr64 = COPY $x0
     %2:gpr32 = COPY $w1
     %3:fpr64 = COPY $d0
     %4:fpr64 = COPY $d1
     %26:gpr32 = COPY $wzr
     FCMPDrr %3, %4, implicit-def $nzcv, implicit $fpcr
     %12:gpr32 = SUBSWrr %2, %26, implicit-def $nzcv
     FCMPDrr %3, %4, implicit-def $nzcv, implicit $fpcr
liveins:
   - { reg: '$x0' }
   - { reg: '$w1' }
 body:             |
   bb.1:
     liveins: $w1, $x0, $s0, $s1
 
     ; CHECK-LABEL: name: test_fcmp_dead_cc_3_fcmps
     ; CHECK: liveins: $w1, $x0, $s0, $s1
-    ; CHECK: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
-    ; CHECK: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
-    ; CHECK: [[COPY2:%[0-9]+]]:fpr32 = COPY $s0
-    ; CHECK: [[COPY3:%[0-9]+]]:fpr32 = COPY $s1
-    ; CHECK: [[COPY4:%[0-9]+]]:gpr32 = COPY $wzr
-    ; CHECK: FCMPSrr [[COPY2]], [[COPY3]], implicit-def dead $nzcv, implicit $fpcr
-    ; CHECK: [[SUBWrr:%[0-9]+]]:gpr32 = SUBWrr [[COPY1]], [[COPY4]]
-    ; CHECK: FCMPSrr [[COPY2]], [[COPY3]], implicit-def dead $nzcv, implicit $fpcr
-    ; CHECK: [[SUBWrr1:%[0-9]+]]:gpr32 = SUBWrr [[COPY1]], [[COPY4]]
-    ; CHECK: FCMPSrr [[COPY2]], [[COPY3]], implicit-def $nzcv, implicit $fpcr
-    ; CHECK: [[UBFMWri:%[0-9]+]]:gpr32common = UBFMWri [[SUBWrr1]], 1, 31
-    ; CHECK: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
-    ; CHECK: [[CSELWr:%[0-9]+]]:gpr32 = CSELWr [[UBFMWri]], [[MOVi32imm]], 8, implicit $nzcv
-    ; CHECK: $w0 = COPY [[CSELWr]]
-    ; CHECK: RET_ReallyLR implicit $w0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:fpr32 = COPY $s0
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:fpr32 = COPY $s1
+    ; CHECK-NEXT: [[COPY4:%[0-9]+]]:gpr32 = COPY $wzr
+    ; CHECK-NEXT: FCMPSrr [[COPY2]], [[COPY3]], implicit-def dead $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[SUBWrr:%[0-9]+]]:gpr32 = SUBWrr [[COPY1]], [[COPY4]]
+    ; CHECK-NEXT: FCMPSrr [[COPY2]], [[COPY3]], implicit-def dead $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[SUBWrr1:%[0-9]+]]:gpr32 = SUBWrr [[COPY1]], [[COPY4]]
+    ; CHECK-NEXT: FCMPSrr [[COPY2]], [[COPY3]], implicit-def $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[UBFMWri:%[0-9]+]]:gpr32common = UBFMWri [[SUBWrr1]], 1, 31
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
+    ; CHECK-NEXT: [[CSELWr:%[0-9]+]]:gpr32 = CSELWr [[UBFMWri]], [[MOVi32imm]], 8, implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[CSELWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %1:gpr64 = COPY $x0
     %2:gpr32 = COPY $w1
     %3:fpr32 = COPY $s0
     %4:fpr32 = COPY $s1
     %26:gpr32 = COPY $wzr
     FCMPSrr %3, %4, implicit-def $nzcv, implicit $fpcr
     %12:gpr32 = SUBSWrr %2, %26, implicit-def $nzcv
     FCMPSrr %3, %4, implicit-def $nzcv, implicit $fpcr
     %12:gpr32 = SUBSWrr %2, %26, implicit-def $nzcv
     FCMPSrr %3, %4, implicit-def $nzcv, implicit $fpcr
     %14:gpr32common = UBFMWri %12, 1, 31
     %60:gpr32 = MOVi32imm 1
     %16:gpr32 = CSELWr %14, %60, 8, implicit $nzcv
     $w0 = COPY %16
     RET_ReallyLR implicit $w0
 
 ...
 ---
-name:            test_impdef_made_dead
+name:            test_impdef_subsx
 alignment:       4
 legalized:       true
 regBankSelected: true
 selected:        true
 tracksRegLiveness: true
-liveins:
-  - { reg: '$x0' }
-  - { reg: '$w1' }
 body:             |
   bb.1:
-    liveins: $w1, $x0, $s0, $s1
-    ; Check that any dead imp-defs of NZCV are marked as such.
-    ; CHECK-LABEL: name: test_impdef_made_dead
-    ; CHECK: liveins: $w1, $x0, $s0, $s1
-    ; CHECK: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
-    ; CHECK: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
-    ; CHECK: [[COPY2:%[0-9]+]]:fpr32 = COPY $s0
-    ; CHECK: [[COPY3:%[0-9]+]]:fpr32 = COPY $s1
-    ; CHECK: [[COPY4:%[0-9]+]]:gpr32 = COPY $wzr
-    ; CHECK: [[SUBSWrr:%[0-9]+]]:gpr32 = SUBSWrr [[COPY1]], [[COPY4]], implicit-def dead $nzcv
-    ; CHECK: FCMPSrr [[COPY2]], [[COPY3]], implicit-def $nzcv, implicit $fpcr
-    ; CHECK: [[UBFMWri:%[0-9]+]]:gpr32common = UBFMWri [[SUBSWrr]], 1, 31
-    ; CHECK: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
-    ; CHECK: [[CSELWr:%[0-9]+]]:gpr32 = CSELWr [[UBFMWri]], [[MOVi32imm]], 8, implicit $nzcv
-    ; CHECK: %ret:gpr32 = SUBSWrr [[CSELWr]], [[SUBSWrr]], implicit-def dead $nzcv
-    ; CHECK: $w0 = COPY [[CSELWr]]
-    ; CHECK: RET_ReallyLR implicit $w0
+    liveins: $x0, $x1
+    ; CHECK-LABEL: name: test_impdef_subsx
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[SUBXrr:%[0-9]+]]:gpr64 = SUBXrr [[COPY]], [[COPY1]]
+    ; CHECK-NEXT: $x0 = COPY [[SUBXrr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0
     %1:gpr64 = COPY $x0
-    %2:gpr32 = COPY $w1
-    %3:fpr32 = COPY $s0
-    %4:fpr32 = COPY $s1
-    %26:gpr32 = COPY $wzr
-    %12:gpr32 = SUBSWrr %2, %26, implicit-def $nzcv
-    FCMPSrr %3, %4, implicit-def $nzcv, implicit $fpcr
-    %14:gpr32common = UBFMWri %12, 1, 31
-    %60:gpr32 = MOVi32imm 1
-    %16:gpr32 = CSELWr %14, %60, 8, implicit $nzcv
-    %ret:gpr32 = SUBSWrr %16, %12, implicit-def $nzcv
-    $w0 = COPY %16
+    %2:gpr64 = COPY $x1
+    %4:gpr64 = SUBSXrr %1, %2, implicit-def $nzcv
+    $x0 = COPY %4
+    RET_ReallyLR implicit $x0
+...
+---
+name:            test_impdef_subsw
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $w0, $w1
+    ; CHECK-LABEL: name: test_impdef_subsw
+    ; CHECK: liveins: $w0, $w1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $x1
+    ; CHECK-NEXT: [[SUBWrr:%[0-9]+]]:gpr32 = SUBWrr [[COPY]], [[COPY1]]
+    ; CHECK-NEXT: $w0 = COPY [[SUBWrr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
+    %1:gpr32 = COPY $x0
+    %2:gpr32 = COPY $x1
+    %4:gpr32 = SUBSWrr %1, %2, implicit-def $nzcv
+    $w0 = COPY %4
+    RET_ReallyLR implicit $w0
+...
+---
+name:            test_impdef_addsx
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1
+    ; CHECK-LABEL: name: test_impdef_addsx
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[ADDXrr:%[0-9]+]]:gpr64 = ADDXrr [[COPY]], [[COPY1]]
+    ; CHECK-NEXT: $x0 = COPY [[ADDXrr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0
+    %1:gpr64 = COPY $x0
+    %2:gpr64 = COPY $x1
+    %4:gpr64 = ADDSXrr %1, %2, implicit-def $nzcv
+    $x0 = COPY %4
+    RET_ReallyLR implicit $x0
+...
+---
+name:            test_impdef_addsw
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $w0, $w1
+    ; CHECK-LABEL: name: test_impdef_addsw
+    ; CHECK: liveins: $w0, $w1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $x1
+    ; CHECK-NEXT: [[ADDWrr:%[0-9]+]]:gpr32 = ADDWrr [[COPY]], [[COPY1]]
+    ; CHECK-NEXT: $w0 = COPY [[ADDWrr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
+    %1:gpr32 = COPY $x0
+    %2:gpr32 = COPY $x1
+    %4:gpr32 = ADDSWrr %1, %2, implicit-def $nzcv
+    $w0 = COPY %4
     RET_ReallyLR implicit $w0
 ...
+---
+name:            test_impdef_adcsx
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+    ; CHECK-LABEL: name: test_impdef_adcsx
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x3
+    ; CHECK-NEXT: [[ADDSXrr:%[0-9]+]]:gpr64 = ADDSXrr [[COPY]], [[COPY2]], implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCXr:%[0-9]+]]:gpr64 = ADCXr [[COPY1]], [[COPY3]], implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[ADDSXrr]]
+    ; CHECK-NEXT: $x1 = COPY [[ADCXr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1
+    %1:gpr64 = COPY $x0
+    %2:gpr64 = COPY $x1
+    %3:gpr64 = COPY $x2
+    %4:gpr64 = COPY $x3
+    %5:gpr64 = ADDSXrr %1, %3, implicit-def $nzcv
+    %6:gpr64 = ADCSXr %2, %4, implicit-def $nzcv, implicit $nzcv
+    $x0 = COPY %5
+    $x1 = COPY %6
+    RET_ReallyLR implicit $x0, implicit $x1
+...
+---
+name:            test_impdef_adcsw
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $w0, $w1, $w2, $w3
+    ; CHECK-LABEL: name: test_impdef_adcsw
+    ; CHECK: liveins: $w0, $w1, $w2, $w3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $w0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr32 = COPY $w2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr32 = COPY $w3
+    ; CHECK-NEXT: [[ADDSWrr:%[0-9]+]]:gpr32 = ADDSWrr [[COPY]], [[COPY2]], implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCWr:%[0-9]+]]:gpr32 = ADCWr [[COPY1]], [[COPY3]], implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[ADDSWrr]]
+    ; CHECK-NEXT: $w1 = COPY [[ADCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0, implicit $w1
+    %1:gpr32 = COPY $w0
+    %2:gpr32 = COPY $w1
+    %3:gpr32 = COPY $w2
+    %4:gpr32 = COPY $w3
+    %5:gpr32 = ADDSWrr %1, %3, implicit-def $nzcv
+    %6:gpr32 = ADCSWr %2, %4, implicit-def $nzcv, implicit $nzcv
+    $w0 = COPY %5
+    $w1 = COPY %6
+    RET_ReallyLR implicit $w0, implicit $w1
+...
+---
+name:            test_impdef_sbcsx
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+    ; CHECK-LABEL: name: test_impdef_sbcsx
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x3
+    ; CHECK-NEXT: [[SUBSXrr:%[0-9]+]]:gpr64 = SUBSXrr [[COPY]], [[COPY2]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCXr:%[0-9]+]]:gpr64 = SBCXr [[COPY1]], [[COPY3]], implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[SUBSXrr]]
+    ; CHECK-NEXT: $x1 = COPY [[SBCXr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1
+    %1:gpr64 = COPY $x0
+    %2:gpr64 = COPY $x1
+    %3:gpr64 = COPY $x2
+    %4:gpr64 = COPY $x3
+    %5:gpr64 = SUBSXrr %1, %3, implicit-def $nzcv
+    %6:gpr64 = SBCSXr %2, %4, implicit-def $nzcv, implicit $nzcv
+    $x0 = COPY %5
+    $x1 = COPY %6
+    RET_ReallyLR implicit $x0, implicit $x1
+...
+---
+name:            test_impdef_sbcsw
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $w0, $w1, $w2, $w3
+    ; CHECK-LABEL: name: test_impdef_sbcsw
+    ; CHECK: liveins: $w0, $w1, $w2, $w3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $w0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr32 = COPY $w2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr32 = COPY $w3
+    ; CHECK-NEXT: [[SUBSWrr:%[0-9]+]]:gpr32 = SUBSWrr [[COPY]], [[COPY2]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCWr:%[0-9]+]]:gpr32 = SBCWr [[COPY1]], [[COPY3]], implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[SUBSWrr]]
+    ; CHECK-NEXT: $w1 = COPY [[SBCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0, implicit $w1
+    %1:gpr32 = COPY $w0
+    %2:gpr32 = COPY $w1
+    %3:gpr32 = COPY $w2
+    %4:gpr32 = COPY $w3
+    %5:gpr32 = SUBSWrr %1, %3, implicit-def $nzcv
+    %6:gpr32 = SBCSWr %2, %4, implicit-def $nzcv, implicit $nzcv
+    $w0 = COPY %5
+    $w1 = COPY %6
+    RET_ReallyLR implicit $w0, implicit $w1
+...

llvm/test/CodeGen/AArch64/GlobalISel/postselectopt-dead-cc-defs.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple aarch64 -run-pass=aarch64-post-select-optimize -verify-machineinstrs %s -o - | FileCheck %s
+---
+name:            test_fcmp_dead_cc
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$w1' }
+body:             |
+  bb.1:
+    liveins: $w1, $x0, $s0, $s1
+
+    ; CHECK-LABEL: name: test_fcmp_dead_cc
+    ; CHECK: liveins: $w1, $x0, $s0, $s1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:fpr32 = COPY $s0
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:fpr32 = COPY $s1
+    ; CHECK-NEXT: [[COPY4:%[0-9]+]]:gpr32 = COPY $wzr
+    ; CHECK-NEXT: FCMPSrr [[COPY2]], [[COPY3]], implicit-def dead $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[SUBWrr:%[0-9]+]]:gpr32 = SUBWrr [[COPY1]], [[COPY4]]
+    ; CHECK-NEXT: FCMPSrr [[COPY2]], [[COPY3]], implicit-def $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[UBFMWri:%[0-9]+]]:gpr32common = UBFMWri [[SUBWrr]], 1, 31
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
+    ; CHECK-NEXT: [[CSELWr:%[0-9]+]]:gpr32 = CSELWr [[UBFMWri]], [[MOVi32imm]], 8, implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[CSELWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
+    %1:gpr64 = COPY $x0
+    %2:gpr32 = COPY $w1
+    %3:fpr32 = COPY $s0
+    %4:fpr32 = COPY $s1
+    %26:gpr32 = COPY $wzr
+    FCMPSrr %3, %4, implicit-def $nzcv, implicit $fpcr
+    %12:gpr32 = SUBSWrr %2, %26, implicit-def $nzcv
+    FCMPSrr %3, %4, implicit-def $nzcv, implicit $fpcr
+    %14:gpr32common = UBFMWri %12, 1, 31
+    %60:gpr32 = MOVi32imm 1
+    %16:gpr32 = CSELWr %14, %60, 8, implicit $nzcv
+    $w0 = COPY %16
+    RET_ReallyLR implicit $w0
+
+...
+---
+name:            test_fcmp_64_dead_cc
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$w1' }
+body:             |
+  bb.1:
+    liveins: $w1, $x0, $d0, $d1
+
+    ; CHECK-LABEL: name: test_fcmp_64_dead_cc
+    ; CHECK: liveins: $w1, $x0, $d0, $d1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:fpr64 = COPY $d0
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:fpr64 = COPY $d1
+    ; CHECK-NEXT: [[COPY4:%[0-9]+]]:gpr32 = COPY $wzr
+    ; CHECK-NEXT: FCMPDrr [[COPY2]], [[COPY3]], implicit-def dead $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[SUBWrr:%[0-9]+]]:gpr32 = SUBWrr [[COPY1]], [[COPY4]]
+    ; CHECK-NEXT: FCMPDrr [[COPY2]], [[COPY3]], implicit-def $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[UBFMWri:%[0-9]+]]:gpr32common = UBFMWri [[SUBWrr]], 1, 31
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
+    ; CHECK-NEXT: [[CSELWr:%[0-9]+]]:gpr32 = CSELWr [[UBFMWri]], [[MOVi32imm]], 8, implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[CSELWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
+    %1:gpr64 = COPY $x0
+    %2:gpr32 = COPY $w1
+    %3:fpr64 = COPY $d0
+    %4:fpr64 = COPY $d1
+    %26:gpr32 = COPY $wzr
+    FCMPDrr %3, %4, implicit-def $nzcv, implicit $fpcr
+    %12:gpr32 = SUBSWrr %2, %26, implicit-def $nzcv
+    FCMPDrr %3, %4, implicit-def $nzcv, implicit $fpcr
+    %14:gpr32common = UBFMWri %12, 1, 31
+    %60:gpr32 = MOVi32imm 1
+    %16:gpr32 = CSELWr %14, %60, 8, implicit $nzcv
+    $w0 = COPY %16
+    RET_ReallyLR implicit $w0
+
+...
+---
+name:            test_fcmp_dead_cc_3_fcmps
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$w1' }
+body:             |
+  bb.1:
+    liveins: $w1, $x0, $s0, $s1
+
+    ; CHECK-LABEL: name: test_fcmp_dead_cc_3_fcmps
+    ; CHECK: liveins: $w1, $x0, $s0, $s1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:fpr32 = COPY $s0
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:fpr32 = COPY $s1
+    ; CHECK-NEXT: [[COPY4:%[0-9]+]]:gpr32 = COPY $wzr
+    ; CHECK-NEXT: FCMPSrr [[COPY2]], [[COPY3]], implicit-def dead $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[SUBWrr:%[0-9]+]]:gpr32 = SUBWrr [[COPY1]], [[COPY4]]
+    ; CHECK-NEXT: FCMPSrr [[COPY2]], [[COPY3]], implicit-def dead $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[SUBWrr1:%[0-9]+]]:gpr32 = SUBWrr [[COPY1]], [[COPY4]]
+    ; CHECK-NEXT: FCMPSrr [[COPY2]], [[COPY3]], implicit-def $nzcv, implicit $fpcr
+    ; CHECK-NEXT: [[UBFMWri:%[0-9]+]]:gpr32common = UBFMWri [[SUBWrr1]], 1, 31
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
+    ; CHECK-NEXT: [[CSELWr:%[0-9]+]]:gpr32 = CSELWr [[UBFMWri]], [[MOVi32imm]], 8, implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[CSELWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
+    %1:gpr64 = COPY $x0
+    %2:gpr32 = COPY $w1
+    %3:fpr32 = COPY $s0
+    %4:fpr32 = COPY $s1
+    %26:gpr32 = COPY $wzr
+    FCMPSrr %3, %4, implicit-def $nzcv, implicit $fpcr
+    %12:gpr32 = SUBSWrr %2, %26, implicit-def $nzcv
+    FCMPSrr %3, %4, implicit-def $nzcv, implicit $fpcr
+    %12:gpr32 = SUBSWrr %2, %26, implicit-def $nzcv
+    FCMPSrr %3, %4, implicit-def $nzcv, implicit $fpcr
+    %14:gpr32common = UBFMWri %12, 1, 31
+    %60:gpr32 = MOVi32imm 1
+    %16:gpr32 = CSELWr %14, %60, 8, implicit $nzcv
+    $w0 = COPY %16
+    RET_ReallyLR implicit $w0
+
+...
+---
+name:            test_impdef_subsx
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1
+    ; CHECK-LABEL: name: test_impdef_subsx
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[SUBXrr:%[0-9]+]]:gpr64 = SUBXrr [[COPY]], [[COPY1]]
+    ; CHECK-NEXT: $x0 = COPY [[SUBXrr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0
+    %1:gpr64 = COPY $x0
+    %2:gpr64 = COPY $x1
+    %4:gpr64 = SUBSXrr %1, %2, implicit-def $nzcv
+    $x0 = COPY %4
+    RET_ReallyLR implicit $x0
+...
+---
+name:            test_impdef_subsw
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $w0, $w1
+    ; CHECK-LABEL: name: test_impdef_subsw
+    ; CHECK: liveins: $w0, $w1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $x1
+    ; CHECK-NEXT: [[SUBWrr:%[0-9]+]]:gpr32 = SUBWrr [[COPY]], [[COPY1]]
+    ; CHECK-NEXT: $w0 = COPY [[SUBWrr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
+    %1:gpr32 = COPY $x0
+    %2:gpr32 = COPY $x1
+    %4:gpr32 = SUBSWrr %1, %2, implicit-def $nzcv
+    $w0 = COPY %4
+    RET_ReallyLR implicit $w0
+...
+---
+name:            test_impdef_addsx
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1
+    ; CHECK-LABEL: name: test_impdef_addsx
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[ADDXrr:%[0-9]+]]:gpr64 = ADDXrr [[COPY]], [[COPY1]]
+    ; CHECK-NEXT: $x0 = COPY [[ADDXrr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0
+    %1:gpr64 = COPY $x0
+    %2:gpr64 = COPY $x1
+    %4:gpr64 = ADDSXrr %1, %2, implicit-def $nzcv
+    $x0 = COPY %4
+    RET_ReallyLR implicit $x0
+...
+---
+name:            test_impdef_addsw
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $w0, $w1
+    ; CHECK-LABEL: name: test_impdef_addsw
+    ; CHECK: liveins: $w0, $w1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $x1
+    ; CHECK-NEXT: [[ADDWrr:%[0-9]+]]:gpr32 = ADDWrr [[COPY]], [[COPY1]]
+    ; CHECK-NEXT: $w0 = COPY [[ADDWrr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
+    %1:gpr32 = COPY $x0
+    %2:gpr32 = COPY $x1
+    %4:gpr32 = ADDSWrr %1, %2, implicit-def $nzcv
+    $w0 = COPY %4
+    RET_ReallyLR implicit $w0
+...
+---
+name:            test_impdef_adcsx
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+    ; CHECK-LABEL: name: test_impdef_adcsx
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x3
+    ; CHECK-NEXT: [[ADDSXrr:%[0-9]+]]:gpr64 = ADDSXrr [[COPY]], [[COPY2]], implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCXr:%[0-9]+]]:gpr64 = ADCXr [[COPY1]], [[COPY3]], implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[ADDSXrr]]
+    ; CHECK-NEXT: $x1 = COPY [[ADCXr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1
+    %1:gpr64 = COPY $x0
+    %2:gpr64 = COPY $x1
+    %3:gpr64 = COPY $x2
+    %4:gpr64 = COPY $x3
+    %5:gpr64 = ADDSXrr %1, %3, implicit-def $nzcv
+    %6:gpr64 = ADCSXr %2, %4, implicit-def $nzcv, implicit $nzcv
+    $x0 = COPY %5
+    $x1 = COPY %6
+    RET_ReallyLR implicit $x0, implicit $x1
+...
+---
+name:            test_impdef_adcsw
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $w0, $w1, $w2, $w3
+    ; CHECK-LABEL: name: test_impdef_adcsw
+    ; CHECK: liveins: $w0, $w1, $w2, $w3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $w0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr32 = COPY $w2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr32 = COPY $w3
+    ; CHECK-NEXT: [[ADDSWrr:%[0-9]+]]:gpr32 = ADDSWrr [[COPY]], [[COPY2]], implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCWr:%[0-9]+]]:gpr32 = ADCWr [[COPY1]], [[COPY3]], implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[ADDSWrr]]
+    ; CHECK-NEXT: $w1 = COPY [[ADCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0, implicit $w1
+    %1:gpr32 = COPY $w0
+    %2:gpr32 = COPY $w1
+    %3:gpr32 = COPY $w2
+    %4:gpr32 = COPY $w3
+    %5:gpr32 = ADDSWrr %1, %3, implicit-def $nzcv
+    %6:gpr32 = ADCSWr %2, %4, implicit-def $nzcv, implicit $nzcv
+    $w0 = COPY %5
+    $w1 = COPY %6
+    RET_ReallyLR implicit $w0, implicit $w1
+...
+---
+name:            test_impdef_sbcsx
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+    ; CHECK-LABEL: name: test_impdef_sbcsx
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x3
+    ; CHECK-NEXT: [[SUBSXrr:%[0-9]+]]:gpr64 = SUBSXrr [[COPY]], [[COPY2]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCXr:%[0-9]+]]:gpr64 = SBCXr [[COPY1]], [[COPY3]], implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[SUBSXrr]]
+    ; CHECK-NEXT: $x1 = COPY [[SBCXr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1
+    %1:gpr64 = COPY $x0
+    %2:gpr64 = COPY $x1
+    %3:gpr64 = COPY $x2
+    %4:gpr64 = COPY $x3
+    %5:gpr64 = SUBSXrr %1, %3, implicit-def $nzcv
+    %6:gpr64 = SBCSXr %2, %4, implicit-def $nzcv, implicit $nzcv
+    $x0 = COPY %5
+    $x1 = COPY %6
+    RET_ReallyLR implicit $x0, implicit $x1
+...
+---
+name:            test_impdef_sbcsw
+alignment:       4
+legalized:       true
+regBankSelected: true
+selected:        true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $w0, $w1, $w2, $w3
+    ; CHECK-LABEL: name: test_impdef_sbcsw
+    ; CHECK: liveins: $w0, $w1, $w2, $w3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $w0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr32 = COPY $w2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr32 = COPY $w3
+    ; CHECK-NEXT: [[SUBSWrr:%[0-9]+]]:gpr32 = SUBSWrr [[COPY]], [[COPY2]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCWr:%[0-9]+]]:gpr32 = SBCWr [[COPY1]], [[COPY3]], implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[SUBSWrr]]
+    ; CHECK-NEXT: $w1 = COPY [[SBCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0, implicit $w1
+    %1:gpr32 = COPY $w0
+    %2:gpr32 = COPY $w1
+    %3:gpr32 = COPY $w2
+    %4:gpr32 = COPY $w3
+    %5:gpr32 = SUBSWrr %1, %3, implicit-def $nzcv
+    %6:gpr32 = SBCSWr %2, %4, implicit-def $nzcv, implicit $nzcv
+    $w0 = COPY %5
+    $w1 = COPY %6
+    RET_ReallyLR implicit $w0, implicit $w1
+...

llvm/test/CodeGen/AArch64/GlobalISel/select-sadde.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
+# RUN: llc -verify-machineinstrs -mtriple aarch64-unknown-uknown -global-isel-abort=1 -run-pass=instruction-select %s -o - | FileCheck %s
+...
+---
+name:            sadde_s64
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1
+
+    ; CHECK-LABEL: name: sadde_s64
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32common = MOVi32imm 1
+    ; CHECK-NEXT: [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri [[MOVi32imm]], 1, 0, implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCSXr:%[0-9]+]]:gpr64 = ADCSXr [[COPY]], [[COPY1]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32 = CSINCWr $wzr, $wzr, 7, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[ADCSXr]]
+    ; CHECK-NEXT: $w1 = COPY [[CSINCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $w1
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s32) = G_CONSTANT i32 1
+    %3:gpr(s64), %4:gpr(s32) = G_SADDE %0, %1, %2
+    $x0 = COPY %3(s64)
+    $w1 = COPY %4(s32)
+    RET_ReallyLR implicit $x0, implicit $w1
+...
+...
+---
+name:            sadde_s32
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $w0, $w1
+
+    ; CHECK-LABEL: name: sadde_s32
+    ; CHECK: liveins: $w0, $w1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $w0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32common = MOVi32imm 1
+    ; CHECK-NEXT: [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri [[MOVi32imm]], 1, 0, implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCSWr:%[0-9]+]]:gpr32 = ADCSWr [[COPY]], [[COPY1]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32 = CSINCWr $wzr, $wzr, 7, implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[ADCSWr]]
+    ; CHECK-NEXT: $w1 = COPY [[CSINCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0, implicit $w1
+    %0:gpr(s32) = COPY $w0
+    %1:gpr(s32) = COPY $w1
+    %2:gpr(s32) = G_CONSTANT i32 1
+    %3:gpr(s32), %4:gpr(s32) = G_SADDE %0, %1, %2
+    $w0 = COPY %3(s32)
+    $w1 = COPY %4(s32)
+    RET_ReallyLR implicit $w0, implicit $w1
+...
+...
+---
+name:            sadde_opt_prev_uaddo
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+
+    ; CHECK-LABEL: name: sadde_opt_prev_uaddo
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x3
+    ; CHECK-NEXT: [[ADDSXrr:%[0-9]+]]:gpr64 = ADDSXrr [[COPY]], [[COPY2]], implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCSXr:%[0-9]+]]:gpr64 = ADCSXr [[COPY1]], [[COPY3]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[ADDSXrr]]
+    ; CHECK-NEXT: $x1 = COPY [[ADCSXr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s64) = COPY $x2
+    %3:gpr(s64) = COPY $x3
+    %8:gpr(s64), %12:gpr(s32) = G_UADDO %0, %2
+    %9:gpr(s64), %13:gpr(s32) = G_SADDE %1, %3, %12
+    $x0 = COPY %8(s64)
+    $x1 = COPY %9(s64)
+    RET_ReallyLR implicit $x0, implicit $x1
+...
+...
+---
+name:            sadde_opt_prev_uadde
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+
+    ; CHECK-LABEL: name: sadde_opt_prev_uadde
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x3
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32common = MOVi32imm 1
+    ; CHECK-NEXT: [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri [[MOVi32imm]], 1, 0, implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCSXr:%[0-9]+]]:gpr64 = ADCSXr [[COPY]], [[COPY2]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[ADCSXr1:%[0-9]+]]:gpr64 = ADCSXr [[COPY1]], [[COPY3]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[ADCSXr]]
+    ; CHECK-NEXT: $x1 = COPY [[ADCSXr1]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s64) = COPY $x2
+    %3:gpr(s64) = COPY $x3
+    %6:gpr(s32) = G_CONSTANT i32 1
+    %8:gpr(s64), %12:gpr(s32) = G_UADDE %0, %2, %6
+    %9:gpr(s64), %13:gpr(s32) = G_SADDE %1, %3, %12
+    $x0 = COPY %8(s64)
+    $x1 = COPY %9(s64)
+    RET_ReallyLR implicit $x0, implicit $x1
+...
+...
+---
+name:            sadde_opt_bail_clobber
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x4, $x5, $x6
+
+    ; CHECK-LABEL: name: sadde_opt_bail_clobber
+    ; CHECK: liveins: $x0, $x1, $x2, $x4, $x5, $x6
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x4
+    ; CHECK-NEXT: [[COPY4:%[0-9]+]]:gpr64 = COPY $x5
+    ; CHECK-NEXT: [[COPY5:%[0-9]+]]:gpr64 = COPY $x6
+    ; CHECK-NEXT: [[ADDSXrr:%[0-9]+]]:gpr64 = ADDSXrr [[COPY]], [[COPY3]], implicit-def $nzcv
+    ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32common = CSINCWr $wzr, $wzr, 3, implicit $nzcv
+    ; CHECK-NEXT: [[ADCSXr:%[0-9]+]]:gpr64 = ADCSXr [[COPY1]], [[COPY4]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri [[CSINCWr]], 1, 0, implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCSXr1:%[0-9]+]]:gpr64 = ADCSXr [[COPY2]], [[COPY5]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[ADDSXrr]]
+    ; CHECK-NEXT: $x1 = COPY [[ADCSXr]]
+    ; CHECK-NEXT: $x2 = COPY [[ADCSXr1]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1, implicit $x2
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s64) = COPY $x2
+    %4:gpr(s64) = COPY $x4
+    %5:gpr(s64) = COPY $x5
+    %6:gpr(s64) = COPY $x6
+    %7:gpr(s64), %11:gpr(s32) = G_UADDO %0, %4
+    %8:gpr(s64), %12:gpr(s32) = G_UADDE %1, %5, %11
+    ; carry-in is not produced by previous instruction
+    %9:gpr(s64), %13:gpr(s32) = G_SADDE %2, %6, %11
+    $x0 = COPY %7(s64)
+    $x1 = COPY %8(s64)
+    $x2 = COPY %9(s64)
+    RET_ReallyLR implicit $x0, implicit $x1, implicit $x2
+...

llvm/test/CodeGen/AArch64/GlobalISel/select-saddo.mir

bb.1.entry:
     liveins: $w0, $w1, $x2
     ; Check that we get ADDSWri when we can fold in a constant.
     ;
     ; CHECK-LABEL: name: saddo_s32_imm
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %copy:gpr32sp = COPY $w0
     ; CHECK-NEXT: %add:gpr32 = ADDSWri %copy, 16, 0, implicit-def $nzcv
-    ; CHECK-NEXT: %overflow:gpr32 = CSINCWr $wzr, $wzr, 7, implicit $nzcv
     ; CHECK-NEXT: $w0 = COPY %add
     ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %copy:gpr(s32) = COPY $w0
     %constant:gpr(s32) = G_CONSTANT i32 16
     %add:gpr(s32), %overflow:gpr(s32) = G_SADDO %copy, %constant
     $w0 = COPY %add(s32)
     RET_ReallyLR implicit $w0
 
bb.1.entry:
     ; Check that we get ADDSWrs when we can fold in a shift.
     ;
     ; CHECK-LABEL: name: saddo_s32_shifted
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %copy1:gpr32 = COPY $w0
     ; CHECK-NEXT: %copy2:gpr32 = COPY $w1
     ; CHECK-NEXT: %add:gpr32 = ADDSWrs %copy1, %copy2, 16, implicit-def $nzcv
-    ; CHECK-NEXT: %overflow:gpr32 = CSINCWr $wzr, $wzr, 7, implicit $nzcv
     ; CHECK-NEXT: $w0 = COPY %add
     ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %copy1:gpr(s32) = COPY $w0
     %copy2:gpr(s32) = COPY $w1
     %constant:gpr(s32) = G_CONSTANT i32 16
     %shift:gpr(s32) = G_SHL %copy2(s32), %constant(s32)
     %add:gpr(s32), %overflow:gpr(s32) = G_SADDO %copy1, %shift
     $w0 = COPY %add(s32)
bb.1.entry:
     liveins: $w0, $w1, $x2
     ; Check that we get SUBSWri when we can fold in a negative constant.
     ;
     ; CHECK-LABEL: name: saddo_s32_neg_imm
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %copy:gpr32sp = COPY $w0
     ; CHECK-NEXT: %add:gpr32 = SUBSWri %copy, 16, 0, implicit-def $nzcv
-    ; CHECK-NEXT: %overflow:gpr32 = CSINCWr $wzr, $wzr, 7, implicit $nzcv
     ; CHECK-NEXT: $w0 = COPY %add
     ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %copy:gpr(s32) = COPY $w0
     %constant:gpr(s32) = G_CONSTANT i32 -16
     %add:gpr(s32), %overflow:gpr(s32) = G_SADDO %copy, %constant
     $w0 = COPY %add(s32)
     RET_ReallyLR implicit $w0
 
bb.1.entry:
     liveins: $w0, $x0
     ; Check that we get ADDSXrx.
     ; CHECK-LABEL: name: saddo_arith_extended
     ; CHECK: liveins: $w0, $x0
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %reg0:gpr64sp = COPY $x0
     ; CHECK-NEXT: %reg1:gpr32 = COPY $w0
     ; CHECK-NEXT: %add:gpr64 = ADDSXrx %reg0, %reg1, 18, implicit-def $nzcv
-    ; CHECK-NEXT: %flags:gpr32 = CSINCWr $wzr, $wzr, 7, implicit $nzcv
     ; CHECK-NEXT: $x0 = COPY %add
     ; CHECK-NEXT: RET_ReallyLR implicit $x0
     %reg0:gpr(s64) = COPY $x0
     %reg1:gpr(s32) = COPY $w0
     %ext:gpr(s64) = G_ZEXT %reg1(s32)
     %cst:gpr(s64) = G_CONSTANT i64 2
     %shift:gpr(s64) = G_SHL %ext, %cst(s64)
     %add:gpr(s64), %flags:gpr(s32) = G_SADDO %reg0, %shift
     $x0 = COPY %add(s64)
     RET_ReallyLR implicit $x0

llvm/test/CodeGen/AArch64/GlobalISel/select-ssube.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
+# RUN: llc -verify-machineinstrs -mtriple aarch64-unknown-uknown -global-isel-abort=1 -run-pass=instruction-select %s -o - | FileCheck %s
+...
+---
+name:            ssube_s64
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1
+
+    ; CHECK-LABEL: name: ssube_s64
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
+    ; CHECK-NEXT: [[SUBSWrr:%[0-9]+]]:gpr32 = SUBSWrr $wzr, [[MOVi32imm]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCSXr:%[0-9]+]]:gpr64 = SBCSXr [[COPY]], [[COPY1]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32 = CSINCWr $wzr, $wzr, 7, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[SBCSXr]]
+    ; CHECK-NEXT: $w1 = COPY [[CSINCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $w1
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s32) = G_CONSTANT i32 1
+    %3:gpr(s64), %4:gpr(s32) = G_SSUBE %0, %1, %2
+    $x0 = COPY %3(s64)
+    $w1 = COPY %4(s32)
+    RET_ReallyLR implicit $x0, implicit $w1
+...
+...
+---
+name:            ssube_s32
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $w0, $w1
+
+    ; CHECK-LABEL: name: ssube_s32
+    ; CHECK: liveins: $w0, $w1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $w0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
+    ; CHECK-NEXT: [[SUBSWrr:%[0-9]+]]:gpr32 = SUBSWrr $wzr, [[MOVi32imm]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCSWr:%[0-9]+]]:gpr32 = SBCSWr [[COPY]], [[COPY1]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32 = CSINCWr $wzr, $wzr, 7, implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[SBCSWr]]
+    ; CHECK-NEXT: $w1 = COPY [[CSINCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0, implicit $w1
+    %0:gpr(s32) = COPY $w0
+    %1:gpr(s32) = COPY $w1
+    %2:gpr(s32) = G_CONSTANT i32 1
+    %3:gpr(s32), %4:gpr(s32) = G_SSUBE %0, %1, %2
+    $w0 = COPY %3(s32)
+    $w1 = COPY %4(s32)
+    RET_ReallyLR implicit $w0, implicit $w1
+...
+...
+---
+name:            ssube_opt_prev_usubo
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+
+    ; CHECK-LABEL: name: ssube_opt_prev_usubo
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x3
+    ; CHECK-NEXT: [[SUBSXrr:%[0-9]+]]:gpr64 = SUBSXrr [[COPY]], [[COPY2]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCSXr:%[0-9]+]]:gpr64 = SBCSXr [[COPY1]], [[COPY3]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[SUBSXrr]]
+    ; CHECK-NEXT: $x1 = COPY [[SBCSXr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s64) = COPY $x2
+    %3:gpr(s64) = COPY $x3
+    %8:gpr(s64), %12:gpr(s32) = G_USUBO %0, %2
+    %9:gpr(s64), %13:gpr(s32) = G_SSUBE %1, %3, %12
+    $x0 = COPY %8(s64)
+    $x1 = COPY %9(s64)
+    RET_ReallyLR implicit $x0, implicit $x1
+...
+...
+---
+name:            ssube_opt_prev_usube
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+
+    ; CHECK-LABEL: name: ssube_opt_prev_usube
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x3
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
+    ; CHECK-NEXT: [[SUBSWrr:%[0-9]+]]:gpr32 = SUBSWrr $wzr, [[MOVi32imm]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCSXr:%[0-9]+]]:gpr64 = SBCSXr [[COPY]], [[COPY2]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[SBCSXr1:%[0-9]+]]:gpr64 = SBCSXr [[COPY1]], [[COPY3]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[SBCSXr]]
+    ; CHECK-NEXT: $x1 = COPY [[SBCSXr1]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s64) = COPY $x2
+    %3:gpr(s64) = COPY $x3
+    %6:gpr(s32) = G_CONSTANT i32 1
+    %8:gpr(s64), %12:gpr(s32) = G_USUBE %0, %2, %6
+    %9:gpr(s64), %13:gpr(s32) = G_SSUBE %1, %3, %12
+    $x0 = COPY %8(s64)
+    $x1 = COPY %9(s64)
+    RET_ReallyLR implicit $x0, implicit $x1
+...
+...
+---
+name:            ssube_opt_bail_clobber
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x4, $x5, $x6
+
+    ; CHECK-LABEL: name: ssube_opt_bail_clobber
+    ; CHECK: liveins: $x0, $x1, $x2, $x4, $x5, $x6
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x4
+    ; CHECK-NEXT: [[COPY4:%[0-9]+]]:gpr64 = COPY $x5
+    ; CHECK-NEXT: [[COPY5:%[0-9]+]]:gpr64 = COPY $x6
+    ; CHECK-NEXT: [[SUBSXrr:%[0-9]+]]:gpr64 = SUBSXrr [[COPY]], [[COPY3]], implicit-def $nzcv
+    ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32 = CSINCWr $wzr, $wzr, 2, implicit $nzcv
+    ; CHECK-NEXT: [[SBCSXr:%[0-9]+]]:gpr64 = SBCSXr [[COPY1]], [[COPY4]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[SUBSWrr:%[0-9]+]]:gpr32 = SUBSWrr $wzr, [[CSINCWr]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCSXr1:%[0-9]+]]:gpr64 = SBCSXr [[COPY2]], [[COPY5]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[SUBSXrr]]
+    ; CHECK-NEXT: $x1 = COPY [[SBCSXr]]
+    ; CHECK-NEXT: $x2 = COPY [[SBCSXr1]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1, implicit $x2
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s64) = COPY $x2
+    %4:gpr(s64) = COPY $x4
+    %5:gpr(s64) = COPY $x5
+    %6:gpr(s64) = COPY $x6
+    %7:gpr(s64), %11:gpr(s32) = G_USUBO %0, %4
+    %8:gpr(s64), %12:gpr(s32) = G_USUBE %1, %5, %11
+    ; carry-in is not produced by previous instruction
+    %9:gpr(s64), %13:gpr(s32) = G_SSUBE %2, %6, %11
+    $x0 = COPY %7(s64)
+    $x1 = COPY %8(s64)
+    $x2 = COPY %9(s64)
+    RET_ReallyLR implicit $x0, implicit $x1, implicit $x2
+...

llvm/test/CodeGen/AArch64/GlobalISel/select-ssubo.mir

bb.1.entry:
     liveins: $w0, $w1, $x2
     ; Check that we get ADDSWri when we can fold in a constant.
     ;
     ; CHECK-LABEL: name: ssubo_s32_imm
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %copy:gpr32sp = COPY $w0
     ; CHECK-NEXT: %add:gpr32 = SUBSWri %copy, 16, 0, implicit-def $nzcv
-    ; CHECK-NEXT: %overflow:gpr32 = CSINCWr $wzr, $wzr, 7, implicit $nzcv
     ; CHECK-NEXT: $w0 = COPY %add
     ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %copy:gpr(s32) = COPY $w0
     %constant:gpr(s32) = G_CONSTANT i32 16
     %add:gpr(s32), %overflow:gpr(s32) = G_SSUBO %copy, %constant
     $w0 = COPY %add(s32)
     RET_ReallyLR implicit $w0
 
bb.1.entry:
     ; Check that we get ADDSWrs when we can fold in a shift.
     ;
     ; CHECK-LABEL: name: ssubo_s32_shifted
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %copy1:gpr32 = COPY $w0
     ; CHECK-NEXT: %copy2:gpr32 = COPY $w1
     ; CHECK-NEXT: %add:gpr32 = SUBSWrs %copy1, %copy2, 16, implicit-def $nzcv
-    ; CHECK-NEXT: %overflow:gpr32 = CSINCWr $wzr, $wzr, 7, implicit $nzcv
     ; CHECK-NEXT: $w0 = COPY %add
     ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %copy1:gpr(s32) = COPY $w0
     %copy2:gpr(s32) = COPY $w1
     %constant:gpr(s32) = G_CONSTANT i32 16
     %shift:gpr(s32) = G_SHL %copy2(s32), %constant(s32)
     %add:gpr(s32), %overflow:gpr(s32) = G_SSUBO %copy1, %shift
     $w0 = COPY %add(s32)
bb.1.entry:
     liveins: $w0, $w1, $x2
     ; Check that we get SUBSWri when we can fold in a negative constant.
     ;
     ; CHECK-LABEL: name: ssubo_s32_neg_imm
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %copy:gpr32sp = COPY $w0
     ; CHECK-NEXT: %add:gpr32 = ADDSWri %copy, 16, 0, implicit-def $nzcv
-    ; CHECK-NEXT: %overflow:gpr32 = CSINCWr $wzr, $wzr, 7, implicit $nzcv
     ; CHECK-NEXT: $w0 = COPY %add
     ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %copy:gpr(s32) = COPY $w0
     %constant:gpr(s32) = G_CONSTANT i32 -16
     %add:gpr(s32), %overflow:gpr(s32) = G_SSUBO %copy, %constant
     $w0 = COPY %add(s32)
     RET_ReallyLR implicit $w0
 
bb.1.entry:
     liveins: $w0, $x0
     ; Check that we get ADDSXrx.
     ; CHECK-LABEL: name: ssubo_arith_extended
     ; CHECK: liveins: $w0, $x0
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %reg0:gpr64sp = COPY $x0
     ; CHECK-NEXT: %reg1:gpr32 = COPY $w0
     ; CHECK-NEXT: %add:gpr64 = SUBSXrx %reg0, %reg1, 18, implicit-def $nzcv
-    ; CHECK-NEXT: %flags:gpr32 = CSINCWr $wzr, $wzr, 7, implicit $nzcv
     ; CHECK-NEXT: $x0 = COPY %add
     ; CHECK-NEXT: RET_ReallyLR implicit $x0
     %reg0:gpr(s64) = COPY $x0
     %reg1:gpr(s32) = COPY $w0
     %ext:gpr(s64) = G_ZEXT %reg1(s32)
     %cst:gpr(s64) = G_CONSTANT i64 2
     %shift:gpr(s64) = G_SHL %ext, %cst(s64)
     %add:gpr(s64), %flags:gpr(s32) = G_SSUBO %reg0, %shift
     $x0 = COPY %add(s64)
     RET_ReallyLR implicit $x0

llvm/test/CodeGen/AArch64/GlobalISel/select-uadde.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
+# RUN: llc -verify-machineinstrs -mtriple aarch64-unknown-uknown -global-isel-abort=1 -run-pass=instruction-select %s -o - | FileCheck %s
+...
+---
+name:            uadde_s64
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1
+
+    ; CHECK-LABEL: name: uadde_s64
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32common = MOVi32imm 1
+    ; CHECK-NEXT: [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri [[MOVi32imm]], 1, 0, implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCSXr:%[0-9]+]]:gpr64 = ADCSXr [[COPY]], [[COPY1]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32 = CSINCWr $wzr, $wzr, 3, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[ADCSXr]]
+    ; CHECK-NEXT: $w1 = COPY [[CSINCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $w1
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s32) = G_CONSTANT i32 1
+    %3:gpr(s64), %4:gpr(s32) = G_UADDE %0, %1, %2
+    $x0 = COPY %3(s64)
+    $w1 = COPY %4(s32)
+    RET_ReallyLR implicit $x0, implicit $w1
+...
+...
+---
+name:            uadde_s32
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $w0, $w1
+
+    ; CHECK-LABEL: name: uadde_s32
+    ; CHECK: liveins: $w0, $w1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $w0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32common = MOVi32imm 1
+    ; CHECK-NEXT: [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri [[MOVi32imm]], 1, 0, implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCSWr:%[0-9]+]]:gpr32 = ADCSWr [[COPY]], [[COPY1]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32 = CSINCWr $wzr, $wzr, 3, implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[ADCSWr]]
+    ; CHECK-NEXT: $w1 = COPY [[CSINCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0, implicit $w1
+    %0:gpr(s32) = COPY $w0
+    %1:gpr(s32) = COPY $w1
+    %2:gpr(s32) = G_CONSTANT i32 1
+    %3:gpr(s32), %4:gpr(s32) = G_UADDE %0, %1, %2
+    $w0 = COPY %3(s32)
+    $w1 = COPY %4(s32)
+    RET_ReallyLR implicit $w0, implicit $w1
+...
+...
+---
+name:            uadde_opt_prev_uaddo
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+
+    ; CHECK-LABEL: name: uadde_opt_prev_uaddo
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x3
+    ; CHECK-NEXT: [[ADDSXrr:%[0-9]+]]:gpr64 = ADDSXrr [[COPY]], [[COPY2]], implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCSXr:%[0-9]+]]:gpr64 = ADCSXr [[COPY1]], [[COPY3]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[ADDSXrr]]
+    ; CHECK-NEXT: $x1 = COPY [[ADCSXr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s64) = COPY $x2
+    %3:gpr(s64) = COPY $x3
+    %8:gpr(s64), %12:gpr(s32) = G_UADDO %0, %2
+    %9:gpr(s64), %13:gpr(s32) = G_UADDE %1, %3, %12
+    $x0 = COPY %8(s64)
+    $x1 = COPY %9(s64)
+    RET_ReallyLR implicit $x0, implicit $x1
+...
+...
+---
+name:            uadde_opt_prev_uadde
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+
+    ; CHECK-LABEL: name: uadde_opt_prev_uadde
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x3
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32common = MOVi32imm 1
+    ; CHECK-NEXT: [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri [[MOVi32imm]], 1, 0, implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCSXr:%[0-9]+]]:gpr64 = ADCSXr [[COPY]], [[COPY2]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[ADCSXr1:%[0-9]+]]:gpr64 = ADCSXr [[COPY1]], [[COPY3]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[ADCSXr]]
+    ; CHECK-NEXT: $x1 = COPY [[ADCSXr1]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s64) = COPY $x2
+    %3:gpr(s64) = COPY $x3
+    %6:gpr(s32) = G_CONSTANT i32 1
+    %8:gpr(s64), %12:gpr(s32) = G_UADDE %0, %2, %6
+    %9:gpr(s64), %13:gpr(s32) = G_UADDE %1, %3, %12
+    $x0 = COPY %8(s64)
+    $x1 = COPY %9(s64)
+    RET_ReallyLR implicit $x0, implicit $x1
+...
+...
+---
+name:            uadde_opt_bail_clobber
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x4, $x5, $x6
+
+    ; CHECK-LABEL: name: uadde_opt_bail_clobber
+    ; CHECK: liveins: $x0, $x1, $x2, $x4, $x5, $x6
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x4
+    ; CHECK-NEXT: [[COPY4:%[0-9]+]]:gpr64 = COPY $x5
+    ; CHECK-NEXT: [[COPY5:%[0-9]+]]:gpr64 = COPY $x6
+    ; CHECK-NEXT: [[ADDSXrr:%[0-9]+]]:gpr64 = ADDSXrr [[COPY]], [[COPY3]], implicit-def $nzcv
+    ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32common = CSINCWr $wzr, $wzr, 3, implicit $nzcv
+    ; CHECK-NEXT: [[ADCSXr:%[0-9]+]]:gpr64 = ADCSXr [[COPY1]], [[COPY4]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri [[CSINCWr]], 1, 0, implicit-def $nzcv
+    ; CHECK-NEXT: [[ADCSXr1:%[0-9]+]]:gpr64 = ADCSXr [[COPY2]], [[COPY5]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[ADDSXrr]]
+    ; CHECK-NEXT: $x1 = COPY [[ADCSXr]]
+    ; CHECK-NEXT: $x2 = COPY [[ADCSXr1]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1, implicit $x2
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s64) = COPY $x2
+    %4:gpr(s64) = COPY $x4
+    %5:gpr(s64) = COPY $x5
+    %6:gpr(s64) = COPY $x6
+    %7:gpr(s64), %11:gpr(s32) = G_UADDO %0, %4
+    %8:gpr(s64), %12:gpr(s32) = G_UADDE %1, %5, %11
+    ; carry-in is not produced by previous instruction
+    %9:gpr(s64), %13:gpr(s32) = G_UADDE %2, %6, %11
+    $x0 = COPY %7(s64)
+    $x1 = COPY %8(s64)
+    $x2 = COPY %9(s64)
+    RET_ReallyLR implicit $x0, implicit $x1, implicit $x2
+...

llvm/test/CodeGen/AArch64/GlobalISel/select-uaddo.mir

bb.1.entry:
     liveins: $w0, $w1, $x2
     ; Check that we get ADDSWri when we can fold in a constant.
     ;
     ; CHECK-LABEL: name: uaddo_s32_imm
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %copy:gpr32sp = COPY $w0
     ; CHECK-NEXT: %add:gpr32 = ADDSWri %copy, 16, 0, implicit-def $nzcv
-    ; CHECK-NEXT: %overflow:gpr32 = CSINCWr $wzr, $wzr, 3, implicit $nzcv
     ; CHECK-NEXT: $w0 = COPY %add
     ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %copy:gpr(s32) = COPY $w0
     %constant:gpr(s32) = G_CONSTANT i32 16
     %add:gpr(s32), %overflow:gpr(s32) = G_UADDO %copy, %constant
     $w0 = COPY %add(s32)
     RET_ReallyLR implicit $w0
 
bb.1.entry:
     ; Check that we get ADDSWrs when we can fold in a shift.
     ;
     ; CHECK-LABEL: name: uaddo_s32_shifted
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %copy1:gpr32 = COPY $w0
     ; CHECK-NEXT: %copy2:gpr32 = COPY $w1
     ; CHECK-NEXT: %add:gpr32 = ADDSWrs %copy1, %copy2, 16, implicit-def $nzcv
-    ; CHECK-NEXT: %overflow:gpr32 = CSINCWr $wzr, $wzr, 3, implicit $nzcv
     ; CHECK-NEXT: $w0 = COPY %add
     ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %copy1:gpr(s32) = COPY $w0
     %copy2:gpr(s32) = COPY $w1
     %constant:gpr(s32) = G_CONSTANT i32 16
     %shift:gpr(s32) = G_SHL %copy2(s32), %constant(s32)
     %add:gpr(s32), %overflow:gpr(s32) = G_UADDO %copy1, %shift
     $w0 = COPY %add(s32)
bb.1.entry:
     liveins: $w0, $w1, $x2
     ; Check that we get SUBSWri when we can fold in a negative constant.
     ;
     ; CHECK-LABEL: name: uaddo_s32_neg_imm
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %copy:gpr32sp = COPY $w0
     ; CHECK-NEXT: %add:gpr32 = SUBSWri %copy, 16, 0, implicit-def $nzcv
-    ; CHECK-NEXT: %overflow:gpr32 = CSINCWr $wzr, $wzr, 3, implicit $nzcv
     ; CHECK-NEXT: $w0 = COPY %add
     ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %copy:gpr(s32) = COPY $w0
     %constant:gpr(s32) = G_CONSTANT i32 -16
     %add:gpr(s32), %overflow:gpr(s32) = G_UADDO %copy, %constant
     $w0 = COPY %add(s32)
     RET_ReallyLR implicit $w0
 
bb.1.entry:
     liveins: $w0, $x0
     ; Check that we get ADDSXrx.
     ; CHECK-LABEL: name: uaddo_arith_extended
     ; CHECK: liveins: $w0, $x0
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %reg0:gpr64sp = COPY $x0
     ; CHECK-NEXT: %reg1:gpr32 = COPY $w0
     ; CHECK-NEXT: %add:gpr64 = ADDSXrx %reg0, %reg1, 18, implicit-def $nzcv
-    ; CHECK-NEXT: %flags:gpr32 = CSINCWr $wzr, $wzr, 3, implicit $nzcv
     ; CHECK-NEXT: $x0 = COPY %add
     ; CHECK-NEXT: RET_ReallyLR implicit $x0
     %reg0:gpr(s64) = COPY $x0
     %reg1:gpr(s32) = COPY $w0
     %ext:gpr(s64) = G_ZEXT %reg1(s32)
     %cst:gpr(s64) = G_CONSTANT i64 2
     %shift:gpr(s64) = G_SHL %ext, %cst(s64)
     %add:gpr(s64), %flags:gpr(s32) = G_UADDO %reg0, %shift
     $x0 = COPY %add(s64)
     RET_ReallyLR implicit $x0

llvm/test/CodeGen/AArch64/GlobalISel/select-usube.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
+# RUN: llc -verify-machineinstrs -mtriple aarch64-unknown-uknown -global-isel-abort=1 -run-pass=instruction-select %s -o - | FileCheck %s
+...
+---
+name:            usube_s64
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1
+
+    ; CHECK-LABEL: name: usube_s64
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
+    ; CHECK-NEXT: [[SUBSWrr:%[0-9]+]]:gpr32 = SUBSWrr $wzr, [[MOVi32imm]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCSXr:%[0-9]+]]:gpr64 = SBCSXr [[COPY]], [[COPY1]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32 = CSINCWr $wzr, $wzr, 2, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[SBCSXr]]
+    ; CHECK-NEXT: $w1 = COPY [[CSINCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $w1
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s32) = G_CONSTANT i32 1
+    %3:gpr(s64), %4:gpr(s32) = G_USUBE %0, %1, %2
+    $x0 = COPY %3(s64)
+    $w1 = COPY %4(s32)
+    RET_ReallyLR implicit $x0, implicit $w1
+...
+...
+---
+name:            usube_s32
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $w0, $w1
+
+    ; CHECK-LABEL: name: usube_s32
+    ; CHECK: liveins: $w0, $w1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $w0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
+    ; CHECK-NEXT: [[SUBSWrr:%[0-9]+]]:gpr32 = SUBSWrr $wzr, [[MOVi32imm]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCSWr:%[0-9]+]]:gpr32 = SBCSWr [[COPY]], [[COPY1]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32 = CSINCWr $wzr, $wzr, 2, implicit $nzcv
+    ; CHECK-NEXT: $w0 = COPY [[SBCSWr]]
+    ; CHECK-NEXT: $w1 = COPY [[CSINCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0, implicit $w1
+    %0:gpr(s32) = COPY $w0
+    %1:gpr(s32) = COPY $w1
+    %2:gpr(s32) = G_CONSTANT i32 1
+    %3:gpr(s32), %4:gpr(s32) = G_USUBE %0, %1, %2
+    $w0 = COPY %3(s32)
+    $w1 = COPY %4(s32)
+    RET_ReallyLR implicit $w0, implicit $w1
+...
+...
+---
+name:            usube_opt_prev_usubo
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+
+    ; CHECK-LABEL: name: usube_opt_prev_usubo
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x3
+    ; CHECK-NEXT: [[SUBSXrr:%[0-9]+]]:gpr64 = SUBSXrr [[COPY]], [[COPY2]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCSXr:%[0-9]+]]:gpr64 = SBCSXr [[COPY1]], [[COPY3]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[SUBSXrr]]
+    ; CHECK-NEXT: $x1 = COPY [[SBCSXr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s64) = COPY $x2
+    %3:gpr(s64) = COPY $x3
+    %8:gpr(s64), %12:gpr(s32) = G_USUBO %0, %2
+    %9:gpr(s64), %13:gpr(s32) = G_USUBE %1, %3, %12
+    $x0 = COPY %8(s64)
+    $x1 = COPY %9(s64)
+    RET_ReallyLR implicit $x0, implicit $x1
+...
+...
+---
+name:            usube_opt_prev_usube
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+
+    ; CHECK-LABEL: name: usube_opt_prev_usube
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x3
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm 1
+    ; CHECK-NEXT: [[SUBSWrr:%[0-9]+]]:gpr32 = SUBSWrr $wzr, [[MOVi32imm]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCSXr:%[0-9]+]]:gpr64 = SBCSXr [[COPY]], [[COPY2]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[SBCSXr1:%[0-9]+]]:gpr64 = SBCSXr [[COPY1]], [[COPY3]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[SBCSXr]]
+    ; CHECK-NEXT: $x1 = COPY [[SBCSXr1]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s64) = COPY $x2
+    %3:gpr(s64) = COPY $x3
+    %6:gpr(s32) = G_CONSTANT i32 1
+    %8:gpr(s64), %12:gpr(s32) = G_USUBE %0, %2, %6
+    %9:gpr(s64), %13:gpr(s32) = G_USUBE %1, %3, %12
+    $x0 = COPY %8(s64)
+    $x1 = COPY %9(s64)
+    RET_ReallyLR implicit $x0, implicit $x1
+...
+...
+---
+name:            usube_opt_bail_clobber
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x4, $x5, $x6
+
+    ; CHECK-LABEL: name: usube_opt_bail_clobber
+    ; CHECK: liveins: $x0, $x1, $x2, $x4, $x5, $x6
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY $x2
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64 = COPY $x4
+    ; CHECK-NEXT: [[COPY4:%[0-9]+]]:gpr64 = COPY $x5
+    ; CHECK-NEXT: [[COPY5:%[0-9]+]]:gpr64 = COPY $x6
+    ; CHECK-NEXT: [[SUBSXrr:%[0-9]+]]:gpr64 = SUBSXrr [[COPY]], [[COPY3]], implicit-def $nzcv
+    ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32 = CSINCWr $wzr, $wzr, 2, implicit $nzcv
+    ; CHECK-NEXT: [[SBCSXr:%[0-9]+]]:gpr64 = SBCSXr [[COPY1]], [[COPY4]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: [[SUBSWrr:%[0-9]+]]:gpr32 = SUBSWrr $wzr, [[CSINCWr]], implicit-def $nzcv
+    ; CHECK-NEXT: [[SBCSXr1:%[0-9]+]]:gpr64 = SBCSXr [[COPY2]], [[COPY5]], implicit-def $nzcv, implicit $nzcv
+    ; CHECK-NEXT: $x0 = COPY [[SUBSXrr]]
+    ; CHECK-NEXT: $x1 = COPY [[SBCSXr]]
+    ; CHECK-NEXT: $x2 = COPY [[SBCSXr1]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $x1, implicit $x2
+    %0:gpr(s64) = COPY $x0
+    %1:gpr(s64) = COPY $x1
+    %2:gpr(s64) = COPY $x2
+    %4:gpr(s64) = COPY $x4
+    %5:gpr(s64) = COPY $x5
+    %6:gpr(s64) = COPY $x6
+    %7:gpr(s64), %11:gpr(s32) = G_USUBO %0, %4
+    %8:gpr(s64), %12:gpr(s32) = G_USUBE %1, %5, %11
+    ; carry-in is not produced by previous instruction
+    %9:gpr(s64), %13:gpr(s32) = G_USUBE %2, %6, %11
+    $x0 = COPY %7(s64)
+    $x1 = COPY %8(s64)
+    $x2 = COPY %9(s64)
+    RET_ReallyLR implicit $x0, implicit $x1, implicit $x2
+...

llvm/test/CodeGen/AArch64/GlobalISel/select-usubo.mir

bb.1.entry:
 
     ; CHECK-LABEL: name: usubo_s32
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr32 = COPY $w0
     ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY $w1
     ; CHECK-NEXT: [[SUBSWrr:%[0-9]+]]:gpr32 = SUBSWrr [[COPY]], [[COPY1]], implicit-def $nzcv
     ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32 = CSINCWr $wzr, $wzr, 2, implicit $nzcv
-    ; CHECK-NEXT: $w0 = COPY [[CSINCWr]]
-    ; CHECK-NEXT: RET_ReallyLR implicit $w0
+    ; CHECK-NEXT: $w0 = COPY [[SUBSWrr]]
+    ; CHECK-NEXT: $w1 = COPY [[CSINCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0, implicit $w1
     %0:gpr(s32) = COPY $w0
     %1:gpr(s32) = COPY $w1
     %3:gpr(s32), %4:gpr(s32) = G_USUBO %0, %1
-    $w0 = COPY %4(s32)
-    RET_ReallyLR implicit $w0
+    $w0 = COPY %3
+    $w1 = COPY %4
+    RET_ReallyLR implicit $w0, implicit $w1
 
 ...
 ---
 name:            usubo_s64
 alignment:       4
 legalized:       true
 regBankSelected: true
 tracksRegLiveness: true
 body:             |
   bb.1.entry:
     liveins: $x0, $x1, $x2
 
     ; CHECK-LABEL: name: usubo_s64
     ; CHECK: liveins: $x0, $x1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $x0
     ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x1
     ; CHECK-NEXT: [[SUBSXrr:%[0-9]+]]:gpr64 = SUBSXrr [[COPY]], [[COPY1]], implicit-def $nzcv
     ; CHECK-NEXT: [[CSINCWr:%[0-9]+]]:gpr32 = CSINCWr $wzr, $wzr, 2, implicit $nzcv
-    ; CHECK-NEXT: $w0 = COPY [[CSINCWr]]
-    ; CHECK-NEXT: RET_ReallyLR implicit $w0
+    ; CHECK-NEXT: $x0 = COPY [[SUBSXrr]]
+    ; CHECK-NEXT: $w1 = COPY [[CSINCWr]]
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $w1
     %0:gpr(s64) = COPY $x0
     %1:gpr(s64) = COPY $x1
     %3:gpr(s64), %4:gpr(s32) = G_USUBO %0, %1
-    $w0 = COPY %4
-    RET_ReallyLR implicit $w0
+    $x0 = COPY %3
+    $w1 = COPY %4
+    RET_ReallyLR implicit $x0, implicit $w1
 
 ...
 ---
 name:            usubo_s32_imm
 alignment:       4
 legalized:       true
 regBankSelected: true
 tracksRegLiveness: true
 body:             |
   bb.1.entry:
     liveins: $w0, $w1, $x2
     ; Check that we get ADDSWri when we can fold in a constant.
     ;
     ; CHECK-LABEL: name: usubo_s32_imm
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %copy:gpr32sp = COPY $w0
     ; CHECK-NEXT: %add:gpr32 = SUBSWri %copy, 16, 0, implicit-def $nzcv
-    ; CHECK-NEXT: %overflow:gpr32 = CSINCWr $wzr, $wzr, 2, implicit $nzcv
     ; CHECK-NEXT: $w0 = COPY %add
-    ; CHECK-NEXT: $w1 = COPY %overflow
-    ; CHECK-NEXT: RET_ReallyLR implicit $w0, implicit $w1
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %copy:gpr(s32) = COPY $w0
     %constant:gpr(s32) = G_CONSTANT i32 16
     %add:gpr(s32), %overflow:gpr(s32) = G_USUBO %copy, %constant
     $w0 = COPY %add
-    $w1 = COPY %overflow
-    RET_ReallyLR implicit $w0, implicit $w1
+    RET_ReallyLR implicit $w0
 
 ...
 ---
 name:            usubo_s32_shifted
 alignment:       4
 legalized:       true
 regBankSelected: true
 tracksRegLiveness: true
 body:             |
   bb.1.entry:
     liveins: $w0, $w1, $x2
     ; Check that we get ADDSWrs when we can fold in a shift.
     ;
     ; CHECK-LABEL: name: usubo_s32_shifted
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %copy1:gpr32 = COPY $w0
     ; CHECK-NEXT: %copy2:gpr32 = COPY $w1
     ; CHECK-NEXT: %add:gpr32 = SUBSWrs %copy1, %copy2, 16, implicit-def $nzcv
-    ; CHECK-NEXT: %overflow:gpr32 = CSINCWr $wzr, $wzr, 2, implicit $nzcv
     ; CHECK-NEXT: $w0 = COPY %add
-    ; CHECK-NEXT: $w1 = COPY %overflow
-    ; CHECK-NEXT: RET_ReallyLR implicit $w0, implicit $w1
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %copy1:gpr(s32) = COPY $w0
     %copy2:gpr(s32) = COPY $w1
     %constant:gpr(s32) = G_CONSTANT i32 16
     %shift:gpr(s32) = G_SHL %copy2(s32), %constant(s32)
     %add:gpr(s32), %overflow:gpr(s32) = G_USUBO %copy1, %shift
     $w0 = COPY %add(s32)
-    $w1 = COPY %overflow(s32)
-    RET_ReallyLR implicit $w0, implicit $w1
+    RET_ReallyLR implicit $w0
 
 ...
 ---
 name:            usubo_s32_neg_imm
 alignment:       4
 legalized:       true
 regBankSelected: true
 tracksRegLiveness: true
 body:             |
   bb.1.entry:
     liveins: $w0, $w1, $x2
     ; Check that we get SUBSWri when we can fold in a negative constant.
     ;
     ; CHECK-LABEL: name: usubo_s32_neg_imm
     ; CHECK: liveins: $w0, $w1, $x2
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %copy:gpr32sp = COPY $w0
     ; CHECK-NEXT: %add:gpr32 = ADDSWri %copy, 16, 0, implicit-def $nzcv
-    ; CHECK-NEXT: %overflow:gpr32 = CSINCWr $wzr, $wzr, 2, implicit $nzcv
     ; CHECK-NEXT: $w0 = COPY %add
-    ; CHECK-NEXT: $w1 = COPY %overflow
-    ; CHECK-NEXT: RET_ReallyLR implicit $w0, implicit $w1
+    ; CHECK-NEXT: RET_ReallyLR implicit $w0
     %copy:gpr(s32) = COPY $w0
     %constant:gpr(s32) = G_CONSTANT i32 -16
     %add:gpr(s32), %overflow:gpr(s32) = G_USUBO %copy, %constant
     $w0 = COPY %add(s32)
-    $w1 = COPY %overflow(s32)
-    RET_ReallyLR implicit $w0, implicit $w1
+    RET_ReallyLR implicit $w0
 
 ...
 ---
 name:            usubo_arith_extended
 alignment:       4
 legalized:       true
 regBankSelected: true
 tracksRegLiveness: true
 body:             |
   bb.1.entry:
     liveins: $w0, $x0
     ; Check that we get ADDSXrx.
     ; CHECK-LABEL: name: usubo_arith_extended
     ; CHECK: liveins: $w0, $x0
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: %reg0:gpr64sp = COPY $x0
     ; CHECK-NEXT: %reg1:gpr32 = COPY $w0
     ; CHECK-NEXT: %add:gpr64 = SUBSXrx %reg0, %reg1, 18, implicit-def $nzcv
-    ; CHECK-NEXT: %flags:gpr32 = CSINCWr $wzr, $wzr, 2, implicit $nzcv
     ; CHECK-NEXT: $x0 = COPY %add
-    ; CHECK-NEXT: $w1 = COPY %flags
-    ; CHECK-NEXT: RET_ReallyLR implicit $x0, implicit $w1
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0
     %reg0:gpr(s64) = COPY $x0
     %reg1:gpr(s32) = COPY $w0
     %ext:gpr(s64) = G_ZEXT %reg1(s32)
     %cst:gpr(s64) = G_CONSTANT i64 2
     %shift:gpr(s64) = G_SHL %ext, %cst(s64)
     %add:gpr(s64), %flags:gpr(s32) = G_USUBO %reg0, %shift
     $x0 = COPY %add(s64)
-    $w1 = COPY %flags
-    RET_ReallyLR implicit $x0, implicit $w1
+    RET_ReallyLR implicit $x0

llvm/test/CodeGen/AArch64/popcount.ll

 ; CHECK-NEXT:    // implicit-def: $q0
 ; CHECK-NEXT:    fmov s0, s1
 ; CHECK-NEXT:    // kill: def $s0 killed $s0 killed $q0
 ; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    mov w8, w0
 ; CHECK-NEXT:    // kill: def $x10 killed $w10
 ; CHECK-NEXT:    bfi x8, x10, #32, #32
 ; CHECK-NEXT:    adds x8, x8, x9
-; CHECK-NEXT:    cset w9, hs
 ; CHECK-NEXT:    mov w0, w8
 ; CHECK-NEXT:    ret
 Entry:
   %1 = load i256, ptr %0, align 16
   %2 = tail call i256 @llvm.ctpop.i256(i256 %1)
   %3 = trunc i256 %2 to i16
   ret i16 %3
 }