[AArch64][GlobalISel] Instruction selection for add/sub with carry-in

[porglezomp]

Summary

[AArch64][GlobalISel] Make G_USUBO legal and select it

The expansion for wide subtractions includes G_USUBO, so it's important
that it's a legal instruction.

[AArch64][GlobalISel] Add selection support for G_UADDE and G_USUBE

These take a carry-in argument. In the common case where it's directly
fed by a carry-out operation in the preceding instruction, we can
translate it to an ADCS/SBCS operation. Otherwise, we re-generate the
condition flags.

[AArch64][GlobalISel] Make G_SADDE and G_SSUBE legal

[porglezomp]

Re-enable G_SADDE and G_SSUBE in selection

[porglezomp]

Update revision again - accidentally pushed only the last commit

[porglezomp]

• [AArch64][GlobalISel] Fix legalize-arith-128 test ABI

[paquette]

Do you think you could split this into 3 patches, one for each commit?

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

2794

Could debug instructions mess this up?

If so, adding a isPredecessorIgnoringDBG helper might be useful here.

2795

Looks like you can fold the variables above into this one?

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp

169

Can you add testcases for the s32/s64 cases here, and for the minScalar case?

llvm/test/CodeGen/AArch64/GlobalISel/legalize-arith-128.ll

2

I think this should be a MIR test rather than a IR test.

This command line should get you 90% there:

llc -mtriple aarch64 -stop-before=legalizer -global-isel -simplify-mir -o -

You'll probably have to tidy it up a little bit after generating it though. (See: test/CodeGen/AArch64/legalize-uaddo.mir as an example)

[porglezomp]

Closing this in favor of separate revisions for the individual changes.

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

private:
   ///
   /// \p Dst is the destination register of the binop to emit.
   /// \p LHS is the left-hand operand of the binop to emit.
   /// \p RHS is the right-hand operand of the binop to emit.
   MachineInstr *emitAddSub(
       const std::array<std::array<unsigned, 2>, 5> &AddrModeAndSizeToOpcode,
       Register Dst, MachineOperand &LHS, MachineOperand &RHS,
       MachineIRBuilder &MIRBuilder) const;
+  MachineInstr *emitAdcSbc(const std::array<unsigned, 2> &SizeToOpcode,
+                           Register Dst, MachineOperand &LHS,
+                           MachineOperand &RHS,
+                           MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitADD(Register DefReg, MachineOperand &LHS,
                         MachineOperand &RHS,
                         MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitADDS(Register Dst, MachineOperand &LHS, MachineOperand &RHS,
                          MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitSUBS(Register Dst, MachineOperand &LHS, MachineOperand &RHS,
                          MachineIRBuilder &MIRBuilder) const;
+  MachineInstr *emitADCS(Register Dst, MachineOperand &LHS, MachineOperand &RHS,
+                         MachineIRBuilder &MIRBuilder) const;
+  MachineInstr *emitSBCS(Register Dst, MachineOperand &LHS, MachineOperand &RHS,
+                         MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitCMN(MachineOperand &LHS, MachineOperand &RHS,
                         MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitTST(MachineOperand &LHS, MachineOperand &RHS,
                         MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitSelect(Register Dst, Register LHS, Register RHS,
                            AArch64CC::CondCode CC,
                            MachineIRBuilder &MIRBuilder) const;
   MachineInstr *emitExtractVectorElt(Optional<Register> DstReg,
private:
   MachineInstr *emitCSetForICMP(Register DefReg, unsigned Pred,
                                 MachineIRBuilder &MIRBuilder) const;
   /// Emit a CSet for a FP compare.
   ///
   /// \p Dst is expected to be a 32-bit scalar register.
   MachineInstr *emitCSetForFCmp(Register Dst, CmpInst::Predicate Pred,
                                 MachineIRBuilder &MIRBuilder) const;
 
+  /// Get the appropriate overflow cond code for \p Opcode.
+  ///
+  /// \p Opcode is expected to be an overflow op's opcode, e.g. G_UADDO,
+  /// G_USUBO, etc.
+  AArch64CC::CondCode condCodeForOverflowOp(unsigned Opcode) const;
+
   /// Emit the overflow op for \p Opcode.
   ///
   /// \p Opcode is expected to be an overflow op's opcode, e.g. G_UADDO,
   /// G_USUBO, etc.
-  std::pair<MachineInstr *, AArch64CC::CondCode>
-  emitOverflowOp(unsigned Opcode, Register Dst, MachineOperand &LHS,
-                 MachineOperand &RHS, MachineIRBuilder &MIRBuilder) const;
+  MachineInstr *emitOverflowOp(unsigned Opcode, Register Dst,
+                               MachineOperand &LHS, MachineOperand &RHS,
+                               MachineIRBuilder &MIRBuilder) const;
 
   /// Emit a TB(N)Z instruction which tests \p Bit in \p TestReg.
   /// \p IsNegative is true if the test should be "not zero".
   /// This will also optimize the test bit instruction when possible.
   MachineInstr *emitTestBit(Register TestReg, uint64_t Bit, bool IsNegative,
                             MachineBasicBlock *DstMBB,
                             MachineIRBuilder &MIB) const;
 
private:
   /// Return true if \p MI is a load or store of \p NumBytes bytes.
   bool isLoadStoreOfNumBytes(const MachineInstr &MI, unsigned NumBytes) const;
 
   /// Returns true if \p MI is guaranteed to have the high-half of a 64-bit
   /// register zeroed out. In other words, the result of MI has been explicitly
   /// zero extended.
   bool isDef32(const MachineInstr &MI) const;
 
+  /// Returns true if \p MI is one of the carrying add/sub generic ops.
+  bool isCarryOp(const MachineInstr &MI) const;
+
   const AArch64TargetMachine &TM;
   const AArch64Subtarget &STI;
   const AArch64InstrInfo &TII;
   const AArch64RegisterInfo &TRI;
   const AArch64RegisterBankInfo &RBI;
 
   bool ProduceNonFlagSettingCondBr = false;
 
case TargetOpcode::G_PTR_ADD: {
     MachineIRBuilder MIRBuilder(I);
     emitADD(I.getOperand(0).getReg(), I.getOperand(1), I.getOperand(2),
             MIRBuilder);
     I.eraseFromParent();
     return true;
   }
   case TargetOpcode::G_SADDO:
   case TargetOpcode::G_UADDO:
-  case TargetOpcode::G_SSUBO: {
-    // Emit the operation and get the correct condition code.
+  case TargetOpcode::G_SSUBO:
+  case TargetOpcode::G_USUBO: {
     MachineIRBuilder MIRBuilder(I);
-    auto OpAndCC = emitOverflowOp(Opcode, I.getOperand(0).getReg(),
-                                  I.getOperand(2), I.getOperand(3), MIRBuilder);
+    emitOverflowOp(Opcode, I.getOperand(0).getReg(), I.getOperand(2),
+                   I.getOperand(3), MIRBuilder);
 
     // Now, put the overflow result in the register given by the first operand
     // to the overflow op. CSINC increments the result when the predicate is
     // false, so to get the increment when it's true, we need to use the
     // inverse. In this case, we want to increment when carry is set.
     Register ZReg = AArch64::WZR;
-    auto CsetMI = MIRBuilder
-                      .buildInstr(AArch64::CSINCWr, {I.getOperand(1).getReg()},
-                                  {ZReg, ZReg})
-                      .addImm(getInvertedCondCode(OpAndCC.second));
+    auto CsetMI =
+        MIRBuilder
+            .buildInstr(AArch64::CSINCWr, {I.getOperand(1).getReg()},
+                        {ZReg, ZReg})
+            .addImm(getInvertedCondCode(condCodeForOverflowOp(Opcode)));
+    constrainSelectedInstRegOperands(*CsetMI, TII, TRI, RBI);
+    I.eraseFromParent();
+    return true;
+  }
+
+  case TargetOpcode::G_SADDE:
+  case TargetOpcode::G_UADDE:
+  case TargetOpcode::G_SSUBE:
+  case TargetOpcode::G_USUBE: {
+    MachineIRBuilder MIRBuilder(I);
+
+    auto CarryIn = I.getOperand(4);
+    MachineInstr *Def = MRI.getVRegDef(CarryIn.getReg());
+    bool DefIsCarryOp = isCarryOp(*Def);
+    bool IsPredecessor = I.getPrevNode() == Def;
+    bool UsableCFlag = IsPredecessor && DefIsCarryOp &&
+                       condCodeForOverflowOp(Def->getOpcode()) == AArch64CC::HS;
+    // If we're not immediately preceeded by an unsigned carry op, we can't
+    // directly rely on the condition to have the C flag set appropriately, so
+    // we need to recompute the condition code.
+    if (!UsableCFlag) {
+      Register Dst = MRI.cloneVirtualRegister(CarryIn.getReg());
+      auto OneImm = select12BitValueWithLeftShift(1);
+      // Sets the C flag to match CarryIn
+      emitInstr(AArch64::SUBWri, {Dst}, {CarryIn}, MIRBuilder, OneImm);
+    }
+
+    emitOverflowOp(Opcode, I.getOperand(0).getReg(), I.getOperand(2),
+                   I.getOperand(3), MIRBuilder);
+
+    // Save the overflow result in a register as in the previous case.
+    Register ZReg = AArch64::WZR;
+    auto CsetMI =
+        MIRBuilder
+            .addImm(getInvertedCondCode(condCodeForOverflowOp(Opcode)));
     constrainSelectedInstRegOperands(*CsetMI, TII, TRI, RBI);
     I.eraseFromParent();
     return true;
   }
 
   case TargetOpcode::G_PTRMASK: {
     Register MaskReg = I.getOperand(2).getReg();
     Optional<int64_t> MaskVal = getConstantVRegSExtVal(MaskReg, MRI);
const std::array<std::array<unsigned, 2>, 5> OpcTable{
       {{AArch64::SUBSXri, AArch64::SUBSWri},
        {AArch64::SUBSXrs, AArch64::SUBSWrs},
        {AArch64::SUBSXrr, AArch64::SUBSWrr},
        {AArch64::ADDSXri, AArch64::ADDSWri},
        {AArch64::SUBSXrx, AArch64::SUBSWrx}}};
   return emitAddSub(OpcTable, Dst, LHS, RHS, MIRBuilder);
 }
 
+MachineInstr *AArch64InstructionSelector::emitAdcSbc(
+    const std::array<unsigned, 2> &SizeToOpcode, Register Dst,
+    MachineOperand &LHS, MachineOperand &RHS,
+    MachineIRBuilder &MIRBuilder) const {
+  MachineRegisterInfo &MRI = MIRBuilder.getMF().getRegInfo();
+  assert(LHS.isReg() && RHS.isReg() && "Expected register operands?");
+  auto Ty = MRI.getType(LHS.getReg());
+  assert(!Ty.isVector() && "Expected a scalar or pointer?");
+  unsigned Size = Ty.getSizeInBits();
+  assert((Size == 32 || Size == 64) && "Expected a 32-bit or 64-bit type only");
+  bool Is32Bit = Size == 32;
+  return emitInstr(SizeToOpcode[Is32Bit], {Dst}, {LHS, RHS}, MIRBuilder);
+}
+
+MachineInstr *
+AArch64InstructionSelector::emitADCS(Register Dst, MachineOperand &LHS,
+                                     MachineOperand &RHS,
+                                     MachineIRBuilder &MIRBuilder) const {
+  const std::array<unsigned, 2> Opcodes = {AArch64::ADCSXr, AArch64::ADCSXr};
+  return emitAdcSbc(Opcodes, Dst, LHS, RHS, MIRBuilder);
+}
+
+MachineInstr *
+AArch64InstructionSelector::emitSBCS(Register Dst, MachineOperand &LHS,
+                                     MachineOperand &RHS,
+                                     MachineIRBuilder &MIRBuilder) const {
+  const std::array<unsigned, 2> Opcodes = {AArch64::SBCSXr, AArch64::SBCSXr};
+  return emitAdcSbc(Opcodes, Dst, LHS, RHS, MIRBuilder);
+}
+
 MachineInstr *
 AArch64InstructionSelector::emitCMN(MachineOperand &LHS, MachineOperand &RHS,
                                     MachineIRBuilder &MIRBuilder) const {
   MachineRegisterInfo &MRI = MIRBuilder.getMF().getRegInfo();
   bool Is32Bit = (MRI.getType(LHS.getReg()).getSizeInBits() == 32);
   auto RC = Is32Bit ? &AArch64::GPR32RegClass : &AArch64::GPR64RegClass;
   return emitADDS(MRI.createVirtualRegister(RC), LHS, RHS, MIRBuilder);
 }
AArch64InstructionSelector::emitCSetForICMP(Register DefReg, unsigned Pred,
   auto I =
       MIRBuilder
     .buildInstr(AArch64::CSINCWr, {DefReg}, {Register(AArch64::WZR), Register(AArch64::WZR)})
           .addImm(InvCC);
   constrainSelectedInstRegOperands(*I, TII, TRI, RBI);
   return &*I;
 }
 
-std::pair<MachineInstr *, AArch64CC::CondCode>
-AArch64InstructionSelector::emitOverflowOp(unsigned Opcode, Register Dst,
-                                           MachineOperand &LHS,
-                                           MachineOperand &RHS,
-                                           MachineIRBuilder &MIRBuilder) const {
+AArch64CC::CondCode
+AArch64InstructionSelector::condCodeForOverflowOp(unsigned Opcode) const {
+  switch (Opcode) {
+  default:
+    llvm_unreachable("Unexpected opcode!");
+  case TargetOpcode::G_SADDO:
+  case TargetOpcode::G_SSUBO:
+  case TargetOpcode::G_SADDE:
+  case TargetOpcode::G_SSUBE:
+    return AArch64CC::VS;
+  case TargetOpcode::G_UADDO:
+  case TargetOpcode::G_USUBO:
+  case TargetOpcode::G_UADDE:
+  case TargetOpcode::G_USUBE:
+    return AArch64CC::HS;
+  }
+}
+
+MachineInstr *AArch64InstructionSelector::emitOverflowOp(
+    unsigned Opcode, Register Dst, MachineOperand &LHS, MachineOperand &RHS,
   switch (Opcode) {
   default:
     llvm_unreachable("Unexpected opcode!");
   case TargetOpcode::G_SADDO:
-    return std::make_pair(emitADDS(Dst, LHS, RHS, MIRBuilder), AArch64CC::VS);
   case TargetOpcode::G_UADDO:
-    return std::make_pair(emitADDS(Dst, LHS, RHS, MIRBuilder), AArch64CC::HS);
+    return emitADDS(Dst, LHS, RHS, MIRBuilder);
   case TargetOpcode::G_SSUBO:
-    return std::make_pair(emitSUBS(Dst, LHS, RHS, MIRBuilder), AArch64CC::VS);
+  case TargetOpcode::G_USUBO:
+    return emitSUBS(Dst, LHS, RHS, MIRBuilder);
+  case TargetOpcode::G_SADDE:
+  case TargetOpcode::G_UADDE:
+    return emitADCS(Dst, LHS, RHS, MIRBuilder);
+  case TargetOpcode::G_SSUBE:
+  case TargetOpcode::G_USUBE:
+    return emitSBCS(Dst, LHS, RHS, MIRBuilder);
   }
 }
 
 bool AArch64InstructionSelector::tryOptSelect(MachineInstr &I) const {
   MachineIRBuilder MIB(I);
   MachineRegisterInfo &MRI = *MIB.getMRI();
   // We want to recognize this pattern:
   //
bool AArch64InstructionSelector::isDef32(const MachineInstr &MI) const {
   case TargetOpcode::COPY:
   case TargetOpcode::G_BITCAST:
   case TargetOpcode::G_TRUNC:
   case TargetOpcode::G_PHI:
     return false;
   }
 }
 
+bool AArch64InstructionSelector::isCarryOp(const MachineInstr &MI) const {
+  switch (MI.getOpcode()) {
+  case TargetOpcode::G_SADDE:
+  case TargetOpcode::G_UADDE:
+  case TargetOpcode::G_SSUBE:
+  case TargetOpcode::G_USUBE:
+  case TargetOpcode::G_SADDO:
+  case TargetOpcode::G_UADDO:
+  case TargetOpcode::G_SSUBO:
+  case TargetOpcode::G_USUBO:
+    return true;
+  default:
+    return false;
+  }
+}
 
 // Perform fixups on the given PHI instruction's operands to force them all
 // to be the same as the destination regbank.
 static void fixupPHIOpBanks(MachineInstr &MI, MachineRegisterInfo &MRI,
                             const AArch64RegisterBankInfo &RBI) {
   assert(MI.getOpcode() == TargetOpcode::G_PHI && "Expected a G_PHI");
   Register DstReg = MI.getOperand(0).getReg();
   const RegisterBank *DstRB = MRI.getRegBankOrNull(DstReg);

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp

AArch64LegalizerInfo::AArch64LegalizerInfo(const AArch64Subtarget &ST)
 
   getActionDefinitionsBuilder({G_SREM, G_UREM})
       .lowerFor({s1, s8, s16, s32, s64});
 
   getActionDefinitionsBuilder({G_SMULO, G_UMULO}).lowerFor({{s64, s1}});
 
   getActionDefinitionsBuilder({G_SMULH, G_UMULH}).legalFor({s32, s64});
 
-  getActionDefinitionsBuilder({G_UADDE, G_USUBE, G_SADDO, G_SSUBO, G_UADDO})
+  getActionDefinitionsBuilder(
+      {G_SADDE, G_SSUBE, G_UADDE, G_USUBE, G_SADDO, G_SSUBO, G_UADDO, G_USUBO})
       .legalFor({{s32, s1}, {s64, s1}})
       .minScalar(0, s32);
 
   getActionDefinitionsBuilder({G_FADD, G_FSUB, G_FMUL, G_FDIV, G_FNEG})
       .legalFor({s32, s64, v2s64, v4s32, v2s32})
       .clampNumElements(0, v2s32, v4s32)
       .clampNumElements(0, v2s64, v2s64);
 

llvm/test/CodeGen/AArch64/GlobalISel/legalize-arith-128.ll

+; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+; RUN: llc -O3 --mtriple aarch64-unknown-unknown -stop-after=legalizer --global-isel=1 %s -o - -verify-machineinstrs | FileCheck %s
+
+define void @add(i128* nocapture, i128, i128) {
+  ; CHECK-LABEL: name: add
+  ; CHECK: bb.1 (%ir-block.3):
+  ; CHECK:   liveins: $x0, $x2, $x3, $x4, $x5
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(s64) = COPY $x2
+  ; CHECK:   [[COPY2:%[0-9]+]]:_(s64) = COPY $x3
+  ; CHECK:   [[COPY3:%[0-9]+]]:_(s64) = COPY $x4
+  ; CHECK:   [[COPY4:%[0-9]+]]:_(s64) = COPY $x5
+  ; CHECK:   [[UADDO:%[0-9]+]]:_(s64), [[UADDO1:%[0-9]+]]:_(s1) = G_UADDO [[COPY3]], [[COPY1]]
+  ; CHECK:   [[UADDE:%[0-9]+]]:_(s64), [[UADDE1:%[0-9]+]]:_(s1) = G_UADDE [[COPY4]], [[COPY2]], [[UADDO1]]
+  ; CHECK:   [[MV:%[0-9]+]]:_(s128) = G_MERGE_VALUES [[UADDO]](s64), [[UADDE]](s64)
+  ; CHECK:   G_STORE [[MV]](s128), [[COPY]](p0) :: (store 16 into %ir.0)
+  ; CHECK:   RET_ReallyLR
+  %4 = add nsw i128 %2, %1
+  store i128 %4, i128* %0, align 16
+  ret void
+}
+
+define void @sub(i128* nocapture, i128, i128) {
+  ; CHECK-LABEL: name: sub
+  ; CHECK: bb.1 (%ir-block.3):
+  ; CHECK:   liveins: $x0, $x2, $x3, $x4, $x5
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(s64) = COPY $x2
+  ; CHECK:   [[COPY2:%[0-9]+]]:_(s64) = COPY $x3
+  ; CHECK:   [[COPY3:%[0-9]+]]:_(s64) = COPY $x4
+  ; CHECK:   [[COPY4:%[0-9]+]]:_(s64) = COPY $x5
+  ; CHECK:   [[USUBO:%[0-9]+]]:_(s64), [[USUBO1:%[0-9]+]]:_(s1) = G_USUBO [[COPY3]], [[COPY1]]
+  ; CHECK:   [[USUBE:%[0-9]+]]:_(s64), [[USUBE1:%[0-9]+]]:_(s1) = G_USUBE [[COPY4]], [[COPY2]], [[USUBO1]]
+  ; CHECK:   [[MV:%[0-9]+]]:_(s128) = G_MERGE_VALUES [[USUBO]](s64), [[USUBE]](s64)
+  ; CHECK:   G_STORE [[MV]](s128), [[COPY]](p0) :: (store 16 into %ir.0)
+  ; CHECK:   RET_ReallyLR
+  %4 = sub nsw i128 %2, %1
+  store i128 %4, i128* %0, align 16
+  ret void
+}
+

llvm/test/CodeGen/AArch64/GlobalISel/legalizer-info-validation.mir

 # DEBUG-NEXT: G_SELECT (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: G_UADDO (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. the first uncovered type index: 2, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_UADDE (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
+# DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. the first uncovered type index: 2, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_USUBO (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
-# DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
-# DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
+# DEBUG-NEXT: .. the first uncovered type index: 2, OK
+# DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_USUBE (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. the first uncovered type index: 2, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_SADDO (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. the first uncovered type index: 2, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_SADDE (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
-# DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
-# DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: .. the first uncovered type index: 2, OK
+# DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_SSUBO (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. the first uncovered type index: 2, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_SSUBE (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
-# DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
-# DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
+# DEBUG-NEXT: .. the first uncovered type index: 2, OK
+# DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_UMULO (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. the first uncovered type index: 2, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_SMULO (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. the first uncovered type index: 2, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_UMULH (opcode {{[0-9]+}}): 1 type index, 0 imm indices

llvm/test/CodeGen/AArch64/GlobalISel/select-add-carry.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -march=aarch64 -run-pass=instruction-select -verify-machineinstrs %s -o - | FileCheck %s
+
+---
+name:            uadd_wide
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+machineFunctionInfo: {}
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2, $x3
+    ; CHECK-LABEL: name: uadd_wide
+    ; CHECK: liveins: $x0, $x1, $x2, $x3
+    ; CHECK: %i0:gpr64 = COPY $x0
+    ; CHECK: %i1:gpr64 = COPY $x1
+    ; CHECK: %i2:gpr64 = COPY $x2
+    ; CHECK: %i3:gpr64 = COPY $x3
+    ; CHECK: %o0:gpr64 = ADDSXrr %i0, %i1, implicit-def $nzcv
+    ; CHECK: %c1:gpr32 = CSINCWr $wzr, $wzr, 3, implicit $nzcv
+    ; CHECK: %o1:gpr64 = ADCSXr %i1, %i2, implicit-def $nzcv, implicit $nzcv
+    ; CHECK: %c2:gpr32 = CSINCWr $wzr, $wzr, 3, implicit $nzcv
+    ; CHECK: %o2:gpr64 = ADCSXr %i2, %i3, implicit-def $nzcv, implicit $nzcv
+    ; CHECK: %c3:gpr32 = CSINCWr $wzr, $wzr, 3, implicit $nzcv
+    ; CHECK: $x0 = COPY %o0
+    ; CHECK: $x1 = COPY %o1
+    ; CHECK: $x2 = COPY %o2
+    %i0:gpr(s64) = COPY $x0
+    %i1:gpr(s64) = COPY $x1
+    %i2:gpr(s64) = COPY $x2
+    %i3:gpr(s64) = COPY $x3
+    %o0:gpr(s64), %c1:gpr(s1) = G_UADDO %i0, %i1
+    %o1:gpr(s64), %c2:gpr(s1) = G_UADDE %i1, %i2, %c1
+    %o2:gpr(s64), %c3:gpr(s1) = G_UADDE %i2, %i3, %c2
+    $x0 = COPY %o0
+    $x1 = COPY %o1
+    $x2 = COPY %o2
+
+...
+---
+name:            uadde_interrupted
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+machineFunctionInfo: {}
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2
+    ; CHECK-LABEL: name: uadde_interrupted
+    ; CHECK: liveins: $x0, $x1, $x2
+    ; CHECK: %i0:gpr64 = COPY $x0
+    ; CHECK: %i1:gpr64 = COPY $x1
+    ; CHECK: %i2:gpr64 = COPY $x2
+    ; CHECK: %o0:gpr64 = ADDSXrr %i0, %i1, implicit-def $nzcv
+    ; CHECK: %c0:gpr32common = CSINCWr $wzr, $wzr, 3, implicit $nzcv
+    ; CHECK: %rude:gpr64 = ADDXrr %i0, %i2
+    ; CHECK: [[SUBWri:%[0-9]+]]:gpr32sp = SUBWri %c0, 1, 0
+    ; CHECK: %o1:gpr64 = ADCSXr %i1, %i2, implicit-def $nzcv, implicit $nzcv
+    ; CHECK: %c1:gpr32 = CSINCWr $wzr, $wzr, 3, implicit $nzcv
+    ; CHECK: $x0 = COPY %o0
+    ; CHECK: $x1 = COPY %o1
+    ; CHECK: $x2 = COPY %rude
+    %i0:gpr(s64) = COPY $x0
+    %i1:gpr(s64) = COPY $x1
+    %i2:gpr(s64) = COPY $x2
+    %o0:gpr(s64), %c0:gpr(s1) = G_UADDO %i0, %i1
+    %rude:gpr(s64) = G_ADD %i0, %i2
+    %o1:gpr(s64), %c1:gpr(s1) = G_UADDE %i1, %i2, %c0
+    $x0 = COPY %o0
+    $x1 = COPY %o1
+    $x2 = COPY %rude
+
+...
+---
+name:            uadde_from_saddo
+alignment:       4
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+machineFunctionInfo: {}
+body:             |
+  bb.1:
+    liveins: $x0, $x1, $x2
+    ; CHECK-LABEL: name: uadde_from_saddo
+    ; CHECK: liveins: $x0, $x1, $x2
+    ; CHECK: %i0:gpr64 = COPY $x0
+    ; CHECK: %i1:gpr64 = COPY $x1
+    ; CHECK: %i2:gpr64 = COPY $x2
+    ; CHECK: %o0:gpr64 = ADDSXrr %i0, %i1, implicit-def $nzcv
+    ; CHECK: %c0:gpr32common = CSINCWr $wzr, $wzr, 7, implicit $nzcv
+    ; CHECK: [[SUBWri:%[0-9]+]]:gpr32sp = SUBWri %c0, 1, 0
+    ; CHECK: %o1:gpr64 = ADCSXr %i1, %i2, implicit-def $nzcv, implicit $nzcv
+    ; CHECK: %c1:gpr32 = CSINCWr $wzr, $wzr, 3, implicit $nzcv
+    ; CHECK: $x0 = COPY %o0
+    ; CHECK: $x1 = COPY %o1
+    %i0:gpr(s64) = COPY $x0
+    %i1:gpr(s64) = COPY $x1
+    %i2:gpr(s64) = COPY $x2
+    %o0:gpr(s64), %c0:gpr(s1) = G_SADDO %i0, %i1
+    %o1:gpr(s64), %c1:gpr(s1) = G_UADDE %i1, %i2, %c0
+    $x0 = COPY %o0
+    $x1 = COPY %o1
+
+...