[GlobalISel][AMDGPU] Legalize saturating add/subtract

[arsenm]

Summary

Add support in LegalizerHelper for lowering G_SADDSAT etc. either using
add/subtract-with-overflow or using max/min instructions.

Enable this lowering for AMDGPU so it can be tested. The legalization
rules are still approximate and skips out on using the clamp bit to
treat these as legal, which has never been used before. This also
doesn't yet try to deal with expanding SALU cases.

[merge_guards_bot]

Unit tests: pass. 62011 tests passed, 0 failed and 783 were skipped.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[foad]

Add legalize tests for signed 16-bit operations.

[merge_guards_bot]

Unit tests: pass. 62011 tests passed, 0 failed and 783 were skipped.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[arsenm]

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

3861–3864

I think this won't produce what we want with VOP3P instructions. For v3+s16, we would want G_UMIN to fewerElementsVector to v2s16, which won't be any of these. We would get vector min after lowering

4536

You can do .getReg(0) instead of ->getOperand(0).getReg()

4549–4550

Just .getReg(0)/1 works

4551–4552

Just passing -1/0 to buildConstant should work

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

309

Bot is wrong here

llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-sat.ll

4

Should probably just use update_mir_test_checks on this

llvm/test/CodeGen/AMDGPU/GlobalISel/legalize-sat.mir

3

This should get an explicit -mcpu. Maybe a runline for with/without i16 instructions

255

Should add at least a v2s16, v3s16, v4s16, v2i32, and s64 test

[arsenm]

llvm/test/CodeGen/AMDGPU/GlobalISel/legalize-sat.mir

3

Really need at least a VI and gfx9+

[arsenm]

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

4532

I think it would also be beneficial to implement this in terms of two helper functions for the two different expansions. That way it's more flexible if a target wants to contextually use different strategies in a custom lowering

[foad]

Rebase and address some review comments.

[foad]

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

3861–3864

Right, this requires more thought. I was hoping to reuse the exisiting isSupported logic without thinking too hard about it.

4532

Are there precedents for that kind of helper function that the target calls directly, instead of via the standard lower() action?

llvm/test/CodeGen/AMDGPU/GlobalISel/legalize-sat.mir

3

What sort of run line do I need to disable i16 instructions?

255

That would require G_SADDO (et al) to be legalizable for all those types, which it currently is not. Do I need to tackle that first, before proceeding with this patch?

[merge_guards_bot]

Unit tests: fail. 62131 tests passed, 1 failed and 808 were skipped.

failed: libc++.std/thread/thread_mutex/thread_mutex_requirements/thread_mutex_requirements_mutex/thread_mutex_class/try_lock.pass.cpp

clang-tidy: fail. clang-tidy found 0 errors and 4 warnings.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[arsenm]

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

4532

We use Helper.lowerFMinNumMaxNum directly in AMDGPULegalizerInfo.

I'm specifically thinking we want to switch which expansion is used on AMDGPU based on whether it's scalar or vector, so we would make these legal and then use the helpers in applyMappingImpl

llvm/test/CodeGen/AMDGPU/GlobalISel/legalize-sat.mir

3

Use a target without one. I usually use a Tahiti, Fiji, and gfx900 run line

255

You can add the tests and use -global-isel-abort=0, and they just won't legalize and defer it to a later patch. Handling the vector cases at least should be as easy as adding the instructions to the fewerElementsVector switch

[aemerson]

When you commit can you split this into one patch to add the opcode/irtranslator support and another for the legalizer changes.

[arsenm]

Can't we also set op_sel on v_add_i32 to get saturating behavior?

[foad]

Split opcode/irtranslator stuff out into D76600.

Use separate helper functions for expanding in terms of max/min or in terms of addo/subo.

[arsenm]

Can you also add some end to end IR tests make sure to stress SALU vs. VALU for a mix of scalar and vector types

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

1969–1970

By scalar vs. vector, I meant SGPR vs. VGPR. I haven't looked at whether this registers a real improvement here

[arsenm]

ping?

[arsenm]

ping. Are you going to get back to this soon, or should I adopt this? This is on the shortlist of remaining operations falling back in the OpenCL conformance tests

[foad]

In D73051#2133179, @arsenm wrote:

ping. Are you going to get back to this soon, or should I adopt this? This is on the shortlist of remaining operations falling back in the OpenCL conformance tests

In the short term I don't have time to work on this, so I would be happy for you to commandeer it.

[arsenm]

Rebase on other patches. Don't use custom lowering, and define legality rules as a placeholder until the clamp modifier is used

[arsenm]

Map to VALU for now

[arsenm]

Add gfx10 checks and expect different legality in the next patch

[arsenm]

ping

[foad]

LGTM, thanks.

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

269

This comment seems to be in an odd place (not next to the saturating operations) and/or redundant with the assert just above?

llvm/test/CodeGen/AMDGPU/GlobalISel/legalize-saddsat.mir

491–492 ↗

(On Diff #278040)

Just curious: can't G_CONSTANT represent vector constants directly?

[arsenm]

b35833b84e95c42ca7166232e193b2b8e221b56a

llvm/test/CodeGen/AMDGPU/GlobalISel/legalize-saddsat.mir

491–492 ↗

(On Diff #278040)

No

llvm/include/llvm/CodeGen/GlobalISel/LegalizerHelper.h

 #include "llvm/CodeGen/GlobalISel/CallLowering.h"
 #include "llvm/CodeGen/GlobalISel/MachineIRBuilder.h"
 #include "llvm/CodeGen/LowLevelType.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/RuntimeLibcalls.h"
 
 namespace llvm {
 // Forward declarations.
+struct LegalityQuery;
 class LegalizerInfo;
 class Legalizer;
 class MachineRegisterInfo;
 class GISelChangeObserver;
 
 class LegalizerHelper {
 public:
   enum LegalizeResult {
private:
 
   /// Perform generic multiplication of values held in multiple registers.
   /// Generated instructions use only types NarrowTy and i1.
   /// Destination can be same or two times size of the source.
   void multiplyRegisters(SmallVectorImpl<Register> &DstRegs,
                          ArrayRef<Register> Src1Regs,
                          ArrayRef<Register> Src2Regs, LLT NarrowTy);
 
+  /// Helper function to test whether an operation is supported by the target.
+  bool isSupported(const LegalityQuery &Q);
+
 public:
   LegalizeResult fewerElementsVectorImplicitDef(MachineInstr &MI,
                                                 unsigned TypeIdx, LLT NarrowTy);
 
   /// Legalize a simple vector instruction where all operands are the same type
   /// by splitting into multiple components.
   LegalizeResult fewerElementsVectorBasic(MachineInstr &MI, unsigned TypeIdx,
                                           LLT NarrowTy);
public:
   LegalizeResult lowerShuffleVector(MachineInstr &MI);
   LegalizeResult lowerDynStackAlloc(MachineInstr &MI);
   LegalizeResult lowerExtract(MachineInstr &MI);
   LegalizeResult lowerInsert(MachineInstr &MI);
   LegalizeResult lowerSADDO_SSUBO(MachineInstr &MI);
   LegalizeResult lowerBswap(MachineInstr &MI);
   LegalizeResult lowerBitreverse(MachineInstr &MI);
   LegalizeResult lowerReadRegister(MachineInstr &MI);
+  LegalizeResult lowerUADDSAT_USUBSAT(MachineInstr &MI);
+  LegalizeResult lowerSADDSAT_SSUBSAT(MachineInstr &MI);
 
 private:
   MachineRegisterInfo &MRI;
   const LegalizerInfo &LI;
   /// To keep track of changes made by the LegalizerHelper.
   GISelChangeObserver &Observer;
 };
 

llvm/include/llvm/Support/TargetOpcodes.def

 // Multiply two numbers at twice the incoming bit width (unsigned) and return
 // the high half of the result.
 HANDLE_TARGET_OPCODE(G_UMULH)
 
 // Multiply two numbers at twice the incoming bit width (signed) and return
 // the high half of the result.
 HANDLE_TARGET_OPCODE(G_SMULH)
 
+/// Generic saturating unsigned addition.
+HANDLE_TARGET_OPCODE(G_UADDSAT)
+
+/// Generic saturating signed addition.
+HANDLE_TARGET_OPCODE(G_SADDSAT)
+
+/// Generic saturating unsigned subtraction.
+HANDLE_TARGET_OPCODE(G_USUBSAT)
+
+/// Generic saturating signed subtraction.
+HANDLE_TARGET_OPCODE(G_SSUBSAT)
+
 /// Generic FP addition.
 HANDLE_TARGET_OPCODE(G_FADD)
 
 /// Generic FP subtraction.
 HANDLE_TARGET_OPCODE(G_FSUB)
 
 /// Generic FP multiplication.
 HANDLE_TARGET_OPCODE(G_FMUL)

llvm/include/llvm/Target/GenericOpcodes.td

 def G_SMULH : GenericInstruction {
   let OutOperandList = (outs type0:$dst);
   let InOperandList = (ins type0:$src1, type0:$src2);
   let hasSideEffects = 0;
   let isCommutable = 1;
 }
 
 //------------------------------------------------------------------------------
+// Saturating ops
+//------------------------------------------------------------------------------
+
+// Generic saturating unsigned addition.
+def G_UADDSAT : GenericInstruction {
+  let OutOperandList = (outs type0:$dst);
+  let InOperandList = (ins type0:$src1, type0:$src2);
+  let hasSideEffects = 0;
+  let isCommutable = 1;
+}
+
+// Generic saturating signed addition.
+def G_SADDSAT : GenericInstruction {
+  let OutOperandList = (outs type0:$dst);
+  let InOperandList = (ins type0:$src1, type0:$src2);
+  let hasSideEffects = 0;
+  let isCommutable = 1;
+}
+
+// Generic saturating unsigned subtraction.
+def G_USUBSAT : GenericInstruction {
+  let OutOperandList = (outs type0:$dst);
+  let InOperandList = (ins type0:$src1, type0:$src2);
+  let hasSideEffects = 0;
+  let isCommutable = 0;
+}
+
+// Generic saturating signed subtraction.
+def G_SSUBSAT : GenericInstruction {
+  let OutOperandList = (outs type0:$dst);
+  let InOperandList = (ins type0:$src1, type0:$src2);
+  let hasSideEffects = 0;
+  let isCommutable = 0;
+}
+
+//------------------------------------------------------------------------------
 // Floating Point Unary Ops.
 //------------------------------------------------------------------------------
 
 def G_FNEG : GenericInstruction {
   let OutOperandList = (outs type0:$dst);
   let InOperandList = (ins type0:$src);
   let hasSideEffects = 0;
 }

llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

bool IRTranslator::translateKnownIntrinsic(const CallInst &CI, Intrinsic::ID ID,
   case Intrinsic::usub_with_overflow:
     return translateOverflowIntrinsic(CI, TargetOpcode::G_USUBO, MIRBuilder);
   case Intrinsic::ssub_with_overflow:
     return translateOverflowIntrinsic(CI, TargetOpcode::G_SSUBO, MIRBuilder);
   case Intrinsic::umul_with_overflow:
     return translateOverflowIntrinsic(CI, TargetOpcode::G_UMULO, MIRBuilder);
   case Intrinsic::smul_with_overflow:
     return translateOverflowIntrinsic(CI, TargetOpcode::G_SMULO, MIRBuilder);
+  case Intrinsic::uadd_sat:
+    return translateBinaryOp(TargetOpcode::G_UADDSAT, CI, MIRBuilder);
+  case Intrinsic::sadd_sat:
+    return translateBinaryOp(TargetOpcode::G_SADDSAT, CI, MIRBuilder);
+  case Intrinsic::usub_sat:
+    return translateBinaryOp(TargetOpcode::G_USUBSAT, CI, MIRBuilder);
+  case Intrinsic::ssub_sat:
+    return translateBinaryOp(TargetOpcode::G_SSUBSAT, CI, MIRBuilder);
   case Intrinsic::fmuladd: {
     const TargetMachine &TM = MF->getTarget();
     const TargetLowering &TLI = *MF->getSubtarget().getTargetLowering();
     Register Dst = getOrCreateVReg(CI);
     Register Op0 = getOrCreateVReg(*CI.getArgOperand(0));
     Register Op1 = getOrCreateVReg(*CI.getArgOperand(1));
     Register Op2 = getOrCreateVReg(*CI.getArgOperand(2));
     if (TM.Options.AllowFPOpFusion != FPOpFusion::Strict &&

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

LegalizerHelper::lower(MachineInstr &MI, unsigned TypeIdx, LLT Ty) {
   case G_INSERT:
     return lowerInsert(MI);
   case G_BSWAP:
     return lowerBswap(MI);
   case G_BITREVERSE:
     return lowerBitreverse(MI);
   case G_READ_REGISTER:
     return lowerReadRegister(MI);
+  case G_UADDSAT:
+  case G_USUBSAT:
+    return lowerUADDSAT_USUBSAT(MI);
+  case G_SADDSAT:
+  case G_SSUBSAT:
+    return lowerSADDSAT_SSUBSAT(MI);
   }
 }
 
 LegalizerHelper::LegalizeResult LegalizerHelper::fewerElementsVectorImplicitDef(
     MachineInstr &MI, unsigned TypeIdx, LLT NarrowTy) {
   SmallVector<Register, 2> DstRegs;
 
   unsigned NarrowSize = NarrowTy.getSizeInBits();
LegalizerHelper::narrowScalarSelect(MachineInstr &MI, unsigned TypeIdx,
 
   insertParts(DstReg, DstTy, NarrowTy, DstRegs,
               LeftoverTy, DstLeftoverRegs);
 
   MI.eraseFromParent();
   return Legalized;
 }
 
+bool LegalizerHelper::isSupported(const LegalityQuery &Q) {
+  auto QAction = LI.getAction(Q).Action;
+  return QAction == Legal || QAction == Libcall || QAction == Custom;
+}
+
 LegalizerHelper::LegalizeResult
 LegalizerHelper::lowerBitCount(MachineInstr &MI, unsigned TypeIdx, LLT Ty) {
   unsigned Opc = MI.getOpcode();
   auto &TII = *MI.getMF()->getSubtarget().getInstrInfo();
-  auto isSupported = [this](const LegalityQuery &Q) {
-    auto QAction = LI.getAction(Q).Action;
-    return QAction == Legal || QAction == Libcall || QAction == Custom;
-  };
   switch (Opc) {
   default:
     return UnableToLegalize;
   case TargetOpcode::G_CTLZ_ZERO_UNDEF: {
     // This trivially expands to CTLZ.
     Observer.changingInstr(MI);
     MI.setDesc(TII.get(TargetOpcode::G_CTLZ));
     Observer.changedInstr(MI);
auto ConditionRHS = MIRBuilder.buildICmp(
       IsAdd ? CmpInst::ICMP_SLT : CmpInst::ICMP_SGT, BoolTy, RHS, Zero);
 
   MIRBuilder.buildXor(Dst1, ConditionRHS, ResultLowerThanLHS);
   MI.eraseFromParent();
   return Legalized;
 }
 
 LegalizerHelper::LegalizeResult
+LegalizerHelper::lowerUADDSAT_USUBSAT(MachineInstr &MI) {
+  Register Res = MI.getOperand(0).getReg();
+  Register LHS = MI.getOperand(1).getReg();
+  Register RHS = MI.getOperand(2).getReg();
+  LLT Ty = MRI.getType(Res);
+  LLT BoolTy = Ty.changeElementSize(1);
+  bool IsAdd = MI.getOpcode() == TargetOpcode::G_UADDSAT;
+
+  if (isSupported({TargetOpcode::G_UMIN, {Ty, Ty}})) {
+    // uadd.sat(a, b) -> a + umin(~a, b)
+    // usub.sat(a, b) -> a - umin(a, b)
+    Register Not =
+        IsAdd ? MIRBuilder.buildNot(Ty, LHS)->getOperand(0).getReg() : LHS;
+    auto Min = MIRBuilder.buildUMin(Ty, Not, RHS);
+    unsigned Opcode = IsAdd ? TargetOpcode::G_ADD : TargetOpcode::G_SUB;
+    MIRBuilder.buildInstr(Opcode, {Res}, {LHS, Min});
+  } else {
+    // uadd.sat(a, b) ->
+    //   {r, ov} = uaddo(a, b)
+    //   ov ? 0xffffffff : r
+    // usub.sat(a, b) ->
+    //   {r, ov} = usubo(a, b)
+    //   ov ? 0 : r
+    unsigned Opcode = IsAdd ? TargetOpcode::G_UADDO : TargetOpcode::G_USUBO;
+    auto Addo = MIRBuilder.buildInstr(Opcode, {Ty, BoolTy}, {LHS, RHS});
+    Register Add = Addo->getOperand(0).getReg();
+    Register Overflow = Addo->getOperand(1).getReg();
+    APInt ClampInt(Ty.getScalarSizeInBits(), IsAdd ? -1 : 0, true);
+    auto Clamp = MIRBuilder.buildConstant(Ty, ClampInt);
+    MIRBuilder.buildSelect(Res, Overflow, Clamp, Add);
+  }
+
+  MI.eraseFromParent();
+  return Legalized;
+}
+
+LegalizerHelper::LegalizeResult
+LegalizerHelper::lowerSADDSAT_SSUBSAT(MachineInstr &MI) {
+  Register Dst = MI.getOperand(0).getReg();
+  Register LHS = MI.getOperand(1).getReg();
+  Register RHS = MI.getOperand(2).getReg();
+  LLT Ty = MRI.getType(Dst);
+  LLT BoolTy = Ty.changeElementSize(1);
+  bool IsAdd = MI.getOpcode() == TargetOpcode::G_SADDSAT;
+
+  // sadd.sat(a, b) ->
+  //   {r, ov} = saddo(a, b)
+  //   ov ? (r >>> 31) + 0x80000000 : r
+  // ssub.sat(a, b) ->
+  //   {r, ov} = ssubo(a, b)
+  //   ov ? (r >>> 31) + 0x80000000 : r
+  auto Addo = MIRBuilder.buildInstr(IsAdd ? TargetOpcode::G_SADDO
+                                          : TargetOpcode::G_SSUBO,
+                                    {Ty, BoolTy}, {LHS, RHS});
+  Register Add = Addo->getOperand(0).getReg();
+  Register Overflow = Addo->getOperand(1).getReg();
+  uint64_t NumBits = Ty.getScalarSizeInBits();
+  auto Amount = MIRBuilder.buildConstant(Ty, NumBits - 1);
+  auto Sign = MIRBuilder.buildAShr(Ty, Add, Amount);
+  auto MinVal = MIRBuilder.buildConstant(Ty, APInt::getSignedMinValue(NumBits));
+  auto Clamp = MIRBuilder.buildAdd(Ty, Sign, MinVal);
+  MIRBuilder.buildSelect(Dst, Overflow, Clamp, Add);
+
+  MI.eraseFromParent();
+  return Legalized;
+}
+
+LegalizerHelper::LegalizeResult
 LegalizerHelper::lowerBswap(MachineInstr &MI) {
   Register Dst = MI.getOperand(0).getReg();
   Register Src = MI.getOperand(1).getReg();
   const LLT Ty = MRI.getType(Src);
   unsigned SizeInBytes = Ty.getSizeInBytes();
   unsigned BaseShiftAmt = (SizeInBytes - 1) * 8;
 
   // Swap most and least significant byte, set remaining bytes in Res to zero.

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

getActionDefinitionsBuilder(G_PHI)
     .widenScalarToNextPow2(0, 32)
     .clampMaxNumElements(0, S32, 16)
     .moreElementsIf(isSmallOddVector(0), oneMoreElement(0))
     .legalIf(isPointer(0));
 
   if (ST.has16BitInsts()) {
     getActionDefinitionsBuilder({G_ADD, G_SUB, G_MUL})
       .legalFor({S32, S16})
       .clampScalar(0, S16, S32)
       .scalarize(0);
   } else {
     getActionDefinitionsBuilder({G_ADD, G_SUB, G_MUL})
       .legalFor({S32})
       .clampScalar(0, S32, S32)
       .scalarize(0);
   }
 
AMDGPULegalizerInfo::AMDGPULegalizerInfo(const GCNSubtarget &ST_,
   getActionDefinitionsBuilder({G_UADDO, G_USUBO,
                                G_UADDE, G_SADDE, G_USUBE, G_SSUBE})
     .legalFor({{S32, S1}, {S32, S32}})
     .clampScalar(0, S32, S32)
     .scalarize(0); // TODO: Implement.
 
   getActionDefinitionsBuilder({G_SADDO, G_SSUBO})
     .lower();
+  getActionDefinitionsBuilder({G_UADDSAT, G_SADDSAT, G_USUBSAT, G_SSUBSAT})
+    .lower();
 
   getActionDefinitionsBuilder(G_BITCAST)
     // Don't worry about the size constraint.
     .legalIf(all(isRegisterType(0), isRegisterType(1)))
     // FIXME: Testing hack
     .legalForCartesianProduct({S16, LLT::vector(2, 8), })
     .lower();
 
bool AMDGPULegalizerInfo::legalizeFastUnsafeFDIV(MachineInstr &MI,
 
   if (!MF.getTarget().Options.UnsafeFPMath && ResTy == S64)
     return false;
 
   if (!Unsafe && ResTy == S32 &&
       MF.getInfo<SIMachineFunctionInfo>()->getMode().FP32Denormals)
     return false;
 
   if (auto CLHS = getConstantFPVRegVal(LHS, MRI)) {
     // 1 / x -> RCP(x)
     if (CLHS->isExactlyValue(1.0)) {
       B.buildIntrinsic(Intrinsic::amdgcn_rcp, Res, false)
         .addUse(RHS)
         .setMIFlags(Flags);
 
       MI.eraseFromParent();
       return true;
     }

llvm/test/CodeGen/AArch64/GlobalISel/legalizer-info-validation.mir

 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_UMULH (opcode {{[0-9]+}}): 1 type index, 0 imm indices
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. the first uncovered type index: 1, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_SMULH (opcode {{[0-9]+}}): 1 type index, 0 imm indices
 # DEBUG-NEXT: .. the first uncovered type index: 1, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
+# DEBUG-NEXT: G_UADDSAT (opcode 117): 1 type index, 0 imm indices
+# DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: G_SADDSAT (opcode 118): 1 type index, 0 imm indices
+# DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: G_USUBSAT (opcode 119): 1 type index, 0 imm indices
+# DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: G_SSUBSAT (opcode 120): 1 type index, 0 imm indices
+# DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_FADD (opcode {{[0-9]+}}): 1 type index, 0 imm indices
 # DEBUG-NEXT: .. the first uncovered type index: 1, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_FSUB (opcode {{[0-9]+}}): 1 type index, 0 imm indices
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. the first uncovered type index: 1, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_FMUL (opcode {{[0-9]+}}): 1 type index, 0 imm indices

llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-sat.ll

+; RUN: llc -march=amdgcn -global-isel -stop-after=irtranslator %s -o - | FileCheck %s
+
+; CHECK-LABEL: name: uaddsat_i16
+; CHECK: G_UADDSAT
+define i16 @uaddsat_i16(i16 %lhs, i16 %rhs) {
+  %res = call i16 @llvm.uadd.sat.i16(i16 %lhs, i16 %rhs)
+  ret i16 %res
+}
+declare i16 @llvm.uadd.sat.i16(i16, i16)
+
+; CHECK-LABEL: name: uaddsat_i32
+; CHECK: G_UADDSAT
+define i32 @uaddsat_i32(i32 %lhs, i32 %rhs) {
+  %res = call i32 @llvm.uadd.sat.i32(i32 %lhs, i32 %rhs)
+  ret i32 %res
+}
+declare i32 @llvm.uadd.sat.i32(i32, i32)
+
+; CHECK-LABEL: name: uaddsat_i64
+; CHECK: G_UADDSAT
+define i64 @uaddsat_i64(i64 %lhs, i64 %rhs) {
+  %res = call i64 @llvm.uadd.sat.i64(i64 %lhs, i64 %rhs)
+  ret i64 %res
+}
+declare i64 @llvm.uadd.sat.i64(i64, i64)
+
+; CHECK-LABEL: name: uaddsat_v2i32
+; CHECK: G_UADDSAT
+define <2 x i32> @uaddsat_v2i32(<2 x i32> %lhs, <2 x i32> %rhs) {
+  %res = call <2 x i32> @llvm.uadd.sat.v2i32(<2 x i32> %lhs, <2 x i32> %rhs)
+  ret <2 x i32> %res
+}
+declare <2 x i32> @llvm.uadd.sat.v2i32(<2 x i32>, <2 x i32>)
+
+; CHECK-LABEL: name: saddsat_i16
+; CHECK: G_SADDSAT
+define i16 @saddsat_i16(i16 %lhs, i16 %rhs) {
+  %res = call i16 @llvm.sadd.sat.i16(i16 %lhs, i16 %rhs)
+  ret i16 %res
+}
+declare i16 @llvm.sadd.sat.i16(i16, i16)
+
+; CHECK-LABEL: name: saddsat_i32
+; CHECK: G_SADDSAT
+define i32 @saddsat_i32(i32 %lhs, i32 %rhs) {
+  %res = call i32 @llvm.sadd.sat.i32(i32 %lhs, i32 %rhs)
+  ret i32 %res
+}
+declare i32 @llvm.sadd.sat.i32(i32, i32)
+
+; CHECK-LABEL: name: saddsat_i64
+; CHECK: G_SADDSAT
+define i64 @saddsat_i64(i64 %lhs, i64 %rhs) {
+  %res = call i64 @llvm.sadd.sat.i64(i64 %lhs, i64 %rhs)
+  ret i64 %res
+}
+declare i64 @llvm.sadd.sat.i64(i64, i64)
+
+; CHECK-LABEL: name: saddsat_v2i32
+; CHECK: G_SADDSAT
+define <2 x i32> @saddsat_v2i32(<2 x i32> %lhs, <2 x i32> %rhs) {
+  %res = call <2 x i32> @llvm.sadd.sat.v2i32(<2 x i32> %lhs, <2 x i32> %rhs)
+  ret <2 x i32> %res
+}
+declare <2 x i32> @llvm.sadd.sat.v2i32(<2 x i32>, <2 x i32>)
+
+; CHECK-LABEL: name: usubsat_i16
+; CHECK: G_USUBSAT
+define i16 @usubsat_i16(i16 %lhs, i16 %rhs) {
+  %res = call i16 @llvm.usub.sat.i16(i16 %lhs, i16 %rhs)
+  ret i16 %res
+}
+declare i16 @llvm.usub.sat.i16(i16, i16)
+
+; CHECK-LABEL: name: usubsat_i32
+; CHECK: G_USUBSAT
+define i32 @usubsat_i32(i32 %lhs, i32 %rhs) {
+  %res = call i32 @llvm.usub.sat.i32(i32 %lhs, i32 %rhs)
+  ret i32 %res
+}
+declare i32 @llvm.usub.sat.i32(i32, i32)
+
+; CHECK-LABEL: name: usubsat_i64
+; CHECK: G_USUBSAT
+define i64 @usubsat_i64(i64 %lhs, i64 %rhs) {
+  %res = call i64 @llvm.usub.sat.i64(i64 %lhs, i64 %rhs)
+  ret i64 %res
+}
+declare i64 @llvm.usub.sat.i64(i64, i64)
+
+; CHECK-LABEL: name: usubsat_v2i32
+; CHECK: G_USUBSAT
+define <2 x i32> @usubsat_v2i32(<2 x i32> %lhs, <2 x i32> %rhs) {
+  %res = call <2 x i32> @llvm.usub.sat.v2i32(<2 x i32> %lhs, <2 x i32> %rhs)
+  ret <2 x i32> %res
+}
+declare <2 x i32> @llvm.usub.sat.v2i32(<2 x i32>, <2 x i32>)
+
+; CHECK-LABEL: name: ssubsat_i16
+; CHECK: G_SSUBSAT
+define i16 @ssubsat_i16(i16 %lhs, i16 %rhs) {
+  %res = call i16 @llvm.ssub.sat.i16(i16 %lhs, i16 %rhs)
+  ret i16 %res
+}
+declare i16 @llvm.ssub.sat.i16(i16, i16)
+
+; CHECK-LABEL: name: ssubsat_i32
+; CHECK: G_SSUBSAT
+define i32 @ssubsat_i32(i32 %lhs, i32 %rhs) {
+  %res = call i32 @llvm.ssub.sat.i32(i32 %lhs, i32 %rhs)
+  ret i32 %res
+}
+declare i32 @llvm.ssub.sat.i32(i32, i32)
+
+; CHECK-LABEL: name: ssubsat_i64
+; CHECK: G_SSUBSAT
+define i64 @ssubsat_i64(i64 %lhs, i64 %rhs) {
+  %res = call i64 @llvm.ssub.sat.i64(i64 %lhs, i64 %rhs)
+  ret i64 %res
+}
+declare i64 @llvm.ssub.sat.i64(i64, i64)
+
+; CHECK-LABEL: name: ssubsat_v2i32
+; CHECK: G_SSUBSAT
+define <2 x i32> @ssubsat_v2i32(<2 x i32> %lhs, <2 x i32> %rhs) {
+  %res = call <2 x i32> @llvm.ssub.sat.v2i32(<2 x i32> %lhs, <2 x i32> %rhs)
+  ret <2 x i32> %res
+}
+declare <2 x i32> @llvm.ssub.sat.v2i32(<2 x i32>, <2 x i32>)

llvm/test/CodeGen/AMDGPU/GlobalISel/legalize-sat.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -march=amdgcn -run-pass=legalizer %s -o - | FileCheck %s
+
+---
+name: uaddsat_s16
+body: |
+  bb.0:
+    liveins: $vgpr0, $vgpr1
+
+    ; CHECK-LABEL: name: uaddsat_s16
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
+    ; CHECK: [[COPY2:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
+    ; CHECK: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 65535
+    ; CHECK: [[COPY3:%[0-9]+]]:_(s32) = COPY [[COPY]](s32)
+    ; CHECK: [[AND:%[0-9]+]]:_(s32) = G_AND [[COPY3]], [[C]]
+    ; CHECK: [[COPY4:%[0-9]+]]:_(s32) = COPY [[COPY1]](s32)
+    ; CHECK: [[AND1:%[0-9]+]]:_(s32) = G_AND [[COPY4]], [[C]]
+    ; CHECK: [[ADD:%[0-9]+]]:_(s32) = G_ADD [[AND]], [[AND1]]
+    ; CHECK: [[AND2:%[0-9]+]]:_(s32) = G_AND [[ADD]], [[C]]
+    ; CHECK: [[ICMP:%[0-9]+]]:_(s1) = G_ICMP intpred(ne), [[ADD]](s32), [[AND2]]
+    ; CHECK: [[TRUNC:%[0-9]+]]:_(s16) = G_TRUNC [[ADD]](s32)
+    ; CHECK: [[C1:%[0-9]+]]:_(s16) = G_CONSTANT i16 -1
+    ; CHECK: [[SELECT:%[0-9]+]]:_(s16) = G_SELECT [[ICMP]](s1), [[C1]], [[TRUNC]]
+    ; CHECK: [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[SELECT]](s16)
+    ; CHECK: $vgpr0 = COPY [[ANYEXT]](s32)
+    %3:_(s32) = COPY $vgpr0
+    %0:_(s16) = G_TRUNC %3(s32)
+    %4:_(s32) = COPY $vgpr1
+    %1:_(s16) = G_TRUNC %4(s32)
+    %2:sgpr_64 = COPY $sgpr30_sgpr31
+    %5:_(s16) = G_UADDSAT %0, %1
+    %7:_(s32) = G_ANYEXT %5(s16)
+    $vgpr0 = COPY %7(s32)
+...
+
+---
+name: uaddsat_s32
+body: |
+  bb.0:
+    liveins: $vgpr0, $vgpr1
+
+    ; CHECK-LABEL: name: uaddsat_s32
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
+    ; CHECK: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 -1
+    ; CHECK: [[XOR:%[0-9]+]]:_(s32) = G_XOR [[COPY]], [[C]]
+    ; CHECK: [[UMIN:%[0-9]+]]:_(s32) = G_UMIN [[XOR]], [[COPY1]]
+    ; CHECK: [[ADD:%[0-9]+]]:_(s32) = G_ADD [[COPY]], [[UMIN]]
+    ; CHECK: $vgpr0 = COPY [[ADD]](s32)
+    %0:_(s32) = COPY $vgpr0
+    %1:_(s32) = COPY $vgpr1
+    %2:_(s32) = G_UADDSAT %0, %1
+    $vgpr0 = COPY %2
+...
+
+---
+name: saddsat_s32
+body: |
+  bb.0:
+    liveins: $vgpr0, $vgpr1
+
+    ; CHECK-LABEL: name: saddsat_s32
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
+    ; CHECK: [[ADD:%[0-9]+]]:_(s32) = G_ADD [[COPY]], [[COPY1]]
+    ; CHECK: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
+    ; CHECK: [[ICMP:%[0-9]+]]:_(s1) = G_ICMP intpred(slt), [[ADD]](s32), [[COPY]]
+    ; CHECK: [[ICMP1:%[0-9]+]]:_(s1) = G_ICMP intpred(slt), [[COPY1]](s32), [[C]]
+    ; CHECK: [[XOR:%[0-9]+]]:_(s1) = G_XOR [[ICMP1]], [[ICMP]]
+    ; CHECK: [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 31
+    ; CHECK: [[ASHR:%[0-9]+]]:_(s32) = G_ASHR [[ADD]], [[C1]](s32)
+    ; CHECK: [[C2:%[0-9]+]]:_(s32) = G_CONSTANT i32 -2147483648
+    ; CHECK: [[ADD1:%[0-9]+]]:_(s32) = G_ADD [[ASHR]], [[C2]]
+    ; CHECK: [[SELECT:%[0-9]+]]:_(s32) = G_SELECT [[XOR]](s1), [[ADD1]], [[ADD]]
+    ; CHECK: $vgpr0 = COPY [[SELECT]](s32)
+    %0:_(s32) = COPY $vgpr0
+    %1:_(s32) = COPY $vgpr1
+    %2:_(s32) = G_SADDSAT %0, %1
+    $vgpr0 = COPY %2
+...
+
+---
+name: usubsat_s16
+body: |
+  bb.0:
+    liveins: $vgpr0, $vgpr1
+
+    ; CHECK-LABEL: name: usubsat_s16
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
+    ; CHECK: [[COPY2:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
+    ; CHECK: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 65535
+    ; CHECK: [[COPY3:%[0-9]+]]:_(s32) = COPY [[COPY]](s32)
+    ; CHECK: [[AND:%[0-9]+]]:_(s32) = G_AND [[COPY3]], [[C]]
+    ; CHECK: [[COPY4:%[0-9]+]]:_(s32) = COPY [[COPY1]](s32)
+    ; CHECK: [[AND1:%[0-9]+]]:_(s32) = G_AND [[COPY4]], [[C]]
+    ; CHECK: [[SUB:%[0-9]+]]:_(s32) = G_SUB [[AND]], [[AND1]]
+    ; CHECK: [[AND2:%[0-9]+]]:_(s32) = G_AND [[SUB]], [[C]]
+    ; CHECK: [[ICMP:%[0-9]+]]:_(s1) = G_ICMP intpred(ne), [[SUB]](s32), [[AND2]]
+    ; CHECK: [[TRUNC:%[0-9]+]]:_(s16) = G_TRUNC [[SUB]](s32)
+    ; CHECK: [[C1:%[0-9]+]]:_(s16) = G_CONSTANT i16 0
+    ; CHECK: [[SELECT:%[0-9]+]]:_(s16) = G_SELECT [[ICMP]](s1), [[C1]], [[TRUNC]]
+    ; CHECK: [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[SELECT]](s16)
+    ; CHECK: $vgpr0 = COPY [[ANYEXT]](s32)
+    %3:_(s32) = COPY $vgpr0
+    %0:_(s16) = G_TRUNC %3(s32)
+    %4:_(s32) = COPY $vgpr1
+    %1:_(s16) = G_TRUNC %4(s32)
+    %2:sgpr_64 = COPY $sgpr30_sgpr31
+    %5:_(s16) = G_USUBSAT %0, %1
+    %7:_(s32) = G_ANYEXT %5(s16)
+    $vgpr0 = COPY %7(s32)
+...
+
+---
+name: usubsat_s32
+body: |
+  bb.0:
+    liveins: $vgpr0, $vgpr1
+
+    ; CHECK-LABEL: name: usubsat_s32
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
+    ; CHECK: [[UMIN:%[0-9]+]]:_(s32) = G_UMIN [[COPY]], [[COPY1]]
+    ; CHECK: [[SUB:%[0-9]+]]:_(s32) = G_SUB [[COPY]], [[UMIN]]
+    ; CHECK: $vgpr0 = COPY [[SUB]](s32)
+    %0:_(s32) = COPY $vgpr0
+    %1:_(s32) = COPY $vgpr1
+    %2:_(s32) = G_USUBSAT %0, %1
+    $vgpr0 = COPY %2
+...