[ARM][TTI] Prevents constants in a min(max) or max(min) pattern from being hoisted when in a loop

[MeeraN]

Summary

Changes TTI function getIntImmCostInst to take an additional Instruction parameter, which enables us to be able to check it is part of a min(max())/max(min()) pattern. We can then mark the constant used as free to prevent it being hoisted so SSAT can still be generated.
Required minor changes in some non-ARM backends to allow for the optional parameter to be included.

[dmgreen]

The TTI changes here look OK to me. I think it is probably worth trying to make the ARM check a bit more specific though. We are trying to match a SSAT, not any min/max. It's worth making sure that the subtarget will have an SSAT instruction, and that the size of the type we will be saturating to will fit into an i32.

I'm not sure about having to match a more specific min(max(..)) and max(min(..)) pattern too. That will depend on how some examples look if they only have the max, but won't produce a ssat.

[MeeraN]

Made the checks more specific to min(max()) and max(min()) patterns which should generate SSAT

[MeeraN]

Improved getIntImmCostInst so that the max(min()) case is properly handled by checking the instruction's operand is a min and a select instruction.

[dmgreen]

Thanks. Looks good.

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

328

Can you add something about matching ssat in this comment. You could consider pulling it out into a separate function too.

329

The condition might be better as ((ST->hasV6Ops() && !ST->isThumb()) || ST->isThumb2()), as there are some cpu's (like the cortex-m0) which have V6Ops but not A32 or SSat.

346–347

If this checks for MinSPF == SPF_SMIN first, then in case that is false we needn't look at the minRHS.

I think getUniqueInteger can just be getValue too, if we know it's a ConstantInt?

[MeeraN]

Added new helper function isSSATMinMaxPattern which checks whether the instruction being passed in is a valid max instruction that is part of a min(max()) or max(min()) pattern. Simplified getIntImmCostInst down to make a call to this function.

[dmgreen]

Very nice, LGTM.

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

297

This can use [&]
(it likely doesn't matter a lot, but can prevent a copy).

llvm/include/llvm/Analysis/TargetTransformInfo.h

public:
 
   /// Return the expected cost of materializing for the given integer
   /// immediate of the specified type.
   int getIntImmCost(const APInt &Imm, Type *Ty, TargetCostKind CostKind) const;
 
   /// Return the expected cost of materialization for the given integer
   /// immediate of the specified type for a given instruction. The cost can be
   /// zero if the immediate can be folded into the specified instruction.
-  int getIntImmCostInst(unsigned Opc, unsigned Idx, const APInt &Imm,
-                        Type *Ty, TargetCostKind CostKind) const;
+  int getIntImmCostInst(unsigned Opc, unsigned Idx, const APInt &Imm, Type *Ty,
+                        TargetCostKind CostKind,
+                        Instruction *Inst = nullptr) const;
   int getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx, const APInt &Imm,
                           Type *Ty, TargetCostKind CostKind) const;
 
   /// Return the expected cost for the given integer when optimising
   /// for size. This is different than the other integer immediate cost
   /// functions in that it is subtarget agnostic. This is useful when you e.g.
   /// target one ISA such as Aarch32 but smaller encodings could be possible
   /// with another such as Thumb. This return value is used as a penalty when
public:
   virtual bool haveFastSqrt(Type *Ty) = 0;
   virtual bool isFCmpOrdCheaperThanFCmpZero(Type *Ty) = 0;
   virtual int getFPOpCost(Type *Ty) = 0;
   virtual int getIntImmCodeSizeCost(unsigned Opc, unsigned Idx,
                                     const APInt &Imm, Type *Ty) = 0;
   virtual int getIntImmCost(const APInt &Imm, Type *Ty,
                             TargetCostKind CostKind) = 0;
   virtual int getIntImmCostInst(unsigned Opc, unsigned Idx, const APInt &Imm,
-                                Type *Ty, TargetCostKind CostKind) = 0;
+                                Type *Ty, TargetCostKind CostKind,
+                                Instruction *Inst = nullptr) = 0;
   virtual int getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx,
                                   const APInt &Imm, Type *Ty,
                                   TargetCostKind CostKind) = 0;
   virtual unsigned getNumberOfRegisters(unsigned ClassID) const = 0;
   virtual unsigned getRegisterClassForType(bool Vector,
                                            Type *Ty = nullptr) const = 0;
   virtual const char *getRegisterClassName(unsigned ClassID) const = 0;
   virtual unsigned getRegisterBitWidth(bool Vector) const = 0;
public:
   int getIntImmCodeSizeCost(unsigned Opc, unsigned Idx, const APInt &Imm,
                             Type *Ty) override {
     return Impl.getIntImmCodeSizeCost(Opc, Idx, Imm, Ty);
   }
   int getIntImmCost(const APInt &Imm, Type *Ty,
                     TargetCostKind CostKind) override {
     return Impl.getIntImmCost(Imm, Ty, CostKind);
   }
-  int getIntImmCostInst(unsigned Opc, unsigned Idx, const APInt &Imm,
-                        Type *Ty, TargetCostKind CostKind) override {
-    return Impl.getIntImmCostInst(Opc, Idx, Imm, Ty, CostKind);
+  int getIntImmCostInst(unsigned Opc, unsigned Idx, const APInt &Imm, Type *Ty,
+                        TargetCostKind CostKind,
+                        Instruction *Inst = nullptr) override {
+    return Impl.getIntImmCostInst(Opc, Idx, Imm, Ty, CostKind, Inst);
   }
   int getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx, const APInt &Imm,
                           Type *Ty, TargetCostKind CostKind) override {
     return Impl.getIntImmCostIntrin(IID, Idx, Imm, Ty, CostKind);
   }
   unsigned getNumberOfRegisters(unsigned ClassID) const override {
     return Impl.getNumberOfRegisters(ClassID);
   }

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

public:
   }
 
   unsigned getIntImmCost(const APInt &Imm, Type *Ty,
                          TTI::TargetCostKind CostKind) {
     return TTI::TCC_Basic;
   }
 
   unsigned getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm,
-                             Type *Ty, TTI::TargetCostKind CostKind) {
+                             Type *Ty, TTI::TargetCostKind CostKind,
+                             Instruction *Inst = nullptr) {
     return TTI::TCC_Free;
   }
 
   unsigned getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx,
                                const APInt &Imm, Type *Ty,
                                TTI::TargetCostKind CostKind) {
     return TTI::TCC_Free;
   }

llvm/lib/Analysis/TargetTransformInfo.cpp

 
 int TargetTransformInfo::getIntImmCost(const APInt &Imm, Type *Ty,
                                        TTI::TargetCostKind CostKind) const {
   int Cost = TTIImpl->getIntImmCost(Imm, Ty, CostKind);
   assert(Cost >= 0 && "TTI should not produce negative costs!");
   return Cost;
 }
 
-int
-TargetTransformInfo::getIntImmCostInst(unsigned Opcode, unsigned Idx,
+int TargetTransformInfo::getIntImmCostInst(unsigned Opcode, unsigned Idx,
                                            const APInt &Imm, Type *Ty,
-                                       TTI::TargetCostKind CostKind) const {
-  int Cost = TTIImpl->getIntImmCostInst(Opcode, Idx, Imm, Ty, CostKind);
+                                           TTI::TargetCostKind CostKind,
+                                           Instruction *Inst) const {
+  int Cost = TTIImpl->getIntImmCostInst(Opcode, Idx, Imm, Ty, CostKind, Inst);
   assert(Cost >= 0 && "TTI should not produce negative costs!");
   return Cost;
 }
 
 int
 TargetTransformInfo::getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx,
                                          const APInt &Imm, Type *Ty,
                                          TTI::TargetCostKind CostKind) const {

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

public:
 
   /// \name Scalar TTI Implementations
   /// @{
 
   using BaseT::getIntImmCost;
   int getIntImmCost(int64_t Val);
   int getIntImmCost(const APInt &Imm, Type *Ty, TTI::TargetCostKind CostKind);
   int getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm,
-                        Type *Ty, TTI::TargetCostKind CostKind);
+                        Type *Ty, TTI::TargetCostKind CostKind,
+                        Instruction *Inst = nullptr);
   int getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx, const APInt &Imm,
                           Type *Ty, TTI::TargetCostKind CostKind);
   TTI::PopcntSupportKind getPopcntSupport(unsigned TyWidth);
 
   /// @}
 
   /// \name Vector TTI Implementations
   /// @{

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

for (unsigned ShiftVal = 0; ShiftVal < BitSize; ShiftVal += 64) {
     Cost += getIntImmCost(Val);
   }
   // We need at least one instruction to materialze the constant.
   return std::max(1, Cost);
 }
 
 int AArch64TTIImpl::getIntImmCostInst(unsigned Opcode, unsigned Idx,
                                       const APInt &Imm, Type *Ty,
-                                      TTI::TargetCostKind CostKind) {
+                                      TTI::TargetCostKind CostKind,
+                                      Instruction *Inst) {
   assert(Ty->isIntegerTy());
 
   unsigned BitSize = Ty->getPrimitiveSizeInBits();
   // There is no cost model for constants with a bit size of 0. Return TCC_Free
   // here, so that constant hoisting will ignore this constant.
   if (BitSize == 0)
     return TTI::TCC_Free;
 

llvm/lib/Target/ARM/ARMTargetTransformInfo.h

public:
 
   int getIntImmCodeSizeCost(unsigned Opcode, unsigned Idx, const APInt &Imm,
                             Type *Ty);
 
   using BaseT::getIntImmCost;
   int getIntImmCost(const APInt &Imm, Type *Ty, TTI::TargetCostKind CostKind);
 
   int getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm,
-                        Type *Ty, TTI::TargetCostKind CostKind);
+                        Type *Ty, TTI::TargetCostKind CostKind,
+                        Instruction *Inst = nullptr);
 
   /// @}
 
   /// \name Vector TTI Implementations
   /// @{
 
   unsigned getNumberOfRegisters(unsigned ClassID) const {
     bool Vector = (ClassID == 1);

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

 int ARMTTIImpl::getIntImmCodeSizeCost(unsigned Opcode, unsigned Idx,
                                       const APInt &Imm, Type *Ty) {
   if (Imm.isNonNegative() && Imm.getLimitedValue() < 256)
     return 0;
 
   return 1;
 }
 
-int ARMTTIImpl::getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm,
-                                  Type *Ty, TTI::TargetCostKind CostKind) {
+int ARMTTIImpl::getIntImmCostInst(unsigned Opcode, unsigned Idx,
+                                  const APInt &Imm, Type *Ty,
+                                  TTI::TargetCostKind CostKind,
+                                  Instruction *Inst) {
   // Division by a constant can be turned into multiplication, but only if we
   // know it's constant. So it's not so much that the immediate is cheap (it's
   // not), but that the alternative is worse.
   // FIXME: this is probably unneeded with GlobalISel.
   if ((Opcode == Instruction::SDiv || Opcode == Instruction::UDiv ||
        Opcode == Instruction::SRem || Opcode == Instruction::URem) &&
       Idx == 1)
     return 0;
 
   if (Opcode == Instruction::And) {
     // UXTB/UXTH
     if (Imm == 255 || Imm == 65535)
       return 0;
     // Conversion to BIC is free, and means we can use ~Imm instead.
     return std::min(getIntImmCost(Imm, Ty, CostKind),
                     getIntImmCost(~Imm, Ty, CostKind));
if (ST->isThumb() && NegImm < 1<<8)
       // icmp X, #-C -> adds X, #C
       return 0;
   }
 
   // xor a, -1 can always be folded to MVN
   if (Opcode == Instruction::Xor && Imm.isAllOnesValue())
     return 0;
 
+  // Ensures constants of min(max()) or max(min()) patterns don't get hoisted
+  if (Inst && (ST->hasV6Ops() || ST->isThumb2()) &&
+      Ty->getIntegerBitWidth() <= 32) {
+    Value *LHS, *RHS;
+    ConstantInt *C;
+    SelectPatternFlavor InstSPF = matchSelectPattern(Inst, LHS, RHS).Flavor;
+
+    if (InstSPF == SPF_SMAX &&
+        PatternMatch::match(RHS, PatternMatch::m_ConstantInt(C)) &&
+        C->getUniqueInteger() == Imm && Imm.isNegative() &&
+        (-Imm).isPowerOf2()) {
+
+      auto isSSatMin = [=](Value *MinInst) {
+        if (isa<SelectInst>(MinInst)) {
+          Value *MinLHS, *MinRHS;
+          ConstantInt *MinC;
+          SelectPatternFlavor MinSPF =
+              matchSelectPattern(MinInst, MinLHS, MinRHS).Flavor;
+          if (PatternMatch::match(MinRHS, PatternMatch::m_ConstantInt(MinC)) &&
+              MinC->getUniqueInteger() == ((-Imm) - 1) && MinSPF == SPF_SMIN)
+            return true;
+        }
+        return false;
+      };
+
+      if (isSSatMin(Inst->getOperand(1)) ||
+          (Inst->hasNUses(2) && (isSSatMin(*Inst->user_begin()) ||
+                                 isSSatMin(*(++Inst->user_begin())))))
+        return 0;
+    }
+  }
+
   return getIntImmCost(Imm, Ty, CostKind);
 }
 
 int ARMTTIImpl::getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind) {
   if (CostKind == TTI::TCK_RecipThroughput &&
       (ST->hasNEON() || ST->hasMVEIntegerOps())) {
     // FIXME: The vectorizer is highly sensistive to the cost of these
     // instructions, which suggests that it may be using the costs incorrectly.

llvm/lib/Target/Lanai/LanaiTargetTransformInfo.h

if (isInt<32>(Imm.getSExtValue())) {
         return TTI::TCC_Basic;
       return 2 * TTI::TCC_Basic;
     }
 
     return 4 * TTI::TCC_Basic;
   }
 
   int getIntImmCostInst(unsigned Opc, unsigned Idx, const APInt &Imm, Type *Ty,
-                        TTI::TargetCostKind CostKind) {
+                        TTI::TargetCostKind CostKind,
+                        Instruction *Inst = nullptr) {
     return getIntImmCost(Imm, Ty, CostKind);
   }
 
   int getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx, const APInt &Imm,
                           Type *Ty, TTI::TargetCostKind CostKind) {
     return getIntImmCost(Imm, Ty, CostKind);
   }
 

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h

public:
   /// \name Scalar TTI Implementations
   /// @{
 
   using BaseT::getIntImmCost;
   int getIntImmCost(const APInt &Imm, Type *Ty,
                     TTI::TargetCostKind CostKind);
 
   int getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm,
-                        Type *Ty, TTI::TargetCostKind CostKind);
+                        Type *Ty, TTI::TargetCostKind CostKind,
+                        Instruction *Inst = nullptr);
   int getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx, const APInt &Imm,
                           Type *Ty, TTI::TargetCostKind CostKind);
 
   unsigned getUserCost(const User *U, ArrayRef<const Value *> Operands,
                        TTI::TargetCostKind CostKind);
 
   TTI::PopcntSupportKind getPopcntSupport(unsigned TyWidth);
   bool isHardwareLoopProfitable(Loop *L, ScalarEvolution &SE,

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

if ((Idx < 4) || (Imm.getBitWidth() <= 64 && isInt<64>(Imm.getSExtValue())))
       return TTI::TCC_Free;
     break;
   }
   return PPCTTIImpl::getIntImmCost(Imm, Ty, CostKind);
 }
 
 int PPCTTIImpl::getIntImmCostInst(unsigned Opcode, unsigned Idx,
                                   const APInt &Imm, Type *Ty,
-                                  TTI::TargetCostKind CostKind) {
+                                  TTI::TargetCostKind CostKind,
+                                  Instruction *Inst) {
   if (DisablePPCConstHoist)
-    return BaseT::getIntImmCostInst(Opcode, Idx, Imm, Ty, CostKind);
+    return BaseT::getIntImmCostInst(Opcode, Idx, Imm, Ty, CostKind, Inst);
 
   assert(Ty->isIntegerTy());
 
   unsigned BitSize = Ty->getPrimitiveSizeInBits();
   if (BitSize == 0)
     return ~0U;
 
   unsigned ImmIdx = ~0U;

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h

class RISCVTTIImpl : public BasicTTIImplBase<RISCVTTIImpl> {
   const RISCVTargetLowering *getTLI() const { return TLI; }
 
 public:
   explicit RISCVTTIImpl(const RISCVTargetMachine *TM, const Function &F)
       : BaseT(TM, F.getParent()->getDataLayout()), ST(TM->getSubtargetImpl(F)),
         TLI(ST->getTargetLowering()) {}
 
   int getIntImmCost(const APInt &Imm, Type *Ty, TTI::TargetCostKind CostKind);
-  int getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm, Type *Ty,
-                        TTI::TargetCostKind CostKind);
+  int getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm,
+                        Type *Ty, TTI::TargetCostKind CostKind,
+                        Instruction *Inst = nullptr);
   int getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx, const APInt &Imm,
                           Type *Ty, TTI::TargetCostKind CostKind);
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_RISCV_RISCVTARGETTRANSFORMINFO_H

llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp

if (Imm == 0)
     return TTI::TCC_Free;
 
   // Otherwise, we check how many instructions it will take to materialise.
   const DataLayout &DL = getDataLayout();
   return RISCVMatInt::getIntMatCost(Imm, DL.getTypeSizeInBits(Ty),
                                     getST()->is64Bit());
 }
 
-int RISCVTTIImpl::getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm,
-                                Type *Ty, TTI::TargetCostKind CostKind) {
+int RISCVTTIImpl::getIntImmCostInst(unsigned Opcode, unsigned Idx,
+                                    const APInt &Imm, Type *Ty,
+                                    TTI::TargetCostKind CostKind,
+                                    Instruction *Inst) {
   assert(Ty->isIntegerTy() &&
          "getIntImmCost can only estimate cost of materialising integers");
 
   // We have a Zero register, so 0 is always free.
   if (Imm == 0)
     return TTI::TCC_Free;
 
   // Some instructions in RISC-V can take a 12-bit immediate. Some of these are

llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.h

public:
   /// \name Scalar TTI Implementations
   /// @{
 
   unsigned getInliningThresholdMultiplier() { return 3; }
 
   int getIntImmCost(const APInt &Imm, Type *Ty, TTI::TargetCostKind CostKind);
 
   int getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm,
-                        Type *Ty, TTI::TargetCostKind CostKind);
+                        Type *Ty, TTI::TargetCostKind CostKind,
+                        Instruction *Inst = nullptr);
   int getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx, const APInt &Imm,
                           Type *Ty, TTI::TargetCostKind CostKind);
 
   TTI::PopcntSupportKind getPopcntSupport(unsigned TyWidth);
 
   void getUnrollingPreferences(Loop *L, ScalarEvolution &SE,
                                TTI::UnrollingPreferences &UP);
 

llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp

if (Imm.getBitWidth() <= 64) {
 
     return 2 * TTI::TCC_Basic;
   }
 
   return 4 * TTI::TCC_Basic;
 }
 
 int SystemZTTIImpl::getIntImmCostInst(unsigned Opcode, unsigned Idx,
                                       const APInt &Imm, Type *Ty,
-                                  TTI::TargetCostKind CostKind) {
+                                      TTI::TargetCostKind CostKind,
+                                      Instruction *Inst) {
   assert(Ty->isIntegerTy());
 
   unsigned BitSize = Ty->getPrimitiveSizeInBits();
   // There is no cost model for constants with a bit size of 0. Return TCC_Free
   // here, so that constant hoisting will ignore this constant.
   if (BitSize == 0)
     return TTI::TCC_Free;
   // No cost model for operations on integers larger than 64 bit implemented yet.

llvm/lib/Target/X86/X86TargetTransformInfo.h

int getInterleavedMemoryOpCostAVX2(
       bool UseMaskForCond = false, bool UseMaskForGaps = false);
 
   int getIntImmCost(int64_t);
 
   int getIntImmCost(const APInt &Imm, Type *Ty, TTI::TargetCostKind CostKind);
 
   unsigned getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind);
 
-  int getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm, Type *Ty,
-                        TTI::TargetCostKind CostKind);
+  int getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm,
+                        Type *Ty, TTI::TargetCostKind CostKind,
+                        Instruction *Inst = nullptr);
   int getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx, const APInt &Imm,
                           Type *Ty, TTI::TargetCostKind CostKind);
   bool isLSRCostLess(TargetTransformInfo::LSRCost &C1,
                      TargetTransformInfo::LSRCost &C2);
   bool canMacroFuseCmp();
   bool isLegalMaskedLoad(Type *DataType, Align Alignment);
   bool isLegalMaskedStore(Type *DataType, Align Alignment);
   bool isLegalNTLoad(Type *DataType, Align Alignment);

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

for (unsigned ShiftVal = 0; ShiftVal < BitSize; ShiftVal += 64) {
     APInt Tmp = ImmVal.ashr(ShiftVal).sextOrTrunc(64);
     int64_t Val = Tmp.getSExtValue();
     Cost += getIntImmCost(Val);
   }
   // We need at least one instruction to materialize the constant.
   return std::max(1, Cost);
 }
 
-int X86TTIImpl::getIntImmCostInst(unsigned Opcode, unsigned Idx, const APInt &Imm,
-                                  Type *Ty, TTI::TargetCostKind CostKind) {
+int X86TTIImpl::getIntImmCostInst(unsigned Opcode, unsigned Idx,
+                                  const APInt &Imm, Type *Ty,
+                                  TTI::TargetCostKind CostKind,
+                                  Instruction *Inst) {
   assert(Ty->isIntegerTy());
 
   unsigned BitSize = Ty->getPrimitiveSizeInBits();
   // There is no cost model for constants with a bit size of 0. Return TCC_Free
   // here, so that constant hoisting will ignore this constant.
   if (BitSize == 0)
     return TTI::TCC_Free;
 

llvm/lib/Transforms/Scalar/ConstantHoisting.cpp

void ConstantHoistingPass::collectConstantCandidates(
   unsigned Cost;
   // Ask the target about the cost of materializing the constant for the given
   // instruction and operand index.
   if (auto IntrInst = dyn_cast<IntrinsicInst>(Inst))
     Cost = TTI->getIntImmCostIntrin(IntrInst->getIntrinsicID(), Idx,
                                     ConstInt->getValue(), ConstInt->getType(),
                                     TargetTransformInfo::TCK_SizeAndLatency);
   else
-    Cost = TTI->getIntImmCostInst(Inst->getOpcode(), Idx, ConstInt->getValue(),
-                                  ConstInt->getType(),
-                                  TargetTransformInfo::TCK_SizeAndLatency);
+    Cost = TTI->getIntImmCostInst(
+        Inst->getOpcode(), Idx, ConstInt->getValue(), ConstInt->getType(),
+        TargetTransformInfo::TCK_SizeAndLatency, Inst);
 
   // Ignore cheap integer constants.
   if (Cost > TargetTransformInfo::TCC_Basic) {
     ConstCandMapType::iterator Itr;
     bool Inserted;
     ConstPtrUnionType Cand = ConstInt;
     std::tie(Itr, Inserted) = ConstCandMap.insert(std::make_pair(Cand, 0));
     if (Inserted) {
void ConstantHoistingPass::collectConstantCandidates(
 
   if (!Offset.isIntN(32))
     return;
 
   // A constant GEP expression that has a GlobalVariable as base pointer is
   // usually lowered to a load from constant pool. Such operation is unlikely
   // to be cheaper than compute it by <Base + Offset>, which can be lowered to
   // an ADD instruction or folded into Load/Store instruction.
-  int Cost = TTI->getIntImmCostInst(Instruction::Add, 1, Offset, PtrIntTy,
-                                    TargetTransformInfo::TCK_SizeAndLatency);
+  int Cost =
+      TTI->getIntImmCostInst(Instruction::Add, 1, Offset, PtrIntTy,
+                             TargetTransformInfo::TCK_SizeAndLatency, Inst);
   ConstCandVecType &ExprCandVec = ConstGEPCandMap[BaseGV];
   ConstCandMapType::iterator Itr;
   bool Inserted;
   ConstPtrUnionType Cand = ConstExpr;
   std::tie(Itr, Inserted) = ConstCandMap.insert(std::make_pair(Cand, 0));
   if (Inserted) {
     ExprCandVec.push_back(ConstantCandidate(
         ConstantInt::get(Type::getInt32Ty(*Ctx), Offset.getLimitedValue()),

llvm/test/CodeGen/Thumb2/mve-satmul-loops.ll

 
 ; i16
 
 define arm_aapcs_vfpcc void @ssatmul_4_q15(i16* nocapture readonly %pSrcA, i16* nocapture readonly %pSrcB, i16* noalias nocapture %pDst, i32 %N) {
 ; CHECK-LABEL: ssatmul_4_q15:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r4, r5, r6, lr}
 ; CHECK-NEXT:    push {r4, r5, r6, lr}
-; CHECK-NEXT:    cmp r3, #0
-; CHECK-NEXT:    beq .LBB5_8
+; CHECK-NEXT:    cbz r3, .LBB5_8
 ; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
 ; CHECK-NEXT:    cmp r3, #3
 ; CHECK-NEXT:    bhi .LBB5_3
 ; CHECK-NEXT:  @ %bb.2:
 ; CHECK-NEXT:    movs r5, #0
 ; CHECK-NEXT:    mov r12, r0
 ; CHECK-NEXT:    mov r6, r1
 ; CHECK-NEXT:    mov r4, r2

 ; CHECK-NEXT:    vqshrnb.s32 q0, q0, #15
 ; CHECK-NEXT:    vstrh.32 q0, [r2], #8
 ; CHECK-NEXT:    le lr, .LBB5_4
 ; CHECK-NEXT:  @ %bb.5: @ %middle.block
 ; CHECK-NEXT:    cmp r5, r3
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r4, r5, r6, pc}
 ; CHECK-NEXT:  .LBB5_6: @ %for.body.preheader21
-; CHECK-NEXT:    movw r0, #32768
 ; CHECK-NEXT:    sub.w lr, r3, r5
-; CHECK-NEXT:    movt r0, #65535
-; CHECK-NEXT:    movw r1, #32767
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB5_7: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldrsh r2, [r12], #2
-; CHECK-NEXT:    ldrsh r3, [r6], #2
-; CHECK-NEXT:    muls r2, r3, r2
-; CHECK-NEXT:    mov r3, r0
-; CHECK-NEXT:    cmp.w r0, r2, asr #15
-; CHECK-NEXT:    it lt
-; CHECK-NEXT:    asrlt r3, r2, #15
-; CHECK-NEXT:    cmp r3, r1
-; CHECK-NEXT:    csel r2, r3, r1, lt
-; CHECK-NEXT:    strh r2, [r4], #2
+; CHECK-NEXT:    ldrsh r0, [r12], #2
+; CHECK-NEXT:    ldrsh r1, [r6], #2
+; CHECK-NEXT:    muls r0, r1, r0
+; CHECK-NEXT:    asrs r0, r0, #15
+; CHECK-NEXT:    ssat r0, #16, r0
+; CHECK-NEXT:    strh r0, [r4], #2
 ; CHECK-NEXT:    le lr, .LBB5_7
 ; CHECK-NEXT:  .LBB5_8: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r4, r5, r6, pc}
 entry:
   %cmp8 = icmp eq i32 %N, 0
   br i1 %cmp8, label %for.cond.cleanup, label %for.body.preheader
 
 for.body.preheader:                               ; preds = %entry

 ; CHECK-NEXT:    vqshrnb.s32 q0, q0, #15
 ; CHECK-NEXT:    vstrh.32 q0, [r2], #16
 ; CHECK-NEXT:    le lr, .LBB6_4
 ; CHECK-NEXT:  @ %bb.5: @ %middle.block
 ; CHECK-NEXT:    cmp r5, r3
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r4, r5, r6, pc}
 ; CHECK-NEXT:  .LBB6_6: @ %for.body.preheader21
-; CHECK-NEXT:    movw r0, #32768
 ; CHECK-NEXT:    sub.w lr, r3, r5
-; CHECK-NEXT:    movt r0, #65535
-; CHECK-NEXT:    movw r1, #32767
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB6_7: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldrsh r2, [r12], #2
-; CHECK-NEXT:    ldrsh r3, [r6], #2
-; CHECK-NEXT:    muls r2, r3, r2
-; CHECK-NEXT:    mov r3, r0
-; CHECK-NEXT:    cmp.w r0, r2, asr #15
-; CHECK-NEXT:    it lt
-; CHECK-NEXT:    asrlt r3, r2, #15
-; CHECK-NEXT:    cmp r3, r1
-; CHECK-NEXT:    csel r2, r3, r1, lt
-; CHECK-NEXT:    strh r2, [r4], #2
+; CHECK-NEXT:    ldrsh r0, [r12], #2
+; CHECK-NEXT:    ldrsh r1, [r6], #2
+; CHECK-NEXT:    muls r0, r1, r0
+; CHECK-NEXT:    asrs r0, r0, #15
+; CHECK-NEXT:    ssat r0, #16, r0
+; CHECK-NEXT:    strh r0, [r4], #2
 ; CHECK-NEXT:    le lr, .LBB6_7
 ; CHECK-NEXT:  .LBB6_8: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r4, r5, r6, pc}
 entry:
   %cmp8 = icmp eq i32 %N, 0
   br i1 %cmp8, label %for.cond.cleanup, label %for.body.preheader
 
 for.body.preheader:                               ; preds = %entry

 ; CHECK-NEXT:    vqshrnt.s32 q0, q2, #15
 ; CHECK-NEXT:    vstrb.8 q0, [r2], #16
 ; CHECK-NEXT:    le lr, .LBB7_4
 ; CHECK-NEXT:  @ %bb.5: @ %middle.block
 ; CHECK-NEXT:    cmp r5, r3
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r4, r5, r6, pc}
 ; CHECK-NEXT:  .LBB7_6: @ %for.body.preheader21
-; CHECK-NEXT:    movw r0, #32768
 ; CHECK-NEXT:    sub.w lr, r3, r5
-; CHECK-NEXT:    movt r0, #65535
-; CHECK-NEXT:    movw r1, #32767
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB7_7: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldrsh r2, [r12], #2
-; CHECK-NEXT:    ldrsh r3, [r6], #2
-; CHECK-NEXT:    muls r2, r3, r2
-; CHECK-NEXT:    mov r3, r0
-; CHECK-NEXT:    cmp.w r0, r2, asr #15
-; CHECK-NEXT:    it lt
-; CHECK-NEXT:    asrlt r3, r2, #15
-; CHECK-NEXT:    cmp r3, r1
-; CHECK-NEXT:    csel r2, r3, r1, lt
-; CHECK-NEXT:    strh r2, [r4], #2
+; CHECK-NEXT:    ldrsh r0, [r12], #2
+; CHECK-NEXT:    ldrsh r1, [r6], #2
+; CHECK-NEXT:    muls r0, r1, r0
+; CHECK-NEXT:    asrs r0, r0, #15
+; CHECK-NEXT:    ssat r0, #16, r0
+; CHECK-NEXT:    strh r0, [r4], #2
 ; CHECK-NEXT:    le lr, .LBB7_7
 ; CHECK-NEXT:  .LBB7_8: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r4, r5, r6, pc}
 entry:
   %cmp8 = icmp eq i32 %N, 0
   br i1 %cmp8, label %for.cond.cleanup, label %for.body.preheader
 
 for.body.preheader:                               ; preds = %entry
for.body:                                         ; preds = %for.body.preheader21, %for.body
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
 define arm_aapcs_vfpcc void @ssatmul_8_q7(i8* nocapture readonly %pSrcA, i8* nocapture readonly %pSrcB, i8* noalias nocapture %pDst, i32 %N) {
 ; CHECK-LABEL: ssatmul_8_q7:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r4, r5, r6, lr}
 ; CHECK-NEXT:    push {r4, r5, r6, lr}
-; CHECK-NEXT:    cmp r3, #0
-; CHECK-NEXT:    beq .LBB14_8
+; CHECK-NEXT:    cbz r3, .LBB14_8
 ; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
 ; CHECK-NEXT:    cmp r3, #7
 ; CHECK-NEXT:    bhi .LBB14_3
 ; CHECK-NEXT:  @ %bb.2:
 ; CHECK-NEXT:    movs r5, #0
 ; CHECK-NEXT:    mov r12, r0
 ; CHECK-NEXT:    mov r6, r1
 ; CHECK-NEXT:    mov r4, r2

 ; CHECK-NEXT:    vstrb.16 q0, [r2], #8
 ; CHECK-NEXT:    le lr, .LBB14_4
 ; CHECK-NEXT:  @ %bb.5: @ %middle.block
 ; CHECK-NEXT:    cmp r5, r3
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r4, r5, r6, pc}
 ; CHECK-NEXT:  .LBB14_6: @ %for.body.preheader23
 ; CHECK-NEXT:    sub.w lr, r3, r5
-; CHECK-NEXT:    mvn r0, #127
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB14_7: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldrsb r1, [r12], #1
-; CHECK-NEXT:    ldrsb r2, [r6], #1
-; CHECK-NEXT:    muls r1, r2, r1
-; CHECK-NEXT:    mvn r2, #127
-; CHECK-NEXT:    cmp.w r0, r1, asr #7
-; CHECK-NEXT:    it lt
-; CHECK-NEXT:    asrlt r2, r1, #7
-; CHECK-NEXT:    cmp r2, #127
-; CHECK-NEXT:    it ge
-; CHECK-NEXT:    movge r2, #127
-; CHECK-NEXT:    strb r2, [r4], #1
+; CHECK-NEXT:    ldrsb r0, [r12], #1
+; CHECK-NEXT:    ldrsb r1, [r6], #1
+; CHECK-NEXT:    muls r0, r1, r0
+; CHECK-NEXT:    asrs r0, r0, #7
+; CHECK-NEXT:    ssat r0, #8, r0
+; CHECK-NEXT:    strb r0, [r4], #1
 ; CHECK-NEXT:    le lr, .LBB14_7
 ; CHECK-NEXT:  .LBB14_8: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r4, r5, r6, pc}
 entry:
   %cmp10 = icmp eq i32 %N, 0
   br i1 %cmp10, label %for.cond.cleanup, label %for.body.preheader
 
 for.body.preheader:                               ; preds = %entry

 ; CHECK-NEXT:    vstrb.16 q0, [r2], #16
 ; CHECK-NEXT:    le lr, .LBB15_4
 ; CHECK-NEXT:  @ %bb.5: @ %middle.block
 ; CHECK-NEXT:    cmp r5, r3
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r4, r5, r6, pc}
 ; CHECK-NEXT:  .LBB15_6: @ %for.body.preheader23
 ; CHECK-NEXT:    sub.w lr, r3, r5
-; CHECK-NEXT:    mvn r0, #127
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB15_7: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldrsb r1, [r12], #1
-; CHECK-NEXT:    ldrsb r2, [r6], #1
-; CHECK-NEXT:    muls r1, r2, r1
-; CHECK-NEXT:    mvn r2, #127
-; CHECK-NEXT:    cmp.w r0, r1, asr #7
-; CHECK-NEXT:    it lt
-; CHECK-NEXT:    asrlt r2, r1, #7
-; CHECK-NEXT:    cmp r2, #127
-; CHECK-NEXT:    it ge
-; CHECK-NEXT:    movge r2, #127
-; CHECK-NEXT:    strb r2, [r4], #1
+; CHECK-NEXT:    ldrsb r0, [r12], #1
+; CHECK-NEXT:    ldrsb r1, [r6], #1
+; CHECK-NEXT:    muls r0, r1, r0
+; CHECK-NEXT:    asrs r0, r0, #7
+; CHECK-NEXT:    ssat r0, #8, r0
+; CHECK-NEXT:    strb r0, [r4], #1
 ; CHECK-NEXT:    le lr, .LBB15_7
 ; CHECK-NEXT:  .LBB15_8: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r4, r5, r6, pc}
 entry:
   %cmp10 = icmp eq i32 %N, 0
   br i1 %cmp10, label %for.cond.cleanup, label %for.body.preheader
 
 for.body.preheader:                               ; preds = %entry

 ; CHECK-NEXT:    vstrb.8 q0, [r2], #16
 ; CHECK-NEXT:    le lr, .LBB16_4
 ; CHECK-NEXT:  @ %bb.5: @ %middle.block
 ; CHECK-NEXT:    cmp r5, r3
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r4, r5, r6, pc}
 ; CHECK-NEXT:  .LBB16_6: @ %for.body.preheader23
 ; CHECK-NEXT:    sub.w lr, r3, r5
-; CHECK-NEXT:    mvn r0, #127
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB16_7: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldrsb r1, [r12], #1
-; CHECK-NEXT:    ldrsb r2, [r6], #1
-; CHECK-NEXT:    muls r1, r2, r1
-; CHECK-NEXT:    mvn r2, #127
-; CHECK-NEXT:    cmp.w r0, r1, asr #7
-; CHECK-NEXT:    it lt
-; CHECK-NEXT:    asrlt r2, r1, #7
-; CHECK-NEXT:    cmp r2, #127
-; CHECK-NEXT:    it ge
-; CHECK-NEXT:    movge r2, #127
-; CHECK-NEXT:    strb r2, [r4], #1
+; CHECK-NEXT:    ldrsb r0, [r12], #1
+; CHECK-NEXT:    ldrsb r1, [r6], #1
+; CHECK-NEXT:    muls r0, r1, r0
+; CHECK-NEXT:    asrs r0, r0, #7
+; CHECK-NEXT:    ssat r0, #8, r0
+; CHECK-NEXT:    strb r0, [r4], #1
 ; CHECK-NEXT:    le lr, .LBB16_7
 ; CHECK-NEXT:  .LBB16_8: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r4, r5, r6, pc}
 entry:
   %cmp10 = icmp eq i32 %N, 0
   br i1 %cmp10, label %for.cond.cleanup, label %for.body.preheader
 
 for.body.preheader:                               ; preds = %entry