[TTI] Add optional VecPred argument to getCmpSelInstrCost.

[fhahn]

Summary

On some targets, like AArch64, vector selects can be efficiently lowered
if the vector condition is a compare with a supported predicate.

This patch adds a new argument to getCmpSelInstrCost, to indicate the
predicate of the feeding select condition. Note that it is not
sufficient to use the context instruction when querying the cost of a
vector select starting from a scalar one, because the condition of the
vector select could be composed of compares with different predicates.

This change greatly improves modeling the costs of certain
compare/select patterns on AArch64.

I am also planning on putting up patches to make use of the new argument in
SLPVectorizer & LV.

[fhahn]

Looks like there's already a related bug report https://bugs.llvm.org/show_bug.cgi?id=40376. Thanks @RKSimon for mentioning the patch there.

[RKSimon]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

702

Would it make sense to do this in TargetTransformInfo::getCmpSelInstrCost? Most platforms are likely to do something similar.

706

isIntPredicate ?

[dmgreen]

AArch64 costs sound OK, especially if we can fix NE codegen to be more likely to be a single instruction.

I would personally keep the args in (Opcode, Types, CondCode, CostKind, CtxI) order, to keep them more consistent with the other cost functions. Now that I look at them though I don't think they are as consistent as I believed they were!

llvm/test/Analysis/CostModel/AArch64/vector-select.ll

43

Huh.

[fhahn]

In D90070#2352567, @dmgreen wrote:

AArch64 costs sound OK, especially if we can fix NE codegen to be more likely to be a single instruction.

I would personally keep the args in (Opcode, Types, CondCode, CostKind, CtxI) order, to keep them more consistent with the other cost functions. Now that I look at them though I don't think they are as consistent as I believed they were!

I think having the CondCode after the types would make sense. One advantage of the current ordering is that we do not need to update all existing callers that pass in CtxI. That convenient initially, but there are not too many users of the function that would need updating. Happy to update to whatever order is preferred before landing the patch (threading things through properly everywhere is a bit of a pain, so I'd prefer updating once everyone is happy, to just do it once).

[fhahn]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

702

I think that makes sense, but unfortunately it seems like TargetTransformInfo::getCmpSelInstrCost is not called for at least -analyze -costmodel, because it uses TargetTransformInfoImplCRTPBase::getUserCost, which delegates directly to the target-specific getCmpSelInstrCost. Not sure if there's a convenient way to work around that?

[dmgreen]

I would put it after the type, but other than that this LGTM

[RKSimon]

Yeah, the TTI/CRTP layout makes code reuse a pain in some places - thanks for trying anyway. LGTM

[fhahn]

In D90070#2358326, @dmgreen wrote:

I would put it after the type, but other than that this LGTM

I updated the patch to pass VecPred before CostKind, which also makes it non-optional. That wasn't too bad and now all the places that would be good to update are more visible. I've the patch for SLP up already and will also update LV. For some of the other users I am not too sure though.

[fhahn]

Rebased after recent changes to SLPVectorizer

[pcc]

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

952

This fails an assertion if the instruction is a constant. If you have a copy of the Android NDK you should be able to reproduce using the in-tree HWASan runtime with something like:

/path/to/build-directory/bin/clang++ -o hwasan_dynamic_shadow.o -c compiler-rt/lib/hwasan/hwasan_dynamic_shadow.cpp -DHWASAN_WITH_INTERCEPTORS=1 -Icompiler-rt/lib --target=aarch64-linux-android29 --sysroot=/path/to/android-ndk-r21/toolchains/llvm/prebuilt/linux-x86_64/sysroot -B/path/to/android-ndk-r21/toolchains/llvm/prebuilt/linux-x86_64 -O3 -fdiagnostics-color -Wall -Wextra -Wno-unused-parameter -Wdelete-non-virtual-dtor -Wstring-conversion -no-canonical-prefixes -Werror=date-time -Wcovered-switch-default -fPIC -funwind-tables -gline-tables-only -fvisibility=hidden -std=c++14 -fvisibility-inlines-hidden -fno-exceptions -fno-rtti

But let me know if you need a repro.

[vitalybuka]

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

952

same crash on linux bot without android http://lab.llvm.org:8011/#/builders/85/builds/383

[fhahn]

Interesting! O reverted the patch for now while I investigate.

[thakis]

Yet another repro at https://bugs.chromium.org/p/chromium/issues/detail?id=1144355#c1 . Thanks for reverting :)

[RKSimon]

FTR this was relanded at rGb3b993a7ad817c3c5801341fa78f34332900eb83

llvm/include/llvm/Analysis/TargetTransformInfo.h

 /// This file defines #2, which is the interface that IR-level transformations
 /// use for querying the codegen.
 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_ANALYSIS_TARGETTRANSFORMINFO_H
 #define LLVM_ANALYSIS_TARGETTRANSFORMINFO_H
 
+#include "llvm/IR/InstrTypes.h"
 #include "llvm/IR/Operator.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/AtomicOrdering.h"
 #include "llvm/Support/DataTypes.h"
 #include <functional>
 
 namespace llvm {
public:
 
   /// \return The expected cost of control-flow related instructions such as
   /// Phi, Ret, Br.
   int getCFInstrCost(unsigned Opcode,
                      TTI::TargetCostKind CostKind = TTI::TCK_SizeAndLatency) const;
 
   /// \returns The expected cost of compare and select instructions. If there
   /// is an existing instruction that holds Opcode, it may be passed in the
-  /// 'I' parameter.
-  int getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy = nullptr,
-                         TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput,
-                         const Instruction *I = nullptr) const;
+  /// 'I' parameter. For vector selects, the optional \p VecPred parameter can
+  /// be used to indicate the select is using a compare with the specified
+  /// predicate as condition.
+  int getCmpSelInstrCost(
+      unsigned Opcode, Type *ValTy, Type *CondTy = nullptr,
+      const Instruction *I = nullptr,
+      CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE) const;
 
   /// \return The expected cost of vector Insert and Extract.
   /// Use -1 to indicate that there is no information on the index value.
   int getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index = -1) const;
 
   /// \return The cost of Load and Store instructions.
   int getMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
                       unsigned AddressSpace,
public:
   virtual int getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
                                CastContextHint CCH,
                                TTI::TargetCostKind CostKind,
                                const Instruction *I) = 0;
   virtual int getExtractWithExtendCost(unsigned Opcode, Type *Dst,
                                        VectorType *VecTy, unsigned Index) = 0;
   virtual int getCFInstrCost(unsigned Opcode,
                              TTI::TargetCostKind CostKind) = 0;
-  virtual int getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
-                                 TTI::TargetCostKind CostKind,
-                                 const Instruction *I) = 0;
+  virtual int getCmpSelInstrCost(
+      unsigned Opcode, Type *ValTy, Type *CondTy, TTI::TargetCostKind CostKind,
+      const Instruction *I,
+      CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE) = 0;
   virtual int getVectorInstrCost(unsigned Opcode, Type *Val,
                                  unsigned Index) = 0;
   virtual int getMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
                               unsigned AddressSpace,
                               TTI::TargetCostKind CostKind,
                               const Instruction *I) = 0;
   virtual int getMaskedMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
                                     unsigned AddressSpace,
public:
   }
   int getExtractWithExtendCost(unsigned Opcode, Type *Dst, VectorType *VecTy,
                                unsigned Index) override {
     return Impl.getExtractWithExtendCost(Opcode, Dst, VecTy, Index);
   }
   int getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind) override {
     return Impl.getCFInstrCost(Opcode, CostKind);
   }
-  int getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
-                         TTI::TargetCostKind CostKind,
-                         const Instruction *I) override {
-    return Impl.getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I);
+  int getCmpSelInstrCost(
+      unsigned Opcode, Type *ValTy, Type *CondTy, TTI::TargetCostKind CostKind,
+      const Instruction *I,
+      CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE) override {
+    return Impl.getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I, VecPred);
   }
   int getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index) override {
     return Impl.getVectorInstrCost(Opcode, Val, Index);
   }
   int getMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
                       unsigned AddressSpace, TTI::TargetCostKind CostKind,
                       const Instruction *I) override {
     return Impl.getMemoryOpCost(Opcode, Src, Alignment, AddressSpace,

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

unsigned getCFInstrCost(unsigned Opcode,
                           TTI::TargetCostKind CostKind) {
     // A phi would be free, unless we're costing the throughput because it
     // will require a register.
     if (Opcode == Instruction::PHI && CostKind != TTI::TCK_RecipThroughput)
       return 0;
     return 1;
   }
 
-  unsigned getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
-                              TTI::TargetCostKind CostKind,
-                              const Instruction *I) const {
+  unsigned getCmpSelInstrCost(
+      unsigned Opcode, Type *ValTy, Type *CondTy, TTI::TargetCostKind CostKind,
+      const Instruction *I,
+      CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE) const {
     return 1;
   }
 
   unsigned getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index) {
     return 1;
   }
 
   unsigned getMemoryOpCost(unsigned Opcode, Type *Src, Align Alignment,
case Instruction::Select: {
       Type *CondTy = U->getOperand(0)->getType();
       return TargetTTI->getCmpSelInstrCost(Opcode, U->getType(), CondTy,
                                            CostKind, I);
     }
     case Instruction::ICmp:
     case Instruction::FCmp: {
       Type *ValTy = U->getOperand(0)->getType();
       return TargetTTI->getCmpSelInstrCost(Opcode, ValTy, U->getType(),
                                            CostKind, I);
     }
     case Instruction::InsertElement: {
       auto *IE = dyn_cast<InsertElementInst>(U);
       if (!IE)
         return TTI::TCC_Basic; // FIXME
       auto *CI = dyn_cast<ConstantInt>(IE->getOperand(2));
       unsigned Idx = CI ? CI->getZExtValue() : -1;
       return TargetTTI->getVectorInstrCost(Opcode, Ty, Idx);

llvm/include/llvm/CodeGen/BasicTTIImpl.h

return thisT()->getVectorInstrCost(Instruction::ExtractElement, VecTy,
            thisT()->getCastInstrCost(Opcode, Dst, VecTy->getElementType(),
                                      TTI::CastContextHint::None, TTI::TCK_RecipThroughput);
   }
 
   unsigned getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind) {
     return BaseT::getCFInstrCost(Opcode, CostKind);
   }
 
-  unsigned getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
-                              TTI::TargetCostKind CostKind,
-                              const Instruction *I = nullptr) {
+  unsigned
+  getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
+                     const Instruction *I = nullptr,
+                     CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE) {
     const TargetLoweringBase *TLI = getTLI();
     int ISD = TLI->InstructionOpcodeToISD(Opcode);
     assert(ISD && "Invalid opcode");
 
     // TODO: Handle other cost kinds.
     if (CostKind != TTI::TCK_RecipThroughput)
-      return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I);
+      return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I,
+                                       VecPred);
 
     // Selects on vectors are actually vector selects.
     if (ISD == ISD::SELECT) {
       assert(CondTy && "CondTy must exist");
       if (CondTy->isVectorTy())
         ISD = ISD::VSELECT;
     }
     std::pair<unsigned, MVT> LT = TLI->getTypeLegalizationCost(DL, ValTy);

llvm/lib/Analysis/TargetTransformInfo.cpp

int TargetTransformInfo::getCFInstrCost(unsigned Opcode,
   int Cost = TTIImpl->getCFInstrCost(Opcode, CostKind);
   assert(Cost >= 0 && "TTI should not produce negative costs!");
   return Cost;
 }
 
 int TargetTransformInfo::getCmpSelInstrCost(unsigned Opcode, Type *ValTy,
                                             Type *CondTy,
                                             TTI::TargetCostKind CostKind,
-                                            const Instruction *I) const {
+                                            const Instruction *I,
+                                            CmpInst::Predicate VecPred) const {
   assert((I == nullptr || I->getOpcode() == Opcode) &&
          "Opcode should reflect passed instruction.");
-  int Cost = TTIImpl->getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I);
+  int Cost =
+      TTIImpl->getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I, VecPred);
   assert(Cost >= 0 && "TTI should not produce negative costs!");
   return Cost;
 }
 
 int TargetTransformInfo::getVectorInstrCost(unsigned Opcode, Type *Val,
                                             unsigned Index) const {
   int Cost = TTIImpl->getVectorInstrCost(Opcode, Val, Index);
   assert(Cost >= 0 && "TTI should not produce negative costs!");

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

int getArithmeticInstrCost(
       TTI::OperandValueKind Opd2Info = TTI::OK_AnyValue,
       TTI::OperandValueProperties Opd1PropInfo = TTI::OP_None,
       TTI::OperandValueProperties Opd2PropInfo = TTI::OP_None,
       ArrayRef<const Value *> Args = ArrayRef<const Value *>(),
       const Instruction *CxtI = nullptr);
 
   int getAddressComputationCost(Type *Ty, ScalarEvolution *SE, const SCEV *Ptr);
 
-  int getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
-                         TTI::TargetCostKind CostKind,
-                         const Instruction *I = nullptr);
+  int getCmpSelInstrCost(
+      unsigned Opcode, Type *ValTy, Type *CondTy, TTI::TargetCostKind CostKind,
+      const Instruction *I = nullptr,
+      CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE);
 
   TTI::MemCmpExpansionOptions enableMemCmpExpansion(bool OptSize,
                                                     bool IsZeroCmp) const;
 
   int getMemoryOpCost(unsigned Opcode, Type *Src, MaybeAlign Alignment,
                       unsigned AddressSpace,
                       TTI::TargetCostKind CostKind,
                       const Instruction *I = nullptr);

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

 //===-- AArch64TargetTransformInfo.cpp - AArch64 specific TTI -------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#include "AArch64ExpandImm.h"
 #include "AArch64TargetTransformInfo.h"
+#include "AArch64ExpandImm.h"
 #include "MCTargetDesc/AArch64AddressingModes.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/CodeGen/BasicTTIImpl.h"
 #include "llvm/CodeGen/CostTable.h"
 #include "llvm/CodeGen/TargetLowering.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/IntrinsicsAArch64.h"
+#include "llvm/IR/PatternMatch.h"
 #include "llvm/Support/Debug.h"
 #include <algorithm>
 using namespace llvm;
+using namespace llvm::PatternMatch;
 
 #define DEBUG_TYPE "aarch64tti"
 
 static cl::opt<bool> EnableFalkorHWPFUnrollFix("enable-falkor-hwpf-unroll-fix",
                                                cl::init(true), cl::Hidden);
 
 bool AArch64TTIImpl::areInlineCompatible(const Function *Caller,
                                          const Function *Callee) const {
int AArch64TTIImpl::getAddressComputationCost(Type *Ty, ScalarEvolution *SE,
   // In many cases the address computation is not merged into the instruction
   // addressing mode.
   return 1;
 }
 
 int AArch64TTIImpl::getCmpSelInstrCost(unsigned Opcode, Type *ValTy,
                                        Type *CondTy,
                                        TTI::TargetCostKind CostKind,
-                                       const Instruction *I) {
+                                       const Instruction *I,
+                                       CmpInst::Predicate VecPred) {
   // TODO: Handle other cost kinds.
   if (CostKind != TTI::TCK_RecipThroughput)
     return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I);
 
   int ISD = TLI->InstructionOpcodeToISD(Opcode);
   // We don't lower some vector selects well that are wider than the register
   // width.
   if (ValTy->isVectorTy() && ISD == ISD::SELECT) {
     // We would need this many instructions to hide the scalarization happening.
     const int AmortizationCost = 20;
+
+    // If VecPred is not set, check if we can get a predicate from the context
+    // instruction, if its type matches the requested ValTy.
+    if (VecPred == CmpInst::BAD_ICMP_PREDICATE && I && I->getType() == ValTy) {
+      CmpInst::Predicate CurrentPred;
+      if (match(I, m_Select(m_Cmp(CurrentPred, m_Value(), m_Value()), m_Value(),
+                            m_Value())))
+        VecPred = CurrentPred;
+    }
+    // Check if we have a compare/select chain that can be lowered using CMxx &
+    // BFI pair.
+    if (CmpInst::isIntPredicate(VecPred)) {
+      static const auto ValidMinMaxTys = {MVT::v8i8,  MVT::v16i8, MVT::v4i16,
+                                          MVT::v8i16, MVT::v2i32, MVT::v4i32,
+                                          MVT::v2i64};
+      auto LT = TLI->getTypeLegalizationCost(DL, ValTy);
+      if (any_of(ValidMinMaxTys, [&LT](MVT M) { return M == LT.second; }))
+        return LT.first;
+    }
+
     static const TypeConversionCostTblEntry
     VectorSelectTbl[] = {
       { ISD::SELECT, MVT::v16i1, MVT::v16i16, 16 },
       { ISD::SELECT, MVT::v8i1, MVT::v8i32, 8 },
       { ISD::SELECT, MVT::v16i1, MVT::v16i32, 16 },
       { ISD::SELECT, MVT::v4i1, MVT::v4i64, 4 * AmortizationCost },
       { ISD::SELECT, MVT::v8i1, MVT::v8i64, 8 * AmortizationCost },
       { ISD::SELECT, MVT::v16i1, MVT::v16i64, 16 * AmortizationCost }

llvm/lib/Target/ARM/ARMTargetTransformInfo.h

public:
 
   int getCFInstrCost(unsigned Opcode,
                      TTI::TargetCostKind CostKind);
 
   int getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
                        TTI::CastContextHint CCH, TTI::TargetCostKind CostKind,
                        const Instruction *I = nullptr);
 
-  int getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
-                         TTI::TargetCostKind CostKind,
-                         const Instruction *I = nullptr);
+  int getCmpSelInstrCost(
+      unsigned Opcode, Type *ValTy, Type *CondTy, TTI::TargetCostKind CostKind,
+      const Instruction *I = nullptr,
+      CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE);
 
   int getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index);
 
   int getAddressComputationCost(Type *Val, ScalarEvolution *SE,
                                 const SCEV *Ptr);
 
   int getArithmeticInstrCost(
       unsigned Opcode, Type *Ty,

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

return std::max(BaseT::getVectorInstrCost(Opcode, ValTy, Index),
            cast<FixedVectorType>(ValTy)->getNumElements() / 2;
   }
 
   return BaseT::getVectorInstrCost(Opcode, ValTy, Index);
 }
 
 int ARMTTIImpl::getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
                                    TTI::TargetCostKind CostKind,
-                                   const Instruction *I) {
+                                   const Instruction *I,
+                                   CmpInst::Predicate VecPred) {
   int ISD = TLI->InstructionOpcodeToISD(Opcode);
 
   // Thumb scalar code size cost for select.
   if (CostKind == TTI::TCK_CodeSize && ISD == ISD::SELECT &&
       ST->isThumb() && !ValTy->isVectorTy()) {
     // Assume expensive structs.
     if (TLI->getValueType(DL, ValTy, true) == MVT::Other)
       return TTI::TCC_Expensive;
if (CostKind == TTI::TCK_CodeSize && ISD == ISD::SELECT &&
     // flag setting instructions.
     if (ValTy->isIntegerTy(1))
       ++Cost;
 
     return Cost;
   }
 
   if (CostKind != TTI::TCK_RecipThroughput)
-    return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I);
+    return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I,
+                                     VecPred);
 
   // On NEON a vector select gets lowered to vbsl.
   if (ST->hasNEON() && ValTy->isVectorTy() && ISD == ISD::SELECT) {
     // Lowering of some vector selects is currently far from perfect.
     static const TypeConversionCostTblEntry NEONVectorSelectTbl[] = {
       { ISD::SELECT, MVT::v4i1, MVT::v4i64, 4*4 + 1*2 + 1 },
       { ISD::SELECT, MVT::v8i1, MVT::v8i64, 50 },
       { ISD::SELECT, MVT::v16i1, MVT::v16i64, 100 }
if (ST->hasNEON() && ValTy->isVectorTy() && ISD == ISD::SELECT) {
 
     std::pair<int, MVT> LT = TLI->getTypeLegalizationCost(DL, ValTy);
     return LT.first;
   }
 
   int BaseCost = ST->hasMVEIntegerOps() && ValTy->isVectorTy()
                      ? ST->getMVEVectorCostFactor()
                      : 1;
-  return BaseCost * BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind,
-                                              I);
+  return BaseCost *
+         BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I, VecPred);
 }
 
 int ARMTTIImpl::getAddressComputationCost(Type *Ty, ScalarEvolution *SE,
                                           const SCEV *Ptr) {
   // Address computations in vectorized code with non-consecutive addresses will
   // likely result in more instructions compared to scalar code where the
   // computation can more often be merged into the index mode. The resulting
   // extra micro-ops can significantly decrease throughput.

llvm/lib/Target/Hexagon/HexagonTargetTransformInfo.h

unsigned getGatherScatterOpCost(unsigned Opcode, Type *DataTy,
                                   const Value *Ptr, bool VariableMask,
                                   Align Alignment, TTI::TargetCostKind CostKind,
                                   const Instruction *I);
   unsigned getInterleavedMemoryOpCost(
       unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,
       Align Alignment, unsigned AddressSpace,
       TTI::TargetCostKind CostKind = TTI::TCK_SizeAndLatency,
       bool UseMaskForCond = false, bool UseMaskForGaps = false);
-  unsigned getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
-                              TTI::TargetCostKind CostKind,
-                              const Instruction *I = nullptr);
+  unsigned
+  getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
+                     const Instruction *I = nullptr,
+                     CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE);
   unsigned getArithmeticInstrCost(
       unsigned Opcode, Type *Ty,
       TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput,
       TTI::OperandValueKind Opd1Info = TTI::OK_AnyValue,
       TTI::OperandValueKind Opd2Info = TTI::OK_AnyValue,
       TTI::OperandValueProperties Opd1PropInfo = TTI::OP_None,
       TTI::OperandValueProperties Opd2PropInfo = TTI::OP_None,
       ArrayRef<const Value *> Args = ArrayRef<const Value *>(),

llvm/lib/Target/Hexagon/HexagonTargetTransformInfo.cpp

return BaseT::getInterleavedMemoryOpCost(Opcode, VecTy, Factor, Indices,
                                              Alignment, AddressSpace,
                                              CostKind,
                                              UseMaskForCond, UseMaskForGaps);
   return getMemoryOpCost(Opcode, VecTy, MaybeAlign(Alignment), AddressSpace,
                          CostKind);
 }
 
 unsigned HexagonTTIImpl::getCmpSelInstrCost(unsigned Opcode, Type *ValTy,
-      Type *CondTy, TTI::TargetCostKind CostKind, const Instruction *I) {
+                                            Type *CondTy,
+                                            TTI::TargetCostKind CostKind,
+                                            const Instruction *I,
+                                            CmpInst::Predicate VecPred) {
   if (ValTy->isVectorTy() && CostKind == TTI::TCK_RecipThroughput) {
     std::pair<int, MVT> LT = TLI.getTypeLegalizationCost(DL, ValTy);
     if (Opcode == Instruction::FCmp)
       return LT.first + FloatFactor * getTypeNumElements(ValTy);
   }
-  return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I);
+  return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I, VecPred);
 }
 
 unsigned HexagonTTIImpl::getArithmeticInstrCost(
     unsigned Opcode, Type *Ty, TTI::TargetCostKind CostKind,
     TTI::OperandValueKind Opd1Info,
     TTI::OperandValueKind Opd2Info, TTI::OperandValueProperties Opd1PropInfo,
     TTI::OperandValueProperties Opd2PropInfo, ArrayRef<const Value *> Args,
     const Instruction *CxtI) {

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h

int getArithmeticInstrCost(
       TTI::OperandValueProperties Opd2PropInfo = TTI::OP_None,
       ArrayRef<const Value *> Args = ArrayRef<const Value *>(),
       const Instruction *CxtI = nullptr);
   int getShuffleCost(TTI::ShuffleKind Kind, Type *Tp, int Index, Type *SubTp);
   int getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
                        TTI::CastContextHint CCH, TTI::TargetCostKind CostKind,
                        const Instruction *I = nullptr);
   int getCFInstrCost(unsigned Opcode, TTI::TargetCostKind CostKind);
-  int getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
-                         TTI::TargetCostKind CostKind,
-                         const Instruction *I = nullptr);
+  int getCmpSelInstrCost(
+      unsigned Opcode, Type *ValTy, Type *CondTy, TTI::TargetCostKind CostKind,
+      const Instruction *I = nullptr,
+      CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE);
   int getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index);
   int getMemoryOpCost(unsigned Opcode, Type *Src, MaybeAlign Alignment,
                       unsigned AddressSpace,
                       TTI::TargetCostKind CostKind,
                       const Instruction *I = nullptr);
   int getInterleavedMemoryOpCost(
       unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,
       Align Alignment, unsigned AddressSpace,

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

int PPCTTIImpl::getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
   // TODO: Allow non-throughput costs that aren't binary.
   if (CostKind != TTI::TCK_RecipThroughput)
     return Cost == 0 ? 0 : 1;
   return Cost;
 }
 
 int PPCTTIImpl::getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
                                    TTI::TargetCostKind CostKind,
-                                   const Instruction *I) {
-  int Cost = BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I);
+                                   const Instruction *I,
+                                   CmpInst::Predicate VecPred) {
+  int Cost =
+      BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I, VecPred);
   // TODO: Handle other cost kinds.
   if (CostKind != TTI::TCK_RecipThroughput)
     return Cost;
   return vectorCostAdjustment(Cost, Opcode, ValTy, nullptr);
 }
 
 int PPCTTIImpl::getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index) {
   assert(Val->isVectorTy() && "This must be a vector type");

llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.h

int getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp, int Index,
                      VectorType *SubTp);
   unsigned getVectorTruncCost(Type *SrcTy, Type *DstTy);
   unsigned getVectorBitmaskConversionCost(Type *SrcTy, Type *DstTy);
   unsigned getBoolVecToIntConversionCost(unsigned Opcode, Type *Dst,
                                          const Instruction *I);
   int getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
                        TTI::CastContextHint CCH, TTI::TargetCostKind CostKind,
                        const Instruction *I = nullptr);
-  int getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
-                         TTI::TargetCostKind CostKind,
-                         const Instruction *I = nullptr);
+  int getCmpSelInstrCost(
+      unsigned Opcode, Type *ValTy, Type *CondTy, TTI::TargetCostKind CostKind,
+      const Instruction *I = nullptr,
+      CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE);
   int getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index);
   bool isFoldableLoad(const LoadInst *Ld, const Instruction *&FoldedValue);
   int getMemoryOpCost(unsigned Opcode, Type *Src, MaybeAlign Alignment,
                       unsigned AddressSpace, TTI::TargetCostKind CostKind,
                       const Instruction *I = nullptr);
 
   int getInterleavedMemoryOpCost(
       unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,

llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp

if (!isa<LoadInst>(Op) && !isa<ConstantInt>(Op))
       ExtCost++;
 
   return ExtCost;
 }
 
 int SystemZTTIImpl::getCmpSelInstrCost(unsigned Opcode, Type *ValTy,
                                        Type *CondTy,
                                        TTI::TargetCostKind CostKind,
-                                       const Instruction *I) {
+                                       const Instruction *I,
+                                       CmpInst::Predicate VecPred) {
   if (CostKind != TTI::TCK_RecipThroughput)
-    return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind);
+    return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, nullptr,
+                                     VecPred);
 
   if (!ValTy->isVectorTy()) {
     switch (Opcode) {
     case Instruction::ICmp: {
       // A loaded value compared with 0 with multiple users becomes Load and
       // Test. The load is then not foldable, so return 0 cost for the ICmp.
       unsigned ScalarBits = ValTy->getScalarSizeInBits();
       if (I != nullptr && ScalarBits >= 32)

llvm/lib/Target/X86/X86TargetTransformInfo.h

int getArithmeticInstrCost(
       TTI::OperandValueProperties Opd2PropInfo = TTI::OP_None,
       ArrayRef<const Value *> Args = ArrayRef<const Value *>(),
       const Instruction *CxtI = nullptr);
   int getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp, int Index,
                      VectorType *SubTp);
   int getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
                        TTI::CastContextHint CCH, TTI::TargetCostKind CostKind,
                        const Instruction *I = nullptr);
-  int getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
-                         TTI::TargetCostKind CostKind,
-                         const Instruction *I = nullptr);
+  int getCmpSelInstrCost(
+      unsigned Opcode, Type *ValTy, Type *CondTy, TTI::TargetCostKind CostKind,
+      const Instruction *I = nullptr,
+      CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE);
   int getVectorInstrCost(unsigned Opcode, Type *Val, unsigned Index);
   unsigned getScalarizationOverhead(VectorType *Ty, const APInt &DemandedElts,
                                     bool Insert, bool Extract);
   int getMemoryOpCost(unsigned Opcode, Type *Src, MaybeAlign Alignment,
                       unsigned AddressSpace,
                       TTI::TargetCostKind CostKind,
                       const Instruction *I = nullptr);
   int getMaskedMemoryOpCost(

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

int X86TTIImpl::getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
   }
 
   return AdjustCost(
       BaseT::getCastInstrCost(Opcode, Dst, Src, CCH, CostKind, I));
 }
 
 int X86TTIImpl::getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
                                    TTI::TargetCostKind CostKind,
-                                   const Instruction *I) {
+                                   const Instruction *I,
+                                   CmpInst::Predicate VecPred) {
   // TODO: Handle other cost kinds.
   if (CostKind != TTI::TCK_RecipThroughput)
-    return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I);
+    return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I,
+                                     VecPred);
 
   // Legalize the type.
   std::pair<int, MVT> LT = TLI->getTypeLegalizationCost(DL, ValTy);
 
   MVT MTy = LT.second;
 
   int ISD = TLI->InstructionOpcodeToISD(Opcode);
   assert(ISD && "Invalid opcode");
int X86TTIImpl::getCmpSelInstrCost(unsigned Opcode, Type *ValTy, Type *CondTy,
   if (ST->hasSSE2())
     if (const auto *Entry = CostTableLookup(SSE2CostTbl, ISD, MTy))
       return LT.first * (ExtraCost + Entry->Cost);
 
   if (ST->hasSSE1())
     if (const auto *Entry = CostTableLookup(SSE1CostTbl, ISD, MTy))
       return LT.first * (ExtraCost + Entry->Cost);
 
-  return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I);
+  return BaseT::getCmpSelInstrCost(Opcode, ValTy, CondTy, CostKind, I, VecPred);
 }
 
 unsigned X86TTIImpl::getAtomicMemIntrinsicMaxElementSize() const { return 16; }
 
 int X86TTIImpl::getTypeBasedIntrinsicInstrCost(
   const IntrinsicCostAttributes &ICA, TTI::TargetCostKind CostKind) {
 
   // Costs should match the codegen from:

llvm/test/Analysis/CostModel/AArch64/vector-select.ll

 ; RUN: opt < %s -mtriple=aarch64--linux-gnu -cost-model -analyze | FileCheck %s --check-prefix=COST
 ; RUN: llc < %s -mtriple=aarch64--linux-gnu | FileCheck %s --check-prefix=CODE
 
 ; COST-LABEL: v8i8_select_eq
 ; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %cmp.1 = icmp eq <8 x i8> %a, %b
-; COST-NEXT:  Cost Model: Found an estimated cost of 29 for instruction:  %s.1 = select <8 x i1> %cmp.1, <8 x i8> %a, <8 x i8> %c
+; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %s.1 = select <8 x i1> %cmp.1, <8 x i8> %a, <8 x i8> %c
 
 ; CODE-LABEL: v8i8_select_eq
 ; CODE:       bb.0
 ; CODE-NEXT:    cmeq  v{{.+}}.8b, v{{.+}}.8b, v{{.+}}.8b
 ; CODE-NEXT:    bif   v{{.+}}.8b, v{{.+}}.8b, v{{.+}}.8b
 ; CODE-NEXT:    ret
 
 define <8 x i8> @v8i8_select_eq(<8 x i8> %a, <8 x i8> %b, <8 x i8> %c) {
   %cmp.1 = icmp eq <8 x i8> %a, %b
   %s.1 = select <8 x i1> %cmp.1, <8 x i8> %a, <8 x i8> %c
   ret <8 x i8> %s.1
 }
 
 ; COST-LABEL: v16i8_select_sgt
 ; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %cmp.1 = icmp sgt <16 x i8> %a, %b
-; COST-NEXT:  Cost Model: Found an estimated cost of 61 for instruction:  %s.1 = select <16 x i1> %cmp.1, <16 x i8> %a, <16 x i8> %c
+; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %s.1 = select <16 x i1> %cmp.1, <16 x i8> %a, <16 x i8> %c
 
 ; CODE-LABEL: v16i8_select_sgt
 ; CODE:       bb.0
 ; CODE-NEXT:    cmgt  v{{.+}}.16b, v{{.+}}.16b, v{{.+}}.16b
 ; CODE-NEXT:    bif   v{{.+}}.16b, v{{.+}}.16b, v{{.+}}.16b
 ; CODE-NEXT:    ret
 
 define <16 x i8> @v16i8_select_sgt(<16 x i8> %a, <16 x i8> %b, <16 x i8> %c) {
   %cmp.1 = icmp sgt <16 x i8> %a, %b
   %s.1 = select <16 x i1> %cmp.1, <16 x i8> %a, <16 x i8> %c
   ret <16 x i8> %s.1
 }
 
 ; COST-LABEL: v4i16_select_ne
 ; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %cmp.1 = icmp ne <4 x i16> %a, %b
-; COST-NEXT:  Cost Model: Found an estimated cost of 13 for instruction:  %s.1 = select <4 x i1> %cmp.1, <4 x i16> %a, <4 x i16> %c
+; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %s.1 = select <4 x i1> %cmp.1, <4 x i16> %a, <4 x i16> %c
 
 ; CODE-LABEL: v4i16_select_ne
 ; CODE:       bb.0
 ; CODE-NEXT:    cmeq  v{{.+}}.4h, v{{.+}}.4h, v{{.+}}.4h
 ; CODE-NEXT:    mvn   v{{.+}}.8b, v{{.+}}.8b
 ; CODE-NEXT:    bif   v{{.+}}.8b, v{{.+}}.8b, v{{.+}}.8b
 ; CODE-NEXT:    ret
 
 define <4 x i16> @v4i16_select_ne(<4 x i16> %a, <4 x i16> %b, <4 x i16> %c) {
   %cmp.1 = icmp ne <4 x i16> %a, %b
   %s.1 = select <4 x i1> %cmp.1, <4 x i16> %a, <4 x i16> %c
   ret <4 x i16> %s.1
 }
 
 ; COST-LABEL: v8i16_select_ugt
 ; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %cmp.1 = icmp ugt <8 x i16> %a, %b
-; COST-NEXT:  Cost Model: Found an estimated cost of 29 for instruction:  %s.1 = select <8 x i1> %cmp.1, <8 x i16> %a, <8 x i16> %c
+; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %s.1 = select <8 x i1> %cmp.1, <8 x i16> %a, <8 x i16> %c
 
 ; CODE-LABEL: v8i16_select_ugt
 ; CODE:       bb.0
 ; CODE-NEXT:    cmhi  v{{.+}}.8h, v{{.+}}.8h, v{{.+}}.8h
 ; CODE-NEXT:    bif   v{{.+}}.16b, v{{.+}}.16b, v{{.+}}.16b
 ; CODE-NEXT:    ret
 
 define <8 x i16> @v8i16_select_ugt(<8 x i16> %a, <8 x i16> %b, <8 x i16> %c) {
   %cmp.1 = icmp ugt <8 x i16> %a, %b
   %s.1 = select <8 x i1> %cmp.1, <8 x i16> %a, <8 x i16> %c
   ret <8 x i16> %s.1
 }
 
 ; COST-LABEL: v2i32_select_ule
 ; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %cmp.1 = icmp ule <2 x i32> %a, %b
-; COST-NEXT:  Cost Model: Found an estimated cost of 5 for instruction:   %s.1 = select <2 x i1> %cmp.1, <2 x i32> %a, <2 x i32> %c
+; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %s.1 = select <2 x i1> %cmp.1, <2 x i32> %a, <2 x i32> %c
 
 ; CODE-LABEL: v2i32_select_ule
 ; CODE:       bb.0
 ; CODE-NEXT:    cmhs  v{{.+}}.2s, v{{.+}}.2s, v{{.+}}.2s
 ; CODE-NEXT:    bif   v{{.+}}.8b, v{{.+}}.8b, v{{.+}}.8b
 ; CODE-NEXT:    ret
 
 define <2 x i32> @v2i32_select_ule(<2 x i32> %a, <2 x i32> %b, <2 x i32> %c) {
   %cmp.1 = icmp ule <2 x i32> %a, %b
   %s.1 = select <2 x i1> %cmp.1, <2 x i32> %a, <2 x i32> %c
   ret <2 x i32> %s.1
 }
 
 ; COST-LABEL: v4i32_select_ult
 ; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %cmp.1 = icmp ult <4 x i32> %a, %b
-; COST-NEXT:  Cost Model: Found an estimated cost of 13 for instruction:  %s.1 = select <4 x i1> %cmp.1, <4 x i32> %a, <4 x i32> %c
+; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %s.1 = select <4 x i1> %cmp.1, <4 x i32> %a, <4 x i32> %c
 
 ; CODE-LABEL: v4i32_select_ult
 ; CODE:       bb.0
 ; CODE-NEXT:    cmhi  v{{.+}}.4s, v{{.+}}.4s, v{{.+}}.4s
 ; CODE-NEXT:    bif   v{{.+}}.16b, v{{.+}}.16b, v{{.+}}.16b
 ; CODE-NEXT:    ret
 
 define <4 x i32> @v4i32_select_ult(<4 x i32> %a, <4 x i32> %b, <4 x i32> %c) {
   %cmp.1 = icmp ult <4 x i32> %a, %b
   %s.1 = select <4 x i1> %cmp.1, <4 x i32> %a, <4 x i32> %c
   ret <4 x i32> %s.1
 }
 
 ; COST-LABEL: v2i64_select_sle
 ; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %cmp.1 = icmp sle <2 x i64> %a, %b
-; COST-NEXT:  Cost Model: Found an estimated cost of 5 for instruction:   %s.1 = select <2 x i1> %cmp.1, <2 x i64> %a, <2 x i64> %c
+; COST-NEXT:  Cost Model: Found an estimated cost of 1 for instruction:   %s.1 = select <2 x i1> %cmp.1, <2 x i64> %a, <2 x i64> %c
 
 ; CODE-LABEL: v2i64_select_sle
 ; CODE:       bb.0
 ; CODE-NEXT:    cmge  v{{.+}}.2d, v{{.+}}.2d, v{{.+}}.2d
 ; CODE-NEXT:    bif   v{{.+}}.16b, v{{.+}}.16b, v{{.+}}.16b
 ; CODE-NEXT:    ret
 
 define <2 x i64> @v2i64_select_sle(<2 x i64> %a, <2 x i64> %b, <2 x i64> %c) {
   %cmp.1 = icmp sle <2 x i64> %a, %b
   %s.1 = select <2 x i1> %cmp.1, <2 x i64> %a, <2 x i64> %c
   ret <2 x i64> %s.1
 }
 
 ; COST-LABEL: v3i64_select_sle
 ; COST-NEXT:  Cost Model: Found an estimated cost of 2 for instruction:   %cmp.1 = icmp sle <3 x i64> %a, %b
-; COST-NEXT:  Cost Model: Found an estimated cost of 6 for instruction:   %s.1 = select <3 x i1> %cmp.1, <3 x i64> %a, <3 x i64> %c
+; COST-NEXT:  Cost Model: Found an estimated cost of 2 for instruction:   %s.1 = select <3 x i1> %cmp.1, <3 x i64> %a, <3 x i64> %c
 
 ; CODE-LABEL: v3i64_select_sle
 ; CODE:       bb.0
 ; CODE:    ldr
 ; CODE:    mov
 ; CODE:    mov
 ; CODE:    mov
 ; CODE:    cmge