[TLI][AArch64] Extend SLEEF vectorized functions mapping with VLA functions

[pawosm01]

Summary

This commit extends D134719 "[AArch64] Enable libm vectorized
functions via SLEEF" with the mappings for the scalable functions.

It also introduces all the necessary changes needed to support masked
interfaces.

[pawosm01]

formatting update only

[pawosm01]

one more formatting change

[sdesmalen]

Hi @pawosm01, the title of this patch peaked my interest so I had a look and left some comments. Thanks for working on this.

nit: you'll probably want to remove the [PATCH] from the title :)

llvm/lib/Transforms/Utils/InjectTLIMappings.cpp

43

nit: Is there a reason you've named this GlobalPredicate instead of just Predicate ?

113–119

nit: Is it worth doing:

for (bool Predicated : { false, true }) {
  for (ElementCount VF = ElementCount::getFixed(2), ...)
    AddVariantDecl(VF, Predicated)
  for (ElementCount VF = ElementCount::getScalable(2), ...)
    AddVariantDecl(VF, Predicated)
}

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

645 ↗

(On Diff #508498)

I feel a bit uncomfortable about this code, because there may be two variants available, a masked and an unmasked variant, where one may be vectorizable and the other one is not. I wonder if isFunctionVectorizable should return true when false is passed there is a predicated variant available. That should be safe, because it is always valid to use a predicated vector function when the vector loop itself doesn't require predication (although the opposite doesn't hold). It would require changing the interface a bit to also return whether the vectorizable function is predicated.

llvm/test/Transforms/LoopVectorize/AArch64/sleef-calls-aarch64.ll

16

Given that these functions are described as masked:

TLI_DEFINE_VECFUNC("acos", "_ZGVsMxv_acos",  SCALABLE(2), MASKED)
TLI_DEFINE_VECFUNC("acosf", "_ZGVsMxv_acosf", SCALABLE(4), MASKED)

Should these calls not pass a predicate mask operand?

[danielkiss]

I'd add note to the documentation too when scalable vectors are used the sleef library is expect to be compiled with it.

llvm/include/llvm/Analysis/VecFuncs.def

21–23

I won't introduce a new argument here, since MASKED is just indicate this interface is scalable so far.
Can we make the declarations less verbose?
e.g.:
#define FIXED(NL) ElementCount::getFixed(NL), false
#define SCALABLE(NL) ElementCount::getScalable(NL), true

llvm/lib/Analysis/VectorUtils.cpp

1532

for me the "Global" naming is a bit misleading or am I missing something?
Masked maybe?

[pawosm01]

post-review changes

[pawosm01]

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

645 ↗

(On Diff #508498)

Considering this is the only place where this overload is used, there is no point in changing or extending the interface. I've reverted to old parameter list and handle your idea inside of the overload of isFunctionVectorizable() called from here.

[pawosm01]

formatting changes again

[sdesmalen]

llvm/include/llvm/Analysis/VecFuncs.def

21–23

Sorry I missed this comment yesterday, but I'm not sure I agree with the suggestion to let SCALABLE imply MASKED. The two concepts are orthogonal, they just happen to have the same value for the set of vector function mappings added in this patch. It's perfectly possible to have an unmasked scalable interface and we may want to add those in the future.

The cost of adding the mask separately seemed quite low, is there another reason you prefer not to add this now?

llvm/lib/CodeGen/ReplaceWithVeclib.cpp

159 ↗

(On Diff #508793)

Can you make false a default option for this argument, so that you don't explicitly have to pass it?

[pawosm01]

Restored to the previous idea. And added default param value as suggested.

[pawosm01]

llvm/include/llvm/Analysis/VecFuncs.def

21–23

I tend to agree with @sdesmalen so I'm reverting to the previous shape.

[pawosm01]

llvm/lib/CodeGen/ReplaceWithVeclib.cpp

159 ↗

(On Diff #508793)

Yes

[pawosm01]

removed extra spaces that suddenly appeared in two lines.

[sdesmalen]

llvm/include/llvm/Analysis/VecFuncs.def

702

Does this need:

#undef MASKED
#undef NOMASK

?

llvm/lib/Analysis/TargetLibraryInfo.cpp

1184

When you #undef MASK/NOMASK in the .def file, this needs changing to something like:

/*MASK=*/false

[pawosm01]

post-review changes again

[pawosm01]

llvm/include/llvm/Analysis/VecFuncs.def

702

It does need it indeed.

[sdesmalen]

LGTM, thanks for addressing all comments.

llvm/test/Transforms/LoopVectorize/AArch64/sleef-calls-aarch64.ll

175

When enabling tail-folding (enabled with -sve-tail-folding=all) the loop no longer vectorizes because there is an Invalid cost for the call to atan2. This is unexpected, because intrinsic is predicated and so could handle the tail-folded loop.

Nothing that needs fixing in this patch, I'm just pointing out that there is more work to follow-on from this patch to benefit from the masked vector-function variants.

[danielkiss]

LGTM, thanks!

llvm/include/llvm/Analysis/VecFuncs.def

21–23

I have no objection since the interface is the struct VecDesc, here the macro would just save some repetition.

llvm/include/llvm/Analysis/TargetLibraryInfo.h

 
 /// Describes a possible vectorization of a function.
 /// Function 'VectorFnName' is equivalent to 'ScalarFnName' vectorized
 /// by a factor 'VectorizationFactor'.
 struct VecDesc {
   StringRef ScalarFnName;
   StringRef VectorFnName;
   ElementCount VectorizationFactor;
+  bool Masked;
 };
 
   enum LibFunc : unsigned {
 #define TLI_DEFINE_ENUM
 #include "llvm/Analysis/TargetLibraryInfo.def"
 
     NumLibFuncs,
     NotLibFunc
public:
   /// Calls addVectorizableFunctions with a known preset of functions for the
   /// given vector library.
   void addVectorizableFunctionsFromVecLib(enum VectorLibrary VecLib,
                                           const llvm::Triple &TargetTriple);
 
   /// Return true if the function F has a vector equivalent with vectorization
   /// factor VF.
   bool isFunctionVectorizable(StringRef F, const ElementCount &VF) const {
-    return !getVectorizedFunction(F, VF).empty();
+    return !(getVectorizedFunction(F, VF, false).empty() &&
+             getVectorizedFunction(F, VF, true).empty());
   }
 
   /// Return true if the function F has a vector equivalent with any
   /// vectorization factor.
   bool isFunctionVectorizable(StringRef F) const;
 
   /// Return the name of the equivalent of F, vectorized with factor VF. If no
   /// such mapping exists, return the empty string.
-  StringRef getVectorizedFunction(StringRef F, const ElementCount &VF) const;
+  StringRef getVectorizedFunction(StringRef F, const ElementCount &VF,
+                                  bool Masked) const;
 
   /// Set to true iff i32 parameters to library functions should have signext
   /// or zeroext attributes if they correspond to C-level int or unsigned int,
   /// respectively.
   void setShouldExtI32Param(bool Val) {
     ShouldExtI32Param = Val;
   }
 
bool has(LibFunc F) const {
     return getState(F) != TargetLibraryInfoImpl::Unavailable;
   }
   bool isFunctionVectorizable(StringRef F, const ElementCount &VF) const {
     return Impl->isFunctionVectorizable(F, VF);
   }
   bool isFunctionVectorizable(StringRef F) const {
     return Impl->isFunctionVectorizable(F);
   }
-  StringRef getVectorizedFunction(StringRef F, const ElementCount &VF) const {
-    return Impl->getVectorizedFunction(F, VF);
+  StringRef getVectorizedFunction(StringRef F, const ElementCount &VF,
+                                  bool Masked = false) const {
+    return Impl->getVectorizedFunction(F, VF, Masked);
   }
 
   /// Tests if the function is both available and a candidate for optimized code
   /// generation.
   bool hasOptimizedCodeGen(LibFunc F) const {
     if (getState(F) == TargetLibraryInfoImpl::Unavailable)
       return false;
     switch (F) {

llvm/include/llvm/Analysis/VecFuncs.def

 // the specified framework or library.
 
 #if defined(TLI_DEFINE_MASSV_VECFUNCS_NAMES)
 #define TLI_DEFINE_MASSV_VECFUNCS
 #define TLI_DEFINE_VECFUNC(SCAL, VEC, VF) VEC,
 #endif
 
 #define FIXED(NL) ElementCount::getFixed(NL)
 #define SCALABLE(NL) ElementCount::getScalable(NL)
+#define NOMASK false
+#define MASKED true
 
 #if !(defined(TLI_DEFINE_VECFUNC))
-#define TLI_DEFINE_VECFUNC(SCAL, VEC, VF) {SCAL, VEC, VF},
+#define TLI_DEFINE_VECFUNC(SCAL, VEC, VF) {SCAL, VEC, VF, NOMASK},
 #endif
 
 #if defined(TLI_DEFINE_ACCELERATE_VECFUNCS)
 // Accelerate framework's Vector Functions
 
 // Floating-Point Arithmetic and Auxiliary Functions
 TLI_DEFINE_VECFUNC("ceilf", "vceilf", FIXED(4))
 TLI_DEFINE_VECFUNC("fabsf", "vfabsf", FIXED(4))

 TLI_DEFINE_VECFUNC( "llvm.tan.f32", "_ZGVnN4v_tanf", FIXED(4))
 
 TLI_DEFINE_VECFUNC( "tanhf", "_ZGVnN4v_tanhf", FIXED(4))
 TLI_DEFINE_VECFUNC( "llvm.tanh.f32", "_ZGVnN4v_tanhf", FIXED(4))
 
 TLI_DEFINE_VECFUNC( "tgammaf", "_ZGVnN4v_tgammaf", FIXED(4))
 TLI_DEFINE_VECFUNC( "llvm.tgamma.f32", "_ZGVnN4v_tgammaf", FIXED(4))
 
+#elif defined(TLI_DEFINE_SLEEFGNUABI_SCALABLE_VECFUNCS)
+
+TLI_DEFINE_VECFUNC("acos", "_ZGVsMxv_acos",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("acosf", "_ZGVsMxv_acosf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("asin", "_ZGVsMxv_asin",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("asinf", "_ZGVsMxv_asinf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("atan", "_ZGVsMxv_atan",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("atanf", "_ZGVsMxv_atanf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("atan2", "_ZGVsMxvv_atan2",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("atan2f", "_ZGVsMxvv_atan2f", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("atanh", "_ZGVsMxv_atanh",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("atanhf", "_ZGVsMxv_atanhf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("cos", "_ZGVsMxv_cos",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("cosf", "_ZGVsMxv_cosf", SCALABLE(4), MASKED)
+TLI_DEFINE_VECFUNC("llvm.cos.f64", "_ZGVsMxv_cos", SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("llvm.cos.f32", "_ZGVsMxv_cosf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("cosh", "_ZGVsMxv_cosh",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("coshf", "_ZGVsMxv_coshf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("exp", "_ZGVsMxv_exp",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("expf", "_ZGVsMxv_expf", SCALABLE(4), MASKED)
+TLI_DEFINE_VECFUNC("llvm.exp.f64", "_ZGVsMxv_exp", SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("llvm.exp.f32", "_ZGVsMxv_expf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("exp2", "_ZGVsMxv_exp2",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("exp2f", "_ZGVsMxv_exp2f", SCALABLE(4), MASKED)
+TLI_DEFINE_VECFUNC("llvm.exp2.f64", "_ZGVsMxv_exp2", SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("llvm.exp2.f32", "_ZGVsMxv_exp2f", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("exp10", "_ZGVsMxv_exp10",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("exp10f", "_ZGVsMxv_exp10f", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("lgamma", "_ZGVsMxv_lgamma",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("lgammaf", "_ZGVsMxv_lgammaf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("log", "_ZGVsMxv_log",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("logf", "_ZGVsMxv_logf", SCALABLE(4), MASKED)
+TLI_DEFINE_VECFUNC("llvm.log.f64", "_ZGVsMxv_log", SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("llvm.log.f32", "_ZGVsMxv_logf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("log10", "_ZGVsMxv_log10",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("log10f", "_ZGVsMxv_log10f", SCALABLE(4), MASKED)
+TLI_DEFINE_VECFUNC("llvm.log10.f64", "_ZGVsMxv_log10", SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("llvm.log10.f32", "_ZGVsMxv_log10f", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("pow", "_ZGVsMxvv_pow", SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("powf", "_ZGVsMxvv_powf", SCALABLE(4), MASKED)
+TLI_DEFINE_VECFUNC("llvm.pow.f64", "_ZGVsMxvv_pow", SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("llvm.pow.f32", "_ZGVsMxvv_powf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("sin", "_ZGVsMxv_sin",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("sinf", "_ZGVsMxv_sinf", SCALABLE(4), MASKED)
+TLI_DEFINE_VECFUNC("llvm.sin.f64", "_ZGVsMxv_sin", SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("llvm.sin.f32", "_ZGVsMxv_sinf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("sinh", "_ZGVsMxv_sinh",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("sinhf", "_ZGVsMxv_sinhf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("sqrt", "_ZGVsMxv_sqrt",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("sqrtf", "_ZGVsMxv_sqrtf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("tan", "_ZGVsMxv_tan",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("tanf", "_ZGVsMxv_tanf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("tanh", "_ZGVsMxv_tanh",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("tanhf", "_ZGVsMxv_tanhf", SCALABLE(4), MASKED)
+
+TLI_DEFINE_VECFUNC("tgamma", "_ZGVsMxv_tgamma",  SCALABLE(2), MASKED)
+TLI_DEFINE_VECFUNC("tgammaf", "_ZGVsMxv_tgammaf", SCALABLE(4), MASKED)
+
 #else
 #error "Must choose which vector library functions are to be defined."
 #endif
 
+#undef MASKED
+#undef NOMASK
+#undef SCALABLE
+#undef FIXED
+
 #undef TLI_DEFINE_VECFUNC
 #undef TLI_DEFINE_ACCELERATE_VECFUNCS
 #undef TLI_DEFINE_DARWIN_LIBSYSTEM_M_VECFUNCS
 #undef TLI_DEFINE_LIBMVEC_X86_VECFUNCS
 #undef TLI_DEFINE_MASSV_VECFUNCS
 #undef TLI_DEFINE_SVML_VECFUNCS
 #undef TLI_DEFINE_SLEEFGNUABI_VF2_VECFUNCS
 #undef TLI_DEFINE_SLEEFGNUABI_VF4_VECFUNCS
+#undef TLI_DEFINE_SLEEFGNUABI_SCALABLE_VECFUNCS
 #undef TLI_DEFINE_MASSV_VECFUNCS_NAMES

llvm/include/llvm/Analysis/VectorUtils.h

 /// the TLI mappings. It is the responsibility of the caller to make sure that
 /// this is only used if all parameters in the vector function are vector type.
 /// This returned string holds scalar-to-vector mapping:
 ///    _ZGV<isa><mask><vlen><vparams>_<scalarname>(<vectorname>)
 ///
 /// where:
 ///
 /// <isa> = "_LLVM_"
-/// <mask> = "N". Note: TLI does not support masked interfaces.
+/// <mask> = "M" if masked, "N" if no mask.
 /// <vlen> = Number of concurrent lanes, stored in the `VectorizationFactor`
 ///          field of the `VecDesc` struct. If the number of lanes is scalable
 ///          then 'x' is printed instead.
 /// <vparams> = "v", as many as are the numArgs.
 /// <scalarname> = the name of the scalar function.
 /// <vectorname> = the name of the vector function.
 std::string mangleTLIVectorName(StringRef VectorName, StringRef ScalarName,
-                                unsigned numArgs, ElementCount VF);
+                                unsigned numArgs, ElementCount VF,
+                                bool Masked = false);
 
 /// Retrieve the `VFParamKind` from a string token.
 VFParamKind getVFParamKindFromString(const StringRef Token);
 
 // Name of the attribute where the variant mappings are stored.
 static constexpr char const *MappingsAttrName = "vector-function-abi-variant";
 
 /// Populates a set of strings representing the Vector Function ABI variants

llvm/lib/Analysis/TargetLibraryInfo.cpp

     #include "llvm/Analysis/VecFuncs.def"
     };
     addVectorizableFunctions(VecFuncs);
     break;
   }
   case SLEEFGNUABI: {
     const VecDesc VecFuncs_VF2[] = {
 #define TLI_DEFINE_SLEEFGNUABI_VF2_VECFUNCS
+#define TLI_DEFINE_VECFUNC(SCAL, VEC, VF) {SCAL, VEC, VF, /* MASK = */ false},
 #include "llvm/Analysis/VecFuncs.def"
     };
     const VecDesc VecFuncs_VF4[] = {
 #define TLI_DEFINE_SLEEFGNUABI_VF4_VECFUNCS
+#define TLI_DEFINE_VECFUNC(SCAL, VEC, VF) {SCAL, VEC, VF, /* MASK = */ false},
+#include "llvm/Analysis/VecFuncs.def"
+    };
+    const VecDesc VecFuncs_VFScalable[] = {
+#define TLI_DEFINE_SLEEFGNUABI_SCALABLE_VECFUNCS
+#define TLI_DEFINE_VECFUNC(SCAL, VEC, VF, MASK) {SCAL, VEC, VF, MASK},
 #include "llvm/Analysis/VecFuncs.def"
     };
 
     switch (TargetTriple.getArch()) {
     default:
       break;
     case llvm::Triple::aarch64:
     case llvm::Triple::aarch64_be:
       addVectorizableFunctions(VecFuncs_VF2);
       addVectorizableFunctions(VecFuncs_VF4);
+      addVectorizableFunctions(VecFuncs_VFScalable);
       break;
     }
     break;
   }
   case NoLibrary:
     break;
   }
 }
 
 bool TargetLibraryInfoImpl::isFunctionVectorizable(StringRef funcName) const {
   funcName = sanitizeFunctionName(funcName);
   if (funcName.empty())
     return false;
 
   std::vector<VecDesc>::const_iterator I =
       llvm::lower_bound(VectorDescs, funcName, compareWithScalarFnName);
   return I != VectorDescs.end() && StringRef(I->ScalarFnName) == funcName;
 }
 
-StringRef
-TargetLibraryInfoImpl::getVectorizedFunction(StringRef F,
-                                             const ElementCount &VF) const {
+StringRef TargetLibraryInfoImpl::getVectorizedFunction(StringRef F,
+                                                       const ElementCount &VF,
+                                                       bool Masked) const {
   F = sanitizeFunctionName(F);
   if (F.empty())
     return F;
   std::vector<VecDesc>::const_iterator I =
       llvm::lower_bound(VectorDescs, F, compareWithScalarFnName);
   while (I != VectorDescs.end() && StringRef(I->ScalarFnName) == F) {
-    if (I->VectorizationFactor == VF)
+    if ((I->VectorizationFactor == VF) && (I->Masked == Masked))
       return I->VectorFnName;
     ++I;
   }
   return StringRef();
 }
 
 TargetLibraryInfo TargetLibraryAnalysis::run(const Function &F,
                                              FunctionAnalysisManager &) {

llvm/lib/Analysis/VectorUtils.cpp

   std::transform(Members.begin(), Members.end(), std::back_inserter(VL),
                  [](std::pair<int, Instruction *> p) { return p.second; });
   propagateMetadata(NewInst, VL);
 }
 }
 
 std::string VFABI::mangleTLIVectorName(StringRef VectorName,
                                        StringRef ScalarName, unsigned numArgs,
-                                       ElementCount VF) {
+                                       ElementCount VF, bool Masked) {
   SmallString<256> Buffer;
   llvm::raw_svector_ostream Out(Buffer);
-  Out << "_ZGV" << VFABI::_LLVM_ << "N";
+  Out << "_ZGV" << VFABI::_LLVM_ << (Masked ? "M" : "N");
   if (VF.isScalable())
     Out << 'x';
   else
     Out << VF.getFixedValue();
   for (unsigned I = 0; I < numArgs; ++I)
     Out << "v";
   Out << "_" << ScalarName << "(" << VectorName << ")";
   return std::string(Out.str());

llvm/lib/Transforms/Utils/InjectTLIMappings.cpp

STATISTIC(NumCompUsedAdded,
           "Number of `@llvm.compiler.used` operands that have been added.");
 
 /// A helper function that adds the vector function declaration that
 /// vectorizes the CallInst CI with a vectorization factor of VF
 /// lanes. The TLI assumes that all parameters and the return type of
 /// CI (other than void) need to be widened to a VectorType of VF
 /// lanes.
 static void addVariantDeclaration(CallInst &CI, const ElementCount &VF,
-                                  const StringRef VFName) {
+                                  bool Predicate, const StringRef VFName) {
   Module *M = CI.getModule();
 
   // Add function declaration.
   Type *RetTy = ToVectorTy(CI.getType(), VF);
   SmallVector<Type *, 4> Tys;
   for (Value *ArgOperand : CI.args())
     Tys.push_back(ToVectorTy(ArgOperand->getType(), VF));
   assert(!CI.getFunctionType()->isVarArg() &&
          "VarArg functions are not supported.");
+  if (Predicate)
+    Tys.push_back(ToVectorTy(Type::getInt1Ty(RetTy->getContext()), VF));
   FunctionType *FTy = FunctionType::get(RetTy, Tys, /*isVarArg=*/false);
   Function *VectorF =
       Function::Create(FTy, Function::ExternalLinkage, VFName, M);
   VectorF->copyAttributesFrom(CI.getCalledFunction());
   ++NumVFDeclAdded;
   LLVM_DEBUG(dbgs() << DEBUG_TYPE << ": Added to the module: `" << VFName
                     << "` of type " << *(VectorF->getType()) << "\n");
 
static void addMappingsFromTLI(const TargetLibraryInfo &TLI, CallInst &CI) {
   if (!TLI.isFunctionVectorizable(ScalarName))
     return;
   SmallVector<std::string, 8> Mappings;
   VFABI::getVectorVariantNames(CI, Mappings);
   Module *M = CI.getModule();
   const SetVector<StringRef> OriginalSetOfMappings(Mappings.begin(),
                                                    Mappings.end());
 
-  auto AddVariantDecl = [&](const ElementCount &VF) {
+  auto AddVariantDecl = [&](const ElementCount &VF, bool Predicate) {
     const std::string TLIName =
-        std::string(TLI.getVectorizedFunction(ScalarName, VF));
+        std::string(TLI.getVectorizedFunction(ScalarName, VF, Predicate));
     if (!TLIName.empty()) {
-      std::string MangledName =
-          VFABI::mangleTLIVectorName(TLIName, ScalarName, CI.arg_size(), VF);
+      std::string MangledName = VFABI::mangleTLIVectorName(
+          TLIName, ScalarName, CI.arg_size(), VF, Predicate);
       if (!OriginalSetOfMappings.count(MangledName)) {
         Mappings.push_back(MangledName);
         ++NumCallInjected;
       }
       Function *VariantF = M->getFunction(TLIName);
       if (!VariantF)
-        addVariantDeclaration(CI, VF, TLIName);
+        addVariantDeclaration(CI, VF, Predicate, TLIName);
     }
   };
 
   //  All VFs in the TLI are powers of 2.
   ElementCount WidestFixedVF, WidestScalableVF;
   TLI.getWidestVF(ScalarName, WidestFixedVF, WidestScalableVF);
 
+  for (bool Predicated : {false, true}) {
     for (ElementCount VF = ElementCount::getFixed(2);
          ElementCount::isKnownLE(VF, WidestFixedVF); VF *= 2)
-    AddVariantDecl(VF);
+      AddVariantDecl(VF, Predicated);
 
-  // TODO: Add scalable variants once we're able to test them.
-  assert(WidestScalableVF.isZero() &&
-         "Scalable vector mappings not yet supported");
+    for (ElementCount VF = ElementCount::getScalable(2);
+         ElementCount::isKnownLE(VF, WidestScalableVF); VF *= 2)
+      AddVariantDecl(VF, Predicated);
+  }
 
   VFABI::setVectorVariantNames(&CI, Mappings);
 }
 
 static bool runImpl(const TargetLibraryInfo &TLI, Function &F) {
   for (auto &I : instructions(F))
     if (auto CI = dyn_cast<CallInst>(&I))
       addMappingsFromTLI(TLI, *CI);

llvm/test/Transforms/LoopVectorize/AArch64/sleef-calls-aarch64.ll

 ; Do NOT use -O3. It will lower exp2 to ldexp, and the test will fail.
-; RUN: opt -vector-library=sleefgnuabi -replace-with-veclib < %s | opt -vector-library=sleefgnuabi -passes=inject-tli-mappings,loop-unroll,loop-vectorize -S | FileCheck %s
+; RUN: opt -vector-library=sleefgnuabi -replace-with-veclib < %s | opt -vector-library=sleefgnuabi -passes=inject-tli-mappings,loop-unroll,loop-vectorize -S | FileCheck %s --check-prefixes=CHECK,NEON
+; RUN: opt -mattr=+sve -vector-library=sleefgnuabi -replace-with-veclib < %s | opt -vector-library=sleefgnuabi -passes=inject-tli-mappings,loop-unroll,loop-vectorize -S | FileCheck %s --check-prefixes=CHECK,SVE
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64-unknown-linux-gnu"
 
 declare double @acos(double) #0
 declare float @acosf(float) #0
 declare double @llvm.acos.f64(double) #0
 declare float @llvm.acos.f32(float) #0
 
 define void @acos_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @acos_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_acos(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_acos(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_acos(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @acos(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @acos_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @acos_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_acosf(<4 x float> [[TMP4:%.*]])
+  ; NEON:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_acosf(<4 x float> [[TMP4:%.*]])
+  ; SVE:     [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_acosf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @asin(double) #0
 declare float @asinf(float) #0
 declare double @llvm.asin.f64(double) #0
 declare float @llvm.asin.f32(float) #0
 
 define void @asin_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @asin_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_asin(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_asin(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_asin(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @asin(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @asin_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @asin_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_asinf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_asinf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_asinf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @atan(double) #0
 declare float @atanf(float) #0
 declare double @llvm.atan.f64(double) #0
 declare float @llvm.atan.f32(float) #0
 
 define void @atan_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @atan_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_atan(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_atan(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_atan(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @atan(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @atan_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @atan_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_atanf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_atanf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_atanf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @atan2(double, double) #0
 declare float @atan2f(float, float) #0
 declare double @llvm.atan2.f64(double, double) #0
 declare float @llvm.atan2.f32(float, float) #0
 
 define void @atan2_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @atan2_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2vv_atan2(<2 x double> [[TMP4:%.*]], <2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2vv_atan2(<2 x double> [[TMP4:%.*]], <2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxvv_atan2(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @atan2(double %conv, double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @atan2_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @atan2_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4vv_atan2f(<4 x float> [[TMP4:%.*]], <4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4vv_atan2f(<4 x float> [[TMP4:%.*]], <4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxvv_atan2f(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @atanh(double) #0
 declare float @atanhf(float) #0
 declare double @llvm.atanh.f64(double) #0
 declare float @llvm.atanh.f32(float) #0
 
 define void @atanh_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @atanh_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_atanh(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_atanh(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_atanh(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @atanh(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @atanh_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @atanh_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_atanhf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_atanhf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_atanhf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @cos(double) #0
 declare float @cosf(float) #0
 declare double @llvm.cos.f64(double) #0
 declare float @llvm.cos.f32(float) #0
 
 define void @cos_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @cos_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_cos(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_cos(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_cos(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @cos(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @cos_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @cos_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_cosf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_cosf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_cosf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @cosh(double) #0
 declare float @coshf(float) #0
 declare double @llvm.cosh.f64(double) #0
 declare float @llvm.cosh.f32(float) #0
 
 define void @cosh_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @cosh_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_cosh(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_cosh(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_cosh(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @cosh(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @cosh_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @cosh_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_coshf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_coshf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_coshf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @exp(double) #0
 declare float @expf(float) #0
 declare double @llvm.exp.f64(double) #0
 declare float @llvm.exp.f32(float) #0
 
 define void @exp_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @exp_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_exp(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_exp(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_exp(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @exp(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @exp_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @exp_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_expf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_expf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_expf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @exp2(double) #0
 declare float @exp2f(float) #0
 declare double @llvm.exp2.f64(double) #0
 declare float @llvm.exp2.f32(float) #0
 
 define void @exp2_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @exp2_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_exp2(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_exp2(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_exp2(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @exp2(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @exp2_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @exp2_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_exp2f(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_exp2f(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_exp2f(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @exp10(double) #0
 declare float @exp10f(float) #0
 declare double @llvm.exp10.f64(double) #0
 declare float @llvm.exp10.f32(float) #0
 
 define void @exp10_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @exp10_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_exp10(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_exp10(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_exp10(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @exp10(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @exp10_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @exp10_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_exp10f(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_exp10f(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_exp10f(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @lgamma(double) #0
 declare float @lgammaf(float) #0
 declare double @llvm.lgamma.f64(double) #0
 declare float @llvm.lgamma.f32(float) #0
 
 define void @lgamma_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @lgamma_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_lgamma(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_lgamma(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_lgamma(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @lgamma(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @lgamma_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @lgamma_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_lgammaf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_lgammaf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_lgammaf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @log10(double) #0
 declare float @log10f(float) #0
 declare double @llvm.log10.f64(double) #0
 declare float @llvm.log10.f32(float) #0
 
 define void @log10_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @log10_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_log10(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_log10(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_log10(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @log10(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @log10_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @log10_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_log10f(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_log10f(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_log10f(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @log(double) #0
 declare float @logf(float) #0
 declare double @llvm.log.f64(double) #0
 declare float @llvm.log.f32(float) #0
 
 define void @log_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @log_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_log(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_log(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_log(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @log(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @log_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @log_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_logf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_logf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_logf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @pow(double, double) #0
 declare float @powf(float, float) #0
 declare double @llvm.pow.f64(double, double) #0
 declare float @llvm.pow.f32(float, float) #0
 
 define void @pow_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @pow_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2vv_pow(<2 x double> [[TMP4:%.*]], <2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2vv_pow(<2 x double> [[TMP4:%.*]], <2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxvv_pow(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @pow(double %conv, double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @pow_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @pow_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4vv_powf(<4 x float> [[TMP4:%.*]], <4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4vv_powf(<4 x float> [[TMP4:%.*]], <4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxvv_powf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @sin(double) #0
 declare float @sinf(float) #0
 declare double @llvm.sin.f64(double) #0
 declare float @llvm.sin.f32(float) #0
 
 define void @sin_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @sin_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_sin(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_sin(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_sin(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @sin(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @sin_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @sin_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_sinf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_sinf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_sinf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @sinh(double) #0
 declare float @sinhf(float) #0
 declare double @llvm.sinh.f64(double) #0
 declare float @llvm.sinh.f32(float) #0
 
 define void @sinh_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @sinh_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_sinh(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_sinh(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_sinh(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @sinh(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @sinh_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @sinh_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_sinhf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_sinhf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_sinhf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @sqrt(double) #0
 declare float @sqrtf(float) #0
 declare double @llvm.sqrt.f64(double) #0
 declare float @llvm.sqrt.f32(float) #0
 
 define void @sqrt_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @sqrt_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_sqrt(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_sqrt(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_sqrt(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @sqrt(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @sqrt_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @sqrt_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_sqrtf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_sqrtf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_sqrtf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to float
   %call = tail call float @sqrtf(float %conv)
   %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
   store float %call, float* %arrayidx, align 4
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
-  
 define void @llvm_sqrt_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @llvm_sqrt_f64(
-  ; CHECK:    [[TMP5:%.*]] = call fast <2 x double> @llvm.sqrt.v2f64(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call fast <2 x double> @llvm.sqrt.v2f64(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call fast <vscale x 2 x double> @llvm.sqrt.nxv2f64(<vscale x 2 x double> [[TMP4:%.*]])
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call fast double @llvm.sqrt.f64(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @llvm_sqrt_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @llvm_sqrt_f32(
-  ; CHECK:    [[TMP5:%.*]] = call fast <4 x float> @llvm.sqrt.v4f32(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call fast <4 x float> @llvm.sqrt.v4f32(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call fast <vscale x 4 x float> @llvm.sqrt.nxv4f32(<vscale x 4 x float> [[TMP4:%.*]])
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @tan(double) #0
 declare float @tanf(float) #0
 declare double @llvm.tan.f64(double) #0
 declare float @llvm.tan.f32(float) #0
 
 define void @tan_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @tan_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_tan(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_tan(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_tan(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @tan(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @tan_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @tan_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_tanf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_tanf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_tanf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @tanh(double) #0
 declare float @tanhf(float) #0
 declare double @llvm.tanh.f64(double) #0
 declare float @llvm.tanh.f32(float) #0
 
 define void @tanh_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @tanh_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_tanh(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_tanh(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_tanh(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @tanh(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @tanh_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @tanh_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_tanhf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_tanhf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_tanhf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32

 
 declare double @tgamma(double) #0
 declare float @tgammaf(float) #0
 declare double @llvm.tgamma.f64(double) #0
 declare float @llvm.tgamma.f32(float) #0
 
 define void @tgamma_f64(double* nocapture %varray) {
   ; CHECK-LABEL: @tgamma_f64(
-  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_tgamma(<2 x double> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_tgamma(<2 x double> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 2 x double> @_ZGVsMxv_tgamma(<vscale x 2 x double> [[TMP4:%.*]], <vscale x 2 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32
   %conv = sitofp i32 %tmp to double
   %call = tail call double @tgamma(double %conv)
   %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
   store double %call, double* %arrayidx, align 8
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond = icmp eq i64 %iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
   for.end:
   ret void
 }
 
 define void @tgamma_f32(float* nocapture %varray) {
   ; CHECK-LABEL: @tgamma_f32(
-  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_tgammaf(<4 x float> [[TMP4:%.*]])
+  ; NEON:     [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_tgammaf(<4 x float> [[TMP4:%.*]])
+  ; SVE:      [[TMP5:%.*]] = call <vscale x 4 x float> @_ZGVsMxv_tgammaf(<vscale x 4 x float> [[TMP4:%.*]], <vscale x 4 x i1> {{.*}})
   ; CHECK:    ret void
   ;
   entry:
   br label %for.body
 
   for.body:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %tmp = trunc i64 %iv to i32