[TLI] Per-function fveclib for math library used for vectorization

[wenlei]

Summary

Encode -fveclib setting as per-function attribute so it can be threaded through to LTO backends. Accordingly, per-function TLI now reads the attribute and populated available vector function list based on that. Note that we expect functions within the same module to share fveclib setting, so vector function list is still shared between functions, as part of the shared TargetLibraryInfoImpl. Inlining between functions with different vect lib attribute is now blocked.

[wenlei]

update

[wenlei]

rebase

[tejohnson]

Needs testing of the inline handling, and of LTO linking IR with different attributes (which is going to hit your assert, see below).

clang/lib/CodeGen/CGCall.cpp

1987

Nit: why not "vec-lib" or just "veclib", to match the option?

llvm/include/llvm/Analysis/TargetLibraryInfo.h

47

Key comment about handling of TLII vs TLI. The former is computed once per module by the analysis (which is going to be the combined module in the case of LTO), the latter is the per-function data structure.

91

To avoid building and storing the VectorDescs and ScalarDescs for every function in the TLI, what I would do is keep 3 sets of VectorDescs/ScalarDescs on the TLII object (one set per possible veclib, built once per module during construction of the TLII), then move the new VectorLibrary member to the TLI and set it there per function based on the attribute, and use it to select which pair of VectorDescs/ScalarDescs is queried.

275–276

I don't think this will do anything currently since the TLII is built once per module by the analysis. You'll hit your assert about incompatibility below first, see comment there.

llvm/lib/Analysis/TargetLibraryInfo.cpp

1549

You'll certainly hit this assert if you try LTO linking two .ll files built with different -fveclib options, because the TLII is built once per module by the analysis.

1631

This is going to override the baseline TLI veclib with whatever is the latest function we build a TLI for (and you'll hit the assert as noted earlier if they conflict).

[wenlei]

Thanks for taking a look and the suggestions, @tejohnson. Yeah, you're right about the potential conflict of attributes. Initially I thought even though we now allow this to be per-function, but since it comes from per-module switch, module level consistency can still be expected which can simplify things a bit (hence the assertion). But I overlooked the combined module from LTO.

I will get back to this later in the week - the change will be a bit more involving as there're a few other places where we populate the module level function list directly for TLII (clang and opt).

[wenlei]

address feedback, allow functions within a module to have different vectlib setting. add test case for inline compatibility check.

[tejohnson]

Great, thanks! A few minor comments below. Looks like it needs a clang-format too.

llvm/include/llvm/Analysis/TargetLibraryInfo.h

239

Suggest moving the implementation of this constructor to the .cpp file, in which case you can just set VectLibrary directly from ClVectorLibrary there and remove the member on the Impl object.

274

This is set via a flag called "inline-caller-superset-nobuiltin". Suggest changing the name to something like "inline-caller-superset-tli" to reflect new larger scope. Also add a check with that option to your new inline test case.

llvm/lib/Analysis/TargetLibraryInfo.cpp

576

Why not just have "i < NumVecLibs"?

587

ditto

1580

Should these two be llvm_unreachable?

[wenlei]

llvm/include/llvm/Analysis/TargetLibraryInfo.h

239

There're utilities that use TargetLibraryInfo, but don't link with TargetLibraryInfo.o. And looking at TargetLibraryInfo, all of the functions are in this header, so I assumed it's intentional to keep this type self-contained in this header, as it's public API, which is why I add ClVectorLibrary to Impl to pass it back to TargetLibraryInfo. For TargetLibraryInfoImpl, it's ok to have the implementation outside of the header. I can give it a try if keeping the class implementation/definition self-contained in the header isn't important.

llvm/lib/Analysis/TargetLibraryInfo.cpp

576

Good catch, thanks..

[tejohnson]

llvm/include/llvm/Analysis/TargetLibraryInfo.h

239

I don't think there should be anything using TLI without linking with libAnalysis, which contains TargetLibraryInfo.o. I don't think it should be important to keep the implementation in the header, any more so than for other headers in the Analysis library.

[wenlei]

address feedback

[wenlei]

llvm/include/llvm/Analysis/TargetLibraryInfo.h

239

Ok, I moved it to .cpp, thanks!

[tejohnson]

lgtm. I think one check is missing in the test, see comment below.

llvm/test/Transforms/Inline/veclib-compat.ll

28 ↗

(On Diff #256437)

I think NOSUPERSET should also check that there is still a call here. You can probably replace some of the duplicated checks with COMMON in this function too.

[wenlei]

rebase, update test

[nikic]

This change causes a ~0.5% compile-time regressions: http://llvm-compile-time-tracker.com/compare.php?from=5b18b6e9a84d985c0a907009fb71de7c1943bc88&to=60c642e74be6af86906d9f3d982728be7bd4329f&stat=instructions This is quite a lot as these things go, so it would be great if you could double check if there's any optimization potential here. In particular I'm wondering why this affects normal builds so much, even though they (presumably?) don't use any veclib at all.

[wenlei]

In D77632#1974015, @nikic wrote:

This change causes a ~0.5% compile-time regressions: http://llvm-compile-time-tracker.com/compare.php?from=5b18b6e9a84d985c0a907009fb71de7c1943bc88&to=60c642e74be6af86906d9f3d982728be7bd4329f&stat=instructions This is quite a lot as these things go, so it would be great if you could double check if there's any optimization potential here. In particular I'm wondering why this affects normal builds so much, even though they (presumably?) don't use any veclib at all.

Thanks for the heads-up. This is surprising but there is a change even when veclib is not used - in order to allow each function to use different veclib without duplicating the work of populating vector function list for each function, we now always pre-populate vector function list for three supported vector libraries for each module. However 0.5% compile-time for that work given it's per-module is not expected. I suspect we may be passing/copying TLII around more than we anticipated (now we always have more stuff to copy). I will take a look. We could also turn this into a lazy initialization - only populate the needed list for module level TLII when it's first queried by a function level TLI.

[tejohnson]

In D77632#1974363, @wenlei wrote:

In D77632#1974015, @nikic wrote:

This change causes a ~0.5% compile-time regressions: http://llvm-compile-time-tracker.com/compare.php?from=5b18b6e9a84d985c0a907009fb71de7c1943bc88&to=60c642e74be6af86906d9f3d982728be7bd4329f&stat=instructions This is quite a lot as these things go, so it would be great if you could double check if there's any optimization potential here. In particular I'm wondering why this affects normal builds so much, even though they (presumably?) don't use any veclib at all.

Thanks for the heads-up. This is surprising but there is a change even when veclib is not used - in order to allow each function to use different veclib without duplicating the work of populating vector function list for each function, we now always pre-populate vector function list for three supported vector libraries for each module. However 0.5% compile-time for that work given it's per-module is not expected. I suspect we may be passing/copying TLII around more than we anticipated (now we always have more stuff to copy). I will take a look. We could also turn this into a lazy initialization - only populate the needed list for module level TLII when it's first queried by a function level TLI.

Hmm, yeah that is surprising, because the TLII should be built once per module per TLI analysis, which is never invalidated. We've gone from populating one set of vec libs to 3, I wouldn't have thought that was particularly expensive, so it would be good to see what is going on here and confirm we are only building this once as expected.

Looking at the compile time data at that link, interestingly the "instructions" metric increased, but not wall time or cycles or task clock - they were all neutral.

[mehdi_amini]

The existing TLI provides a very convenient way to define a VecLib without LLVM knowing about it ahead of time. This feature is important for any embedded use of LLVM as a library out-of-tree (I'll add a unit-test in-tree).
I don't think it is a big change to this patch to preserve the current ability but I wanted to check first (and in the meantime I reverted in temporarily in https://reviews.llvm.org/D77925 to avoid the feature regression).

At the moment the place where you seem to use this knowledge is with the enum VectorLibrary in the TargetLibraryInfoImpl class, and the VecLibDescs array which statically contains the known VecLib.
It seems to me that if we replace this enum with a string instead to identify the VecLib everything should still hold together and this would fit with minor changes to this path. The VecLibDescs could just be a StringMap<VectorLibraryDescriptors> in this case.

That was a third-party (in my case the XLA compiler) can still register its own "XLA" VecLib and add all the descriptors.

How does it sound?

[tejohnson]

In D77632#1975619, @mehdi_amini wrote:

The existing TLI provides a very convenient way to define a VecLib without LLVM knowing about it ahead of time. This feature is important for any embedded use of LLVM as a library out-of-tree (I'll add a unit-test in-tree).
I don't think it is a big change to this patch to preserve the current ability but I wanted to check first (and in the meantime I reverted in temporarily in https://reviews.llvm.org/D77925 to avoid the feature regression).

At the moment the place where you seem to use this knowledge is with the enum VectorLibrary in the TargetLibraryInfoImpl class, and the VecLibDescs array which statically contains the known VecLib.
It seems to me that if we replace this enum with a string instead to identify the VecLib everything should still hold together and this would fit with minor changes to this path. The VecLibDescs could just be a StringMap<VectorLibraryDescriptors> in this case.

That was a third-party (in my case the XLA compiler) can still register its own "XLA" VecLib and add all the descriptors.

How does it sound?

I think this should work. Just reiterating something we chatted about off patch yesterday, we really need a unit test that mimics the behavior utilized by the XLA compiler, for regression testing.

[mehdi_amini]

In D77632#1976126, @tejohnson wrote:

I think this should work. Just reiterating something we chatted about off patch yesterday, we really need a unit test that mimics the behavior utilized by the XLA compiler, for regression testing.

Yes I pinged some of the XLA folks to make it happen.

[wenlei]

In D77632#1975619, @mehdi_amini wrote:

The existing TLI provides a very convenient way to define a VecLib without LLVM knowing about it ahead of time. This feature is important for any embedded use of LLVM as a library out-of-tree (I'll add a unit-test in-tree).
I don't think it is a big change to this patch to preserve the current ability but I wanted to check first (and in the meantime I reverted in temporarily in https://reviews.llvm.org/D77925 to avoid the feature regression).

At the moment the place where you seem to use this knowledge is with the enum VectorLibrary in the TargetLibraryInfoImpl class, and the VecLibDescs array which statically contains the known VecLib.
It seems to me that if we replace this enum with a string instead to identify the VecLib everything should still hold together and this would fit with minor changes to this path. The VecLibDescs could just be a StringMap<VectorLibraryDescriptors> in this case.

That was a third-party (in my case the XLA compiler) can still register its own "XLA" VecLib and add all the descriptors.

How does it sound?

Thanks for the explanation about the revert. The proposal of using a StringMap to maintain the openness sounds good to me. And agree with @tejohnson, if the openness is a feature, it should be covered in tests, otherwise it can feel somewhat like a loophole and prone to breakage, though I can see how it can be useful.. Hope this patch can be restored with tweaks soon (we have workloads with very visible vectorization that relies on this).

[wenlei]

In D77632#1974409, @tejohnson wrote:

In D77632#1974363, @wenlei wrote:

In D77632#1974015, @nikic wrote:

This change causes a ~0.5% compile-time regressions: http://llvm-compile-time-tracker.com/compare.php?from=5b18b6e9a84d985c0a907009fb71de7c1943bc88&to=60c642e74be6af86906d9f3d982728be7bd4329f&stat=instructions This is quite a lot as these things go, so it would be great if you could double check if there's any optimization potential here. In particular I'm wondering why this affects normal builds so much, even though they (presumably?) don't use any veclib at all.

Thanks for the heads-up. This is surprising but there is a change even when veclib is not used - in order to allow each function to use different veclib without duplicating the work of populating vector function list for each function, we now always pre-populate vector function list for three supported vector libraries for each module. However 0.5% compile-time for that work given it's per-module is not expected. I suspect we may be passing/copying TLII around more than we anticipated (now we always have more stuff to copy). I will take a look. We could also turn this into a lazy initialization - only populate the needed list for module level TLII when it's first queried by a function level TLI.

Hmm, yeah that is surprising, because the TLII should be built once per module per TLI analysis, which is never invalidated. We've gone from populating one set of vec libs to 3, I wouldn't have thought that was particularly expensive, so it would be good to see what is going on here and confirm we are only building this once as expected.

Looking at the compile time data at that link, interestingly the "instructions" metric increased, but not wall time or cycles or task clock - they were all neutral.

Turns out there're a few places where we call copy ctor for TLI unnecessarily. Made some changes in D77952 to use move when possible. In addition, I should have used move for TLI.VecLibDescs in move ctor of TargetLibraryInfoImpl too.

[mehdi_amini]

In D77632#1976231, @wenlei wrote:

And agree with @tejohnson, if the openness is a feature, it should be covered in tests, otherwise it can feel somewhat like a loophole and prone to breakage

The thing is that LLVM does not have much C++ unittests in general, so most of the "features" like this one that LLVM offers as a library are just an artifact of being "designed as a library" and being mindful about the layering.
From this point of view this patch is changing the design of a component that was modular/pluggable into a closed system. I'm perfectly fine with trying to add a unit-test, I just don't know yet where it would fit in the LLVM testing though.

[tejohnson]

In D77632#1976240, @wenlei wrote:

In D77632#1974409, @tejohnson wrote:

In D77632#1974363, @wenlei wrote:

In D77632#1974015, @nikic wrote:

This change causes a ~0.5% compile-time regressions: http://llvm-compile-time-tracker.com/compare.php?from=5b18b6e9a84d985c0a907009fb71de7c1943bc88&to=60c642e74be6af86906d9f3d982728be7bd4329f&stat=instructions This is quite a lot as these things go, so it would be great if you could double check if there's any optimization potential here. In particular I'm wondering why this affects normal builds so much, even though they (presumably?) don't use any veclib at all.

Thanks for the heads-up. This is surprising but there is a change even when veclib is not used - in order to allow each function to use different veclib without duplicating the work of populating vector function list for each function, we now always pre-populate vector function list for three supported vector libraries for each module. However 0.5% compile-time for that work given it's per-module is not expected. I suspect we may be passing/copying TLII around more than we anticipated (now we always have more stuff to copy). I will take a look. We could also turn this into a lazy initialization - only populate the needed list for module level TLII when it's first queried by a function level TLI.

Hmm, yeah that is surprising, because the TLII should be built once per module per TLI analysis, which is never invalidated. We've gone from populating one set of vec libs to 3, I wouldn't have thought that was particularly expensive, so it would be good to see what is going on here and confirm we are only building this once as expected.

Looking at the compile time data at that link, interestingly the "instructions" metric increased, but not wall time or cycles or task clock - they were all neutral.

Turns out there're a few places where we call copy ctor for TLI unnecessarily.

I assume you mean the TargetLibraryInfoImpl (TLII) here, not the TargetLibraryInfo (TLI), right? The latter should be cheap to copy. Are these the changes in BackendUtil.cpp in D77952? I had a question about that on that patch as I think we will be calling the initializer more. Mostly we should only be copying the TargetLibraryInfo during optimization though, and not the TLII impl object.

Made some changes in D77952 to use move when possible. In addition, I should have used move for TLI.VecLibDescs in move ctor of TargetLibraryInfoImpl too.

[nikic]

I gave D77952 a try (on top of this one), but didn't see a significant improvement from that change.

Looking at the callgrind output for compilation of a small file, I see 52M total instructions, 4 calls to TLII initialization, where addition of the vector functions takes up the majority of the time, at 0.7M. Most of the cost is in the sorting. 2 of the initialization calls are default-constructed TLII without target triple, which seems suspect to me (are we not adding TLI early enough, and something pulls it in via analysis dependency?)

So for small files, just registering the vector functions does make up a non-trivial fraction of time, and lazy initialization might make sense. This isn't the whole truth though: While the largest regressions are indeed on small files, there are also quite a few > 1% regressions on very large files.

For a mid-size file with ~6000M instructions retried, the main difference I see is TargetLibraryAnalysis::run() going up from 82M to 126M, with the cost coming from the extra getFnAttribute("veclib") call in the TargetLibraryInfo constructor. Fetching attributes is surprisingly expensive, as it performs an iteration over all attributes internally. As this code is iterating over all attributes anyway in order to handle no-builtin-*, it might make sense to move the checks for "veclib" and "no-builtins" into that loop as well, which should make them essentially free.

[tejohnson]

In D77632#1976308, @mehdi_amini wrote:

In D77632#1976231, @wenlei wrote:

And agree with @tejohnson, if the openness is a feature, it should be covered in tests, otherwise it can feel somewhat like a loophole and prone to breakage

The thing is that LLVM does not have much C++ unittests in general, so most of the "features" like this one that LLVM offers as a library are just an artifact of being "designed as a library" and being mindful about the layering.
From this point of view this patch is changing the design of a component that was modular/pluggable into a closed system.

The interfaces being relied on were in the underlying Impl class, I think if that is expected to be pluggable and stable it really needs unit testing to reflect that usage.

I'm perfectly fine with trying to add a unit-test, I just don't know yet where it would fit in the LLVM testing though.

Presumably the testing should be in llvm/unittests/Analysis/TargetLibraryInfoTest.cpp, which already exists but only tests the LibFuncs (builtins) interfaces.

clang/lib/CodeGen/CGCall.cpp

 //===--- CGCall.cpp - Encapsulate calling convention details --------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
if (llvm::is_contained(NBA->builtinNames(), "*")) {
     FuncAttrs.addAttribute("no-builtins");
     return;
   }
 
   // And last, add the rest of the builtin names.
   llvm::for_each(NBA->builtinNames(), AddNoBuiltinAttr);
 }
 
+static void addVectLibAttributes(llvm::AttrBuilder &FuncAttrs,
+                                 const CodeGenOptions &CodeGenOpts) {
+  StringRef AttributeName = "vect-lib";
+  switch (CodeGenOpts.getVecLib()) {
+  case CodeGenOptions::Accelerate:
+    FuncAttrs.addAttribute(AttributeName, "Accelerate");
+    break;
+  case CodeGenOptions::MASSV:
+    FuncAttrs.addAttribute(AttributeName, "MASSV");
+    break;
+  case CodeGenOptions::SVML:
+    FuncAttrs.addAttribute(AttributeName, "SVML");
+    break;
+  case CodeGenOptions::NoLibrary:
+    break;
+  }
+}
+
 void CodeGenModule::ConstructAttributeList(
     StringRef Name, const CGFunctionInfo &FI, CGCalleeInfo CalleeInfo,
     llvm::AttributeList &AttrList, unsigned &CallingConv, bool AttrOnCallSite) {
   llvm::AttrBuilder FuncAttrs;
   llvm::AttrBuilder RetAttrs;
 
   CallingConv = FI.getEffectiveCallingConvention();
   if (FI.isNoReturn())
void CodeGenModule::ConstructAttributeList(
   // Attach "no-builtins" attributes to:
   // * call sites: both `nobuiltin` and "no-builtins" or "no-builtin-<name>".
   // * definitions: "no-builtins" or "no-builtin-<name>" only.
   // The attributes can come from:
   // * LangOpts: -ffreestanding, -fno-builtin, -fno-builtin-<name>
   // * FunctionDecl attributes: __attribute__((no_builtin(...)))
   addNoBuiltinAttributes(FuncAttrs, getLangOpts(), NBA);
 
+  // Attach "vect-lib" attribute to function based on '-fveclib' setting.
+  addVectLibAttributes(FuncAttrs, getCodeGenOpts());
+
   ConstructDefaultFnAttrList(Name, HasOptnone, AttrOnCallSite, FuncAttrs);
 
   // This must run after constructing the default function attribute list
   // to ensure that the speculative load hardening attribute is removed
   // in the case where the -mspeculative-load-hardening flag was passed.
   if (TargetDecl) {
     if (TargetDecl->hasAttr<NoSpeculativeLoadHardeningAttr>())
       FuncAttrs.removeAttribute(llvm::Attribute::SpeculativeLoadHardening);

clang/test/CodeGen/libcalls-veclib.c

+// RUN: %clang_cc1 -S -emit-llvm -fveclib=SVML -o - %s | FileCheck --check-prefixes=SVML %s
+// RUN: %clang_cc1 -S -emit-llvm -fveclib=Accelerate -o - %s | FileCheck --check-prefixes=ACCELERATE %s
+// RUN: %clang_cc1 -S -emit-llvm -fveclib=MASSV -o - %s | FileCheck --check-prefixes=MASSV %s
+// RUN: %clang_cc1 -S -emit-llvm -fveclib=none -o - %s | FileCheck --check-prefixes=NOLIB %s
+// RUN: %clang_cc1 -S -emit-llvm -o - %s | FileCheck --check-prefixes=NOLIB %s
+
+int main() {
+  return 0;
+}
+
+// SVML: "vect-lib"="SVML"
+// ACCELERATE: "vect-lib"="Accelerate"
+// MASSV: "vect-lib"="MASSV"
+// NOLIB-NOT: "vect-lib"
 No newline at end of file

llvm/include/llvm/Analysis/TargetLibraryInfo.h

 //===-- TargetLibraryInfo.h - Library information ---------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
#include "llvm/Analysis/TargetLibraryInfo.def"
     NotLibFunc
   };
 
 /// Implementation of the target library information.
 ///
 /// This class constructs tables that hold the target library information and
 /// make it available. However, it is somewhat expensive to compute and only
 /// depends on the triple. So users typically interact with the \c
 /// TargetLibraryInfo wrapper below.
 class TargetLibraryInfoImpl {
   friend class TargetLibraryInfo;
 
+public:
+  /// List of known vector-functions libraries.
+  ///
+  /// The vector-functions library defines, which functions are vectorizable
+  /// and with which factor. The library can be specified by either frontend,
+  /// or a commandline option, and then used by
+  /// addVectorizableFunctionsFromVecLib for filling up the tables of
+  /// vectorizable functions.
+  enum VectorLibrary {
+    NoLibrary,  // Don't use any vector library.
+    Accelerate, // Use Accelerate framework.
+    MASSV,      // IBM MASS vector library.
+    SVML        // Intel short vector math library.
+  };
+
+private:
   unsigned char AvailableArray[(NumLibFuncs+3)/4];
   llvm::DenseMap<unsigned, std::string> CustomNames;
   static StringLiteral const StandardNames[NumLibFuncs];
   bool ShouldExtI32Param, ShouldExtI32Return, ShouldSignExtI32Param;
 
   enum AvailabilityState {
     StandardName = 3, // (memset to all ones)
     CustomName = 1,
     Unavailable = 0  // (memset to all zeros)
   };
   void setState(LibFunc F, AvailabilityState State) {
     AvailableArray[F/4] &= ~(3 << 2*(F&3));
     AvailableArray[F/4] |= State << 2*(F&3);
   }
   AvailabilityState getState(LibFunc F) const {
     return static_cast<AvailabilityState>((AvailableArray[F/4] >> 2*(F&3)) & 3);
   }
 
   /// Vectorization descriptors - sorted by ScalarFnName.
   std::vector<VecDesc> VectorDescs;
   /// Scalarization descriptors - same content as VectorDescs but sorted based
   /// on VectorFnName rather than ScalarFnName.
   std::vector<VecDesc> ScalarDescs;
+  /// Vector library available for vectorization.
+  VectorLibrary VectLibrary = NoLibrary;
 
   /// Return true if the function type FTy is valid for the library function
   /// F, regardless of whether the function is available.
   bool isValidProtoForLibFunc(const FunctionType &FTy, LibFunc F,
                               const DataLayout *DL) const;
 
 public:
-  /// List of known vector-functions libraries.
-  ///
-  /// The vector-functions library defines, which functions are vectorizable
-  /// and with which factor. The library can be specified by either frontend,
-  /// or a commandline option, and then used by
-  /// addVectorizableFunctionsFromVecLib for filling up the tables of
-  /// vectorizable functions.
-  enum VectorLibrary {
-    NoLibrary,  // Don't use any vector library.
-    Accelerate, // Use Accelerate framework.
-    MASSV,      // IBM MASS vector library.
-    SVML        // Intel short vector math library.
-  };
-
   TargetLibraryInfoImpl();
   explicit TargetLibraryInfoImpl(const Triple &T);
 
   // Provide value semantics.
   TargetLibraryInfoImpl(const TargetLibraryInfoImpl &TLI);
   TargetLibraryInfoImpl(TargetLibraryInfoImpl &&TLI);
   TargetLibraryInfoImpl &operator=(const TargetLibraryInfoImpl &TLI);
   TargetLibraryInfoImpl &operator=(TargetLibraryInfoImpl &&TLI);
public:
 
   /// Add a set of scalar -> vector mappings, queryable via
   /// getVectorizedFunction and getScalarizedFunction.
   void addVectorizableFunctions(ArrayRef<VecDesc> Fns);
 
   /// Calls addVectorizableFunctions with a known preset of functions for the
   /// given vector library.
   void addVectorizableFunctionsFromVecLib(enum VectorLibrary VecLib);
+  void addVectorizableFunctionsFromVecLib(const StringRef &VecLibName);
 
   /// Return true if the function F has a vector equivalent with vectorization
   /// factor VF.
   bool isFunctionVectorizable(StringRef F, unsigned VF) const {
     return !getVectorizedFunction(F, VF).empty();
   }
 
   /// Return true if the function F has a vector equivalent with any
explicit TargetLibraryInfo(const TargetLibraryInfoImpl &Impl,
       : Impl(&Impl), OverrideAsUnavailable(NumLibFuncs) {
     if (!F)
       return;
     if ((*F)->hasFnAttribute("no-builtins"))
       disableAllFunctions();
     else {
       // Disable individual libc/libm calls in TargetLibraryInfo.
       LibFunc LF;
       AttributeSet FnAttrs = (*F)->getAttributes().getFnAttributes();
       for (const Attribute &Attr : FnAttrs) {
         if (!Attr.isStringAttribute())
           continue;
         auto AttrStr = Attr.getKindAsString();
         if (!AttrStr.consume_front("no-builtin-"))
           continue;
         if (getLibFunc(AttrStr, LF))
           setUnavailable(LF);
public:
   }
   TargetLibraryInfo &operator=(TargetLibraryInfo &&TLI) {
     Impl = TLI.Impl;
     OverrideAsUnavailable = TLI.OverrideAsUnavailable;
     return *this;
   }
 
   /// Determine whether a callee with the given TLI can be inlined into
-  /// caller with this TLI, based on 'nobuiltin' attributes. When requested,
-  /// allow inlining into a caller with a superset of the callee's nobuiltin
-  /// attributes, which is conservatively correct.
+  /// caller with this TLI, based on 'nobuiltin', `vect-lib` attributes.
+  /// When requested, allow inlining into a caller with a superset of the
+  /// callee's attributes, which is conservatively correct.
   bool areInlineCompatible(const TargetLibraryInfo &CalleeTLI,
                            bool AllowCallerSuperset) const {
     if (!AllowCallerSuperset)
-      return OverrideAsUnavailable == CalleeTLI.OverrideAsUnavailable;
+      return Impl->VectLibrary == CalleeTLI.Impl->VectLibrary &&
+             OverrideAsUnavailable == CalleeTLI.OverrideAsUnavailable;
     BitVector B = OverrideAsUnavailable;
     B |= CalleeTLI.OverrideAsUnavailable;
-    // We can inline if the union of the caller and callee's nobuiltin
-    // attributes is no stricter than the caller's nobuiltin attributes.
-    return B == OverrideAsUnavailable;
+    // We can inline if the union of the caller and callee's attributes
+    // is no stricter than the caller's attributes.
+    return B == OverrideAsUnavailable &&
+           CalleeTLI.Impl->VectLibrary == TargetLibraryInfoImpl::NoLibrary;
   }
 
   /// Searches for a particular function name.
   ///
   /// If it is one of the known library functions, return true and set F to the
   /// corresponding value.
   bool getLibFunc(StringRef funcName, LibFunc &F) const {
     return Impl->getLibFunc(funcName, F);

llvm/lib/Analysis/TargetLibraryInfo.cpp

 //===-- TargetLibraryInfo.cpp - Runtime library information ----------------==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 }
 
 TargetLibraryInfoImpl::TargetLibraryInfoImpl(const TargetLibraryInfoImpl &TLI)
     : CustomNames(TLI.CustomNames), ShouldExtI32Param(TLI.ShouldExtI32Param),
       ShouldExtI32Return(TLI.ShouldExtI32Return),
       ShouldSignExtI32Param(TLI.ShouldSignExtI32Param) {
   memcpy(AvailableArray, TLI.AvailableArray, sizeof(AvailableArray));
   VectorDescs = TLI.VectorDescs;
   ScalarDescs = TLI.ScalarDescs;
 }
 
 TargetLibraryInfoImpl::TargetLibraryInfoImpl(TargetLibraryInfoImpl &&TLI)
     : CustomNames(std::move(TLI.CustomNames)),
       ShouldExtI32Param(TLI.ShouldExtI32Param),
       ShouldExtI32Return(TLI.ShouldExtI32Return),
       ShouldSignExtI32Param(TLI.ShouldSignExtI32Param) {
   std::move(std::begin(TLI.AvailableArray), std::end(TLI.AvailableArray),
             AvailableArray);
   VectorDescs = TLI.VectorDescs;
   ScalarDescs = TLI.ScalarDescs;
 }
 
 TargetLibraryInfoImpl &TargetLibraryInfoImpl::operator=(const TargetLibraryInfoImpl &TLI) {
   CustomNames = TLI.CustomNames;
   ShouldExtI32Param = TLI.ShouldExtI32Param;
   ShouldExtI32Return = TLI.ShouldExtI32Return;
   ShouldSignExtI32Param = TLI.ShouldSignExtI32Param;
   memcpy(AvailableArray, TLI.AvailableArray, sizeof(AvailableArray));
void TargetLibraryInfoImpl::addVectorizableFunctions(ArrayRef<VecDesc> Fns) {
   VectorDescs.insert(VectorDescs.end(), Fns.begin(), Fns.end());
   llvm::sort(VectorDescs, compareByScalarFnName);
 
   ScalarDescs.insert(ScalarDescs.end(), Fns.begin(), Fns.end());
   llvm::sort(ScalarDescs, compareByVectorFnName);
 }
 
 void TargetLibraryInfoImpl::addVectorizableFunctionsFromVecLib(
+    const StringRef &VecLibName) {
+  VectorLibrary VecLib = NoLibrary;
+  if (VecLibName == "Accelerate")
+    VecLib = Accelerate;
+  else if (VecLibName == "MASSV")
+    VecLib = MASSV;
+  else if (VecLibName == "SVML")
+    VecLib = SVML;
+  else
+    return;
+  addVectorizableFunctionsFromVecLib(VecLib);
+}
+
+void TargetLibraryInfoImpl::addVectorizableFunctionsFromVecLib(
     enum VectorLibrary VecLib) {
+  if (VectLibrary != NoLibrary) {
+    assert(VectLibrary == VecLib && 
+           "Conflicting VectorLibrary detected");
+    return;
+  }
+
+  VectLibrary = VecLib;
   switch (VecLib) {
   case Accelerate: {
     const VecDesc VecFuncs[] = {
     #define TLI_DEFINE_ACCELERATE_VECFUNCS
     #include "llvm/Analysis/VecFuncs.def"
     };
     addVectorizableFunctions(VecFuncs);
     break;
case SVML: {
     const VecDesc VecFuncs[] = {
     #define TLI_DEFINE_SVML_VECFUNCS
     #include "llvm/Analysis/VecFuncs.def"
     };
     addVectorizableFunctions(VecFuncs);
     break;
   }
   case NoLibrary:
     break;
   }
 }
 
 bool TargetLibraryInfoImpl::isFunctionVectorizable(StringRef funcName) const {
   funcName = sanitizeFunctionName(funcName);
   if (funcName.empty())
     return false;
 
StringRef TargetLibraryInfoImpl::getScalarizedFunction(StringRef F,
   return I->ScalarFnName;
 }
 
 TargetLibraryInfo TargetLibraryAnalysis::run(const Function &F,
                                              FunctionAnalysisManager &) {
   if (!BaselineInfoImpl)
     BaselineInfoImpl =
         TargetLibraryInfoImpl(Triple(F.getParent()->getTargetTriple()));
+
+  StringRef VectLibName = F.getFnAttribute("vect-lib").getValueAsString();
+  if (!VectLibName.empty())
+    BaselineInfoImpl->addVectorizableFunctionsFromVecLib(VectLibName);
+
   return TargetLibraryInfo(*BaselineInfoImpl, &F);
 }
 
 unsigned TargetLibraryInfoImpl::getWCharSize(const Module &M) const {
   if (auto *ShortWChar = cast_or_null<ConstantAsMetadata>(
       M.getModuleFlag("wchar_size")))
     return cast<ConstantInt>(ShortWChar->getValue())->getZExtValue();
   return 0;