Intel SVML calling conventions

[dvnagorny]

Summary

This patch fixes the problem with improper calls to SVML library as it has non-standard calling conventions.
So accordingly it has SVML calling conventions definitions and code to set CC to the vectorized calls.
As SVML provides several implementations for the math functions we also took into consideration fast attribute and select more fast implementation in such case.
This work is based on original Matt Masten's work.

[lebedev.ri]

include/llvm/IR/CallingConv.h

225

unneeded comment

lib/Analysis/TargetLibraryInfo.cpp

55

This memory allocation looks unfortunate :/

utils/TableGen/SVMLEmitter.cpp

63

This surely can be StringRef

[dvnagorny]

lib/Analysis/TargetLibraryInfo.cpp

55

Could you explain your comment more detailed please. svmlMangle() returns not a reference but the std::string.

[lebedev.ri]

lib/Analysis/TargetLibraryInfo.cpp

55

Not much to explain, that already summarizes it quite well.

It will return a std::string, not a reference,
so unless small string optimization within std::string
happens, every call to svmlMangle() will cause an allocation.

And this also caused getVectorizedFunction() to return
the std:;string, and so on.

[dvnagorny]

lib/Analysis/TargetLibraryInfo.cpp

55

I can't agree with you here. Beside of RVO we have move semantics for std::string now. So there shouldn't be no any extra memory allocations. Does LLVM coding standard imply anything special on move semantics?

[lebedev.ri]

lib/Analysis/TargetLibraryInfo.cpp

1495

Sure, here, move will happen.

1497

But i don't see how it can happen here.

[dvnagorny]

lib/Analysis/TargetLibraryInfo.cpp

1497

So as the name is dynamically constructed now it should be stored somewhere. I think it's good enough if it occures not more then once.

[hfinkel]

include/llvm/Analysis/TargetLibraryInfo.h

159

I think just "Ignored" is fine.

include/llvm/IR/SVML.td

53

What does this mean? Why would a _ha variant of floor, for example, be needed for vectorization? Or, to put it another way, how would a _ha variant of floor differ from the _ep version?

[dvnagorny]

include/llvm/Analysis/TargetLibraryInfo.h

159

Will fix, thank you.

include/llvm/IR/SVML.td

53

Really I don't expect any difference in behaviour, However I think that link-time weak aliases for these symbols are more preferable here. It will not require any additional logic in function name mangling code.

[hfinkel]

include/llvm/IR/SVML.td

53

Do these aliases exist currently? I'm concerned about the comment saying that we're disabling vectorization of floor, etc., and I'm prefer that we not disable vectorization unnecessarily.

[craig.topper]

lib/Analysis/TargetLibraryInfo.cpp

1485

This looks like it passes 80 columns.

1501

This should probably be std::string() now.

utils/TableGen/SVMLEmitter.cpp

51

Mark these 'const'?

56

This looks to be a dead variable in release builds.

62

I think this should be Records.getAllDerivedDefinitions("SvmlVariant")

[dvnagorny]

include/llvm/IR/SVML.td

53

Really this code will work when vector-library SVML option will be explicitly passed. So it doesn't disable any default vectorizations. On the other hand in the current LLVM state these function's vectorization isn't enabled yet. So I keep their vectorization the same. From SVML library side aliases aren't available yet however this library not the only potential source of aliases.
It's quite simple to provide them from the application code.

[hfinkel]

include/llvm/IR/SVML.td

53

Okay, but providing them in application code doesn't help autovectorization, does it? Passing -fveclib=SVML should enable autovectorization of calls to floor, etc. You're saying that it doesn't now, and so I can certainly see that it might be best to change that in a separate patch, but saying that we'll wait for aliases to appear in the library doesn't sound like a good plan (and doesn't help users of existing versions of the library). The vectorizer can call some version which is already there.

In short, would it make sense for the comment to read?

// TODO: SVML does not currently provide _ha varients of these fucnctions. We should call the _ep variants of these functions in all cases instead.

[hsaito]

lib/Analysis/TargetLibraryInfo.cpp

55

I have a problem for the use of "_ha" interface in the non-fast case. Unless the compiler is in a reasonably relaxed mode, I'd like vector computation and scalar computation to be bitwise-identical. "_ha" interface of SVML doesn't produce the bitwise identical result as scalar call. I'm sure there is a room for using "_ha" interface, but we need to carefully design how to enable it. By simply using SVML, I don't think the programmer gave the compiler a license to deviate from bitwise identical results. I may be too much of a paranoid about it, but there is a fair number of people who ask for bitwise identity.

[dvnagorny]

lib/Analysis/TargetLibraryInfo.cpp

55

Quite interesting problem here. AFAIK "_ha" interface provides "high accuracy" versions of functions. (1 ulp), "default" SVML interface provides less precise functions (probably 4ulp).
So it means that scalar implementation is lesser precise than vectorized one.
On the other hand I'm not sure if it is really possible to have differ libm implementations providing fully bitwise-idenctical results for the whole float or double domain within 4ulps precision.
As not exact units should differ by definition.

include/llvm/Analysis/TargetLibraryInfo.h

struct VecDesc {
 
   enum LibFunc {
 #define TLI_DEFINE_ENUM
 #include "llvm/Analysis/TargetLibraryInfo.def"
 
     NumLibFuncs
   };
 
+enum SVMLAccuracy {
+  SVML_DEFAULT,
+  SVML_HA,
+  SVML_EP
+};
+
 /// Implementation of the target library information.
 ///
 /// This class constructs tables that hold the target library information and
 /// make it available. However, it is somewhat expensive to compute and only
 /// depends on the triple. So users typically interact with the \c
 /// TargetLibraryInfo wrapper below.
 class TargetLibraryInfoImpl {
   friend class TargetLibraryInfo;
public:
 
   /// Calls addVectorizableFunctions with a known preset of functions for the
   /// given vector library.
   void addVectorizableFunctionsFromVecLib(enum VectorLibrary VecLib);
 
   /// Return true if the function F has a vector equivalent with vectorization
   /// factor VF.
   bool isFunctionVectorizable(StringRef F, unsigned VF) const {
-    return !getVectorizedFunction(F, VF).empty();
+     bool IgnoreMeThere;
+     return !getVectorizedFunction(F, VF, IgnoreMeThere, false).empty();
   }
 
   /// Return true if the function F has a vector equivalent with any
   /// vectorization factor.
   bool isFunctionVectorizable(StringRef F) const;
 
   /// Return the name of the equivalent of F, vectorized with factor VF. If no
   /// such mapping exists, return the empty string.
-  StringRef getVectorizedFunction(StringRef F, unsigned VF) const;
+  std::string getVectorizedFunction(StringRef F, unsigned VF, bool &FromSVML,
+                                    bool IsFast) const;
 
   /// Return true if the function F has a scalar equivalent, and set VF to be
   /// the vectorization factor.
   bool isFunctionScalarizable(StringRef F, unsigned &VF) const {
     return !getScalarizedFunction(F, VF).empty();
   }
 
   /// Return the name of the equivalent of F, scalarized. If no such mapping
bool has(LibFunc F) const {
     return Impl->getState(F) != TargetLibraryInfoImpl::Unavailable;
   }
   bool isFunctionVectorizable(StringRef F, unsigned VF) const {
     return Impl->isFunctionVectorizable(F, VF);
   }
   bool isFunctionVectorizable(StringRef F) const {
     return Impl->isFunctionVectorizable(F);
   }
-  StringRef getVectorizedFunction(StringRef F, unsigned VF) const {
-    return Impl->getVectorizedFunction(F, VF);
+  std::string getVectorizedFunction(StringRef F, unsigned VF, bool &FromSVML,
+                                    bool IsFast) const {
+    return Impl->getVectorizedFunction(F, VF, FromSVML, IsFast);
   }
 
   /// Tests if the function is both available and a candidate for optimized code
   /// generation.
   bool hasOptimizedCodeGen(LibFunc F) const {
     if (Impl->getState(F) == TargetLibraryInfoImpl::Unavailable)
       return false;
     switch (F) {

include/llvm/IR/CMakeLists.txt

 set(LLVM_TARGET_DEFINITIONS Attributes.td)
 tablegen(LLVM Attributes.inc -gen-attrs)
 
 set(LLVM_TARGET_DEFINITIONS Intrinsics.td)
 tablegen(LLVM Intrinsics.inc -gen-intrinsic)
 add_public_tablegen_target(intrinsics_gen)
+
+set(LLVM_TARGET_DEFINITIONS SVML.td)
+tablegen(LLVM SVML.inc -gen-svml)
+add_public_tablegen_target(svml_gen)

include/llvm/IR/CallingConv.h

enum {
     /// use.
     AMDGPU_LS = 95,
 
     /// Calling convention used for AMDPAL shader stage before geometry shader
     /// if geometry is in use. So either the domain (= tessellation evaluation)
     /// shader if tessellation is in use, or otherwise the vertex shader.
     AMDGPU_ES = 96,
 
+    /// Intel_SVML - Calling conventions for Intel Short Math Vector Library
+    Intel_SVML = 97,
+    //
     /// The highest possible calling convention ID. Must be some 2^k - 1.
     MaxID = 1023
   };
 
 } // end namespace CallingConv
 
 } // end namespace llvm
 
 #endif // LLVM_IR_CALLINGCONV_H

include/llvm/IR/SVML.td

+//===-- Intel_SVML.td - Defines SVML call variants ---------*- tablegen -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is used by TableGen to define the different typs of SVML function
+// variants used with -fveclib=SVML.
+//
+//===----------------------------------------------------------------------===//
+
+class SvmlVariant;
+
+def sin        : SvmlVariant;
+def cos        : SvmlVariant;
+def pow        : SvmlVariant;
+def exp        : SvmlVariant;
+def log        : SvmlVariant;
+def acos       : SvmlVariant;
+def acosh      : SvmlVariant;
+def asin       : SvmlVariant;
+def asinh      : SvmlVariant;
+def atan2      : SvmlVariant;
+def atan       : SvmlVariant;
+def atanh      : SvmlVariant;
+def cbrt       : SvmlVariant;
+def cdfnorm    : SvmlVariant;
+def cdfnorminv : SvmlVariant;
+def cosd       : SvmlVariant;
+def cosh       : SvmlVariant;
+def erf        : SvmlVariant;
+def erfc       : SvmlVariant;
+def erfcinv    : SvmlVariant;
+def erfinv     : SvmlVariant;
+def exp10      : SvmlVariant;
+def exp2       : SvmlVariant;
+def expm1      : SvmlVariant;
+def hypot      : SvmlVariant;
+def invsqrt    : SvmlVariant;
+def log10      : SvmlVariant;
+def log1p      : SvmlVariant;
+def log2       : SvmlVariant;
+def sind       : SvmlVariant;
+def sinh       : SvmlVariant;
+def sqrt       : SvmlVariant;
+def tan        : SvmlVariant;
+def tanh       : SvmlVariant;
+
+// While SVML doesn't provide _ha versions of
+// the following symbols let's disable their vectorization.
+
+// def nearbyint  : SvmlVariant;
+// def logb       : SvmlVariant;
+// def floor      : SvmlVariant;
+// def fmod       : SvmlVariant;
+// def ceil       : SvmlVariant;
+// def trunc      : SvmlVariant;
+// def rint       : SvmlVariant;
+// def round      : SvmlVariant;

lib/Analysis/CMakeLists.txt

add_llvm_library(LLVMAnalysis
   ValueTracking.cpp
   VectorUtils.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${LLVM_MAIN_INCLUDE_DIR}/llvm/Analysis
 
   DEPENDS
   intrinsics_gen
+  svml_gen
   )

lib/Analysis/TargetLibraryInfo.cpp

if (T.isMacOSX() && T.isMacOSXVersionLT(10, 9))
     return false;
 
   if (T.isiOS() && T.isOSVersionLT(7, 0))
     return false;
 
   return true;
 }
 
+std::string svmlMangle(StringRef FnName, const bool IsFast) {
+  std::string FullName = FnName;
+  return IsFast ? FullName : FullName + "_ha";
+}
+
 /// Initialize the set of available library functions based on the specified
 /// target triple. This should be carefully written so that a missing target
 /// triple gets a sane set of defaults.
 static void initialize(TargetLibraryInfoImpl &TLI, const Triple &T,
                        ArrayRef<StringRef> StandardNames) {
   // Verify that the StandardNames array is in alphabetical order.
   assert(std::is_sorted(StandardNames.begin(), StandardNames.end(),
                         [](StringRef LHS, StringRef RHS) {
const VecDesc VecFuncs[] = {
         {"acoshf", "vacoshf", 4},
         {"atanhf", "vatanhf", 4},
     };
     addVectorizableFunctions(VecFuncs);
     break;
   }
   case SVML: {
     const VecDesc VecFuncs[] = {
-        {"sin", "__svml_sin2", 2},
-        {"sin", "__svml_sin4", 4},
-        {"sin", "__svml_sin8", 8},
-
-        {"sinf", "__svml_sinf4", 4},
-        {"sinf", "__svml_sinf8", 8},
-        {"sinf", "__svml_sinf16", 16},
-
-        {"cos", "__svml_cos2", 2},
-        {"cos", "__svml_cos4", 4},
-        {"cos", "__svml_cos8", 8},
-
-        {"cosf", "__svml_cosf4", 4},
-        {"cosf", "__svml_cosf8", 8},
-        {"cosf", "__svml_cosf16", 16},
-
-        {"pow", "__svml_pow2", 2},
-        {"pow", "__svml_pow4", 4},
-        {"pow", "__svml_pow8", 8},
-
-        {"powf", "__svml_powf4", 4},
-        {"powf", "__svml_powf8", 8},
-        {"powf", "__svml_powf16", 16},
-
-        { "__pow_finite", "__svml_pow2", 2 },
-        { "__pow_finite", "__svml_pow4", 4 },
-        { "__pow_finite", "__svml_pow8", 8 },
-
-        { "__powf_finite", "__svml_powf4", 4 },
-        { "__powf_finite", "__svml_powf8", 8 },
-        { "__powf_finite", "__svml_powf16", 16 },
-
-        {"llvm.pow.f64", "__svml_pow2", 2},
-        {"llvm.pow.f64", "__svml_pow4", 4},
-        {"llvm.pow.f64", "__svml_pow8", 8},
-
-        {"llvm.pow.f32", "__svml_powf4", 4},
-        {"llvm.pow.f32", "__svml_powf8", 8},
-        {"llvm.pow.f32", "__svml_powf16", 16},
-
-        {"exp", "__svml_exp2", 2},
-        {"exp", "__svml_exp4", 4},
-        {"exp", "__svml_exp8", 8},
-
-        {"expf", "__svml_expf4", 4},
-        {"expf", "__svml_expf8", 8},
-        {"expf", "__svml_expf16", 16},
-
-        { "__exp_finite", "__svml_exp2", 2 },
-        { "__exp_finite", "__svml_exp4", 4 },
-        { "__exp_finite", "__svml_exp8", 8 },
-
-        { "__expf_finite", "__svml_expf4", 4 },
-        { "__expf_finite", "__svml_expf8", 8 },
-        { "__expf_finite", "__svml_expf16", 16 },
-
-        {"llvm.exp.f64", "__svml_exp2", 2},
-        {"llvm.exp.f64", "__svml_exp4", 4},
-        {"llvm.exp.f64", "__svml_exp8", 8},
-
-        {"llvm.exp.f32", "__svml_expf4", 4},
-        {"llvm.exp.f32", "__svml_expf8", 8},
-        {"llvm.exp.f32", "__svml_expf16", 16},
-
-        {"log", "__svml_log2", 2},
-        {"log", "__svml_log4", 4},
-        {"log", "__svml_log8", 8},
-
-        {"logf", "__svml_logf4", 4},
-        {"logf", "__svml_logf8", 8},
-        {"logf", "__svml_logf16", 16},
-
-        { "__log_finite", "__svml_log2", 2 },
-        { "__log_finite", "__svml_log4", 4 },
-        { "__log_finite", "__svml_log8", 8 },
-
-        { "__logf_finite", "__svml_logf4", 4 },
-        { "__logf_finite", "__svml_logf8", 8 },
-        { "__logf_finite", "__svml_logf16", 16 },
-
-        {"llvm.log.f64", "__svml_log2", 2},
-        {"llvm.log.f64", "__svml_log4", 4},
-        {"llvm.log.f64", "__svml_log8", 8},
-
-        {"llvm.log.f32", "__svml_logf4", 4},
-        {"llvm.log.f32", "__svml_logf8", 8},
-        {"llvm.log.f32", "__svml_logf16", 16},
+#define GET_SVML_VARIANTS
+#include "llvm/IR/SVML.inc"
+#undef GET_SVML_VARIANTS
     };
     addVectorizableFunctions(VecFuncs);
     break;
   }
   case NoLibrary:
     break;
   }
 }
 
 bool TargetLibraryInfoImpl::isFunctionVectorizable(StringRef funcName) const {
   funcName = sanitizeFunctionName(funcName);
   if (funcName.empty())
     return false;
 
   std::vector<VecDesc>::const_iterator I = std::lower_bound(
       VectorDescs.begin(), VectorDescs.end(), funcName,
       compareWithScalarFnName);
   return I != VectorDescs.end() && StringRef(I->ScalarFnName) == funcName;
 }
 
-StringRef TargetLibraryInfoImpl::getVectorizedFunction(StringRef F,
-                                                       unsigned VF) const {
+std::string TargetLibraryInfoImpl::getVectorizedFunction(StringRef F,
+                                                         unsigned VF, bool &FromSVML, bool IsFast) const {
+  FromSVML = ClVectorLibrary == SVML;
   F = sanitizeFunctionName(F);
   if (F.empty())
     return F;
   std::vector<VecDesc>::const_iterator I = std::lower_bound(
       VectorDescs.begin(), VectorDescs.end(), F, compareWithScalarFnName);
   while (I != VectorDescs.end() && StringRef(I->ScalarFnName) == F) {
-    if (I->VectorizationFactor == VF)
+    if (I->VectorizationFactor == VF) {
+      if (FromSVML) {
+        return svmlMangle(I->VectorFnName, IsFast);
+      }
       return I->VectorFnName;
+    }
     ++I;
   }
   return StringRef();
 }
 
 StringRef TargetLibraryInfoImpl::getScalarizedFunction(StringRef F,
                                                        unsigned &VF) const {
   F = sanitizeFunctionName(F);
   if (F.empty())
     return F;
 

lib/AsmParser/LLLexer.cpp

#define KEYWORD(STR)                                                           \
   KEYWORD(msp430_intrcc);
   KEYWORD(avr_intrcc);
   KEYWORD(avr_signalcc);
   KEYWORD(ptx_kernel);
   KEYWORD(ptx_device);
   KEYWORD(spir_kernel);
   KEYWORD(spir_func);
   KEYWORD(intel_ocl_bicc);
+  KEYWORD(intel_svmlcc);
   KEYWORD(x86_64_sysvcc);
   KEYWORD(win64cc);
   KEYWORD(x86_regcallcc);
   KEYWORD(webkit_jscc);
   KEYWORD(swiftcc);
   KEYWORD(anyregcc);
   KEYWORD(preserve_mostcc);
   KEYWORD(preserve_allcc);

lib/AsmParser/LLParser.cpp

void LLParser::ParseOptionalDLLStorageClass(unsigned &Res) {
   Lex.Lex();
 }
 
 /// ParseOptionalCallingConv
 ///   ::= /*empty*/
 ///   ::= 'ccc'
 ///   ::= 'fastcc'
 ///   ::= 'intel_ocl_bicc'
+///   ::= 'intel_svmlcc'
 ///   ::= 'coldcc'
 ///   ::= 'x86_stdcallcc'
 ///   ::= 'x86_fastcallcc'
 ///   ::= 'x86_thiscallcc'
 ///   ::= 'x86_vectorcallcc'
 ///   ::= 'arm_apcscc'
 ///   ::= 'arm_aapcscc'
 ///   ::= 'arm_aapcs_vfpcc'
bool LLParser::ParseOptionalCallingConv(unsigned &CC) {
   case lltok::kw_msp430_intrcc:  CC = CallingConv::MSP430_INTR; break;
   case lltok::kw_avr_intrcc:     CC = CallingConv::AVR_INTR; break;
   case lltok::kw_avr_signalcc:   CC = CallingConv::AVR_SIGNAL; break;
   case lltok::kw_ptx_kernel:     CC = CallingConv::PTX_Kernel; break;
   case lltok::kw_ptx_device:     CC = CallingConv::PTX_Device; break;
   case lltok::kw_spir_kernel:    CC = CallingConv::SPIR_KERNEL; break;
   case lltok::kw_spir_func:      CC = CallingConv::SPIR_FUNC; break;
   case lltok::kw_intel_ocl_bicc: CC = CallingConv::Intel_OCL_BI; break;
+  case lltok::kw_intel_svmlcc:   CC = CallingConv::Intel_SVML; break;
   case lltok::kw_x86_64_sysvcc:  CC = CallingConv::X86_64_SysV; break;
   case lltok::kw_win64cc:        CC = CallingConv::Win64; break;
   case lltok::kw_webkit_jscc:    CC = CallingConv::WebKit_JS; break;
   case lltok::kw_anyregcc:       CC = CallingConv::AnyReg; break;
   case lltok::kw_preserve_mostcc:CC = CallingConv::PreserveMost; break;
   case lltok::kw_preserve_allcc: CC = CallingConv::PreserveAll; break;
   case lltok::kw_ghccc:          CC = CallingConv::GHC; break;
   case lltok::kw_swiftcc:        CC = CallingConv::Swift; break;

lib/AsmParser/LLToken.h

enum Kind {
   kw_prologue,
   kw_c,
 
   kw_cc,
   kw_ccc,
   kw_fastcc,
   kw_coldcc,
   kw_intel_ocl_bicc,
+  kw_intel_svmlcc,
   kw_x86_stdcallcc,
   kw_x86_fastcallcc,
   kw_x86_thiscallcc,
   kw_x86_vectorcallcc,
   kw_x86_regcallcc,
   kw_arm_apcscc,
   kw_arm_aapcscc,
   kw_arm_aapcs_vfpcc,

lib/IR/AsmWriter.cpp

static void PrintCallingConv(unsigned cc, raw_ostream &Out) {
   case CallingConv::CXX_FAST_TLS:  Out << "cxx_fast_tlscc"; break;
   case CallingConv::GHC:           Out << "ghccc"; break;
   case CallingConv::X86_StdCall:   Out << "x86_stdcallcc"; break;
   case CallingConv::X86_FastCall:  Out << "x86_fastcallcc"; break;
   case CallingConv::X86_ThisCall:  Out << "x86_thiscallcc"; break;
   case CallingConv::X86_RegCall:   Out << "x86_regcallcc"; break;
   case CallingConv::X86_VectorCall:Out << "x86_vectorcallcc"; break;
   case CallingConv::Intel_OCL_BI:  Out << "intel_ocl_bicc"; break;
+  case CallingConv::Intel_SVML:    Out << "intel_svmlcc"; break;
   case CallingConv::ARM_APCS:      Out << "arm_apcscc"; break;
   case CallingConv::ARM_AAPCS:     Out << "arm_aapcscc"; break;
   case CallingConv::ARM_AAPCS_VFP: Out << "arm_aapcs_vfpcc"; break;
   case CallingConv::MSP430_INTR:   Out << "msp430_intrcc"; break;
   case CallingConv::AVR_INTR:      Out << "avr_intrcc "; break;
   case CallingConv::AVR_SIGNAL:    Out << "avr_signalcc "; break;
   case CallingConv::PTX_Kernel:    Out << "ptx_kernel"; break;
   case CallingConv::PTX_Device:    Out << "ptx_device"; break;

lib/IR/Verifier.cpp

void Verifier::visitFunction(const Function &F) {
   case CallingConv::AMDGPU_PS:
   case CallingConv::AMDGPU_CS:
     Assert(!F.hasStructRetAttr(),
            "Calling convention does not allow sret", &F);
     LLVM_FALLTHROUGH;
   case CallingConv::Fast:
   case CallingConv::Cold:
   case CallingConv::Intel_OCL_BI:
+  case CallingConv::Intel_SVML:
   case CallingConv::PTX_Kernel:
   case CallingConv::PTX_Device:
     Assert(!F.isVarArg(), "Calling convention does not support varargs or "
                           "perfect forwarding!",
            &F);
     break;
   }
 

lib/Target/X86/X86CallingConv.td

def RetCC_X86_64 : CallingConv<[
           
   // Mingw64 and native Win64 use Win64 CC
   CCIfSubtarget<"isTargetWin64()", CCDelegateTo<RetCC_X86_Win64_C>>,
 
   // Otherwise, drop to normal X86-64 CC
   CCDelegateTo<RetCC_X86_64_C>
 ]>;
 
+// Intel_SVML return-value convention.
+def RetCC_Intel_SVML : CallingConv<[
+  // Vector types are returned in XMM0,XMM1
+  CCIfType<[v4f32, v2f64],
+            CCAssignToReg<[XMM0,XMM1]>>,
+
+  // 256-bit FP vectors
+  CCIfType<[v8f32, v4f64],
+            CCAssignToReg<[YMM0,YMM1]>>,
+
+  // 512-bit FP vectors
+  CCIfType<[v16f32, v8f64],
+            CCAssignToReg<[ZMM0,ZMM1]>>
+]>;
+
 // This is the return-value convention used for the entire X86 backend.
 def RetCC_X86 : CallingConv<[
 
   // Check if this is the Intel OpenCL built-ins calling convention
   CCIfCC<"CallingConv::Intel_OCL_BI", CCDelegateTo<RetCC_Intel_OCL_BI>>,
 
+  CCIfCC<"CallingConv::Intel_SVML", CCDelegateTo<RetCC_Intel_SVML>>,
+
   CCIfSubtarget<"is64Bit()", CCDelegateTo<RetCC_X86_64>>,
   CCDelegateTo<RetCC_X86_32>
 ]>;
 
 //===----------------------------------------------------------------------===//
 // X86-64 Argument Calling Conventions
 //===----------------------------------------------------------------------===//
 
def CC_Intel_OCL_BI : CallingConv<[
   // Pass masks in mask registers
   CCIfType<[v16i1, v8i1], CCAssignToReg<[K1]>>,
 
   CCIfSubtarget<"isTargetWin64()", CCDelegateTo<CC_X86_Win64_C>>,
   CCIfSubtarget<"is64Bit()",       CCDelegateTo<CC_X86_64_C>>,
   CCDelegateTo<CC_X86_32_C>
 ]>;
 
+// X86-64 Intel Short Vector Math Library calling convention.
+def CC_Intel_SVML : CallingConv<[
+
+  // The SSE vector arguments are passed in XMM registers.
+  CCIfType<[v4f32, v2f64],
+           CCAssignToReg<[XMM0, XMM1, XMM2]>>,
+
+  // The 256-bit vector arguments are passed in YMM registers.
+  CCIfType<[v8f32, v4f64],
+           CCAssignToReg<[YMM0, YMM1, YMM2]>>,
+
+  // The 512-bit vector arguments are passed in ZMM registers.
+  CCIfType<[v16f32, v8f64],
+           CCAssignToReg<[ZMM0, ZMM1, ZMM2]>>
+]>;
+
 def CC_X86_32_Intr : CallingConv<[
   CCAssignToStack<4, 4>
 ]>;
 
 def CC_X86_64_Intr : CallingConv<[
   CCAssignToStack<8, 8>
 ]>;
 
def CC_X86_64 : CallingConv<[
 
   // Otherwise, drop to normal X86-64 CC
   CCDelegateTo<CC_X86_64_C>
 ]>;
 
 // This is the argument convention used for the entire X86 backend.
 def CC_X86 : CallingConv<[
   CCIfCC<"CallingConv::Intel_OCL_BI", CCDelegateTo<CC_Intel_OCL_BI>>,
+  CCIfCC<"CallingConv::Intel_SVML", CCDelegateTo<CC_Intel_SVML>>,
   CCIfSubtarget<"is64Bit()", CCDelegateTo<CC_X86_64>>,
   CCDelegateTo<CC_X86_32>
 ]>;
 
 //===----------------------------------------------------------------------===//
 // Callee-saved Registers.
 //===----------------------------------------------------------------------===//
 

 def CSR_Win64_RegCall_NoSSE : CalleeSavedRegs<(add RBX, RBP, RSP,
                                               (sequence "R%u", 10, 15))>;
 def CSR_Win64_RegCall       : CalleeSavedRegs<(add CSR_Win64_RegCall_NoSSE,                                  
                                               (sequence "XMM%u", 8, 15))>;
 def CSR_SysV64_RegCall_NoSSE : CalleeSavedRegs<(add RBX, RBP, RSP,
                                                (sequence "R%u", 12, 15))>;
 def CSR_SysV64_RegCall       : CalleeSavedRegs<(add CSR_SysV64_RegCall_NoSSE,               
                                                (sequence "XMM%u", 8, 15))>;
-                                               
+// SVML calling convention 
+def CSR_32_Intel_SVML        : CalleeSavedRegs<(add CSR_32_RegCall_NoSSE)>;
+def CSR_32_Intel_SVML_AVX512 : CalleeSavedRegs<(add CSR_32_Intel_SVML,
+                                                K4, K5, K6, K7)>;
+
+def CSR_64_Intel_SVML_NoSSE : CalleeSavedRegs<(add RBX, RSI, RDI, RBP, RSP, R12, R13, R14, R15)>;
+
+def CSR_64_Intel_SVML       : CalleeSavedRegs<(add CSR_64_Intel_SVML_NoSSE,
+                                               (sequence "XMM%u", 8, 15))>;
+def CSR_Win64_Intel_SVML    : CalleeSavedRegs<(add CSR_64_Intel_SVML_NoSSE,
+                                               (sequence "XMM%u", 6, 15))>;
+
+def CSR_64_Intel_SVML_AVX        : CalleeSavedRegs<(add CSR_64_Intel_SVML_NoSSE,
+                                                    (sequence "YMM%u", 8, 15))>;
+def CSR_Win64_Intel_SVML_AVX     : CalleeSavedRegs<(add CSR_64_Intel_SVML_NoSSE,
+                                                    (sequence "YMM%u", 6, 15))>;
+
+def CSR_64_Intel_SVML_AVX512     : CalleeSavedRegs<(add CSR_64_Intel_SVML_NoSSE,
+                                                    (sequence "ZMM%u", 16, 31),
+                                                    K4, K5, K6, K7)>;
+def CSR_Win64_Intel_SVML_AVX512  : CalleeSavedRegs<(add CSR_64_Intel_SVML_NoSSE,
+                                                    (sequence "ZMM%u", 6, 21),
+                                                    K4, K5, K6, K7)>;

lib/Target/X86/X86ISelLowering.cpp

SDValue X86TargetLowering::LowerFormalArguments(
   }
 
   if (isVarArg && MFI.hasMustTailInVarArgFunc()) {
     // Find the largest legal vector type.
     MVT VecVT = MVT::Other;
     // FIXME: Only some x86_32 calling conventions support AVX512.
     if (Subtarget.hasAVX512() &&
         (Is64Bit || (CallConv == CallingConv::X86_VectorCall ||
-                     CallConv == CallingConv::Intel_OCL_BI)))
+                     CallConv == CallingConv::Intel_OCL_BI   ||
+                     CallConv == CallingConv::Intel_SVML)))
       VecVT = MVT::v16f32;
     else if (Subtarget.hasAVX())
       VecVT = MVT::v8f32;
     else if (Subtarget.hasSSE2())
       VecVT = MVT::v4f32;
 
     // We forward some GPRs and some vector types.
     SmallVector<MVT, 2> RegParmTypes;

lib/Target/X86/X86RegisterInfo.cpp

case CallingConv::Intel_OCL_BI: {
     if (HasAVX && IsWin64)
       return CSR_Win64_Intel_OCL_BI_AVX_SaveList;
     if (HasAVX && Is64Bit)
       return CSR_64_Intel_OCL_BI_AVX_SaveList;
     if (!HasAVX && !IsWin64 && Is64Bit)
       return CSR_64_Intel_OCL_BI_SaveList;
     break;
   }
+  case CallingConv::Intel_SVML: {
+    if (Is64Bit) {
+      if (HasAVX512)
+        return IsWin64 ? CSR_Win64_Intel_SVML_AVX512_SaveList :
+                         CSR_64_Intel_SVML_AVX512_SaveList;
+      if (HasAVX)
+        return IsWin64 ? CSR_Win64_Intel_SVML_AVX_SaveList :
+                         CSR_64_Intel_SVML_AVX_SaveList;
+
+      return IsWin64 ? CSR_Win64_Intel_SVML_SaveList :
+                       CSR_64_Intel_SVML_SaveList;
+    } else { // Is32Bit
+        if (HasAVX512)
+            return CSR_32_Intel_SVML_AVX512_SaveList;
+        return CSR_32_Intel_SVML_SaveList;
+    }
+  }
   case CallingConv::HHVM:
     return CSR_64_HHVM_SaveList;
   case CallingConv::X86_RegCall:
     if (Is64Bit) {
       if (IsWin64) {
         return (HasSSE ? CSR_Win64_RegCall_SaveList :
                          CSR_Win64_RegCall_NoSSE_SaveList);
       } else {
case CallingConv::Intel_OCL_BI: {
     if (HasAVX && IsWin64)
       return CSR_Win64_Intel_OCL_BI_AVX_RegMask;
     if (HasAVX && Is64Bit)
       return CSR_64_Intel_OCL_BI_AVX_RegMask;
     if (!HasAVX && !IsWin64 && Is64Bit)
       return CSR_64_Intel_OCL_BI_RegMask;
     break;
   }
+  case CallingConv::Intel_SVML: {
+    if (Is64Bit) {
+      if (HasAVX512)
+        return IsWin64 ? CSR_Win64_Intel_SVML_AVX512_RegMask :
+                         CSR_64_Intel_SVML_AVX512_RegMask;
+      if (HasAVX)
+        return IsWin64 ? CSR_Win64_Intel_SVML_AVX_RegMask :
+                         CSR_64_Intel_SVML_AVX_RegMask;
+
+      return IsWin64 ? CSR_Win64_Intel_SVML_RegMask :
+                       CSR_64_Intel_SVML_RegMask;
+    } else { // Is32Bit
+        if (HasAVX512)
+            return CSR_32_Intel_SVML_AVX512_RegMask;
+        return CSR_32_Intel_SVML_RegMask;
+    }
+  }
   case CallingConv::HHVM:
     return CSR_64_HHVM_RegMask;
   case CallingConv::X86_RegCall:
     if (Is64Bit) {
       if (IsWin64) {
         return (HasSSE ? CSR_Win64_RegCall_RegMask :
                          CSR_Win64_RegCall_NoSSE_RegMask);
       } else {

lib/Target/X86/X86Subtarget.h

bool isCallingConvWin64(CallingConv::ID CC) const {
     case CallingConv::C:
     case CallingConv::Fast:
     case CallingConv::Swift:
     case CallingConv::X86_FastCall:
     case CallingConv::X86_StdCall:
     case CallingConv::X86_ThisCall:
     case CallingConv::X86_VectorCall:
     case CallingConv::Intel_OCL_BI:
+    case CallingConv::Intel_SVML:
       return isTargetWin64();
     // This convention allows using the Win64 convention on other targets.
     case CallingConv::Win64:
       return true;
     // This convention allows using the SysV convention on Windows targets.
     case CallingConv::X86_64_SysV:
       return false;
     // Otherwise, who knows what this is.

lib/Transforms/Vectorize/LoopVectorize.cpp

for (unsigned Part = 0; Part < UF; ++Part) {
         // Some intrinsics have a scalar argument - don't replace it with a
         // vector.
         if (!UseVectorIntrinsic || !hasVectorInstrinsicScalarOpd(ID, i))
           Arg = getOrCreateVectorValue(CI->getArgOperand(i), Part);
         Args.push_back(Arg);
       }
 
       Function *VectorF;
+      bool FromSVML = false;
       if (UseVectorIntrinsic) {
         // Use vector version of the intrinsic.
         Type *TysForDecl[] = {CI->getType()};
         if (VF > 1)
           TysForDecl[0] = VectorType::get(CI->getType()->getScalarType(), VF);
         VectorF = Intrinsic::getDeclaration(M, ID, TysForDecl);
       } else {
         // Use vector version of the library call.
-        StringRef VFnName = TLI->getVectorizedFunction(FnName, VF);
+        bool IsFast = CI->getFastMathFlags().isFast();
+        std::string VFnName = TLI->getVectorizedFunction(FnName, VF, FromSVML, IsFast);
         assert(!VFnName.empty() && "Vector function name is empty.");
         VectorF = M->getFunction(VFnName);
         if (!VectorF) {
           // Generate a declaration
           FunctionType *FTy = FunctionType::get(RetTy, Tys, false);
           VectorF =
               Function::Create(FTy, Function::ExternalLinkage, VFnName, M);
           VectorF->copyAttributesFrom(F);
         }
       }
       assert(VectorF && "Can't create vector function.");
 
       SmallVector<OperandBundleDef, 1> OpBundles;
       CI->getOperandBundlesAsDefs(OpBundles);
       CallInst *V = Builder.CreateCall(VectorF, Args, OpBundles);
 
       if (isa<FPMathOperator>(V))
         V->copyFastMathFlags(CI);
-
+      if (FromSVML) V->setCallingConv(CallingConv::Intel_SVML);
       VectorLoopValueMap.setVectorValue(&I, Part, V);
       addMetadata(V, &I);
     }
 
     break;
   }
 
   default:

test/Transforms/LoopVectorize/X86/svml-calls.ll

for.body:                                         ; preds = %for.body, %entry
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
   %exitcond = icmp eq i64 %indvars.iv.next, 1000
   br i1 %exitcond, label %for.end, label %for.body
 
 for.end:                                          ; preds = %for.body
   ret void
 }
 
+; CHECK-LABEL: @atan2_finite
+; CHECK: <8 x double> @__svml_atan28
+; CHECK: ret
+
+declare double @__atan2_finite(double, double) local_unnamed_addr #0
+
+define void @atan2_finite([100 x double]* nocapture %varray) local_unnamed_addr #0 {
+entry:
+  br label %for.cond1.preheader
+
+for.cond1.preheader:                              ; preds = %for.inc7, %entry
+  %indvars.iv19 = phi i64 [ 0, %entry ], [ %indvars.iv.next20, %for.inc7 ]
+  %0 = trunc i64 %indvars.iv19 to i32
+  %conv = sitofp i32 %0 to double
+  br label %for.body3
+
+for.body3:                                        ; preds = %for.body3, %for.cond1.preheader
+  %indvars.iv = phi i64 [ 0, %for.cond1.preheader ], [ %indvars.iv.next, %for.body3 ]
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
+  %1 = trunc i64 %indvars.iv.next to i32
+  %conv4 = sitofp i32 %1 to double
+  %call = tail call fast double @__atan2_finite(double %conv, double %conv4)
+  %arrayidx6 = getelementptr inbounds [100 x double], [100 x double]* %varray, i64 %indvars.iv19, i64 %indvars.iv
+  store double %call, double* %arrayidx6, align 8
+  %exitcond = icmp eq i64 %indvars.iv.next, 100
+  br i1 %exitcond, label %for.inc7, label %for.body3, !llvm.loop !5
+
+for.inc7:                                         ; preds = %for.body3
+  %indvars.iv.next20 = add nuw nsw i64 %indvars.iv19, 1
+  %exitcond21 = icmp eq i64 %indvars.iv.next20, 100
+  br i1 %exitcond21, label %for.end9, label %for.cond1.preheader
+
+for.end9:                                         ; preds = %for.inc7
+  ret void
+}
+
 attributes #0 = { nounwind readnone }
+
+!5 = distinct !{!5, !6, !7}
+!6 = !{!"llvm.loop.vectorize.width", i32 8}
+!7 = !{!"llvm.loop.vectorize.enable", i1 true}

utils/TableGen/CMakeLists.txt

add_tablegen(llvm-tblgen LLVM
   PseudoLoweringEmitter.cpp
   RISCVCompressInstEmitter.cpp
   RegisterBankEmitter.cpp
   RegisterInfoEmitter.cpp
   SDNodeProperties.cpp
   SearchableTableEmitter.cpp
   SubtargetEmitter.cpp
   SubtargetFeatureInfo.cpp
+  SVMLEmitter.cpp
   TableGen.cpp
   Types.cpp
   X86DisassemblerTables.cpp
   X86EVEX2VEXTablesEmitter.cpp
   X86FoldTablesEmitter.cpp
   X86ModRMFilters.cpp
   X86RecognizableInstr.cpp
   WebAssemblyDisassemblerEmitter.cpp
   CTagsEmitter.cpp
   )
 set_target_properties(llvm-tblgen PROPERTIES FOLDER "Tablegenning")

utils/TableGen/SVMLEmitter.cpp

+//===------ SVMLEmitter.cpp - Generate SVML function variants -------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This tablegen backend emits the scalar to svml function map for TLI.
+//
+//===----------------------------------------------------------------------===//
+
+#include "CodeGenTarget.h"
+#include "llvm/Support/Format.h"
+#include "llvm/TableGen/Error.h"
+#include "llvm/TableGen/Record.h"
+#include "llvm/TableGen/TableGenBackend.h"
+#include <map>
+#include <vector>
+
+using namespace llvm;
+
+#define DEBUG_TYPE "SVMLVariants"
+#include "llvm/Support/Debug.h"
+
+namespace {
+
+class SVMLVariantsEmitter {
+
+  RecordKeeper &Records;
+
+private:
+  void emitSVMLVariants(raw_ostream &OS);
+
+public:
+  SVMLVariantsEmitter(RecordKeeper &R) : Records(R) {}
+
+  void run(raw_ostream &OS);
+};
+} // End anonymous namespace
+
+/// \brief Emit the set of SVML variant function names.
+// The default is to emit the high accuracy SVML variants until a mechanism is
+// introduced to allow a selection of different variants through precision
+// requirements specified by the user. This code generates mappings to svml
+// that are in the scalar form of llvm intrinsics, math library calls, or the
+// finite variants of math library calls.
+void SVMLVariantsEmitter::emitSVMLVariants(raw_ostream &OS) {
+
+  unsigned MinSinglePrecVL = 4;
+  unsigned MaxSinglePrecVL = 16;
+  unsigned MinDoublePrecVL = 2;
+  unsigned MaxDoublePrecVL = 8;
+
+  Record *SvmlVariantsClass = Records.getClass("SvmlVariant");
+  assert(SvmlVariantsClass &&
+         "SvmlVariant class not found in target description file!");
+
+  OS << "#ifdef GET_SVML_VARIANTS\n";
+
+  for (const auto &D : Records.getDefs()) {
+    std::string SvmlVariantNameStr = D.first;
+    // Single Precision SVML
+    for (unsigned VL = MinSinglePrecVL; VL <= MaxSinglePrecVL; VL *= 2) {
+      // Emit the scalar math library function to svml function entry.
+      OS << "{\"" << SvmlVariantNameStr << "f" << "\", ";
+      OS << "\"" << "__svml_" << SvmlVariantNameStr << "f" << VL << "\", "
+         << VL << "},\n";
+
+      // Emit the scalar intrinsic to svml function entry.
+      OS << "{\"" << "llvm." << SvmlVariantNameStr << ".f32" << "\", ";
+      OS << "\"" << "__svml_" << SvmlVariantNameStr << "f" << VL << "\", "
+         << VL << "},\n";
+
+      // Emit the finite math library function to svml function entry.
+      OS << "{\"__" << SvmlVariantNameStr << "f_finite" << "\", ";
+      OS << "\"" << "__svml_" << SvmlVariantNameStr << "f" << VL << "\", "
+         << VL << "},\n";
+    }
+
+    // Double Precision SVML
+    for (unsigned VL = MinDoublePrecVL; VL <= MaxDoublePrecVL; VL *= 2) {
+      // Emit the scalar math library function to svml function entry.
+      OS << "{\"" << SvmlVariantNameStr << "\", ";
+      OS << "\"" << "__svml_" << SvmlVariantNameStr << VL << "\", " << VL
+         << "},\n";
+
+      // Emit the scalar intrinsic to svml function entry.
+      OS << "{\"" << "llvm." << SvmlVariantNameStr << ".f64" << "\", ";
+      OS << "\"" << "__svml_" << SvmlVariantNameStr << VL << "\", " << VL
+         << "},\n";
+
+      // Emit the finite math library function to svml function entry.
+      OS << "{\"__" << SvmlVariantNameStr << "_finite" << "\", ";
+      OS << "\"" << "__svml_" << SvmlVariantNameStr << VL << "\", "
+         << VL << "},\n";
+    }
+  }
+
+  OS << "#endif // GET_SVML_VARIANTS\n\n";
+}
+
+void SVMLVariantsEmitter::run(raw_ostream &OS) {
+  emitSVMLVariants(OS);
+}
+
+namespace llvm {
+
+void EmitSVMLVariants(RecordKeeper &RK, raw_ostream &OS) {
+  SVMLVariantsEmitter(RK).run(OS);
+}
+
+} // End llvm namespace

utils/TableGen/TableGen.cpp

enum ActionType {
   GenOptParserDefs,
   GenCTags,
   GenAttributes,
   GenSearchableTables,
   GenGlobalISel,
   GenX86EVEX2VEXTables,
   GenX86FoldTables,
   GenRegisterBank,
+  GenSVMLVariants,
 };
 
 namespace {
   cl::opt<ActionType>
   Action(cl::desc("Action to perform:"),
          cl::values(clEnumValN(PrintRecords, "print-records",
                                "Print all records to stdout (default)"),
                     clEnumValN(GenEmitter, "gen-emitter",
Action(cl::desc("Action to perform:"),
                                "Generate generic binary-searchable table"),
                     clEnumValN(GenGlobalISel, "gen-global-isel",
                                "Generate GlobalISel selector"),
                     clEnumValN(GenX86EVEX2VEXTables, "gen-x86-EVEX2VEX-tables",
                                "Generate X86 EVEX to VEX compress tables"),
                     clEnumValN(GenX86FoldTables, "gen-x86-fold-tables",
                                "Generate X86 fold tables"),
                     clEnumValN(GenRegisterBank, "gen-register-bank",
-                               "Generate registers bank descriptions")));
+                               "Generate registers bank descriptions"),
+                    clEnumValN(GenSVMLVariants, "gen-svml",
+                               "Generate SVML variant function names")));
 
   cl::OptionCategory PrintEnumsCat("Options for -print-enums");
   cl::opt<std::string>
   Class("class", cl::desc("Print Enum list for this class"),
         cl::value_desc("class name"), cl::cat(PrintEnumsCat));
 
 bool LLVMTableGenMain(raw_ostream &OS, RecordKeeper &Records) {
   switch (Action) {
case GenRegisterBank:
     EmitRegisterBank(Records, OS);
     break;
   case GenX86EVEX2VEXTables:
     EmitX86EVEX2VEXTables(Records, OS);
     break;
   case GenX86FoldTables:
     EmitX86FoldTables(Records, OS);
     break;
+  case GenSVMLVariants:
+    EmitSVMLVariants(Records, OS);
+    break;
   }
 
   return false;
 }
 }
 
 int main(int argc, char **argv) {
   sys::PrintStackTraceOnErrorSignal(argv[0]);

utils/TableGen/TableGenBackends.h

 void EmitOptParser(RecordKeeper &RK, raw_ostream &OS);
 void EmitCTags(RecordKeeper &RK, raw_ostream &OS);
 void EmitAttributes(RecordKeeper &RK, raw_ostream &OS);
 void EmitSearchableTables(RecordKeeper &RK, raw_ostream &OS);
 void EmitGlobalISel(RecordKeeper &RK, raw_ostream &OS);
 void EmitX86EVEX2VEXTables(RecordKeeper &RK, raw_ostream &OS);
 void EmitX86FoldTables(RecordKeeper &RK, raw_ostream &OS);
 void EmitRegisterBank(RecordKeeper &RK, raw_ostream &OS);
+void EmitSVMLVariants(RecordKeeper &RK, raw_ostream &OS);
 
 } // End llvm namespace
 
 #endif

utils/vim/syntax/llvm.vim

syn keyword llvmKeyword
       \ hhvmcc
       \ hhvm_ccc
       \ hidden
       \ initialexec
       \ inlinehint
       \ inreg
       \ inteldialect
       \ intel_ocl_bicc
+      \ intel_svmlcc
       \ internal
       \ linkonce
       \ linkonce_odr
       \ localdynamic
       \ localexec
       \ local_unnamed_addr
       \ minsize
       \ module