[AArch64] Enable libm vectorized functions via SLEEF

[steleman]

Summary

This changeset is modeled after Intel's submission for SVML. It enables trigonometry functions vectorization via SLEEF: http://sleef.org/.

• A new vectorization library enum is added to TargetLibraryInfo.h: SLEEF.

• A new option is added to TargetLibraryInfoImpl - ClVectorLibrary: SLEEF.

• A comprehensive test case is included in this changeset.

• In a separate changeset (for clang), a new vectorization library argument is added to -fveclib: -fveclib=SLEEF.

Trigonometry functions that are vectorized by sleef:

acos
asin
atan
atanh
cos
cosh
exp
exp2
exp10
lgamma
log10
log2
log
sin
sinh
sqrt
tan
tanh
tgamma

[steleman]

The corresponding Clang changeset is: https://reviews.llvm.org/D53928.

[fpetrogalli]

Hi @steleman , thank you for working on this!

Before doing a detailed review, I have a couple of comments and requests.

1. Please add full context to the patch (git show -U999999).
2. You are using Vector Function ABI mangled names, which means you ware assuming to link agains libsleefgnuabi.so instead of libsleef.so. For this reason, I think it is worth to replace the -fveclib=SLEEF option with -fveclib=SLEEFGNUABI. This request applies to all places where you have used SLEEF.
3. Given that SLEEF has some degree of target independence, I would recommend using the TargetTriple if that selects the list of functions into a switch statement that sets the architectural extension token in the mangled names. On the long term, the list of available vector functions should be generated by setting such token instead of hard coding all names for all targets. Notice that I am not saying that you should add all targets, I am just saying that you should prepare the ground for extending this, as I think that -fveclib=SLEEFGNUABI should work the same for all the extensions supported by SLEEF.
4. On the longer term, I want to make you aware of the fact that there is plan to replace the TLI implementation of -fveclib with an OpenMP based one (to decouple backend and frontend). We discussed this at the last LLVM dev meeting, I will circulate an RFC soon. See https://llvm.org/devmtg/2018-10/talk-abstracts.html#bof7.
5. Please change your commit messages header, the function you are adding are not just trigonometric functions. May I suggest "[AArch64] Enable libm vectorized functions via SLEEFGNUABI"?

Kind regards,

Francesco

[steleman]

In D53927#1282380, @fpetrogalli wrote:

Hi @steleman , thank you for working on this!

Before doing a detailed review, I have a couple of comments and requests.

2. You are using Vector Function ABI mangled names, which means you ware assuming to link agains libsleefgnuabi.so instead of libsleef.so. For this reason, I think it is worth to replace the -fveclib=SLEEF option with -fveclib=SLEEFGNUABI. This request applies to all places where you have used SLEEF.

Sorry but I wholeheartedly disagree with this idea.

For starters, saying -fveclib=SLEEFGNUABI is just ugly IMO.

Furthermore, I do not understand why this change is necessary. It provides nothing that the current naming convention doesn't already provide. So, really, it's a matter of personal preference.

Also: Intel did not name their veclib argument "SVMLINTELABI". They named it "SVML".

3. Given that SLEEF has some degree of target independence, I would recommend using the TargetTriple if that selects the list of functions into a switch statement that sets the architectural extension token in the mangled names. On the long term, the list of available vector functions should be generated by setting such token instead of hard coding all names for all targets. Notice that I am not saying that you should add all targets, I am just saying that you should prepare the ground for extending this, as I think that -fveclib=SLEEFGNUABI should work the same for all the extensions supported by SLEEF.

I don't understand any of this. It seems to me that you are describing what I've already done.

The TargetTriple wasn't present in the TargetLibraryInfoImpl class. I had to add it, specifically because it is needed by the SLEEF switch statement case for AArch64.

On the long term, the list of available vector functions should be generated by setting such token instead of hard coding all names for all targets.

I don't understand what this means.

Are you suggesting that the VecLib array of function name mappings should be generated by tblgen? Yes, I agree. It should be. But it currently isn't. And I don't know when tblgen will be extended to do this job. Or if it ever will be extended to do this job.

Also, we (Cavium) will not be extending the SLEEF mappings to other ISA's. Simply because we do not have the capability of testing PPC64, or Windows.

So, really, providing additional SLEEF mappings for other ISA's or platforms is best suited for those who are best versed in the subtle details of those other ISA's or platforms.

The SLEEFGNUABI ABI has very little CPU independence when compiling for for X86_64. It is, in fact, very CPU-dependent.

In fact, in the current implementation of TargetLibraryInfo, it is impossible to provide X86_64 mappings for SLEEF using the GNUABI. That is because the X86_64 GNUABI mappings require knowledge of the capabilities of the specific CPU that the object is being compiled for. These CPU capabilities are hardcoded by SLEEF, in the SLEEF GNU ABI mangled name:

'b' -> SSE
'c' -> AVX
'd' -> AVX2
'e' -> AVX512

So, at a minimum, on X86_64, you would need to know the values passed to -march= and -mtune=. But these CPU capabilities are currently inaccessible from TargetLibraryInfo.

TargetLibraryInfo could be extended to acquire the CPU capability information, but that's for another changeset. Maybe after TargetLibraryInfo is extended to support this capability we can revisit the construction of the GNU ABI mangled name.

4. On the longer term, I want to make you aware of the fact that there is plan to replace the TLI implementation of -fveclib with an OpenMP based one (to decouple backend and frontend). We discussed this at the last LLVM dev meeting, I will circulate an RFC soon. See https://llvm.org/devmtg/2018-10/talk-abstracts.html#bof7.

That's an interesting plan.

I'd like to make you aware that brute-forcing OpenMP for vectorization will probably run afoul of SPEC rules.

SPEC allows OpenMP in SPECspeed, but not in SPECrate, and that is clearly stated at the SPEC web site. Not to mention that the SPEC benchmarks code cannot be modified to add #pragma omp simd. And having clang silently and secretly modify emitted code to insert OpenMP simd will likely not fly at SPEC.

I am also not a fan of OpenMP being used as a giant hammer that can hit any nail. You would be introducing a dependency on OpenMP for programs that do not otherwise use, or need, OpenMP. So, in reality, there is no "decoupling" here. There is only a dependency substitution: the existing dependency on SLEEF is replaced by the future dependency on OpenMP.

5. Please change your commit messages header, the function you are adding are not just trigonometric functions. May I suggest "[AArch64] Enable libm vectorized functions via SLEEFGNUABI"?

I'll settle for "[AArch64] Enable libm vectorized functions via SLEEF".

[llvm-commits]

2. You are using Vector Function ABI mangled names, which means you ware assuming to link agains libsleefgnuabi.so instead of libsleef.so. For this reason, I think it is worth to replace the -fveclib=SLEEF option with -fveclib=SLEEFGNUABI. This request applies to all places where you have used SLEEF.

Sorry but I wholeheartedly disagree with this idea.

For starters, saying -fveclib=SLEEFGNUABI is just ugly IMO.

Furthermore, I do not understand why this change is necessary. It provides nothing that the current naming convention doesn't already provide. So, really, it's a matter of personal preference.

Fair enough. :)
I just think SLEEFGNUABI is more precise, and avoids confusion given that SLEEF is packaged in two version, with incompatible naming conventions and signatures. You are clearly using the conventions in libsleefgnuabi.so, I think I just think it is better to use -fveclib=SLEEFGNUABI.
Moreover, someone might want to add -fveclib for the non GNUABI version of the library - for example, they might prefer the sincos signatures of libsleef over libsleefgnuabi. I agree that -ffveclib=SLEEFGNUABI might be ugly to someone, but not more than -fveclib=SLEEFNONGNUABi.

I will not enforce this though, it is just a name.

Also: Intel did not name their veclib argument "SVMLINTELABI". They named it "SVML”.

I think this could have happened whether they would have come up with a version of the library that had new signatures and name mangling scheme.

3. Given that SLEEF has some degree of target independence, I would recommend using the TargetTriple if that selects the list of functions into a switch statement that sets the architectural extension token in the mangled names. On the long term, the list of available vector functions should be generated by setting such token instead of hard coding all names for all targets. Notice that I am not saying that you should add all targets, I am just saying that you should prepare the ground for extending this, as I think that -fveclib=SLEEFGNUABI should work the same for all the extensions supported by SLEEF.

I don't understand any of this. It seems to me that you are describing what I've already done.

The TargetTriple wasn't present in the TargetLibraryInfoImpl class. I had to add it, specifically because it is needed by the SLEEF switch statement case for AArch64.

On the long term, the list of available vector functions should be generated by setting such token instead of hard coding all names for all targets.

I don't understand what this means.

Are you suggesting that the VecLib array of function name mappings should be generated by tblgen? Yes, I agree. It should be. But it currently isn't. And I don't know when tblgen will be extended to do this job. Or if it ever will be extended to do this job.

Also, we (Cavium) will not be extending the SLEEF mappings to other ISA's. Simply because we do not have the capability of testing PPC64, or Windows.

So, really, providing additional SLEEF mappings for other ISA's or platforms is best suited for those who are best versed in the subtle details of those other ISA's or platforms.

The SLEEFGNUABI ABI has very little CPU independence when compiling for for X86_64. It is, in fact, very CPU-dependent.

In fact, in the current implementation of TargetLibraryInfo, it is impossible to provide X86_64 mappings for SLEEF using the GNUABI. That is because the X86_64 GNUABI mappings require knowledge of the capabilities of the specific CPU that the object is being compiled for. These CPU capabilities are hardcoded by SLEEF, in the SLEEF GNU ABI mangled name:

'b' -> SSE
'c' -> AVX
'd' -> AVX2
'e' -> AVX512

So, at a minimum, on X86_64, you would need to know the values passed to -march= and -mtune=. But these CPU capabilities are currently inaccessible from TargetLibraryInfo.

TargetLibraryInfo could be extended to acquire the CPU capability information, but that's for another changeset. Maybe after TargetLibraryInfo is extended to support this capability we can revisit the construction of the GNU ABI mangled name.

I think there is no need to use tablegen for this. For the rest, I agree with all you said. Let me explain what I exactly mean by “prepare the ground to extend this”.

You have encoded the list of function for AArch64 as follows (allow me to use pseudocode).

If (AArch64) {
  List ={ {“sin”, “_ZGVn2v_sin”, 2},
             {“cos”, “_ZGVn2v_cos”, 2},
             ….
};
}

I think you should rewrite your code as

  List ={ {“sin”, “_ZGV<simdext>2v_sin”, 2},
             {“cos”, “_ZGV<simdext>2v_cos”, 2},
             ….
};
If (aarch64)
   ext_token = “n”.

Loop through (list) and replace “<simdext>” with ext_token.

With this change, if anyone decide to add any of the other architectures (of course, adding was is missing in target triple to discern the right extension token), all they have to do is to add new logic to set the ext_token variable correctly.

Let me stress again the fact that I am not asking you do do the non aarch64 work. I just think it make sense to make sure other people can extend it easily.

4. On the longer term, I want to make you aware of the fact that there is plan to replace the TLI implementation of -fveclib with an OpenMP based one (to decouple backend and frontend). We discussed this at the last LLVM dev meeting, I will circulate an RFC soon. See https://llvm.org/devmtg/2018-10/talk-abstracts.html#bof7.

That's an interesting plan.

I'd like to make you aware that brute-forcing OpenMP for vectorization will probably run afoul of SPEC rules.

SPEC allows OpenMP in SPECspeed, but not in SPECrate, and that is clearly stated at the SPEC web site. Not to mention that the SPEC benchmarks code cannot be modified to add #pragma omp simd. And having clang silently and secretly modify emitted code to insert OpenMP simd will likely not fly at SPEC.

I am also not a fan of OpenMP being used as a giant hammer that can hit any nail. You would be introducing a dependency on OpenMP for programs that do not otherwise use, or need, OpenMP. So, in reality, there is no "decoupling" here. There is only a dependency substitution: the existing dependency on SLEEF is replaced by the future dependency on OpenMP.

The OpenMP based implementation will pull in only the simd capabilities of openmp (-fopenmp-simd), not the whole OpenMP specs. I will be more detailed on this in the RFC, and add you in the discussion to make sure your concerns are addressed.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

---

llvm-commits mailing list
[e-mail]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits

[fpetrogalli]

Apologies for the confidentiality notice at the end my last message. It wasn't supposed to be there, please ignore it. Nothing of what I have written is confidential.

[steleman]

In D53927#1282601, @llvm-commits wrote:

The SLEEFGNUABI ABI has very little CPU independence when compiling for for X86_64. It is, in fact, very CPU-dependent.

In fact, in the current implementation of TargetLibraryInfo, it is impossible to provide X86_64 mappings for SLEEF using the GNUABI. That is because the X86_64 GNUABI mappings require knowledge of the capabilities of the specific CPU that the object is being compiled for. These CPU capabilities are hardcoded by SLEEF, in the SLEEF GNU ABI mangled name:

'b' -> SSE
'c' -> AVX
'd' -> AVX2
'e' -> AVX512

So, at a minimum, on X86_64, you would need to know the values passed to -march= and -mtune=. But these CPU capabilities are currently inaccessible from TargetLibraryInfo.

TargetLibraryInfo could be extended to acquire the CPU capability information, but that's for another changeset. Maybe after TargetLibraryInfo is extended to support this capability we can revisit the construction of the GNU ABI mangled name.

I think there is no need to use tablegen for this. For the rest, I agree with all you said. Let me explain what I exactly mean by “prepare the ground to extend this”.

You have encoded the list of function for AArch64 as follows (allow me to use pseudocode).

If (AArch64) {
  List ={ {“sin”, “_ZGVn2v_sin”, 2},
             {“cos”, “_ZGVn2v_cos”, 2},
             ….
};
}

I think you should rewrite your code as

  List ={ {“sin”, “_ZGV<simdext>2v_sin”, 2},
             {“cos”, “_ZGV<simdext>2v_cos”, 2},
             ….
};
If (aarch64)
   ext_token = “n”.

Loop through (list) and replace “<simdext>” with ext_token.

With this change, if anyone decide to add any of the other architectures (of course, adding was is missing in target triple to discern the right extension token), all they have to do is to add new logic to set the ext_token variable correctly.

OK, I like the idea of having a generalized way of expressing the libm <-> SLEEF bindings.

But, there's always a 'but':

1. Looping through this static array of struct bindings, and doing the replacement of the ${ext_token} with the appropriate mangling symbol requires an extra function call. Do we want that extra function call?

2. This gets even more complicated. Here's why:

• Here are the available SLEEF functions for atan on AArch64. We get them with

readelf -sW libsleefgnuabi.so.3.3 | awk '{ printf "%s\t%s\n", $5, $8 }' | egrep atan | egrep -v 'atan2|atanh

GLOBAL _ZGVnN2v_atan
GLOBAL _ZGVnN4v_atanf_u35
GLOBAL _ZGVnN4v_atanf
GLOBAL _ZGVnN2v_atan_u35
GLOBAL _ZGVnN2v_atan_u35
GLOBAL _ZGVnN2v_atan
GLOBAL _ZGVnN4v_atanf
GLOBAL _ZGVnN4v_atanf_u35

• And here are the SLEEF functions for atan for X86_64, which was compiled with -march=core-avx2:

WEAK _ZGVeM8v___atan_finite
GLOBAL _ZGVeM8v_atan
GLOBAL _ZGVeM8v_atan_u35
GLOBAL _ZGVbN4v_atanf
GLOBAL _ZGVbN2v_atan_u35
GLOBAL _ZGVcN4v_atan
GLOBAL _ZGVeN8v_atan_u35
GLOBAL _ZGVdN4v_atan_u35
GLOBAL _ZGVbN2v_atan
GLOBAL _ZGVbN4v_atanf_u35
WEAK _ZGVeM16v___atanf_finite_u35
WEAK _ZGVeM16v___atanf_finite
GLOBAL _ZGVdN8v_atanf_u35
WEAK _ZGVeM8v___atan_finite_u35
GLOBAL _ZGVcN4v_atan_u35
GLOBAL _ZGVdN8v_atanf
GLOBAL _ZGVcN8v_atanf
GLOBAL _ZGVcN8v_atanf_u35
GLOBAL _ZGVdN4v_atan
GLOBAL _ZGVeN16v_atanf_u35
GLOBAL _ZGVeN16v_atanf
GLOBAL _ZGVeM16v_atanf
GLOBAL _ZGVeM16v_atanf_u35
GLOBAL _ZGVeN8v_atan
GLOBAL _ZGVeN16v_atanf
WEAK _ZGVeM8v___atan_finite
WEAK _ZGVeM16v___atanf_finite_u35
GLOBAL _ZGVeM8v_atan_u35
GLOBAL _ZGVdN4v_atan_u35
GLOBAL _ZGVbN4v_atanf
WEAK _ZGVeM16v___atanf_finite
WEAK _ZGVeM8v___atan_finite_u35
GLOBAL _ZGVeM16v_atanf
GLOBAL _ZGVeM8v_atan
GLOBAL _ZGVeN16v_atanf_u35
GLOBAL _ZGVcN4v_atan_u35
GLOBAL _ZGVcN4v_atan
GLOBAL _ZGVeN8v_atan_u35
GLOBAL _ZGVbN4v_atanf_u35
GLOBAL _ZGVdN4v_atan
GLOBAL _ZGVbN2v_atan_u35
GLOBAL _ZGVdN8v_atanf
GLOBAL _ZGVbN2v_atan
GLOBAL _ZGVcN8v_atanf
GLOBAL _ZGVcN8v_atanf_u35
GLOBAL _ZGVdN8v_atanf_u35
GLOBAL _ZGVeN8v_atan
GLOBAL _ZGVeM16v_atanf_u35

As you can see, for X86_64, the bindings are not as straightforward (read: 1-1) as they currently are on AArch64. On X86_64, there's several variants for atan, and for the same vectorization factor. The discriminating factor between the several variants being the CPU capability.

This will happen on AArch64 too when SVE is introduced. We'll end up having several different versions of atan: one for SVE, the other for non-SVE.

I have a strong suspicion - that I have not verified in practice - that PPC/PPC64 is exactly the same. There will be several different versions for every function, depending on which vectorization model / CPU capability model was chosen at compile-time (-march= | -mtune=).

Right now, it is not possible to make the distinction between different CPU capabilities inside TargetLibraryInfo. That's because the information from -march= and -mtune= isn't captured in TargetLibraryInfo. The only thing we have - right now - is the TargetTriple. And the TargetTriple tells us nothing about the -march= and/or -mcpu= micro-arch tuning.

So, we need to add some new information to TargetLibraryInfo: namely the -march= and -mtune= information that was passed to clang. I think implementing this bit is part of a larger - and different - discussion. And I am very reluctant to mix the TargetLibraryInfo enhancement with SLEEF support in AArch64, in the same changeset.

So, what I would suggest for now, is that we postpone this generalization of the libm <-> SLEEF bindings until we have a complete picture of what are the various - and subtle - differences between the various CPU capabilities mangling. We don't yet have a complete picture - I can't build SLEEF on PPC/PPC64 simply because I don't have access to a PPC/PPC64 box.

What do you think?

[fpetrogalli]

OK, I like the idea of having a generalized way of expressing the libm <-> SLEEF bindings.

Great! But… (yes, I also have a but now!)

But, there's always a 'but':

1. Looping through this static array of struct bindings, and doing the replacement of the ${ext_token} with the appropriate mangling symbol requires an extra function call. Do we want that extra function call?

2. This gets even more complicated. Here's why:

• Here are the available SLEEF functions for atan on AArch64. We get them with

readelf -sW libsleefgnuabi.so.3.3 | awk '{ printf "%s\t%s\n", $5, $8 }' | egrep atan | egrep -v 'atan2|atanh

GLOBAL _ZGVnN2v_atan
GLOBAL _ZGVnN4v_atanf_u35
GLOBAL _ZGVnN4v_atanf
GLOBAL _ZGVnN2v_atan_u35
GLOBAL _ZGVnN2v_atan_u35
GLOBAL _ZGVnN2v_atan
GLOBAL _ZGVnN4v_atanf
GLOBAL _ZGVnN4v_atanf_u35

• And here are the SLEEF functions for atan for X86_64, which was compiled with -march=core-avx2:

WEAK _ZGVeM8v___atan_finite
GLOBAL _ZGVeM8v_atan
GLOBAL _ZGVeM8v_atan_u35
GLOBAL _ZGVbN4v_atanf
GLOBAL _ZGVbN2v_atan_u35
GLOBAL _ZGVcN4v_atan
GLOBAL _ZGVeN8v_atan_u35
GLOBAL _ZGVdN4v_atan_u35
GLOBAL _ZGVbN2v_atan
GLOBAL _ZGVbN4v_atanf_u35
WEAK _ZGVeM16v___atanf_finite_u35
WEAK _ZGVeM16v___atanf_finite
GLOBAL _ZGVdN8v_atanf_u35
WEAK _ZGVeM8v___atan_finite_u35
GLOBAL _ZGVcN4v_atan_u35
GLOBAL _ZGVdN8v_atanf
GLOBAL _ZGVcN8v_atanf
GLOBAL _ZGVcN8v_atanf_u35
GLOBAL _ZGVdN4v_atan
GLOBAL _ZGVeN16v_atanf_u35
GLOBAL _ZGVeN16v_atanf
GLOBAL _ZGVeM16v_atanf
GLOBAL _ZGVeM16v_atanf_u35
GLOBAL _ZGVeN8v_atan
GLOBAL _ZGVeN16v_atanf
WEAK _ZGVeM8v___atan_finite
WEAK _ZGVeM16v___atanf_finite_u35
GLOBAL _ZGVeM8v_atan_u35
GLOBAL _ZGVdN4v_atan_u35
GLOBAL _ZGVbN4v_atanf
WEAK _ZGVeM16v___atanf_finite
WEAK _ZGVeM8v___atan_finite_u35
GLOBAL _ZGVeM16v_atanf
GLOBAL _ZGVeM8v_atan
GLOBAL _ZGVeN16v_atanf_u35
GLOBAL _ZGVcN4v_atan_u35
GLOBAL _ZGVcN4v_atan
GLOBAL _ZGVeN8v_atan_u35
GLOBAL _ZGVbN4v_atanf_u35
GLOBAL _ZGVdN4v_atan
GLOBAL _ZGVbN2v_atan_u35
GLOBAL _ZGVdN8v_atanf
GLOBAL _ZGVbN2v_atan
GLOBAL _ZGVcN8v_atanf
GLOBAL _ZGVcN8v_atanf_u35
GLOBAL _ZGVdN8v_atanf_u35
GLOBAL _ZGVeN8v_atan
GLOBAL _ZGVeM16v_atanf_u35

As you can see, for X86_64, the bindings are not as straightforward (read: 1-1) as they currently are on AArch64. On X86_64, there's several variants for atan, and for the same vectorization factor. The discriminating factor between the several variants being the CPU capability.

… but you made a good point here in listing the variants with different lanes on x86. I missed this in my reasoning yesterday. Even if we find a way of generalizing the list of function, we still have to make sure the NumberOfLanes integral field of the mapping is appropriate for the function being used, which means the list of AArch64 functions cannot be shared with the x86 one. The hypothetical function call you mentioned before for setting the name would have to set the number of lanes, and actually add new records to the list… too convoluted when compared to a clear static list.

This will happen on AArch64 too when SVE is introduced. We'll end up having several different versions of atan: one for SVE, the other for non-SVE.

I have a strong suspicion - that I have not verified in practice - that PPC/PPC64 is exactly the same. There will be several different versions for every function, depending on which vectorization model / CPU capability model was chosen at compile-time (-march= | -mtune=).

Right now, it is not possible to make the distinction between different CPU capabilities inside TargetLibraryInfo. That's because the information from -march= and -mtune= isn't captured in TargetLibraryInfo. The only thing we have - right now - is the TargetTriple. And the TargetTriple tells us nothing about the -march= and/or -mcpu= micro-arch tuning.

So, we need to add some new information to TargetLibraryInfo: namely the -march= and -mtune= information that was passed to clang. I think implementing this bit is part of a larger - and different - discussion. And I am very reluctant to mix the TargetLibraryInfo enhancement with SLEEF support in AArch64, in the same changeset.

Yes, and the way we will go will probably be to skip the TLI and give this responsibility to the LoopVectorizer. But as you said, this is a discussion that needs to be taken at a different level, I will talk about this in the openMP sims-only based proposal.

So, what I would suggest for now, is that we postpone this generalization of the libm <-> SLEEF bindings until we have a complete picture of what are the various - and subtle - differences between the various CPU capabilities mangling. We don't yet have a complete picture - I can't build SLEEF on PPC/PPC64 simply because I don't have access to a PPC/PPC64 box.

What do you think?

I agree with you. The design of the patch is OK as it is. Thank you for your patience. I will go through it carefully.

But, if I may, I would like to stress the fact that you should use -fveclib=SLEEFGNUABI. I think we can agree that, although SLEEFGNUABI might be ugly, it is the right name to use for uniformity with the linker flags it implies (-fsleefgnuabi).

Kind regards,

Francesco

[shibatch]

Is it easy to add pow and atan2?
Since these functions are relatively frequently used, I think people would appreciate if you could also add these two functions.

[steleman]

In D53927#1285092, @shibatch wrote:

Is it easy to add pow and atan2?
Since these functions are relatively frequently used, I think people would appreciate if you could also add these two functions.

I will add them. They will be in the next revision of the changeset.

[fpetrogalli]

GLOBAL _ZGVnN2v_atan
GLOBAL _ZGVnN4v_atanf_u35
GLOBAL _ZGVnN4v_atanf
GLOBAL _ZGVnN2v_atan_u35
GLOBAL _ZGVnN2v_atan_u35
GLOBAL _ZGVnN2v_atan
GLOBAL _ZGVnN4v_atanf
GLOBAL _ZGVnN4v_atanf_u35

@steleman , one more thing. You might want to map the functions to the _u35 version when it is available, because the one with no _u35 postfix are using the 1.0ULP implementation which is slower than the 3.5ULP one. Of course, assuming that 3.5 ULP is good enough for you.

Francesco

[steleman]

• changed the -fveclib=<X> argument value to 'sleefgnuabi'.
• added atan2 and pow
• spreadsheet with comparison between libm and sleef is here:

https://docs.google.com/spreadsheets/d/1lcpESCnuzEoTl_XHBqE9FLL0tXJB_tZGR8yciCx1yjg/edit?usp=sharing

• comprehensive test case in C with timings is here:

https://drive.google.com/open?id=1PGKRUdL29_ANoYebOo3Q59syhKp_mNSj

[shibatch]

I am sorry for not reading the comments carefully, but I think 3.5-ULP functions should be used only when a fast-math option is specified.

[steleman]

In D53927#1288314, @shibatch wrote:

I am sorry for not reading the comments carefully, but I think 3.5-ULP functions should be used only when a fast-math option is specified.

Which means we have to use the non 3.5-ULP functions because we have no way of knowing from inside TargetLibraryInfo if -ffast-math was specified on compile-line or not.

I will revert it back to the non 3.5-ULP functions tomorrow.

[steleman]

• Reverted to using the non _u35 SLEEF function names as per comment from @shibatch.

[steleman]

Ping!!

[fpetrogalli]

Hi @steleman,

sorry for the delay in getting back to you. I have a couple of observations, for you and @shibatch.

1. @steleman I don't understand some of the values in your benchmarks. In particular, sin and cos should have similar timings, not differ so much as in your report. I wonder whether the choice of the CLOCK_PROCESS_CPUTIME_ID might have caused this. I think that CLOCK_PROCESS_CPUTIME_ID might translate in a syscall, and therefore cause much overhead in the measurement. I'd rather use CLOCK_MONOTONIC. Also, to make sure you are just measuring the function latency, I think you should invoke the benchmark on array of smaller size, and invoke the call a couple of times before actually starting the time measurement, to reduce the amount of noise causes by warm up effects.

2. @shibatch, do you know for sure that we shouldn't use the 3.5 ULP version of the function? I have just run the following example with both -O3 and -Ofast, they both produce the same SVML calls. As far as I know SVML guarantee a 4ULP precision, so I think we are good in doing SLEEF. @shibatch, so you have any advice here?

// compile with "clang -fveclib=SVML ~/test.c -o - -c -S -emit-llvm [-Ofast|-O3]"
#include <math.h>

void f(double * restrict x, double * restrict y, unsigned n) {
  int i;
  for (i = 0; i < n; ++i) {
    x[i]=sin(y[i]);
  }
}

On a final note, you'll have to excuse again delays in my replies in the coming week, because of SC18. If you will be there too, it would be great to meet.

Kind regards,

Francesco

[shibatch]

@shibatch, do you know for sure that we shouldn't use the 3.5 ULP version of the function?

For example, wikepedia says "Reputable numeric libraries compute the basic transcendental functions to between 0.5 and about 1 ULP".

https://en.wikipedia.org/wiki/Unit_in_the_last_place

As far as I know SVML guarantee a 4ULP precision, so I think we are good in doing SLEEF.

I think there is a room for discussion here. Since your OpemMP patch will have the same problem, it is better to post an RFC for this.

[steleman]

1. @steleman I don't understand some of the values in your benchmarks. In particular, sin and cos should have similar timings, not differ so much as in your report. I wonder whether the choice of the CLOCK_PROCESS_CPUTIME_ID might have caused this. I think that CLOCK_PROCESS_CPUTIME_ID might translate in a syscall, and therefore cause much overhead in the measurement. I'd rather use CLOCK_MONOTONIC. Also, to make sure you are just measuring the function latency, I think you should invoke the benchmark on array of smaller size, and invoke the call a couple of times before actually starting the time measurement, to reduce the amount of noise causes by warm up effects.

I don't understand this. Are you suggesting that decreasing the size of the statistical sample increases the accuracy?

At any rate, the test benchmark and the results spreadsheet aren't part of this changeset.

So I'm confused as to what's being asked here.

[steleman]

If there are no more comments directly related to this changeset, can we move this along?

[steleman]

Ping!!!

[fpetrogalli]

Hi @steleman , thank you for your patience.

I don't understand this. Are you suggesting that decreasing the size of the statistical sample increases the accuracy?

yes, reducing the size of the array increases the accuracy of the measurement, if what you are measuring is latency of the function. You should also run each measurement multiple times and get the best.

At any rate, the test benchmark and the results spreadsheet aren't part of this changeset.
So I'm confused as to what's being asked here.

Yes, the benchmarks and the results are not part of the submission. I think your results are quite unstable, I was suggesting changes to get them more stable (sin shouldn't differ so much from cos, because their algorithm is very similar). If I may, I would like to point you at the open source benchmarks that our math library (libm) team uses: https://github.com/ARM-software/optimized-routines/blob/master/test/mathbench.c#L253-L274

Having said that, the implementation of the functionality looks good to me. My only concerns are about the actual performance, that is why I was suggesting to use the 3.5 ULP version of the functions. But I'll let you sort out which version is best for you.

As per submitting this code, I have to redirect you to the maintainer of this part of the back-end for submitting your changes.

Kind regards,

Francesco

[steleman]

In D53927#1301191, @fpetrogalli wrote:

Hi @steleman , thank you for your patience.

As per submitting this code, I have to redirect you to the maintainer of this part of the back-end for submitting your changes.

It would have been much more appropriate to make this re-direction a week ago, and not let this changeset rot unnecessarily for a week.

[joel_k_jones]

@fpetrogalli Who do you recommend as the maintainer? Looking at the history for TargetLibraryInfo.{cpp,h} doesn't indicate an obvious candidate.

[fpetrogalli]

@joel_k_jones , @steleman

I am sorry for having kept you waiting for the patch. I was not fully aware of the approval process upstream (didn’t do much contributions upstream myself), and before being able to approve I had to understand the process.

I will approve the patch and you will be able to submit it, but before doing that I would like to ask another tiny change.

I would like to change the value of the option from -fveclib=SLEEFGNUABI to -fveclib=SLEEFGNUABI10. The rest of the patch can stay the same.

By specifying “10” at the end of the option value, it will be clear to the user that they are using high precision functions. In the future, we might add the 3.5 ULP ones with the option SLEEFGNUABI35.

Thank you for promoting SLEEF in clang!

I hope this makes sense.

Francesco

[steleman]

In D53927#1304262, @fpetrogalli wrote:

@joel_k_jones , @steleman

I will approve the patch and you will be able to submit it, but before doing that I would like to ask another tiny change.

I would like to change the value of the option from -fveclib=SLEEFGNUABI to -fveclib=SLEEFGNUABI10. The rest of the patch can stay the same.

By specifying “10” at the end of the option value, it will be clear to the user that they are using high precision functions. In the future, we might add the 3.5 ULP ones with the option SLEEFGNUABI35.

This is completely unnecessary.

Whether or not SLEEF uses ULP 1.0 or ULP 3.5 will be determined by -ffast-math passed on compile-line. There is no need to invent two or more different argument values for SLEEF.

For now, SLEEF will use ULP 1.0 exclusively.

When TargetLibraryInfo is expanded to capture the information about -mcpu=, -march=, -mtune= and -ffast-math, the choice between ULP 1.0 and ULP 3.5 will happen silently at compile-time.

There is no need for having two or more different argument values for -fveclib=sleefgnuabi.

This opens the door for having potentially N number of possible values for -fveclib=sleefgnuabi. That's confusing to the end-user and it doesn't follow the existing model already implemented by Intel with SVML.

[fpetrogalli]

In D53927#1304274, @steleman wrote:

For now, SLEEF will use ULP 1.0 exclusively.

When TargetLibraryInfo is expanded to capture the information about -mcpu=, -march=, -mtune= and -ffast-math, the choice between ULP 1.0 and ULP 3.5 will happen silently at compile-time.

Ah, this sounds reasonable to me, thank you for pointing it out, apologies in advance in case you already mentioned it and I missed it.

Kind regards,

Francesco

[joel_k_jones]

Reverted in rL347511

[chandlerc]

In D53927#1304262, @fpetrogalli wrote:

@joel_k_jones , @steleman

I am sorry for having kept you waiting for the patch. I was not fully aware of the approval process upstream (didn’t do much contributions upstream myself), and before being able to approve I had to understand the process.

I will approve the patch and you will be able to submit it, but before doing that I would like to ask another tiny change.

FWIW, I'm sorry to push on this, but I think it would be better to have review from folks more familiar w/ generic LLVM optimizations and with SLEEF to review this. The obvious candidate, IMO, would be Hal. If Hal is unavailable, we could have a number of other people that would be very effective at reviewing (Tim, others...) with deep and londstanding experience in this place.

Given that this is a pretty significant change for AArch64 at least, and potentially for LLVM as a whole, so I think it is reasonable to expect a bit more wide review before it gets landed. Essentially, I'd suggest being a bit more reluctant to approve this kind of patch. =D

That said, I also want to say a huge thank you for actually reviewing (and doing a really detailed review!) of patches like this. It's really helpful, and please don't stop doing taht. It's simply amazing and definitely helps our process scale.

[steleman]

In D53927#1307070, @chandlerc wrote:

FWIW, I'm sorry to push on this, but I think it would be better to have review from folks more familiar w/ generic LLVM optimizations and with SLEEF to review this. The obvious candidate, IMO, would be Hal. If Hal is unavailable, we could have a number of other people that would be very effective at reviewing (Tim, others...) with deep and londstanding experience in this place.

Given that this is a pretty significant change for AArch64 at least, and potentially for LLVM as a whole, so I think it is reasonable to expect a bit more wide review before it gets landed.

To make things perfectly clear:

• This changeset doesn't do anything that wasn't already in LLVM. -fveclib=SVML support was added by Intel prior to this changeset.
• There is nothing here that qualifies as this is a pretty significant change for AArch64 at least, and potentially for LLVM as a whole, as -fveclib=<X> is not the default in clang.
• The only difference is in the veclib library: SLEEF instead of SVML. And in the vector function names, obviously.
• @shibatch - a.k.a. the author of SLEEF - is subscribed to this changeset, and has commented on it already.

Essentially, I'd suggest being a bit more reluctant to approve this kind of patch.

I'd like a better understanding of what this really means.

[joel_k_jones]

Adding Hal Finkel and Tim Northover as reviewers as I discussed with Chandler in IRC.

I don't think that this is such a major change, at least in terms of possibility of breaking anything without notice or in terms of introducing a new kind of infrastructure.

[chandlerc]

Adding new support for a vector math library seems worth getting reviewed / approved by someone reasonably deeply familiar w/ our vectorization infra.

There are a bunch of possibilities here. I suggest Hal because I know he has worked specifically with SLEEF and vectorization before. But there are plenty of others. You can look at folks who have touched lib/Transforms/Vectorize if you'd like to see others.

I'm not trying to slow anything down, and maybe the code change is super simple and this will just be a quick stamp. But it seems quite useful to double check with folks that might have context on this area, etc.

[rengolin]

Overall, this change looks ok to me. The tests look good, too.

However, I have a question that I have asked before and don't seem to see a solution here: how do you deal with evolving library support?

Today, we don't have SVE, so we don't generate SVE bindings, all fine. Say, next year you upstream SVE bindings in SLEEF and then change LLVM to emit those too.

The objects compiled with LLVM next year won't link against SLEEF compiled today. You can say "compile SLEEF again", but Linux distros have different schedule for different packages, especially when they have different maintainers.

They cope with the main libraries by validating and releasing a clear set of compatible libraries, and the GNU community participates heavily in that process by testing and patching gcc, glibc, binutils, etc.

I don't think SLEEF maintainers will have the same stamina, so it's possible that an update in either distro or SLEEF breaks things, they reject the new package, but now, LLVM is broken, too.

This is not an LLVM problem, per se, but by supporting SLEEF, you'll be transferring the problem to the LLVM community, as we'll be the ones receiving the complaints first.

This also happens in LLVM proper, when we have just released a new version and people haven't updated SLEEF yet, and we receive a burst of complaints.

The real question is: is the SLEEF community ready to take on the task of fixing compatibility / linkage issues as soon as they occur? The alternative is what we do with other broken code: deletion. I want to avoid that.

The other question, inline, is about the argument name. We really want to use the same as existing one, for the sake of least surprise.

Some additional random comments...

Adding the triple here shows the real dependency that was never added, because SVML assumes Intel hardware, so no problems there.

The discussion on how to simplify the pattern matching is valid, but at this stage, I think we can keep the existing mechanism.

I agree that this is bound to increase with SVE but I'm not sure writing a whole new table-gen back end just for this is worthy.

The code would be slightly longer on the src side (instead of build side) but it's well organised and clear, so no big deal.

I tend to agree with @steleman that auto-generating as a loop+replace can be tricky if there are support gaps in the library.

But further cleanup changes to this patch could prove me wrong. :)

cheers,
--renato

llvm/trunk/lib/Analysis/TargetLibraryInfo.cpp

29

A quick google search told me GCC uses -fveclib=SLEEF. We don't want to be different or that would create big problems in build systems. It seems, at least right now, there's only one ABI, so there's no point in premature classification.

[shibatch]

Hello Renato,

Today, we don't have SVE, so we don't generate SVE bindings, all fine. Say, next year you upstream SVE bindings in SLEEF and then change LLVM to emit those too.

SLEEF already has support for SVE.

The objects compiled with LLVM next year won't link against SLEEF compiled today. You can say "compile SLEEF again", but Linux distros have different schedule for different packages, especially when they have different maintainers.

This is also a problem in SVML.
My understanding is that this patch is kind of tentative, which will be used until the OpenMP patch ( https://reviews.llvm.org/D54412 ) will be merged.
After that, the include file that is contained in the vector math library is referred by the compiler to see which math functions are available. So this problem won't occur.

[rengolin]

In D53927#1326908, @shibatch wrote:

Today, we don't have SVE, so we don't generate SVE bindings, all fine. Say, next year you upstream SVE bindings in SLEEF and then change LLVM to emit those too.

SLEEF already has support for SVE.

I see, but the point is still the same.

Implementation will evolve, new functions will be added, and testing all variants will still lead to the same maintenance issues.

The bottom line is: this is a long term commitment, and I want to make sure everyone involved is aware of that.

This is also a problem in SVML.

Yes, and Intel has been very forthcoming in maintaining their code.

My understanding is that this patch is kind of tentative, which will be used until the OpenMP patch ( https://reviews.llvm.org/D54412 ) will be merged.
After that, the include file that is contained in the vector math library is referred by the compiler to see which math functions are available. So this problem won't occur.

In that case, I propose we either make this experimental, or we wait until that patch lands and we can change this one accordingly.

I am worried to walk into a situation where we generate OpenMP code where no pragmas were written, as we don't (and shouldn't) include libomp by default.

I know this patch doesn't do that, but some of the comments here and there make allusion to that fact, and we should be careful how we proceed.

I would welcome @hfinkel's opinion.

[steleman]

Hi Renato,

My answers inline.

In D53927#1326846, @rengolin wrote:

Overall, this change looks ok to me. The tests look good, too.

However, I have a question that I have asked before and don't seem to see a solution here: how do you deal with evolving library support?

I'm not. I'm punting, simply because I can't control SLEEF's potential ABI changes from LLVM. :-) And absent a formal definiton of SLEEF's mangling and ABI, I don't see how this problem can be dealt with in a definitive way.

I am taking it on faith that SLEEF won't keep changing their mangling and ABI.

AFAICS, SLEEF isn't included in any Linux distro. I can't find any reference to it in RH/CentOS/Fedora, or at rpmfind. Or Ubuntu or SuSE.

Which means that those environments who will want to use SLEEF will have to build their own.

The upside is that SLEEF is likely of interest only to the rarefied world of HPC/Super-computing. Those environments can definitely build their own SLEEF and maintain their own binary copy.

Today, we don't have SVE, so we don't generate SVE bindings, all fine. Say, next year you upstream SVE bindings in SLEEF and then change LLVM to emit those too.

The objects compiled with LLVM next year won't link against SLEEF compiled today. You can say "compile SLEEF again", but Linux distros have different schedule for different packages, especially when they have different maintainers.

The objects compiled with LLVM next year will still link with SLEEF from today. They just won't be binding to the SVE version of SLEEF. They'll keep binding to the non-SVE SLEEF.

Meaning, ThunderX2/T99 - for example - will never use SVE, because SVE is not available in T99 silicon.

So, future SVE becomes a source of problems only for AArch64 silicon that has SVE support. For those CPU's it becomes a compile-time/link-time choice.

If they want to use a SVE-enabled SLEEF, they'll have to build a SVE-enabled SLEEF to link against. And I suspect they'll have to say '-march=armv8.2+sve' - or something like that - on compile-line.

Those environments can always use two different versions of SLEEF - one with SVE, the other one without, just by using different shared object names and different binding SONAMEs: one for SVE, the other one for not SVE.

clang doesn't auto-magically pass -lsleefgnuabi on link-line, so that's left up to the consumer.

They cope with the main libraries by validating and releasing a clear set of compatible libraries, and the GNU community participates heavily in that process by testing and patching gcc, glibc, binutils, etc.

I don't think SLEEF maintainers will have the same stamina, so it's possible that an update in either distro or SLEEF breaks things, they reject the new package, but now, LLVM is broken, too.

This is not an LLVM problem, per se, but by supporting SLEEF, you'll be transferring the problem to the LLVM community, as we'll be the ones receiving the complaints first.

This also happens in LLVM proper, when we have just released a new version and people haven't updated SLEEF yet, and we receive a burst of complaints.

The real question is: is the SLEEF community ready to take on the task of fixing compatibility / linkage issues as soon as they occur? The alternative is what we do with other broken code: deletion. I want to avoid that.

The other question, inline, is about the argument name. We really want to use the same as existing one, for the sake of least surprise.

I agree with you. ;-) I will change it back to fveclib=SLEEF. ;-)

Some additional random comments...

Adding the triple here shows the real dependency that was never added, because SVML assumes Intel hardware, so no problems there.

The discussion on how to simplify the pattern matching is valid, but at this stage, I think we can keep the existing mechanism.

I agree that this is bound to increase with SVE but I'm not sure writing a whole new table-gen back end just for this is worthy.

The code would be slightly longer on the src side (instead of build side) but it's well organised and clear, so no big deal.

I tend to agree with @steleman that auto-generating as a loop+replace can be tricky if there are support gaps in the library.

But further cleanup changes to this patch could prove me wrong. :)

Now about the -fveclib=OpenMP + #pragma omp simd story:

I am fully aware of the OpenMP SIMD Plan[tm] and of the not-really-a-RFC RFC at https://reviews.llvm.org/D54412.

Here are my thoughts on that - i've already expressed some vague version of those thoughts earlier in this changeset:

1. If the -fveclib=OpenMP feature requires changes to existing code to work -- namely inserting #pragma omp simd directives in existing production code -- then it's a non-starter by design. There are many environments in HPC/Super-computing where making changes to existing production code requires an Act Of God and half a million $USD in code re-certification costs.

2. Not every program wants to become dependent on OpenMP. Some programs really really don't want OpenMP. There should exist a mechanism for those kinds of programs to vectorize libm functions without involving OpenMP.

3. The story with the "magic" header file. Some environments might consider this "magic" header file a sneaky way of implementing [1] -- namely production source code changes. A header file that wasn't there before is now automagically included, and there's nothing they can do to not include it. Boom! - source code change. Happy re-certification planning.

4. I am very happy about the thought of OpenMP being able - at some point in the future - to vectorize things. Programs that already use OpenMP will happily take advantage of this future feature. But, based on my understanding of the current OpenMP reality, this new OpenMP feature won't be available for a while. It's certainly not available now. So, how do we accomplish libm vectorization in the meanwhile?

[shibatch]

I am taking it on faith that SLEEF won't keep changing their mangling and ABI.

The name mangling rule in SLEEF is basically the vector function ABI for AArch64.
The functions for Intel ISA conform to the corresponding ABI for x86.

https://developer.arm.com/products/software-development-tools/hpc/arm-compiler-for-hpc/vector-function-abi

AFAICS, SLEEF isn't included in any Linux distro. I can't find any reference to it in RH/CentOS/Fedora, or at rpmfind. Or Ubuntu or SuSE.

Debian now includes SLEEF as a package. https://tracker.debian.org/pkg/sleef

Which means that those environments who will want to use SLEEF will have to build their own.
The upside is that SLEEF is likely of interest only to the rarefied world of HPC/Super-computing. Those environments can definitely build their own SLEEF and maintain their own binary copy.

Several open source projects including PyTorch have already adopted SLEEF.

[steleman]

In D53927#1327040, @shibatch wrote:

I am taking it on faith that SLEEF won't keep changing their mangling and ABI.

The name mangling rule in SLEEF is basically the vector function ABI for AArch64.
The functions for Intel ISA conform to the corresponding ABI for x86.

https://developer.arm.com/products/software-development-tools/hpc/arm-compiler-for-hpc/vector-function-abi

I know. :-)

Speaking from my own experience: I have never met an ABI that didn't have to change after it was adopted and ratified as a PDF. ;-)

So yes, the AArch64 Vector ABI is probably stable, with a 99.3% level of confidence.

[rengolin]

In D53927#1327020, @steleman wrote:

I am taking it on faith that SLEEF won't keep changing their mangling and ABI.
AFAICS, SLEEF isn't included in any Linux distro. I can't find any reference to it in RH/CentOS/Fedora, or at rpmfind. Or Ubuntu or SuSE.

As @shibatch said, they use Intel's and Arm's ABIs, so that should be fine. But also, that this is not just a distro issue, but bundling into other libraries.

Which means that those environments who will want to use SLEEF will have to build their own.
The upside is that SLEEF is likely of interest only to the rarefied world of HPC/Super-computing. Those environments can definitely build their own SLEEF and maintain their own binary copy.

The signal I'm getting from all labs (in US, Europe and Asia) is that they really *don't* want to maintain their own binaries. They just cope with it because we don't do a good job at the tools/packaging level.

I'm not willing to add more chaos into the mix.

Those environments can always use two different versions of SLEEF - one with SVE, the other one without, just by using different shared object names and different binding SONAMEs: one for SVE, the other one for not SVE.
clang doesn't auto-magically pass -lsleefgnuabi on link-line, so that's left up to the consumer.

SVE was just an example of completely missing options, but new functions and potentially renames can raise much harder issues to fix. We had such issues with glibc, libgcc, compiler-rt in the past and it takes awfully long to go away.

The link flag issue is minor, as people passing -fveclib=sleef to CFLAGS will also pass -lsleef to LDFLAGS, but that doesn't solve the versioning problem.

4. I am very happy about the thought of OpenMP being able - at some point in the future - to vectorize things. Programs that already use OpenMP will happily take advantage of this future feature. But, based on my understanding of the current OpenMP reality, this new OpenMP feature won't be available for a while. It's certainly not available now. So, how do we accomplish libm vectorization in the meanwhile?

The alternative is to call it experimental and take slow steps. This is mostly a political change, not technical, so it won't involve a lot of red tape. As I said before, the code looks fine to me.

What this normally means is:

• Commit messages and code comments have "experimental" in them
• A small paragraph of comments on the code explaining the problems with the current approach
• Developer availability to move code around or re-write in the near future

The bad side is that:

• Experimental features have less priority than other changes
• Other developers are far less likely to keep that part of the code in good shape
• If this code breaks some other, "official" code, it could be reverted or changed
• If no one takes the maintenance role, the code will bit rot and be eventually removed

A code moves from experimental to official when:

• There are enough users / developers maintaining it for months / year
• There are enough validation efforts (buildbots, new tests, distro use)
• The code, and its tests, don't break other unrelated changes often enough
• Developers usually respond promptly to such breakages

As I said, I still want to hear from Hal/Tim, but I'd be happy to call this experimental, as long as we're agreeing that there is a commitment to look after this code (and all ramifications, such as for OMP).

The commitment doesn't need to be from you directly, but from enough developers to make this worthwhile.

I just want to make clear I do want SLEEF support in LLVM, but we need to make it right from the start.

cheers,
--renato

[steleman]

In D53927#1327091, @rengolin wrote:

Which means that those environments who will want to use SLEEF will have to build their own.
The upside is that SLEEF is likely of interest only to the rarefied world of HPC/Super-computing. Those environments can definitely build their own SLEEF and maintain their own binary copy.

The signal I'm getting from all labs (in US, Europe and Asia) is that they really *don't* want to maintain their own binaries. They just cope with it because we don't do a good job at the tools/packaging level.

I'm not willing to add more chaos into the mix.

Then they are in trouble with a hypothetical distro SLEEF.

That's because no commercial distro - if they ever decide to include SLEEF - will compile SLEEF for the maximum CPU feature-set possible. They do the exact opposite - they compile it for the lowest possible common denominator.

For example, on AArch64, you won't get distro libraries compiled specifically for -mcpu=thunderx2t99. Or on x86_64 you'll never get -mavx512ifma, for example.

[shibatch]

That's because no commercial distro - if they ever decide to include SLEEF - will compile SLEEF for the maximum CPU feature-set possible. They do the exact opposite - they compile it for the lowest possible common denominator.

For example, on AArch64, you won't get distro libraries compiled specifically for -mcpu=thunderx2t99. Or on x86_64 you'll never get -mavx512ifma, for example.

Many applications and libraries including SLEEF use dispatchers to choose a code with which the available vector extension can be utilized.

[rengolin]

In D53927#1327204, @steleman wrote:

Then they are in trouble with a hypothetical distro SLEEF.

That's because no commercial distro - if they ever decide to include SLEEF - will compile SLEEF for the maximum CPU feature-set possible. They do the exact opposite - they compile it for the lowest possible common denominator.

For example, on AArch64, you won't get distro libraries compiled specifically for -mcpu=thunderx2t99. Or on x86_64 you'll never get -mavx512ifma, for example.

A bit off topic, but there are two solutions to that problem:

1. Dynamic Arch

The likes of IFUNC and virtual dispatch, which compiles all variants, but chooses the right one at run time. OpenBLAS uses this, for example.

This is easy on the compiler, which knows all variables at build time and no user are harmed in the process. But it's hard to maintain and has additional runtime costs.

2. Modules / Overlays

LMod, EasyBuild, Spack can choose the appropriate libraries at build / run time as needed.

That's not so easy on the compiler, especially if the modules are shared libraries, linked at runtime. It gets worse if users build half local and half use the modules.

Both are preferable to having every cluster user re-compile the whole stack every time, or having cluster admins curate a random number of random builds at random places (which is usually the end result).

That is also assuming the user knows how to best compile the libraries, better than the developers or packagers, which is usually not true.

For example, just adding march / mtune to OpenBLAS doesn't do much good and only improves a fraction of forcing the right arch via TARGET=ARCH, which then will select the right flags anyway.

[fpetrogalli]

llvm/trunk/lib/Analysis/TargetLibraryInfo.cpp

29

Hi @rengolin - where did you find this option? I am struggling to find it in the gcc code base and in the gcc developer mailing lists. GCC uses -mveclibabi, but it is only for x86.

[rengolin]

llvm/trunk/lib/Analysis/TargetLibraryInfo.cpp

29

Hum, now I'm struggling to find it, too. I may have hit a sweet spot between sleef from clang and mveclibabi from gcc and assumed it worked.

This page [1] tells me only svml and acml are supported, so I stand corrected. However, if/when gcc supports it, mveclibabi=sleefgnuabi will be very weird. So I still prefer sleef instead of sleefgnuabi.

Also, why do we use fveclib and not mveclibabi? This isn't really helpful for build systems...

[1] https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html

[fpetrogalli]

llvm/trunk/lib/Analysis/TargetLibraryInfo.cpp

29

I proposed to use -whateveroption=SLEEFGNUABI instead of -whateveroption=SLEEF because SLEEF is produced in two version: libsleef.so and libsleefgnuabi.so. The two libraries have different signatures and naming conventions. The GNUABI version is compatible with the names of the Vector Function ABI of x86 and AArch64, and with the signature of the functions in glibc, which makes it a superior choice as it adheres to standards. That is why I think this patch is making the right choice in choosing the GNUABI names.

Now, although we might have personal preferences on how to name things (I agree with you that mveclibabi|fveclib=sleefgnuabi is not pretty), I think the ultimate criteria for our choice is correctness and clarity. Having -fveclib=SLEEF when the compiler will be really using libsleefgnuabi.so would just be confusing, especially if some day we decide that we want to support also libsleef.so (for example, some workloads might prefer the sincos signature in there over the one provided in libsleefgnuabi).

[rengolin]

llvm/trunk/lib/Analysis/TargetLibraryInfo.cpp

29

Is there any realistic expectation that we should support two ABIs for the same library with one being incompatible with the other ABIs (x86, Arm, Gnu)?

If these are just aliases, we don't even need to have two SO objects, and just add the aliases and wrappers to the main SO. We do that in Compiler-RT because it would be a nightmare otherwise.

[shibatch]

A header file that wasn't there before is now automagically included, and there's nothing they can do to not include it. Boom! - source code change. Happy re-certification planning.

@steleman Could you explain a little bit about when a re-certification is required?
Is it required when a standard header file(e.g. math.h) is changed?

gcc/glibc is doing a similar thing. In math-vector.h, which is included from math.h, there is declaration like below.

# if defined _OPENMP && _OPENMP >= 201307
/* OpenMP case.  */
#  define __DECL_SIMD_x86_64 _Pragma ("omp declare simd notinbranch")
# elif __GNUC_PREREQ (6,0)
/* W/o OpenMP use GCC 6.* __attribute__ ((__simd__)).  */
#  define __DECL_SIMD_x86_64 __attribute__ ((__simd__ ("notinbranch")))
# endif

# ifdef __DECL_SIMD_x86_64
#  undef __DECL_SIMD_cos
#  define __DECL_SIMD_cos __DECL_SIMD_x86_64
...

[steleman]

In D53927#1327983, @shibatch wrote:

A header file that wasn't there before is now automagically included, and there's nothing they can do to not include it. Boom! - source code change. Happy re-certification planning.

@steleman Could you explain a little bit about when a re-certification is required?
Is it required when a standard header file(e.g. math.h) is changed?

Yes. Distro upgrades must be (and are) certified. Once they're certified, they stay immutable for a very long time.

gcc/glibc is doing a similar thing. In math-vector.h, which is included from math.h, there is declaration like below.

# if defined _OPENMP && _OPENMP >= 201307
/* OpenMP case.  */
#  define __DECL_SIMD_x86_64 _Pragma ("omp declare simd notinbranch")
# elif __GNUC_PREREQ (6,0)
/* W/o OpenMP use GCC 6.* __attribute__ ((__simd__)).  */
#  define __DECL_SIMD_x86_64 __attribute__ ((__simd__ ("notinbranch")))
# endif

# ifdef __DECL_SIMD_x86_64
#  undef __DECL_SIMD_cos
#  define __DECL_SIMD_cos __DECL_SIMD_x86_64
...

So the non-openmp case doesn't change code, and doesn't introduce external dependencies that may not have been present in the initial implementation. It just adds a function attribute. Which, in GCC-land, translates to a function call to a (possibly inlined) vectorized function.

I am not, by any means, suggesting that there is something inherently wrong with using OpenMP for SIMD libm vectorization. There is absolutely nothing wrong with that approach.

However, OpenMP cannot be the only method of obtaining vectorization of libm functions, for reasons already mentioned. There has to be an alternate method, one that does not involve importing OpenMP pragmas and interfaces, and their implicit trigger of code changes. GCC provides that alternative with attribute((simd)).

[shibatch]

The RFC for reimplementing -fveclib with OpenMP is titled "OpenMP" because the corresponding code for ARM HPC compiler requires OpenMP.
However, the proposal in RFC does not require OpenMP, as quoted below.

The new proposed #pragma directive are:

1. #pragma veclib declare simd.
2. #pragma veclib declare variant.

Both directive follows the syntax of the declare simd and the
declare variant directives of OpenMP, with the exception that
declare variant is used only for the simd context.

[steleman]

In D53927#1328001, @shibatch wrote:

The RFC for reimplementing -fveclib with OpenMP is titled "OpenMP" because the corresponding code for ARM HPC compiler requires OpenMP.
However, the proposal in RFC does not require OpenMP, as quoted below.

The new proposed #pragma directive are:

1. #pragma veclib declare simd.
2. #pragma veclib declare variant.

Both directive follows the syntax of the declare simd and the
declare variant directives of OpenMP, with the exception that
declare variant is used only for the simd context.

1. Where, when, how and by whom are these #pragmas inserted? Are they silently inserted by clang? Do they have to be manually inserted in existing code?

2. What is the effect of these #pragmas? The ARM closed-source commercial compiler requires OpenMP. How will these #pragmas be implemented in the open-source clang? Will they import OpenMP interfaces in the open-source clang as well?

3. If importing OpenMP interfaces is not triggered by the insertion of these #pragmas, why is the argument to -fveclib called 'openmp'? I would find it baffling to have to type -fveclib=openmp on compile-line, but ending up not using OpenMP.

[shibatch]

Where, when, how and by whom are these #pragmas inserted? Are they silently inserted by clang? Do they have to be manually inserted in existing code?
What is the effect of these #pragmas? The ARM closed-source commercial compiler requires OpenMP. How will these #pragmas be implemented in the open-source clang? Will they import OpenMP interfaces in the open-source clang as well?
If importing OpenMP interfaces is not triggered by the insertion of these #pragmas, why is the argument to -fveclib called 'openmp'? I would find it baffling to have to type -fveclib=openmp on compile-line, but ending up not using OpenMP.

I think you should post these questions as a reply to the RFC.

[rengolin]

In D53927#1328003, @shibatch wrote:

Where, when, how and by whom are these #pragmas inserted? Are they silently inserted by clang? Do they have to be manually inserted in existing code?
What is the effect of these #pragmas? The ARM closed-source commercial compiler requires OpenMP. How will these #pragmas be implemented in the open-source clang? Will they import OpenMP interfaces in the open-source clang as well?
If importing OpenMP interfaces is not triggered by the insertion of these #pragmas, why is the argument to -fveclib called 'openmp'? I would find it baffling to have to type -fveclib=openmp on compile-line, but ending up not using OpenMP.

I think you should post these questions as a reply to the RFC.

Yes, but bear in mind that the proprietary compiler options will always have less weight than other OSS compilers.

Putting it bluntly, if it doesn't work in GCC, people will not use it. Regardless of what any proprietary compiler does, based or not in LLVM.

We already have some clang pragmas for vectorisation, we don't need yet another syntax, and if we do, we should use OpenMPs.

Using OMP pragmas doesn't necessarily means we need to link with OMP libraries, and that's a special case for OMP SIMD.

Any new pragma MUST be done though an RFC process in the llvm-dev list, that's for sure.

[steleman]

Updated version of the patch/changeset, as per comments from Renato:

• veclib argument reverted back to SLEEF.
• Corresponding enum names reverted back to SLEEF.

• New benchmark comparing SLEEF performance when built with GCC 8.2 vs. clang ToT from 12/15/2018:

https://docs.google.com/spreadsheets/d/1QgdJjgCTFA9I2_5MvJU0uR4JT4knoW5xu-ujxszRbd0/edit?usp=sharing

[rengolin]

Thanks! I've updated the reviewers on the Clang review (D53928).

[steleman]

Ping!

Yes, I realize people were on holidays. :-)

Is this OK to go in (the second time around)?

Is anyone else interested in commenting?

[rengolin]

Perhaps best to ping Clang's commit, as that's the review that is blocking this one.

As long as we agree on the behaviour in Clang, this patch should be fine.

[steleman]

In D53927#1352433, @rengolin wrote:

Perhaps best to ping Clang's commit, as that's the review that is blocking this one.

As long as we agree on the behaviour in Clang, this patch should be fine.

OK, so I ping'ed Clang and no-one is biting. :-)

So, what are the next steps for getting this into LLVM?

[rengolin]

Hal approved the Clang side, so this one looks good, too.

Please commit this one first, then Clang's, to make sure fast bots and bisects don't break.

Thanks!

[steleman]

Thank you Hal and Renato.

[steleman]

This is a significant update to the original version of the SLEEF
changeset.

While testing LLVM+SLEEF on AArch64, I discovered that, for programs
such as the following simple test case:

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <time.h>

int main()
{
  double D;
  double R;

  srand48(time(0));

  D = drand48();
  R = tgamma(D);
  (void) fprintf(stderr, "tgamma: D=%lf R=%lf\n", D, R);

  D = drand48();
  R = acos(D);
  (void) fprintf(stderr, "acos: D=%lf R=%lf\n", D, R);

  D = drand48();
  R = acosh(D);
  (void) fprintf(stderr, "acosh: D=%lf R=%lf\n", D, R);

  D = drand48();
  R = asin(D);
  (void) fprintf(stderr, "asin: D=%lf R=%lf\n", D, R);

  return 0;
}

Compile with:

./clang -O3 -mcpu=thunderx2t99 -fvectorize -fveclib=SLEEF -ffast-math -ffp-contract=fast -c t.c -o t.o

compilation will fail in LLVM:

fatal error: error in backend: Cannot select: intrinsic %llvm.asin
clang-8: error: clang frontend command failed with exit code 70 (use -v to see invocation)
[ ... etc etc ... ]

This failure is triggered by -ffast-math -ffp-contract=fast. Without
these flags, the fatal error does not occur.

This only happens when the newly-enabled intrinsics (tgamma, acos,
acosh, asin, etc) are called on a scalar -- as in the test program above.

When called on a vector, with or without -ffast-math -ffp-contract=fast,
these new intrinsics always succeed.

This updated version of the patch addresses this problem.

I have also added a second test case for SLEEF AArch64 which tests for

%call = call fast float @acosf(float %conv)

I realize that this patch has grown in size significantly, and that
it may be a candidate for splitting it into smaller parts.

The problem is, I do not quite see how it can be split into several
smaller parts, simply because without this entire patch, the new
intrinsics will cause a fatal error in the LLVM backend, as shown above.

The patch is based on LLVM ToT from 02/06/2019.

[rengolin]

Right, this is a completely different patch and it seems that the original proposal hadn't been tested thoroughly.

It did feel as the initial patch was far too simple for what it enabled, but I thought this was due to implementations in the intrinsic selection that I didn't follow for the past year or two.

By looking over this new patch, it feels as though most changes are mechanical, but I'm afraid I don't have enough familiarity with the code base any more to be sure without further testing.

Given that this review is "approved" and "closed" and then reverted, and that the implementation is now totally different than the original approved patch, I'll reopen this one and request revision again.

Also, the Clang part (D53928) seems to have changed significantly after approved, so that, too, would need revisiting. I have updated that review to look at it again.

[rengolin]

Hi,

I have a few comments inline, mostly harmless, as the change is almost entirely mechanical and makes sense.

I'm happy with the change as is, no need to split (as I also can't see how), after we address all the comments.

That'd also give some time for other people to review the new changes.

cheers,
--renato

include/llvm/Analysis/TargetLibraryInfo.def

315 ↗

(On Diff #185576)

Probably a silly question but, why "tgamma" when the function name is "gamma"?

include/llvm/IR/RuntimeLibcalls.def

258 ↗

(On Diff #185576)

Looks like someone should one day look into generating this file, an OPcode section and TargetLibraryInfo from the intrinsics table-gen... Not now, though.

lib/Analysis/TargetLibraryInfo.cpp

368 ↗

(On Diff #185576)

This would probably fail in many non-Arm64 arches, too.

Best to restrict to what this was meant for, which I'm guessing is x86.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

1108 ↗

(On Diff #185576)

Please clean up all "new intrinsics" comments, as they're not necessary for the patch.

I imagine they were helpful while developing... :)

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

4952 ↗

(On Diff #185576)

I think we can have these in the same place as the others, but with an "experimental" comment before.

The main idea behind the experimental tag is so that people don't rely on it until it's complete, but once it is, they can.

So if we create them in different places, we'd have to move them in once they're no longer experimental and that would be a pain.

I'm not expecting this to be in experimental for too long and am hoping to get this official by the next release (july-ish 2019), so, let's keep the experimental status on comments only and then remove the comments when everything is working as expected.

test/Transforms/LoopVectorize/AArch64/sleef-calls-aarch64-fast.ll

1 ↗

(On Diff #185576)

We don't generally recommend O3 in tests anyway, as this introduces test noise when the passes in O3 change (add/del/move around).

3 ↗

(On Diff #185576)

does fp-contract=fast really make a difference for these transformations?

6 ↗

(On Diff #185576)

is this really specific to Arm64? At least the generation could be done for any target, if the library supports it or not is not relevant for this test, no?

test/Transforms/LoopVectorize/AArch64/sleef-calls-aarch64.ll

3 ↗

(On Diff #185576)

What's the difference between these two test files?

[steleman]

Hi Renato,

Thank you very much for the comments.

I will create a new changeset with all your suggestions, and then re-submit.

[bryanpkc]

include/llvm/Analysis/TargetLibraryInfo.def

315 ↗

(On Diff #185576)

This didn't work in my test build. The function names must be in alphabetical order, so __gamma_r_finite etc. cannot come after __isoc99_scanf.

[rengolin]

include/llvm/Analysis/TargetLibraryInfo.def

315 ↗

(On Diff #185576)

Better move to alphabetical order, then. Would that remove the need to call it "tgamma"?

[shibatch]

tgamma stands for "true gamma", and it is the function name specified in the ANSI C standard.

[rengolin]

In D53927#1397785, @shibatch wrote:

tgamma stands for "true gamma", and it is the function name specified in the ANSI C standard.

Hm, if this is the same as tgamma, why is it __gamma_r_finite and not __tgamma_r_finite?

[steleman]

In D53927#1397790, @rengolin wrote:

In D53927#1397785, @shibatch wrote:

tgamma stands for "true gamma", and it is the function name specified in the ANSI C standard.

Hm, if this is the same as tgamma, why is it __gamma_r_finite and not __tgamma_r_finite?

Because __tgamma_r_finite doesn't exist, at least on Linux. :-)

See /usr/include/bits/math-finite.h.

[rengolin]

In D53927#1397870, @steleman wrote:

Because __tgamma_r_finite doesn't exist, at least on Linux. :-)

See /usr/include/bits/math-finite.h.

Ah! Sounds good. :)

Thanks!

[steleman]

Added inline comments/answers to Renato's questions.

lib/Analysis/TargetLibraryInfo.cpp

368 ↗

(On Diff #185576)

I am a bit worried about this one. The original set exp10|exp10f|exp10l as unavailable on all arches. I originally enabled these functions for AArch64 only.

If we restrict this to x86|x86_64 only, then these functions will become available
on MIPS, PPC, PPC64, etc. I have no way of testing these ISA's to determine if they're broken or not.

So I disabled these functions for x86|x86_64 only. Let me know if you'd like to revert this to the original, which was to enable them on AArch64 only.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

4952 ↗

(On Diff #185576)

Renamed this function to expandNonVectorIntrinsics.

test/Transforms/LoopVectorize/AArch64/sleef-calls-aarch64-fast.ll

3 ↗

(On Diff #185576)

It does not. Which is good. :-) That's the point of the test - the transformations should stay identical, and pass, with or without -fp-contract=fast.

6 ↗

(On Diff #185576)

SLEEF is only currently enabled for AArch64 in TargetLibraryInfo. So, for any other ISA, all the tests will fail, because the transformations won't happen.

Plus, for any ISA other than AArch64, the SLEEF function name mangling will be different than on AArch64. Which means this particular test - as written for AArch64 - can't be used on any other ISA; all the CHECKs will fail.

test/Transforms/LoopVectorize/AArch64/sleef-calls-aarch64.ll

3 ↗

(On Diff #185576)

sleef-calls-aarch64-fast.ll calls

%call = call fast <type> <function-name>(<argument>)

sleef-calls-aarch64.ll calls

%call = tail call <type> <function-name>(<argument>)

[steleman]

Updated changeset to reflect comments/questions from Renato.

This new changeset is based on LLVM ToT from 02/14/2019.

[rengolin]

Thanks for your answers, they make sense. Other than the mistake I made on disabling exp calls (you were right), looks fine.

lib/Analysis/TargetLibraryInfo.cpp

368 ↗

(On Diff #185576)

My bad, I read it the other way round. Your original patch was correct.

[steleman]

Reverted change in TargetLibraryInfo.cpp re: exp10|exp10f|exp10l on Linux with GLIBC.

These functions are now allowed on AArch64 or AArch64_be only.

[rengolin]

Thanks Stephan, now looks good.

Did you see the comment on lgamma coming after isoc99? I wonder if that's reproducible in all systems...

include/llvm/Analysis/TargetLibraryInfo.def

315 ↗

(On Diff #185576)

@bryanpkc What's your environment?

@steleman Any idea why you don't see that on your builds?

[steleman]

In D53927#1399263, @rengolin wrote:

Thanks Stephan, now looks good.

Did you see the comment on lgamma coming after isoc99? I wonder if that's reproducible in all systems...

Yup I saw it. But I don't get any errors here (Ubuntu 16.04 / Ubuntu 18.something) AArch64.

And it's not clear to me how that sorted alphabetical order is supposed to work. There's a bunch of other functions that come after lgamma/tgamma that aren't sorted.

For example:

/// char * __strtok_r(char *s, const char *delim, char **save_ptr);
TLI_DEFINE_ENUM_INTERNAL(dunder_strtok_r)
TLI_DEFINE_STRING_INTERNAL("__strtok_r")
/// int abs(int j);
TLI_DEFINE_ENUM_INTERNAL(abs)
TLI_DEFINE_STRING_INTERNAL("abs")
/// int access(const char *path, int amode);
TLI_DEFINE_ENUM_INTERNAL(access)
TLI_DEFINE_STRING_INTERNAL("access")

[steleman]

In D53927#1399263, @rengolin wrote:

Thanks Stephan, now looks good.

Did you see the comment on lgamma coming after isoc99? I wonder if that's reproducible in all systems...

Just did a rebuild after moving lgamma after isoc99. No warnings, no errors, and no difference in my test results ...

[rengolin]

In D53927#1399306, @steleman wrote:

Just did a rebuild after moving lgamma after isoc99. No warnings, no errors, and no difference in my test results ...

It seems like a build issue (or a different library issue), that we can fix after commit.

LGTM, thanks!

[steleman]

In D53927#1399318, @rengolin wrote:

In D53927#1399306, @steleman wrote:

Just did a rebuild after moving lgamma after isoc99. No warnings, no errors, and no difference in my test results ...

It seems like a build issue (or a different library issue), that we can fix after commit.

LGTM, thanks!

Thank you Sir! :-)

[bryanpkc]

In D53927#1399283, @steleman wrote:

In D53927#1399263, @rengolin wrote:

Thanks Stephan, now looks good.

Did you see the comment on lgamma coming after isoc99? I wonder if that's reproducible in all systems...

Yup I saw it. But I don't get any errors here (Ubuntu 16.04 / Ubuntu 18.something) AArch64.

And it's not clear to me how that sorted alphabetical order is supposed to work. There's a bunch of other functions that come after lgamma/tgamma that aren't sorted.

For example:

/// char * __strtok_r(char *s, const char *delim, char **save_ptr);
TLI_DEFINE_ENUM_INTERNAL(dunder_strtok_r)
TLI_DEFINE_STRING_INTERNAL("__strtok_r")
/// int abs(int j);
TLI_DEFINE_ENUM_INTERNAL(abs)
TLI_DEFINE_STRING_INTERNAL("abs")
/// int access(const char *path, int amode);
TLI_DEFINE_ENUM_INTERNAL(access)
TLI_DEFINE_STRING_INTERNAL("access")

The build was fine, but when running the compiler, this assertion in lib/Analysis/TargetLibraryInfo.cpp will fail:

static void initialize(TargetLibraryInfoImpl &TLI, const Triple &T,
                       ArrayRef<StringRef> StandardNames) {
  // Verify that the StandardNames array is in alphabetical order.
  assert(std::is_sorted(StandardNames.begin(), StandardNames.end(),
                        [](StringRef LHS, StringRef RHS) {
                          return LHS < RHS;
                        }) &&
         "TargetLibraryInfoImpl function names must be sorted");

You probably don't see it because you are only building in Release mode.

P.S. The examples you listed are sorted (_ is less than a).

[steleman]

In D53927#1400072, @bryanpkc wrote:

In D53927#1399283, @steleman wrote:

In D53927#1399263, @rengolin wrote:

Thanks Stephan, now looks good.

Did you see the comment on lgamma coming after isoc99? I wonder if that's reproducible in all systems...

Yup I saw it. But I don't get any errors here (Ubuntu 16.04 / Ubuntu 18.something) AArch64.

And it's not clear to me how that sorted alphabetical order is supposed to work. There's a bunch of other functions that come after lgamma/tgamma that aren't sorted.

For example:

/// char * __strtok_r(char *s, const char *delim, char **save_ptr);
TLI_DEFINE_ENUM_INTERNAL(dunder_strtok_r)
TLI_DEFINE_STRING_INTERNAL("__strtok_r")
/// int abs(int j);
TLI_DEFINE_ENUM_INTERNAL(abs)
TLI_DEFINE_STRING_INTERNAL("abs")
/// int access(const char *path, int amode);
TLI_DEFINE_ENUM_INTERNAL(access)
TLI_DEFINE_STRING_INTERNAL("access")

The build was fine, but when running the compiler, this assertion in lib/Analysis/TargetLibraryInfo.cpp will fail:

static void initialize(TargetLibraryInfoImpl &TLI, const Triple &T,
                       ArrayRef<StringRef> StandardNames) {
  // Verify that the StandardNames array is in alphabetical order.
  assert(std::is_sorted(StandardNames.begin(), StandardNames.end(),
                        [](StringRef LHS, StringRef RHS) {
                          return LHS < RHS;
                        }) &&
         "TargetLibraryInfoImpl function names must be sorted");

You probably don't see it because you are only building in Release mode.

P.S. The examples you listed are sorted (_ is less than a).

I think the correct fix here is to use an associative container, not an array, and not rely on writing a text file containing sorted character strings.

std::set comes to mind.

[rengolin]

@bryanpkc debug build makes sense, thanks! @steleman, I'd rather just order now and try to rector later. This patch has changed enough tunes already. :-)

[steleman]

Corrected manual sorting in TargetLibraryInfo.def as per comments.

[steleman]

In D53927#1400128, @rengolin wrote:

@bryanpkc debug build makes sense, thanks! @steleman, I'd rather just order now and try to rector later. This patch has changed enough tunes already. :-)

OK, changeset updated. Tested with Debug+Assertions+ExpensiveChecks.

[rengolin]

Perfect, thanks!

[steleman]

One minor update:

${top_srcdir}/unittests/Analysis/TargetLibraryInfoTest.cpp

needs an update to account for lgamma, tgamma, lgamma_r_finite and gamma_r_finite.

[arsenm]

include/llvm/IR/Intrinsics.td

516–533 ↗

(On Diff #187507)

I would split adding the new intrinsics into a separate patch

[steleman]

include/llvm/IR/Intrinsics.td

516–533 ↗

(On Diff #187507)

That won't work.

[echristo]

include/llvm/IR/Intrinsics.td

516–533 ↗

(On Diff #187507)

"Why not" is always a good follow up to that sort of response :)

[steleman]

include/llvm/IR/Intrinsics.td

516–533 ↗

(On Diff #187507)

Because:

• It adds unnecessary complexity to this changeset and to D53928.
• There is no obvious benefit to this added complexity.
• There are many drawbacks to this added complexity.
• Splitting it based on new vs. existing intrinsics seems arbitrary. One might as well ask for a separate changeset for each and every new intrinsic.

And, now it's my turn to ask:

• Why should it be split?
• What is the benefit of splitting it?
• What will be accomplished by splitting it?
• Is increased complexity - with all its drawbacks and pitfalls - a goal?
• Does integrating it as is prevent, or hinder something else?
• Why does this come up now, after this changeset - and its friend D53928 - have been reviewed for 4+ months, and already approved, and re-approved, and there are 21 subscribers to this changeset?

[rengolin]

Hi @steleman, that wasn't a "loaded question", it was an honest one. Let me try to clarify what that means for all parties.

Because:

It adds unnecessary complexity to this changeset and to D53928.
There is no obvious benefit to this added complexity.
There are many drawbacks to this added complexity.

Every split or merge adds complexity, but when submitting patches upstream, you are required to explain not only what you've done, but why you haven't done some other way. This is the dynamics of upstream review and is not meant to push back or demerit, but to make sure all assumptions are shared and there is consensus.

A better answer would be:

• because puling it apart would essentially create two very similar patch sets, possibly the second overwriting behaviour
• because we would need to make sure the intermediate state is sane (to avoid crashes in bisects) and that means testing a state that doesn't make sense
• because the final subset is too small, next to the refactoring needed, and the refactoring is part of "adding" the functionality

in addition to your own:

Splitting it based on new vs. existing intrinsics seems arbitrary. One might as well ask for a separate changeset for each and every new intrinsic.

To answer your questions...

And, now it's my turn to ask:

Why should it be split?
What is the benefit of splitting it?
What will be accomplished by splitting it?
Does integrating it as is prevent, or hinder something else?

Again, that was a standard question in upstream reviews, especially LLVM, as we have a more aggressive commit policy (more trunk breakages, more bisects, more reverts). For that to work, we must introduce the least amount of breakages, even in intermediate states, so they have to be small ans testable.

Furthermore, reverting the refactoring plus the actual change is more complicated than the actual change. If the breakage is in your usage of the refactoring, we can only revert that, leaving the refactoring, and being able to test the refactoring without the new behaviour and with, to find if the original assumption was broken, or if it was some silly mistake on the usage of it.

Is increased complexity - with all its drawbacks and pitfalls - a goal?

Don't assume ill intent, those questions are always worth asking and rarely meant as anything else other than double (or triple) checking consensus.

Why does this come up now, after this changeset - and its friend D53928 - have been reviewed for 4+ months, and already approved, and re-approved, and there are 21 subscribers to this changeset?

Again, another standard upstream behaviour (on Linux, GCC and others as well).

People are copied in all sorts of reviews, code owners even more. If we were to review everything that people copy us in straight away we wouldn't be able to do anything else, including our day jobs.

Matt's and Eric's comments here are very welcome, and I would appreciate their review on our comments about the design decisions. If they agree, we have more hope that this was the right thing all along. If they have more questions, proposals and change requests, we will be relieved that they caught something we didn't before users started relying on it.

It does take some time, but when it goes in, there's a higher probability of being right.

Given that this change (and its Clang counterpart) will add external user-visible behaviour that will have to be maintained for a long time, it's only right that more people chime in when they have time.

I'm glad Matt and Eric took the time before it went in.

cheers,
--renato

[steleman]

In D53927#1411714, @rengolin wrote:

Why does this come up now, after this changeset - and its friend D53928 - have been reviewed for 4+ months, and already approved, and re-approved, and there are 21 subscribers to this changeset?

Again, another standard upstream behaviour (on Linux, GCC and others as well).

People are copied in all sorts of reviews, code owners even more. If we were to review everything that people copy us in straight away we wouldn't be able to do anything else, including our day jobs.

Matt's and Eric's comments here are very welcome, and I would appreciate their review on our comments about the design decisions. If they agree, we have more hope that this was the right thing all along. If they have more questions, proposals and change requests, we will be relieved that they caught something we didn't before users started relying on it.

It does take some time, but when it goes in, there's a higher probability of being right.

Given that this change (and its Clang counterpart) will add external user-visible behaviour that will have to be maintained for a long time, it's only right that more people chime in when they have time.

I'm glad Matt and Eric took the time before it went in.

Renato,

I don't have a problem with various people chiming in and offering comments and ideas.

I am, however, confused, about the concept of open-ended discussion that never reaches a terminal point, even though it would appear that it did.

[joelkevinjones]

I'm not sure what the proposed split would consist of. Would it be

• A patch consisting of only the changes in Intrinsics.td which would be applied first. Wouldn't the changes in SelectionDAG be required as well?
• Everything but the changes in TargetLibraryInfo.cpp etc. for supporting SLEEF and including any related test cases that would be upstreamed 2nd?

Would the second choice be useful by itself? Either in functionality or improved cross-platform testing before the SLEEF support went in?

[rengolin]

In D53927#1412003, @joelkevinjones wrote:

I'm not sure what the proposed split would consist of.

I can see a few different concepts introduced here:

• Lowering sin/cos/tan, etc and all related mechanical changes (large)
• Adding the STRICT variants, with aliases on the lowering switches (medium)
• Changing the log10 behaviour and code movement (small)
• Adding SLEEF at the end, making use of all three patches above (tiny)

The lines between those concepts may be blurred, so it's never a clear cut, and trying to split the patch in that sequence may show problems I haven't consider.

But Git is quite helpful in splitting patches into a branch, and then testing the branch, and then one patch at a time.

From a high point of view, I can't see why any intermediate state would be harmful to builds or tests, given they only add unused code, which will be enabled by the last one.

I also imagine the tests can only work after the last patch is applied, and that's the main reason why I was ok with the patch being a single bulk.

The other reason is because there's a Clang patch, which needs to be applied after all these, and it may cause bisect issues to have too many depending patches across repos.

Hopefully the monorepo will help solve that problem, but we're not there yet.

[joel_k_jones]

I'm trying to understand what log10 behavior got changed. Could someone out where that change is?

[rengolin]

In D53927#1412486, @joel_k_jones wrote:

I'm trying to understand what log10 behavior got changed. Could someone out where that change is?

Sorry, I mixed things up.

I was referring to the log10 order change in the list, but that was done *after* lgamma/tgamma was introduced, to build in the right order (in debug builds), so that was a change on gamma order, not log10.

Just ignore that item.

[Artoria2e5]

This should probably be reopened following commit 7459398a436f67f304c8653f39b8d82109778fec which reverted this change. Not doing that myself considering I may have missed something like a resubmission.

Update: searched the site to make sure, and it looks like there really is none. D53928 is accepted, but obviously can't be merged unless this one is there.

[georgemitenkov]

Hi all!

I was looking throughout LLVM repo to see if there is any SLEEF support here, and this is exactly what I need. Any updates on this patch?

Thanks,
George

[rengolin]

Hi @georgemitenkov,

This was a long time ago, I completely forgot about this.

I was happy with the patch before and I have no reason to hot be happy now, but we need someone else to have a look to make sure we don't revert it back again.

@echristo @arsenm @hfinkel would be good candidates.

That is, of course, if @steleman and @joelkevinjones are still interested in pushing this forward (and its clang counterpart, which now can be atomically committed together due to the monorepo).

The main pending issue I see, reading back the comments, is a split on the intrinsics. I personally don't think it's a good idea, but am not opposed to such a change. It'd be up to the authors and other reviewers to decide on that, though.

As it stands, despite having been approved by me and @fpetrogalli, it would need an additional approval by another reviewer (preferably one of the three mentioned above) to be committed.

Thanks,
--renato

[georgemitenkov]

Thanks @rengolin! I guess in this case we have to see what @steleman and @joelkevinjones think and if they are interested (As well as other reviewers).

It would be really great to have this functionality. With the recent patch (https://reviews.llvm.org/D95373) math intrinsics can be changed into calls for SIMD libraries, and having Sleef there just completes the picture. I am wondering if may be it is better to discuss this on the mailing list? (Sometimes reverted patches are lost for reviewers)

[steleman]

In D53927#2786879, @georgemitenkov wrote:

Thanks @rengolin! I guess in this case we have to see what @steleman and @joelkevinjones think and if they are interested (As well as other reviewers).

It would be really great to have this functionality. With the recent patch (https://reviews.llvm.org/D95373) math intrinsics can be changed into calls for SIMD libraries, and having Sleef there just completes the picture. I am wondering if may be it is better to discuss this on the mailing list? (Sometimes reverted patches are lost for reviewers)

1. I no longer have access to an AArch64 server to re-do and test this patch - I am no longer at Marvell. The patch in its current form will not apply to trunk because it is much too old.

2. I can't use IBM TJWatson's infrastructure or IBM time to work on this patch. It's outside my responsibilities. I am willing to work on it in my free time, as a hobby.

3. If I somehow obtain access to an AArch64 box to re-do this patch (I'm fuzzy on that at the moment), and I spend my own free time doing the work, I would like to know ahead of time that it will not degenerate again into what went on here from 2018 through 2019.

[rengolin]

In D53927#2786927, @steleman wrote:

3. If I somehow obtain access to an AArch64 box to re-do this patch (I'm fuzzy on that at the moment), and I spend my own free time doing the work, I would like to know ahead of time that it will not degenerate again into what went on here from 2018 through 2019.

I'd like to get feedback from @echristo @ararslan @hfinkel and others before we start working on it.

But assuming they're ok with it I can perhaps give you access to an AArch64 machine in my garage, or you can ask @maxim-kuvyrkov for some more beefy server access.

llvm/trunk/include/llvm/Analysis/TargetLibraryInfo.h

 ///
 /// This class constructs tables that hold the target library information and
 /// make it available. However, it is somewhat expensive to compute and only
 /// depends on the triple. So users typically interact with the \c
 /// TargetLibraryInfo wrapper below.
 class TargetLibraryInfoImpl {
   friend class TargetLibraryInfo;
 
+  Triple TT;
   unsigned char AvailableArray[(NumLibFuncs+3)/4];
   llvm::DenseMap<unsigned, std::string> CustomNames;
   static StringRef const StandardNames[NumLibFuncs];
   bool ShouldExtI32Param, ShouldExtI32Return, ShouldSignExtI32Param;
 
   enum AvailabilityState {
     StandardName = 3, // (memset to all ones)
     CustomName = 1,
public:
   /// The vector-functions library defines, which functions are vectorizable
   /// and with which factor. The library can be specified by either frontend,
   /// or a commandline option, and then used by
   /// addVectorizableFunctionsFromVecLib for filling up the tables of
   /// vectorizable functions.
   enum VectorLibrary {
     NoLibrary,  // Don't use any vector library.
     Accelerate, // Use Accelerate framework.
-    SVML        // Intel short vector math library.
+    SVML,       // Intel short vector math library.
+    SLEEFGNUABI // SLEEF - SIMD Library for Evaluating Elementary Functions.
   };
 
   TargetLibraryInfoImpl();
   explicit TargetLibraryInfoImpl(const Triple &T);
 
   // Provide value semantics.
   TargetLibraryInfoImpl(const TargetLibraryInfoImpl &TLI);
   TargetLibraryInfoImpl(TargetLibraryInfoImpl &&TLI);

llvm/trunk/include/llvm/IR/Intrinsics.td

let IntrProperties = [IntrNoMem, IntrSpeculatable] in {
 
   // These functions do not read memory, but are sensitive to the
   // rounding mode. LLVM purposely does not model changes to the FP
   // environment so they can be treated as readnone.
   def int_sqrt : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_powi : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>, llvm_i32_ty]>;
   def int_sin  : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_cos  : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+  def int_acos : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+  def int_asin : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+  def int_atan : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+  def int_atan2 : Intrinsic<[llvm_anyfloat_ty],
+                            [LLVMMatchType<0>, LLVMMatchType<0>]>;
+  def int_tan  : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_pow  : Intrinsic<[llvm_anyfloat_ty],
                            [LLVMMatchType<0>, LLVMMatchType<0>]>;
+
+  def int_cosh : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+  def int_sinh : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+  def int_tanh : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+
+  def int_asinh : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+  def int_acosh : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+  def int_atanh : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+
+  def int_lgamma : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+  def int_tgamma : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+
   def int_log  : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_log10: Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_log2 : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_exp  : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_exp2 : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+  def int_exp10 : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_fabs : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
+
   def int_copysign : Intrinsic<[llvm_anyfloat_ty],
                                [LLVMMatchType<0>, LLVMMatchType<0>]>;
   def int_floor : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_ceil  : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_trunc : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_rint  : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_nearbyint : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;
   def int_round : Intrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>]>;

llvm/trunk/lib/Analysis/TargetLibraryInfo.cpp

 static cl::opt<TargetLibraryInfoImpl::VectorLibrary> ClVectorLibrary(
     "vector-library", cl::Hidden, cl::desc("Vector functions library"),
     cl::init(TargetLibraryInfoImpl::NoLibrary),
     cl::values(clEnumValN(TargetLibraryInfoImpl::NoLibrary, "none",
                           "No vector functions library"),
                clEnumValN(TargetLibraryInfoImpl::Accelerate, "Accelerate",
                           "Accelerate framework"),
                clEnumValN(TargetLibraryInfoImpl::SVML, "SVML",
-                          "Intel SVML library")));
+                          "Intel SVML library"),
+               clEnumValN(TargetLibraryInfoImpl::SLEEFGNUABI, "sleefgnuabi",
+                          "SIMD Library for Evaluating Elementary Functions")));
 
 StringRef const TargetLibraryInfoImpl::StandardNames[LibFunc::NumLibFuncs] = {
 #define TLI_DEFINE_STRING
 #include "llvm/Analysis/TargetLibraryInfo.def"
 };
 
 static bool hasSinCosPiStret(const Triple &T) {
   // Only Darwin variants have _stret versions of combined trig functions.

 
 TargetLibraryInfoImpl::TargetLibraryInfoImpl() {
   // Default to everything being available.
   memset(AvailableArray, -1, sizeof(AvailableArray));
 
   initialize(*this, Triple(), StandardNames);
 }
 
-TargetLibraryInfoImpl::TargetLibraryInfoImpl(const Triple &T) {
+TargetLibraryInfoImpl::TargetLibraryInfoImpl(const Triple &T)
+  : TT(T) {
   // Default to everything being available.
   memset(AvailableArray, -1, sizeof(AvailableArray));
 
   initialize(*this, T, StandardNames);
 }
 
 TargetLibraryInfoImpl::TargetLibraryInfoImpl(const TargetLibraryInfoImpl &TLI)
-    : CustomNames(TLI.CustomNames), ShouldExtI32Param(TLI.ShouldExtI32Param),
+    : TT(TLI.TT), CustomNames(TLI.CustomNames),
+      ShouldExtI32Param(TLI.ShouldExtI32Param),
       ShouldExtI32Return(TLI.ShouldExtI32Return),
       ShouldSignExtI32Param(TLI.ShouldSignExtI32Param) {
   memcpy(AvailableArray, TLI.AvailableArray, sizeof(AvailableArray));
   VectorDescs = TLI.VectorDescs;
   ScalarDescs = TLI.ScalarDescs;
 }
 
 TargetLibraryInfoImpl::TargetLibraryInfoImpl(TargetLibraryInfoImpl &&TLI)
-    : CustomNames(std::move(TLI.CustomNames)),
+    : TT(std::move(TLI.TT)), CustomNames(std::move(TLI.CustomNames)),
       ShouldExtI32Param(TLI.ShouldExtI32Param),
       ShouldExtI32Return(TLI.ShouldExtI32Return),
       ShouldSignExtI32Param(TLI.ShouldSignExtI32Param) {
   std::move(std::begin(TLI.AvailableArray), std::end(TLI.AvailableArray),
             AvailableArray);
   VectorDescs = TLI.VectorDescs;
   ScalarDescs = TLI.ScalarDescs;
 }
 
 TargetLibraryInfoImpl &TargetLibraryInfoImpl::operator=(const TargetLibraryInfoImpl &TLI) {
+  TT = TLI.TT;
   CustomNames = TLI.CustomNames;
   ShouldExtI32Param = TLI.ShouldExtI32Param;
   ShouldExtI32Return = TLI.ShouldExtI32Return;
   ShouldSignExtI32Param = TLI.ShouldSignExtI32Param;
   memcpy(AvailableArray, TLI.AvailableArray, sizeof(AvailableArray));
   return *this;
 }
 
 TargetLibraryInfoImpl &TargetLibraryInfoImpl::operator=(TargetLibraryInfoImpl &&TLI) {
+  TT = std::move(TLI.TT);
   CustomNames = std::move(TLI.CustomNames);
   ShouldExtI32Param = TLI.ShouldExtI32Param;
   ShouldExtI32Return = TLI.ShouldExtI32Return;
   ShouldSignExtI32Param = TLI.ShouldSignExtI32Param;
   std::move(std::begin(TLI.AvailableArray), std::end(TLI.AvailableArray),
             AvailableArray);
   return *this;
 }
const VecDesc VecFuncs[] = {
 
         {"llvm.log.f32", "__svml_logf4", 4},
         {"llvm.log.f32", "__svml_logf8", 8},
         {"llvm.log.f32", "__svml_logf16", 16},
     };
     addVectorizableFunctions(VecFuncs);
     break;
   }
+  case SLEEFGNUABI: {
+    if (TT.getArch() == llvm::Triple::aarch64 ||
+        TT.getArch() == llvm::Triple::aarch64_be) {
+      const VecDesc AArch64TwoAndFourLaneVecFuncs[] = {
+        { "acos", "_ZGVnN2v_acos", 2 },
+        { "acos", "_ZGVnN4v_acosf", 4 },
+        { "acosf", "_ZGVnN4v_acosf", 4 },
+        { "llvm.acos.f64", "_ZGVnN2v_acos", 2 },
+        { "llvm.acos.f32", "_ZGVnN4v_acosf", 4 },
+
+        { "asin", "_ZGVnN2v_asin", 2 },
+        { "asin", "_ZGVnN4v_asinf", 4 },
+        { "asinf", "_ZGVnN4v_asinf", 4 },
+        { "llvm.asin.f64", "_ZGVnN2v_asin", 2 },
+        { "llvm.asin.f32", "_ZGVnN4v_asinf", 4 },
+
+        { "atan", "_ZGVnN2v_atan", 2 },
+        { "atan", "_ZGVnN4v_atanf", 4 },
+        { "atanf", "_ZGVnN4v_atanf", 4 },
+        { "llvm.atan.f64", "_ZGVnN2v_atan", 2 },
+        { "llvm.atan.f32", "_ZGVnN4v_atanf", 4 },
+
+        { "atan2", "_ZGVnN2vv_atan2", 2 },
+        { "atan2", "_ZGVnN4vv_atan2f", 4 },
+        { "atan2f", "_ZGVnN4vv_atan2f", 4 },
+        { "llvm.atan2.f64", "_ZGVnN2vv_atan2", 2 },
+        { "llvm.atan2.f32", "_ZGVnN4vv_atan2f", 4 },
+        { "llvm.atan2.v2f64", "_ZGVnN2vv_atan2", 2 },
+        { "llvm.atan2.v4f32", "_ZGVnN4vv_atan2f", 4 },
+
+        { "atanh", "_ZGVnN2v_atanh", 2 },
+        { "atanh", "_ZGVnN4v_atanhf", 4 },
+        { "atanhf", "_ZGVnN4v_atanhf", 4 },
+        { "llvm.atanh.f64", "_ZGVnN2v_atanh", 2 },
+        { "llvm.atanh.f32", "_ZGVnN4v_atanhf", 4 },
+
+        { "cos", "_ZGVnN2v_cos", 2 },
+        { "cos", "_ZGVnN4v_cosf", 4 },
+        { "cosf", "_ZGVnN4v_cosf", 4 },
+        { "llvm.cos.f64", "_ZGVnN2v_cos", 2 },
+        { "llvm.cos.f32", "_ZGVnN4v_cosf", 4 },
+
+        { "cosh", "_ZGVnN2v_cosh", 2 },
+        { "cosh", "_ZGVnN4v_coshf", 4 },
+        { "coshf", "_ZGVnN4v_coshf", 4 },
+        { "llvm.cosh.f64", "_ZGVnN2v_cosh", 2 },
+        { "llvm.cosh.f32", "_ZGVnN4v_coshf", 4 },
+
+        { "exp", "_ZGVnN2v_exp", 2 },
+        { "exp", "_ZGVnN4v_expf", 4 },
+        { "expf", "_ZGVnN4v_expf", 4 },
+        { "llvm.exp.f64", "_ZGVnN2v_exp", 2 },
+        { "llvm.exp.f32", "_ZGVnN4v_expf", 4 },
+        { "llvm.exp.v2f64", "_ZGVnN2v_exp", 2 },
+        { "llvm.exp.v4f32", "_ZGVnN4v_expf", 4 },
+
+        { "exp2", "_ZGVnN2v_exp2", 2 },
+        { "exp2", "_ZGVnN4v_exp2f", 4 },
+        { "exp2f", "_ZGVnN4v_exp2f", 4 },
+        { "llvm.exp2.f64", "_ZGVnN2v_exp2", 2 },
+        { "llvm.exp2.f32", "_ZGVnN4v_exp2f", 4 },
+        { "llvm.exp2.v2f64", "_ZGVnN2v_exp2", 2 },
+        { "llvm.exp2.v4f32", "_ZGVnN4v_exp2f", 4 },
+
+        { "exp10", "_ZGVnN2v_exp10", 2 },
+        { "exp10", "_ZGVnN4v_exp10f", 4 },
+        { "exp10f", "_ZGVnN4v_exp10f", 4 },
+        { "llvm.exp10.f64", "_ZGVnN2v_exp10", 2 },
+        { "llvm.exp10.f32", "_ZGVnN4v_exp10f", 4 },
+        { "llvm.exp10.v2f64", "_ZGVnN2v_exp10", 2 },
+        { "llvm.exp10.v4f32", "_ZGVnN4v_exp10f", 4 },
+
+        { "lgamma", "_ZGVnN2v_lgamma", 2 },
+        { "lgamma", "_ZGVnN4v_lgammaf", 4 },
+        { "lgammaf", "_ZGVnN4v_lgammaf", 4 },
+        { "llvm.lgamma.f64", "_ZGVnN2v_lgamma", 2 },
+        { "llvm.lgamma.f32", "_ZGVnN4v_lgammaf", 4 },
+
+        { "log", "_ZGVnN2v_log", 2 },
+        { "log", "_ZGVnN4v_logf", 4 },
+        { "logf", "_ZGVnN4v_logf", 4 },
+        { "llvm.log.f64", "_ZGVnN2v_log", 2 },
+        { "llvm.log.f32", "_ZGVnN4v_logf", 4 },
+
+        { "log2", "_ZGVnN2v_log2", 2 },
+        { "log2", "_ZGVnN4v_log2f", 4 },
+        { "log2f", "_ZGVnN4v_log2f", 4 },
+        { "llvm.log2.f64", "_ZGVnN2v_log2", 2 },
+        { "llvm.log2.f32", "_ZGVnN4v_log2f", 4 },
+
+        { "log10", "_ZGVnN2v_log10", 2 },
+        { "log10", "_ZGVnN4v_log10f", 4 },
+        { "log10f", "_ZGVnN4v_log10f", 4 },
+        { "llvm.log10.f64", "_ZGVnN2v_log10", 2 },
+        { "llvm.log10.f32", "_ZGVnN4v_log10f", 4 },
+
+        { "pow", "_ZGVnN2vv_pow", 2 },
+        { "pow", "_ZGVnN4vv_powf", 4 },
+        { "powf", "_ZGVnN4vv_powf", 4 },
+        { "llvm.pow.f64", "_ZGVnN2vv_pow", 2 },
+        { "llvm.pow.f32", "_ZGVnN4vv_powf", 4 },
+        { "llvm.pow.v2f64", "_ZGVnN2vv_pow", 2 },
+        { "llvm.pow.v4f32", "_ZGVnN4vv_powf", 4 },
+
+        { "sin", "_ZGVnN2v_sin", 2 },
+        { "sin", "_ZGVnN4v_sinf", 4 },
+        { "sinf", "_ZGVnN4v_sinf", 4 },
+        { "llvm.sin.f64", "_ZGVnN2v_sin", 2 },
+        { "llvm.sin.f32", "_ZGVnN4v_sinf", 4 },
+
+        { "sinh", "_ZGVnN2v_sinh", 2 },
+        { "sinh", "_ZGVnN4v_sinhf", 4 },
+        { "sinhf", "_ZGVnN4v_sinhf", 4 },
+        { "llvm.sinh.f64", "_ZGVnN2v_sinh", 2 },
+        { "llvm.sinh.f32", "_ZGVnN4v_sinhf", 4 },
+
+        { "sqrt", "_ZGVnN2v_sqrt", 2 },
+        { "sqrt", "_ZGVnN4v_sqrtf", 4 },
+        { "sqrtf", "_ZGVnN4v_sqrtf", 4 },
+        { "llvm.sqrt.f64", "_ZGVnN2v_sqrt", 2 },
+        { "llvm.sqrt.f32", "_ZGVnN4v_sqrtf", 4 },
+
+        { "tan", "_ZGVnN2v_tan", 2 },
+        { "tan", "_ZGVnN4v_tanf", 4 },
+        { "tanf", "_ZGVnN4v_tanf", 4 },
+        { "llvm.tan.f64", "_ZGVnN2v_tan", 2 },
+        { "llvm.tan.f32", "_ZGVnN4v_tanf", 4 },
+
+        { "tanh", "_ZGVnN2v_tanh", 2 },
+        { "tanh", "_ZGVnN4v_tanhf", 4 },
+        { "tanhf", "_ZGVnN4v_tanhf", 4 },
+        { "llvm.tanh.f64", "_ZGVnN2v_tanh", 2 },
+        { "llvm.tanh.f32", "_ZGVnN4v_tanhf", 4 },
+
+        { "tgamma", "_ZGVnN2v_tgamma", 2 },
+        { "tgamma", "_ZGVnN4v_tgammaf", 4 },
+        { "tgammaf", "_ZGVnN4v_tgammaf", 4 },
+        { "llvm.tgamma.f64", "_ZGVnN2v_tgamma", 2 },
+        { "llvm.tgamma.f32", "_ZGVnN4v_tgammaf", 4 },
+      };
+      addVectorizableFunctions(AArch64TwoAndFourLaneVecFuncs);
+    }
+    break;
+  }
   case NoLibrary:
     break;
   }
 }
 
 bool TargetLibraryInfoImpl::isFunctionVectorizable(StringRef funcName) const {
   funcName = sanitizeFunctionName(funcName);
   if (funcName.empty())

llvm/trunk/test/Transforms/LoopVectorize/AArch64/sleef-calls-aarch64.ll

+; Do NOT use -O3. It will lower exp2 to ldexp, and the test will fail.
+; RUN: opt -vector-library=sleefgnuabi -loop-unroll -loop-vectorize -S < %s | FileCheck %s
+
+target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64-unknown-linux-gnu"
+
+declare double @acos(double) #0
+declare float @acosf(float) #0
+declare double @llvm.acos.f64(double) #0
+declare float @llvm.acos.f32(float) #0
+
+define void @acos_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @acos_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_acos(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @acos(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @acos_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @acos_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_acosf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @acosf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @asin(double) #0
+declare float @asinf(float) #0
+declare double @llvm.asin.f64(double) #0
+declare float @llvm.asin.f32(float) #0
+
+define void @asin_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @asin_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_asin(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @asin(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @asin_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @asin_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_asinf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @asinf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @atan(double) #0
+declare float @atanf(float) #0
+declare double @llvm.atan.f64(double) #0
+declare float @llvm.atan.f32(float) #0
+
+define void @atan_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @atan_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_atan(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @atan(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @atan_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @atan_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_atanf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @atanf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @atan2(double, double) #0
+declare float @atan2f(float, float) #0
+declare double @llvm.atan2.f64(double, double) #0
+declare float @llvm.atan2.f32(float, float) #0
+
+define void @atan2_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @atan2_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2vv_atan2(<2 x double> [[TMP4:%.*]], <2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @atan2(double %conv, double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @atan2_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @atan2_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4vv_atan2f(<4 x float> [[TMP4:%.*]], <4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @atan2f(float %conv, float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @atanh(double) #0
+declare float @atanhf(float) #0
+declare double @llvm.atanh.f64(double) #0
+declare float @llvm.atanh.f32(float) #0
+
+define void @atanh_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @atanh_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_atanh(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @atanh(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @atanh_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @atanh_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_atanhf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @atanhf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @cos(double) #0
+declare float @cosf(float) #0
+declare double @llvm.cos.f64(double) #0
+declare float @llvm.cos.f32(float) #0
+
+define void @cos_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @cos_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_cos(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @cos(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @cos_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @cos_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_cosf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @cosf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @cosh(double) #0
+declare float @coshf(float) #0
+declare double @llvm.cosh.f64(double) #0
+declare float @llvm.cosh.f32(float) #0
+
+define void @cosh_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @cosh_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_cosh(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @cosh(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @cosh_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @cosh_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_coshf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @coshf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @exp(double) #0
+declare float @expf(float) #0
+declare double @llvm.exp.f64(double) #0
+declare float @llvm.exp.f32(float) #0
+
+define void @exp_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @exp_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_exp(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @exp(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @exp_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @exp_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_expf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @expf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @exp2(double) #0
+declare float @exp2f(float) #0
+declare double @llvm.exp2.f64(double) #0
+declare float @llvm.exp2.f32(float) #0
+
+define void @exp2_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @exp2_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_exp2(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @exp2(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @exp2_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @exp2_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_exp2f(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @exp2f(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @exp10(double) #0
+declare float @exp10f(float) #0
+declare double @llvm.exp10.f64(double) #0
+declare float @llvm.exp10.f32(float) #0
+
+define void @exp10_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @exp10_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_exp10(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @exp10(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @exp10_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @exp10_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_exp10f(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @exp10f(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @lgamma(double) #0
+declare float @lgammaf(float) #0
+declare double @llvm.lgamma.f64(double) #0
+declare float @llvm.lgamma.f32(float) #0
+
+define void @lgamma_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @lgamma_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_lgamma(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @lgamma(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @lgamma_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @lgamma_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_lgammaf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @lgammaf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @log10(double) #0
+declare float @log10f(float) #0
+declare double @llvm.log10.f64(double) #0
+declare float @llvm.log10.f32(float) #0
+
+define void @log10_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @log10_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_log10(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @log10(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @log10_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @log10_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_log10f(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @log10f(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @log2(double) #0
+declare float @log2f(float) #0
+declare double @llvm.log2.f64(double) #0
+declare float @llvm.log2.f32(float) #0
+
+define void @log2_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @log2_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_log2(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @log2(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @log2_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @log2_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_log2f(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @log2f(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @log(double) #0
+declare float @logf(float) #0
+declare double @llvm.log.f64(double) #0
+declare float @llvm.log.f32(float) #0
+
+define void @log_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @log_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_log(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @log(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @log_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @log_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_logf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @logf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @pow(double, double) #0
+declare float @powf(float, float) #0
+declare double @llvm.pow.f64(double, double) #0
+declare float @llvm.pow.f32(float, float) #0
+
+define void @pow_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @pow_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2vv_pow(<2 x double> [[TMP4:%.*]], <2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @pow(double %conv, double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @pow_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @pow_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4vv_powf(<4 x float> [[TMP4:%.*]], <4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @powf(float %conv, float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @sin(double) #0
+declare float @sinf(float) #0
+declare double @llvm.sin.f64(double) #0
+declare float @llvm.sin.f32(float) #0
+
+define void @sin_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @sin_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_sin(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @sin(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @sin_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @sin_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_sinf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @sinf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @sinh(double) #0
+declare float @sinhf(float) #0
+declare double @llvm.sinh.f64(double) #0
+declare float @llvm.sinh.f32(float) #0
+
+define void @sinh_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @sinh_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_sinh(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @sinh(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @sinh_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @sinh_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_sinhf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @sinhf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @sqrt(double) #0
+declare float @sqrtf(float) #0
+declare double @llvm.sqrt.f64(double) #0
+declare float @llvm.sqrt.f32(float) #0
+
+define void @sqrt_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @sqrt_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_sqrt(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @sqrt(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @sqrt_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @sqrt_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_sqrtf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @sqrtf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @tan(double) #0
+declare float @tanf(float) #0
+declare double @llvm.tan.f64(double) #0
+declare float @llvm.tan.f32(float) #0
+
+define void @tan_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @tan_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_tan(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @tan(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @tan_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @tan_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_tanf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @tanf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @tanh(double) #0
+declare float @tanhf(float) #0
+declare double @llvm.tanh.f64(double) #0
+declare float @llvm.tanh.f32(float) #0
+
+define void @tanh_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @tanh_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_tanh(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @tanh(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @tanh_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @tanh_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_tanhf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @tanhf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+declare double @tgamma(double) #0
+declare float @tgammaf(float) #0
+declare double @llvm.tgamma.f64(double) #0
+declare float @llvm.tgamma.f32(float) #0
+
+define void @tgamma_f64(double* nocapture %varray) {
+  ; CHECK-LABEL: @tgamma_f64(
+  ; CHECK:    [[TMP5:%.*]] = call <2 x double> @_ZGVnN2v_tgamma(<2 x double> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to double
+  %call = tail call double @tgamma(double %conv)
+  %arrayidx = getelementptr inbounds double, double* %varray, i64 %iv
+  store double %call, double* %arrayidx, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+
+define void @tgamma_f32(float* nocapture %varray) {
+  ; CHECK-LABEL: @tgamma_f32(
+  ; CHECK:    [[TMP5:%.*]] = call <4 x float> @_ZGVnN4v_tgammaf(<4 x float> [[TMP4:%.*]])
+  ; CHECK:    ret void
+  ;
+  entry:
+  br label %for.body
+
+  for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %tmp = trunc i64 %iv to i32
+  %conv = sitofp i32 %tmp to float
+  %call = tail call float @tgammaf(float %conv)
+  %arrayidx = getelementptr inbounds float, float* %varray, i64 %iv
+  store float %call, float* %arrayidx, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, 1000
+  br i1 %exitcond, label %for.end, label %for.body
+
+  for.end:
+  ret void
+}
+