[SLP] Support llvm.isnan in vectorizer

[sepavloff]

Summary

This change adds llvm.isnan to the set of trivially vectorizable
intrinsic functions. This function differs from other functions by
signature, so processing Call instruction in vectorizeTree was
changed so that overloaded argument types of intrinsic functions were
restored using information from intrinsic description tables.

This change fixes PR51556 (Failure to vectorize llvm.isnan calls).

[xbolva00]

But there are concerns in https://reviews.llvm.org/D104854 whether the base change should not be reverted at all…

[RKSimon]

In D109049#2976449, @xbolva00 wrote:

But there are concerns in https://reviews.llvm.org/D104854 whether the base change should not be reverted at all…

This needs to be addressed first, I regret putting these isnan cost/vectorization tests/bugs in at all atm as I hadn't realised there were still so many open questions about these new fpclassify intrinsics......

llvm/include/llvm/IR/Intrinsics.h

192

The creation of hasBoundType and the refactor of these asserts could be pulled out as a preliminary NFC - independent of this patch. Since you are updating the asserts we should add assert messages as well so that they match the style guidelines.

[sepavloff]

In D109049#2976449, @xbolva00 wrote:

But there are concerns in https://reviews.llvm.org/D104854 whether the base change should not be reverted at all…

There is a discussion: https://lists.llvm.org/pipermail/llvm-dev/2021-August/152257.html, devoted to this topic. There are no additional feedback during the last week, so I would think the problem is closed.

There are no arguments against the intrinsic, this is the only way to implement isnan, which would satisfy IEEE-754 and C standard in the case when FP exceptions are not ignored.

There were some hesitation whether this intrinsic should be optimized out in fast-math mode. So far there are no sound arguments in favor of such optimization, but there is a reference to the relevant discussion in GCC mail list, which demonstrates difficulties that rise if the effect of fast-math is understood in "naive" way. No matter if we decided optimize out this intrinsic or not, necessity of the intrinsic is not doubted.

[lebedev.ri]

In D109049#2976512, @sepavloff wrote:

In D109049#2976449, @xbolva00 wrote:

But there are concerns in https://reviews.llvm.org/D104854 whether the base change should not be reverted at all…

There is a discussion: https://lists.llvm.org/pipermail/llvm-dev/2021-August/152257.html, devoted to this topic. There are no additional feedback during the last week, so I would think the problem is closed.

Au contraire.
It looks like goalpost shifting to me.
What is going on the is basic design disscussion, with no clear favorite,
even though a design has already been "selected" and committed.
I would like to once again urge you to restart this process.

There are no arguments against the intrinsic, this is the only way to implement isnan, which would satisfy IEEE-754 and C standard in the case when FP exceptions are not ignored.

There were some hesitation whether this intrinsic should be optimized out in fast-math mode. So far there are no sound arguments in favor of such optimization, but there is a reference to the relevant discussion in GCC mail list, which demonstrates difficulties that rise if the effect of fast-math is understood in "naive" way. No matter if we decided optimize out this intrinsic or not, necessity of the intrinsic is not doubted.

[sepavloff]

In D109049#2976514, @lebedev.ri wrote:

In D109049#2976512, @sepavloff wrote:

In D109049#2976449, @xbolva00 wrote:

But there are concerns in https://reviews.llvm.org/D104854 whether the base change should not be reverted at all…

There is a discussion: https://lists.llvm.org/pipermail/llvm-dev/2021-August/152257.html, devoted to this topic. There are no additional feedback during the last week, so I would think the problem is closed.

Au contraire.
It looks like goalpost shifting to me.
What is going on the is basic design disscussion, with no clear favorite,
even though a design has already been "selected" and committed.
I would like to once again urge you to restart this process.

What exactly do you want me to restart? There is already a discussion on this topic, it could be continued. You could express your concerns there to warm the discussion.

[xbolva00]

In D109049#2976566, @sepavloff wrote:

In D109049#2976514, @lebedev.ri wrote:

In D109049#2976512, @sepavloff wrote:

In D109049#2976449, @xbolva00 wrote:

But there are concerns in https://reviews.llvm.org/D104854 whether the base change should not be reverted at all…

There is a discussion: https://lists.llvm.org/pipermail/llvm-dev/2021-August/152257.html, devoted to this topic. There are no additional feedback during the last week, so I would think the problem is closed.

Au contraire.
It looks like goalpost shifting to me.
What is going on the is basic design disscussion, with no clear favorite,
even though a design has already been "selected" and committed.
I would like to once again urge you to restart this process.

What exactly do you want me to restart? There is already a discussion on this topic, it could be continued. You could express your concerns there to warm the discussion.

Revert original patch while there is a ongoing discussion

[sepavloff]

In D109049#2976649, @xbolva00 wrote:

In D109049#2976566, @sepavloff wrote:

In D109049#2976514, @lebedev.ri wrote:

In D109049#2976512, @sepavloff wrote:

In D109049#2976449, @xbolva00 wrote:

But there are concerns in https://reviews.llvm.org/D104854 whether the base change should not be reverted at all…

There is a discussion: https://lists.llvm.org/pipermail/llvm-dev/2021-August/152257.html, devoted to this topic. There are no additional feedback during the last week, so I would think the problem is closed.

Au contraire.
It looks like goalpost shifting to me.
What is going on the is basic design disscussion, with no clear favorite,
even though a design has already been "selected" and committed.
I would like to once again urge you to restart this process.

What exactly do you want me to restart? There is already a discussion on this topic, it could be continued. You could express your concerns there to warm the discussion.

Revert original patch while there is a ongoing discussion

There is no an ongoing discussion at present. The discussion was in https://lists.llvm.org/pipermail/llvm-dev/2021-August/152257.html. No essential objections were provided. I would say we reached consensus that llvm.isnan is necessary.

If you have any objection or concern please publish it in the discussion on the mail list.

[arsenm]

Can this be abandoned now? We have is.fpclass and I believe vectorization of it works

[sepavloff]

In D109049#4387907, @arsenm wrote:

Can this be abandoned now? We have is.fpclass and I believe vectorization of it works

Sure, it is not actual anymore.

llvm/include/llvm/IR/Intrinsics.h

enum ArgKind {
       AK_Any,
       AK_AnyInteger,
       AK_AnyFloat,
       AK_AnyVector,
       AK_AnyPointer,
       AK_MatchType = 7
     };
 
+    /// Returns true if type of this argument or return value depends on the
+    /// type of other argument.
+    bool hasBoundType() const {
+      switch (Kind) {
+      case Argument:
+      case ExtendArgument:
+      case TruncArgument:
+      case HalfVecArgument:
+      case SameVecWidthArgument:
+      case PtrToArgument:
+      case PtrToElt:
+      case VecOfAnyPtrsToElt:
+      case VecElementArgument:
+      case Subdivide2Argument:
+      case Subdivide4Argument:
+      case VecOfBitcastsToInt:
+        return true;
+      default:
+        return false;
+      }
+      llvm_unreachable("unhandled");
+    }
+
     unsigned getArgumentNumber() const {
-      assert(Kind == Argument || Kind == ExtendArgument ||
-             Kind == TruncArgument || Kind == HalfVecArgument ||
-             Kind == SameVecWidthArgument || Kind == PtrToArgument ||
-             Kind == PtrToElt || Kind == VecElementArgument ||
-             Kind == Subdivide2Argument || Kind == Subdivide4Argument ||
-             Kind == VecOfBitcastsToInt);
+      assert(hasBoundType() && Kind != VecOfAnyPtrsToElt);
       return Argument_Info >> 3;
     }
     ArgKind getArgumentKind() const {
-      assert(Kind == Argument || Kind == ExtendArgument ||
-             Kind == TruncArgument || Kind == HalfVecArgument ||
-             Kind == SameVecWidthArgument || Kind == PtrToArgument ||
-             Kind == VecElementArgument || Kind == Subdivide2Argument ||
-             Kind == Subdivide4Argument || Kind == VecOfBitcastsToInt);
+      assert(hasBoundType() && Kind != VecOfAnyPtrsToElt);
       return (ArgKind)(Argument_Info & 7);
     }
 
     // VecOfAnyPtrsToElt uses both an overloaded argument (for address space)
     // and a reference argument (for matching vector width and element types)
     unsigned getOverloadArgNumber() const {
       assert(Kind == VecOfAnyPtrsToElt);
       return Argument_Info >> 16;
static IITDescriptor getVector(unsigned Width, bool IsScalable) {
       return Result;
     }
   };
 
   /// Return the IIT table descriptor for the specified intrinsic into an array
   /// of IITDescriptors.
   void getIntrinsicInfoTableEntries(ID id, SmallVectorImpl<IITDescriptor> &T);
 
+  /// Given the IIT table descriptor skips the descriptors corresponding to one
+  /// type element (either return value or an argument).
+  ///
+  /// The function is used to iterate through intrinsic description without
+  /// building types.
+  void popFirstElement(ArrayRef<Intrinsic::IITDescriptor> &Infos);
+
   enum MatchIntrinsicTypesResult {
     MatchIntrinsicTypes_Match = 0,
     MatchIntrinsicTypes_NoMatchRet = 1,
     MatchIntrinsicTypes_NoMatchArg = 2,
   };
 
   /// Match the specified function type with the type constraints specified by
   /// the .td file. If the given type is an overloaded type it is pushed to the

llvm/lib/Analysis/VectorUtils.cpp

bool llvm::isTriviallyVectorizable(Intrinsic::ID ID) {
   case Intrinsic::nearbyint:
   case Intrinsic::round:
   case Intrinsic::roundeven:
   case Intrinsic::pow:
   case Intrinsic::fma:
   case Intrinsic::fmuladd:
   case Intrinsic::powi:
   case Intrinsic::canonicalize:
+  case Intrinsic::isnan:
     return true;
   default:
     return false;
   }
 }
 
 /// Identifies if the vector form of the intrinsic has a scalar operand.
 bool llvm::hasVectorInstrinsicScalarOpd(Intrinsic::ID ID,

llvm/lib/IR/Function.cpp

void Intrinsic::getIntrinsicInfoTableEntries(ID id,
   }
 
   // Okay, decode the table into the output vector of IITDescriptors.
   DecodeIITType(NextElt, IITEntries, IIT_Done, T);
   while (NextElt != IITEntries.size() && IITEntries[NextElt] != 0)
     DecodeIITType(NextElt, IITEntries, IIT_Done, T);
 }
 
+void Intrinsic::popFirstElement(ArrayRef<Intrinsic::IITDescriptor> &Infos) {
+  assert(!Infos.empty());
+
+  IITDescriptor D = Infos.front();
+  Infos = Infos.slice(1);
+  switch (D.Kind) {
+  case IITDescriptor::Void:
+  case IITDescriptor::VarArg:
+  case IITDescriptor::MMX:
+  case IITDescriptor::AMX:
+  case IITDescriptor::Token:
+  case IITDescriptor::Metadata:
+  case IITDescriptor::Half:
+  case IITDescriptor::BFloat:
+  case IITDescriptor::Float:
+  case IITDescriptor::Double:
+  case IITDescriptor::Quad:
+  case IITDescriptor::Integer:
+  case IITDescriptor::Argument:
+  case IITDescriptor::ExtendArgument:
+  case IITDescriptor::TruncArgument:
+  case IITDescriptor::Subdivide2Argument:
+  case IITDescriptor::Subdivide4Argument:
+  case IITDescriptor::HalfVecArgument:
+  case IITDescriptor::PtrToArgument:
+  case IITDescriptor::PtrToElt:
+  case IITDescriptor::VecElementArgument:
+  case IITDescriptor::VecOfBitcastsToInt:
+  case IITDescriptor::VecOfAnyPtrsToElt:
+    return;
+  case IITDescriptor::Vector:
+  case IITDescriptor::Pointer:
+  case IITDescriptor::SameVecWidthArgument:
+    popFirstElement(Infos);
+    return;
+  case IITDescriptor::Struct:
+    for (unsigned i = 0, e = D.Struct_NumElements; i != e; ++i)
+      popFirstElement(Infos);
+    return;
+  }
+  llvm_unreachable("unhandled");
+}
+
 static Type *DecodeFixedType(ArrayRef<Intrinsic::IITDescriptor> &Infos,
                              ArrayRef<Type*> Tys, LLVMContext &Context) {
   using namespace Intrinsic;
 
   IITDescriptor D = Infos.front();
   Infos = Infos.slice(1);
 
   switch (D.Kind) {

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

case Instruction::Call: {
         IID = FI->getIntrinsicID();
 
       Intrinsic::ID ID = getVectorIntrinsicIDForCall(CI, TLI);
 
       auto VecCallCosts = getVectorCallCosts(CI, VecTy, TTI, TLI);
       bool UseIntrinsic = ID != Intrinsic::not_intrinsic &&
                           VecCallCosts.first <= VecCallCosts.second;
 
+      SmallVector<Intrinsic::IITDescriptor, 8> Table;
+      ArrayRef<Intrinsic::IITDescriptor> TableRef;
+      if (UseIntrinsic) {
+        getIntrinsicInfoTableEntries(ID, Table);
+        TableRef = Table;
+      }
+      SmallVector<Type *, 2> TysForDecl;
+
+      const auto UpdateTysForDecl = [&](Value *V, bool IsScalar) {
+        if (!UseIntrinsic)
+          return;
+        const Intrinsic::IITDescriptor &Item = TableRef.front();
+        switch (Item.Kind) {
+        case Intrinsic::IITDescriptor::Argument:
+          if (Item.getArgumentNumber() >= TysForDecl.size()) {
+            if (IsScalar)
+              TysForDecl.push_back(V->getType());
+            else
+              TysForDecl.push_back(
+                  FixedVectorType::get(V->getType(), E->Scalars.size()));
+          }
+          break;
+        case Intrinsic::IITDescriptor::SameVecWidthArgument:
+          break;
+        default:
+          assert(!Item.hasBoundType());
+        }
+        popFirstElement(TableRef);
+      };
+
+      UpdateTysForDecl(CI, false);
       Value *ScalarArg = nullptr;
       std::vector<Value *> OpVecs;
-      SmallVector<Type *, 2> TysForDecl =
-          {FixedVectorType::get(CI->getType(), E->Scalars.size())};
       for (int j = 0, e = CI->getNumArgOperands(); j < e; ++j) {
-        ValueList OpVL;
+        if (UseIntrinsic) {
           // Some intrinsics have scalar arguments. This argument should not be
           // vectorized.
-        if (UseIntrinsic && hasVectorInstrinsicScalarOpd(IID, j)) {
+          bool IsScalar = hasVectorInstrinsicScalarOpd(IID, j);
+          if (IsScalar) {
             CallInst *CEI = cast<CallInst>(VL0);
             ScalarArg = CEI->getArgOperand(j);
             OpVecs.push_back(CEI->getArgOperand(j));
-          if (hasVectorInstrinsicOverloadedScalarOpd(IID, j))
-            TysForDecl.push_back(ScalarArg->getType());
+          }
+          UpdateTysForDecl(CI->getArgOperand(j), IsScalar);
+          if (IsScalar)
             continue;
         }
 
         Value *OpVec = vectorizeTree(E->getOperand(j));
         LLVM_DEBUG(dbgs() << "SLP: OpVec[" << j << "]: " << *OpVec << "\n");
         OpVecs.push_back(OpVec);
       }
 
       Function *CF;

llvm/test/Transforms/SLPVectorizer/X86/intrinsic.ll

 }
 
 define void @vec_ctlz_i32_neg(i32* %a, i32* %b, i32* %c, i1) {
 ; CHECK-LABEL: @vec_ctlz_i32_neg(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[I0:%.*]] = load i32, i32* [[A:%.*]], align 4
 ; CHECK-NEXT:    [[I1:%.*]] = load i32, i32* [[B:%.*]], align 4
 ; CHECK-NEXT:    [[ADD1:%.*]] = add i32 [[I0]], [[I1]]
-; CHECK-NEXT:    [[CALL1:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[ADD1]], i1 true) #[[ATTR4:[0-9]+]]
+; CHECK-NEXT:    [[CALL1:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[ADD1]], i1 true) #[[ATTR5:[0-9]+]]
 ; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds i32, i32* [[A]], i32 1
 ; CHECK-NEXT:    [[I2:%.*]] = load i32, i32* [[ARRAYIDX2]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds i32, i32* [[B]], i32 1
 ; CHECK-NEXT:    [[I3:%.*]] = load i32, i32* [[ARRAYIDX3]], align 4
 ; CHECK-NEXT:    [[ADD2:%.*]] = add i32 [[I2]], [[I3]]
-; CHECK-NEXT:    [[CALL2:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[ADD2]], i1 false) #[[ATTR4]]
+; CHECK-NEXT:    [[CALL2:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[ADD2]], i1 false) #[[ATTR5]]
 ; CHECK-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds i32, i32* [[A]], i32 2
 ; CHECK-NEXT:    [[I4:%.*]] = load i32, i32* [[ARRAYIDX4]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds i32, i32* [[B]], i32 2
 ; CHECK-NEXT:    [[I5:%.*]] = load i32, i32* [[ARRAYIDX5]], align 4
 ; CHECK-NEXT:    [[ADD3:%.*]] = add i32 [[I4]], [[I5]]
-; CHECK-NEXT:    [[CALL3:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[ADD3]], i1 true) #[[ATTR4]]
+; CHECK-NEXT:    [[CALL3:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[ADD3]], i1 true) #[[ATTR5]]
 ; CHECK-NEXT:    [[ARRAYIDX6:%.*]] = getelementptr inbounds i32, i32* [[A]], i32 3
 ; CHECK-NEXT:    [[I6:%.*]] = load i32, i32* [[ARRAYIDX6]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX7:%.*]] = getelementptr inbounds i32, i32* [[B]], i32 3
 ; CHECK-NEXT:    [[I7:%.*]] = load i32, i32* [[ARRAYIDX7]], align 4
 ; CHECK-NEXT:    [[ADD4:%.*]] = add i32 [[I6]], [[I7]]
-; CHECK-NEXT:    [[CALL4:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[ADD4]], i1 false) #[[ATTR4]]
+; CHECK-NEXT:    [[CALL4:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[ADD4]], i1 false) #[[ATTR5]]
 ; CHECK-NEXT:    store i32 [[CALL1]], i32* [[C:%.*]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX8:%.*]] = getelementptr inbounds i32, i32* [[C]], i32 1
 ; CHECK-NEXT:    store i32 [[CALL2]], i32* [[ARRAYIDX8]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX9:%.*]] = getelementptr inbounds i32, i32* [[C]], i32 2
 ; CHECK-NEXT:    store i32 [[CALL3]], i32* [[ARRAYIDX9]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX10:%.*]] = getelementptr inbounds i32, i32* [[C]], i32 3
 ; CHECK-NEXT:    store i32 [[CALL4]], i32* [[ARRAYIDX10]], align 4
 ; CHECK-NEXT:    ret void

 }
 
 define void @vec_cttz_i32_neg(i32* %a, i32* %b, i32* %c, i1) {
 ; CHECK-LABEL: @vec_cttz_i32_neg(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[I0:%.*]] = load i32, i32* [[A:%.*]], align 4
 ; CHECK-NEXT:    [[I1:%.*]] = load i32, i32* [[B:%.*]], align 4
 ; CHECK-NEXT:    [[ADD1:%.*]] = add i32 [[I0]], [[I1]]
-; CHECK-NEXT:    [[CALL1:%.*]] = tail call i32 @llvm.cttz.i32(i32 [[ADD1]], i1 true) #[[ATTR4]]
+; CHECK-NEXT:    [[CALL1:%.*]] = tail call i32 @llvm.cttz.i32(i32 [[ADD1]], i1 true) #[[ATTR5]]
 ; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds i32, i32* [[A]], i32 1
 ; CHECK-NEXT:    [[I2:%.*]] = load i32, i32* [[ARRAYIDX2]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds i32, i32* [[B]], i32 1
 ; CHECK-NEXT:    [[I3:%.*]] = load i32, i32* [[ARRAYIDX3]], align 4
 ; CHECK-NEXT:    [[ADD2:%.*]] = add i32 [[I2]], [[I3]]
-; CHECK-NEXT:    [[CALL2:%.*]] = tail call i32 @llvm.cttz.i32(i32 [[ADD2]], i1 false) #[[ATTR4]]
+; CHECK-NEXT:    [[CALL2:%.*]] = tail call i32 @llvm.cttz.i32(i32 [[ADD2]], i1 false) #[[ATTR5]]
 ; CHECK-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds i32, i32* [[A]], i32 2
 ; CHECK-NEXT:    [[I4:%.*]] = load i32, i32* [[ARRAYIDX4]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds i32, i32* [[B]], i32 2
 ; CHECK-NEXT:    [[I5:%.*]] = load i32, i32* [[ARRAYIDX5]], align 4
 ; CHECK-NEXT:    [[ADD3:%.*]] = add i32 [[I4]], [[I5]]
-; CHECK-NEXT:    [[CALL3:%.*]] = tail call i32 @llvm.cttz.i32(i32 [[ADD3]], i1 true) #[[ATTR4]]
+; CHECK-NEXT:    [[CALL3:%.*]] = tail call i32 @llvm.cttz.i32(i32 [[ADD3]], i1 true) #[[ATTR5]]
 ; CHECK-NEXT:    [[ARRAYIDX6:%.*]] = getelementptr inbounds i32, i32* [[A]], i32 3
 ; CHECK-NEXT:    [[I6:%.*]] = load i32, i32* [[ARRAYIDX6]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX7:%.*]] = getelementptr inbounds i32, i32* [[B]], i32 3
 ; CHECK-NEXT:    [[I7:%.*]] = load i32, i32* [[ARRAYIDX7]], align 4
 ; CHECK-NEXT:    [[ADD4:%.*]] = add i32 [[I6]], [[I7]]
-; CHECK-NEXT:    [[CALL4:%.*]] = tail call i32 @llvm.cttz.i32(i32 [[ADD4]], i1 false) #[[ATTR4]]
+; CHECK-NEXT:    [[CALL4:%.*]] = tail call i32 @llvm.cttz.i32(i32 [[ADD4]], i1 false) #[[ATTR5]]
 ; CHECK-NEXT:    store i32 [[CALL1]], i32* [[C:%.*]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX8:%.*]] = getelementptr inbounds i32, i32* [[C]], i32 1
 ; CHECK-NEXT:    store i32 [[CALL2]], i32* [[ARRAYIDX8]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX9:%.*]] = getelementptr inbounds i32, i32* [[C]], i32 2
 ; CHECK-NEXT:    store i32 [[CALL3]], i32* [[ARRAYIDX9]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX10:%.*]] = getelementptr inbounds i32, i32* [[C]], i32 3
 ; CHECK-NEXT:    store i32 [[CALL4]], i32* [[ARRAYIDX10]], align 4
 ; CHECK-NEXT:    ret void

 
 
 define void @vec_powi_f32_neg(float* %a, float* %b, float* %c, i32 %P, i32 %Q) {
 ; CHECK-LABEL: @vec_powi_f32_neg(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[I0:%.*]] = load float, float* [[A:%.*]], align 4
 ; CHECK-NEXT:    [[I1:%.*]] = load float, float* [[B:%.*]], align 4
 ; CHECK-NEXT:    [[ADD1:%.*]] = fadd float [[I0]], [[I1]]
-; CHECK-NEXT:    [[CALL1:%.*]] = tail call float @llvm.powi.f32.i32(float [[ADD1]], i32 [[P:%.*]]) #[[ATTR4]]
+; CHECK-NEXT:    [[CALL1:%.*]] = tail call float @llvm.powi.f32.i32(float [[ADD1]], i32 [[P:%.*]]) #[[ATTR5]]
 ; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds float, float* [[A]], i32 1
 ; CHECK-NEXT:    [[I2:%.*]] = load float, float* [[ARRAYIDX2]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds float, float* [[B]], i32 1
 ; CHECK-NEXT:    [[I3:%.*]] = load float, float* [[ARRAYIDX3]], align 4
 ; CHECK-NEXT:    [[ADD2:%.*]] = fadd float [[I2]], [[I3]]
-; CHECK-NEXT:    [[CALL2:%.*]] = tail call float @llvm.powi.f32.i32(float [[ADD2]], i32 [[Q:%.*]]) #[[ATTR4]]
+; CHECK-NEXT:    [[CALL2:%.*]] = tail call float @llvm.powi.f32.i32(float [[ADD2]], i32 [[Q:%.*]]) #[[ATTR5]]
 ; CHECK-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds float, float* [[A]], i32 2
 ; CHECK-NEXT:    [[I4:%.*]] = load float, float* [[ARRAYIDX4]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds float, float* [[B]], i32 2
 ; CHECK-NEXT:    [[I5:%.*]] = load float, float* [[ARRAYIDX5]], align 4
 ; CHECK-NEXT:    [[ADD3:%.*]] = fadd float [[I4]], [[I5]]
-; CHECK-NEXT:    [[CALL3:%.*]] = tail call float @llvm.powi.f32.i32(float [[ADD3]], i32 [[P]]) #[[ATTR4]]
+; CHECK-NEXT:    [[CALL3:%.*]] = tail call float @llvm.powi.f32.i32(float [[ADD3]], i32 [[P]]) #[[ATTR5]]
 ; CHECK-NEXT:    [[ARRAYIDX6:%.*]] = getelementptr inbounds float, float* [[A]], i32 3
 ; CHECK-NEXT:    [[I6:%.*]] = load float, float* [[ARRAYIDX6]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX7:%.*]] = getelementptr inbounds float, float* [[B]], i32 3
 ; CHECK-NEXT:    [[I7:%.*]] = load float, float* [[ARRAYIDX7]], align 4
 ; CHECK-NEXT:    [[ADD4:%.*]] = fadd float [[I6]], [[I7]]
-; CHECK-NEXT:    [[CALL4:%.*]] = tail call float @llvm.powi.f32.i32(float [[ADD4]], i32 [[Q]]) #[[ATTR4]]
+; CHECK-NEXT:    [[CALL4:%.*]] = tail call float @llvm.powi.f32.i32(float [[ADD4]], i32 [[Q]]) #[[ATTR5]]
 ; CHECK-NEXT:    store float [[CALL1]], float* [[C:%.*]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX8:%.*]] = getelementptr inbounds float, float* [[C]], i32 1
 ; CHECK-NEXT:    store float [[CALL2]], float* [[ARRAYIDX8]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX9:%.*]] = getelementptr inbounds float, float* [[C]], i32 2
 ; CHECK-NEXT:    store float [[CALL3]], float* [[ARRAYIDX9]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX10:%.*]] = getelementptr inbounds float, float* [[C]], i32 3
 ; CHECK-NEXT:    store float [[CALL4]], float* [[ARRAYIDX10]], align 4
 ; CHECK-NEXT:    ret void

 }
 
 
 declare i1 @llvm.isnan.f64(double)
 
 define void @vec_isnan_f64(double* %a, double* %b, double* %c, double* %d) {
 ; CHECK-LABEL: @vec_isnan_f64(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[AIDX1:%.*]] = getelementptr inbounds double, double* [[A:%.*]], i64 1
-; CHECK-NEXT:    [[A0:%.*]] = load double, double* [[A]], align 8
-; CHECK-NEXT:    [[A1:%.*]] = load double, double* [[AIDX1]], align 8
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast double* [[B:%.*]] to <2 x double>*
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast double* [[A:%.*]] to <2 x double>*
 ; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, <2 x double>* [[TMP0]], align 8
-; CHECK-NEXT:    [[TMP2:%.*]] = bitcast double* [[C:%.*]] to <2 x double>*
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast double* [[B:%.*]] to <2 x double>*
 ; CHECK-NEXT:    [[TMP3:%.*]] = load <2 x double>, <2 x double>* [[TMP2]], align 8
-; CHECK-NEXT:    [[ISNAN0:%.*]] = tail call i1 @llvm.isnan.f64(double [[A0]])
-; CHECK-NEXT:    [[ISNAN1:%.*]] = tail call i1 @llvm.isnan.f64(double [[A1]])
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x i1> poison, i1 [[ISNAN0]], i32 0
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x i1> [[TMP4]], i1 [[ISNAN1]], i32 1
-; CHECK-NEXT:    [[TMP6:%.*]] = select <2 x i1> [[TMP5]], <2 x double> [[TMP1]], <2 x double> [[TMP3]]
-; CHECK-NEXT:    [[TMP7:%.*]] = bitcast double* [[D:%.*]] to <2 x double>*
-; CHECK-NEXT:    store <2 x double> [[TMP6]], <2 x double>* [[TMP7]], align 8
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast double* [[C:%.*]] to <2 x double>*
+; CHECK-NEXT:    [[TMP5:%.*]] = load <2 x double>, <2 x double>* [[TMP4]], align 8
+; CHECK-NEXT:    [[TMP6:%.*]] = call <2 x i1> @llvm.isnan.v2f64(<2 x double> [[TMP1]])
+; CHECK-NEXT:    [[TMP7:%.*]] = select <2 x i1> [[TMP6]], <2 x double> [[TMP3]], <2 x double> [[TMP5]]
+; CHECK-NEXT:    [[TMP8:%.*]] = bitcast double* [[D:%.*]] to <2 x double>*
+; CHECK-NEXT:    store <2 x double> [[TMP7]], <2 x double>* [[TMP8]], align 8
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %aidx1 = getelementptr inbounds double, double* %a, i64 1
   %bidx1 = getelementptr inbounds double, double* %b, i64 1
   %cidx1 = getelementptr inbounds double, double* %c, i64 1
   %didx1 = getelementptr inbounds double, double* %d, i64 1
   %a0 = load double, double* %a, align 8