[VP] Add vp.powi and a pass for expanding vp.powi before DAG.

[fakepaper56]

Summary

The patch uses different expanding way for vp.powi from the method of powi.
Vector powi is unrolled to multiple powi() libary calls in SelectionDAG, but the
method is not work for scalable vectors.
To support scalable vectors, the patch expands vp.powi at IR level. The
expanding way of vp.powi is based on compiler-rt/__powidf2.

[frasercrmck]

Maybe I missed the rationale, but why not use the ExpandVectorPredicationPass for this?

[fakepaper56]

Maybe I missed the rationale, but why not use the ExpandVectorPredicationPass for this?

LLVM could not expanding scalable vector type powi now. So this pass is not only for vp.powi, but also expanding scalable vector type powi in the future.

[craig.topper]

llvm/lib/CodeGen/ExpandPowi.cpp

35

expansion*

70

CreatePHI returns a PHINode*, can we use that to avoid casts?

124

support*

157

Why does this require AA?

[fakepaper56]

Address Craig's comment and add missing test case.

[fakepaper56]

llvm/lib/CodeGen/ExpandPowi.cpp

157

Sorry, they are my misuse.

[craig.topper]

llvm/lib/CodeGen/ExpandPowi.cpp

17

GlobalsModRef Probably uneeded?

[craig.topper]

llvm/lib/CodeGen/ExpandPowi.cpp

83

What's preventing using vp.icmp?

[fakepaper56]

Cleanup headers.

[fakepaper56]

llvm/lib/CodeGen/ExpandPowi.cpp

83

It is only that I don't know how to construct vp.icmp/vp.fcmp instructions.

[fakepaper56]

llvm/lib/CodeGen/ExpandPowi.cpp

83

I don't understand how make predicate to a pointer of Value.

[craig.topper]

llvm/lib/CodeGen/ExpandPowi.cpp

83

Should be something like this code from IRBuilder with the assert removed.

Value *getConstrainedFPPredicate(CmpInst::Predicate Predicate) {               
  assert(CmpInst::isFPPredicate(Predicate) &&                                  
         Predicate != CmpInst::FCMP_FALSE &&                                   
         Predicate != CmpInst::FCMP_TRUE &&                                    
         "Invalid constrained FP comparison predicate!");                      
                                                                               
  StringRef PredicateStr = CmpInst::getPredicateName(Predicate);               
  auto *PredicateMDS = MDString::get(Context, PredicateStr);                   
                                                                               
  return MetadataAsValue::get(Context, PredicateMDS);                          
}

[fakepaper56]

Use vp.icmp instead of icmp.

[fakepaper56]

llvm/lib/CodeGen/ExpandPowi.cpp

83

Thank you for the recommendation.

[fakepaper56]

Rebase and ping.

[craig.topper]

llvm/lib/CodeGen/ExpandPowi.cpp

114

old fixme?

132

Drop curly braces.

[craig.topper]

llvm/lib/CodeGen/ExpandPowi.cpp

58

why "forward"?

[fakepaper56]

Address Craig's comment.

[fakepaper56]

llvm/lib/CodeGen/ExpandPowi.cpp

58

Sorry, it didn't make sense. I changed it to powi-expansion-loop.

[frasercrmck]

In D143578#4113149, @fakepaper56 wrote:

Maybe I missed the rationale, but why not use the ExpandVectorPredicationPass for this?

LLVM could not expanding scalable vector type powi now. So this pass is not only for vp.powi, but also expanding scalable vector type powi in the future.

Apologies, I was away on holiday.

Thanks - I missed that the plan was also to support llvm.powi. I guess I just find ExpandPowi and ExpandVectorPredicationPass to be doing two very similar things (in this patch) with regards to vp.powi: expanding it into an equivalent set of operations; that seems unfortunate.

I get that scalable-vector llvm.powi is different, but so would many other scalable-vector intrinsics if the target doesn't support that operation: llvm.sin, llvm.cos, etc. So would we have passes for each intrinsic? If not, ExpandPowi seems too restrictive in its scope.

If we're supporting intrinsics, what about plain scalable-vector add on a target without scalable vectors, like x86?

I'd basically like to know how this fits in with some longer-term strategy about what we want to support for illegal scalable-vector operations, rather than this specific powi use-case. If we start to open the door to specific intrinsics, I think it'd help to have a well-defined rationale and plan in mind.

[fakepaper56]

In D143578#4140944, @frasercrmck wrote:

I get that scalable-vector llvm.powi is different, but so would many other scalable-vector intrinsics if the target doesn't support that operation: llvm.sin, llvm.cos, etc. So would we have passes for each intrinsic? If not, ExpandPowi seems too restrictive in its scope.

If we're supporting intrinsics, what about plain scalable-vector add on a target without scalable vectors, like x86?

I agree with you that only expanding powi is too restrictive. I think at least we should expand all the math function in a pass. But I don't have no idea that whether we should expand scalable operations for target without scalable vectors?

[craig.topper]

In D143578#4140944, @frasercrmck wrote:

In D143578#4113149, @fakepaper56 wrote:

Maybe I missed the rationale, but why not use the ExpandVectorPredicationPass for this?

LLVM could not expanding scalable vector type powi now. So this pass is not only for vp.powi, but also expanding scalable vector type powi in the future.

Apologies, I was away on holiday.

Thanks - I missed that the plan was also to support llvm.powi. I guess I just find ExpandPowi and ExpandVectorPredicationPass to be doing two very similar things (in this patch) with regards to vp.powi: expanding it into an equivalent set of operations; that seems unfortunate.

I get that scalable-vector llvm.powi is different, but so would many other scalable-vector intrinsics if the target doesn't support that operation: llvm.sin, llvm.cos, etc. So would we have passes for each intrinsic? If not, ExpandPowi seems too restrictive in its scope.

If we're supporting intrinsics, what about plain scalable-vector add on a target without scalable vectors, like x86?

I'd basically like to know how this fits in with some longer-term strategy about what we want to support for illegal scalable-vector operations, rather than this specific powi use-case. If we start to open the door to specific intrinsics, I think it'd help to have a well-defined rationale and plan in mind.

Note that this pass doesn't scalarize

In D143578#4141678, @fakepaper56 wrote:

In D143578#4140944, @frasercrmck wrote:

I get that scalable-vector llvm.powi is different, but so would many other scalable-vector intrinsics if the target doesn't support that operation: llvm.sin, llvm.cos, etc. So would we have passes for each intrinsic? If not, ExpandPowi seems too restrictive in its scope.

If we're supporting intrinsics, what about plain scalable-vector add on a target without scalable vectors, like x86?

I agree with you that only expanding powi is too restrictive. I think at least we should expand all the math function in a pass. But I don't have no idea that whether we should expand scalable operations for target without scalable vectors?

How would we expand the other math functions? Many of them are large and probably difficult to keep in vector form. We could scalarize them with a loop and use scalar libcalls. But that makes it very different than what we're doing for powi here.

How do envision sharing this code for llvm.powi. A lot of this code creates VP intrinsics. Do you have an abstraction plan?

[fakepaper56]

In D143578#4142322, @craig.topper wrote:

How do envision sharing this code for llvm.powi. A lot of this code creates VP intrinsics. Do you have an abstraction plan?

My plan is use same expanding function but use true mask for its mask and the elementcount for its evl.

[fakepaper56]

Also expanding llvm.powi.

[craig.topper]

No test for RISC-V?

llvm/test/CodeGen/Generic/expand-powi.ll

3

This needs a REQUIRES: x86-registered-target or it needs to be moved into the X86 directory.

[craig.topper]

llvm/lib/CodeGen/ExpandPowi.cpp

128

I think we should do a vp_icmp followed by a mask vp_reduce_or.

[craig.topper]

All the existing tests for llvm.powi use a scalar exponent even when the result is a vector. Should vp.powi only accept scalar exponent?

[fakepaper56]

I think we should follow rule of llvm.powi first.

[fakepaper56]

This update does,

1. Make vp.powi follows llvm.powi to only accept scalar exponent.
2. Add tests for RISC-V.
3. Update test cases.

But it still a test fail for ir unit test. I don't know how to debug it. I even
can not use gdb to trace it.
The below command about the test fails.

$ LLVM_SYMBOLIZER_PATH=./build/bin/llvm-symbolizer ./build/unittests/IR/./IRTests
...
[ RUN      ] VPIntrinsicTest.VPIntrinsicDeclarationForParams
IRTests: /home/yeting/x86-riscv-llvm/llvm/include/llvm/ADT/ArrayRef.h:255: const T& llvm::ArrayRef<T>::operator[](size_t) const [with T = llvm::Type*; size_t = long unsigned int]: Assertion `Index < Length && "Invalid index!"' failed.
 #0 0x00005620c5f909ee llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /home/yeting/x86-riscv-llvm/llvm/lib/Support/Unix/Signals.inc:567:22
 #1 0x00005620c5f90dc0 PrintStackTraceSignalHandler(void*) /home/yeting/x86-riscv-llvm/llvm/lib/Support/Unix/Signals.inc:641:1
 #2 0x00005620c5f8e4fe llvm::sys::RunSignalHandlers() /home/yeting/x86-riscv-llvm/llvm/lib/Support/Signals.cpp:104:20
 #3 0x00005620c5f9033f SignalHandler(int) /home/yeting/x86-riscv-llvm/llvm/lib/Support/Unix/Signals.inc:412:1
 #4 0x00007f3c933e0980 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12980)
 #5 0x00007f3c91d92e87 raise /build/glibc-uZu3wS/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:51:0
 #6 0x00007f3c91d947f1 abort /build/glibc-uZu3wS/glibc-2.27/stdlib/abort.c:81:0
 #7 0x00007f3c91d843fa __assert_fail_base /build/glibc-uZu3wS/glibc-2.27/assert/assert.c:89:0
 #8 0x00007f3c91d84472 (/lib/x86_64-linux-gnu/libc.so.6+0x30472)
 #9 0x00005620c5b9b1c6 llvm::ArrayRef<llvm::Type*>::operator[](unsigned long) const /home/yeting/x86-riscv-llvm/llvm/include/llvm/ADT/ArrayRef.h:256:14
#10 0x00005620c5cde47b DecodeFixedType(llvm::ArrayRef<llvm::Intrinsic::IITDescriptor>&, llvm::ArrayRef<llvm::Type*>, llvm::LLVMContext&) /home/yeting/x86-riscv-llvm/llvm/lib/IR/Function.cpp:1401:37
#11 0x00005620c5cdeae6 llvm::Intrinsic::getType(llvm::LLVMContext&, unsigned int, llvm::ArrayRef<llvm::Type*>) /home/yeting/x86-riscv-llvm/llvm/lib/IR/Function.cpp:1480:21
#12 0x00005620c5cf70c8 llvm::Intrinsic::getDeclaration(llvm::Module*, unsigned int, llvm::ArrayRef<llvm::Type*>) /home/yeting/x86-riscv-llvm/llvm/lib/IR/Function.cpp:1505:21
#13 0x00005620c5d49863 llvm::VPIntrinsic::getDeclarationForParams(llvm::Module*, unsigned int, llvm::Type*, llvm::ArrayRef<llvm::Value*>) /home/yeting/x86-riscv-llvm/llvm/lib/IR/IntrinsicInst.cpp:594:39
#14 0x00005620c56b2283 (anonymous namespace)::VPIntrinsicTest_VPIntrinsicDeclarationForParams_Test::TestBody() /home/yeting/x86-riscv-llvm/llvm/unittests/IR/VPIntrinsicTest.cpp:367:72

[craig.topper]

llvm/docs/LangRef.rst

19934 ↗

(On Diff #505490)

Predicated version of raising a vector of floating-point values to an integer power.

[craig.topper]

Fixed crash by adding special case in llvm::VPIntrinsic::getDeclarationForParams

[reames]

In D143578#4142322, @craig.topper wrote:

How would we expand the other math functions? Many of them are large and probably difficult to keep in vector form. We could scalarize them with a loop and use scalar libcalls.

I want to second this point. I think doing the fancy expansion here is a bad idea at this time. We can come back to that, but an initial implementation should scalarize via a loop. The lowering works for all of the lane-wise math routines. Only once we have correct lowering for the majority of the routines should we bother optimizing any of them.

Even then, I'm not convinced that inlining this loop is profitable over generating a runtime call to a new routine.

llvm/lib/CodeGen/ExpandPowi.cpp

36

This appears to correspond to the recently introduced IRBuilder::CreateElementCount.

[craig.topper]

In D143578#4197721, @reames wrote:

In D143578#4142322, @craig.topper wrote:

How would we expand the other math functions? Many of them are large and probably difficult to keep in vector form. We could scalarize them with a loop and use scalar libcalls.

I want to second this point. I think doing the fancy expansion here is a bad idea at this time. We can come back to that, but an initial implementation should scalarize via a loop. The lowering works for all of the lane-wise math routines. Only once we have correct lowering for the majority of the routines should we bother optimizing any of them.

Even then, I'm not convinced that inlining this loop is profitable over generating a runtime call to a new routine.

I want to mention that powi is weird and does not correspond to a real math routine. It's a fast math optimization for pow with an integer argument. The scalar version of powi is provided in libgcc/compiler-rt while pow itself is in libm. This almost makes it a compiler implementation detail. Should a vector math library provide this function?

[reames]

In D143578#4197800, @craig.topper wrote:

Even then, I'm not convinced that inlining this loop is profitable over generating a runtime call to a new routine.

I want to mention that powi is weird and does not correspond to a real math routine. It's a fast math optimization for pow with an integer argument. The scalar version of powi is provided in libgcc/compiler-rt while pow itself is in libm. This almost makes it a compiler implementation detail. Should a vector math library provide this function?

One of the options which was mentioned in the recent compiler-rt thread on discourse was to have a weak definition defined in each object file so that the linker could pick one (including the runtime libs if available). I'd lean towards something like that.

[fakepaper56]

Use CreateElementCount and fix typos in LangRef.rst.

[fakepaper56]

In D143578#4197817, @reames wrote:

One of the options which was mentioned in the recent compiler-rt thread on discourse was to have a weak definition defined in each object file so that the linker could pick one (including the runtime libs if available). I'd lean towards something like that.

Could you provide the link of the discourse you mentioned?

llvm/include/llvm/CodeGen/MachinePassRegistry.def

 FUNCTION_PASS("unreachableblockelim", UnreachableBlockElimPass, ())
 FUNCTION_PASS("consthoist", ConstantHoistingPass, ())
 FUNCTION_PASS("replace-with-veclib", ReplaceWithVeclib, ())
 FUNCTION_PASS("partially-inline-libcalls", PartiallyInlineLibCallsPass, ())
 FUNCTION_PASS("ee-instrument", EntryExitInstrumenterPass, (false))
 FUNCTION_PASS("post-inline-ee-instrument", EntryExitInstrumenterPass, (true))
 FUNCTION_PASS("expand-large-div-rem", ExpandLargeDivRemPass, ())
 FUNCTION_PASS("expand-large-fp-convert", ExpandLargeFpConvertPass, ())
+FUNCTION_PASS("expand-powi", ExpandPowiPass, ())
 FUNCTION_PASS("expand-reductions", ExpandReductionsPass, ())
 FUNCTION_PASS("expandvp", ExpandVectorPredicationPass, ())
 FUNCTION_PASS("lowerinvoke", LowerInvokePass, ())
 FUNCTION_PASS("scalarize-masked-mem-intrin", ScalarizeMaskedMemIntrinPass, ())
 FUNCTION_PASS("tlshoist", TLSVariableHoistPass, ())
 FUNCTION_PASS("verify", VerifierPass, ())
 #undef FUNCTION_PASS
 

llvm/include/llvm/CodeGen/Passes.h

namespace llvm {
   FunctionPass *createExpandVectorPredicationPass();
 
   // Expands large div/rem instructions.
   FunctionPass *createExpandLargeDivRemPass();
 
   // Expands large div/rem instructions.
   FunctionPass *createExpandLargeFpConvertPass();
 
+  // Expands powi instructions.
+  FunctionPass *createExpandPowiPass();
+
   // This pass expands memcmp() to load/stores.
   FunctionPass *createExpandMemCmpPass();
 
   /// Creates Break False Dependencies pass. \see BreakFalseDeps.cpp
   FunctionPass *createBreakFalseDeps();
 
   // This pass expands indirectbr instructions.
   FunctionPass *createIndirectBrExpandPass();

llvm/include/llvm/IR/Intrinsics.td

let IntrProperties = [IntrNoMem, IntrNoSync, IntrWillReturn] in {
   def int_vp_rint : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
                              [ LLVMMatchType<0>,
                                LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
                                llvm_i32_ty]>;
   def int_vp_nearbyint : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
                              [ LLVMMatchType<0>,
                                LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
                                llvm_i32_ty]>;
+  def int_vp_powi : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
+                             [ LLVMMatchType<0>,
+                               llvm_anyvector_ty,
+                               LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
+                               llvm_i32_ty]>;
 
   // Casts
   def int_vp_trunc : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
                              [ llvm_anyvector_ty,
                                LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
                                llvm_i32_ty]>;
   def int_vp_zext : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
                              [ llvm_anyvector_ty,

llvm/include/llvm/IR/VPIntrinsics.def

 // llvm.vp.rint(x,mask,vlen)
 BEGIN_REGISTER_VP(vp_rint, 1, 2, VP_FRINT, -1)
 END_REGISTER_VP(vp_rint, VP_FRINT)
 
 // llvm.vp.nearbyint(x,mask,vlen)
 BEGIN_REGISTER_VP(vp_nearbyint, 1, 2, VP_FNEARBYINT, -1)
 END_REGISTER_VP(vp_nearbyint, VP_FNEARBYINT)
 
+// llvm.vp.powi(x, y, mask,vlen)
+BEGIN_REGISTER_VP_INTRINSIC(vp_powi, 2, 3)
+VP_PROPERTY_BINARYOP
+END_REGISTER_VP_INTRINSIC(vp_powi)
 ///// } Floating-Point Arithmetic
 
 ///// Type Casts {
 // Specialized helper macro for type conversions.
 // <operation>(%x, %mask, %evl).
 #ifdef HELPER_REGISTER_FP_CAST_VP
 #error                                                                         \
     "The internal helper macro HELPER_REGISTER_FP_CAST_VP is already defined!"

llvm/include/llvm/InitializePasses.h

 void initializeEarlyTailDuplicatePass(PassRegistry&);
 void initializeEdgeBundlesPass(PassRegistry&);
 void initializeEHContGuardCatchretPass(PassRegistry &);
 void initializeEliminateAvailableExternallyLegacyPassPass(PassRegistry&);
 void initializeExpandLargeFpConvertLegacyPassPass(PassRegistry&);
 void initializeExpandLargeDivRemLegacyPassPass(PassRegistry&);
 void initializeExpandMemCmpPassPass(PassRegistry&);
 void initializeExpandPostRAPass(PassRegistry&);
+void initializeExpandPowiLegacyPassPass(PassRegistry &);
 void initializeExpandReductionsPass(PassRegistry&);
 void initializeExpandVectorPredicationPass(PassRegistry &);
 void initializeMakeGuardsExplicitLegacyPassPass(PassRegistry&);
 void initializeExternalAAWrapperPassPass(PassRegistry&);
 void initializeFEntryInserterPass(PassRegistry&);
 void initializeFinalizeISelPass(PassRegistry&);
 void initializeFinalizeMachineBundlesPass(PassRegistry&);
 void initializeFixIrreduciblePass(PassRegistry &);

llvm/lib/CodeGen/CMakeLists.txt

add_llvm_component_library(LLVMCodeGen
   EarlyIfConversion.cpp
   EdgeBundles.cpp
   EHContGuardCatchret.cpp
   ExecutionDomainFix.cpp
   ExpandLargeDivRem.cpp
   ExpandLargeFpConvert.cpp
   ExpandMemCmp.cpp
   ExpandPostRAPseudos.cpp
+  ExpandPowi.cpp
   ExpandReductions.cpp
   ExpandVectorPredication.cpp
   FaultMaps.cpp
   FEntryInserter.cpp
   FinalizeISel.cpp
   FixupStatepointCallerSaved.cpp
   FuncletLayout.cpp
   GCMetadata.cpp

llvm/lib/CodeGen/ExpandPowi.cpp

+//===--- ExpandPowi.cpp - Expand Powi intrinsics ---------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass implements IR expansion for powi/vp.powi. The expansion is based on
+// compiler-rt/__powidf2.c.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/CodeGen/TargetLowering.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/IR/Intrinsics.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+
+#define DEBUG_TYPE "expand-powi"
+
+using namespace llvm;
+
+// Helper function to generate Value for CmpInst::Predicate.
+// FIXME: Support createVPCmp in IRBuilderBase.
+static Value *getPredicateValue(LLVMContext &Context,
+                                CmpInst::Predicate Predicate) {
+  StringRef PredicateStr = CmpInst::getPredicateName(Predicate);
+  auto *PredicateMDS = MDString::get(Context, PredicateStr);
+  return MetadataAsValue::get(Context, PredicateMDS);
+}
+
+// The expansion is based on the c code of compiler-rt/__powidf2.c,
+// const int recip = b < 0;
+// double r = 1;
+// while (1) {
+//   if (b & 1)
+//     r *= a;
+//   b /= 2;
+//   if (b == 0)
+//     break;
+//   a *= a;
+// }
+// return recip ? 1 / r : r;
+static void expandPowi(IntrinsicInst *II) {
+  Value *OrigBase = II->getOperand(0);
+  Value *OrigExp = II->getOperand(1);
+  Value *Mask = II->getOperand(2);
+  Value *EVL = II->getOperand(3);
+
+  BasicBlock *PreLoopBB = II->getParent();
+  BasicBlock *PostLoopBB = PreLoopBB->splitBasicBlock(II, "powi-post-loop");
+  BasicBlock *LoopBody =
+      BasicBlock::Create(PreLoopBB->getContext(), "powi-forward-loop",
+                         PreLoopBB->getParent(), PostLoopBB);
+
+  IRBuilder<> Builder(PreLoopBB->getTerminator());
+  Builder.CreateBr(LoopBody);
+  PreLoopBB->getTerminator()->eraseFromParent();
+
+  Type *BaseTy = OrigBase->getType();
+  Type *ExpTy = OrigExp->getType();
+  Type *CondTy = ExpTy->getWithNewBitWidth(1);
+  Value *True = ConstantInt::get(CondTy, 1);
+  LLVMContext &C = II->getContext();
+
+  Builder.SetInsertPoint(LoopBody);
+  // Create phi of base.
+  PHINode *Base = Builder.CreatePHI(BaseTy, 2, "base");
+  Base->addIncoming(OrigBase, PreLoopBB);
+  // Create phi of exponent.
+  PHINode *Exp = Builder.CreatePHI(ExpTy, 2, "exp");
+  Exp->addIncoming(OrigExp, PreLoopBB);
+  // Create phi of res.
+  PHINode *Res = Builder.CreatePHI(BaseTy, 2, "res");
+  Res->addIncoming(ConstantFP::get(BaseTy, 1.), PreLoopBB);
+  // Res *= Base if Exp is odd.
+  Value *Tmp = Builder.CreateIntrinsic(BaseTy, Intrinsic::vp_fmul,
+                                       {Res, Base, True, EVL});
+  Value *And1 = Builder.CreateIntrinsic(
+      ExpTy, Intrinsic::vp_and, {Exp, ConstantInt::get(ExpTy, 1), True, EVL});
+  Value *PredicateNE = getPredicateValue(C, CmpInst::ICMP_NE);
+  Value *IsOdd = Builder.CreateIntrinsic(
+      CondTy, Intrinsic::vp_icmp,
+      {And1, ConstantInt::get(ExpTy, 0), PredicateNE, True, EVL});
+  Value *NewRes = Builder.CreateIntrinsic(BaseTy, Intrinsic::vp_select,
+                                          {IsOdd, Tmp, Res, EVL});
+  Res->addIncoming(NewRes, LoopBody);
+  // Update Exp.
+  Value *NewExp = Builder.CreateIntrinsic(
+      ExpTy, Intrinsic::vp_lshr, {Exp, ConstantInt::get(ExpTy, 1), True, EVL});
+  Exp->addIncoming(NewExp, LoopBody);
+  // Update Base.
+  Value *NewBase = Builder.CreateIntrinsic(BaseTy, Intrinsic::vp_fmul,
+                                           {Base, Base, True, EVL});
+  Base->addIncoming(NewBase, LoopBody);
+  // Check whether the elements of Exp are all zeros.
+  Type *ExpScalarTy = ExpTy->getScalarType();
+  Value *ScalarZero = ConstantInt::get(ExpScalarTy, 0);
+  Value *OrSum = Builder.CreateIntrinsic(ExpScalarTy, Intrinsic::vp_reduce_or,
+                                         {ScalarZero, NewExp, Mask, EVL});
+  Builder.CreateCondBr(Builder.CreateICmpEQ(OrSum, ScalarZero), PostLoopBB,
+                       LoopBody);
+
+  Builder.SetInsertPoint(&PostLoopBB->front());
+  // Use reciprocal if power is negative.
+  Value *Recip =
+      Builder.CreateIntrinsic(BaseTy, Intrinsic::vp_fdiv,
+                              {ConstantFP::get(BaseTy, 1.), NewRes, Mask, EVL});
+  // FIXME: Use vp.icmp.
+  Value *PredicateSLT = getPredicateValue(C, CmpInst::ICMP_SLT);
+  Value *IsNegative = Builder.CreateIntrinsic(
+      CondTy, Intrinsic::vp_icmp,
+      {OrigExp, ConstantInt::get(ExpTy, 0), PredicateSLT, True, EVL});
+  Value *Powi = Builder.CreateIntrinsic(BaseTy, Intrinsic::vp_select,
+                                        {IsNegative, Recip, NewRes, EVL});
+
+  II->replaceAllUsesWith(Powi);
+  II->eraseFromParent();
+}
+
+// TODO: Add cost model to skip small fixed vectors powi.
+static bool runImpl(Function &F) {
+  SmallVector<IntrinsicInst *, 4> Replace;
+  for (auto &I : instructions(F)) {
+    if (auto *II = dyn_cast<IntrinsicInst>(&I)) {
+      // TODO: Also support llvm.powi.
+      if (II->getIntrinsicID() == Intrinsic::vp_powi) {
+        Replace.push_back(II);
+      }
+    }
+  }
+
+  if (Replace.empty())
+    return false;
+
+  for (IntrinsicInst *II : Replace)
+    expandPowi(II);
+
+  return true;
+}
+
+namespace {
+class ExpandPowiLegacyPass : public FunctionPass {
+public:
+  static char ID;
+
+  ExpandPowiLegacyPass() : FunctionPass(ID) {
+    initializeExpandPowiLegacyPassPass(*PassRegistry::getPassRegistry());
+  }
+
+  bool runOnFunction(Function &F) override { return runImpl(F); }
+};
+} // namespace
+
+char ExpandPowiLegacyPass::ID = 0;
+INITIALIZE_PASS_BEGIN(ExpandPowiLegacyPass, "expand-powi",
+                      "Expand powi functions", false, false)
+INITIALIZE_PASS_END(ExpandPowiLegacyPass, "expand-powi",
+                    "Expand powi functions", false, false)
+
+FunctionPass *llvm::createExpandPowiPass() {
+  return new ExpandPowiLegacyPass();
+}

llvm/lib/CodeGen/TargetPassConfig.cpp

 bool TargetPassConfig::addISelPasses() {
   if (TM->useEmulatedTLS())
     addPass(createLowerEmuTLSPass());
 
   addPass(createPreISelIntrinsicLoweringPass());
   PM->add(createTargetTransformInfoWrapperPass(TM->getTargetIRAnalysis()));
   addPass(createExpandLargeDivRemPass());
   addPass(createExpandLargeFpConvertPass());
+  addPass(createExpandPowiPass());
   addIRPasses();
   addCodeGenPrepare();
   addPassesToHandleExceptions();
   addISelPrepare();
 
   return addCoreISelPasses();
 }
 

llvm/test/CodeGen/Generic/expand-powi.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=x86_64-unknown-linux-gnu -expand-powi -S < %s | FileCheck %s
+declare <vscale x 1 x float> @llvm.vp.powi.nxv1f32.nxv1i32(<vscale x 1 x float>, <vscale x 1 x i32>, <vscale x 1 x i1>, i32)
+define <vscale x 1 x float> @foo(<vscale x 1 x float> %a, <vscale x 1 x i32> %b, <vscale x 1 x i1> %m, i32 %evl) {
+; CHECK-LABEL: @foo(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[POWI_FORWARD_LOOP:%.*]]
+; CHECK:       powi-forward-loop:
+; CHECK-NEXT:    [[BASE:%.*]] = phi <vscale x 1 x float> [ [[A:%.*]], [[ENTRY:%.*]] ], [ [[TMP5:%.*]], [[POWI_FORWARD_LOOP]] ]
+; CHECK-NEXT:    [[EXP:%.*]] = phi <vscale x 1 x i32> [ [[B:%.*]], [[ENTRY]] ], [ [[TMP4:%.*]], [[POWI_FORWARD_LOOP]] ]
+; CHECK-NEXT:    [[RES:%.*]] = phi <vscale x 1 x float> [ shufflevector (<vscale x 1 x float> insertelement (<vscale x 1 x float> poison, float 1.000000e+00, i64 0), <vscale x 1 x float> poison, <vscale x 1 x i32> zeroinitializer), [[ENTRY]] ], [ [[TMP3:%.*]], [[POWI_FORWARD_LOOP]] ]
+; CHECK-NEXT:    [[TMP0:%.*]] = call <vscale x 1 x float> @llvm.vp.fmul.nxv1f32(<vscale x 1 x float> [[RES]], <vscale x 1 x float> [[BASE]], <vscale x 1 x i1> shufflevector (<vscale x 1 x i1> insertelement (<vscale x 1 x i1> poison, i1 true, i64 0), <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer), i32 [[EVL:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = call <vscale x 1 x i32> @llvm.vp.and.nxv1i32(<vscale x 1 x i32> [[EXP]], <vscale x 1 x i32> shufflevector (<vscale x 1 x i32> insertelement (<vscale x 1 x i32> poison, i32 1, i64 0), <vscale x 1 x i32> poison, <vscale x 1 x i32> zeroinitializer), <vscale x 1 x i1> shufflevector (<vscale x 1 x i1> insertelement (<vscale x 1 x i1> poison, i1 true, i64 0), <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer), i32 [[EVL]])
+; CHECK-NEXT:    [[TMP2:%.*]] = call <vscale x 1 x i1> @llvm.vp.icmp.nxv1i32(<vscale x 1 x i32> [[TMP1]], <vscale x 1 x i32> zeroinitializer, metadata !"ne", <vscale x 1 x i1> shufflevector (<vscale x 1 x i1> insertelement (<vscale x 1 x i1> poison, i1 true, i64 0), <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer), i32 [[EVL]])
+; CHECK-NEXT:    [[TMP3]] = call <vscale x 1 x float> @llvm.vp.select.nxv1f32(<vscale x 1 x i1> [[TMP2]], <vscale x 1 x float> [[TMP0]], <vscale x 1 x float> [[RES]], i32 [[EVL]])
+; CHECK-NEXT:    [[TMP4]] = call <vscale x 1 x i32> @llvm.vp.lshr.nxv1i32(<vscale x 1 x i32> [[EXP]], <vscale x 1 x i32> shufflevector (<vscale x 1 x i32> insertelement (<vscale x 1 x i32> poison, i32 1, i64 0), <vscale x 1 x i32> poison, <vscale x 1 x i32> zeroinitializer), <vscale x 1 x i1> shufflevector (<vscale x 1 x i1> insertelement (<vscale x 1 x i1> poison, i1 true, i64 0), <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer), i32 [[EVL]])
+; CHECK-NEXT:    [[TMP5]] = call <vscale x 1 x float> @llvm.vp.fmul.nxv1f32(<vscale x 1 x float> [[BASE]], <vscale x 1 x float> [[BASE]], <vscale x 1 x i1> shufflevector (<vscale x 1 x i1> insertelement (<vscale x 1 x i1> poison, i1 true, i64 0), <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer), i32 [[EVL]])
+; CHECK-NEXT:    [[TMP6:%.*]] = call i32 @llvm.vp.reduce.or.nxv1i32(i32 0, <vscale x 1 x i32> [[TMP4]], <vscale x 1 x i1> [[M:%.*]], i32 [[EVL]])
+; CHECK-NEXT:    [[TMP7:%.*]] = icmp eq i32 [[TMP6]], 0
+; CHECK-NEXT:    br i1 [[TMP7]], label [[POWI_POST_LOOP:%.*]], label [[POWI_FORWARD_LOOP]]
+; CHECK:       powi-post-loop:
+; CHECK-NEXT:    [[TMP8:%.*]] = call <vscale x 1 x float> @llvm.vp.fdiv.nxv1f32(<vscale x 1 x float> shufflevector (<vscale x 1 x float> insertelement (<vscale x 1 x float> poison, float 1.000000e+00, i64 0), <vscale x 1 x float> poison, <vscale x 1 x i32> zeroinitializer), <vscale x 1 x float> [[TMP3]], <vscale x 1 x i1> [[M]], i32 [[EVL]])
+; CHECK-NEXT:    [[TMP9:%.*]] = call <vscale x 1 x i1> @llvm.vp.icmp.nxv1i32(<vscale x 1 x i32> [[B]], <vscale x 1 x i32> zeroinitializer, metadata !"slt", <vscale x 1 x i1> shufflevector (<vscale x 1 x i1> insertelement (<vscale x 1 x i1> poison, i1 true, i64 0), <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer), i32 [[EVL]])
+; CHECK-NEXT:    [[TMP10:%.*]] = call <vscale x 1 x float> @llvm.vp.select.nxv1f32(<vscale x 1 x i1> [[TMP9]], <vscale x 1 x float> [[TMP8]], <vscale x 1 x float> [[TMP3]], i32 [[EVL]])
+; CHECK-NEXT:    ret <vscale x 1 x float> [[TMP10]]
+;
+entry:
+  %0 = call <vscale x 1 x float> @llvm.vp.powi.nxv1f32.nxv1i32(<vscale x 1 x float> %a, <vscale x 1 x i32> %b, <vscale x 1 x i1> %m, i32 %evl)
+  ret <vscale x 1 x float> %0
+}

llvm/tools/opt/opt.cpp

std::vector<StringRef> PassNameExact = {
       "view-regions",
       "view-regions-only",
       "select-optimize",
       "expand-large-div-rem",
       "structurizecfg",
       "fix-irreducible",
       "expand-large-fp-convert",
       "callbrprepare",
+      "expand-powi",
   };
   for (const auto &P : PassNamePrefix)
     if (Pass.startswith(P))
       return true;
   for (const auto &P : PassNameContain)
     if (Pass.contains(P))
       return true;
   return llvm::is_contained(PassNameExact, Pass);
int main(int argc, char **argv) {
   initializeTransformUtils(Registry);
   initializeInstCombine(Registry);
   initializeTarget(Registry);
   // For codegen passes, only passes that do IR to IR transformation are
   // supported.
   initializeExpandLargeDivRemLegacyPassPass(Registry);
   initializeExpandLargeFpConvertLegacyPassPass(Registry);
   initializeExpandMemCmpPassPass(Registry);
+  initializeExpandPowiLegacyPassPass(Registry);
   initializeScalarizeMaskedMemIntrinLegacyPassPass(Registry);
   initializeSelectOptimizePass(Registry);
   initializeCallBrPreparePass(Registry);
   initializeCodeGenPreparePass(Registry);
   initializeAtomicExpandPass(Registry);
   initializeRewriteSymbolsLegacyPassPass(Registry);
   initializeWinEHPreparePass(Registry);
   initializeDwarfEHPrepareLegacyPassPass(Registry);