Expand constrained FP POWI

[cameron.mcinally]

Summary

This is a continuation of D47491...

POWI is an oddball math intrinsic since its 2nd argument is always a scalar integer. Add a helper function to expand STRICT_FPOWI.

Also, update the SmallVector value for both ExpandStrictFPPOWI(...) and ExpandStrictFPOp(...). The new value, 32, was chosen assuming 1024b vectors with 32b FP being the smallest element size. Special thanks to Andy for decoding how SmallVector works.

[craig.topper]

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

1154

Why couldn't we just detect that the type of Op.getOperand(j) is a scalar type and skip the extract? Then you could handle POWI in in this code without the separate function. Maybe with some asserts to ensure that type is vector everything but the last operand of POWI.

1192

There's no reason for Opers to be a SmallVector.

You could just do

SDValue Opers[] = { Chain, Base, Power };

test/CodeGen/X86/vector-constrained-fp-intrinsics.ll

337

FMA is missing? I didn't catch it in the other patch.

342

sqrt.v4f64 doesn't appear to be used.

[craig.topper]

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

729

Should STRICT_FMUL/DIV/ADD/SUB be here even though I don't know how to test them? If some target did it someday, we would end up the default case which would do the wrong thing.

[cameron.mcinally]

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

729

Ah, you're right. That's a typo.

There could definitely be a RISC-like target without vector arith ops.

1154

Good call. I was hung up on checking the FPOWI opcode and missed the obvious...

test/CodeGen/X86/vector-constrained-fp-intrinsics.ll

337

I'll add those...

342

It's debugging junk. I'll remove it.

[cameron.mcinally]

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

729

These are also missing from SelectionDAGLegalize::LegalizeOp(...). I'll roll that change into this one, if there are no objections.

[cameron.mcinally]

Suffering a little bit of project creep fixing up bits of code I pass on the way, but I think I've incorporated all of Craig's suggestions. Thanks again, Craig.

Also added two FMA tests and pulled the RUN lines from test/CodeGen/X86/fp-intrinsics.ll. I hope that is okay...

[cameron.mcinally]

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

1154

I made this change, but purposefully did not add an ASSERT checking for POWI. I suspect the fix I made is general enough. And, we might possibly see similar arguments when we get to the CMP/CVT/etc intrinsics. Okay with you, Craig?

1192

This whole function was removed.

[cameron.mcinally]

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

1154

Should Op.getOperand(j) be snapped into a temp since there is more than one use? I'm not sure of the LLVM coding standards on that.

[craig.topper]

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

1154

Yeah start with

SDValue Oper = Op.getOperand(j)

Then if its a vector just

Oper = DAG.getNodE(ISD::EXTRACT_VECTOR_ELT, dl, EltVT, Oper, Idx)

Then you don't need the else at all.

[cameron.mcinally]

Create a temp for Op.getOperand(j).

[craig.topper]

LGTM

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

class VectorLegalizer {
   SDValue ExpandLoad(SDValue Op);
   SDValue ExpandStore(SDValue Op);
   SDValue ExpandFNEG(SDValue Op);
   SDValue ExpandFSUB(SDValue Op);
   SDValue ExpandBITREVERSE(SDValue Op);
   SDValue ExpandCTLZ(SDValue Op);
   SDValue ExpandCTTZ_ZERO_UNDEF(SDValue Op);
   SDValue ExpandStrictFPOp(SDValue Op);
+  SDValue ExpandStrictFPPOWI(SDValue Op);
   
   /// Implements vector promotion.
   ///
   /// This is essentially just bitcasting the operands to a different type and
   /// bitcasting the result back to the original type.
   SDValue Promote(SDValue Op);
 
   /// Implements [SU]INT_TO_FP vector promotion.
case ISD::SETCC:
     return UnrollVSETCC(Op);
   case ISD::BITREVERSE:
     return ExpandBITREVERSE(Op);
   case ISD::CTLZ:
   case ISD::CTLZ_ZERO_UNDEF:
     return ExpandCTLZ(Op);
   case ISD::CTTZ_ZERO_UNDEF:
     return ExpandCTTZ_ZERO_UNDEF(Op);
   case ISD::STRICT_FSQRT:
   case ISD::STRICT_FMA:
   case ISD::STRICT_FPOW:
   case ISD::STRICT_FSIN:
   case ISD::STRICT_FCOS:
   case ISD::STRICT_FEXP:
   case ISD::STRICT_FEXP2:
   case ISD::STRICT_FLOG:
   case ISD::STRICT_FLOG10:
   case ISD::STRICT_FLOG2:
   case ISD::STRICT_FRINT:
   case ISD::STRICT_FNEARBYINT:
     return ExpandStrictFPOp(Op);
+  case ISD::STRICT_FPOWI:
+    return ExpandStrictFPPOWI(Op);
   default:
     return DAG.UnrollVectorOp(Op.getNode());
   }
 }
 
 SDValue VectorLegalizer::ExpandSELECT(SDValue Op) {
   // Lower a select instruction where the condition is a scalar and the
   // operands are vectors. Lower this select to VSELECT and implement it
SDValue VectorLegalizer::ExpandStrictFPOp(SDValue Op) {
   EVT EltVT = VT.getVectorElementType();
   unsigned NumElems = VT.getVectorNumElements();
   unsigned NumOpers = Op.getNumOperands();
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   EVT ValueVTs[] = {EltVT, MVT::Other};
   SDValue Chain = Op.getOperand(0);
   SDLoc dl(Op);
 
-  SmallVector<SDValue, 8> OpValues;
-  SmallVector<SDValue, 8> OpChains;
+  SmallVector<SDValue, 32> OpValues;
+  SmallVector<SDValue, 32> OpChains;
   for (unsigned i = 0; i < NumElems; ++i) {
     SmallVector<SDValue, 4> Opers;
     SDValue Idx = DAG.getConstant(i, dl, 
                                   TLI.getVectorIdxTy(DAG.getDataLayout()));
 
     // The Chain is the first operand.
     Opers.push_back(Chain);
 
     // Now process the remaining operands. 
     for (unsigned j = 1; j < NumOpers; ++j) {
       SDValue Oper = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, 
                                  EltVT, Op.getOperand(j), Idx);
       Opers.push_back(Oper);
     }
  
     SDValue ScalarOp = DAG.getNode(Op->getOpcode(), dl, ValueVTs, Opers);
 
     OpValues.push_back(ScalarOp.getValue(0));
     OpChains.push_back(ScalarOp.getValue(1));
   }
 
   SDValue Result = DAG.getBuildVector(VT, dl, OpValues);
   SDValue NewChain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, OpChains);
 
   AddLegalizedOperand(Op.getValue(0), Result);
   AddLegalizedOperand(Op.getValue(1), NewChain);
 
   return NewChain;
 }
 
+SDValue VectorLegalizer::ExpandStrictFPPOWI(SDValue Op) {
+  EVT VT = Op.getValueType();
+  EVT EltVT = VT.getVectorElementType();
+  unsigned NumElems = VT.getVectorNumElements();
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+  EVT ValueVTs[] = {EltVT, MVT::Other};
+  SDValue Chain = Op.getOperand(0);
+  SDValue BaseVec = Op.getOperand(1);
+  SDValue Power = Op.getOperand(2);
+  SDLoc dl(Op);
+
+  SmallVector<SDValue, 32> OpValues;
+  SmallVector<SDValue, 32> OpChains;
+  for (unsigned i = 0; i < NumElems; ++i) {
+    SmallVector<SDValue, 3> Opers;
+    SDValue Idx = DAG.getConstant(i, dl,
+                                  TLI.getVectorIdxTy(DAG.getDataLayout()));
+    SDValue Base = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl,
+                               EltVT, BaseVec, Idx);
+
+    Opers.push_back(Chain);
+    Opers.push_back(Base);
+    Opers.push_back(Power);
+
+    SDValue ScalarOp = DAG.getNode(Op->getOpcode(), dl, ValueVTs, Opers);
+
+    OpValues.push_back(ScalarOp.getValue(0));
+    OpChains.push_back(ScalarOp.getValue(1));
+  }
+
+  SDValue Result = DAG.getBuildVector(VT, dl, OpValues);
+  SDValue NewChain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, OpChains);
+
+  AddLegalizedOperand(Op.getValue(0), Result);
+  AddLegalizedOperand(Op.getValue(1), NewChain);
+
+  return NewChain;
+}
+
 SDValue VectorLegalizer::UnrollVSETCC(SDValue Op) {
   EVT VT = Op.getValueType();
   unsigned NumElems = VT.getVectorNumElements();
   EVT EltVT = VT.getVectorElementType();
   SDValue LHS = Op.getOperand(0), RHS = Op.getOperand(1), CC = Op.getOperand(2);
   EVT TmpEltVT = LHS.getValueType().getVectorElementType();
   SDLoc dl(Op);
   SmallVector<SDValue, 8> Ops(NumElems);

test/CodeGen/X86/vector-constrained-fp-intrinsics.ll

entry:
   %pow = call <2 x double> @llvm.experimental.constrained.pow.v2f64(
                              <2 x double> <double 42.1, double 42.2>,
                              <2 x double> <double 3.0, double 3.0>,
                              metadata !"round.dynamic",
                              metadata !"fpexcept.strict")
   ret <2 x double> %pow
 }
 
+define <2 x double> @constrained_vector_powi() {
+; CHECK-LABEL: constrained_vector_powi:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    subq $24, %rsp
+; CHECK-NEXT:    .cfi_def_cfa_offset 32
+; CHECK-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
+; CHECK-NEXT:    movl $3, %edi
+; CHECK-NEXT:    callq __powidf2
+; CHECK-NEXT:    movaps %xmm0, (%rsp) # 16-byte Spill
+; CHECK-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
+; CHECK-NEXT:    movl $3, %edi
+; CHECK-NEXT:    callq __powidf2
+; CHECK-NEXT:    unpcklpd (%rsp), %xmm0 # 16-byte Folded Reload
+; CHECK-NEXT:    # xmm0 = xmm0[0],mem[0]
+; CHECK-NEXT:    addq $24, %rsp
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    retq
+entry:
+  %powi = call <2 x double> @llvm.experimental.constrained.powi.v2f64(
+                              <2 x double> <double 42.1, double 42.2>,
+                              i32 3,
+                              metadata !"round.dynamic",
+                              metadata !"fpexcept.strict")
+  ret <2 x double> %powi
+}
+
 define <2 x double> @constrained_vector_sin() {
 ; CHECK-LABEL: constrained_vector_sin:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    subq $24, %rsp
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
 ; CHECK-NEXT:    callq sin
 ; CHECK-NEXT:    movaps %xmm0, (%rsp) # 16-byte Spill
entry:
   %nearby = call <2 x double> @llvm.experimental.constrained.nearbyint.v2f64(
                                 <2 x double> <double 42.1, double 42.0>,
                                 metadata !"round.dynamic",
                                 metadata !"fpexcept.strict")
   ret <2 x double> %nearby
 }
 
 
 declare <2 x double> @llvm.experimental.constrained.fdiv.v2f64(<2 x double>, <2 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.fmul.v2f64(<2 x double>, <2 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.fadd.v2f64(<2 x double>, <2 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.fsub.v2f64(<2 x double>, <2 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.sqrt.v2f64(<2 x double>, metadata, metadata)
 declare <4 x double> @llvm.experimental.constrained.sqrt.v4f64(<4 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.pow.v2f64(<2 x double>, <2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.powi.v2f64(<2 x double>, i32, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.sin.v2f64(<2 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.cos.v2f64(<2 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.exp.v2f64(<2 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.exp2.v2f64(<2 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.log.v2f64(<2 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.log10.v2f64(<2 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.log2.v2f64(<2 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.rint.v2f64(<2 x double>, metadata, metadata)
 declare <2 x double> @llvm.experimental.constrained.nearbyint.v2f64(<2 x double>, metadata, metadata)