Scalarizer: Support scalarizing intrinsics

[arsenm]

ping

[arsenm]

ping

[mehdi_amini]

lib/Transforms/Scalar/Scalarizer.cpp

110

It is not clear to me what is this doing?
(Why do you need the CI member for instance)

Looks like you mimic'd struct BinarySplitter, but simplifications make it moot.

412

Document the function

431

Is this dead code?

437

This seems to assume that every vector operand must have the same number of elements as the returned type?

439

Would this loop be equivalent to:

for (auto &Operand : I->operands()) {
  if (!Operand.getType()->isVectorTy())
    ScalarOperands[I] = CI.getOperand(I);
  else {
    Scattered[I] = scatter(&CI, CI.getOperand(I));
    assert(Scattered[I].size() == NumElems && "mismatched call operands");
  }
}

I'm not sure about hasVectorInstrinsicScalarOpd though...

445

I think you could pass the initial size to the ctor.
It may even be better to call reserve() on ScalarCallOps though as the next thing is clearing it.
I'd probably use reserve on Res as well and use push_back in the loop below.

462

A comment above the loop can be appreciated.

[arsenm]

Address comments

lib/Transforms/Scalar/Scalarizer.cpp

437

This is true for all of the current intrinsics, I can add a comment

439

Yes, the hasVectorInstrinsicScalarOpd is why this is over the index

445

That's what I originally did, but the scalar operand complicated using push_back, so now it always sets the appropriate index. I would expect such an intrinsic to fail isTriviallyScalarizable since a different vector size probably should be treated like the scalar operand index

[mehdi_amini]

LGTM, with two comments

lib/Transforms/Scalar/Scalarizer.cpp

437

The comment seems truncated (and too long for a single line), clang-format?

I'm surprised no intrinsic are breaking this, is it an assumption baked everywhere in the optimizer?
It makes the code fragile to future intrinsics.

439

Why using this hasVectorInstrinsicScalarOpd()?
I'm not even sure to understand its implementation by the way, it says Identifies if the intrinsic has a scalar operand but only check the intrinsic id for a specific list. That sounds quite fragile? The type seems much more likely to be "the source of truth".

[arsenm]

lib/Transforms/Scalar/Scalarizer.cpp

437

I don't think it's that surprising. There's no obvious scalarizaton/vectorization for something like that. Not many places really care about the general types of intrinsics besides the vectorizers.

439

It also checks the operand index. This is just the existing mechanism, I guess it would also work to check the type

[arsenm]

lib/Transforms/Scalar/Scalarizer.cpp

439

The use below needs it because it doesn't check the actual value, it only knows the index when picking between the vector and scalar args

[arsenm]

r276681

lib/Transforms/Scalar/Scalarizer.cpp

 // to expose optimization opportunities on the individual scalar operations.
 // It is mainly intended for targets that do not have vector units, but it
 // may also be useful for revectorizing code to different vector widths.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Transforms/Scalar.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/Analysis/VectorUtils.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstVisitor.h"
 #include "llvm/Pass.h"
 #include "llvm/Transforms/Utils/BasicBlockUtils.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "scalarizer"

 
 // Information about a load or store that we're scalarizing.
 struct VectorLayout {
   VectorLayout() : VecTy(nullptr), ElemTy(nullptr), VecAlign(0), ElemSize(0) {}
 
   // Return the alignment of element I.
   uint64_t getElemAlign(unsigned I) {
     return MinAlign(VecAlign, I * ElemSize);
   }
 
   // The type of the vector.
   VectorType *VecTy;
 
   // The type of each element.
   Type *ElemTy;
 
   // The alignment of the vector.
public:
   bool visitBinaryOperator(BinaryOperator &);
   bool visitGetElementPtrInst(GetElementPtrInst &);
   bool visitCastInst(CastInst &);
   bool visitBitCastInst(BitCastInst &);
   bool visitShuffleVectorInst(ShuffleVectorInst &);
   bool visitPHINode(PHINode &);
   bool visitLoadInst(LoadInst &);
   bool visitStoreInst(StoreInst &);
+  bool visitCallInst(CallInst &I);
 
   static void registerOptions() {
     // This is disabled by default because having separate loads and stores
     // makes it more likely that the -combiner-alias-analysis limits will be
     // reached.
     OptionRegistry::registerOption<bool, Scalarizer,
                                  &Scalarizer::ScalarizeLoadStore>(
         "scalarize-load-store",
         "Allow the scalarizer pass to scalarize loads and store", false);
   }
 
 private:
   Scatterer scatter(Instruction *, Value *);
   void gather(Instruction *, const ValueVector &);
   bool canTransferMetadata(unsigned Kind);
   void transferMetadata(Instruction *, const ValueVector &);
   bool getVectorLayout(Type *, unsigned, VectorLayout &, const DataLayout &);
   bool finish();
 
   template<typename T> bool splitBinary(Instruction &, const T &);
 
+  bool splitCall(CallInst &CI);
+
   ScatterMap Scattered;
   GatherList Gathered;
   unsigned ParallelLoopAccessMDKind;
   bool ScalarizeLoadStore;
 };
 
 char Scalarizer::ID = 0;
 } // end anonymous namespace
bool Scalarizer::splitBinary(Instruction &I, const Splitter &Split) {
   Res.resize(NumElems);
   for (unsigned Elem = 0; Elem < NumElems; ++Elem)
     Res[Elem] = Split(Builder, Op0[Elem], Op1[Elem],
                       I.getName() + ".i" + Twine(Elem));
   gather(&I, Res);
   return true;
 }
 
+static bool isTriviallyScalariable(Intrinsic::ID ID) {
+  return isTriviallyVectorizable(ID);
+}
+
+// All of the current scalarizable intrinsics only have one mangled type.
+static Function *getScalarIntrinsicDeclaration(Module *M,
+                                               Intrinsic::ID ID,
+                                               VectorType *Ty) {
+  return Intrinsic::getDeclaration(M, ID, { Ty->getScalarType() });
+}
+
+/// If a call to a vector typed intrinsic function, split into a scalar call per
+/// element if possible for the intrinsic.
+bool Scalarizer::splitCall(CallInst &CI) {
+  VectorType *VT = dyn_cast<VectorType>(CI.getType());
+  if (!VT)
+    return false;
+
+  Function *F = CI.getCalledFunction();
+  if (!F)
+    return false;
+
+  Intrinsic::ID ID = F->getIntrinsicID();
+  if (ID == Intrinsic::not_intrinsic || !isTriviallyScalariable(ID))
+    return false;
+
+  unsigned NumElems = VT->getNumElements();
+  unsigned NumArgs = CI.getNumArgOperands();
+
+  ValueVector ScalarOperands(NumArgs);
+  SmallVector<Scatterer, 8> Scattered(NumArgs);
+
+  Scattered.resize(NumArgs);
+
+  // Assumes that any vector type has the same number of elements as the return vector type, which is true for
+  for (unsigned I = 0; I != NumArgs; ++I) {
+    if (hasVectorInstrinsicScalarOpd(ID, I))
+      ScalarOperands[I] = CI.getOperand(I);
+    else {
+      Scattered[I] = scatter(&CI, CI.getOperand(I));
+      assert(Scattered[I].size() == NumElems && "mismatched call operands");
+    }
+  }
+
+  ValueVector Res(NumElems);
+  ValueVector ScalarCallOps(NumArgs);
+
+  Function *NewIntrin = getScalarIntrinsicDeclaration(F->getParent(), ID, VT);
+  IRBuilder<> Builder(&CI);
+
+  // Perform actual scalarization, taking care to preserve any scalar operands.
+  for (unsigned Elem = 0; Elem < NumElems; ++Elem) {
+    ScalarCallOps.clear();
+
+    for (unsigned J = 0; J != NumArgs; ++J) {
+      if (hasVectorInstrinsicScalarOpd(ID, J))
+        ScalarCallOps.push_back(ScalarOperands[J]);
+      else
+        ScalarCallOps.push_back(Scattered[J][Elem]);
+    }
+
+    Res[Elem] = Builder.CreateCall(NewIntrin, ScalarCallOps,
+                                   CI.getName() + ".i" + Twine(Elem));
+  }
+
+  gather(&CI, Res);
+  return true;
+}
+
 bool Scalarizer::visitSelectInst(SelectInst &SI) {
   VectorType *VT = dyn_cast<VectorType>(SI.getType());
   if (!VT)
     return false;
 
   unsigned NumElems = VT->getNumElements();
   IRBuilder<> Builder(&SI);
   Scatterer Op1 = scatter(&SI, SI.getOperand(1));
bool Scalarizer::visitStoreInst(StoreInst &SI) {
   for (unsigned I = 0; I < NumElems; ++I) {
     unsigned Align = Layout.getElemAlign(I);
     Stores[I] = Builder.CreateAlignedStore(Val[I], Ptr[I], Align);
   }
   transferMetadata(&SI, Stores);
   return true;
 }
 
+bool Scalarizer::visitCallInst(CallInst &CI) {
+  return splitCall(CI);
+}
+
 // Delete the instructions that we scalarized.  If a full vector result
 // is still needed, recreate it using InsertElements.
 bool Scalarizer::finish() {
   // The presence of data in Gathered or Scattered indicates changes
   // made to the Function.
   if (Gathered.empty() && Scattered.empty())
     return false;
   for (const auto &GMI : Gathered) {

test/Transforms/Scalarizer/intrinsics.ll

+; RUN: opt -S -scalarizer %s | FileCheck %s
+
+; Unary fp
+declare <2 x float> @llvm.sqrt.v2f32(<2 x float>)
+
+; Binary fp
+declare <2 x float> @llvm.minnum.v2f32(<2 x float>, <2 x float>)
+
+; Ternary fp
+declare <2 x float> @llvm.fma.v2f32(<2 x float>, <2 x float>, <2 x float>)
+
+; Binary int
+declare <2 x i32> @llvm.bswap.v2i32(<2 x i32>)
+
+; Unary int plus constant scalar operand
+declare <2 x i32> @llvm.ctlz.v2i32(<2 x i32>, i1)
+
+; Unary fp plus any scalar operand
+declare <2 x float> @llvm.powi.v2f32(<2 x float>, i32)
+
+; CHECK-LABEL: @scalarize_sqrt_v2f32(
+; CHECK: %sqrt.i0 = call float @llvm.sqrt.f32(float %x.i0)
+; CHECK: %sqrt.i1 = call float @llvm.sqrt.f32(float %x.i1)
+; CHECK: %sqrt.upto0 = insertelement <2 x float> undef, float %sqrt.i0, i32 0
+; CHECK: %sqrt = insertelement <2 x float> %sqrt.upto0, float %sqrt.i1, i32 1
+; CHECK: ret <2 x float> %sqrt
+define <2 x float> @scalarize_sqrt_v2f32(<2 x float> %x) #0 {
+  %sqrt = call <2 x float> @llvm.sqrt.v2f32(<2 x float> %x)
+  ret <2 x float> %sqrt
+}
+
+; CHECK-LABEL: @scalarize_minnum_v2f32(
+; CHECK: %minnum.i0 = call float @llvm.minnum.f32(float %x.i0, float %y.i0)
+; CHECK: %minnum.i1 = call float @llvm.minnum.f32(float %x.i1, float %y.i1)
+; CHECK: %minnum.upto0 = insertelement <2 x float> undef, float %minnum.i0, i32 0
+; CHECK: %minnum = insertelement <2 x float> %minnum.upto0, float %minnum.i1, i32 1
+; CHECK: ret <2 x float> %minnum
+define <2 x float> @scalarize_minnum_v2f32(<2 x float> %x, <2 x float> %y) #0 {
+  %minnum = call <2 x float> @llvm.minnum.v2f32(<2 x float> %x, <2 x float> %y)
+  ret <2 x float> %minnum
+}
+
+; CHECK-LABEL: @scalarize_fma_v2f32(
+; CHECK: %fma.i0 = call float @llvm.fma.f32(float %x.i0, float %y.i0, float %z.i0)
+; CHECK: %fma.i1 = call float @llvm.fma.f32(float %x.i1, float %y.i1, float %z.i1)
+; CHECK: %fma.upto0 = insertelement <2 x float> undef, float %fma.i0, i32 0
+; CHECK: %fma = insertelement <2 x float> %fma.upto0, float %fma.i1, i32 1
+; CHECK: ret <2 x float> %fma
+define <2 x float> @scalarize_fma_v2f32(<2 x float> %x, <2 x float> %y, <2 x float> %z) #0 {
+  %fma = call <2 x float> @llvm.fma.v2f32(<2 x float> %x, <2 x float> %y, <2 x float> %z)
+  ret <2 x float> %fma
+}
+
+; CHECK-LABEL: @scalarize_bswap_v2i32(
+; CHECK: %bswap.i0 = call i32 @llvm.bswap.i32(i32 %x.i0)
+; CHECK: %bswap.i1 = call i32 @llvm.bswap.i32(i32 %x.i1)
+; CHECK: %bswap.upto0 = insertelement <2 x i32> undef, i32 %bswap.i0, i32 0
+; CHECK: %bswap = insertelement <2 x i32> %bswap.upto0, i32 %bswap.i1, i32 1
+; CHECK: ret <2 x i32> %bswap
+define <2 x i32> @scalarize_bswap_v2i32(<2 x i32> %x) #0 {
+  %bswap = call <2 x i32> @llvm.bswap.v2i32(<2 x i32> %x)
+  ret <2 x i32> %bswap
+}
+
+; CHECK-LABEL: @scalarize_ctlz_v2i32(
+; CHECK: %ctlz.i0 = call i32 @llvm.ctlz.i32(i32 %x.i0, i1 true)
+; CHECK: %ctlz.i1 = call i32 @llvm.ctlz.i32(i32 %x.i1, i1 true)
+; CHECK: %ctlz.upto0 = insertelement <2 x i32> undef, i32 %ctlz.i0, i32 0
+; CHECK: %ctlz = insertelement <2 x i32> %ctlz.upto0, i32 %ctlz.i1, i32 1
+; CHECK: ret <2 x i32> %ctlz
+define <2 x i32> @scalarize_ctlz_v2i32(<2 x i32> %x) #0 {
+  %ctlz = call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> %x, i1 true)
+  ret <2 x i32> %ctlz
+}
+
+; CHECK-LABEL: @scalarize_powi_v2f32(
+; CHECK: %powi.i0 = call float @llvm.powi.f32(float %x.i0, i32 %y)
+; CHECK: %powi.i1 = call float @llvm.powi.f32(float %x.i1, i32 %y)
+; CHECK: %powi.upto0 = insertelement <2 x float> undef, float %powi.i0, i32 0
+; CHECK: %powi = insertelement <2 x float> %powi.upto0, float %powi.i1, i32 1
+; CHECK: ret <2 x float> %powi
+define <2 x float> @scalarize_powi_v2f32(<2 x float> %x, i32 %y) #0 {
+  %powi = call <2 x float> @llvm.powi.v2f32(<2 x float> %x, i32 %y)
+  ret <2 x float> %powi
+}