[TTI] Add a hook for specifying per-target defaults for Interleaved Accesses

[sbaranga]

Summary

This adds a hook to TTI which enables us to selectively turn on by default
interleaved access vectorization for targets on which we have have performed
the required benchmarking.

[rengolin]

Hi Silviu,

Do you plan to enable this for ARM/AArch64 soon?

lib/Transforms/Vectorize/LoopVectorize.cpp

3886

Why the use of getNumOccurrences?

[sbaranga]

In D11901#220399, @rengolin wrote:

Hi Silviu,

Do you plan to enable this for ARM/AArch64 soon?

Hi Renato,

I think we are very close. As far as I can see we only need a slight tuning of the cost model (increase the cost of insert/extract element for both backends).

Cheers,
Silviu

[sbaranga]

lib/Transforms/Vectorize/LoopVectorize.cpp

3886

We want to use EnableInterleavedMemAccess only when the user has specified this option on the command line, otherwise we want to use the target defaults.

As far as I can tell, we need to use getNumOccurences to do that.

[rengolin]

lib/Transforms/Vectorize/LoopVectorize.cpp

3886

But EnableInterleavedMemAccess is a boolean, so just using it as before should work, no?

if (EnableInterleavedMemAccesses ||
    TTI->enableInterleavedAccessVectorization())
  InterleaveInfo.analyzeInterleaving(Strides);

[sbaranga]

lib/Transforms/Vectorize/LoopVectorize.cpp

3886

I think the behaviour here is slightly different. With the above implementation you wouldn't be able to disable interleaved accesses for targets which have enabled it by default. On the other hand, we could when using getNumOccurences.

[rengolin]

lib/Transforms/Vectorize/LoopVectorize.cpp

3886

So, setting -no-enable-interleaved-mem-accesses would make that count to zero?

[sbaranga]

lib/Transforms/Vectorize/LoopVectorize.cpp

3886

Should -no-enable-interleaved-mem-accesses work?

I thought it was -enable-interleaved-mem-accesses=false, in which case the counter is 1 (and UseInterleaved is false).
And if we don't pass it in the counter is 0 and UseInterleaved is the target default.

[rengolin]

lib/Transforms/Vectorize/LoopVectorize.cpp

3886

Right, makes sense.

[sbaranga]

Thanks, r244449

include/llvm/Analysis/TargetTransformInfo.h

public:
 
   /// \brief Return true if switches should be turned into lookup tables for the
   /// target.
   bool shouldBuildLookupTables() const;
 
   /// \brief Don't restrict interleaved unrolling to small loops.
   bool enableAggressiveInterleaving(bool LoopHasReductions) const;
 
+  /// \brief Enable matching of interleaved access groups.
+  bool enableInterleavedAccessVectorization() const;
+
   /// \brief Return hardware support for population count.
   PopcntSupportKind getPopcntSupport(unsigned IntTyWidthInBit) const;
 
   /// \brief Return true if the hardware has a fast square-root instruction.
   bool haveFastSqrt(Type *Ty) const;
 
   /// \brief Return the expected cost of supporting the floating point operation
   /// of the specified type.
public:
   virtual bool isTruncateFree(Type *Ty1, Type *Ty2) = 0;
   virtual bool isZExtFree(Type *Ty1, Type *Ty2) = 0;
   virtual bool isProfitableToHoist(Instruction *I) = 0;
   virtual bool isTypeLegal(Type *Ty) = 0;
   virtual unsigned getJumpBufAlignment() = 0;
   virtual unsigned getJumpBufSize() = 0;
   virtual bool shouldBuildLookupTables() = 0;
   virtual bool enableAggressiveInterleaving(bool LoopHasReductions) = 0;
+  virtual bool enableInterleavedAccessVectorization() = 0;
   virtual PopcntSupportKind getPopcntSupport(unsigned IntTyWidthInBit) = 0;
   virtual bool haveFastSqrt(Type *Ty) = 0;
   virtual int getFPOpCost(Type *Ty) = 0;
   virtual int getIntImmCost(const APInt &Imm, Type *Ty) = 0;
   virtual int getIntImmCost(unsigned Opc, unsigned Idx, const APInt &Imm,
                             Type *Ty) = 0;
   virtual int getIntImmCost(Intrinsic::ID IID, unsigned Idx, const APInt &Imm,
                             Type *Ty) = 0;
public:
   unsigned getJumpBufAlignment() override { return Impl.getJumpBufAlignment(); }
   unsigned getJumpBufSize() override { return Impl.getJumpBufSize(); }
   bool shouldBuildLookupTables() override {
     return Impl.shouldBuildLookupTables();
   }
   bool enableAggressiveInterleaving(bool LoopHasReductions) override {
     return Impl.enableAggressiveInterleaving(LoopHasReductions);
   }
+  bool enableInterleavedAccessVectorization() override {
+    return Impl.enableInterleavedAccessVectorization();
+  }
   PopcntSupportKind getPopcntSupport(unsigned IntTyWidthInBit) override {
     return Impl.getPopcntSupport(IntTyWidthInBit);
   }
   bool haveFastSqrt(Type *Ty) override { return Impl.haveFastSqrt(Ty); }
 
   int getFPOpCost(Type *Ty) override { return Impl.getFPOpCost(Ty); }
 
   int getIntImmCost(const APInt &Imm, Type *Ty) override {

include/llvm/Analysis/TargetTransformInfoImpl.h

public:
   unsigned getJumpBufAlignment() { return 0; }
 
   unsigned getJumpBufSize() { return 0; }
 
   bool shouldBuildLookupTables() { return true; }
 
   bool enableAggressiveInterleaving(bool LoopHasReductions) { return false; }
 
+  bool enableInterleavedAccessVectorization() { return false; }
+
   TTI::PopcntSupportKind getPopcntSupport(unsigned IntTyWidthInBit) {
     return TTI::PSK_Software;
   }
 
   bool haveFastSqrt(Type *Ty) { return false; }
 
   unsigned getFPOpCost(Type *Ty) { return TargetTransformInfo::TCC_Basic; }
 

lib/Analysis/TargetTransformInfo.cpp

 bool TargetTransformInfo::shouldBuildLookupTables() const {
   return TTIImpl->shouldBuildLookupTables();
 }
 
 bool TargetTransformInfo::enableAggressiveInterleaving(bool LoopHasReductions) const {
   return TTIImpl->enableAggressiveInterleaving(LoopHasReductions);
 }
 
+bool TargetTransformInfo::enableInterleavedAccessVectorization() const {
+  return TTIImpl->enableInterleavedAccessVectorization();
+}
+
 TargetTransformInfo::PopcntSupportKind
 TargetTransformInfo::getPopcntSupport(unsigned IntTyWidthInBit) const {
   return TTIImpl->getPopcntSupport(IntTyWidthInBit);
 }
 
 bool TargetTransformInfo::haveFastSqrt(Type *Ty) const {
   return TTIImpl->haveFastSqrt(Ty);
 }

lib/Transforms/Vectorize/LoopVectorize.cpp

bool LoopVectorizationLegality::canVectorize() {
   collectLoopUniforms();
 
   DEBUG(dbgs() << "LV: We can vectorize this loop"
                << (LAI->getRuntimePointerChecking()->Need
                        ? " (with a runtime bound check)"
                        : "")
                << "!\n");
 
+  bool UseInterleaved = TTI->enableInterleavedAccessVectorization();
+
+  // If an override option has been passed in for interleaved accesses, use it.
+  if (EnableInterleavedMemAccesses.getNumOccurrences() > 0)
+    UseInterleaved = EnableInterleavedMemAccesses;
+
   // Analyze interleaved memory accesses.
-  if (EnableInterleavedMemAccesses)
+  if (UseInterleaved)
      InterleaveInfo.analyzeInterleaving(Strides);
 
   // Okay! We can vectorize. At this point we don't have any other mem analysis
   // which may limit our maximum vectorization factor, so just return true with
   // no restrictions.
   return true;
 }
 
 static Type *convertPointerToIntegerType(const DataLayout &DL, Type *Ty) {