Align load/store for llvm.loop.aligned metadata

[m-happy]

Summary

This patch adds functionality 'llvm.loop.aligned' metadata. The load/store accesses are aligned to 32 bit for loops marked with 'llvm.loop.aligned'.
This is used by D69897

[Meinersbur]

This seems to be only be used by the vectorizer. Are there plans for meanings beyond LoopVectorize? If not, it should be reflected in the metadata name: llvm.loop.vectorize.aligned.enable.

The semantics just assume that the data is aligned, but does not make it aligned. This should be reflected in the metadata name.

I do not see why an alignment of 32 fits everybody. Why not making it configurable, e.g. { !"llvm.loop.vectorize.assume_alignment", i64 32 }?

A regression test is missing.

llvm/docs/LangRef.rst

5443

Unrelated?

Also, I think that reStructured Spinx warns if the underline is not long enough, and the bots are configured with fail-on-warnings.

llvm/lib/Transforms/Vectorize/LoadStoreVectorizer.cpp

207–208

[nit] Please complete remove commented source

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

806

[nit] unrelated

7316

Since this function is only used withing this translation unit, make it static.

Also add a doxygen comment explaining what this function does.

7316–7317

[style] Please run clang-format over your code (git clang-format HEAD)

7322

[style] The LLVM coding standard does not use almost-always-auto.

7328–7329

Please completely remove commented source

[hfinkel]

Do we need this at all? Can't we just emit @llvm.assume calls before the loop if we know the alignment? I've done this in the past and it's worked well (by using __builtin_assume_aligned from Clang).

[jdoerfert]

In D69900#1736112, @hfinkel wrote:

Do we need this at all? Can't we just emit @llvm.assume calls before the loop if we know the alignment? I've done this in the past and it's worked well (by using __builtin_assume_aligned from Clang).

I'm in favor of doing this (llvm.assume) in the frontedn so other passes can use the information as well without requiring them to understand yet another encoding.

llvm/docs/LangRef.rst

 0 disables vectorization:
 
 .. code-block:: llvm
 
    !0 = !{!"llvm.loop.vectorize.enable", i1 0}
    !1 = !{!"llvm.loop.vectorize.enable", i1 1}
 
 '``llvm.loop.vectorize.predicate.enable``' Metadata
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This metadata selectively enables or disables creating predicated instructions
 for the loop, which can enable folding of the scalar epilogue loop into the
 main loop. The first operand is the string
 ``llvm.loop.vectorize.predicate.enable`` and the second operand is a bit. If
 the bit operand value is 1 vectorization is enabled. A value of 0 disables
 vectorization:
 
 .. code-block:: llvm
 
    !0 = !{!"llvm.loop.vectorize.predicate.enable", i1 0}
    !1 = !{!"llvm.loop.vectorize.predicate.enable", i1 1}
 
+'``llvm.loop.aligned.enable``' Metadata
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This metadata selectively enables or disables creating aligned 
+instructions
+for the loop, the aligned store instruction can be generated for 
+load/store into the 
+main loop. The first operand is the string 
+``llvm.loop.aligned.enable`` and the second operand is a bit. If the
+bit operand value is 1 aligned store instruction is enabled. A value
+of 0 disables aligned load/store:
+
+.. code-block::llvm
+
+    !0 = !{!"llvm.loop.aligned.enable", i1 1}
+    !1 = !{!"llvm.loop.aligned.disable", i1 0}
+
+
+
 '``llvm.loop.vectorize.width``' Metadata
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This metadata sets the target width of the vectorizer. The first
 operand is the string ``llvm.loop.vectorize.width`` and the second
 operand is an integer specifying the width. For example:
 
 .. code-block:: llvm

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

 /// and can, upon request, write them back as metadata on the loop. It will
 /// initially scan the loop for existing metadata, and will update the local
 /// values based on information in the loop.
 /// We cannot write all values to metadata, as the mere presence of some info,
 /// for example 'force', means a decision has been made. So, we need to be
 /// careful NOT to add them if the user hasn't specifically asked so.
 class LoopVectorizeHints {
   enum HintKind { HK_WIDTH, HK_UNROLL, HK_FORCE, HK_ISVECTORIZED,
-                  HK_PREDICATE };
+                  HK_PREDICATE, HK_ALIGNED };
 
   /// Hint - associates name and validation with the hint value.
   struct Hint {
     const char *Name;
     unsigned Value; // This may have to change for non-numeric values.
     HintKind Kind;
 
     Hint(const char *Name, unsigned Value, HintKind Kind)
class LoopVectorizeHints {
   Hint Force;
 
   /// Already Vectorized
   Hint IsVectorized;
 
   /// Vector Predicate
   Hint Predicate;
 
+  /// references aligned
+  Hint IsAligned;
+
   /// Return the loop metadata prefix.
   static StringRef Prefix() { return "llvm.loop."; }
 
   /// True if there is any unsafe math in the loop.
   bool PotentiallyUnsafe = false;
 
 public:
   enum ForceKind {
public:
 
   /// Dumps all the hint information.
   void emitRemarkWithHints() const;
 
   unsigned getWidth() const { return Width.Value; }
   unsigned getInterleave() const { return Interleave.Value; }
   unsigned getIsVectorized() const { return IsVectorized.Value; }
   unsigned getPredicate() const { return Predicate.Value; }
+  unsigned getAligned() const { return IsAligned.Value; }
   enum ForceKind getForce() const {
     if ((ForceKind)Force.Value == FK_Undefined &&
         hasDisableAllTransformsHint(TheLoop))
       return FK_Disabled;
     return (ForceKind)Force.Value;
   }
 
   /// If hints are provided that force vectorization, use the AlwaysPrint

llvm/lib/Transforms/Vectorize/LoadStoreVectorizer.cpp

private:
   /// Vectorizes the store instructions in Chain.
   bool
   vectorizeStoreChain(ArrayRef<Instruction *> Chain,
                       SmallPtrSet<Instruction *, 16> *InstructionsProcessed);
 
   /// Check if this load/store access is misaligned accesses.
   bool accessIsMisaligned(unsigned SzInBytes, unsigned AddressSpace,
                           unsigned Alignment);
+
+  // /// Align all the array references
+  // bool alignArrayAccess(ArrayRef<Instruction *> Instrs, const DataLayout &DL);
 };
 
 class LoadStoreVectorizerLegacyPass : public FunctionPass {
 public:
   static char ID;
 
   LoadStoreVectorizerLegacyPass() : FunctionPass(ID) {
     initializeLoadStoreVectorizerLegacyPassPass(*PassRegistry::getPassRegistry());

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

bool LoopVectorizeHints::Hint::validate(unsigned Val) {
   case HK_WIDTH:
     return isPowerOf2_32(Val) && Val <= VectorizerParams::MaxVectorWidth;
   case HK_UNROLL:
     return isPowerOf2_32(Val) && Val <= MaxInterleaveFactor;
   case HK_FORCE:
     return (Val <= 1);
   case HK_ISVECTORIZED:
   case HK_PREDICATE:
+  case HK_ALIGNED:
     return (Val == 0 || Val == 1);
   }
   return false;
 }
 
 LoopVectorizeHints::LoopVectorizeHints(const Loop *L,
                                        bool InterleaveOnlyWhenForced,
                                        OptimizationRemarkEmitter &ORE)
     : Width("vectorize.width", VectorizerParams::VectorizationFactor, HK_WIDTH),
       Interleave("interleave.count", InterleaveOnlyWhenForced, HK_UNROLL),
       Force("vectorize.enable", FK_Undefined, HK_FORCE),
       IsVectorized("isvectorized", 0, HK_ISVECTORIZED),
-      Predicate("vectorize.predicate.enable", 0, HK_PREDICATE), TheLoop(L),
+      Predicate("vectorize.predicate.enable", 0, HK_PREDICATE),
+      IsAligned("aligned.enable", 0, HK_ALIGNED), TheLoop(L),
       ORE(ORE) {
   // Populate values with existing loop metadata.
   getHintsFromMetadata();
 
   // force-vector-interleave overrides DisableInterleaving.
   if (VectorizerParams::isInterleaveForced())
     Interleave.Value = VectorizerParams::VectorizationInterleave;
 
if (!Name.startswith(Prefix()))
     return;
   Name = Name.substr(Prefix().size(), StringRef::npos);
 
   const ConstantInt *C = mdconst::dyn_extract<ConstantInt>(Arg);
   if (!C)
     return;
   unsigned Val = C->getZExtValue();
 
-  Hint *Hints[] = {&Width, &Interleave, &Force, &IsVectorized, &Predicate};
+  Hint *Hints[] = {&Width, &Interleave, &Force, &IsVectorized, &Predicate,
+                    &IsAligned };
   for (auto H : Hints) {
     if (Name == H->Name) {
       if (H->validate(Val))
         H->Value = Val;
       else
         LLVM_DEBUG(dbgs() << "LV: ignoring invalid hint '" << Name << "'\n");
       break;
     }

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

if (DIL && Inst->getFunction()->isDebugInfoForProfiling() &&
                    << DIL->getFilename() << " Line: " << DIL->getLine());
     }
     else
       B.SetCurrentDebugLocation(DIL);
   } else
     B.SetCurrentDebugLocation(DebugLoc());
 }
 
+
 /// Write a record \p DebugMsg about vectorization failure to the debug
 /// output stream. If \p I is passed, it is an instruction that prevents
 /// vectorization.
 #ifndef NDEBUG
 static void debugVectorizationFailure(const StringRef DebugMsg,
     Instruction *I) {
   dbgs() << "LV: Not vectorizing: " << DebugMsg;
   if (I != nullptr)
if (Hints.getForce() != LoopVectorizeHints::FK_Enabled &&
        llvm::shouldOptimizeForSize(L->getHeader(), PSI, BFI)))
     SEL = CM_ScalarEpilogueNotAllowedOptSize;
   else if (PreferPredicateOverEpilog || Hints.getPredicate()) 
     SEL = CM_ScalarEpilogueNotNeededUsePredicate;
 
   return SEL;
 }
 
+void alignAccess(Loop *L,
+          LoopVectorizeHints &Hints){
+
+  if(!(Hints.getAligned())){
+    return;
+  }
+  for(auto *BB : L->getBlocks()){
+    for(auto &I : *BB){
+      LoadInst *LI = dyn_cast<LoadInst>(&I);
+      StoreInst *SI = dyn_cast<StoreInst>(&I);
+      //Handle stores
+      if(SI){
+        // LLVM_DEBUG(dbgs()<<"Store "<<++st<<"\n");
+        // CreateAlignedLoad(SI->getType, SI->getPointerElementType(), 32, "aligned.load");
+        SI->setAlignment(32);
+        // unsigned Alignment = getLoadStoreAlignment(SI);
+        //   LLVM_DEBUG(dbgs()<<" alignment of store after" << Alignment<<"\n");
+      }
+      //Handle load/store
+      if(LI){
+        // LLVM_DEBUG(dbgs()<<"load "<<++ld<<"\n");
+        LI->setAlignment(32);
+          // unsigned Alignment = getLoadStoreAlignment(LI);
+          // LLVM_DEBUG(dbgs()<<" alignment of load after setting" << Alignment<<"\n");
+      }
+
+    }
+  }
+}
+
+
 // Process the loop in the VPlan-native vectorization path. This path builds
 // VPlan upfront in the vectorization pipeline, which allows to apply
 // VPlan-to-VPlan transformations from the very beginning without modifying the
 // input LLVM IR.
 static bool processLoopInVPlanNativePath(
     Loop *L, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT,
     LoopVectorizationLegality *LVL, TargetTransformInfo *TTI,
     TargetLibraryInfo *TLI, DemandedBits *DB, AssumptionCache *AC,
LLVM_DEBUG(
       dbgs() << "LV: Loop hints:"
              << " force="
              << (Hints.getForce() == LoopVectorizeHints::FK_Disabled
                      ? "disabled"
                      : (Hints.getForce() == LoopVectorizeHints::FK_Enabled
                             ? "enabled"
                             : "?"))
              << " width=" << Hints.getWidth()
-             << " unroll=" << Hints.getInterleave() << "\n");
+             << " unroll=" << Hints.getInterleave()
+             << " aligned=" << Hints.getAligned()<<"\n");
 
   // Function containing loop
   Function *F = L->getHeader()->getParent();
 
   // Looking at the diagnostic output is the only way to determine if a loop
   // was vectorized (other than looking at the IR or machine code), so it
   // is important to generate an optimization remark for each loop. Most of
   // these messages are generated as OptimizationRemarkAnalysis. Remarks
ORE->emit([&]() {
                                 L->getHeader())
              << "interleaved loop (interleaved count: "
              << NV("InterleaveCount", IC) << ")";
     });
   } else {
     // If we decided that it is *legal* to vectorize the loop, then do it.
     InnerLoopVectorizer LB(L, PSE, LI, DT, TLI, TTI, AC, ORE, VF.Width, IC,
                            &LVL, &CM);
+    alignAccess(L, Hints);
     LVP.executePlan(LB, DT);
     ++LoopsVectorized;
 
     // Add metadata to disable runtime unrolling a scalar loop when there are
     // no runtime checks about strides and memory. A scalar loop that is
     // rarely used is not worth unrolling.
     if (!LB.areSafetyChecksAdded())
       DisableRuntimeUnroll = true;