[LoopDataPrefetch] Update an existing prefetch to same address to 'write' for a store

[jonpa]

Summary

If a previous read access in the loop has gotten a prefetch, that prefetch should be updated to 'write' in case a store is later seen to the same address.

I believe one should be careful not to emit store prefetches without relative certainty of benefit (targeted cache line), so this patch only does this in case the address is actually the same (PD == 0). I however also tried to do this with different spaces between the load and the store. On SystemZ (with '-min-prefetch-stride=128 -loop-prefetch-writes') I see

(unpatched) 1606 read / 601 write prefetches.
patch (same address): 1503 / 704.
patch, but within Cache Line Size: 1421 / 786
patch, but within CLS / 16: 1459 / 748

In summary, this seems to be worthwhile and safe even for just the identical address case. Perhaps CLS/16 is beneficial..?

[jonpa]

ping

[jonpa]

This change is included instead in https://reviews.llvm.org/D70228.

lib/Transforms/Scalar/LoopDataPrefetch.cpp

for (auto &I : *BB) {
             /// executed each iteration MemI is reached, by moving the
             /// prefetch up in the CFG if needed.
             BasicBlock *PrefetchBB = Pref.first->getParent();
             if (PrefetchBB != BB && DomPref) {
               BasicBlock *DomBB = DT->findNearestCommonDominator(PrefetchBB, BB);
               if (DomBB != PrefetchBB)
                 Pref.first->moveBefore(DomBB->getTerminator());
             }
+            /// If MemI is a store but the already emitted prefetch is for a
+            /// 'read', update the prefetch argument to 'write'.
+            bool HasLoadPrefetch =
+              cast<ConstantInt>(Pref.first->getOperand(1))->getZExtValue() == 0;
+            if (PD == 0 && HasLoadPrefetch && MemI->mayWriteToMemory()) {
+              Type *I32 = Type::getInt32Ty(BB->getContext());
+              Pref.first->setOperand(1, ConstantInt::get(I32, 1));
+            }
             DupPref = true;
             break;
           }
         }
       }
       if (DupPref)
         continue;
 

test/CodeGen/SystemZ/prefetch-04.ll

+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z14 -prefetch-distance=50 \
+; RUN:   -loop-prefetch-writes -stop-after=loop-data-prefetch | FileCheck %s
+;
+; Check that for a load followed by a store to the same address gets a single
+; write prefetch.
+;
+; CHECK-LABEL: for.body
+; CHECK: call void @llvm.prefetch.p0i8(i8* %scevgep{{.*}}, i32 1, i32 3, i32 1
+; CHECK-not: call void @llvm.prefetch
+
+define void @fun(i32* nocapture %Src, i32* nocapture readonly %Dst) {
+entry:
+  br label %for.body
+
+for.body:
+  %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next.9, %for.body ]
+  %arrayidx = getelementptr inbounds i32, i32* %Dst, i64 %indvars.iv
+  %0 = load i32, i32* %arrayidx, align 4
+  %a = add i32 %0, 128
+  store i32 %a, i32* %arrayidx, align 4
+  %indvars.iv.next.9 = add nuw nsw i64 %indvars.iv, 1600
+  %cmp.9 = icmp ult i64 %indvars.iv.next.9, 11200
+  br i1 %cmp.9, label %for.body, label %for.cond.cleanup
+
+for.cond.cleanup:
+  ret void
+}
+