[LV] Improve register pressure estimation if MaxLocalUsers is zero

[yrouban]

Summary

Do not limit LoopVectorize interleave count using MaxLocalUsers in case MaxLocalUsers is zero.

[fhahn]

Do you have any performance data motivating the change and ruling out any regressions?

llvm/test/Transforms/LoopVectorize/X86/interleave-count.ll

38

Please update the test to use opaque pointers. Also, it would be good to put up a patch to just add the test separately and then only include the changes caused by the patch in the diff

[yrouban]

In D147434#4243671, @fhahn wrote:

Do you have any performance data motivating the change and ruling out any regressions?

No. I investigated an AVX512 memset code generated by a non-llvm compiler with vectorized move instruction using zmm register and unrolled as if it had interleave count 16. Trying to achieve the same result with LoopVectorize I have found this nit. This is not a problem as long as the number of vector registers is big enough, that is even if decremented and bit-floored but still bigger than the other interleave count limits (e.g. X86TTIImpl::getMaxInterleaveFactor() returns 4 for AVX).

llvm/test/Transforms/LoopVectorize/X86/interleave-count.ll

38

Done. See D147588.

[yrouban]

extracted the test to a separate patch.

[yrouban]

just rebased over the updated D147588

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

if (VF.isScalar()) {
       if (ForceTargetNumVectorRegs.getNumOccurrences() > 0)
         TargetNumRegisters = ForceTargetNumVectorRegs;
     }
     unsigned MaxLocalUsers = pair.second;
     unsigned LoopInvariantRegs = 0;
     if (R.LoopInvariantRegs.find(pair.first) != R.LoopInvariantRegs.end())
       LoopInvariantRegs = R.LoopInvariantRegs[pair.first];
 
-    unsigned TmpIC = llvm::bit_floor((TargetNumRegisters - LoopInvariantRegs) /
-                                     MaxLocalUsers);
-    // Don't count the induction variable as interleaved.
     if (EnableIndVarRegisterHeur) {
-      TmpIC = llvm::bit_floor((TargetNumRegisters - LoopInvariantRegs - 1) /
-                              std::max(1U, (MaxLocalUsers - 1)));
-    }
-
+      // Treat the induction variable as a LoopInvariantReg.
+      assert(MaxLocalUsers);
+      --MaxLocalUsers;
+      ++LoopInvariantRegs;
+    }
+    if (MaxLocalUsers > 0) {
+      unsigned TmpIC = llvm::bit_floor(
+          (TargetNumRegisters - LoopInvariantRegs) / MaxLocalUsers);
       IC = std::min(IC, TmpIC);
     }
+  }
 
   // Clamp the interleave ranges to reasonable counts.
   unsigned MaxInterleaveCount = TTI.getMaxInterleaveFactor(VF);
 
   // Check if the user has overridden the max.
   if (VF.isScalar()) {
     if (ForceTargetMaxScalarInterleaveFactor.getNumOccurrences() > 0)
       MaxInterleaveCount = ForceTargetMaxScalarInterleaveFactor;

llvm/test/Transforms/LoopVectorize/X86/interleave-count.ll

+; REQUIRES: asserts
+;
+; RUN: opt -S -passes=loop-vectorize -debug-only=loop-vectorize -debug-only=loop-accesses \
+; RUN:     -mtriple=x86_64-pc_linux -mcpu=cascadelake \
+; RUN:     -force-target-max-vector-interleave=16 -force-target-num-vector-regs=16 \
+; RUN:     %s 2>&1 | FileCheck %s
+
+define void @test(double *%dst, i64 %len) {
+; CHECK-LABEL: LV: Checking a loop in 'test'
+; CHECK:       LV: IC is 16
+;
+; CHECK-LABEL: define void @test
+;
+; Number of @llvm.masked.scatter() calls is 16.
+; CHECK:         call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+;
+entry:
+  %size = shl i64 %len, 3
+  %p.end = getelementptr inbounds double, ptr %dst, i64 %size
+  br label %loop
+
+loop:
+  %p = phi double * [%dst, %entry], [%p.next, %loop]
+  store double -1.000000e+00, ptr %p, align 8
+  %p.next = getelementptr inbounds double, ptr %p, i64 8
+  %done = icmp eq ptr %p.next, %p.end
+  br i1 %done, label %exit, label %loop, !prof !{!"branch_weights", i32 1, i32 1000000}
+
+exit:
+  ret void
+}