[LV] Improve register pressure estimation if MaxLocalUsers is zero

[yrouban]

Summary

Do not limit LoopVectorize interleave count using MaxLocalUsers in case MaxLocalUsers is zero.

[fhahn]

Do you have any performance data motivating the change and ruling out any regressions?

llvm/test/Transforms/LoopVectorize/X86/interleave-count.ll

38

Please update the test to use opaque pointers. Also, it would be good to put up a patch to just add the test separately and then only include the changes caused by the patch in the diff

[yrouban]

In D147434#4243671, @fhahn wrote:

Do you have any performance data motivating the change and ruling out any regressions?

No. I investigated an AVX512 memset code generated by a non-llvm compiler with vectorized move instruction using zmm register and unrolled as if it had interleave count 16. Trying to achieve the same result with LoopVectorize I have found this nit. This is not a problem as long as the number of vector registers is big enough, that is even if decremented and bit-floored but still bigger than the other interleave count limits (e.g. X86TTIImpl::getMaxInterleaveFactor() returns 4 for AVX).

llvm/test/Transforms/LoopVectorize/X86/interleave-count.ll

38

Done. See D147588.

[yrouban]

extracted the test to a separate patch.

[yrouban]

just rebased over the updated D147588

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

if (VF.isScalar()) {
       if (ForceTargetNumVectorRegs.getNumOccurrences() > 0)
         TargetNumRegisters = ForceTargetNumVectorRegs;
     }
     unsigned MaxLocalUsers = pair.second;
     unsigned LoopInvariantRegs = 0;
     if (R.LoopInvariantRegs.find(pair.first) != R.LoopInvariantRegs.end())
       LoopInvariantRegs = R.LoopInvariantRegs[pair.first];
 
-    unsigned TmpIC = llvm::bit_floor((TargetNumRegisters - LoopInvariantRegs) /
-                                     MaxLocalUsers);
-    // Don't count the induction variable as interleaved.
     if (EnableIndVarRegisterHeur) {
-      TmpIC = llvm::bit_floor((TargetNumRegisters - LoopInvariantRegs - 1) /
-                              std::max(1U, (MaxLocalUsers - 1)));
-    }
-
+      // Treat the induction variable as a LoopInvariantReg.
+      assert(MaxLocalUsers);
+      --MaxLocalUsers;
+      ++LoopInvariantRegs;
+    }
+    if (MaxLocalUsers > 0) {
+      unsigned TmpIC = llvm::bit_floor(
+          (TargetNumRegisters - LoopInvariantRegs) / MaxLocalUsers);
       IC = std::min(IC, TmpIC);
     }
+  }
 
   // Clamp the interleave ranges to reasonable counts.
   unsigned MaxInterleaveCount = TTI.getMaxInterleaveFactor(VF);
 
   // Check if the user has overridden the max.
   if (VF.isScalar()) {
     if (ForceTargetMaxScalarInterleaveFactor.getNumOccurrences() > 0)
       MaxInterleaveCount = ForceTargetMaxScalarInterleaveFactor;

llvm/test/Transforms/LoopVectorize/X86/interleave-count.ll

 ; REQUIRES: asserts
 ;
 ; RUN: opt -S -passes=loop-vectorize -debug-only=loop-vectorize \
 ; RUN:     -mtriple=x86_64-pc_linux -mcpu=cascadelake \
 ; RUN:     -force-target-max-vector-interleave=16 -force-target-num-vector-regs=16 \
 ; RUN:     %s 2>&1 | FileCheck %s
 
-; Check that the interleave count is limited by 8 even if there is no
+; Check that the interleave count is not limited by 8 if there is no
 ; register use except one induction variable.
 define void @test(ptr %dst, i64 %size) {
 ; CHECK-LABEL: LV: Checking a loop in 'test'
-; CHECK:       LV: IC is 8
+; CHECK:       LV: IC is 16
 ;
 ; CHECK-LABEL: define void @test
 ;
-; Number of @llvm.masked.scatter() calls is 8.
+; Number of @llvm.masked.scatter() calls is 16.
 ; CHECK:         call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
 ; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
 ; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
 ; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
 ; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
 ; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
 ; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
 ; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
 ; CHECK-NOT:     call void @llvm.masked.scatter.v4f64.v4p0(<4 x double>
 ;
 entry:
   %p.end = getelementptr inbounds double, ptr %dst, i64 %size
   br label %loop
 
 loop:
   %p = phi ptr [%dst, %entry], [%p.next, %loop]
   store double -1.000000e+00, ptr %p, align 8
   %p.next = getelementptr inbounds double, ptr %p, i64 8
   %done = icmp eq ptr %p.next, %p.end
   br i1 %done, label %exit, label %loop
 
 exit:
   ret void
 }