Unroll for uncountable loops

[evstupac]

Summary

The patch introduces 1st heuristic to unroll uncountable loops.
The heuristics targeted on loops working with list structures.

make check passed
spec2000 has unchanged performance
There are several improvements on applications working with list structures.

[mzolotukhin]

Hi Evgeny,

I haven't looked into the details of the patch yet, but (at least from the function name) it looks like this is very similar to what LoopUnrollAnalyzer does (see lib/Analysis/LoopUnrollAnalyzer.cpp). Have you considered adding this functionality there (and what's missing there to catch this)?

Michael

[evstupac]

Hi Michael,

The new function is more about a decision if we need unroll or not. lib/Transforms/Scalar/LoopUnrollPass.cpp looks like a good place for it.

Thanks,
Evgeny

[evstupac]

PING.

[mzolotukhin]

Hi Evgeny,

Why functionality from lib/Analysis/LoopUnrollAnalyzer.cpp isn't enough for your case? It should be able to predict that phis would be removed after unrolling (possibly along with other instructions).

Michael

[evstupac]

Add recurrent xor case.
Move calculations to cost estimation function.

[evstupac]

1. Set "disable unroll" for loops unrolled by "force unroll". The reason is that on LTO unroll pass could be called multiple times and we will try to unroll unless exceed threshold. This is not critical, but makes unroll behavior dependent on LTO.
2. Add missed unit tests.
3. Fix calculations for phis used outside of the loop only.

[evstupac]

PING.

[evstupac]

PING 2.

[evstupac]

PING 3.

[mzolotukhin]

lib/Transforms/Scalar/LoopUnrollPass.cpp

505–509

This look very hacky. What is the motivation of this change? How often do we see such loops?

If we do want to handle such cases (which I'm not convinced now), then we should do it in a general way. That is, the logic should be in instruction visitors, and we should automagically deduce that these instructions are free. There are more cases than just xor - we can multiply by -1, and we should be able to handle in a similar way. In the current form the code is not easily extensible to handle new cases.

[evstupac]

lib/Transforms/Scalar/LoopUnrollPass.cpp

505–509

What is the motivation of this change? How often do we see such loops?

Not each test looks like this, but there are couple where we switch states:

state = st[s^=1];

which becomes invariant with some other calculations after unroll.

If we do want to handle such cases (which I'm not convinced now), then we should do it in a general way. That is, the logic should be in instruction visitors, and we should automagically deduce that these instructions are free.

The same is valid for the code above where complete unroll simplify phi. Why phi is simplified here?
I just did the same for XOR.
And yes we can multiply by -1, do i&1, i/2,... but we need a start point which depends on unroll factor and iteration.
The other solution is to pass, unroll factor in addition to iteration number and move simplification there.
Do you think this is better?

[mzolotukhin]

What is the motivation of this change? How often do we see such loops?

Not each test looks like this, but there are couple where we switch states:

state = st[s^=1];
which becomes invariant with some other calculations after unroll.

I realize that it's possible to write such a test. My question was how frequent are such cases in real programs. Can you provide statistics from SPEC/llvm-testsuite that would justify the change?

If we do want to handle such cases (which I'm not convinced now), then we should do it in a general way. That is, the logic should be in instruction visitors, and we should automagically deduce that these instructions are free.

The same is valid for the code above where complete unroll simplify phi. Why phi is simplified here?

PHIs are present in every loop, in contrast to XORs.

The other solution is to pass, unroll factor in addition to iteration number and move simplification there.
Do you think this is better?

I haven't put much thought to this yet, because I'm still not convinced in the need of this change. It's possible though that if we decide to go this way, we'll need to do some prep-work first to make the code look nice. The main part of simplification code is already there (with PHI-nodes getting an extra attention due to their special and very important nature).

[evstupac]

(was unable to answer in phabricator previously, so just put this from mail thread here)

I realize that it's possible to write such a test. My question was how frequent are such cases in real programs. Can you provide statistics from SPEC/llvm-testsuite that would justify the change?

I'll get the results for llvm test-suite. There is not much difference
on spec2000. There are gains on eembc benhmarks.

PHIs are present in every loop, in contrast to XORs.

The patch touches only recursive XORs which also go through PHI.
PHIs for optimization after complete unroll are in countable loops. My
patch deals with uncountable (a list loop for example).

[evstupac]

I'm not sure I can share all the cases, but I've found good public example of "xor" in the loop from jpeg (h2v1_downsample and h2v2_downsample):
https://github.com/cloudflare/jpegtran/blob/master/jcsample.c

For the case above runtime unroll should work fine, but there are cases where loop upper bound is structure field load. If so loop becomes uncountable and we are unable to apply runtime unroll.
Generally there multiple cases when runtime unroll can fail, so why not to force unroll if we can prove profitability?

[evstupac]

Add phi cycle function to explain phi simplification.

[evstupac]

Regarding performance data.
Initially I want unroll only loops where previous iteration value is reused. Just with this change I get the following.
197.parser up to 5% depending on CPU and 10% to 30% on set of internal benchmarks.
With xor and sub enabled most of benchmarks the results got buildsame or no performance change. However I still believe this fits to current functionality.

One thing I noticed, adding cost to regular unroll (not complete). All loops which counts something got unrolled:

while(smth)

n++;

Use(n);

And unroll safes at least 1 instruction for such loops.

[evstupac]

PING.
Let's come to a conclusion if this is acceptable or not.
The cases that covered here:

1. Uncountable loops where previous value is reused (save one+ instruction for each value)
2. uncountable loops, that counts smth:

while(smth) {
  body();
  n++;
}

to:

while(smth) {
  body();
  if (!smth) goto exit2;
  body();
  n+=2;
}
exit2:
n++;
loop exit:

Saves one+ "add" in the loop.

3. Uncountable loops switching states (not that frequent):

while(smth) {
  body();
  s ^= n;
}
while(smth) {
  body();
  s = -s;
}

Potentially saves a lot (as constant values could simplify several instructions).

[mzolotukhin]

Hi,

I'm still not convinced that we need this functionality. The loops that you mentioned can easily be handled by using unroll pragma if users really care about their performance. The patch might introduce no performance/compile-time regressions, but we shouldn't forget about the increased source code complexity.

Michael

[evstupac]

Hi Michael,

but we shouldn't forget about the increased source code complexity.

The patch mostly reuse existing code for full unroll.

Regarding PHI Cycle it even improve current heuristic, as full unroll consider
"s^=1;" is constant "1" at all iterations >0. However it is 0 on even and 1 on odd.
And I'm ok to drop this part to make code shorter.

Maybe we can add uncountable loops unroll under -O3?

Thanks,
Evgeny

[hfinkel]

In D21720#691221, @evstupac wrote:

Hi Michael,

but we shouldn't forget about the increased source code complexity.

The patch mostly reuse existing code for full unroll.

Regarding PHI Cycle it even improve current heuristic, as full unroll consider
"s^=1;" is constant "1" at all iterations >0. However it is 0 on even and 1 on odd.
And I'm ok to drop this part to make code shorter.

Maybe we can add uncountable loops unroll under -O3?

I agree. -O3 would be the right place for this.

Thanks,
Evgeny

In D21720#691193, @mzolotukhin wrote:

Hi,

I'm still not convinced that we need this functionality. The loops that you mentioned can easily be handled by using unroll pragma if users really care about their performance. The patch might introduce no performance/compile-time regressions, but we shouldn't forget about the increased source code complexity.

Michael

I don't think that "easily be handled by using unroll pragma" is the right standard here. The same argument can be made for anything. We should only really require a pragma when we cannot design a reasonable heuristic (at this stage in the pipeline). This patch does not seem that large. I can review next week.

[mzolotukhin]

full unroll consider "s^=1;" is constant "1" at all iterations >0. However it is 0 on even and 1 on odd.

If that's the case, I'd be happy to review a separate patch that just fixes that.

I don't think that "easily be handled by using unroll pragma" is the right standard here. The same argument can be made for anything. We should only really require a pragma when we cannot design a reasonable heuristic (at this stage in the pipeline). This patch does not seem that large. I can review next week.

My biggest concern about this patch is that it doesn't solve the problem in a general way, but instead only catches two cases: s ^= 1 and s = -s. While it looks kind of generic, I can see no way how it can be extended in future. E.g. it doesn't seem to be able to support another way to write the same idiom: s = i%2 or s = i&1, and if we'd like to support it, we'll need yet another approach.

Your point about using pragma is generally valid, but here is my view on it. While in general we do need to try to cover as many cases without pragmas as possible, we need to provide a general implementation that potentially can help other cases as well. Patching the compiler here and there just to cover cases one by one would lead to unmaintainable and inefficient code in the end. Also, unlike many other loops in a loop with s ^=1 the user would more likely correctly guess a favorable unroll factor (2), just by the nature of the computation. In more complicated loops a help from the compiler is needed much more, because it might be hard to guess the best unroll factor.

IMHO, a general approach here would be to teach unroller (or maybe SCEV) to analyze N sequential iterations. E.g. for the xor case and for 2 sequential iterations starting at the i-th iteration, we'd get something like S_i_plus_2 = S_i_plus_1 ^ 1 = (S_i ^ 1) ^ 1 = S_i. Honestly, I don't know how hard it would be to implement this, and maybe the efforts and complexity won't be worth the gain, but that's what I'd call a general approach, rather than targeting a single case.

Michael

[evstupac]

My biggest concern about this patch is that it doesn't solve the problem in a general way, but instead only catches two cases: s ^= 1 and s = -s.

Why PhiCycle is not general? That could be InstCycle instead (for i&3, i&7, i%N,....).
"i%2", "i&1" are convertible to "s^=1".

IMHO, a general approach here would be to teach unroller (or maybe SCEV) to analyze N sequential iterations. E.g. for the xor case and for 2 sequential iterations starting at the i-th iteration, we'd get something like S_i_plus_2 = S_i_plus_1 ^ 1 = (S_i ^ 1) ^ 1 = S_i.

Current approach do this, but starting from 0 iteration (taking in account that some instruction are cycle).

Suppose we skip this, what about:

1. Uncountable loops where previous value is reused (save one+ instruction for each value)?
2. Uncountable loops, that counts smth?

[hfinkel]

In D21720#696992, @evstupac wrote:

My biggest concern about this patch is that it doesn't solve the problem in a general way, but instead only catches two cases: s ^= 1 and s = -s.

Why PhiCycle is not general? That could be InstCycle instead (for i&3, i&7, i%N,....).
"i%2", "i&1" are convertible to "s^=1".

Sure, but "are convertible to" is not helpful if they're not. Whether that's a useful canonical form might be a separate discussion (maybe it is if that's the only use of the induction variable). In any case, I agree that we should have something that is insensitive to how this is written.

However, we still need to make sure we're doing this in a way that is profitable. The point of making sure to unroll by an even factor for these cycle-2 recurrences is that it allows us to completely remove the PHI. It is not clear to me that the more-general case shares this property without additional work (by which I mean canonicalization work).

IMHO, a general approach here would be to teach unroller (or maybe SCEV) to analyze N sequential iterations. E.g. for the xor case and for 2 sequential iterations starting at the i-th iteration, we'd get something like S_i_plus_2 = S_i_plus_1 ^ 1 = (S_i ^ 1) ^ 1 = S_i.

But you need more than this. You need to know how the intermediate iterations are related so that you can eliminate the PHI. This is especially true for the remainder case (because remainders are expensive compared to &, for example).

I suspect that SCEV would have a hard time doing this because the relations are not algebraic (although it might be able to do something interesting with the remainder case).

In any case, there is definitely a more-general case we can handle here, but it is the following: there needs to either be a cycle PHI, or a use of an otherwise-unused instruction variable that repeats in a fixed pattern, such that unrolling the loop by the length of that pattern allows us to eliminate the PHI. i&C and i%C have that property. Is there anything else?

Current approach do this, but starting from 0 iteration (taking in account that some instruction are cycle).

Suppose we skip this, what about:

1. Uncountable loops where previous value is reused (save one+ instruction for each value)?
2. Uncountable loops, that counts smth?

[evstupac]

In any case, there is definitely a more-general case we can handle here, but it is the following: there needs to either be a cycle PHI, or a use of an otherwise-unused instruction variable that repeats in a fixed pattern, such that unrolling the loop by the length of that pattern allows us to eliminate the PHI. i&C and i%C have that property. Is there anything else?

"i/C" could also be optimized when "i" starts from "k*C". However we'll need to lower it (replace with new IV) in later optimizations (for example LSR).

What about cases 1. and 2. ?

[hfinkel]

In D21720#697188, @evstupac wrote:

In any case, there is definitely a more-general case we can handle here, but it is the following: there needs to either be a cycle PHI, or a use of an otherwise-unused instruction variable that repeats in a fixed pattern, such that unrolling the loop by the length of that pattern allows us to eliminate the PHI. i&C and i%C have that property. Is there anything else?

"i/C" could also be optimized when "i" starts from "k*C". However we'll need to lower it (replace with new IV) in later optimizations (for example LSR).

What about cases 1. and 2. ?

Case 1 sounds good, but maybe our existing heuristic should get it? I don't understand what you mean in case 2.

[evstupac]

Case 1 sounds good, but maybe our existing heuristic should get it? I don't understand what you mean in case 2.

Actually case 1 is the most simple and profitable. Maybe we shall start with it only?
Regarding case 2:

It covers loops which counts how long some condition is true. For example (how many elements in the list):
while (curr) {

len++;
curr = curr->next;

}

After unroll by 2:

while(curr) {

curr1 = curr->next;
if (!curr1) goto exit1;
curr = curr1->next;
len+=2;

}
exit1:
len++;
loop exit:

[evstupac]

but maybe our existing heuristic should get it?

No. We don't do unroll of uncountable loops at all.

[zzheng]

lib/Transforms/Scalar/LoopUnrollPass.cpp

94

Is this description copied from above flag?

allow-force sounds like forced unrolling... how about name it UnrollAllowUncountable (unroll-allow-uncountable)?

test/Transforms/LoopUnroll/unroll-force.ll

1 ↗

(On Diff #88080)

Maybe name this file 'unroll-uncountable.ll'?

[mzolotukhin]

What is the case 1?

For loops with unknown trip count (that's the case 2, right?) I don't see why we need to add any additional analyzers (analyzeLoopUnrollCost). Removing one instruction by unrolling by 2 is not better than the worst case for partial unroll of a loop with a known trip count:

for (i = 0; i < N; i++) {
  ...
}

But since we sometimes partially unroll such loops, I don't currently see a reason why we can't do this for loops with unknown trip-count as well. However, we don't need additional thresholds/knobs for it, thresholds for partial unroll should do it already.

[evstupac]

What is the case 1?

Uncountable loops where previous value is reused (save one+ instruction for each value)
The example is in lit test (motivation comes from list loops).

For loops with unknown trip count (that's the case 2, right?)

No. All cases are about uncountable loops.

I don't see why we need to add any additional analyzers (analyzeLoopUnrollCost). Removing one instruction by unrolling by 2 is not better than the worst case for partial unroll of a loop with a known trip count:

We can skip this and do like in my initial patch (just run a function that return unroll count if it is profitable to unroll). That way cases 2 and 3 are skipped. I change the patch to call `analyzeLoopUnrollCost' to make more general.

But since we sometimes partially unroll such loops, I don't currently see a reason why we can't do this for loops with unknown trip-count as well. However, we don't need additional thresholds/knobs for it, thresholds for partial unroll should do it already.

The patch do not introduce a new threshold. If we enable unroll for all uncountable loops it will result in dramatic code size increase (for most cases without performance value).

[evstupac]

lib/Transforms/Scalar/LoopUnrollPass.cpp

94

Right. Thanks for catching this. Will fix. "unroll-allow-uncountable" looks good.

test/Transforms/LoopUnroll/unroll-force.ll

1 ↗

(On Diff #88080)

Yes. Make sense.

[mzolotukhin]

Uncountable loops where previous value is reused (save one+ instruction for each value)

What do you mean by uncountable? How is it different from a loop with unknown trip count?
What is a previous value? Is it a value from a previous iteration? If that's the case, any induction variable satisfies that condition.

No. All cases are about uncountable loops.

What's the difference between these two cases? I've gotten totally confused now.

We can skip this and do like in my initial patch (just run a function that return unroll count if it is profitable to unroll). That way cases 2 and 3 are skipped. I change the patch to call `analyzeLoopUnrollCost' to make more general.

What is the case 3?

The patch do not introduce a new threshold. If we enable unroll for all uncountable loops it will result in dramatic code size increase (for most cases without performance value).

Exactly. And unroll of any loop that has at least one IV will save at least one instruction, because we'll have a phi for the IV and we can fold this phi in between unrolled iterations. So, my point was that "saving one instruction" should not be enough to unroll a loop.

[evstupac]

What do you mean by uncountable? How is it different from a loop with unknown trip count?

L is uncountble <=> isa<SCEVCouldNotCompute>(SE->getBackedgeTakenCount(L)) is true.
For unknown trip isa<SCEVCouldNotCompute>(SE->getBackedgeTakenCount(L)) is false.

What's the difference between these two cases? I've gotten totally confused now.
What is the case 3?

The cases that covered here:

1. Uncountable loops where previous value is reused (save one+ instruction for each value)
2. uncountable loops, that counts smth:

while(smth) {
  body();
  n++;
}

to:

while(smth) {
  body();
  if (!smth) goto exit2;
  body();
  n+=2;
}
exit2:
n++;
loop exit:

Saves one+ "add" in the loop.

3. Uncountable loops switching states (not that frequent):

while(smth) {
  body();
  s ^= n;
}
while(smth) {
  body();
  s = -s;
}

Potentially saves a lot (as constant values could simplify several instructions).

[efriedma]

Currently, there's a profitability check hidden in llvm::UnrollLoop: if the "Force" parameter is false, it will refuse to unroll any loop unless it can make the loop latch in the unrolled iterations an unconditional branch. (See the debug message "Wont unroll; remainder loop could not be generated" in lib/Transforms/Utils/LoopUnroll.cpp.) This prevents the unroll pass from going crazy unrolling every loop in the program, but it's also not a very good way to measure profitability.

Ideally, we want to tie unrolling to a dynamic cost discount. If the unrolled loop is substantially cheaper than the original, we should unroll, whether or not the savings come from eliminating the conditional branch in the latch. Conversely, unrolling a large loop just to eliminate one branch is probably a bad idea.

This patch is trying to solve the problem for certain cases... but the approach is kind of awkward: rather than actually moving the profitability check, it adds a special case to set the "Force" bit for certain loops.

[evstupac]

Conversely, unrolling a large loop just to eliminate one branch is probably a bad idea.

That is bounded by thresholds.

This patch is trying to solve the problem for certain cases... but the approach is kind of awkward: rather than actually moving the profitability check,
it adds a special case to set the "Force" bit for certain loops.

Isn't it the same for Runtime, Partial? Setting count means almost the same. Count of 1 or 0 means we'll act in the same way if Runtime is false. I can rewrite this and set Force to true by default (but set count to 0 or 1 for uncountable loops, that do not meet some conditions).
And yes you are right we need to move all profitability checks from Utils/LoopUnrollRuntime.cpp to Scalar/LoopUnrollPass.cpp (I plan to do this in another patch).,

[efriedma]

Conversely, unrolling a large loop just to eliminate one branch is probably a bad idea.

That is bounded by thresholds.

Not really... I mean, we have PartialThreshold as a maximum size for the whole unrolled loop, but that isn't sensitive to the dynamic cost savings. (Compare to getFullUnrollBoostingFactor for complete unrolling.)

[evstupac]

Not really... I mean, we have PartialThreshold as a maximum size for the whole unrolled loop, but that isn't sensitive to the dynamic cost savings. (Compare to getFullUnrollBoostingFactor for complete unrolling.)

Right. But full unroll is the only exception.
And yes this threshold is not that accurate, but it bounds loop unrolling. And it will bound uncountable loops unroll as well.

[hfinkel]

In D21720#698373, @evstupac wrote:

Not really... I mean, we have PartialThreshold as a maximum size for the whole unrolled loop, but that isn't sensitive to the dynamic cost savings. (Compare to getFullUnrollBoostingFactor for complete unrolling.)

Right. But full unroll is the only exception.
And yes this threshold is not that accurate, but it bounds loop unrolling. And it will bound uncountable loops unroll as well.

Okay, so is the general plan here to?

1. Generally allow unrolling of uncountable loops
2. Adjust the profitability heuristic to account for the cost of the branches for the uncountable case
3. Include, as necessary, in our general profitability heuristic (if we don't get this naturally), savings from phi-derived values which are cyclic with a small cycle length (like the s = -s case).

[mzolotukhin]

Okay, so is the general plan here to?

1. Generally allow unrolling of uncountable loops
2. Adjust the profitability heuristic to account for the cost of the branches for the uncountable case
3. Include, as necessary, in our general profitability heuristic (if we don't get this naturally), savings from phi-derived values which are cyclic with a small cycle length (like the s = -s case).

I agree that we should start from (1). But I don't understand, why (2) (and (3) ) is considered special for the uncountable case. AFAIU, the current heuristic aims at simulating these savings (removed branches) already - even though we do not try to accurately predict cost of rolled and unrolled version of the loop. Am I missing something?

Michael

[evstupac]

Okay, so is the general plan here to?

1. Generally allow unrolling of uncountable loops

That seems reasonable. I'll enable this at O3.

2. Adjust the profitability heuristic to account for the cost of the branches for the uncountable case

In most cases we can not remove branches in uncountable (isa<SCEVCouldNotCompute>(SE->getBackedgeTakenCount(L)) is true) loops. And it is hard to detect cases when we can.
To be able to reduce IVs (if there are such) we'll need to count them and do some basic analysis. A lot of uncountable loops don't have IVs at all.
So the heuristic we have for Runtime unrolling (with unknown trip count) is not ok here.

3. Include, as necessary, in our general profitability heuristic (if we don't get this naturally), savings from phi-derived values which are cyclic with a small cycle length (like the s = -s case).

Right now this is the only way to see if unroll for uncountable loop is profitable or not.

I agree that we should start from (1). But I don't understand, why (2) (and (3) ) is considered special for the uncountable case. AFAIU, the current heuristic aims at simulating these savings (removed branches) already - even though we do not try to accurately predict cost of rolled and unrolled version of the loop. Am I missing something?

Current heuristic works for loops with unknown trip count (when we are able to calculate it, but runtime only). It is not applicable for uncountable loops.

[evstupac]

Enabled uncountable (unable to calculate trip count) unroll by default at O3 and higher.
Renamed test (according to inline comments).

[evstupac]

PING

[efriedma]

The use of the "Force" bit here is still really confusing... we need a better way of expressing the profitability of unrolling.

include/llvm/CodeGen/BasicTTIImpl.h

332

What effect does this have on performance?

[evstupac]

In D21720#766168, @efriedma wrote:

The use of the "Force" bit here is still really confusing... we need a better way of expressing the profitability of unrolling.

The naming was introduced earlier. Changing it in this patch would be considered as "unrelated change".
"Force" mean we'll unroll loop anyway. For example if user set "pragma unroll(2)" we need to force unrolling (no matter: countable loop or not, it is expensive to calculate tripcount or not...).

include/llvm/CodeGen/BasicTTIImpl.h

332

spec performance is unchanged.
However I see performance gain on small loops working with list structures (that is the main reason I'm trying to apply this).
Generally I don't see why we are missing unroll of uncountable loops now. When we are able to prove profitability we should do this.

[evstupac]

PING.

[efriedma]

Changing it in this patch would be considered as "unrelated change".

It's still messy, and still needs to change before this lands. Changing it in a separate patch is fine, if that makes sense.

---

This change needs testcases, including a couple examples of loops where the unrolling is profitable.

lib/Transforms/Scalar/LoopUnrollPass.cpp

413

Indentation. And I'm not sure why you're adding an instruction to the worklist that might not even be in the loop.

466

Maybe this would be more clear if it returned a boolean? I'm not sure the rest of this code makes sense if PhiCycle isn't either 0 or 2.

608

What is this change doing? (Please put comments in the code.)

987

"Cost->UnrolledCost >= Cost->RolledDynamicCost" is the profitability check? Needs a comment to explain what that means.

Do we care how large the improvement is vs. the size of the loop?

990

Maybe rearrange this? UP.Force is only true on one codepath out of the previous if statement.

include/llvm/Analysis/LoopUnrollAnalyzer.h

class UnrolledInstAnalyzer : private InstVisitor<UnrolledInstAnalyzer, bool> {
   struct SimplifiedAddress {
     Value *Base = nullptr;
     ConstantInt *Offset = nullptr;
   };
 
 public:
   UnrolledInstAnalyzer(unsigned Iteration,
                        DenseMap<Value *, Constant *> &SimplifiedValues,
-                       ScalarEvolution &SE, const Loop *L)
-      : SimplifiedValues(SimplifiedValues), SE(SE), L(L) {
+                       ScalarEvolution &SE, const Loop *L, bool CompleteUnroll)
+      : SimplifiedValues(SimplifiedValues), SE(SE),
+        L(L), CompleteUnroll(CompleteUnroll) {
       IterationNumber = SE.getConstant(APInt(64, Iteration));
   }
 
   // Allow access to the initial visit method.
   using Base::visit;
 
 private:
   /// \brief A cache of pointer bases and constant-folded offsets corresponding
private:
   /// of simplified values specific to this iteration.  The idea is to propagate
   /// any special information we have about loads that can be replaced with
   /// constants after complete unrolling, and account for likely simplifications
   /// post-unrolling.
   DenseMap<Value *, Constant *> &SimplifiedValues;
 
   ScalarEvolution &SE;
   const Loop *L;
+  bool CompleteUnroll;
 
   bool simplifyInstWithSCEV(Instruction *I);
 
   bool visitInstruction(Instruction &I) { return simplifyInstWithSCEV(&I); }
   bool visitBinaryOperator(BinaryOperator &I);
   bool visitLoad(LoadInst &I);
   bool visitCastInst(CastInst &I);
   bool visitCmpInst(CmpInst &I);
   bool visitPHINode(PHINode &PN);
 };
 }
 #endif

include/llvm/CodeGen/BasicTTIImpl.h

for (Loop::block_iterator I = L->block_begin(), E = L->block_end(); I != E;
         }
     }
 
     // Enable runtime and partial unrolling up to the specified size.
     // Enable using trip count upper bound to unroll loops.
     UP.Partial = UP.Runtime = UP.UpperBound = true;
     UP.PartialThreshold = MaxOps;
 
+    const TargetMachine &TM = getTLI()->getTargetMachine();
+    // Unroll uncountable inner loops at O3 and higher.
+    if (L->getSubLoops().size() == 0 && TM.getOptLevel() > CodeGenOpt::Default)
+      UP.Force = true;
+
     // Avoid unrolling when optimizing for size.
     UP.OptSizeThreshold = 0;
     UP.PartialOptSizeThreshold = 0;
 
     // Set number of instructions optimized when "back edge"
     // becomes "fall through" to default value of 2.
     UP.BEInsns = 2;
   }

lib/Analysis/LoopUnrollAnalyzer.cpp

bool UnrolledInstAnalyzer::simplifyInstWithSCEV(Instruction *I) {
   if (!SE.isSCEVable(I->getType()))
     return false;
 
   const SCEV *S = SE.getSCEV(I);
   if (auto *SC = dyn_cast<SCEVConstant>(S)) {
     SimplifiedValues[I] = SC->getValue();
     return true;
   }
+  if (!CompleteUnroll)
+    return false;
 
   auto *AR = dyn_cast<SCEVAddRecExpr>(S);
   if (!AR || AR->getLoop() != L)
     return false;
 
   const SCEV *ValueAtIteration = AR->evaluateAtIteration(IterationNumber, SE);
   // Check if the AddRec expression becomes a constant.
   if (auto *SC = dyn_cast<SCEVConstant>(ValueAtIteration)) {

 }
 
 bool UnrolledInstAnalyzer::visitPHINode(PHINode &PN) {
   // Run base visitor first. This way we can gather some useful for later
   // analysis information.
   if (Base::visitPHINode(PN))
     return true;
 
+  // Consider PHI is foldable only after complete unroll.
+  if (!CompleteUnroll)
+    return false;
+
   // The loop induction PHI nodes are definitionally free.
   return PN.getParent() == L->getHeader();
 }

lib/Transforms/Scalar/LoopUnrollPass.cpp

UnrollAllowPartial("unroll-allow-partial", cl::Hidden,
                        cl::desc("Allows loops to be partially unrolled until "
                                 "-unroll-threshold loop size is reached."));
 
 static cl::opt<bool> UnrollAllowRemainder(
     "unroll-allow-remainder", cl::Hidden,
     cl::desc("Allow generation of a loop remainder (extra iterations) "
              "when unrolling a loop."));
 
+static cl::opt<bool> UnrollUncountable(
+    "unroll-uncountable", cl::Hidden, cl::init(false),
+    cl::desc("Allow unroll of uncountable (trip count is not countable) "
+             "loops when profitable."));
+
 static cl::opt<bool>
     UnrollRuntime("unroll-runtime", cl::ZeroOrMore, cl::Hidden,
                   cl::desc("Unroll loops with run-time trip counts"));
 
 static cl::opt<unsigned> UnrollMaxUpperBound(
     "unroll-max-upperbound", cl::init(8), cl::Hidden,
     cl::desc(
         "The max of trip count upper bound that is considered in unrolling"));
static TargetTransformInfo::UnrollingPreferences gatherUnrollingPreferences(
   if (UnrollAllowPartial.getNumOccurrences() > 0)
     UP.Partial = UnrollAllowPartial;
   if (UnrollAllowRemainder.getNumOccurrences() > 0)
     UP.AllowRemainder = UnrollAllowRemainder;
   if (UnrollRuntime.getNumOccurrences() > 0)
     UP.Runtime = UnrollRuntime;
   if (UnrollMaxUpperBound == 0)
     UP.UpperBound = false;
+  if (UnrollUncountable.getNumOccurrences() > 0)
+    UP.Force = true;
   if (UnrollAllowPeeling.getNumOccurrences() > 0)
     UP.AllowPeeling = UnrollAllowPeeling;
 
   // Apply user values provided by argument
   if (UserThreshold.hasValue()) {
     UP.Threshold = *UserThreshold;
     UP.PartialThreshold = *UserThreshold;
   }
struct UnrolledInstStateKeyInfo {
   }
   static inline bool isEqual(const UnrolledInstState &LHS,
                              const UnrolledInstState &RHS) {
     return PairInfo::isEqual({LHS.I, LHS.Iteration}, {RHS.I, RHS.Iteration});
   }
 };
 }
 
+/// Some instructions get the same value during loop execution.
+/// The function find a cycle length for such.
+static unsigned getPhiCycleLength (PHINode *PHI, const Loop *L) {
+  Value *V = PHI->getIncomingValueForBlock(L->getLoopLatch());
+  Instruction *I = dyn_cast<Instruction>(V);
+  if (!I)
+    return 0;
+  switch (I->getOpcode()) {
+  case Instruction::Sub:
+    if (ConstantInt *CS = dyn_cast<ConstantInt>(I->getOperand(0)))
+      if (CS->isZero() && I->getOperand(1) == PHI)
+        return 2;
+  case Instruction::Xor:
+    if (I->getOperand(0) == PHI && L->isLoopInvariant(I->getOperand(1)))
+      return 2;
+  default:
+    return 0;
+  }
+  return 0;
+}
+
 namespace {
 struct EstimatedUnrollCost {
   /// \brief The estimated cost after unrolling.
   unsigned UnrolledCost;
 
   /// \brief The estimated dynamic cost of executing the instructions in the
   /// rolled form.
   unsigned RolledDynamicCost;

 /// dynamic cost we mean that we won't count costs of blocks that are known not
 /// to be executed (i.e. if we have a branch in the loop and we know that at the
 /// given iteration its condition would be resolved to true, we won't add up the
 /// cost of the 'false'-block).
 /// \returns Optional value, holding the RolledDynamicCost and UnrolledCost. If
 /// the analysis failed (no benefits expected from the unrolling, or the loop is
 /// too big to analyze), the returned value is None.
 static Optional<EstimatedUnrollCost>
-analyzeLoopUnrollCost(const Loop *L, unsigned TripCount, DominatorTree &DT,
+analyzeLoopUnrollCost(const Loop *L, unsigned UnrollCount, DominatorTree &DT,
                       ScalarEvolution &SE, const TargetTransformInfo &TTI,
-                      unsigned MaxUnrolledLoopSize) {
+                      unsigned MaxUnrolledLoopSize, bool CompleteUnroll = true) {
   // We want to be able to scale offsets by the trip count and add more offsets
   // to them without checking for overflows, and we already don't want to
   // analyze *massive* trip counts, so we force the max to be reasonably small.
   assert(UnrollMaxIterationsCountToAnalyze < (INT_MAX / 2) &&
          "The unroll iterations max is too large!");
 
   // Only analyze inner loops. We can't properly estimate cost of nested loops
   // and we won't visit inner loops again anyway.
   if (!L->empty())
     return None;
 
-  // Don't simulate loops with a big or unknown tripcount
-  if (!UnrollMaxIterationsCountToAnalyze || !TripCount ||
-      TripCount > UnrollMaxIterationsCountToAnalyze)
+  // Don't simulate loops with a big or unknown UnrollCount
+  if (!UnrollMaxIterationsCountToAnalyze || !UnrollCount ||
+      UnrollCount > UnrollMaxIterationsCountToAnalyze)
     return None;
 
   SmallSetVector<BasicBlock *, 16> BBWorklist;
   SmallSetVector<std::pair<BasicBlock *, BasicBlock *>, 4> ExitWorklist;
   DenseMap<Value *, Constant *> SimplifiedValues;
   SmallVector<std::pair<Value *, Constant *>, 4> SimplifiedInputValues;
 
   // The estimated cost of the unrolled form of the loop. We try to estimate
for (;; --Iteration) {
           continue;
 
         // Mark that we are counting the cost of this instruction now.
         Cost.IsCounted = true;
 
         // If this is a PHI node in the loop header, just add it to the PHI set.
         if (auto *PhiI = dyn_cast<PHINode>(I))
           if (PhiI->getParent() == L->getHeader()) {
+            if (CompleteUnroll)
               assert(Cost.IsFree && "Loop PHIs shouldn't be evaluated as they "
-                                  "inherently simplify during unrolling.");
+                                    "inherently simplify during complete "
+                                    "unrolling.");
             if (Iteration == 0)
               continue;
 
             // Push the incoming value from the backedge into the PHI used list
             // if it is an in-loop instruction. We'll use this to populate the
             // cost worklist for the next iteration (as we count backwards).
             if (auto *OpI = dyn_cast<Instruction>(
                     PhiI->getIncomingValueForBlock(L->getLoopLatch())))
for (;; --Iteration) {
         // recursively. If we reach a loop PHI node, simply add it to the set
         // to be considered on the next iteration (backwards!).
         for (Value *Op : I->operands()) {
           // Check whether this operand is free due to being a constant or
           // outside the loop.
           auto *OpI = dyn_cast<Instruction>(Op);
           if (!OpI || !L->contains(OpI))
             continue;
-
+          // For regular unroll instruction that depends on PHI also costs.
+          if (!CompleteUnroll && isa<PHINode>(OpI)) {
+            auto *OpIP = dyn_cast<PHINode>(OpI);
+            if(OpIP->getParent() == L->getHeader())
+              if (auto *OpIPI =
+                      dyn_cast<Instruction>(
+                          OpIP->getIncomingValueForBlock(L->getLoopLatch())))
+            CostWorklist.push_back(OpIPI);
+          }
           // Otherwise accumulate its cost.
           CostWorklist.push_back(OpI);
         }
       } while (!CostWorklist.empty());
 
       if (PHIUsedList.empty())
         // We've exhausted the search.
         break;
assert(L->isLCSSAForm(DT) &&
          "Must have loops in LCSSA form to track live-out values.");
 
   DEBUG(dbgs() << "Starting LoopUnroll profitability analysis...\n");
 
   // Simulate execution of each iteration of the loop counting instructions,
   // which would be simplified.
   // Since the same load will take different values on different iterations,
   // we literally have to go through all loop's iterations.
-  for (unsigned Iteration = 0; Iteration < TripCount; ++Iteration) {
+  for (unsigned Iteration = 0; Iteration < UnrollCount; ++Iteration) {
     DEBUG(dbgs() << " Analyzing iteration " << Iteration << "\n");
 
     // Prepare for the iteration by collecting any simplified entry or backedge
     // inputs.
     for (Instruction &I : *L->getHeader()) {
       auto *PHI = dyn_cast<PHINode>(&I);
       if (!PHI)
         break;
 
       // The loop header PHI nodes must have exactly two input: one from the
       // loop preheader and one from the loop latch.
       assert(
           PHI->getNumIncomingValues() == 2 &&
           "Must have an incoming value only for the preheader and the latch.");
 
+      // If incoming value for PHI is another loop PHI, that means we need to
+      // store previous value (without unroll).
+      if (PHINode *PHIL = dyn_cast<PHINode>(
+          PHI->getIncomingValueForBlock(L->getLoopLatch())))
+        if (Iteration && L->contains(PHIL))
+          RolledDynamicCost++;
+
+      unsigned PhiCycle = getPhiCycleLength (PHI, L);
+      // For complete unroll if we were unable to get PHI cycle length
+      // consider cycle length as full unroll count (TripCount).
+      if (CompleteUnroll && PhiCycle == 0)
+        PhiCycle = UnrollCount;
+      // If PHI has no cycle we are unable to simplify it.
+      if (PhiCycle == 0)
+        continue;
+
       Value *V = PHI->getIncomingValueForBlock(
-          Iteration == 0 ? L->getLoopPreheader() : L->getLoopLatch());
+          Iteration % PhiCycle ? L->getLoopLatch() : L->getLoopPreheader());
       Constant *C = dyn_cast<Constant>(V);
-      if (Iteration != 0 && !C)
+      if ((Iteration % PhiCycle) && !C)
         C = SimplifiedValues.lookup(V);
       if (C)
         SimplifiedInputValues.push_back({PHI, C});
     }
 
     // Now clear and re-populate the map for the next iteration.
     SimplifiedValues.clear();
     while (!SimplifiedInputValues.empty())
       SimplifiedValues.insert(SimplifiedInputValues.pop_back_val());
+    UnrolledInstAnalyzer Analyzer(Iteration, SimplifiedValues,
+                                  SE, L, CompleteUnroll);
 
-    UnrolledInstAnalyzer Analyzer(Iteration, SimplifiedValues, SE, L);
 
     BBWorklist.clear();
     BBWorklist.insert(L->getHeader());
     // Note that we *must not* cache the size, this loop grows the worklist.
     for (unsigned Idx = 0; Idx != BBWorklist.size(); ++Idx) {
       BasicBlock *BB = BBWorklist[Idx];
 
       // Visit all instructions in the given basic block and try to simplify
       // it.  We don't change the actual IR, just count optimization
       // opportunities.
       for (Instruction &I : *BB) {
         if (isa<DbgInfoIntrinsic>(I))
           continue;
 
         // Track this instruction's expected baseline cost when executing the
         // rolled loop form.
         RolledDynamicCost += TTI.getUserCost(&I);
 
         // Visit the instruction to analyze its loop cost after unrolling,
         // and if the visitor returns true, mark the instruction as free after
         // unrolling and continue.
         bool IsFree = Analyzer.visit(I);
         bool Inserted = InstCostMap.insert({&I, (int)Iteration,
                                            (unsigned)IsFree,
                                            /*IsCounted*/ false}).second;
         (void)Inserted;
         assert(Inserted && "Cannot have a state for an unvisited instruction!");
for (unsigned Idx = 0; Idx != BBWorklist.size(); ++Idx) {
           BBWorklist.insert(Succ);
         else
           ExitWorklist.insert({BB, Succ});
       AddCostRecursively(*TI, Iteration);
     }
 
     // If we found no optimization opportunities on the first iteration, we
     // won't find them on later ones too.
-    if (UnrolledCost == RolledDynamicCost) {
+    if (CompleteUnroll && UnrolledCost == RolledDynamicCost) {
       DEBUG(dbgs() << "  No opportunities found.. exiting.\n"
                    << "  UnrolledCost: " << UnrolledCost << "\n");
       return None;
     }
   }
 
   while (!ExitWorklist.empty()) {
     BasicBlock *ExitingBB, *ExitBB;
     std::tie(ExitingBB, ExitBB) = ExitWorklist.pop_back_val();
 
     for (Instruction &I : *ExitBB) {
       auto *PN = dyn_cast<PHINode>(&I);
       if (!PN)
         break;
 
       Value *Op = PN->getIncomingValueForBlock(ExitingBB);
+      if (auto *OpPN = dyn_cast<PHINode>(Op))
+        if (OpPN->getParent() == L->getHeader())
+          Op = OpPN->getIncomingValueForBlock(L->getLoopLatch());
       if (auto *OpI = dyn_cast<Instruction>(Op))
         if (L->contains(OpI))
-          AddCostRecursively(*OpI, TripCount - 1);
+          AddCostRecursively(*OpI, UnrollCount - 1);
     }
   }
 
   DEBUG(dbgs() << "Analysis finished:\n"
                << "UnrolledCost: " << UnrolledCost << ", "
                << "RolledDynamicCost: " << RolledDynamicCost << "\n");
   return {{UnrolledCost, RolledDynamicCost}};
 }
if (PragmaCount > 0 && !UP.AllowRemainder)
              "unroll_count pragma because remainder loop is restricted "
              "(that could architecture specific or because the loop "
              "contains a convergent instruction) and so must have an unroll "
              "count that divides the loop trip multiple of "
           << NV("TripMultiple", TripMultiple) << ".  Unrolling instead "
           << NV("UnrollCount", UP.Count) << " time(s).");
   }
 
+  // Estimate if Force unroll could be profitable.
+  if (!isa<SCEVCouldNotCompute>(SE->getBackedgeTakenCount(L)) && UP.Count >= 2
+      && !ExplicitUnroll) {
+    UP.Force = false;
+  } else {
+    Optional<EstimatedUnrollCost> Cost =
+        analyzeLoopUnrollCost(L, 2, DT, *SE, TTI,
+                              UP.PartialThreshold + UP.BEInsns, false);
+    if (!Cost || Cost->UnrolledCost >= Cost->RolledDynamicCost ||
+        (unsigned)Cost->UnrolledCost > UP.PartialThreshold)
+      UP.Force = false;
+  }
+  if (UP.Force) {
+    // Currently force unroll only by 2.
+    UP.Count = 2;
+    DEBUG(dbgs() << "  unrolling with count: " << UP.Count << "\n");
+    // Unroll uncountable loops only once.
+    return true;
+  }
   if (UP.Count > UP.MaxCount)
     UP.Count = UP.MaxCount;
   DEBUG(dbgs() << "  partially unrolling with count: " << UP.Count << "\n");
   if (UP.Count < 2)
     UP.Count = 0;
   return ExplicitUnroll;
 }
 

unittests/Analysis/UnrollAnalyzer.cpp

bool runOnFunction(Function &F) override {
     BasicBlock *Header = &*FI++;
     Loop *L = LI->getLoopFor(Header);
     BasicBlock *Exiting = L->getExitingBlock();
 
     SimplifiedValuesVector.clear();
     TripCount = SE->getSmallConstantTripCount(L, Exiting);
     for (unsigned Iteration = 0; Iteration < TripCount; Iteration++) {
       DenseMap<Value *, Constant *> SimplifiedValues;
-      UnrolledInstAnalyzer Analyzer(Iteration, SimplifiedValues, *SE, L);
+      UnrolledInstAnalyzer Analyzer(Iteration, SimplifiedValues, *SE, L, true);
       for (auto *BB : L->getBlocks())
         for (Instruction &I : *BB)
           Analyzer.visit(I);
       SimplifiedValuesVector.push_back(SimplifiedValues);
     }
     return false;
   }
   void getAnalysisUsage(AnalysisUsage &AU) const override {