Page MenuHomePhabricator

[JumpThreading] Only write back branch-weight MDs for blocks that originally had PGO info
ClosedPublic

Authored by anemet on Aug 31 2016, 8:02 PM.

Details

Summary

Currently the pass updates branch weights in the IR if the function has
any PGO info (entry frequency is set). However we could still have
regions of the CFG that does not have branch weights collected (e.g. a
cold region). In this case we'd use static estimates. Since static
estimates for branches are determined independently, they are
inconsistent. Updating them can "randomly" inflate block frequencies.

I've run into this in a completely cold loop of h264ref from
SPEC. -Rpass-with-hotness showed the loop to be completely cold during
inlining (before JT) but completely hot during vectorization (after JT).

The new testcase demonstrate the problem. We check array elements
against 1, 2 and 3 in a loop. The check against 3 is the loop-exiting
check. The block names should be self-explanatory.

In this example, jump threading incorrectly updates the weight of the
loop-exiting branch to 0, drastically inflating the frequency of the
loop (in the range of billions).

There is no run-time profile info for edges inside the loop, so branch
probabilities are estimated. These are the resulting branch and block
frequencies for the loop body:

         check_1 (16)
     (8) /  |
     eq_1   | (8)
         \  |
         check_2 (16)
     (8) /  |
     eq_2   | (8)
         \  |
         check_3 (16)
     (1) /  |
(loop exit) | (15)
            |
       (back edge)

First we thread eq_1 -> check_2 to check_3. Frequencies are updated to
remove the frequency of eq_1 from check_2 and then from the false edge
leaving check_2. Changed frequencies are highlighted with * *:

         check_1 (16)
     (8) /  |
     eq_1   | (8)
    /       |
   /     check_2 (*8*)
  /  (8) /  |
  \  eq_2   | (*0*)
   \     \  |
    ` --- check_3 (16)
     (1) /  |
(loop exit) | (15)
            |
       (back edge)

Next we thread eq_1 -> check_3 and eq_2 -> check_3 to check_1 as new
back edges. Frequencies are updated to remove the frequency of eq_1 and
eq_3 from check_3 and then the false edge leaving check_3 (changed
frequencies are highlighted with * *):

                check_1 (16)
            (8) /  |
            eq_1   | (8)
           /       |
          /     check_2 (*8*)
         /  (8) /  |
        /-- eq_2   | (*0*)
(back edge)        |
                check_3 (*0*)
          (*0*) /  |
       (loop exit) | (*0*)
                   |
              (back edge)

As a result, the loop exit edge ends up with 0 frequency which in turn makes
the loop header to have maximum frequency.

There are a few potential problems here:

  1. The profile data seems odd. There is a single profile sample of the

loop being entered. On the other hand, there are no weights inside the
loop.

  1. Based on static estimation we shouldn't set edges to "extreme"

values, i.e. extremely likely or unlikely.

  1. We shouldn't create profile metadata that is calculated from static

estimation. I am not sure what policy is but it seems to make sense to
treat profile metadata as something that is known to originate from
profiling. Estimated probabilities should only be reflected in BPI/BFI.

Any one of these would probably fix the immediate problem. I went for 3
because I think it's a good policy to have and added a FIXME about 2.

Diff Detail

Repository
rL LLVM

Event Timeline

anemet updated this revision to Diff 69943.Aug 31 2016, 8:02 PM
anemet retitled this revision from to [JumpThreading] Only write back branch-weight MDs for blocks that originally had PGO info.
anemet updated this object.
anemet added a subscriber: llvm-commits.
davidxl edited edge metadata.Sep 1 2016, 11:32 AM

Can you show me an example of

"The profile data seems odd. There is a single profile sample of the
loop being entered. On the other hand, there are no weights inside the
loop."

I want to know what is going on? Are your using FE PGO or IR PGO?

anemet added a comment.Sep 1 2016, 2:58 PM

Can you show me an example of

"The profile data seems odd. There is a single profile sample of the
loop being entered. On the other hand, there are no weights inside the
loop."

I want to know what is going on? Are your using FE PGO or IR PGO?

It's FE-based.

The CFG of the function with the branch weights from the IR looks like this: https://reviews.llvm.org/F2391730. I believe the IR was produced with -disable-llvm-optzns.

I can also send you the PGO data file privately if you're interested.

There is a fundamental problem in BFI that it can not handle 0 weight -- to workaround it

  1. FE PGO annotator will always add 1 to weights of both targets unconditionally when annotating the branch
  2. BFI always add 1 to the weight if it is zero.

The end result is that

  1. we will never see code region annoated with zero frequency/count
  2. for FE PGO, all loop trip count appears to be half of the real trip count.

I will try to improve this situation independently.

davidxl added inline comments.Sep 1 2016, 3:41 PM
lib/Transforms/Scalar/JumpThreading.cpp
1633 ↗(On Diff #69943)

Check the 'branch_weight' string.

anemet added a comment.Sep 1 2016, 6:23 PM

There is a fundamental problem in BFI that it can not handle 0 weight -- to workaround it

  1. FE PGO annotator will always add 1 to weights of both targets unconditionally when annotating the branch
  2. BFI always add 1 to the weight if it is zero.

    The end result is that
  3. we will never see code region annoated with zero frequency/count

Ah, that totally explains what's happening here. Is there a PR?

  1. for FE PGO, all loop trip count appears to be half of the real trip count.

Wow, this was going to be next thing for me to investigate, thanks for the insights!

There is no PR for the first problem.

For the loop trip count problem, see https://llvm.org/bugs/show_bug.cgi?id=27791. I have cc'ed you in that bug.

The right solution for problem #1 is

  1. in BPI, when reading meta data to create BP, if there is zero weight, BPI should assign an extreme branch probability value instead of using zero
  2. in BFI, remove the code that add 1 to the zero weight
  3. In FE PGO, do not add one to weight.

The handling in 2) is not ideal for a very special case -- loop with exit edge not executed (because for instance call to no return function). In such as, BFI will still produce huge frequency for the loop body .. A real solution may require PGOUse to annotate the loop body with the real count of the loop header.

anemet updated this revision to Diff 70206.Sep 2 2016, 12:16 PM
anemet edited edge metadata.

Addressed David's comment.

davidxl accepted this revision.Sep 3 2016, 12:25 PM
davidxl edited edge metadata.

lgtm with some test change suggestion.

test/Transforms/JumpThreading/static-profile.ll
79 ↗(On Diff #70206)

Perhaps check threading happens here and this branch has no meta data

CHECK: br .... check2_thread, .... label %check_2$
CHECK-NEXT: check_2.thread:

87 ↗(On Diff #70206)

similarly as above : check threading happens and no meta data is annotated here.

This revision is now accepted and ready to land.Sep 3 2016, 12:25 PM
anemet added inline comments.Sep 6 2016, 9:09 AM
test/Transforms/JumpThreading/static-profile.ll
79 ↗(On Diff #70206)

Right!

Thanks for the review!

This revision was automatically updated to reflect the committed changes.