This is an archive of the discontinued LLVM Phabricator instance.

Differential D109551

[AutoFDO][llvm-profgen] Profile generation for LBR(non-CS) sample
ClosedPublic

Authored by wlei on Sep 9 2021, 3:34 PM.

Details

Reviewers
wenlei
hoy
wmi
Commits
rGd5f2013004ef: [AutoFDO][llvm-profgen] Profile generation for LBR(non-CS) sample
Summary

This patch introduces non-CS AutoFDO profile generation into LLVM. The profile is supposed to be well consumed by compiler using -fprofile-sample-use=[profile].

After range and branch counters are extracted from the LBR sample, here we go through each addresses for symbolization, create FunctionSamples and populate its sub fields like TotalSamples, BodySamples and HeadSamples etc. For inlined code, as we need to map back to original code, so we always add body samples to the leaf frame's function sample.

Diff Detail

Event Timeline

wlei created this revision.Sep 9 2021, 3:34 PM
Herald added subscribers: hoy, wenlei, lxfind. · View Herald TranscriptSep 9 2021, 3:34 PM
wlei requested review of this revision.Sep 9 2021, 3:34 PM
Herald added a project: Restricted Project. · View Herald TranscriptSep 9 2021, 3:34 PM
Herald added a subscriber: llvm-commits. · View Herald Transcript
wlei retitled this revision from [llvm-profgen] Profile generation for LBR(non-CS) sample to [AutoFDO][llvm-profgen] Profile generation for LBR(non-CS) sample.Sep 9 2021, 4:10 PM
wlei edited the summary of this revision. (Show Details)
wlei added reviewers: wenlei, hoy, wmi.
Harbormaster completed remote builds in B123320: Diff 371735.Sep 9 2021, 4:30 PM
hoy added inline comments.Sep 10 2021, 11:22 AM
llvm/tools/llvm-profgen/ProfileGenerator.cpp
250

nit: use setContext. It's also consistent with getTopLevelFunctionProfile.

277

This is covered by the following while loop?

305

I just noticed that this call may result in empty entries populated in FuncStartAddrMap. Maybe refactor it to use find?

StringRef getFuncFromStartOffset(uint64_t Offset) {
   return FuncStartAddrMap[Offset];
}
wlei updated this revision to Diff 372022.Sep 10 2021, 3:29 PM
wlei marked an inline comment as done.

addressing Hongtao's feedback

llvm/tools/llvm-profgen/ProfileGenerator.cpp
250

Done!

277

Good catch, also remove the one in CSProfileGenerator.

305

Good catch!

Harbormaster completed remote builds in B123516: Diff 372022.Sep 10 2021, 4:05 PM
hoy added inline comments.Sep 10 2021, 5:28 PM
llvm/tools/llvm-profgen/ProfileGenerator.cpp
294

may still be good to keep the comment.

llvm/tools/llvm-profgen/ProfileGenerator.h
25–28

Since we are now using this base class to generate non-CS non-probe profile, add a header comment to clarify that?

64

Make these new functions private since they are not supposed to be called by the derived classes?

68

Nit: a helper function ...

wlei updated this revision to Diff 372291.Sep 13 2021, 10:11 AM

addressing Hongtao's feedback

hoy accepted this revision.Sep 13 2021, 10:25 AM

lgtm

This revision is now accepted and ready to land.Sep 13 2021, 10:25 AM
Harbormaster completed remote builds in B123703: Diff 372291.Sep 13 2021, 11:07 AM
wenlei added inline comments.Sep 21 2021, 12:49 PM
llvm/test/tools/llvm-profgen/inline-noprobe.test
11

Add a test case to cover call site profile for inlinees?

llvm/test/tools/llvm-profgen/noinline-noprobe.test
12

We may want to strengthen the tests a bit since breaking llvm-profgen would be quite impactful for AutoFDO performance.

  • Add a test to cover call site profile with multiple targets?
  • Test discriminator: we have correct base discriminator as well as duplication factor in the generated profile.
  • Different profile format, extended binary format has profiled symbol list (it's ok to deal with symbol list later and separately)
  • Cover these cases for nested inlinee profile too.
llvm/tools/llvm-profgen/ProfileGenerator.cpp
317

move this assertion to the top?

wenlei added inline comments.Sep 22 2021, 6:05 PM
llvm/tools/llvm-profgen/ProfileGenerator.h
25–28

I am thinking about whether profile generation should follow the structure of perf reader. It looks like there's not much overlap for CS and non-CS profile as the profile format is different -pretty much updateBodySamplesforFunctionProfile is the only function from the base that is being reused? If that's the case, would it be better to have a separate ProfileGenerator base as interface then two subclass, one for CS one for non-CS?

49–50

I was initially a bit confused about why these are not virtual since CSProfileGenerator defines them too. Then I found they have different params (still confused why).. Now I understand here it's for all functions, but in CSProfileGenerator it's for one function. Perhaps we can name them populateBodySamplesForFunction vs populateBodySamplesForAllFunctions? Same for populateFunctionBoundarySamples?

wlei added inline comments.Sep 24 2021, 9:17 AM
llvm/test/tools/llvm-profgen/noinline-noprobe.test
12

added a large test case (quick_sort), it will cover the call site with multiple targets and nested inlinee profile.

Test discriminator: we have correct base discriminator as well as duplication factor in the generated profile

I had a standalone test case for discriminator test , see https://reviews.llvm.org/D109934

Different profile format, extended binary format has profiled symbol list (it's ok to deal with symbol list later and separately)

Sounds good, will add it shortly.

llvm/tools/llvm-profgen/ProfileGenerator.cpp
317

fixed.

llvm/tools/llvm-profgen/ProfileGenerator.h
25–28

Good point, four sub-class is really harder to maintain and extend, refactored.

49–50

Sounds good!

wlei updated this revision to Diff 374873.Sep 24 2021, 9:19 AM

Addressing feedback:

  1. add a stonger test case through a quick sort program
  2. refactor ProfileGenerator breaking into two subclass: CS and non-CS
Harbormaster completed remote builds in B125589: Diff 374873.Sep 24 2021, 9:50 AM
wenlei added a comment.Sep 24 2021, 10:24 AM

Thanks for refactoring, looks good. and we can deal with other format and profiled symbol list later in separate patch. Two more comments inline.

llvm/tools/llvm-profgen/ProfileGenerator.cpp
220

This might be a behavior change for AutoFDO. I did see negative offsets in the past - if the code is from macro expansion which is defined before the function start. We would not have any profile if we filter them out.

llvm/tools/llvm-profgen/ProfileGenerator.h
34

why do we need this?

wenlei added subscribers: xur, davidxl.Sep 24 2021, 10:47 AM

I just looked through features we have in internal tool, here's a list of things we need in addition to the basic support here and duplication factor. We can port them in later patches.

  • profiled symbol list as mentioned.
  • coroutine support to attribute sample on coroutine funclets back to main function.
  • split function support to handle profiling on BOLT binaries.
  • heuristic to compute and report profile density and make profiling suggestion.

cc @wmi @davidxl @xur hope this will soon be the unified tool for sample PGO profile generation including FS-AFDO too. We plan to switch to llvm-profgen for AutoFDO production use when it's ready.

davidxl added a subscriber: rajeshwarv.Sep 24 2021, 11:38 AM
wlei added a comment.Sep 24 2021, 1:14 PM
In D109551#3021298, @wenlei wrote:

I just looked through features we have in internal tool, here's a list of things we need in addition to the basic support here and duplication factor. We can port them in later patches.

  • profiled symbol list as mentioned.
  • coroutine support to attribute sample on coroutine funclets back to main function.
  • split function support to handle profiling on BOLT binaries.
  • heuristic to compute and report profile density and make profiling suggestion.

Patches are on the way, thanks for the summary.

llvm/tools/llvm-profgen/ProfileGenerator.cpp
220

The invalid number will be fixed in https://reviews.llvm.org/D110081. As we discuss offline, we might have the invalid case in non-macro case.

llvm/tools/llvm-profgen/ProfileGenerator.h
34

We need a virtual destructor for the inheritance class.

wenlei accepted this revision.Sep 24 2021, 1:19 PM

lgtm, thanks.

This revision was landed with ongoing or failed builds.Sep 24 2021, 1:56 PM
Closed by commit rGd5f2013004ef: [AutoFDO][llvm-profgen] Profile generation for LBR(non-CS) sample (authored by wlei). · Explain Why
This revision was automatically updated to reflect the committed changes.
wlei added a commit: rGd5f2013004ef: [AutoFDO][llvm-profgen] Profile generation for LBR(non-CS) sample.

Revision Contents

Diff 374954

llvm/test/tools/llvm-profgen/Inputs/inline-noprobe.perfbin

  • This binary file was added.
PropertyOld ValueNew Value
File Modenull100755

llvm/test/tools/llvm-profgen/Inputs/inline-noprobe.perfscript

  • This file was added.
400691 0x400691/0x400669/P/-/-/4 0x400691/0x400669/P/-/-/6 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/4 0x400691/0x400669/P/-/-/5 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/4 0x400691/0x400669/P/-/-/5 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/4 0x400691/0x400669/P/-/-/6 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/4 0x400691/0x400669/P/-/-/5 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/4 0x400691/0x400669/P/-/-/5 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/5 0x400691/0x400669/P/-/-/6 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/5 0x400691/0x400669/P/-/-/5 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/4 0x400691/0x400669/P/-/-/7 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/5 0x400691/0x400669/P/-/-/6 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/6 0x400691/0x400669/P/-/-/6
400691 0x400691/0x400669/P/-/-/6 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/3 0x400691/0x400669/P/-/-/5 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/3 0x400691/0x400669/P/-/-/5 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/3 0x400691/0x400669/P/-/-/5 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/4 0x400691/0x400669/P/-/-/6 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/4 0x400691/0x400669/P/-/-/5 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/5 0x400691/0x400669/P/-/-/6 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/4 0x400691/0x400669/P/-/-/6 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/3 0x400691/0x400669/P/-/-/5 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/4 0x400691/0x400669/P/-/-/5 0x400677/0x400650/P/-/-/1 0x400691/0x400669/P/-/-/4 0x400691/0x400669/P/-/-/6 0x400677/0x400650/P/-/-/1

llvm/test/tools/llvm-profgen/Inputs/inline-noprobe2.perfbin

  • This binary file was added.
PropertyOld ValueNew Value
File Modenull100755

llvm/test/tools/llvm-profgen/Inputs/inline-noprobe2.perfscript

  • This file was added.
4008bf 0x4008bf/0x400870/P/-/-/7 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/4 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/2 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1 0x400540/0x7f7448e88cb0/P/-/-/1 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/12 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/7 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/6 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1 0x400540/0x7f7448e88cb0/P/-/-/1 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/9 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/6 0x7f7448e88840/0x7f7448e8885c/P/-/-/2 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889b0/0x7f7448e88826/P/-/-/11
7f7448e889e4 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/4 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/2 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1 0x400540/0x7f7448e88cb0/P/-/-/1 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/6 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/4 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/2 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1 0x400540/0x7f7448e88cb0/P/-/-/1 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/7 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/4 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/2 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1
400788 0x400788/0x4007a8/P/-/-/22 0x400753/0x400770/M/-/-/1 0x40075b/0x400743/P/-/-/3 0x40075b/0x400743/P/-/-/20 0x40075b/0x400743/P/-/-/3 0x40075b/0x400743/P/-/-/3 0x40075b/0x400743/P/-/-/20 0x40075b/0x400743/P/-/-/16 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/1 0x40075b/0x400743/P/-/-/1 0x40072f/0x400755/P/-/-/3 0x4007a6/0x400710/P/-/-/3 0x4007b8/0x400790/P/-/-/2 0x400788/0x4007a8/P/-/-/24 0x400753/0x400770/M/-/-/1 0x40076e/0x400740/P/-/-/27 0x40075b/0x400743/P/-/-/19 0x40075b/0x400743/M/-/-/2 0x40076e/0x400740/P/-/-/8 0x40076e/0x400740/P/-/-/26 0x40075b/0x400743/M/-/-/28 0x40075b/0x400743/M/-/-/2 0x40076e/0x400740/P/-/-/3 0x40076e/0x400740/P/-/-/4 0x40076e/0x400740/P/-/-/27 0x40076e/0x400740/P/-/-/25 0x40076e/0x400740/P/-/-/46 0x40075b/0x400743/P/-/-/19 0x40076e/0x400740/P/-/-/65 0x40075b/0x400743/M/-/-/7 0x40075f/0x400740/P/-/-/17
4007ca 0x4007ca/0x400790/P/-/-/3 0x4007d7/0x4007bd/P/-/-/3 0x4007d7/0x4007cf/P/-/-/3 0x4007d7/0x4007cf/P/-/-/1 0x400792/0x4007d7/P/-/-/1 0x4007ca/0x400790/P/-/-/2 0x4007d7/0x4007bd/P/-/-/1 0x400792/0x4007d7/P/-/-/1 0x4007b8/0x400790/P/-/-/2 0x400788/0x4007a8/P/-/-/30 0x400753/0x400770/M/-/-/16 0x40075b/0x400743/M/-/-/1 0x40072f/0x400755/P/-/-/3 0x4007a6/0x400710/P/-/-/27 0x4007ca/0x400790/P/-/-/3 0x4007d7/0x4007bd/P/-/-/3 0x4007d7/0x4007cf/P/-/-/1 0x400792/0x4007d7/P/-/-/1 0x4007ca/0x400790/P/-/-/2 0x4007d7/0x4007bd/P/-/-/1 0x400792/0x4007d7/P/-/-/1 0x4007b8/0x400790/P/-/-/2 0x400788/0x4007a8/P/-/-/29 0x400753/0x400770/M/-/-/4 0x40075f/0x400740/P/-/-/14 0x40072f/0x400755/P/-/-/5 0x4007a6/0x400710/P/-/-/22 0x4007b8/0x400790/P/-/-/2 0x400788/0x4007a8/P/-/-/26 0x400753/0x400770/M/-/-/1 0x40075b/0x400743/P/-/-/24 0x40075b/0x400743/M/-/-/7
400700 0x400700/0x4006b0/P/-/-/47 0x4006bf/0x4006d0/M/-/-/1 0x4006c8/0x4006b0/P/-/-/2 0x4006ea/0x4006b0/P/-/-/26 0x4006bf/0x4006d0/M/-/-/3 0x4006c8/0x4006b0/P/-/-/25 0x4006c8/0x4006b0/P/-/-/44 0x4006ea/0x4006b0/M/-/-/26 0x4006bf/0x4006d0/M/-/-/3 0x4006c8/0x4006b0/P/-/-/23 0x400700/0x4006b0/P/-/-/9 0x4006ca/0x4006ec/P/-/-/39 0x400700/0x4006b0/P/-/-/9 0x4006ca/0x4006ec/P/-/-/37 0x4006ea/0x4006b0/M/-/-/6 0x4006bf/0x4006d0/P/-/-/3 0x4006c8/0x4006b0/P/-/-/23 0x400700/0x4006b0/P/-/-/27 0x4006bf/0x4006d0/P/-/-/1 0x400700/0x4006b0/P/-/-/9 0x4006ca/0x4006ec/P/-/-/40 0x4006ea/0x4006b0/P/-/-/2 0x4006bf/0x4006d0/P/-/-/1 0x4006ea/0x4006b0/P/-/-/25 0x4006bf/0x4006d0/M/-/-/6 0x400689/0x4006b9/P/-/-/10 0x40066d/0x400686/M/-/-/2 0x4007a6/0x400650/P/-/-/3 0x4007ca/0x400790/P/-/-/9 0x4007d7/0x4007bd/P/-/-/3 0x4007d7/0x4007cf/P/-/-/3 0x4007d7/0x4007cf/P/-/-/3
4007a6 0x4007a6/0x400650/M/-/-/3 0x4007ca/0x400790/P/-/-/1 0x4007d7/0x4007bd/P/-/-/3 0x4007d7/0x4007cf/P/-/-/2 0x4007d7/0x4007cf/P/-/-/4 0x4007d7/0x4007cf/P/-/-/1 0x4007d7/0x4007cf/P/-/-/2 0x4007d7/0x4007cf/P/-/-/1 0x4007d7/0x4007cf/P/-/-/1 0x400792/0x4007d7/P/-/-/1 0x4007ca/0x400790/P/-/-/1 0x4007d7/0x4007bd/P/-/-/1 0x400792/0x4007d7/P/-/-/1 0x4007b8/0x400790/P/-/-/2 0x4006a2/0x4007a8/P/-/-/4 0x400675/0x400682/P/-/-/3 0x4007a6/0x400650/P/-/-/9 0x4007ca/0x400790/P/-/-/1 0x4007d7/0x4007bd/P/-/-/1 0x400792/0x4007d7/P/-/-/1 0x4007b8/0x400790/P/-/-/2 0x4006a2/0x4007a8/P/-/-/1 0x4006b7/0x40068b/P/-/-/1 0x4006c8/0x4006b0/P/-/-/1 0x400689/0x4006b9/P/-/-/1 0x40066d/0x400686/P/-/-/2 0x4007a6/0x400650/P/-/-/9 0x4007ca/0x400790/P/-/-/8 0x4007d7/0x4007bd/P/-/-/1 0x400792/0x4007d7/P/-/-/1 0x4007b8/0x400790/P/-/-/2 0x4006a2/0x4007a8/P/-/-/3
40076e 0x40076e/0x400740/P/-/-/25 0x40075b/0x400743/M/-/-/2 0x40076e/0x400740/P/-/-/5 0x40076e/0x400740/P/-/-/26 0x40076e/0x400740/P/-/-/24 0x40075b/0x400743/P/-/-/1 0x40072f/0x400755/P/-/-/2 0x4007a6/0x400710/P/-/-/4 0x4007b8/0x400790/P/-/-/2 0x400788/0x4007a8/P/-/-/23 0x400753/0x400770/M/-/-/2 0x40076e/0x400740/P/-/-/3 0x40076e/0x400740/P/-/-/4 0x40076e/0x400740/P/-/-/5 0x40076e/0x400740/P/-/-/4 0x40076e/0x400740/P/-/-/26 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40076e/0x400740/P/-/-/25 0x40076e/0x400740/P/-/-/25 0x40076e/0x400740/P/-/-/4 0x40075b/0x400743/P/-/-/1 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40076e/0x400740/P/-/-/49 0x40075b/0x400743/M/-/-/22 0x40075b/0x400743/M/-/-/2 0x40076e/0x400740/P/-/-/25 0x40076e/0x400740/P/-/-/24 0x40075b/0x400743/P/-/-/2 0x40076e/0x400740/P/-/-/48 0x40075b/0x400743/M/-/-/2
400870 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/7 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/4 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/2 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1 0x400540/0x7f7448e88cb0/P/-/-/1 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/5 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/4 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/2 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1 0x400540/0x7f7448e88cb0/P/-/-/1 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/6 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1
4007d7 0x4007d7/0x4007cf/P/-/-/19 0x400792/0x4007d7/M/-/-/1 0x4007ca/0x400790/P/-/-/3 0x4007d7/0x4007bd/P/-/-/2 0x4007d7/0x4007cf/P/-/-/2 0x400792/0x4007d7/P/-/-/1 0x4007ca/0x400790/P/-/-/1 0x4007d7/0x4007bd/P/-/-/1 0x400792/0x4007d7/P/-/-/1 0x4007b8/0x400790/P/-/-/2 0x4006a2/0x4007a8/P/-/-/27 0x4007a6/0x400650/P/-/-/3 0x4007b8/0x400790/P/-/-/2 0x4006a2/0x4007a8/P/-/-/24 0x4006b7/0x40068b/M/-/-/1 0x400700/0x4006b0/P/-/-/48 0x4006bf/0x4006d0/M/-/-/6 0x400689/0x4006b9/P/-/-/8 0x40066d/0x400686/M/-/-/2 0x4007a6/0x400650/P/-/-/3 0x4007b8/0x400790/P/-/-/2 0x4006a2/0x4007a8/P/-/-/30 0x4006b7/0x40068b/M/-/-/19 0x4006c8/0x4006b0/M/-/-/20 0x400700/0x4006b0/P/-/-/6 0x4006bf/0x4006d0/P/-/-/2 0x400689/0x4006b9/P/-/-/21 0x4007a6/0x400650/P/-/-/10 0x4007ca/0x400790/P/-/-/7 0x4007d7/0x4007bd/P/-/-/1 0x400792/0x4007d7/P/-/-/1 0x4007b8/0x400790/P/-/-/2
400870 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/7 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/4 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/2 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1 0x400540/0x7f7448e88cb0/P/-/-/1 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/7 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/4 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/2 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1 0x400540/0x7f7448e88cb0/P/-/-/1 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/13 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1
4008bf 0x4008bf/0x400870/P/-/-/6 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/4 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/2 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1 0x400540/0x7f7448e88cb0/P/-/-/1 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/7 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/4 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/2 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1 0x400540/0x7f7448e88cb0/P/-/-/1 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/7 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1
40075b 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/1 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/1 0x40075b/0x400743/P/-/-/3 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/3 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/2 0x40075b/0x400743/P/-/-/3 0x40075b/0x400743/P/-/-/19 0x40075b/0x400743/P/-/-/2 0x40072f/0x400755/P/-/-/13 0x4007a6/0x400710/M/-/-/12 0x40082f/0x400790/P/-/-/7
4008bf 0x4008bf/0x400870/P/-/-/6 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/4 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/2 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1 0x400540/0x7f7448e88cb0/P/-/-/1 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/7 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1 0x7f7448e8899f/0x7f7448e889d8/P/-/-/4 0x7f7448e88821/0x7f7448e88960/P/-/-/1 0x7f7448e887f9/0x7f7448e88815/P/-/-/2 0x7f7448e887e4/0x7f7448e887f2/P/-/-/2 0x7f7448e88cb8/0x7f7448e887c0/P/-/-/1 0x400540/0x7f7448e88cb0/P/-/-/1 0x400870/0x400540/P/-/-/1 0x4008bf/0x400870/P/-/-/7 0x7f7448e88cc1/0x400875/P/-/-/1 0x7f7448e88875/0x7f7448e88cbd/P/-/-/2 0x7f7448e88840/0x7f7448e8885c/P/-/-/1 0x7f7448e8882d/0x7f7448e8883a/P/-/-/1 0x7f7448e889e4/0x7f7448e88826/P/-/-/1

llvm/test/tools/llvm-profgen/inline-noprobe.test

  • This file was added.
; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-noprobe.perfscript --binary=%S/Inputs/inline-noprobe.perfbin --output=%t --skip-symbolization
; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-RAW-PROFILE
; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-noprobe.perfscript --binary=%S/Inputs/inline-noprobe.perfbin --output=%t
; RUN: FileCheck %s --input-file %t --check-prefix=CHECK
CHECK: main:836:0
CHECK: 1: foo:836
CHECK: 2.1: 42
CHECK: 3: 62
CHECK: 3.2: 21
CHECK: 3.1: bar:252
wenleiUnsubmitted
Not Done
ReplyInline Actions

Add a test case to cover call site profile for inlinees?

wenlei: Add a test case to cover call site profile for inlinees?
CHECK: 1: 42
CHECK: 3.2: bar:63
CHECK: 1: 21
CHECK-RAW-PROFILE: 3
CHECK-RAW-PROFILE-NEXT: 650-691:21
CHECK-RAW-PROFILE-NEXT: 669-677:20
CHECK-RAW-PROFILE-NEXT: 669-691:21
CHECK-RAW-PROFILE-NEXT: 2
CHECK-RAW-PROFILE-NEXT: 677->650:21
CHECK-RAW-PROFILE-NEXT: 691->669:43
; original code:
; clang -O3 -g -fdebug-info-for-profiling test.c -o a.out
#include <stdio.h>
int bar(int x, int y) {
if (x % 2) {
return x - y;
}
return x + y;
}
void foo() {
int s, i = 0;
while (i++ < 4000 * 4000)
if (i % 3) s = bar(i, s); else s += bar(s, i);
printf("sum is %d\n", s);
}
int main() {
foo();
return 0;
}

llvm/test/tools/llvm-profgen/inline-noprobe2.test

  • This file was added.
; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-noprobe2.perfscript --binary=%S/Inputs/inline-noprobe2.perfbin --output=%t
; RUN: FileCheck %s --input-file %t --check-prefix=CHECK
;CHECK: partition_pivot_first:1050:5
;CHECK-NEXT: 0: 5
;CHECK-NEXT: 1: 5
;CHECK-NEXT: 2: 5
;CHECK-NEXT: 3: 5
;CHECK-NEXT: 3.1: 83
;CHECK-NEXT: 4: 82
;CHECK-NEXT: 4.1: 26
;CHECK-NEXT: 4.2: 25
;CHECK-NEXT: 4.3: 26
;CHECK-NEXT: 5: 6
;CHECK-NEXT: 6: 6
;CHECK-NEXT: 4.2: swap:100
;CHECK-NEXT: 1: 25
;CHECK-NEXT: 2: 25
;CHECK-NEXT: 3: 25
;CHECK-NEXT: 5: swap:24
;CHECK-NEXT: 1: 6
;CHECK-NEXT: 2: 6
;CHECK-NEXT: 3: 6
;CHECK-NEXT: quick_sort:414:25
;CHECK-NEXT: 1: 24
;CHECK-NEXT: 2: 12 partition_pivot_last:7 partition_pivot_first:5
;CHECK-NEXT: 3: 11 quick_sort:12
;CHECK-NEXT: 4: 12 quick_sort:12
;CHECK-NEXT: 6: 24
;CHECK-NEXT: partition_pivot_last:391:7
;CHECK-NEXT: 1: 6
;CHECK-NEXT: 2: 6
;CHECK-NEXT: 3: 6
;CHECK-NEXT: 3.1: 18
;CHECK-NEXT: 3.3: 18
;CHECK-NEXT: 4: 19
;CHECK-NEXT: 5: 9
;CHECK-NEXT: 6: 5
;CHECK-NEXT: 7: 5
;CHECK-NEXT: 5: swap:61
;CHECK-NEXT: 1: 9
;CHECK-NEXT: 2: 9
;CHECK-NEXT: 3: 9
;CHECK-NEXT: 6: swap:20
;CHECK-NEXT: 1: 5
;CHECK-NEXT: 2: 5
;CHECK-NEXT: 3: 5
;CHECK-NEXT: main:213:0
;CHECK-NEXT: 5.1: 10
;CHECK-NEXT: 5.3: 10
;CHECK-NEXT: 6: 10
;CHECK-NEXT: 6.1: 12
;CHECK-NEXT: 6.3: 10
;CHECK-NEXT: 8: 0 quick_sort:1
; original code:
; clang -O3 -g -fno-optimize-sibling-calls -fdebug-info-for-profiling qsort.c -o a.out
#include <stdio.h>
#include <stdlib.h>
void swap(int *a, int *b) {
int t = *a;
*a = *b;
*b = t;
}
int partition_pivot_last(int* array, int low, int high) {
int pivot = array[high];
int i = low - 1;
for (int j = low; j < high; j++)
if (array[j] < pivot)
swap(&array[++i], &array[j]);
swap(&array[i + 1], &array[high]);
return (i + 1);
}
int partition_pivot_first(int* array, int low, int high) {
int pivot = array[low];
int i = low + 1;
for (int j = low + 1; j <= high; j++)
if (array[j] < pivot) { if (j != i) swap(&array[i], &array[j]); i++;}
swap(&array[i - 1], &array[low]);
return i - 1;
}
void quick_sort(int* array, int low, int high, int (*partition_func)(int *, int, int)) {
if (low < high) {
int pi = (*partition_func)(array, low, high);
quick_sort(array, low, pi - 1, partition_func);
quick_sort(array, pi + 1, high, partition_func);
}
}
int main() {
const int size = 200;
int sum = 0;
int *array = malloc(size * sizeof(int));
for(int i = 0; i < 100 * 1000; i++) {
for(int j = 0; j < size; j++)
array[j] = j % 10 ? rand() % size: j;
int (*fptr)(int *, int, int) = i % 3 ? partition_pivot_last : partition_pivot_first;
quick_sort(array, 0, size - 1, fptr);
sum += array[i % size];
}
printf("sum=%d\n", sum);
return 0;
}

llvm/test/tools/llvm-profgen/noinline-noprobe.test

  • This file was moved from llvm/test/tools/llvm-profgen/noprobe.test.
; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noprobe.perfscript --binary=%S/Inputs/noprobe.perfbin --output=%t --skip-symbolization ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noprobe.perfscript --binary=%S/Inputs/noprobe.perfbin --output=%t --skip-symbolization
; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-RAW-PROFILE ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-RAW-PROFILE
; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noprobe.mmap.perfscript --binary=%S/Inputs/noprobe.perfbin --output=%t --skip-symbolization ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noprobe.mmap.perfscript --binary=%S/Inputs/noprobe.perfbin --output=%t --skip-symbolization
; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-RAW-PROFILE ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-RAW-PROFILE
; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noprobe.aggperfscript --binary=%S/Inputs/noprobe.perfbin --output=%t --skip-symbolization ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noprobe.aggperfscript --binary=%S/Inputs/noprobe.perfbin --output=%t --skip-symbolization
; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-RAW-PROFILE-AGG ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-RAW-PROFILE-AGG
; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noprobe.perfscript --binary=%S/Inputs/noprobe.perfbin --output=%t
; RUN: FileCheck %s --input-file %t --check-prefix=CHECK
CHECK: foo:360:0
CHECK: 2: 19
CHECK: 3: 19 bar:21
wenleiUnsubmitted
Not Done
ReplyInline Actions

We may want to strengthen the tests a bit since breaking llvm-profgen would be quite impactful for AutoFDO performance.

  • Add a test to cover call site profile with multiple targets?
  • Test discriminator: we have correct base discriminator as well as duplication factor in the generated profile.
  • Different profile format, extended binary format has profiled symbol list (it's ok to deal with symbol list later and separately)
  • Cover these cases for nested inlinee profile too.
wenlei: We may want to strengthen the tests a bit since breaking llvm-profgen would be quite impactful…
wleiAuthorUnsubmitted
Done
ReplyInline Actions

added a large test case (quick_sort), it will cover the call site with multiple targets and nested inlinee profile.

Test discriminator: we have correct base discriminator as well as duplication factor in the generated profile

I had a standalone test case for discriminator test , see https://reviews.llvm.org/D109934

Different profile format, extended binary format has profiled symbol list (it's ok to deal with symbol list later and separately)

Sounds good, will add it shortly.

wlei: added a large test case (quick_sort), it will cover the call site with multiple targets and…
CHECK: bar:324:21
CHECK: 0: 20
CHECK: 1: 20
CHECK: 2: 13
CHECK: 4: 6
CHECK: 5: 18
CHECK-RAW-PROFILE: 7 CHECK-RAW-PROFILE: 7
CHECK-RAW-PROFILE-NEXT: 5b0-5c8:7 CHECK-RAW-PROFILE-NEXT: 5b0-5c8:7
CHECK-RAW-PROFILE-NEXT: 5b0-5d7:13 CHECK-RAW-PROFILE-NEXT: 5b0-5d7:13
CHECK-RAW-PROFILE-NEXT: 5dc-5e9:6 CHECK-RAW-PROFILE-NEXT: 5dc-5e9:6
CHECK-RAW-PROFILE-NEXT: 5e5-5e9:12 CHECK-RAW-PROFILE-NEXT: 5e5-5e9:12
CHECK-RAW-PROFILE-NEXT: 5ff-62f:19 CHECK-RAW-PROFILE-NEXT: 5ff-62f:19
CHECK-RAW-PROFILE-NEXT: 634-637:18 CHECK-RAW-PROFILE-NEXT: 634-637:18
Show All 20 Lines
CHECK-RAW-PROFILE-AGG-NEXT: 5d7->5e5:21 CHECK-RAW-PROFILE-AGG-NEXT: 5d7->5e5:21
CHECK-RAW-PROFILE-AGG-NEXT: 5e9->634:30 CHECK-RAW-PROFILE-AGG-NEXT: 5e9->634:30
CHECK-RAW-PROFILE-AGG-NEXT: 62f->5b0:35 CHECK-RAW-PROFILE-AGG-NEXT: 62f->5b0:35
CHECK-RAW-PROFILE-AGG-NEXT: 637->645:30 CHECK-RAW-PROFILE-AGG-NEXT: 637->645:30
CHECK-RAW-PROFILE-AGG-NEXT: 645->5ff:31 CHECK-RAW-PROFILE-AGG-NEXT: 645->5ff:31
; original code: ; original code:
; clang -O3 -g -debug-info-for-profiling test.c -o a.out ; clang -O3 -g -fdebug-info-for-profiling test.c -fno-inline -o a.out
#include <stdio.h> #include <stdio.h>
int bar(int x, int y) { int bar(int x, int y) {
if (x % 3) { if (x % 3) {
return x - y; return x - y;
} }
return x + y; return x + y;
} }
Show All 12 Lines

llvm/test/tools/llvm-profgen/noprobe.test

  • This file was moved to llvm/test/tools/llvm-profgen/noinline-noprobe.test.

llvm/tools/llvm-profgen/ProfileGenerator.h

Show All 16 Lines
#include <unordered_set> #include <unordered_set>
using namespace llvm; using namespace llvm;
using namespace sampleprof; using namespace sampleprof;
namespace llvm { namespace llvm {
namespace sampleprof { namespace sampleprof {
class ProfileGenerator { // This base class for profile generation of sample-based PGO. We reuse all
// structures relating to function profiles and profile writers as seen in
// /ProfileData/SampleProf.h.
class ProfileGeneratorBase {
hoyUnsubmitted
Not Done
ReplyInline Actions

Since we are now using this base class to generate non-CS non-probe profile, add a header comment to clarify that?

hoy: Since we are now using this base class to generate non-CS non-probe profile, add a header…
wenleiUnsubmitted
Not Done
ReplyInline Actions

I am thinking about whether profile generation should follow the structure of perf reader. It looks like there's not much overlap for CS and non-CS profile as the profile format is different -pretty much updateBodySamplesforFunctionProfile is the only function from the base that is being reused? If that's the case, would it be better to have a separate ProfileGenerator base as interface then two subclass, one for CS one for non-CS?

wenlei: I am thinking about whether profile generation should follow the structure of perf reader. It…
wleiAuthorUnsubmitted
Done
ReplyInline Actions

Good point, four sub-class is really harder to maintain and extend, refactored.

wlei: Good point, four sub-class is really harder to maintain and extend, refactored.
public: public:
ProfileGenerator(ProfiledBinary *B) : Binary(B){}; ProfileGeneratorBase(ProfiledBinary *Binary,
virtual ~ProfileGenerator() = default; const ContextSampleCounterMap &Counters)
static std::unique_ptr<ProfileGenerator> : Binary(Binary), SampleCounters(Counters){};
virtual ~ProfileGeneratorBase() = default;
wenleiUnsubmitted
Not Done
ReplyInline Actions

why do we need this?

wenlei: why do we need this?
wleiAuthorUnsubmitted
Done
ReplyInline Actions

We need a virtual destructor for the inheritance class.

wlei: We need a virtual destructor for the inheritance class.
static std::unique_ptr<ProfileGeneratorBase>
create(ProfiledBinary *Binary, const ContextSampleCounterMap &SampleCounters, create(ProfiledBinary *Binary, const ContextSampleCounterMap &SampleCounters,
enum PerfScriptType SampleType); enum PerfScriptType SampleType);
virtual void generateProfile() = 0; virtual void generateProfile() = 0;
// Use SampleProfileWriter to serialize profile map
virtual void write(std::unique_ptr<SampleProfileWriter> Writer,
SampleProfileMap &ProfileMap);
void write(); void write();
protected: protected:
// Use SampleProfileWriter to serialize profile map
void write(std::unique_ptr<SampleProfileWriter> Writer,
SampleProfileMap &ProfileMap);
/* /*
For each region boundary point, mark if it is begin or end (or both) of For each region boundary point, mark if it is begin or end (or both) of
the region. Boundary points are inclusive. Log the sample count as well the region. Boundary points are inclusive. Log the sample count as well
so we can use it when we compute the sample count of each disjoint region so we can use it when we compute the sample count of each disjoint region
later. Note that there might be multiple ranges with different sample later. Note that there might be multiple ranges with different sample
count that share same begin/end point. We need to accumulate the sample count that share same begin/end point. We need to accumulate the sample
wenleiUnsubmitted
Not Done
ReplyInline Actions

I was initially a bit confused about why these are not virtual since CSProfileGenerator defines them too. Then I found they have different params (still confused why).. Now I understand here it's for all functions, but in CSProfileGenerator it's for one function. Perhaps we can name them populateBodySamplesForFunction vs populateBodySamplesForAllFunctions? Same for populateFunctionBoundarySamples?

wenlei: I was initially a bit confused about why these are not virtual since CSProfileGenerator defines…
wleiAuthorUnsubmitted
Done
ReplyInline Actions

Sounds good!

wlei: Sounds good!
count for the boundary point for such case, because for the example count for the boundary point for such case, because for the example
below, below,
|<--100-->| |<--100-->|
|<------200------>| |<------200------>|
A B C A B C
sample count for disjoint region [A,B] would be 300. sample count for disjoint region [A,B] would be 300.
*/ */
void findDisjointRanges(RangeSample &DisjointRanges, void findDisjointRanges(RangeSample &DisjointRanges,
const RangeSample &Ranges); const RangeSample &Ranges);
// Helper function for updating body sample for a leaf location in
// FunctionProfile
void updateBodySamplesforFunctionProfile(FunctionSamples &FunctionProfile,
hoyUnsubmitted
Not Done
ReplyInline Actions

Make these new functions private since they are not supposed to be called by the derived classes?

hoy: Make these new functions private since they are not supposed to be called by the derived…
const SampleContextFrame &LeafLoc,
uint64_t Count);
// Used by SampleProfileWriter // Used by SampleProfileWriter
hoyUnsubmitted
Not Done
ReplyInline Actions

Nit: a helper function ...

hoy: Nit: a helper function ...
SampleProfileMap ProfileMap; SampleProfileMap ProfileMap;
ProfiledBinary *Binary = nullptr; ProfiledBinary *Binary = nullptr;
};
class CSProfileGenerator : public ProfileGenerator {
protected:
const ContextSampleCounterMap &SampleCounters; const ContextSampleCounterMap &SampleCounters;
};
class ProfileGenerator : public ProfileGeneratorBase {
public: public:
CSProfileGenerator(ProfiledBinary *Binary, ProfileGenerator(ProfiledBinary *Binary,
const ContextSampleCounterMap &Counters) const ContextSampleCounterMap &Counters)
: ProfileGenerator(Binary), SampleCounters(Counters){}; : ProfileGeneratorBase(Binary, Counters){};
void generateProfile() override;
private:
void generateLineNumBasedProfile();
FunctionSamples &getTopLevelFunctionProfile(StringRef FuncName);
// Helper function to get the leaf frame's FunctionProfile by traversing the
// inline stack and meanwhile it adds the total samples for each frame's
// function profile.
FunctionSamples &
getLeafProfileAndAddTotalSamples(const SampleContextFrameVector &FrameVec,
uint64_t Count);
void populateBodySamplesForAllFunctions(const RangeSample &RangeCounter);
void
populateBoundarySamplesForAllFunctions(const BranchSample &BranchCounters);
};
using ProbeCounterMap =
std::unordered_map<const MCDecodedPseudoProbe *, uint64_t>;
class CSProfileGenerator : public ProfileGeneratorBase {
public: public:
CSProfileGenerator(ProfiledBinary *Binary,
const ContextSampleCounterMap &Counters)
: ProfileGeneratorBase(Binary, Counters){};
void generateProfile() override; void generateProfile() override;
// Trim the context stack at a given depth. // Trim the context stack at a given depth.
template <typename T> template <typename T>
static void trimContext(SmallVectorImpl<T> &S, int Depth = MaxContextDepth) { static void trimContext(SmallVectorImpl<T> &S, int Depth = MaxContextDepth) {
if (Depth < 0 || static_cast<size_t>(Depth) >= S.size()) if (Depth < 0 || static_cast<size_t>(Depth) >= S.size())
return; return;
std::copy(S.begin() + S.size() - static_cast<size_t>(Depth), S.end(), std::copy(S.begin() + S.size() - static_cast<size_t>(Depth), S.end(),
▲ Show 20 Lines • Show All 101 Lines • ▼ Show 20 Lines while (I <= MaxDedupSize) {
std::copy(BeginIter + Right + 1, Context.end(), BeginIter + End); std::copy(BeginIter + Right + 1, Context.end(), BeginIter + End);
End += Context.size() - Right - 1; End += Context.size() - Right - 1;
I++; I++;
Context.resize(End); Context.resize(End);
MaxDedupSize = std::min(static_cast<uint32_t>(End / 2), MaxDedupSize); MaxDedupSize = std::min(static_cast<uint32_t>(End / 2), MaxDedupSize);
} }
} }
protected: private:
void generateLineNumBasedProfile();
// Lookup or create FunctionSamples for the context // Lookup or create FunctionSamples for the context
FunctionSamples & FunctionSamples &
getFunctionProfileForContext(const SampleContextFrameVector &Context, getFunctionProfileForContext(const SampleContextFrameVector &Context,
bool WasLeafInlined = false); bool WasLeafInlined = false);
// Post processing for profiles before writing out, such as mermining // Post processing for profiles before writing out, such as mermining
// and trimming cold profiles, running preinliner on profiles. // and trimming cold profiles, running preinliner on profiles.
void postProcessProfiles(); void postProcessProfiles();
void computeSummaryAndThreshold();
void write(std::unique_ptr<SampleProfileWriter> Writer,
SampleProfileMap &ProfileMap) override;
// Thresholds from profile summary to answer isHotCount/isColdCount queries. void computeSummaryAndThreshold();
uint64_t HotCountThreshold;
uint64_t ColdCountThreshold;
// Underlying context table serves for sample profile writer.
std::unordered_set<SampleContextFrameVector, SampleContextFrameHash> Contexts;
private: void populateBodySamplesForFunction(FunctionSamples &FunctionProfile,
// Helper function for updating body sample for a leaf location in
// FunctionProfile
void updateBodySamplesforFunctionProfile(FunctionSamples &FunctionProfile,
const SampleContextFrame &LeafLoc,
uint64_t Count);
void populateFunctionBodySamples(FunctionSamples &FunctionProfile,
const RangeSample &RangeCounters); const RangeSample &RangeCounters);
void populateFunctionBoundarySamples(SampleContextFrames ContextId, void populateBoundarySamplesForFunction(SampleContextFrames ContextId,
FunctionSamples &FunctionProfile, FunctionSamples &FunctionProfile,
const BranchSample &BranchCounters); const BranchSample &BranchCounters);
void populateInferredFunctionSamples(); void populateInferredFunctionSamples();
public: void generateProbeBasedProfile();
// Deduplicate adjacent repeated context sequences up to a given sequence
// length. -1 means no size limit.
static int32_t MaxCompressionSize;
static int MaxContextDepth;
};
using ProbeCounterMap =
std::unordered_map<const MCDecodedPseudoProbe *, uint64_t>;
class PseudoProbeCSProfileGenerator : public CSProfileGenerator {
public:
PseudoProbeCSProfileGenerator(ProfiledBinary *Binary,
const ContextSampleCounterMap &Counters)
: CSProfileGenerator(Binary, Counters) {}
void generateProfile() override;
private:
// Go through each address from range to extract the top frame probe by // Go through each address from range to extract the top frame probe by
// looking up in the Address2ProbeMap // looking up in the Address2ProbeMap
void extractProbesFromRange(const RangeSample &RangeCounter, void extractProbesFromRange(const RangeSample &RangeCounter,
ProbeCounterMap &ProbeCounter); ProbeCounterMap &ProbeCounter);
// Fill in function body samples from probes // Fill in function body samples from probes
void populateBodySamplesWithProbes(const RangeSample &RangeCounter, void populateBodySamplesWithProbes(const RangeSample &RangeCounter,
SampleContextFrames ContextStack); SampleContextFrames ContextStack);
// Fill in boundary samples for a call probe // Fill in boundary samples for a call probe
void populateBoundarySamplesWithProbes(const BranchSample &BranchCounter, void populateBoundarySamplesWithProbes(const BranchSample &BranchCounter,
SampleContextFrames ContextStack); SampleContextFrames ContextStack);
// Helper function to get FunctionSamples for the leaf probe // Helper function to get FunctionSamples for the leaf probe
FunctionSamples & FunctionSamples &
getFunctionProfileForLeafProbe(SampleContextFrames ContextStack, getFunctionProfileForLeafProbe(SampleContextFrames ContextStack,
const MCDecodedPseudoProbe *LeafProbe); const MCDecodedPseudoProbe *LeafProbe);
// Thresholds from profile summary to answer isHotCount/isColdCount queries.
uint64_t HotCountThreshold;
uint64_t ColdCountThreshold;
// Underlying context table serves for sample profile writer.
std::unordered_set<SampleContextFrameVector, SampleContextFrameHash> Contexts;
public:
// Deduplicate adjacent repeated context sequences up to a given sequence
// length. -1 means no size limit.
static int32_t MaxCompressionSize;
static int MaxContextDepth;
}; };
} // end namespace sampleprof } // end namespace sampleprof
} // end namespace llvm } // end namespace llvm
#endif #endif

llvm/tools/llvm-profgen/ProfileGenerator.cpp

Show First 20 Lines • Show All 68 Lines • ▼ Show 20 Lines
namespace llvm { namespace llvm {
namespace sampleprof { namespace sampleprof {
// Initialize the MaxCompressionSize to -1 which means no size limit // Initialize the MaxCompressionSize to -1 which means no size limit
int32_t CSProfileGenerator::MaxCompressionSize = -1; int32_t CSProfileGenerator::MaxCompressionSize = -1;
int CSProfileGenerator::MaxContextDepth = -1; int CSProfileGenerator::MaxContextDepth = -1;
std::unique_ptr<ProfileGenerator> std::unique_ptr<ProfileGeneratorBase>
ProfileGenerator::create(ProfiledBinary *Binary, ProfileGeneratorBase::create(ProfiledBinary *Binary,
const ContextSampleCounterMap &SampleCounters, const ContextSampleCounterMap &SampleCounters,
enum PerfScriptType SampleType) { enum PerfScriptType SampleType) {
std::unique_ptr<ProfileGenerator> ProfileGenerator; std::unique_ptr<ProfileGeneratorBase> Generator;
if (SampleType == PERF_LBR_STACK) { if (SampleType == PERF_LBR) {
if (Binary->usePseudoProbes()) { // TODO: Support probe based profile generation
ProfileGenerator.reset( Generator.reset(new ProfileGenerator(Binary, SampleCounters));
new PseudoProbeCSProfileGenerator(Binary, SampleCounters)); } else if (SampleType == PERF_LBR_STACK) {
} else { Generator.reset(new CSProfileGenerator(Binary, SampleCounters));
ProfileGenerator.reset(new CSProfileGenerator(Binary, SampleCounters));
}
} else { } else {
// TODO:
llvm_unreachable("Unsupported perfscript!"); llvm_unreachable("Unsupported perfscript!");
} }
return ProfileGenerator; return Generator;
} }
void ProfileGenerator::write(std::unique_ptr<SampleProfileWriter> Writer, void ProfileGeneratorBase::write(std::unique_ptr<SampleProfileWriter> Writer,
SampleProfileMap &ProfileMap) { SampleProfileMap &ProfileMap) {
if (std::error_code EC = Writer->write(ProfileMap)) if (std::error_code EC = Writer->write(ProfileMap))
exitWithError(std::move(EC)); exitWithError(std::move(EC));
} }
void ProfileGenerator::write() { void ProfileGeneratorBase::write() {
auto WriterOrErr = SampleProfileWriter::create(OutputFilename, OutputFormat); auto WriterOrErr = SampleProfileWriter::create(OutputFilename, OutputFormat);
if (std::error_code EC = WriterOrErr.getError()) if (std::error_code EC = WriterOrErr.getError())
exitWithError(EC, OutputFilename); exitWithError(EC, OutputFilename);
if (UseMD5) { if (UseMD5) {
if (OutputFormat != SPF_Ext_Binary) if (OutputFormat != SPF_Ext_Binary)
WithColor::warning() << "-use-md5 is ignored. Specify " WithColor::warning() << "-use-md5 is ignored. Specify "
"--format=extbinary to enable it\n"; "--format=extbinary to enable it\n";
else else
WriterOrErr.get()->setUseMD5(); WriterOrErr.get()->setUseMD5();
} }
write(std::move(WriterOrErr.get()), ProfileMap); write(std::move(WriterOrErr.get()), ProfileMap);
} }
void ProfileGenerator::findDisjointRanges(RangeSample &DisjointRanges, void ProfileGeneratorBase::findDisjointRanges(RangeSample &DisjointRanges,
const RangeSample &Ranges) { const RangeSample &Ranges) {
/* /*
Regions may overlap with each other. Using the boundary info, find all Regions may overlap with each other. Using the boundary info, find all
disjoint ranges and their sample count. BoundaryPoint contains the count disjoint ranges and their sample count. BoundaryPoint contains the count
multiple samples begin/end at this points. multiple samples begin/end at this points.
|<--100-->| Sample1 |<--100-->| Sample1
|<------200------>| Sample2 |<------200------>| Sample2
▲ Show 20 Lines • Show All 82 Lines • ▼ Show 20 Lines if (Point.EndCount) {
DisjointRanges[{BeginAddress, Address}] = Count; DisjointRanges[{BeginAddress, Address}] = Count;
assert(Count >= Point.EndCount && "Mismatched live ranges"); assert(Count >= Point.EndCount && "Mismatched live ranges");
Count -= Point.EndCount; Count -= Point.EndCount;
BeginAddress = Address + 1; BeginAddress = Address + 1;
} }
} }
} }
void ProfileGeneratorBase::updateBodySamplesforFunctionProfile(
FunctionSamples &FunctionProfile, const SampleContextFrame &LeafLoc,
uint64_t Count) {
// Filter out invalid negative(int type) lineOffset
if (LeafLoc.Callsite.LineOffset & 0x80000000)
wenleiUnsubmitted
Not Done
ReplyInline Actions

This might be a behavior change for AutoFDO. I did see negative offsets in the past - if the code is from macro expansion which is defined before the function start. We would not have any profile if we filter them out.

wenlei: This might be a behavior change for AutoFDO. I did see negative offsets in the past - if the…
wleiAuthorUnsubmitted
Done
ReplyInline Actions

The invalid number will be fixed in https://reviews.llvm.org/D110081. As we discuss offline, we might have the invalid case in non-macro case.

wlei: The invalid number will be fixed in https://reviews.llvm.org/D110081. As we discuss offline, we…
return;
// Use the maximum count of samples with same line location
ErrorOr<uint64_t> R = FunctionProfile.findSamplesAt(
LeafLoc.Callsite.LineOffset, LeafLoc.Callsite.Discriminator);
uint64_t PreviousCount = R ? R.get() : 0;
if (PreviousCount < Count) {
FunctionProfile.addBodySamples(LeafLoc.Callsite.LineOffset,
LeafLoc.Callsite.Discriminator,
Count - PreviousCount);
}
}
FunctionSamples &
ProfileGenerator::getTopLevelFunctionProfile(StringRef FuncName) {
SampleContext Context(FuncName);
auto Ret = ProfileMap.emplace(Context, FunctionSamples());
if (Ret.second) {
FunctionSamples &FProfile = Ret.first->second;
FProfile.setContext(Context);
}
return Ret.first->second;
}
void ProfileGenerator::generateProfile() {
if (Binary->usePseudoProbes()) {
// TODO: Support probe based profile generation
} else {
generateLineNumBasedProfile();
}
}
hoyUnsubmitted
Done
ReplyInline Actions

nit: use setContext. It's also consistent with getTopLevelFunctionProfile.

hoy: nit: use `setContext`. It's also consistent with `getTopLevelFunctionProfile`.
wleiAuthorUnsubmitted
Done
ReplyInline Actions

Done!

wlei: Done!
void ProfileGenerator::generateLineNumBasedProfile() {
assert(SampleCounters.size() == 1 &&
"Must have one entry for profile generation.");
const SampleCounter &SC = SampleCounters.begin()->second;
// Fill in function body samples
populateBodySamplesForAllFunctions(SC.RangeCounter);
// Fill in boundary sample counts as well as call site samples for calls
populateBoundarySamplesForAllFunctions(SC.BranchCounter);
}
FunctionSamples &ProfileGenerator::getLeafProfileAndAddTotalSamples(
const SampleContextFrameVector &FrameVec, uint64_t Count) {
// Get top level profile
FunctionSamples *FunctionProfile =
&getTopLevelFunctionProfile(FrameVec[0].CallerName);
FunctionProfile->addTotalSamples(Count);
for (size_t I = 1; I < FrameVec.size(); I++) {
FunctionSamplesMap &SamplesMap =
FunctionProfile->functionSamplesAt(FrameVec[I - 1].Callsite);
auto Ret =
SamplesMap.emplace(FrameVec[I].CallerName.str(), FunctionSamples());
if (Ret.second) {
SampleContext Context(FrameVec[I].CallerName);
Ret.first->second.setContext(Context);
}
hoyUnsubmitted
Not Done
ReplyInline Actions

This is covered by the following while loop?

hoy: This is covered by the following while loop?
wleiAuthorUnsubmitted
Done
ReplyInline Actions

Good catch, also remove the one in CSProfileGenerator.

wlei: Good catch, also remove the one in CSProfileGenerator.
FunctionProfile = &Ret.first->second;
FunctionProfile->addTotalSamples(Count);
}
return *FunctionProfile;
}
void ProfileGenerator::populateBodySamplesForAllFunctions(
const RangeSample &RangeCounter) {
RangeSample Ranges;
findDisjointRanges(Ranges, RangeCounter);
for (auto Range : Ranges) {
uint64_t RangeBegin = Binary->offsetToVirtualAddr(Range.first.first);
uint64_t RangeEnd = Binary->offsetToVirtualAddr(Range.first.second);
uint64_t Count = Range.second;
// Disjoint ranges have introduce zero-filled gap that
// doesn't belong to current context, filter them out.
if (Count == 0)
continue;
InstructionPointer IP(Binary, RangeBegin, true);
// Disjoint ranges may have range in the middle of two instr,
// e.g. If Instr1 at Addr1, and Instr2 at Addr2, disjoint range
// can be Addr1+1 to Addr2-1. We should ignore such range.
while (IP.Address <= RangeEnd) {
uint64_t Offset = Binary->virtualAddrToOffset(IP.Address);
const SampleContextFrameVector &FrameVec =
Binary->getFrameLocationStack(Offset);
hoyUnsubmitted
Not Done
ReplyInline Actions

I just noticed that this call may result in empty entries populated in FuncStartAddrMap. Maybe refactor it to use find?

StringRef getFuncFromStartOffset(uint64_t Offset) {
   return FuncStartAddrMap[Offset];
}
hoy: I just noticed that this call may result in empty entries populated in `FuncStartAddrMap`.
wleiAuthorUnsubmitted
Done
ReplyInline Actions

Good catch!

wlei: Good catch!
if (!FrameVec.empty()) {
FunctionSamples &FunctionProfile =
getLeafProfileAndAddTotalSamples(FrameVec, Count);
updateBodySamplesforFunctionProfile(FunctionProfile, FrameVec.back(),
Count);
}
// Move to next IP within the range.
IP.advance();
}
}
}
wenleiUnsubmitted
Not Done
ReplyInline Actions

move this assertion to the top?

wenlei: move this assertion to the top?
wleiAuthorUnsubmitted
Done
ReplyInline Actions

fixed.

wlei: fixed.
void ProfileGenerator::populateBoundarySamplesForAllFunctions(
const BranchSample &BranchCounters) {
for (auto Entry : BranchCounters) {
uint64_t SourceOffset = Entry.first.first;
uint64_t TargetOffset = Entry.first.second;
uint64_t Count = Entry.second;
assert(Count != 0 && "Unexpected zero weight branch");
// Get the callee name by branch target if it's a call branch.
StringRef CalleeName = FunctionSamples::getCanonicalFnName(
Binary->getFuncFromStartOffset(TargetOffset));
if (CalleeName.size() == 0)
continue;
// Record called target sample and its count.
const SampleContextFrameVector &FrameVec =
Binary->getFrameLocationStack(SourceOffset);
if (!FrameVec.empty()) {
FunctionSamples &FunctionProfile =
getLeafProfileAndAddTotalSamples(FrameVec, Count);
FunctionProfile.addCalledTargetSamples(
FrameVec.back().Callsite.LineOffset,
FrameVec.back().Callsite.Discriminator, CalleeName, Count);
}
// Add head samples for callee.
FunctionSamples &CalleeProfile = getTopLevelFunctionProfile(CalleeName);
CalleeProfile.addHeadSamples(Count);
}
}
FunctionSamples &CSProfileGenerator::getFunctionProfileForContext( FunctionSamples &CSProfileGenerator::getFunctionProfileForContext(
const SampleContextFrameVector &Context, bool WasLeafInlined) { const SampleContextFrameVector &Context, bool WasLeafInlined) {
auto I = ProfileMap.find(SampleContext(Context)); auto I = ProfileMap.find(SampleContext(Context));
if (I == ProfileMap.end()) { if (I == ProfileMap.end()) {
// Save the new context for future references. // Save the new context for future references.
SampleContextFrames NewContext = *Contexts.insert(Context).first; SampleContextFrames NewContext = *Contexts.insert(Context).first;
SampleContext FContext(NewContext, RawContext); SampleContext FContext(NewContext, RawContext);
auto Ret = ProfileMap.emplace(FContext, FunctionSamples()); auto Ret = ProfileMap.emplace(FContext, FunctionSamples());
if (WasLeafInlined) if (WasLeafInlined)
FContext.setAttribute(ContextWasInlined); FContext.setAttribute(ContextWasInlined);
FunctionSamples &FProfile = Ret.first->second; FunctionSamples &FProfile = Ret.first->second;
FProfile.setContext(FContext); FProfile.setContext(FContext);
return Ret.first->second; return Ret.first->second;
} }
return I->second; return I->second;
} }
void CSProfileGenerator::generateProfile() { void CSProfileGenerator::generateProfile() {
FunctionSamples::ProfileIsCS = true; FunctionSamples::ProfileIsCS = true;
if (Binary->usePseudoProbes()) {
// Enable pseudo probe functionalities in SampleProf
FunctionSamples::ProfileIsProbeBased = true;
generateProbeBasedProfile();
} else {
generateLineNumBasedProfile();
}
postProcessProfiles();
}
void CSProfileGenerator::generateLineNumBasedProfile() {
for (const auto &CI : SampleCounters) { for (const auto &CI : SampleCounters) {
const StringBasedCtxKey *CtxKey = const StringBasedCtxKey *CtxKey =
dyn_cast<StringBasedCtxKey>(CI.first.getPtr()); dyn_cast<StringBasedCtxKey>(CI.first.getPtr());
// Get or create function profile for the range // Get or create function profile for the range
FunctionSamples &FunctionProfile = FunctionSamples &FunctionProfile =
getFunctionProfileForContext(CtxKey->Context, CtxKey->WasLeafInlined); getFunctionProfileForContext(CtxKey->Context, CtxKey->WasLeafInlined);
// Fill in function body samples // Fill in function body samples
populateFunctionBodySamples(FunctionProfile, CI.second.RangeCounter); populateBodySamplesForFunction(FunctionProfile, CI.second.RangeCounter);
// Fill in boundary sample counts as well as call site samples for calls // Fill in boundary sample counts as well as call site samples for calls
populateFunctionBoundarySamples(CtxKey->Context, FunctionProfile, populateBoundarySamplesForFunction(CtxKey->Context, FunctionProfile,
CI.second.BranchCounter); CI.second.BranchCounter);
} }
// Fill in call site value sample for inlined calls and also use context to // Fill in call site value sample for inlined calls and also use context to
// infer missing samples. Since we don't have call count for inlined // infer missing samples. Since we don't have call count for inlined
// functions, we estimate it from inlinee's profile using the entry of the // functions, we estimate it from inlinee's profile using the entry of the
// body sample. // body sample.
populateInferredFunctionSamples(); populateInferredFunctionSamples();
postProcessProfiles();
}
void CSProfileGenerator::updateBodySamplesforFunctionProfile(
FunctionSamples &FunctionProfile, const SampleContextFrame &LeafLoc,
uint64_t Count) {
// Filter out invalid negative(int type) lineOffset
if (LeafLoc.Callsite.LineOffset & 0x80000000)
return;
// Use the maximum count of samples with same line location
ErrorOr<uint64_t> R = FunctionProfile.findSamplesAt(
LeafLoc.Callsite.LineOffset, LeafLoc.Callsite.Discriminator);
uint64_t PreviousCount = R ? R.get() : 0;
if (PreviousCount < Count) {
FunctionProfile.addBodySamples(LeafLoc.Callsite.LineOffset,
LeafLoc.Callsite.Discriminator,
Count - PreviousCount);
}
} }
void CSProfileGenerator::populateFunctionBodySamples( void CSProfileGenerator::populateBodySamplesForFunction(
FunctionSamples &FunctionProfile, const RangeSample &RangeCounter) { FunctionSamples &FunctionProfile, const RangeSample &RangeCounter) {
// Compute disjoint ranges first, so we can use MAX // Compute disjoint ranges first, so we can use MAX
// for calculating count for each location. // for calculating count for each location.
RangeSample Ranges; RangeSample Ranges;
findDisjointRanges(Ranges, RangeCounter); findDisjointRanges(Ranges, RangeCounter);
for (auto Range : Ranges) { for (auto Range : Ranges) {
uint64_t RangeBegin = Binary->offsetToVirtualAddr(Range.first.first); uint64_t RangeBegin = Binary->offsetToVirtualAddr(Range.first.first);
uint64_t RangeEnd = Binary->offsetToVirtualAddr(Range.first.second); uint64_t RangeEnd = Binary->offsetToVirtualAddr(Range.first.second);
uint64_t Count = Range.second; uint64_t Count = Range.second;
// Disjoint ranges have introduce zero-filled gap that // Disjoint ranges have introduce zero-filled gap that
// doesn't belong to current context, filter them out. // doesn't belong to current context, filter them out.
if (Count == 0) if (Count == 0)
continue; continue;
InstructionPointer IP(Binary, RangeBegin, true); InstructionPointer IP(Binary, RangeBegin, true);
// Disjoint ranges may have range in the middle of two instr, // Disjoint ranges may have range in the middle of two instr,
hoyUnsubmitted
Not Done
ReplyInline Actions

may still be good to keep the comment.

hoy: may still be good to keep the comment.
// e.g. If Instr1 at Addr1, and Instr2 at Addr2, disjoint range // e.g. If Instr1 at Addr1, and Instr2 at Addr2, disjoint range
// can be Addr1+1 to Addr2-1. We should ignore such range. // can be Addr1+1 to Addr2-1. We should ignore such range.
if (IP.Address > RangeEnd)
continue;
while (IP.Address <= RangeEnd) { while (IP.Address <= RangeEnd) {
uint64_t Offset = Binary->virtualAddrToOffset(IP.Address); uint64_t Offset = Binary->virtualAddrToOffset(IP.Address);
auto LeafLoc = Binary->getInlineLeafFrameLoc(Offset); auto LeafLoc = Binary->getInlineLeafFrameLoc(Offset);
if (LeafLoc.hasValue()) { if (LeafLoc.hasValue()) {
// Recording body sample for this specific context // Recording body sample for this specific context
updateBodySamplesforFunctionProfile(FunctionProfile, *LeafLoc, Count); updateBodySamplesforFunctionProfile(FunctionProfile, *LeafLoc, Count);
} }
// Accumulate total sample count even it's a line with invalid debug info // Accumulate total sample count even it's a line with invalid debug info
FunctionProfile.addTotalSamples(Count); FunctionProfile.addTotalSamples(Count);
// Move to next IP within the range // Move to next IP within the range
IP.advance(); IP.advance();
} }
} }
} }
void CSProfileGenerator::populateFunctionBoundarySamples( void CSProfileGenerator::populateBoundarySamplesForFunction(
SampleContextFrames ContextId, FunctionSamples &FunctionProfile, SampleContextFrames ContextId, FunctionSamples &FunctionProfile,
const BranchSample &BranchCounters) { const BranchSample &BranchCounters) {
for (auto Entry : BranchCounters) { for (auto Entry : BranchCounters) {
uint64_t SourceOffset = Entry.first.first; uint64_t SourceOffset = Entry.first.first;
uint64_t TargetOffset = Entry.first.second; uint64_t TargetOffset = Entry.first.second;
uint64_t Count = Entry.second; uint64_t Count = Entry.second;
assert(Count != 0 && "Unexpected zero weight branch");
// Get the callee name by branch target if it's a call branch // Get the callee name by branch target if it's a call branch
StringRef CalleeName = FunctionSamples::getCanonicalFnName( StringRef CalleeName = FunctionSamples::getCanonicalFnName(
Binary->getFuncFromStartOffset(TargetOffset)); Binary->getFuncFromStartOffset(TargetOffset));
if (CalleeName.size() == 0) if (CalleeName.size() == 0)
continue; continue;
// Record called target sample and its count // Record called target sample and its count
auto LeafLoc = Binary->getInlineLeafFrameLoc(SourceOffset); auto LeafLoc = Binary->getInlineLeafFrameLoc(SourceOffset);
if (!LeafLoc.hasValue()) if (!LeafLoc.hasValue())
continue; continue;
FunctionProfile.addCalledTargetSamples(LeafLoc->Callsite.LineOffset, FunctionProfile.addCalledTargetSamples(LeafLoc->Callsite.LineOffset,
LeafLoc->Callsite.Discriminator, LeafLoc->Callsite.Discriminator,
CalleeName, Count); CalleeName, Count);
// Record head sample for called target(callee) // Record head sample for called target(callee)
SampleContextFrameVector CalleeCtx(ContextId.begin(), ContextId.end()); SampleContextFrameVector CalleeCtx(ContextId.begin(), ContextId.end());
assert(CalleeCtx.back().CallerName == LeafLoc->CallerName && assert(CalleeCtx.back().CallerName == LeafLoc->CallerName &&
"Leaf function name doesn't match"); "Leaf function name doesn't match");
CalleeCtx.back() = *LeafLoc; CalleeCtx.back() = *LeafLoc;
CalleeCtx.emplace_back(CalleeName, LineLocation(0, 0)); CalleeCtx.emplace_back(CalleeName, LineLocation(0, 0));
FunctionSamples &CalleeProfile = getFunctionProfileForContext(CalleeCtx); FunctionSamples &CalleeProfile = getFunctionProfileForContext(CalleeCtx);
assert(Count != 0 && "Unexpected zero weight branch");
CalleeProfile.addHeadSamples(Count); CalleeProfile.addHeadSamples(Count);
} }
} }
static SampleContextFrame static SampleContextFrame
getCallerContext(SampleContextFrames CalleeContext, getCallerContext(SampleContextFrames CalleeContext,
SampleContextFrameVector &CallerContext) { SampleContextFrameVector &CallerContext) {
assert(CalleeContext.size() > 1 && "Unexpected empty context"); assert(CalleeContext.size() > 1 && "Unexpected empty context");
▲ Show 20 Lines • Show All 76 Lines • ▼ Show 20 Linesvoid CSProfileGenerator::computeSummaryAndThreshold() {
SampleProfileSummaryBuilder Builder(ProfileSummaryBuilder::DefaultCutoffs); SampleProfileSummaryBuilder Builder(ProfileSummaryBuilder::DefaultCutoffs);
auto Summary = Builder.computeSummaryForProfiles(ProfileMap); auto Summary = Builder.computeSummaryForProfiles(ProfileMap);
HotCountThreshold = ProfileSummaryBuilder::getHotCountThreshold( HotCountThreshold = ProfileSummaryBuilder::getHotCountThreshold(
(Summary->getDetailedSummary())); (Summary->getDetailedSummary()));
ColdCountThreshold = ProfileSummaryBuilder::getColdCountThreshold( ColdCountThreshold = ProfileSummaryBuilder::getColdCountThreshold(
(Summary->getDetailedSummary())); (Summary->getDetailedSummary()));
} }
void CSProfileGenerator::write(std::unique_ptr<SampleProfileWriter> Writer,
SampleProfileMap &ProfileMap) {
if (std::error_code EC = Writer->write(ProfileMap))
exitWithError(std::move(EC));
}
// Helper function to extract context prefix string stack // Helper function to extract context prefix string stack
// Extract context stack for reusing, leaf context stack will // Extract context stack for reusing, leaf context stack will
// be added compressed while looking up function profile // be added compressed while looking up function profile
static void extractPrefixContextStack( static void extractPrefixContextStack(
SampleContextFrameVector &ContextStack, SampleContextFrameVector &ContextStack,
const SmallVectorImpl<const MCDecodedPseudoProbe *> &Probes, const SmallVectorImpl<const MCDecodedPseudoProbe *> &Probes,
ProfiledBinary *Binary) { ProfiledBinary *Binary) {
for (const auto *P : Probes) { for (const auto *P : Probes) {
Binary->getInlineContextForProbe(P, ContextStack, true); Binary->getInlineContextForProbe(P, ContextStack, true);
} }
} }
void PseudoProbeCSProfileGenerator::generateProfile() { void CSProfileGenerator::generateProbeBasedProfile() {
// Enable pseudo probe functionalities in SampleProf
FunctionSamples::ProfileIsProbeBased = true;
FunctionSamples::ProfileIsCS = true;
for (const auto &CI : SampleCounters) { for (const auto &CI : SampleCounters) {
const ProbeBasedCtxKey *CtxKey = const ProbeBasedCtxKey *CtxKey =
dyn_cast<ProbeBasedCtxKey>(CI.first.getPtr()); dyn_cast<ProbeBasedCtxKey>(CI.first.getPtr());
SampleContextFrameVector ContextStack; SampleContextFrameVector ContextStack;
extractPrefixContextStack(ContextStack, CtxKey->Probes, Binary); extractPrefixContextStack(ContextStack, CtxKey->Probes, Binary);
// Fill in function body samples from probes, also infer caller's samples // Fill in function body samples from probes, also infer caller's samples
// from callee's probe // from callee's probe
populateBodySamplesWithProbes(CI.second.RangeCounter, ContextStack); populateBodySamplesWithProbes(CI.second.RangeCounter, ContextStack);
// Fill in boundary samples for a call probe // Fill in boundary samples for a call probe
populateBoundarySamplesWithProbes(CI.second.BranchCounter, ContextStack); populateBoundarySamplesWithProbes(CI.second.BranchCounter, ContextStack);
} }
postProcessProfiles();
} }
void PseudoProbeCSProfileGenerator::extractProbesFromRange( void CSProfileGenerator::extractProbesFromRange(const RangeSample &RangeCounter,
const RangeSample &RangeCounter, ProbeCounterMap &ProbeCounter) { ProbeCounterMap &ProbeCounter) {
RangeSample Ranges; RangeSample Ranges;
findDisjointRanges(Ranges, RangeCounter); findDisjointRanges(Ranges, RangeCounter);
for (const auto &Range : Ranges) { for (const auto &Range : Ranges) {
uint64_t RangeBegin = Binary->offsetToVirtualAddr(Range.first.first); uint64_t RangeBegin = Binary->offsetToVirtualAddr(Range.first.first);
uint64_t RangeEnd = Binary->offsetToVirtualAddr(Range.first.second); uint64_t RangeEnd = Binary->offsetToVirtualAddr(Range.first.second);
uint64_t Count = Range.second; uint64_t Count = Range.second;
// Disjoint ranges have introduce zero-filled gap that // Disjoint ranges have introduce zero-filled gap that
// doesn't belong to current context, filter them out. // doesn't belong to current context, filter them out.
Show All 20 Lines while (IP.Address <= RangeEnd) {
} }
} }
IP.advance(); IP.advance();
} }
} }
} }
void PseudoProbeCSProfileGenerator::populateBodySamplesWithProbes( void CSProfileGenerator::populateBodySamplesWithProbes(
const RangeSample &RangeCounter, SampleContextFrames ContextStack) { const RangeSample &RangeCounter, SampleContextFrames ContextStack) {
ProbeCounterMap ProbeCounter; ProbeCounterMap ProbeCounter;
// Extract the top frame probes by looking up each address among the range in // Extract the top frame probes by looking up each address among the range in
// the Address2ProbeMap // the Address2ProbeMap
extractProbesFromRange(RangeCounter, ProbeCounter); extractProbesFromRange(RangeCounter, ProbeCounter);
std::unordered_map<MCDecodedPseudoProbeInlineTree *, std::unordered_map<MCDecodedPseudoProbeInlineTree *,
std::unordered_set<FunctionSamples *>> std::unordered_set<FunctionSamples *>>
FrameSamples; FrameSamples;
▲ Show 20 Lines • Show All 42 Lines • ▼ Show 20 Lines for (auto &I : FrameSamples) {
for (auto *FunctionProfile : I.second) { for (auto *FunctionProfile : I.second) {
for (auto *Probe : I.first->getProbes()) { for (auto *Probe : I.first->getProbes()) {
FunctionProfile->addBodySamplesForProbe(Probe->getIndex(), 0); FunctionProfile->addBodySamplesForProbe(Probe->getIndex(), 0);
} }
} }
} }
} }
void PseudoProbeCSProfileGenerator::populateBoundarySamplesWithProbes( void CSProfileGenerator::populateBoundarySamplesWithProbes(
const BranchSample &BranchCounter, SampleContextFrames ContextStack) { const BranchSample &BranchCounter, SampleContextFrames ContextStack) {
for (auto BI : BranchCounter) { for (auto BI : BranchCounter) {
uint64_t SourceOffset = BI.first.first; uint64_t SourceOffset = BI.first.first;
uint64_t TargetOffset = BI.first.second; uint64_t TargetOffset = BI.first.second;
uint64_t Count = BI.second; uint64_t Count = BI.second;
uint64_t SourceAddress = Binary->offsetToVirtualAddr(SourceOffset); uint64_t SourceAddress = Binary->offsetToVirtualAddr(SourceOffset);
const MCDecodedPseudoProbe *CallProbe = const MCDecodedPseudoProbe *CallProbe =
Binary->getCallProbeForAddr(SourceAddress); Binary->getCallProbeForAddr(SourceAddress);
if (CallProbe == nullptr) if (CallProbe == nullptr)
continue; continue;
FunctionSamples &FunctionProfile = FunctionSamples &FunctionProfile =
getFunctionProfileForLeafProbe(ContextStack, CallProbe); getFunctionProfileForLeafProbe(ContextStack, CallProbe);
FunctionProfile.addBodySamples(CallProbe->getIndex(), 0, Count); FunctionProfile.addBodySamples(CallProbe->getIndex(), 0, Count);
FunctionProfile.addTotalSamples(Count); FunctionProfile.addTotalSamples(Count);
StringRef CalleeName = FunctionSamples::getCanonicalFnName( StringRef CalleeName = FunctionSamples::getCanonicalFnName(
Binary->getFuncFromStartOffset(TargetOffset)); Binary->getFuncFromStartOffset(TargetOffset));
if (CalleeName.size() == 0) if (CalleeName.size() == 0)
continue; continue;
FunctionProfile.addCalledTargetSamples(CallProbe->getIndex(), 0, CalleeName, FunctionProfile.addCalledTargetSamples(CallProbe->getIndex(), 0, CalleeName,
Count); Count);
} }
} }
FunctionSamples &PseudoProbeCSProfileGenerator::getFunctionProfileForLeafProbe( FunctionSamples &CSProfileGenerator::getFunctionProfileForLeafProbe(
SampleContextFrames ContextStack, const MCDecodedPseudoProbe *LeafProbe) { SampleContextFrames ContextStack, const MCDecodedPseudoProbe *LeafProbe) {
// Explicitly copy the context for appending the leaf context // Explicitly copy the context for appending the leaf context
SampleContextFrameVector NewContextStack(ContextStack.begin(), SampleContextFrameVector NewContextStack(ContextStack.begin(),
ContextStack.end()); ContextStack.end());
Binary->getInlineContextForProbe(LeafProbe, NewContextStack, true); Binary->getInlineContextForProbe(LeafProbe, NewContextStack, true);
// For leaf inlined context with the top frame, we should strip off the top // For leaf inlined context with the top frame, we should strip off the top
// frame's probe id, like: // frame's probe id, like:
Show All 19 Lines

llvm/tools/llvm-profgen/ProfiledBinary.h

Show First 20 Lines • Show All 226 Lines • ▼ Show 20 Linesclass ProfiledBinary {
/// Decode the interesting parts of the binary and build internal data /// Decode the interesting parts of the binary and build internal data
/// structures. On high level, the parts of interest are: /// structures. On high level, the parts of interest are:
/// 1. Text sections, including the main code section and the PLT /// 1. Text sections, including the main code section and the PLT
/// entries that will be used to handle cross-module call transitions. /// entries that will be used to handle cross-module call transitions.
/// 2. The .debug_line section, used by Dwarf-based profile generation. /// 2. The .debug_line section, used by Dwarf-based profile generation.
/// 3. Pseudo probe related sections, used by probe-based profile /// 3. Pseudo probe related sections, used by probe-based profile
/// generation. /// generation.
void load(); void load();
const SampleContextFrameVector &getFrameLocationStack(uint64_t Offset) const {
auto I = Offset2LocStackMap.find(Offset);
assert(I != Offset2LocStackMap.end() &&
"Can't find location for offset in the binary");
return I->second;
}
public: public:
ProfiledBinary(const StringRef Path) ProfiledBinary(const StringRef Path)
: Path(Path), ProEpilogTracker(this), : Path(Path), ProEpilogTracker(this),
TrackFuncContextSize(EnableCSPreInliner && TrackFuncContextSize(EnableCSPreInliner &&
UseContextCostForPreInliner) { UseContextCostForPreInliner) {
setupSymbolizer(); setupSymbolizer();
load(); load();
▲ Show 20 Lines • Show All 56 Lines • ▼ Show 20 Lines uint64_t getCallAddrFromFrameAddr(uint64_t FrameAddr) const {
auto I = getIndexForAddr(FrameAddr); auto I = getIndexForAddr(FrameAddr);
FrameAddr = I ? getAddressforIndex(I - 1) : 0; FrameAddr = I ? getAddressforIndex(I - 1) : 0;
if (FrameAddr && addressIsCall(FrameAddr)) if (FrameAddr && addressIsCall(FrameAddr))
return FrameAddr; return FrameAddr;
return 0; return 0;
} }
StringRef getFuncFromStartOffset(uint64_t Offset) { StringRef getFuncFromStartOffset(uint64_t Offset) {
return FuncStartAddrMap[Offset]; auto I = FuncStartAddrMap.find(Offset);
if (I == FuncStartAddrMap.end())
return StringRef();
return I->second;
} }
uint32_t getFuncSizeForContext(SampleContext &Context) { uint32_t getFuncSizeForContext(SampleContext &Context) {
return FuncSizeTracker.getFuncSizeForContext(Context); return FuncSizeTracker.getFuncSizeForContext(Context);
} }
const SampleContextFrameVector &getFrameLocationStack(uint64_t Offset) const {
auto I = Offset2LocStackMap.find(Offset);
assert(I != Offset2LocStackMap.end() &&
"Can't find location for offset in the binary");
return I->second;
}
Optional<SampleContextFrame> getInlineLeafFrameLoc(uint64_t Offset) { Optional<SampleContextFrame> getInlineLeafFrameLoc(uint64_t Offset) {
const auto &Stack = getFrameLocationStack(Offset); const auto &Stack = getFrameLocationStack(Offset);
if (Stack.empty()) if (Stack.empty())
return {}; return {};
return Stack.back(); return Stack.back();
} }
// Compare two addresses' inline context // Compare two addresses' inline context
▲ Show 20 Lines • Show All 49 Lines • Show Last 20 Lines

llvm/tools/llvm-profgen/llvm-profgen.cpp

Show First 20 Lines • Show All 87 Lines • ▼ Show 20 Linesint main(int argc, const char *argv[]) {
// Parse perf events and samples // Parse perf events and samples
std::unique_ptr<PerfReaderBase> Reader = std::unique_ptr<PerfReaderBase> Reader =
PerfReaderBase::create(Binary.get(), PerfTraceFilenames); PerfReaderBase::create(Binary.get(), PerfTraceFilenames);
Reader->parsePerfTraces(PerfTraceFilenames); Reader->parsePerfTraces(PerfTraceFilenames);
if (SkipSymbolization) if (SkipSymbolization)
return EXIT_SUCCESS; return EXIT_SUCCESS;
// TBD std::unique_ptr<ProfileGeneratorBase> Generator =
if (Reader->getPerfScriptType() == PERF_LBR) { ProfileGeneratorBase::create(Binary.get(), Reader->getSampleCounters(),
WithColor::warning() << "Currently LBR only perf script is not supported!"; Reader->getPerfScriptType());
return EXIT_SUCCESS;
}
std::unique_ptr<ProfileGenerator> Generator = ProfileGenerator::create(
Binary.get(), Reader->getSampleCounters(), Reader->getPerfScriptType());
Generator->generateProfile(); Generator->generateProfile();
Generator->write(); Generator->write();
return EXIT_SUCCESS; return EXIT_SUCCESS;
} }