[PGO] context sensitive PGO

[xur]

Summary

Current PGO profile counts are not context sensitive. The branch probabilities for the inlined functions are kept the same for all call-sites, and they might be very different from the actual branch probabilities. These suboptimal profile can greatly affect some downstream optimizations, in particular for machine basic block placement optimization.

In this patch, we propose to have a post-inline PGO instrumentation/use pass, which we called Context Sensitive PGO (CSPGO). For the users who want the best possible performance, they can perform a second round of PGO instrument/use on the top of the regular PGO. They will have two sets of profile counts. The first pass profile will be manly for inline, indirect-call promotion, and CGSCC simplification pass optimizations. The second pass profile is for post-inline optimizations and code-gen optimizations.

The typical usage models are:
// Regular PGO instrumentation and generate pass1 profile.

> clang -O2 -fprofile-generate source.c -o gen
> ./gen
> llvm-profdata merge default.*profraw -o pass1.profdata

// CSPGO instrumentation.

> clang -O2 -fprofile-use=pass1.profdata -fcs-profile-generate -o gen2
> ./gen2

// Merge two sets of profiles

> llvm-profdata merge default.*profraw pass1.profdata -o profile.profdata

// Use the combined profile. Pass manager will invoke two PGO use passes.

> clang -O2 -fprofile-use=profile.profdata -o use

The major changes are the following:

• bump index profile version to indicate if there are context-sensitive counts.
• change the FuncHash and reserve bits for context-sensitive counts.
• change the ProfileSummary interface as we might have two sets of profile summaries.
• change pass-manager (legacy and new)
• change llvm-profdata to handle two kind of profiles.
• change instrprofiling lowing for more aggressive register promotion for CSPGO.

Note that other than the FuncHash change, there is no effect on existing PGO pass.

This patches is functional tested using SPEC2006, with the various combinations of the following:

• legacy-pass manager
• new pass manager
• thin-lto (through gold plugin)
• thin-lto (though driver command line)
• lto (through plugin)
• non-lto

We performance-tested this patch with a few large Google benchmarks and saw good performance improvements.

I did not include any tests in this patch. I will add tests once the interfaces are decided in the code review.

[davidxl]

include/llvm/ProfileData/InstrProf.h

777 ↗

(On Diff #172824)

nit: FlaginHash --> FlagInHash

780 ↗

(On Diff #172824)

FlaginHash -> FlagInHash.

902 ↗

(On Diff #172824)

There is probably no need to bump the version number if there is a new variant bit set in the profile data. The format of existing variant do not change.

include/llvm/ProfileData/InstrProfReader.h

148 ↗

(On Diff #172824)

Why? It would be nice to add support in text reader as well.

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

618 ↗

(On Diff #172824)

This won't work well -- the non-CS FunctionHash may collide with CS hash. How do you differentiate?

Also doing this for nonCS profiling also breaks the backward compatibility -- old profile with high bits set in hash won't be found anymore. It may also create more hash conflicts.

[xur]

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

618 ↗

(On Diff #172824)

That is the reason I bump the version number. I use the version number to tell if this is old Hash format or new Hash format.

[davidxl]

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

618 ↗

(On Diff #172824)

Say there exists an old version profile with the 60-63 bits set for some functions. When this profile is used, the new compiler won't be able to find those functions.

Also where is the code that checks the version number?

[davidxl]

Test cases are missing in the patch.

[vsk]

Hi Rong, at a high-level, I like what this patch is doing. I'll try to leave in-depth comments by next week -- please ping the review otherwise.

These suboptimal profile can greatly affect some downstream optimizations, in particular for machine basic block placement optimization.

If it's possible to get these numbers, it'd be interesting to know how the improvements from this patch compare to link-time or post-link block ordering tools (Bolt).

[xur]

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

618 ↗

(On Diff #172824)

or this we will use one VARIANT bit in header to distinguish. AnnotateAllFunction will check if the VARIANT flag is set when reading CS profiles.
For old index profile, this won't be a problem as this bit will not be set.

This could be a problem for old raw profile. As of now, I did not change instrumentation run-time. So I could only use FuncHash to tell if this regular count or CS count. Setting of VARIANT bit in index profile relies on the new FuncHash scheme.

I can add change to profile runtime and set the VARIANT for raw profile. Then setting the VARIANT bit for index profile will not rely on check of FuncHash. This will fully backward compatible.

The new FuncHash scheme will be guarded by a check of VARIANT bit. Maybe we don't need bump the profile version.

Right now, It's only in IndexedInstrProfReader::readSummary(). I agree that we probably need to place more checks on this.

[xur]

In D54175#1291940, @vsk wrote:

Hi Rong, at a high-level, I like what this patch is doing. I'll try to leave in-depth comments by next week -- please ping the review otherwise.

Sounds good. Looking forward to your reviews.

These suboptimal profile can greatly affect some downstream optimizations, in particular for machine basic block placement optimization.

If it's possible to get these numbers, it'd be interesting to know how the improvements from this patch compare to link-time or post-link block ordering tools (Bolt).

We mainly use a key google benchmark for performance evaluation. Note that this benchmark has been highly tuned. A performance gain of 0.5% is considered significant.

Our experiments shows 1.5% to 2% from CSPGO. The improvement is mainly from machine basicblock placement. CSPGO also sets the accurate function entry count (for the cases that inline are cross module). That will make the hot-text section more dense. This can be seen from instruction heat map. But the performance improvement was small.

BOLT can have similar performance boosts from basic block reordering if it's applied to regular PGO binary (i.e. ~2%)
BOLT can improve the CSPGO binary by 0.5% and that is from function splitting, which is not currently enabled in llvm.

[davidxl]

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

618 ↗

(On Diff #172824)

I think it is better to emit variant bit in raw profile data. Relying on func hashing content sounds fragile.

The masking and adding bit in function hash is only needed for CS instrumentation and annotation. With this, the old profile (non CS) will still be valid. There is a slight downside that nonCS and CS profile can collide -- but this is very unlikely given all the inlining/cfg transformations that had happened.

[vsk]

include/llvm/Analysis/ProfileSummaryInfo.h

48 ↗

(On Diff #172824)

It doesn't seem necessary to add a flag to computeSummary or to add recomputeSummary to PSI's API.

Why not call ProfileSummaryInfo::invalidate when attaching CSPGO data to a Module? invalidate() could be modified to recompute the summary. That keeps the APIs a bit simpler.

include/llvm/IR/Module.h

859 ↗

(On Diff #172824)

I think {set, get}ProfileSummary would be safer & easier to understand if clients had to pass in a non-optional enum specifying which profile data kind they need (maybe use ProfileSummary::Kind for this?). That should make it impossible to mix-and-match CS + the wrong format.

include/llvm/IR/ProfileSummary.h

50 ↗

(On Diff #172824)

Does this really need to be in the header? Could we just take the opportunity to delete this declaration, and make the definition static?

include/llvm/Passes/PassBuilder.h

47 ↗

(On Diff #172824)

I think this could use a little more explanation (why would CSProfileGen not need ProfileUse?).

603 ↗

(On Diff #172824)

I think it'd be best to consolidate flags here, and replace {bool RunProfileGen, bool IsCS} with {enum InstrumentionKind} (or similar, there must be an enum for this already).

include/llvm/ProfileData/InstrProfReader.h

473 ↗

(On Diff #172824)

I'm not sure that the right behavior here should be to default to the non-context sensitive data. Why not either make the bool argument non-optional, or, take the max of the two?

500 ↗

(On Diff #172824)

I don't think the bool argument here should have a default value. Clients should explicitly pick which kind of summary they need.

include/llvm/ProfileData/InstrProfWriter.h

87 ↗

(On Diff #172824)

Using xor in this context is a bit unconventional. Could you rewrite this using ands/ors to make it easier to understand?

lib/ProfileData/InstrProfReader.cpp

768 ↗

(On Diff #172824)

No need to assert on the version, it's checked before readSummary is called.

768 ↗

(On Diff #172824)

Please write out the type of Summary here for clarity.

[xur]

Thanks vsk and david for the reviews. I will update the patch to integrate the comments.

include/llvm/Analysis/ProfileSummaryInfo.h

48 ↗

(On Diff #172824)

I agree that change invlidate() would be more clean (and better). My concern was that would change current behavior.
invalidate() be called by passmanager. Currently it does nothing, If I change that, profilesummary could be recomputed many time as most passes will invalidate all the analysis.

include/llvm/IR/Module.h

859 ↗

(On Diff #172824)

I will change to use an enum parameter (and no default).

include/llvm/IR/ProfileSummary.h

50 ↗

(On Diff #172824)

I think this is a good idea. the strings are only used in ProfileSummary.cpp. I will change.

include/llvm/Passes/PassBuilder.h

47 ↗

(On Diff #172824)

Sorry. This will be deleted.
I used this to work around an issue with out internal tool.

Thanks for catching this.

603 ↗

(On Diff #172824)

yes. I agree.

include/llvm/ProfileData/InstrProf.h

902 ↗

(On Diff #172824)

OK. no version change.

include/llvm/ProfileData/InstrProfReader.h

148 ↗

(On Diff #172824)

Text format is supported (mostly). I say mostly because there is not text string for CS format. But the flat is encoded in header (and in checksum).

473 ↗

(On Diff #172824)

I can change the take bool argument.

Taking max is not a good idea as the client need to know what kind of summary it uses. Mixing these two will cause wrong hot/cold attributes.

include/llvm/ProfileData/InstrProfWriter.h

87 ↗

(On Diff #172824)

OK. I will change to use and/or

lib/ProfileData/InstrProfReader.cpp

768 ↗

(On Diff #172824)

Do you mean comments?

[vsk]

include/llvm/Analysis/ProfileSummaryInfo.h

48 ↗

(On Diff #172824)

Would it be feasible to update any passes which inadvertently invalidate PSI to preserve it instead?

include/llvm/ProfileData/InstrProfReader.h

473 ↗

(On Diff #172824)

Sounds good.

lib/ProfileData/InstrProfReader.cpp

768 ↗

(On Diff #172824)

I meant 'ProfileSummary &Summary', instead of 'auto &Summary'. I think the coding guidelines suggest writing out the type explicitly when it may not be obvious from the immediate context. This is subjective though, I don't have a strong opinion about this.

[xur]

include/llvm/Analysis/ProfileSummaryInfo.h

48 ↗

(On Diff #172824)

I found that we don't need to invalidate summary.
Normal PGO-use pass is before per-function optimization passes
and CS pass of PGO-use is in per-module optimization passes.
They do not share the analyse pass result -- All the results in
per-function optimization pass are discarded and recomputed
again in per-module optimization pass.

I will revert this change.

include/llvm/Passes/PassBuilder.h

603 ↗

(On Diff #172824)

hmm. IsCS is also used by ProfileUse. Even I combine like this, I still need two flags.

include/llvm/ProfileData/InstrProfReader.h

148 ↗

(On Diff #172824)

Now. "csir" is supported keyword in proftext format.

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

618 ↗

(On Diff #172824)

Now the raw profile also emit variant.

But I still relying on funchash to tell if the profile for this function is CS or not.

I agree the chance of collision is low. I think what matter more is the summary computation. Functions with lots of select instruction will counts as CS profile and used in CS profile summary. This definitively happens but I'm not sure if matters for performance.

I lean to change the hash unconditionally. For old profiles, we will report a hash-mismatch and the users need to recollect the profile. I think this could happen for other compiler change (like changes before pgo instrumentation).

[xur]

This new version integrated the review comments from David and Vsk. Noticeable changes are:
(1) Combined the following 3 field in PGOOptions struct

bool RunProfileGen;
bool RunCSProfileGen;
bool RunCSProfileUse;

into an enum. There three are mutually exclusive.

(2) Not use default arguments for a set of functions.

(3) Don't bump up index profile version

(4) CS variant flag is set in raw profile and use that to tell if the profile has CS counters.

(5) Better handle in profile text format.

(7) Revert ProfileSummary recomputation as it is not needed.

(8) Fix a bug in the change to instrprofiling promotion.

I will also update
https://reviews.llvm.org/D54177
and
https://reviews.llvm.org/D54176

[xur]

Fix an issue in LegacyPassManager where CSPGO is not separated from -fprofile-use cleanly.

[tejohnson]

lib/LTO/LTOBackend.cpp

152 ↗

(On Diff #176208)

Presumably need a corresponding change in runOldPMPasses?

You'll also want to have a way to test via llvm-lto2. E.g. llvm-lto2 has an option for passing a sample profile.

tools/gold/gold-plugin.cpp

898 ↗

(On Diff #176208)

Needs tests

Also, support should be added to lld too - is there a separate patch for that?

tools/llvm-profdata/llvm-profdata.cpp

231 ↗

(On Diff #176208)

leftover debug output

[xur]

lib/LTO/LTOBackend.cpp

152 ↗

(On Diff #176208)

You are right. I need to set the options for oldPM.

tools/gold/gold-plugin.cpp

898 ↗

(On Diff #176208)

I haven't done lld patch yet. There are too many moving parts at this stage.

tools/llvm-profdata/llvm-profdata.cpp

231 ↗

(On Diff #176208)

Removed.

[xur]

Integrated with Teresa's review comments.

I will add tests in a later patch.

[tejohnson]

In D54175#1317088, @xur wrote:

Integrated with Teresa's review comments.

I will add tests in a later patch.

I really think that tests for gold and for LTO should be added with this patch, since it is adding the functionality for those.

lib/LTO/LTOBackend.cpp

284 ↗

(On Diff #176433)

Is it deliberate that here in the old PM the RunCSIRInstr and CSIRProfile are handled regardless of whether SampleProfile is set, whereas in the new PM they are only handled when SampleProfile is not set? What should be the behavior if sample profile is specified along with CS profiling?

[xur]

As for the tests, I meant to add test cases to this patch later, not through a different patch.
I can add tests now.

lib/LTO/LTOBackend.cpp

284 ↗

(On Diff #176433)

It's not deliberate. I should have make the behavior consistent.
I think I would like to change new PM and make CSProfile independent of SampleProfile.
The reason is that -fprofile-use and -fprofile-sample-use are independent.
This usage does not make a lot of sense, and we should probably give some warnings. But that should be in command line handling, not here.

What do you think?

[tejohnson]

In D54175#1317243, @xur wrote:

As for the tests, I meant to add test cases to this patch later, not through a different patch.

Oh got it - that's totally fine. I misunderstood.

I can add tests now.

include/llvm/Passes/PassBuilder.h

44 ↗

(On Diff #176433)

Shouldn't this be (Action == NoAction || ...)? I.e. either we are doing no PGO action, or we better have one of the other fields set?

lib/LTO/LTOBackend.cpp

284 ↗

(On Diff #176433)

Yeah it would be good to make them consistent. Looking at the new PM though, I'm not sure how you can make them independent with the existing PGOOptions interface.

I noticed a similar inconsistency between new and old PM on the clang side patch, although it looks like it predates your changes (with the newPM the sample profile is only looked at if no IR profiling enabled, in the old PM it is looked at unconditionally).

[xur]

include/llvm/Passes/PassBuilder.h

44 ↗

(On Diff #176433)

There are 4 possible values for Action: NoAction, IRInstr, CSIRInstr, CSIRUse
If NoAction, we need to have SampleProfileFile nonempty or ProfileUseFile nonempty. They are for sample fdo and fdo-use.
If the value is of the other 3 values, we don't check SampleProfile and ProfileUseFile.

I think this is correct but I agree the check can be improved.

lib/LTO/LTOBackend.cpp

284 ↗

(On Diff #176433)

SampleProfileFile, ProfileGenFile and ProfileUseFile are of different fields in PGOOptions interface. I think technically they can be independently set /checked. But it's really confusing if SampleProifle and Profile{Gen|Use}File are both on.

[tejohnson]

include/llvm/Passes/PassBuilder.h

44 ↗

(On Diff #176433)

I see. I think it would be a lot clearer if the Action was extended then to cover all the possible pgo types, it's confusing to have NoAction be set when e.g. sample profile or IR profile use is enabled.

lib/LTO/LTOBackend.cpp

284 ↗

(On Diff #176433)

Agree. Probably add some checking into PGOOptions that only one is set (maybe in combination with the extension to Action that I suggested above)?

[xur]

Synced to newest change and fixed a few issues.

[xur]

Ping.

If the reviewers are OK with the high level structures and the interfaces, I will start to add test cases.
I have also done the lld patch. I will submit a separated patch for review.

[tejohnson]

include/llvm/Passes/PassBuilder.h

44 ↗

(On Diff #176433)

Please address this comment (it's really unintuitive to have NoAction mean sample profile or IR profile use).

lib/LTO/LTOBackend.cpp

284 ↗

(On Diff #176433)

ping on this suggestion.

lib/Passes/PassBuilder.cpp

768 ↗

(On Diff #177973)

Add comment about the LTOPreLink check.

lib/Transforms/IPO/PassManagerBuilder.cpp

566 ↗

(On Diff #177973)

Document constant parameter (here and elsewhere). E.g. /*IsCS=*/true.

[xur]

This new pass addresses the review comments from Teresa. Main changes are:
(1) used two enums in PGOOptions:

enum PGOAction { NoAction, IRInstr, IRUse, SampleUse };
enum CSPGOAction { NoCSAction, CSIRInstr, CSIRUse };

The first enum is for regular PGO. We will use one ProfileFile string to represent the profile (instrument path or profile path).
CSPGOAction can be set when PGOAction == NaAction, or PGOAction==IRUse.
for CSIRUse, the profile is the same as ProfileFile. For CSIRInstr, we need another string CSProfileGenFile to represent instrumentation file name.

NewPMDDriver already use a similar method for regular PGO.

(2) Changed the clang options handling to give error message when -fsample-profile-use is used with -fprofile-use or -fprofile-generate.

(3) added support in tools/opt/NewPMDriver.cpp. This will be used in tests.

[xur]

Fixed the issue with lld (where lld is not happy with the COMDAT variables created
after LTO/ThinLTO)

Add the support of clang bootstrap for CSPGO.

Update the patch.

Tested this patch with google internal benchmarks (mostly for distributed ThinLTO).

Tested this patch with clang bootstrap for CSPGO.
(1) plain CSPGO (new and legacy pass manager).
(2) Thinlto CSPGO with plugin-opt to lld (new-experimental and legacy pass manager).
(3) Fulllto CSPGO with plugin-opt to lld (new-experimental and legacy pass manager).
CSPGO improves 1%-4% (wall time of ninja clang) upon regular PGO in clang bootstrap.
Thinlto and Fulllto is ~4% (wall time of ninja clang) in the new-experimental pass manager.
Compiling of individual files can improve up to 20%.

I'm working to add tests to this patch

Clang patch:
https://reviews.llvm.org/D54176

Compiler-rt patch:
https://reviews.llvm.org/D54177

lld patch:
https://reviews.llvm.org/D56675

[tejohnson]

Bunch of mostly nit suggestions about names and commenting, and a few questions. I like the improved profile action expression.

CMakeLists.txt

575 ↗

(On Diff #181595)

nit: indentation seems off

cmake/modules/HandleLLVMOptions.cmake

816 ↗

(On Diff #181595)

s/Nedd/Need

Also, I don't understand how the comment about needing to pass it to the backend compiler after ld-plugin relates to the handling below.

826 ↗

(On Diff #181595)

This part seems unrelated - probably split into a different patch?

include/llvm/Passes/PassBuilder.h

35 ↗

(On Diff #181595)

document different values (in both enums)

45 ↗

(On Diff #181595)

This ordering doesn't seem to match the ordering of the checks in the assert below. In any case, rather than have the documentation just duplicate the code (it doesn't really add anything this way), it would be better to split up the following assert into multiple, with more of a plain english explanation of each one.

include/llvm/ProfileData/InstrProf.h

770 ↗

(On Diff #181595)

document

1016 ↗

(On Diff #181595)

this is just the addition of bit 60 being the CS flag, right? Maybe just say that?

include/llvm/ProfileData/InstrProfReader.h

438 ↗

(On Diff #181595)

Document new parameter here and elsewhere. I assume it is legal to have both a CS and a non-CS summary in this object? In that case, maybe the parameter name should be something like UseCS.

include/llvm/Transforms/Instrumentation/InstrProfiling.h

64 ↗

(On Diff #181595)

document variable

include/llvm/Transforms/Instrumentation/PGOInstrumentation.h

39 ↗

(On Diff #181595)

document const parameter

lib/Passes/PassBuilder.cpp

523 ↗

(On Diff #181595)

Why do we skip size optimizations with CS?

[tejohnson]

lib/Passes/PassBuilder.cpp

777 ↗

(On Diff #181595)

document const parameter (here and in below call)

1022 ↗

(On Diff #181595)

document const parameter

1145 ↗

(On Diff #181595)

document const parameter (here and below call)

lib/ProfileData/InstrProf.cpp

1045 ↗

(On Diff #181595)

nit - in createProfileNameVar the linkage initially set to weak any, then reset to external if supports comdat. Here you do the opposite - suggest making the logic the same in both places for consistency

lib/ProfileData/InstrProfReader.cpp

815 ↗

(On Diff #181595)

document const parameter here and in below call

lib/Transforms/IPO/PassManagerBuilder.cpp

537 ↗

(On Diff #181595)

s/proifle/profile/

538 ↗

(On Diff #181595)

Can you add a better explanation about linker issue - i.e. the linker wants to see all variables before the LTO link since it needs to resolve symbols/comdats.

575 ↗

(On Diff #181595)

document const parameter (here and below)

lib/Transforms/Instrumentation/InstrProfiling.cpp

276 ↗

(On Diff #181595)

document rationale for this equation

998 ↗

(On Diff #181595)

Comment needed

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

410 ↗

(On Diff #181595)

document

435 ↗

(On Diff #181595)

document

554 ↗

(On Diff #181595)

document

666 ↗

(On Diff #181595)

Probably needs a slight expansion on why that other info shouldn't affect function hash.

678 ↗

(On Diff #181595)

why reorder this?

1459 ↗

(On Diff #181595)

document const parameter. also add comment for why this is not needed when IsCS

1507 ↗

(On Diff #181595)

I don't see where the new parameter PSI is getting used.

1642 ↗

(On Diff #181595)

Here and below, needs comment about why PSI when IsCS. But see comment in annotateAllFunctions - I don't see it getting used there?

tools/llvm-profdata/llvm-profdata.cpp

716 ↗

(On Diff #181595)

How about replace these 4 lines with just:

if (FuncIsCS != ShowCS)

continue;

tools/opt/NewPMDriver.cpp

116 ↗

(On Diff #181595)

Can you do something similar to D56749 (moving and consolidating these opts between PMs)?

249 ↗

(On Diff #181595)

Is this still TODO? I see handling further below.

284 ↗

(On Diff #181595)

As an aside - theoretically someone might want to do CS profiling without regular PGO. Is it worth supporting that combination?

unittests/ProfileData/InstrProfTest.cpp

179 ↗

(On Diff #181595)

document const param

805 ↗

(On Diff #181595)

ditto

[xur]

cmake/modules/HandleLLVMOptions.cmake

816 ↗

(On Diff #181595)

Fixed the typo.

I think currently -profile-instr-use flag is only added to CMAKE_CXX_FLAGS and CMAKE_C_FLAGS. I need them to pass to linker and shared linker also. Since there is no flag for CSIR use build. I have to pass them unconditionally. For regular PGO, the flag is void operation.

I'll remove the line of CMAKE_CXX_FLAGS and CMAKE_C_FLAGS

826 ↗

(On Diff #181595)

Moved the change to
https://reviews.llvm.org/D57068

include/llvm/Passes/PassBuilder.h

45 ↗

(On Diff #181595)

Make sense. I will change this part.

lib/Passes/PassBuilder.cpp

523 ↗

(On Diff #181595)

This is for the early inline. We skip the early inline if for optimization for size.
For CS, we will not do early inline either.

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

666 ↗

(On Diff #181595)

This change actually will affect function hash -- if an old profile is passed here, an non-cs record could be recognized as CS profile. But that will most likely result in a CFG hash mismatch where the user will get a warning.

678 ↗

(On Diff #181595)

not intentionally. I will revert this change.

1507 ↗

(On Diff #181595)

Good catch! This is from my other changes that fixes BB profile count. They should be removed.

1642 ↗

(On Diff #181595)

Fixed. PSI is not used here.

tools/llvm-profdata/llvm-profdata.cpp

716 ↗

(On Diff #181595)

Yes!

tools/opt/NewPMDriver.cpp

249 ↗

(On Diff #181595)

Yes. we should remove this TODO.

284 ↗

(On Diff #181595)

Since both CSprofile and non-CS-profile are merged into one profile. I think it's earlier to assume regular PGO is enabled for CSPGO. If there is a need for support the other way, I will add later.

[xur]

Integrated Teresa's review comments.

Also fixed a bug that can cause mismatch in new pass manager -- some optimization passes are under guard of PGOOptions::IRUse. LTOBackend sets PGOAction to NOAction for CSIRInstr (because regular profile is not passed down for CSIRInstr). This leads to different IR for CSIRInstr and CSIRUse.

This patch sets the PGOAction to IRUSE for CSIRInstr.

[xur]

Fixed a pass ordering issue in legacy pass manager for ThinLTO: Need to place CS Instrumentation after EliminateAvailableExternallyPass. This can cause duplicate profile variables that leads to bad profile for online profile merge.

Also synced the trunk and fixed a merge issue.

[tejohnson]

Mostly minor comments inline. But still missing tests. Needs testing via new gold-plugin and opt options. Should also add options to llvm-lto2 to enable testing the LTOBackend path without needing gold or any other specific linker.

include/llvm/Passes/PassBuilder.h

55 ↗

(On Diff #183899)

Should we also have IRUse when CSAction is CSIRInstr?

include/llvm/ProfileData/InstrProfReader.h

438 ↗

(On Diff #181595)

The same change should be done elsewhere in this class (document and change to UseCS). There are a couple other interfaces with the same option.

include/llvm/ProfileData/InstrProfWriter.h

77 ↗

(On Diff #183899)

Document new parameter (there is currently nowhere in the file that indicates CS means context sensitive).

include/llvm/Transforms/Instrumentation.h

91 ↗

(On Diff #183899)

Document new parameter here and further down in createInstrProfilingLegacyPass (there is currently nowhere in the file that indicates CS means context sensitive).

lib/Passes/PassBuilder.cpp

523 ↗

(On Diff #181595)

Please add comment to that effect.

lib/Transforms/IPO/PassManagerBuilder.cpp

283 ↗

(On Diff #183899)

Similar suggestion to the one I had for new PM (add comment about now not when IsCS).

lib/Transforms/Instrumentation/InstrProfiling.cpp

999 ↗

(On Diff #183899)

typo in instrumentation (missing n in the middle).

tools/opt/opt.cpp

290 ↗

(On Diff #183899)

The reason I suggested moving this here was to use the same set of options to set up CSPGO for the old PM. But it doesn't seem like it is being set up when invoked via opt. I'd expect something down in AddOptimizationPasses like where it sets up regular IR PGO PassManagerBuilder options.

[davidxl]

Please also add a profile dumping/reading round trip test with isCS flag (there are existing similar test for llvm-profdata)

text-format --> index-format-> text-format : diff text-format

include/llvm/ProfileData/InstrProfWriter.h

91 ↗

(On Diff #183899)

What is this (promotion)'s use case?

include/llvm/Transforms/Instrumentation/PGOInstrumentation.h

31 ↗

(On Diff #183899)

Add documentation for the new class. Also explain why it needs to be created in a pass.

lib/Analysis/ProfileSummaryInfo.cpp

89 ↗

(On Diff #183899)

This is confusing. It is better to just pass 'IsCS' flag to getProfileSummary method.

lib/ProfileData/InstrProf.cpp

1015 ↗

(On Diff #183899)

nit: createProfileFileNameVar.

Can this one be split out as a NFC change?

tools/llvm-profdata/llvm-profdata.cpp

909 ↗

(On Diff #183899)

New option needs to be documented in command line guide.

[xur]

Integrated the review comments from Teresa and David.
Added unit tests for
(1) llvm-profdata
(2) context-sensitive profile summary
(3) opt new options and passes
(4) gold plugin
(5) using lto2 to test csgen and csuse.

[davidxl]

include/llvm/ProfileData/InstrProfWriter.h

91 ↗

(On Diff #183899)

Can you explain the 'promotion' case here (as comments)?

lib/Analysis/ProfileSummaryInfo.cpp

89 ↗

(On Diff #183899)

how about this comment?

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

497 ↗

(On Diff #185907)

Remove this.

[xur]

Fixed the missing changes David mentioned.

include/llvm/Passes/PassBuilder.h

55 ↗

(On Diff #183899)

CSIRInstr can run without IRUSE. CSIRUse needs to IRUse (the only reason is they share the profile).

include/llvm/ProfileData/InstrProfWriter.h

91 ↗

(On Diff #183899)

when merging a non-cs profile with cs profile, the result is cs profile.

lib/ProfileData/InstrProf.cpp

1015 ↗

(On Diff #183899)

split the change to nfc r353230.

tools/opt/opt.cpp

290 ↗

(On Diff #183899)

OK. Got it.

[xur]

Forgot to include one more test:
A test/Other/cspgo-O2-pipeline.ll

[xur]

Removed the unrelated instructions (comments) from the tests.

[tejohnson]

LGTM, but wait for @davidxl to do final approval since I'm not sure his last comments addressed.

[davidxl]

lgtm

[ABataev]

llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h

41

Seems to me, it breaks the build with the shared libraries. Could you take a look at this?
lib/Passes/CMakeFiles/LLVMPasses.dir/PassBuilder.cpp.o: In function `llvm::PGOInstrumentationGenCreateVar::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&)':
/build/llvm/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h:42: undefined reference to `llvm::createProfileFileNameVar(llvm::Module&, llvm::StringRef)'
/build/llvm/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h:43: undefined reference to `llvm::createIRLevelProfileFlagVar(llvm::Module&, bool)'

[rupprecht]

llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h

41

Not OP, but I also hit this. rL355346 should fix it.

[ABataev]

llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h

41

Thanks for the fix!

[hctim]

Looks like this commit has broken the clang-hexagon-elf#23291: build. PTAL :)

FAIL: LLVM :: Transforms/PGOProfile/thinlto_cspgo_gen.ll (41188 of 44167)
******************** TEST 'LLVM :: Transforms/PGOProfile/thinlto_cspgo_gen.ll' FAILED ********************
Script:
--
: 'RUN: at line 1';   /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/bin/opt -module-summary /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/llvm/test/Transforms/PGOProfile/thinlto_cspgo_gen.ll -o /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp1.bc
: 'RUN: at line 2';   /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/bin/opt -module-summary /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/llvm/test/Transforms/PGOProfile/Inputs/thinlto_cspgo_bar_gen.ll -o /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp2.bc
: 'RUN: at line 3';   /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/bin/llvm-lto2 run -lto-cspgo-profile-file=alloc -lto-cspgo-gen -save-temps -o /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp1.bc /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp2.bc    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp1.bc,foo,pl    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp1.bc,bar,l    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp1.bc,main,plx    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp1.bc,__llvm_profile_filename,plx    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp1.bc,__llvm_profile_raw_version,plx    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp2.bc,bar,pl    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp2.bc,odd,pl    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp2.bc,even,pl    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp2.bc,__llvm_profile_filename,x    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp2.bc,__llvm_profile_raw_version,x
: 'RUN: at line 14';   /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/bin/llvm-dis /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_gen.ll.tmp.1.4.opt.bc -o - | /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/bin/FileCheck /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/llvm/test/Transforms/PGOProfile/thinlto_cspgo_gen.ll --check-prefix=CSGEN
--
Exit Code: 1

Command Output (stderr):
--
llvm-lto2: LTO::run failed: No available targets are compatible with triple "x86_64-unknown-linux-gnu"

--

********************
PASS: LLVM :: Transforms/PartiallyInlineLibCalls/nobuiltin.ll (41189 of 44167)
PASS: LLVM :: Transforms/PartiallyInlineLibCalls/bad-prototype.ll (41190 of 44167)
PASS: LLVM :: Transforms/PGOProfile/memop_size_annotation.ll (41191 of 44167)
PASS: LLVM :: Transforms/PGOProfile/loop1.ll (41192 of 44167)
PASS: LLVM :: Transforms/PhaseOrdering/2010-03-22-empty-baseclass.ll (41193 of 44167)
PASS: LLVM :: Transforms/PGOProfile/loop2.ll (41194 of 44167)
FAIL: LLVM :: Transforms/PGOProfile/thinlto_cspgo_use.ll (41195 of 44167)
******************** TEST 'LLVM :: Transforms/PGOProfile/thinlto_cspgo_use.ll' FAILED ********************
Script:
--
: 'RUN: at line 1';   /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/bin/opt -module-summary /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/llvm/test/Transforms/PGOProfile/thinlto_cspgo_use.ll -o /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp1.bc
: 'RUN: at line 2';   /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/bin/opt -module-summary /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/llvm/test/Transforms/PGOProfile/Inputs/thinlto_cspgo_bar_use.ll -o /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp2.bc
: 'RUN: at line 3';   /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/bin/llvm-profdata merge /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/llvm/test/Transforms/PGOProfile/Inputs/thinlto_cs.proftext -o /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp3.profdata
: 'RUN: at line 4';   /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/bin/llvm-lto2 run -lto-cspgo-profile-file=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp3.profdata -save-temps -o /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp1.bc /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp2.bc    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp1.bc,foo,pl    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp1.bc,bar,l    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp1.bc,main,plx    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp2.bc,bar,pl    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp2.bc,odd,pl    -r=/var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp2.bc,even,pl
: 'RUN: at line 11';   /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/bin/llvm-dis /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/test/Transforms/PGOProfile/Output/thinlto_cspgo_use.ll.tmp.1.4.opt.bc -o - | /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/stage1/bin/FileCheck /var/lib/buildbot/slaves/hexagon-build-03/clang-hexagon-elf/llvm/test/Transforms/PGOProfile/thinlto_cspgo_use.ll --check-prefix=CSUSE
--
Exit Code: 1

Command Output (stderr):
--
llvm-lto2: LTO::run failed: No available targets are compatible with triple "x86_64-unknown-linux-gnu"

--

********************

[pcc]

llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp

673

Hi Rong, I discovered a problem with this line of code. If the number of select instructions when the profile was collected differs by a multiple of 16 from the number when the profile is used, we get a hash collision which results in us hitting an assertion here:
http://llvm-cs.pcc.me.uk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp#1042

Granted, the same problem seems to have already existed for differences that are multiples of 256, but this seems to make the problem more likely to occur.

[xur]

llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp

673

I sent this patch for review.

https://reviews.llvm.org/D60154

llvm/trunk/include/llvm/InitializePasses.h

 void initializeObjCARCOptPass(PassRegistry&);
 void initializeOptimizationRemarkEmitterWrapperPassPass(PassRegistry&);
 void initializeOptimizePHIsPass(PassRegistry&);
 void initializePAEvalPass(PassRegistry&);
 void initializePEIPass(PassRegistry&);
 void initializePGOIndirectCallPromotionLegacyPassPass(PassRegistry&);
 void initializePGOInstrumentationGenLegacyPassPass(PassRegistry&);
 void initializePGOInstrumentationUseLegacyPassPass(PassRegistry&);
+void initializePGOInstrumentationGenCreateVarLegacyPassPass(PassRegistry&);
 void initializePGOMemOPSizeOptLegacyPassPass(PassRegistry&);
 void initializePHIEliminationPass(PassRegistry&);
 void initializePartialInlinerLegacyPassPass(PassRegistry&);
 void initializePartiallyInlineLibCallsLegacyPassPass(PassRegistry&);
 void initializePatchableFunctionPass(PassRegistry&);
 void initializePeepholeOptimizerPass(PassRegistry&);
 void initializePhiValuesWrapperPassPass(PassRegistry&);
 void initializePhysicalRegisterUsageInfoPass(PassRegistry&);

llvm/trunk/include/llvm/LTO/Config.h

struct Config {
 
   /// Flag to indicate that the optimizer should not assume builtins are present
   /// on the target.
   bool Freestanding = false;
 
   /// Disable entirely the optimizer, including importing for ThinLTO
   bool CodeGenOnly = false;
 
+  /// Run PGO context sensitive IR instrumentation.
+  bool RunCSIRInstr = false;
+
   /// If this field is set, the set of passes run in the middle-end optimizer
   /// will be the one specified by the string. Only works with the new pass
   /// manager as the old one doesn't have this ability.
   std::string OptPipeline;
 
   // If this field is set, it has the same effect of specifying an AA pipeline
   // identified by the string. Only works with the new pass manager, in
   // conjunction OptPipeline.
   std::string AAPipeline;
 
   /// Setting this field will replace target triples in input files with this
   /// triple.
   std::string OverrideTriple;
 
   /// Setting this field will replace unspecified target triples in input files
   /// with this triple.
   std::string DefaultTriple;
 
+  /// Context Sensitive PGO profile path.
+  std::string CSIRProfile;
+
   /// Sample PGO profile path.
   std::string SampleProfile;
 
   /// Name remapping file for profile data.
   std::string ProfileRemapping;
 
   /// The directory to store .dwo files.
   std::string DwoDir;

llvm/trunk/include/llvm/LinkAllPasses.h

ForcePassLinking() {
       (void) llvm::createDependenceAnalysisWrapperPass();
       (void) llvm::createDomOnlyPrinterPass();
       (void) llvm::createDomPrinterPass();
       (void) llvm::createDomOnlyViewerPass();
       (void) llvm::createDomViewerPass();
       (void) llvm::createGCOVProfilerPass();
       (void) llvm::createPGOInstrumentationGenLegacyPass();
       (void) llvm::createPGOInstrumentationUseLegacyPass();
+      (void) llvm::createPGOInstrumentationGenCreateVarLegacyPass();
       (void) llvm::createPGOIndirectCallPromotionLegacyPass();
       (void) llvm::createPGOMemOPSizeOptLegacyPass();
       (void) llvm::createInstrProfilingLegacyPass();
       (void) llvm::createFunctionImportPass();
       (void) llvm::createFunctionInliningPass();
       (void) llvm::createAlwaysInlinerLegacyPass();
       (void) llvm::createGlobalDCEPass();
       (void) llvm::createGlobalOptimizerPass();

llvm/trunk/include/llvm/ProfileData/InstrProf.h

private:
   void scaleValueProfData(uint32_t ValueKind, uint64_t Weight,
                           function_ref<void(instrprof_error)> Warn);
 };
 
 struct NamedInstrProfRecord : InstrProfRecord {
   StringRef Name;
   uint64_t Hash;
 
+  // We reserve this bit as the flag for context sensitive profile record.
+  static const int CS_FLAG_IN_FUNC_HASH = 60;
+
   NamedInstrProfRecord() = default;
   NamedInstrProfRecord(StringRef Name, uint64_t Hash,
                        std::vector<uint64_t> Counts)
       : InstrProfRecord(std::move(Counts)), Name(Name), Hash(Hash) {}
+
+  static bool hasCSFlagInHash(uint64_t FuncHash) {
+    return ((FuncHash >> CS_FLAG_IN_FUNC_HASH) & 1);
+  }
+  static void setCSFlagInHash(uint64_t &FuncHash) {
+    FuncHash |= ((uint64_t)1 << CS_FLAG_IN_FUNC_HASH);
+  }
 };
 
 uint32_t InstrProfRecord::getNumValueKinds() const {
   uint32_t NumValueKinds = 0;
   for (uint32_t Kind = IPVK_First; Kind <= IPVK_Last; ++Kind)
     NumValueKinds += !(getValueSitesForKind(Kind).empty());
   return NumValueKinds;
 }

 
 // Version 1: First version
 // Version 2: Added value profile data section. Per-function control data
 // struct has more fields to describe value profile information.
 // Version 3: Compressed name section support. Function PGO name reference
 // from control data struct is changed from raw pointer to Name's MD5 value.
 // Version 4: ValueDataBegin and ValueDataSizes fields are removed from the
 // raw header.
+// Version 5: Bit 60 of FuncHash is reserved for the flag for the context
+// sensitive records.
 const uint64_t Version = INSTR_PROF_RAW_VERSION;
 
 template <class IntPtrT> inline uint64_t getMagic();
 template <> inline uint64_t getMagic<uint64_t>() {
   return INSTR_PROF_RAW_MAGIC_64;
 }
 
 template <> inline uint64_t getMagic<uint32_t>() {

 };
 
 } // end namespace RawInstrProf
 
 // Parse MemOP Size range option.
 void getMemOPSizeRangeFromOption(StringRef Str, int64_t &RangeStart,
                                  int64_t &RangeLast);
 
+// Create a COMDAT variable INSTR_PROF_RAW_VERSION_VAR to make the runtime
+// aware this is an ir_level profile so it can set the version flag.
+void createIRLevelProfileFlagVar(Module &M, bool IsCS);
+
 // Create the variable for the profile file name.
 void createProfileFileNameVar(Module &M, StringRef InstrProfileOutput);
 
 } // end namespace llvm
 
 #endif // LLVM_PROFILEDATA_INSTRPROF_H

llvm/trunk/include/llvm/ProfileData/InstrProfData.inc

 #define INSTR_PROF_INDEX_VERSION 5
 /* Coverage mapping format vresion (start from 0). */
 #define INSTR_PROF_COVMAP_VERSION 2
 
 /* Profile version is always of type uint64_t. Reserve the upper 8 bits in the
  * version for other variants of profile. We set the lowest bit of the upper 8
  * bits (i.e. bit 56) to 1 to indicate if this is an IR-level instrumentaiton
  * generated profile, and 0 if this is a Clang FE generated profile.
+ * 1 in bit 57 indicates there are context-sensitive records in the profile.
  */
 #define VARIANT_MASKS_ALL 0xff00000000000000ULL
 #define GET_VERSION(V) ((V) & ~VARIANT_MASKS_ALL)
 #define VARIANT_MASK_IR_PROF (0x1ULL << 56)
+#define VARIANT_MASK_CSIR_PROF (0x1ULL << 57)
 #define INSTR_PROF_RAW_VERSION_VAR __llvm_profile_raw_version
 #define INSTR_PROF_PROFILE_RUNTIME_VAR __llvm_profile_runtime
 
 /* The variable that holds the name of the profile data
  * specified via command line. */
 #define INSTR_PROF_PROFILE_NAME_VAR __llvm_profile_filename
 
 /* section name strings common to all targets other

llvm/trunk/include/llvm/Transforms/Instrumentation.h

struct GCOVOptions {
 
   // Regexes separated by a semi-colon to filter the files to not instrument.
   std::string Exclude;
 };
 
 ModulePass *createGCOVProfilerPass(const GCOVOptions &Options =
                                    GCOVOptions::getDefault());
 
-// PGO Instrumention
-ModulePass *createPGOInstrumentationGenLegacyPass();
+// PGO Instrumention. Parameter IsCS indicates if this is the context senstive
+// instrumentation.
+ModulePass *createPGOInstrumentationGenLegacyPass(bool IsCS = false);
 ModulePass *
-createPGOInstrumentationUseLegacyPass(StringRef Filename = StringRef(""));
+createPGOInstrumentationUseLegacyPass(StringRef Filename = StringRef(""),
+                                      bool IsCS = false);
+ModulePass *createPGOInstrumentationGenCreateVarLegacyPass(
+    StringRef CSInstrName = StringRef(""));
 ModulePass *createPGOIndirectCallPromotionLegacyPass(bool InLTO = false,
                                                      bool SamplePGO = false);
 FunctionPass *createPGOMemOPSizeOptLegacyPass();
 
 // The pgo-specific indirect call promotion function declared below is used by
 // the pgo-driven indirect call promotion and sample profile passes. It's a
 // wrapper around llvm::promoteCall, et al. that additionally computes !prof
 // metadata. We place it in a pgo namespace so it's not confused with the
struct InstrProfOptions {
   bool NoRedZone = false;
 
   // Do counter register promotion
   bool DoCounterPromotion = false;
 
   // Use atomic profile counter increments.
   bool Atomic = false;
 
+  // Use BFI to guide register promotion
+  bool UseBFIInPromotion = false;
+
   // Name of the profile file to use as output
   std::string InstrProfileOutput;
 
   InstrProfOptions() = default;
 };
 
-/// Insert frontend instrumentation based profiling.
+/// Insert frontend instrumentation based profiling. Parameter IsCS indicates if
+// this is the context senstive instrumentation.
 ModulePass *createInstrProfilingLegacyPass(
-    const InstrProfOptions &Options = InstrProfOptions());
+    const InstrProfOptions &Options = InstrProfOptions(), bool IsCS = false);
 
 FunctionPass *createHWAddressSanitizerPass(bool CompileKernel = false,
                                            bool Recover = false);
 
 // Insert DataFlowSanitizer (dynamic data flow analysis) instrumentation
 ModulePass *createDataFlowSanitizerPass(
     const std::vector<std::string> &ABIListFiles = std::vector<std::string>(),
     void *(*getArgTLS)() = nullptr, void *(*getRetValTLS)() = nullptr);

llvm/trunk/include/llvm/Transforms/Instrumentation/InstrProfiling.h

 using LoadStorePair = std::pair<Instruction *, Instruction *>;
 
 /// Instrumentation based profiling lowering pass. This pass lowers
 /// the profile instrumented code generated by FE or the IR based
 /// instrumentation pass.
 class InstrProfiling : public PassInfoMixin<InstrProfiling> {
 public:
   InstrProfiling() = default;
-  InstrProfiling(const InstrProfOptions &Options) : Options(Options) {}
+  InstrProfiling(const InstrProfOptions &Options, bool IsCS)
+      : Options(Options), IsCS(IsCS) {}
 
   PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
   bool run(Module &M, const TargetLibraryInfo &TLI);
 
 private:
   InstrProfOptions Options;
   Module *M;
   Triple TT;
   const TargetLibraryInfo *TLI;
   struct PerFunctionProfileData {
     uint32_t NumValueSites[IPVK_Last + 1];
     GlobalVariable *RegionCounters = nullptr;
     GlobalVariable *DataVar = nullptr;
 
     PerFunctionProfileData() {
       memset(NumValueSites, 0, sizeof(uint32_t) * (IPVK_Last + 1));
     }
   };
   DenseMap<GlobalVariable *, PerFunctionProfileData> ProfileDataMap;
   std::vector<GlobalValue *> UsedVars;
   std::vector<GlobalVariable *> ReferencedNames;
   GlobalVariable *NamesVar;
   size_t NamesSize;
 
+  // Is this lowering for the context-sensitive instrumentation.
+  bool IsCS;
+
   // vector of counter load/store pairs to be register promoted.
   std::vector<LoadStorePair> PromotionCandidates;
 
   // The start value of precise value profile range for memory intrinsic sizes.
   int64_t MemOPSizeRangeStart;
   // The end value of precise value profile range for memory intrinsic sizes.
   int64_t MemOPSizeRangeLast;
 

llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h

 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TRANSFORMS_PGOINSTRUMENTATION_H
 #define LLVM_TRANSFORMS_PGOINSTRUMENTATION_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/IR/PassManager.h"
+#include "llvm/ProfileData/InstrProf.h"
 #include <cstdint>
 #include <string>
 
 namespace llvm {
 
 class Function;
 class Instruction;
 class Module;
 
 /// The instrumentation (profile-instr-gen) pass for IR based PGO.
+// We use this pass to create COMDAT profile variables for context
+// sensitive PGO (CSPGO). The reason to have a pass for this is CSPGO
+// can be run after LTO/ThinLTO linking. Lld linker needs to see
+// all the COMDAT variables before linking. So we have this pass
+// always run before linking for CSPGO.
+class PGOInstrumentationGenCreateVar
+    : public PassInfoMixin<PGOInstrumentationGenCreateVar> {
+public:
+  PGOInstrumentationGenCreateVar(std::string CSInstrName = "")
+      : CSInstrName(CSInstrName) {}
+  PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM) {
+    createProfileFileNameVar(M, CSInstrName);
+    createIRLevelProfileFlagVar(M, /* IsCS */ true);
+    return PreservedAnalyses::all();
+  }
+
+private:
+  std::string CSInstrName;
+};
+
+/// The instrumentation (profile-instr-gen) pass for IR based PGO.
 class PGOInstrumentationGen : public PassInfoMixin<PGOInstrumentationGen> {
 public:
+  PGOInstrumentationGen(bool IsCS = false) : IsCS(IsCS) {}
   PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
+
+private:
+  // If this is a context sensitive instrumentation.
+  bool IsCS;
 };
 
 /// The profile annotation (profile-instr-use) pass for IR based PGO.
 class PGOInstrumentationUse : public PassInfoMixin<PGOInstrumentationUse> {
 public:
   PGOInstrumentationUse(std::string Filename = "",
-                        std::string RemappingFilename = "");
+                        std::string RemappingFilename = "", bool IsCS = false);
 
   PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
 
 private:
   std::string ProfileFileName;
   std::string ProfileRemappingFileName;
+  // If this is a context sensitive instrumentation.
+  bool IsCS;
 };
 
 /// The indirect function call promotion pass.
 class PGOIndirectCallPromotion : public PassInfoMixin<PGOIndirectCallPromotion> {
 public:
   PGOIndirectCallPromotion(bool IsInLTO = false, bool SamplePGO = false)
       : InLTO(IsInLTO), SamplePGO(SamplePGO) {}
 

llvm/trunk/lib/Passes/PassBuilder.cpp

FPM.addPass(
         createFunctionToLoopPassAdaptor(LoopRotatePass(), DebugLogging));
     MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM)));
 
     // Add the profile lowering pass.
     InstrProfOptions Options;
     if (!ProfileGenFile.empty())
       Options.InstrProfileOutput = ProfileGenFile;
     Options.DoCounterPromotion = true;
-    MPM.addPass(InstrProfiling(Options));
+    Options.UseBFIInPromotion = false;
+    MPM.addPass(InstrProfiling(Options, false));
   }
 
   if (!ProfileUseFile.empty())
     MPM.addPass(PGOInstrumentationUse(ProfileUseFile, ProfileRemappingFile));
 }
 
 static InlineParams
 getInlineParamsFromOptLevel(PassBuilder::OptimizationLevel Level) {

llvm/trunk/lib/ProfileData/InstrProf.cpp

if (Pos != std::string::npos) {
       if (Pos < MemOPSizeRange.size() - 1)
         MemOPSizeRange.substr(Pos + 1).getAsInteger(10, RangeLast);
     } else
       MemOPSizeRange.getAsInteger(10, RangeLast);
   }
   assert(RangeLast >= RangeStart);
 }
 
+// Create a COMDAT variable INSTR_PROF_RAW_VERSION_VAR to make the runtime
+// aware this is an ir_level profile so it can set the version flag.
+void createIRLevelProfileFlagVar(Module &M, bool IsCS) {
+  const StringRef VarName(INSTR_PROF_QUOTE(INSTR_PROF_RAW_VERSION_VAR));
+  Type *IntTy64 = Type::getInt64Ty(M.getContext());
+  uint64_t ProfileVersion = (INSTR_PROF_RAW_VERSION | VARIANT_MASK_IR_PROF);
+  if (IsCS)
+    ProfileVersion |= VARIANT_MASK_CSIR_PROF;
+  auto IRLevelVersionVariable = new GlobalVariable(
+      M, IntTy64, true, GlobalValue::WeakAnyLinkage,
+      Constant::getIntegerValue(IntTy64, APInt(64, ProfileVersion)), VarName);
+  IRLevelVersionVariable->setVisibility(GlobalValue::DefaultVisibility);
+  Triple TT(M.getTargetTriple());
+  if (TT.supportsCOMDAT()) {
+    IRLevelVersionVariable->setLinkage(GlobalValue::ExternalLinkage);
+    IRLevelVersionVariable->setComdat(M.getOrInsertComdat(VarName));
+  }
+}
+
 // Create the variable for the profile file name.
 void createProfileFileNameVar(Module &M, StringRef InstrProfileOutput) {
   if (InstrProfileOutput.empty())
     return;
   Constant *ProfileNameConst =
       ConstantDataArray::getString(M.getContext(), InstrProfileOutput, true);
   GlobalVariable *ProfileNameVar = new GlobalVariable(
       M, ProfileNameConst->getType(), true, GlobalValue::WeakAnyLinkage,

llvm/trunk/lib/Transforms/Instrumentation/InstrProfiling.cpp

 //===----------------------------------------------------------------------===//
 
 #include "llvm/Transforms/Instrumentation/InstrProfiling.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/ADT/Twine.h"
+#include "llvm/Analysis/BlockFrequencyInfo.h"
+#include "llvm/Analysis/BranchProbabilityInfo.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/Constant.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Dominators.h"

 
 class InstrProfilingLegacyPass : public ModulePass {
   InstrProfiling InstrProf;
 
 public:
   static char ID;
 
   InstrProfilingLegacyPass() : ModulePass(ID) {}
-  InstrProfilingLegacyPass(const InstrProfOptions &Options)
-      : ModulePass(ID), InstrProf(Options) {}
+  InstrProfilingLegacyPass(const InstrProfOptions &Options, bool IsCS)
+      : ModulePass(ID), InstrProf(Options, IsCS) {}
 
   StringRef getPassName() const override {
     return "Frontend instrumentation-based coverage lowering";
   }
 
   bool runOnModule(Module &M) override {
     return InstrProf.run(M, getAnalysis<TargetLibraryInfoWrapperPass>().getTLI());
   }

 
 /// A helper class to do register promotion for all profile counter
 /// updates in a loop.
 ///
 class PGOCounterPromoter {
 public:
   PGOCounterPromoter(
       DenseMap<Loop *, SmallVector<LoadStorePair, 8>> &LoopToCands,
-      Loop &CurLoop, LoopInfo &LI)
+      Loop &CurLoop, LoopInfo &LI, BlockFrequencyInfo *BFI)
       : LoopToCandidates(LoopToCands), ExitBlocks(), InsertPts(), L(CurLoop),
-        LI(LI) {
+        LI(LI), BFI(BFI) {
 
     SmallVector<BasicBlock *, 8> LoopExitBlocks;
     SmallPtrSet<BasicBlock *, 8> BlockSet;
     L.getExitBlocks(LoopExitBlocks);
 
     for (BasicBlock *ExitBlock : LoopExitBlocks) {
       if (BlockSet.insert(ExitBlock).second) {
         ExitBlocks.push_back(ExitBlock);
bool run(int64_t *NumPromoted) {
 
     unsigned Promoted = 0;
     for (auto &Cand : LoopToCandidates[&L]) {
 
       SmallVector<PHINode *, 4> NewPHIs;
       SSAUpdater SSA(&NewPHIs);
       Value *InitVal = ConstantInt::get(Cand.first->getType(), 0);
 
+      // If BFI is set, we will use it to guide the promotions.
+      if (BFI) {
+        auto *BB = Cand.first->getParent();
+        auto InstrCount = BFI->getBlockProfileCount(BB);
+        if (!InstrCount)
+          continue;
+        auto PreheaderCount = BFI->getBlockProfileCount(L.getLoopPreheader());
+        // If the average loop trip count is not greater than 1.5, we skip
+        // promotion.
+        if (PreheaderCount &&
+            (PreheaderCount.getValue() * 3) >= (InstrCount.getValue() * 2))
+          continue;
+      }
+
       PGOCounterPromoterHelper Promoter(Cand.first, Cand.second, SSA, InitVal,
                                         L.getLoopPreheader(), ExitBlocks,
                                         InsertPts, LoopToCandidates, LI);
       Promoter.run(SmallVector<Instruction *, 2>({Cand.first, Cand.second}));
       Promoted++;
       if (Promoted >= MaxProm)
         break;
 
if (!LP->hasDedicatedExits())
       return 0;
 
     BasicBlock *PH = LP->getLoopPreheader();
     if (!PH)
       return 0;
 
     SmallVector<BasicBlock *, 8> ExitingBlocks;
     LP->getExitingBlocks(ExitingBlocks);
+
+    // If BFI is set, we do more aggressive promotions based on BFI.
+    if (BFI)
+      return (unsigned)-1;
+
     // Not considierered speculative.
     if (ExitingBlocks.size() == 1)
       return MaxNumOfPromotionsPerLoop;
 
     if (ExitingBlocks.size() > SpeculativeCounterPromotionMaxExiting)
       return 0;
 
     // Whether the target block is in a loop does not matter:
unsigned getMaxNumOfPromotionsInLoop(Loop *LP) {
     return MaxProm;
   }
 
   DenseMap<Loop *, SmallVector<LoadStorePair, 8>> &LoopToCandidates;
   SmallVector<BasicBlock *, 8> ExitBlocks;
   SmallVector<Instruction *, 8> InsertPts;
   Loop &L;
   LoopInfo &LI;
+  BlockFrequencyInfo *BFI;
 };
 
 } // end anonymous namespace
 
 PreservedAnalyses InstrProfiling::run(Module &M, ModuleAnalysisManager &AM) {
   auto &TLI = AM.getResult<TargetLibraryAnalysis>(M);
   if (!run(M, TLI))
     return PreservedAnalyses::all();
 
   return PreservedAnalyses::none();
 }
 
 char InstrProfilingLegacyPass::ID = 0;
 INITIALIZE_PASS_BEGIN(
     InstrProfilingLegacyPass, "instrprof",
     "Frontend instrumentation-based coverage lowering.", false, false)
 INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
 INITIALIZE_PASS_END(
     InstrProfilingLegacyPass, "instrprof",
     "Frontend instrumentation-based coverage lowering.", false, false)
 
 ModulePass *
-llvm::createInstrProfilingLegacyPass(const InstrProfOptions &Options) {
-  return new InstrProfilingLegacyPass(Options);
+llvm::createInstrProfilingLegacyPass(const InstrProfOptions &Options,
+                                     bool IsCS) {
+  return new InstrProfilingLegacyPass(Options, IsCS);
 }
 
 static InstrProfIncrementInst *castToIncrementInst(Instruction *Instr) {
   InstrProfIncrementInst *Inc = dyn_cast<InstrProfIncrementInstStep>(Instr);
   if (Inc)
     return Inc;
   return dyn_cast<InstrProfIncrementInst>(Instr);
 }

 void InstrProfiling::promoteCounterLoadStores(Function *F) {
   if (!isCounterPromotionEnabled())
     return;
 
   DominatorTree DT(*F);
   LoopInfo LI(DT);
   DenseMap<Loop *, SmallVector<LoadStorePair, 8>> LoopPromotionCandidates;
 
+  std::unique_ptr<BlockFrequencyInfo> BFI;
+  if (Options.UseBFIInPromotion) {
+    std::unique_ptr<BranchProbabilityInfo> BPI;
+    BPI.reset(new BranchProbabilityInfo(*F, LI, TLI));
+    BFI.reset(new BlockFrequencyInfo(*F, *BPI, LI));
+  }
+
   for (const auto &LoadStore : PromotionCandidates) {
     auto *CounterLoad = LoadStore.first;
     auto *CounterStore = LoadStore.second;
     BasicBlock *BB = CounterLoad->getParent();
     Loop *ParentLoop = LI.getLoopFor(BB);
     if (!ParentLoop)
       continue;
     LoopPromotionCandidates[ParentLoop].emplace_back(CounterLoad, CounterStore);
   }
 
   SmallVector<Loop *, 4> Loops = LI.getLoopsInPreorder();
 
   // Do a post-order traversal of the loops so that counter updates can be
   // iteratively hoisted outside the loop nest.
   for (auto *Loop : llvm::reverse(Loops)) {
-    PGOCounterPromoter Promoter(LoopPromotionCandidates, *Loop, LI);
+    PGOCounterPromoter Promoter(LoopPromotionCandidates, *Loop, LI, BFI.get());
     Promoter.run(&TotalCountersPromoted);
   }
 }
 
 /// Check if the module contains uses of any profiling intrinsics.
 static bool containsProfilingIntrinsics(Module &M) {
   if (auto *F = M.getFunction(
           Intrinsic::getName(llvm::Intrinsic::instrprof_increment)))
static inline bool shouldRecordFunctionAddr(Function *F) {
   // indirect call target info.
   return F->hasAddressTaken() || F->hasLinkOnceLinkage();
 }
 
 static bool needsRuntimeRegistrationOfSectionRange(const Triple &TT) {
   // Don't do this for Darwin.  compiler-rt uses linker magic.
   if (TT.isOSDarwin())
     return false;
-
   // Use linker script magic to get data/cnts/name start/end.
   if (TT.isOSLinux() || TT.isOSFreeBSD() || TT.isOSNetBSD() ||
       TT.isOSFuchsia() || TT.isPS4CPU() || TT.isOSWindows())
     return false;
 
   return true;
 }
 

 }
 
 void InstrProfiling::emitUses() {
   if (!UsedVars.empty())
     appendToUsed(*M, UsedVars);
 }
 
 void InstrProfiling::emitInitialization() {
-  // Create variable for profile name.
+  // Create ProfileFileName variable. Don't don't this for the
+  // context-sensitive instrumentation lowering: This lowering is after
+  // LTO/ThinLTO linking. Pass PGOInstrumentationGenCreateVar should
+  // have already create the variable before LTO/ThinLTO linking.
+  if (!IsCS)
     createProfileFileNameVar(*M, Options.InstrProfileOutput);
   Function *RegisterF = M->getFunction(getInstrProfRegFuncsName());
   if (!RegisterF)
     return;
 
   // Create the initialization function.
   auto *VoidTy = Type::getVoidTy(M->getContext());
   auto *F = Function::Create(FunctionType::get(VoidTy, false),
                              GlobalValue::InternalLinkage,

llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp

 #include "llvm/ADT/iterator.h"
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/Analysis/BlockFrequencyInfo.h"
 #include "llvm/Analysis/BranchProbabilityInfo.h"
 #include "llvm/Analysis/CFG.h"
 #include "llvm/Analysis/IndirectCallVisitor.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/ProfileSummaryInfo.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CFG.h"
 #include "llvm/IR/CallSite.h"
 #include "llvm/IR/Comdat.h"
 #include "llvm/IR/Constant.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DiagnosticInfo.h"

 STATISTIC(NumOfPGOMemIntrinsics, "Number of mem intrinsics instrumented.");
 STATISTIC(NumOfPGOEdge, "Number of edges.");
 STATISTIC(NumOfPGOBB, "Number of basic-blocks.");
 STATISTIC(NumOfPGOSplit, "Number of critical edge splits.");
 STATISTIC(NumOfPGOFunc, "Number of functions having valid profile counts.");
 STATISTIC(NumOfPGOMismatch, "Number of functions having mismatch profile.");
 STATISTIC(NumOfPGOMissing, "Number of functions without profile.");
 STATISTIC(NumOfPGOICall, "Number of indirect call value instrumentations.");
+STATISTIC(NumOfCSPGOInstrument, "Number of edges instrumented in CSPGO.");
+STATISTIC(NumOfCSPGOSelectInsts,
+          "Number of select instruction instrumented in CSPGO.");
+STATISTIC(NumOfCSPGOMemIntrinsics,
+          "Number of mem intrinsics instrumented in CSPGO.");
+STATISTIC(NumOfCSPGOEdge, "Number of edges in CSPGO.");
+STATISTIC(NumOfCSPGOBB, "Number of basic-blocks in CSPGO.");
+STATISTIC(NumOfCSPGOSplit, "Number of critical edge splits in CSPGO.");
+STATISTIC(NumOfCSPGOFunc,
+          "Number of functions having valid profile counts in CSPGO.");
+STATISTIC(NumOfCSPGOMismatch,
+          "Number of functions having mismatch profile in CSPGO.");
+STATISTIC(NumOfCSPGOMissing, "Number of functions without profile in CSPGO.");
 
 // Command line option to specify the file to read profile from. This is
 // mainly used for testing.
 static cl::opt<std::string>
     PGOTestProfileFile("pgo-test-profile-file", cl::init(""), cl::Hidden,
                        cl::value_desc("filename"),
                        cl::desc("Specify the path of profile data file. This is"
                                 "mainly for test purpose."));
struct MemIntrinsicVisitor : public InstVisitor<MemIntrinsicVisitor> {
 
   unsigned getNumOfMemIntrinsics() const { return NMemIs; }
 };
 
 class PGOInstrumentationGenLegacyPass : public ModulePass {
 public:
   static char ID;
 
-  PGOInstrumentationGenLegacyPass() : ModulePass(ID) {
+  PGOInstrumentationGenLegacyPass(bool IsCS = false)
+      : ModulePass(ID), IsCS(IsCS) {
     initializePGOInstrumentationGenLegacyPassPass(
         *PassRegistry::getPassRegistry());
   }
 
   StringRef getPassName() const override { return "PGOInstrumentationGenPass"; }
 
 private:
+  // Is this is context-sensitive instrumentation.
+  bool IsCS;
   bool runOnModule(Module &M) override;
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addRequired<BlockFrequencyInfoWrapperPass>();
   }
 };
 
 class PGOInstrumentationUseLegacyPass : public ModulePass {
 public:
   static char ID;
 
   // Provide the profile filename as the parameter.
-  PGOInstrumentationUseLegacyPass(std::string Filename = "")
-      : ModulePass(ID), ProfileFileName(std::move(Filename)) {
+  PGOInstrumentationUseLegacyPass(std::string Filename = "", bool IsCS = false)
+      : ModulePass(ID), ProfileFileName(std::move(Filename)), IsCS(IsCS) {
     if (!PGOTestProfileFile.empty())
       ProfileFileName = PGOTestProfileFile;
     initializePGOInstrumentationUseLegacyPassPass(
         *PassRegistry::getPassRegistry());
   }
 
   StringRef getPassName() const override { return "PGOInstrumentationUsePass"; }
 
 private:
   std::string ProfileFileName;
+  // Is this is context-sensitive instrumentation use.
+  bool IsCS;
 
   bool runOnModule(Module &M) override;
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.addRequired<ProfileSummaryInfoWrapperPass>();
     AU.addRequired<BlockFrequencyInfoWrapperPass>();
   }
 };
 
+class PGOInstrumentationGenCreateVarLegacyPass : public ModulePass {
+public:
+  static char ID;
+  StringRef getPassName() const override {
+    return "PGOInstrumentationGenCreateVarPass";
+  }
+  PGOInstrumentationGenCreateVarLegacyPass(std::string CSInstrName = "")
+      : ModulePass(ID), InstrProfileOutput(CSInstrName) {
+    initializePGOInstrumentationGenCreateVarLegacyPassPass(
+        *PassRegistry::getPassRegistry());
+  }
+
+private:
+  bool runOnModule(Module &M) override {
+    createProfileFileNameVar(M, InstrProfileOutput);
+    createIRLevelProfileFlagVar(M, true);
+    return false;
+  }
+  std::string InstrProfileOutput;
+};
+
 } // end anonymous namespace
 
 char PGOInstrumentationGenLegacyPass::ID = 0;
 
 INITIALIZE_PASS_BEGIN(PGOInstrumentationGenLegacyPass, "pgo-instr-gen",
                       "PGO instrumentation.", false, false)
 INITIALIZE_PASS_DEPENDENCY(BlockFrequencyInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(BranchProbabilityInfoWrapperPass)
 INITIALIZE_PASS_END(PGOInstrumentationGenLegacyPass, "pgo-instr-gen",
                     "PGO instrumentation.", false, false)
 
-ModulePass *llvm::createPGOInstrumentationGenLegacyPass() {
-  return new PGOInstrumentationGenLegacyPass();
+ModulePass *llvm::createPGOInstrumentationGenLegacyPass(bool IsCS) {
+  return new PGOInstrumentationGenLegacyPass(IsCS);
 }
 
 char PGOInstrumentationUseLegacyPass::ID = 0;
 
 INITIALIZE_PASS_BEGIN(PGOInstrumentationUseLegacyPass, "pgo-instr-use",
                       "Read PGO instrumentation profile.", false, false)
 INITIALIZE_PASS_DEPENDENCY(BlockFrequencyInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(BranchProbabilityInfoWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(ProfileSummaryInfoWrapperPass)
 INITIALIZE_PASS_END(PGOInstrumentationUseLegacyPass, "pgo-instr-use",
                     "Read PGO instrumentation profile.", false, false)
 
-ModulePass *llvm::createPGOInstrumentationUseLegacyPass(StringRef Filename) {
-  return new PGOInstrumentationUseLegacyPass(Filename.str());
+ModulePass *llvm::createPGOInstrumentationUseLegacyPass(StringRef Filename,
+                                                        bool IsCS) {
+  return new PGOInstrumentationUseLegacyPass(Filename.str(), IsCS);
+}
+
+char PGOInstrumentationGenCreateVarLegacyPass::ID = 0;
+
+INITIALIZE_PASS(PGOInstrumentationGenCreateVarLegacyPass,
+                "pgo-instr-gen-create-var",
+                "Create PGO instrumentation version variable for CSPGO.", false,
+                false)
+
+ModulePass *
+llvm::createPGOInstrumentationGenCreateVarLegacyPass(StringRef CSInstrName) {
+  return new PGOInstrumentationGenCreateVarLegacyPass(CSInstrName);
 }
 
 namespace {
 
 /// An MST based instrumentation for PGO
 ///
 /// Implements a Minimum Spanning Tree (MST) based instrumentation for PGO
 /// in the function level.
struct BBInfo {
   }
 };
 
 // This class implements the CFG edges. Note the CFG can be a multi-graph.
 template <class Edge, class BBInfo> class FuncPGOInstrumentation {
 private:
   Function &F;
 
+  // Is this is context-sensitive instrumentation.
+  bool IsCS;
+
   // A map that stores the Comdat group in function F.
   std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers;
 
   void computeCFGHash();
   void renameComdatFunction();
 
 public:
   std::vector<std::vector<Instruction *>> ValueSites;
void dumpInfo(std::string Str = "") const {
     MST.dumpEdges(dbgs(), Twine("Dump Function ") + FuncName + " Hash: " +
                               Twine(FunctionHash) + "\t" + Str);
   }
 
   FuncPGOInstrumentation(
       Function &Func,
       std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers,
       bool CreateGlobalVar = false, BranchProbabilityInfo *BPI = nullptr,
-      BlockFrequencyInfo *BFI = nullptr)
-      : F(Func), ComdatMembers(ComdatMembers), ValueSites(IPVK_Last + 1),
-        SIVisitor(Func), MIVisitor(Func), MST(F, BPI, BFI) {
+      BlockFrequencyInfo *BFI = nullptr, bool IsCS = false)
+      : F(Func), IsCS(IsCS), ComdatMembers(ComdatMembers),
+        ValueSites(IPVK_Last + 1), SIVisitor(Func), MIVisitor(Func),
+        MST(F, BPI, BFI) {
     // This should be done before CFG hash computation.
     SIVisitor.countSelects(Func);
     MIVisitor.countMemIntrinsics(Func);
+    if (!IsCS) {
       NumOfPGOSelectInsts += SIVisitor.getNumOfSelectInsts();
       NumOfPGOMemIntrinsics += MIVisitor.getNumOfMemIntrinsics();
+      NumOfPGOBB += MST.BBInfos.size();
       ValueSites[IPVK_IndirectCallTarget] = findIndirectCalls(Func);
+    } else {
+      NumOfCSPGOSelectInsts += SIVisitor.getNumOfSelectInsts();
+      NumOfCSPGOMemIntrinsics += MIVisitor.getNumOfMemIntrinsics();
+      NumOfCSPGOBB += MST.BBInfos.size();
+    }
     ValueSites[IPVK_MemOPSize] = MIVisitor.findMemIntrinsics(Func);
 
     FuncName = getPGOFuncName(F);
     computeCFGHash();
     if (!ComdatMembers.empty())
       renameComdatFunction();
     LLVM_DEBUG(dumpInfo("after CFGMST"));
 
-    NumOfPGOBB += MST.BBInfos.size();
     for (auto &E : MST.AllEdges) {
       if (E->Removed)
         continue;
-      NumOfPGOEdge++;
+      IsCS ? NumOfCSPGOEdge++ : NumOfPGOEdge++;
       if (!E->InMST)
-        NumOfPGOInstrument++;
+        IsCS ? NumOfCSPGOInstrument++ : NumOfPGOInstrument++;
     }
 
     if (CreateGlobalVar)
       FuncNameVar = createPGOFuncNameVar(F, FuncName);
   }
 
   // Return the number of profile counters needed for the function.
   unsigned getNumCounters() {
for (unsigned I = 0, E = TI->getNumSuccessors(); I != E; ++I) {
       if (BI == nullptr)
         continue;
       uint32_t Index = BI->Index;
       for (int J = 0; J < 4; J++)
         Indexes.push_back((char)(Index >> (J * 8)));
     }
   }
   JC.update(Indexes);
+
+  // Hash format for context sensitive profile. Reserve 4 bits for other
+  // information.
   FunctionHash = (uint64_t)SIVisitor.getNumOfSelectInsts() << 56 |
                  (uint64_t)ValueSites[IPVK_IndirectCallTarget].size() << 48 |
+                 //(uint64_t)ValueSites[IPVK_MemOPSize].size() << 40 |
                  (uint64_t)MST.AllEdges.size() << 32 | JC.getCRC();
+  // Reserve bit 60-63 for other information purpose.
+  FunctionHash &= 0x0FFFFFFFFFFFFFFF;
+  if (IsCS)
+    NamedInstrProfRecord::setCSFlagInHash(FunctionHash);
   LLVM_DEBUG(dbgs() << "Function Hash Computation for " << F.getName() << ":\n"
                     << " CRC = " << JC.getCRC()
                     << ", Selects = " << SIVisitor.getNumOfSelectInsts()
                     << ", Edges = " << MST.AllEdges.size() << ", ICSites = "
                     << ValueSites[IPVK_IndirectCallTarget].size()
                     << ", Hash = " << FunctionHash << "\n";);
 }
 
BasicBlock *FuncPGOInstrumentation<Edge, BBInfo>::getInstrBB(Edge *E) {
   Instruction *TI = SrcBB->getTerminator();
   if (TI->getNumSuccessors() <= 1)
     return SrcBB;
   if (!E->IsCritical)
     return DestBB;
 
   // For a critical edge, we have to split. Instrument the newly
   // created BB.
-  NumOfPGOSplit++;
+  IsCS ? NumOfCSPGOSplit++ : NumOfPGOSplit++;
   LLVM_DEBUG(dbgs() << "Split critical edge: " << getBBInfo(SrcBB).Index
                     << " --> " << getBBInfo(DestBB).Index << "\n");
   unsigned SuccNum = GetSuccessorNumber(SrcBB, DestBB);
   BasicBlock *InstrBB = SplitCriticalEdge(TI, SuccNum);
   assert(InstrBB && "Critical edge is not split");
 
   E->Removed = true;
   return InstrBB;
 }
 
 // Visit all edge and instrument the edges not in MST, and do value profiling.
 // Critical edges will be split.
 static void instrumentOneFunc(
     Function &F, Module *M, BranchProbabilityInfo *BPI, BlockFrequencyInfo *BFI,
-    std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers) {
+    std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers,
+    bool IsCS) {
   // Split indirectbr critical edges here before computing the MST rather than
   // later in getInstrBB() to avoid invalidating it.
   SplitIndirectBrCriticalEdges(F, BPI, BFI);
+
   FuncPGOInstrumentation<PGOEdge, BBInfo> FuncInfo(F, ComdatMembers, true, BPI,
-                                                   BFI);
+                                                   BFI, IsCS);
   unsigned NumCounters = FuncInfo.getNumCounters();
 
   uint32_t I = 0;
   Type *I8PtrTy = Type::getInt8PtrTy(M->getContext());
   for (auto &E : FuncInfo.MST.AllEdges) {
     BasicBlock *InstrBB = FuncInfo.getInstrBB(E.get());
     if (!InstrBB)
       continue;

 
 namespace {
 
 class PGOUseFunc {
 public:
   PGOUseFunc(Function &Func, Module *Modu,
              std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers,
              BranchProbabilityInfo *BPI = nullptr,
-             BlockFrequencyInfo *BFIin = nullptr)
+             BlockFrequencyInfo *BFIin = nullptr, bool IsCS = false)
       : F(Func), M(Modu), BFI(BFIin),
-        FuncInfo(Func, ComdatMembers, false, BPI, BFIin),
-        FreqAttr(FFA_Normal) {}
+        FuncInfo(Func, ComdatMembers, false, BPI, BFIin, IsCS),
+        FreqAttr(FFA_Normal), IsCS(IsCS) {}
 
   // Read counts for the instrumented BB from profile.
   bool readCounters(IndexedInstrProfReader *PGOReader, bool &AllZeros);
 
   // Populate the counts for all BBs.
   void populateCounters();
 
   // Set the branch weights based on the count values.
private:
   uint32_t ProfileCountSize = 0;
 
   // ProfileRecord for this function.
   InstrProfRecord ProfileRecord;
 
   // Function hotness info derived from profile.
   FuncFreqAttr FreqAttr;
 
+  // Is to use the context sensitive profile.
+  bool IsCS;
+
   // Find the Instrumented BB and set the value.
   void setInstrumentedCounts(const std::vector<uint64_t> &CountFromProfile);
 
   // Set the edge counter value for the unknown edge -- there should be only
   // one unknown edge.
   void setEdgeCount(DirectEdges &Edges, uint64_t Value);
 
   // Return FuncName string;

 bool PGOUseFunc::readCounters(IndexedInstrProfReader *PGOReader, bool &AllZeros) {
   auto &Ctx = M->getContext();
   Expected<InstrProfRecord> Result =
       PGOReader->getInstrProfRecord(FuncInfo.FuncName, FuncInfo.FunctionHash);
   if (Error E = Result.takeError()) {
     handleAllErrors(std::move(E), [&](const InstrProfError &IPE) {
       auto Err = IPE.get();
       bool SkipWarning = false;
+      LLVM_DEBUG(dbgs() << "Error in reading profile for Func "
+                        << FuncInfo.FuncName << ": ");
       if (Err == instrprof_error::unknown_function) {
-        NumOfPGOMissing++;
+        IsCS ? NumOfCSPGOMissing++ : NumOfPGOMissing++;
         SkipWarning = !PGOWarnMissing;
+        LLVM_DEBUG(dbgs() << "unknown function");
       } else if (Err == instrprof_error::hash_mismatch ||
                  Err == instrprof_error::malformed) {
-        NumOfPGOMismatch++;
+        IsCS ? NumOfCSPGOMismatch++ : NumOfPGOMismatch++;
         SkipWarning =
             NoPGOWarnMismatch ||
             (NoPGOWarnMismatchComdat &&
              (F.hasComdat() ||
               F.getLinkage() == GlobalValue::AvailableExternallyLinkage));
+        LLVM_DEBUG(dbgs() << "hash mismatch (skip=" << SkipWarning << ")");
       }
 
+      LLVM_DEBUG(dbgs() << " IsCS=" << IsCS << "\n");
       if (SkipWarning)
         return;
 
-      std::string Msg = IPE.message() + std::string(" ") + F.getName().str();
+      std::string Msg = IPE.message() + std::string(" ") + F.getName().str() +
+                        std::string(" Hash = ") +
+                        std::to_string(FuncInfo.FunctionHash);
+
       Ctx.diagnose(
           DiagnosticInfoPGOProfile(M->getName().data(), Msg, DS_Warning));
     });
     return false;
   }
   ProfileRecord = std::move(Result.get());
   std::vector<uint64_t> &CountFromProfile = ProfileRecord.Counts;
 
-  NumOfPGOFunc++;
+  IsCS ? NumOfCSPGOFunc++ : NumOfPGOFunc++;
   LLVM_DEBUG(dbgs() << CountFromProfile.size() << " counts\n");
   uint64_t ValueSum = 0;
   for (unsigned I = 0, S = CountFromProfile.size(); I < S; I++) {
     LLVM_DEBUG(dbgs() << "  " << I << ": " << CountFromProfile[I] << "\n");
     ValueSum += CountFromProfile[I];
   }
   AllZeros = (ValueSum == 0);
 
   LLVM_DEBUG(dbgs() << "SUM =  " << ValueSum << "\n");
 
   getBBInfo(nullptr).UnknownCountOutEdge = 2;
   getBBInfo(nullptr).UnknownCountInEdge = 2;
 
   setInstrumentedCounts(CountFromProfile);
+#if 0
+  ProgramMaxCount = PGOReader->getMaximumFunctionCount(IsCS);
+#else
   ProgramMaxCount = PGOReader->getMaximumFunctionCount();
+#endif
   return true;
 }
 
 // Populate the counters from instrumented BBs to all BBs.
 // In the end of this operation, all BBs should have a valid count value.
 void PGOUseFunc::populateCounters() {
   // First set up Count variable for all BBs.
   for (auto &E : FuncInfo.MST.AllEdges) {
#endif
   assert(CountPosition == ProfileCountSize);
 
   LLVM_DEBUG(FuncInfo.dumpInfo("after reading profile."));
 }
 
 // Assign the scaled count values to the BB with multiple out edges.
 void PGOUseFunc::setBranchWeights() {
   // Generate MD_prof metadata for every branch instruction.
-  LLVM_DEBUG(dbgs() << "\nSetting branch weights.\n");
+  LLVM_DEBUG(dbgs() << "\nSetting branch weights for func " << F.getName()
+                    << " IsCS=" << IsCS << "\n");
   for (auto &BB : F) {
     Instruction *TI = BB.getTerminator();
     if (TI->getNumSuccessors() < 2)
       continue;
     if (!(isa<BranchInst>(TI) || isa<SwitchInst>(TI) ||
           isa<IndirectBrInst>(TI)))
       continue;
+
     if (getBBInfo(&BB).CountValue == 0)
       continue;
 
     // We have a non-zero Branch BB.
     const UseBBInfo &BBCountInfo = getBBInfo(&BB);
     unsigned Size = BBCountInfo.OutEdges.size();
     SmallVector<uint64_t, 2> EdgeCounts(Size, 0);
     uint64_t MaxCount = 0;
for (auto &I : ValueSites) {
     annotateValueSite(*M, *I, ProfileRecord,
                       static_cast<InstrProfValueKind>(Kind), ValueSiteIndex,
                       Kind == IPVK_MemOPSize ? MaxNumMemOPAnnotations
                                              : MaxNumAnnotations);
     ValueSiteIndex++;
   }
 }
 
-// Create a COMDAT variable INSTR_PROF_RAW_VERSION_VAR to make the runtime
-// aware this is an ir_level profile so it can set the version flag.
-static void createIRLevelProfileFlagVariable(Module &M) {
-  Type *IntTy64 = Type::getInt64Ty(M.getContext());
-  uint64_t ProfileVersion = (INSTR_PROF_RAW_VERSION | VARIANT_MASK_IR_PROF);
-  auto IRLevelVersionVariable = new GlobalVariable(
-      M, IntTy64, true, GlobalVariable::ExternalLinkage,
-      Constant::getIntegerValue(IntTy64, APInt(64, ProfileVersion)),
-      INSTR_PROF_QUOTE(INSTR_PROF_RAW_VERSION_VAR));
-  IRLevelVersionVariable->setVisibility(GlobalValue::DefaultVisibility);
-  Triple TT(M.getTargetTriple());
-  if (!TT.supportsCOMDAT())
-    IRLevelVersionVariable->setLinkage(GlobalValue::WeakAnyLinkage);
-  else
-    IRLevelVersionVariable->setComdat(M.getOrInsertComdat(
-        StringRef(INSTR_PROF_QUOTE(INSTR_PROF_RAW_VERSION_VAR))));
-}
-
 // Collect the set of members for each Comdat in module M and store
 // in ComdatMembers.
 static void collectComdatMembers(
     Module &M,
     std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers) {
   if (!DoComdatRenaming)
     return;
   for (Function &F : M)
     if (Comdat *C = F.getComdat())
       ComdatMembers.insert(std::make_pair(C, &F));
   for (GlobalVariable &GV : M.globals())
     if (Comdat *C = GV.getComdat())
       ComdatMembers.insert(std::make_pair(C, &GV));
   for (GlobalAlias &GA : M.aliases())
     if (Comdat *C = GA.getComdat())
       ComdatMembers.insert(std::make_pair(C, &GA));
 }
 
 static bool InstrumentAllFunctions(
     Module &M, function_ref<BranchProbabilityInfo *(Function &)> LookupBPI,
-    function_ref<BlockFrequencyInfo *(Function &)> LookupBFI) {
-  createIRLevelProfileFlagVariable(M);
+    function_ref<BlockFrequencyInfo *(Function &)> LookupBFI, bool IsCS) {
+  // For the context-sensitve instrumentation, we should have a separated pass
+  // (before LTO/ThinLTO linking) to create these variables.
+  if (!IsCS)
+    createIRLevelProfileFlagVar(M, /* IsCS */ false);
   std::unordered_multimap<Comdat *, GlobalValue *> ComdatMembers;
   collectComdatMembers(M, ComdatMembers);
 
   for (auto &F : M) {
     if (F.isDeclaration())
       continue;
     auto *BPI = LookupBPI(F);
     auto *BFI = LookupBFI(F);
-    instrumentOneFunc(F, &M, BPI, BFI, ComdatMembers);
+    instrumentOneFunc(F, &M, BPI, BFI, ComdatMembers, IsCS);
   }
   return true;
 }
 
 bool PGOInstrumentationGenLegacyPass::runOnModule(Module &M) {
   if (skipModule(M))
     return false;
 
   auto LookupBPI = [this](Function &F) {
     return &this->getAnalysis<BranchProbabilityInfoWrapperPass>(F).getBPI();
   };
   auto LookupBFI = [this](Function &F) {
     return &this->getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI();
   };
-  return InstrumentAllFunctions(M, LookupBPI, LookupBFI);
+  return InstrumentAllFunctions(M, LookupBPI, LookupBFI, IsCS);
 }
 
 PreservedAnalyses PGOInstrumentationGen::run(Module &M,
                                              ModuleAnalysisManager &AM) {
   auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
   auto LookupBPI = [&FAM](Function &F) {
     return &FAM.getResult<BranchProbabilityAnalysis>(F);
   };
 
   auto LookupBFI = [&FAM](Function &F) {
     return &FAM.getResult<BlockFrequencyAnalysis>(F);
   };
 
-  if (!InstrumentAllFunctions(M, LookupBPI, LookupBFI))
+  if (!InstrumentAllFunctions(M, LookupBPI, LookupBFI, IsCS))
     return PreservedAnalyses::all();
 
   return PreservedAnalyses::none();
 }
 
 static bool annotateAllFunctions(
     Module &M, StringRef ProfileFileName, StringRef ProfileRemappingFileName,
     function_ref<BranchProbabilityInfo *(Function &)> LookupBPI,
-    function_ref<BlockFrequencyInfo *(Function &)> LookupBFI) {
+    function_ref<BlockFrequencyInfo *(Function &)> LookupBFI, bool IsCS) {
   LLVM_DEBUG(dbgs() << "Read in profile counters: ");
   auto &Ctx = M.getContext();
   // Read the counter array from file.
   auto ReaderOrErr =
       IndexedInstrProfReader::create(ProfileFileName, ProfileRemappingFileName);
   if (Error E = ReaderOrErr.takeError()) {
     handleAllErrors(std::move(E), [&](const ErrorInfoBase &EI) {
       Ctx.diagnose(
           DiagnosticInfoPGOProfile(ProfileFileName.data(), EI.message()));
     });
     return false;
   }
 
   std::unique_ptr<IndexedInstrProfReader> PGOReader =
       std::move(ReaderOrErr.get());
   if (!PGOReader) {
     Ctx.diagnose(DiagnosticInfoPGOProfile(ProfileFileName.data(),
                                           StringRef("Cannot get PGOReader")));
     return false;
   }
+#if 0
+  if (!PGOReader->hasCSIRLevelProfile() && IsCS)
+    return false;
+#endif
+
   // TODO: might need to change the warning once the clang option is finalized.
   if (!PGOReader->isIRLevelProfile()) {
     Ctx.diagnose(DiagnosticInfoPGOProfile(
         ProfileFileName.data(), "Not an IR level instrumentation profile"));
     return false;
   }
 
   std::unordered_multimap<Comdat *, GlobalValue *> ComdatMembers;
   collectComdatMembers(M, ComdatMembers);
   std::vector<Function *> HotFunctions;
   std::vector<Function *> ColdFunctions;
   for (auto &F : M) {
     if (F.isDeclaration())
       continue;
     auto *BPI = LookupBPI(F);
     auto *BFI = LookupBFI(F);
     // Split indirectbr critical edges here before computing the MST rather than
     // later in getInstrBB() to avoid invalidating it.
     SplitIndirectBrCriticalEdges(F, BPI, BFI);
-    PGOUseFunc Func(F, &M, ComdatMembers, BPI, BFI);
+    PGOUseFunc Func(F, &M, ComdatMembers, BPI, BFI, IsCS);
     bool AllZeros = false;
     if (!Func.readCounters(PGOReader.get(), AllZeros))
       continue;
     if (AllZeros) {
       F.setEntryCount(ProfileCount(0, Function::PCT_Real));
       if (Func.getProgramMaxCount() != 0)
         ColdFunctions.push_back(&F);
       continue;
if (PGOViewRawCounts != PGOVCT_None &&
         else
           ViewGraph(&Func, Twine("PGORawCounts_") + Func.getFunc().getName());
       else if (PGOViewRawCounts == PGOVCT_Text) {
         dbgs() << "pgo-view-raw-counts: " << Func.getFunc().getName() << "\n";
         Func.dumpInfo();
       }
     }
   }
+#if 0
+  M.setProfileSummary(PGOReader->getSummary(IsCS).getMD(M.getContext()),
+                      IsCS ? ProfileSummary::PSK_CSInstr
+                           : ProfileSummary::PSK_Instr);
+#else
   M.setProfileSummary(PGOReader->getSummary().getMD(M.getContext()));
+#endif
+
   // Set function hotness attribute from the profile.
   // We have to apply these attributes at the end because their presence
   // can affect the BranchProbabilityInfo of any callers, resulting in an
   // inconsistent MST between prof-gen and prof-use.
   for (auto &F : HotFunctions) {
     F->addFnAttr(Attribute::InlineHint);
     LLVM_DEBUG(dbgs() << "Set inline attribute to function: " << F->getName()
                       << "\n");
   }
   for (auto &F : ColdFunctions) {
     F->addFnAttr(Attribute::Cold);
     LLVM_DEBUG(dbgs() << "Set cold attribute to function: " << F->getName()
                       << "\n");
   }
   return true;
 }
 
 PGOInstrumentationUse::PGOInstrumentationUse(std::string Filename,
-                                             std::string RemappingFilename)
+                                             std::string RemappingFilename,
+                                             bool IsCS)
     : ProfileFileName(std::move(Filename)),
-      ProfileRemappingFileName(std::move(RemappingFilename)) {
+      ProfileRemappingFileName(std::move(RemappingFilename)), IsCS(IsCS) {
   if (!PGOTestProfileFile.empty())
     ProfileFileName = PGOTestProfileFile;
   if (!PGOTestProfileRemappingFile.empty())
     ProfileRemappingFileName = PGOTestProfileRemappingFile;
 }
 
 PreservedAnalyses PGOInstrumentationUse::run(Module &M,
                                              ModuleAnalysisManager &AM) {
 
   auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
   auto LookupBPI = [&FAM](Function &F) {
     return &FAM.getResult<BranchProbabilityAnalysis>(F);
   };
 
   auto LookupBFI = [&FAM](Function &F) {
     return &FAM.getResult<BlockFrequencyAnalysis>(F);
   };
 
   if (!annotateAllFunctions(M, ProfileFileName, ProfileRemappingFileName,
-                            LookupBPI, LookupBFI))
+                            LookupBPI, LookupBFI, IsCS))
     return PreservedAnalyses::all();
 
   return PreservedAnalyses::none();
 }
 
 bool PGOInstrumentationUseLegacyPass::runOnModule(Module &M) {
   if (skipModule(M))
     return false;
 
   auto LookupBPI = [this](Function &F) {
     return &this->getAnalysis<BranchProbabilityInfoWrapperPass>(F).getBPI();
   };
   auto LookupBFI = [this](Function &F) {
     return &this->getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI();
   };
 
-  return annotateAllFunctions(M, ProfileFileName, "", LookupBPI, LookupBFI);
+  return annotateAllFunctions(M, ProfileFileName, "", LookupBPI, LookupBFI,
+                              IsCS);
 }
 
 static std::string getSimpleNodeName(const BasicBlock *Node) {
   if (!Node->getName().empty())
     return Node->getName();
 
   std::string SimpleNodeName;
   raw_string_ostream OS(SimpleNodeName);

llvm/trunk/tools/gold/gold-plugin.cpp

#endif
   static std::string dwo_dir;
   /// Statistics output filename.
   static std::string stats_file;
 
   // Optimization remarks filename and hotness options
   static std::string OptRemarksFilename;
   static bool OptRemarksWithHotness = false;
 
+  // Context sensitive PGO options.
+  static std::string cs_profile_path;
+  static bool cs_pgo_gen = false;
+
   static void process_plugin_option(const char *opt_)
   {
     if (opt_ == nullptr)
       return;
     llvm::StringRef opt = opt_;
 
     if (opt.startswith("mcpu=")) {
       mcpu = opt.substr(strlen("mcpu="));
if (opt.startswith("mcpu=")) {
         message(LDPL_FATAL, "Invalid parallelism level: %s", opt_ + 5);
     } else if (opt.startswith("lto-partitions=")) {
       if (opt.substr(strlen("lto-partitions="))
               .getAsInteger(10, ParallelCodeGenParallelismLevel))
         message(LDPL_FATAL, "Invalid codegen partition level: %s", opt_ + 5);
     } else if (opt == "disable-verify") {
       DisableVerify = true;
     } else if (opt.startswith("sample-profile=")) {
-      sample_profile= opt.substr(strlen("sample-profile="));
+      sample_profile = opt.substr(strlen("sample-profile="));
+    } else if (opt == "cs-profile-generate") {
+      cs_pgo_gen = true;
+    } else if (opt.startswith("cs-profile-path=")) {
+      cs_profile_path = opt.substr(strlen("cs-profile-path="));
     } else if (opt == "new-pass-manager") {
       new_pass_manager = true;
     } else if (opt == "debug-pass-manager") {
       debug_pass_manager = true;
     } else if (opt.startswith("dwo_dir=")) {
       dwo_dir = opt.substr(strlen("dwo_dir="));
     } else if (opt.startswith("opt-remarks-filename=")) {
       OptRemarksFilename = opt.substr(strlen("opt-remarks-filename="));
static std::unique_ptr<LTO> createLTO(IndexWriteCallback OnIndexWrite,
   case options::OT_ASM_ONLY:
     Conf.CGFileType = TargetMachine::CGFT_AssemblyFile;
     break;
   }
 
   if (!options::sample_profile.empty())
     Conf.SampleProfile = options::sample_profile;
 
+  if (!options::cs_profile_path.empty())
+    Conf.CSIRProfile = options::cs_profile_path;
+  Conf.RunCSIRInstr = options::cs_pgo_gen;
+
   Conf.DwoDir = options::dwo_dir;
 
   // Set up optimization remarks handling.
   Conf.RemarksFilename = options::OptRemarksFilename;
   Conf.RemarksWithHotness = options::OptRemarksWithHotness;
 
   // Use new pass manager if set in driver
   Conf.UseNewPM = options::new_pass_manager;

llvm/trunk/tools/llvm-lto2/llvm-lto2.cpp

static cl::opt<bool> OptRemarksWithHotness(
     "pass-remarks-with-hotness",
     cl::desc("Whether to include hotness informations in the remarks.\n"
              "Has effect only if -pass-remarks-output is specified."));
 
 static cl::opt<std::string>
     SamplePGOFile("lto-sample-profile-file",
                   cl::desc("Specify a SamplePGO profile file"));
 
+static cl::opt<std::string>
+    CSPGOFile("lto-cspgo-profile-file",
+              cl::desc("Specify a context sensitive PGO profile file"));
+
+static cl::opt<bool>
+    RunCSIRInstr("lto-cspgo-gen",
+                 cl::desc("Run PGO context sensitive IR instrumentation"),
+                 cl::init(false), cl::Hidden);
+
 static cl::opt<bool>
     UseNewPM("use-new-pm",
              cl::desc("Run LTO passes using the new pass manager"),
              cl::init(false), cl::Hidden);
 
 static cl::opt<bool>
     DebugPassManager("debug-pass-manager", cl::init(false), cl::Hidden,
                      cl::desc("Print pass management debugging information"));
if (SaveTemps)
     check(Conf.addSaveTemps(OutputFilename + "."),
           "Config::addSaveTemps failed");
 
   // Optimization remarks.
   Conf.RemarksFilename = OptRemarksOutput;
   Conf.RemarksWithHotness = OptRemarksWithHotness;
 
   Conf.SampleProfile = SamplePGOFile;
+  Conf.CSIRProfile = CSPGOFile;
+  Conf.RunCSIRInstr = RunCSIRInstr;
 
   // Run a custom pipeline, if asked for.
   Conf.OptPipeline = OptPipeline;
   Conf.AAPipeline = AAPipeline;
 
   Conf.OptLevel = OptLevel - '0';
   Conf.UseNewPM = UseNewPM;
   switch (CGOptLevel) {