[IndirectCallPromotion][SimpifyCFG] Preserve indirect callsites in hoist/sink when merging !prof is unprofitable and merge these callsites after ICP completes in a build pipeline

[mingmingl]

Summary

The problem to solve:

• When simplify-cfg could hoist / sink indirect callsites with inherently different target values, there isn't a simple lossless way to merge !prof value profile metadata.
  • hoist of indirect call https://gcc.godbolt.org/z/o6G68rn3v, sink of indirect call https://gcc.godbolt.org/z/79E3onono. The C++ comments and test cases have brief comments to explain why/when not preserving is sub-optimal. Basically, ICP heuristics (code) promotes when a target value show up enough times, and merging the sum changes the distribution.

This patch:

1. Before indirect-call-promotion happens, preserve the indirect callsites with target value profiles in simplify-cfg when there isn't a good way to merge.
   • Basically, when two indirect callsites each has value profiles, preserve them; or else it's fine to hoist or sink before ICP.
2. After indirect-call-promotion happens, always sink or hoist (when possible) and merge value profiles.

Implementation:

1. Set a module flag after the last ICP pass completes in a binary build pipeline.
   • Pass in ThinOrFullLTOPhase to ICP pass constructor. IndirectCallProm transformations all happen after ICP pass completes in a {thin, regular} LTO postlink pipeline or in a non-LTO pipeline.
2. Add helper functions to decide whether it's profitable to merge and get merged results.
3. SimplifyCFG calls helper functions above to preserve callsites for indirect-call-prom to happen, or hoist/sink indirect calls && merge value profiles.
4. Clearing value profiles after ICP makes use of them won't work since CGProfile pass still uses <target-value, counter> pairs to compute function hotness.

[tejohnson]

Would be good to explain motivation (maybe reference other patch?)

Also, the new tests only test the behavior with the internal option. It would be good to have tests with various pipelines to check the logic on when it gets cleared in those different cases simply based on the new pipeline logic.

llvm/include/llvm/IR/Instructions.h

4018 ↗

(On Diff #515602)

Nothing in this patch uses the IndirectCallMD value set here - is it needed?

llvm/lib/Transforms/Instrumentation/IndirectCallPromotion.cpp

307

This is unnecessary if !MDProf

311

Can the CallBase be anything other than a CallInst or InvokeInst?

385

Comment describing why

[wenlei]

cc @hoy to make sure this works for csspgo.

[mingmingl]

In D148876#4288286, @tejohnson wrote:

Would be good to explain motivation (maybe reference other patch?)

Makes sense, will do.

Also, the new tests only test the behavior with the internal option. It would be good to have tests with various pipelines to check the logic on when it gets cleared in those different cases simply based on the new pipeline logic.

I see your point. I'm going to write a test like https://github.com/llvm/llvm-project/blob/main/llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll to test pipeline logic.

In D148876#4288429, @wenlei wrote:

cc @hoy to make sure this works for csspgo.

thanks! The assumption here is that ICP runs once per pipeline and there are no other passes that uses value-profiles. If the assumption doesn't hold, I'm thinking about an explicit boolean in ICP pass constructor (which makes pipeline logic a little bit more complex)

[hoy]

In D148876#4288504, @mingmingl wrote:

In D148876#4288286, @tejohnson wrote:

Would be good to explain motivation (maybe reference other patch?)

Makes sense, will do.

Also, the new tests only test the behavior with the internal option. It would be good to have tests with various pipelines to check the logic on when it gets cleared in those different cases simply based on the new pipeline logic.

I see your point. I'm going to write a test like https://github.com/llvm/llvm-project/blob/main/llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll to test pipeline logic.

In D148876#4288429, @wenlei wrote:

cc @hoy to make sure this works for csspgo.

thanks! The assumption here is that ICP runs once per pipeline and there are no other passes that uses value-profiles. If the assumption doesn't hold, I'm thinking about an explicit boolean in ICP pass constructor (which makes pipeline logic a little bit more complex)

Value profiles for indirect call sites can still be used by lld call graph sorting after ICP. Please see https://github.com/llvm/llvm-project/blob/26202a57e5e78b5473ef657737d181f0a68ed56d/llvm/lib/Transforms/Instrumentation/CGProfile.cpp#L81

[mingmingl]

In D148876#4288686, @hoy wrote:

In D148876#4288504, @mingmingl wrote:

In D148876#4288286, @tejohnson wrote:

Would be good to explain motivation (maybe reference other patch?)

Makes sense, will do.

Also, the new tests only test the behavior with the internal option. It would be good to have tests with various pipelines to check the logic on when it gets cleared in those different cases simply based on the new pipeline logic.

I see your point. I'm going to write a test like https://github.com/llvm/llvm-project/blob/main/llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll to test pipeline logic.

In D148876#4288429, @wenlei wrote:

cc @hoy to make sure this works for csspgo.

thanks! The assumption here is that ICP runs once per pipeline and there are no other passes that uses value-profiles. If the assumption doesn't hold, I'm thinking about an explicit boolean in ICP pass constructor (which makes pipeline logic a little bit more complex)

Value profiles for indirect call sites can still be used by lld call graph sorting after ICP. Please see https://github.com/llvm/llvm-project/blob/26202a57e5e78b5473ef657737d181f0a68ed56d/llvm/lib/Transforms/Instrumentation/CGProfile.cpp#L81

oh thanks for this pointer! Clearing is not an option then.

CGProfile.cpp uses <counter, target-val> pairs but not 'total-count', I'm suggesting this option. Let me know your thoughts!

In this patch

1. extend its format from !1 = !{!"VP", IndirectCallKind , Sum, counter1, val1, counter2, val2} to !1 = !{!"VP", Kind , Sum, counter1, val1, counter2, val2, boolean/int 'indirect-call-promotion-all-done'}
2. indirect-call-promotion-all-done is initially 0, and set to 1 after the last ICP in a compiler pipeline

In D147954

3. simplify-cfg bails out from merge indirect-calls when seeing indirect-call-promotion-all-done as 0, could merge target value profiles after seeing indirect-call-promotion-all-done being set to 1, so that CGProfile.cpp could get the <counter, target-val> pairs.

Meanwhile, the current ICP should probably still preserve counts as opposed to clearing when all targets are used.

cc @tejohnson @davidxl

[hoy]

In D148876#4288764, @mingmingl wrote:

In D148876#4288686, @hoy wrote:

In D148876#4288504, @mingmingl wrote:

In D148876#4288286, @tejohnson wrote:

Would be good to explain motivation (maybe reference other patch?)

Makes sense, will do.

Also, the new tests only test the behavior with the internal option. It would be good to have tests with various pipelines to check the logic on when it gets cleared in those different cases simply based on the new pipeline logic.

I see your point. I'm going to write a test like https://github.com/llvm/llvm-project/blob/main/llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll to test pipeline logic.

In D148876#4288429, @wenlei wrote:

cc @hoy to make sure this works for csspgo.

thanks! The assumption here is that ICP runs once per pipeline and there are no other passes that uses value-profiles. If the assumption doesn't hold, I'm thinking about an explicit boolean in ICP pass constructor (which makes pipeline logic a little bit more complex)

Value profiles for indirect call sites can still be used by lld call graph sorting after ICP. Please see https://github.com/llvm/llvm-project/blob/26202a57e5e78b5473ef657737d181f0a68ed56d/llvm/lib/Transforms/Instrumentation/CGProfile.cpp#L81

oh thanks for this pointer! Clearing is not an option then.

CGProfile.cpp uses <counter, target-val> pairs but not 'total-count', I'm suggesting this option. Let me know your thoughts!

In this patch

1. extend its format from !1 = !{!"VP", IndirectCallKind , Sum, counter1, val1, counter2, val2} to !1 = !{!"VP", Kind , Sum, counter1, val1, counter2, val2, boolean/int 'indirect-call-promotion-all-done'}
2. indirect-call-promotion-all-done is initially 0, and set to 1 after the last ICP in a compiler pipeline

In D147954

3. simplify-cfg bails out from merge indirect-calls when seeing indirect-call-promotion-all-done as 0, could merge target value profiles after seeing indirect-call-promotion-all-done being set to 1, so that CGProfile.cpp could get the <counter, target-val> pairs.

Meanwhile, the current ICP should probably still preserve counts as opposed to clearing when all targets are used.

cc @tejohnson @davidxl

I guess I don't get the motivation of the original change. Can you please remind me why you'd like to cleanup VP metadata after ICP?

[mingmingl]

In D148876#4288774, @hoy wrote:

In D148876#4288764, @mingmingl wrote:

In D148876#4288686, @hoy wrote:

In D148876#4288504, @mingmingl wrote:

In D148876#4288286, @tejohnson wrote:

Would be good to explain motivation (maybe reference other patch?)

Makes sense, will do.

Also, the new tests only test the behavior with the internal option. It would be good to have tests with various pipelines to check the logic on when it gets cleared in those different cases simply based on the new pipeline logic.

I see your point. I'm going to write a test like https://github.com/llvm/llvm-project/blob/main/llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll to test pipeline logic.

In D148876#4288429, @wenlei wrote:

cc @hoy to make sure this works for csspgo.

thanks! The assumption here is that ICP runs once per pipeline and there are no other passes that uses value-profiles. If the assumption doesn't hold, I'm thinking about an explicit boolean in ICP pass constructor (which makes pipeline logic a little bit more complex)

Value profiles for indirect call sites can still be used by lld call graph sorting after ICP. Please see https://github.com/llvm/llvm-project/blob/26202a57e5e78b5473ef657737d181f0a68ed56d/llvm/lib/Transforms/Instrumentation/CGProfile.cpp#L81

oh thanks for this pointer! Clearing is not an option then.

CGProfile.cpp uses <counter, target-val> pairs but not 'total-count', I'm suggesting this option. Let me know your thoughts!

In this patch

1. extend its format from !1 = !{!"VP", IndirectCallKind , Sum, counter1, val1, counter2, val2} to !1 = !{!"VP", Kind , Sum, counter1, val1, counter2, val2, boolean/int 'indirect-call-promotion-all-done'}
2. indirect-call-promotion-all-done is initially 0, and set to 1 after the last ICP in a compiler pipeline

In D147954

3. simplify-cfg bails out from merge indirect-calls when seeing indirect-call-promotion-all-done as 0, could merge target value profiles after seeing indirect-call-promotion-all-done being set to 1, so that CGProfile.cpp could get the <counter, target-val> pairs.

Meanwhile, the current ICP should probably still preserve counts as opposed to clearing when all targets are used.

cc @tejohnson @davidxl

I guess I don't get the motivation of the original change. Can you please remind me why you'd like to cleanup VP metadata after ICP?

(D147954 is about the problem to solve and has more discussions) In brief, https://gcc.godbolt.org/z/Pxoh8PrGE shows a case where simplify-cfg aggressively drops profile metadata while it shouldn't. This would happen when prelink simplify-cfg enables instruction sink and postlink ICP couldn't see the value profiles.
The motivation of this patch and D147954 is to preserve value profiles attached on indirect calls for indirect-call-promotion to use.

Turning off simplify-cfg sinking/hoisting completely in prelink pipeline would be more invasive -> for 'llvm/test/Transforms/SimplifyCFG/merge-direct-call-branch-weights-in-hoist.ll' in D147954, it's desired to hoist address calculation instructions (from if.then and if.else in https://gcc.godbolt.org/z/b4Y9Mqrq1) in prelink stage.

[hoy]

In D148876#4288824, @mingmingl wrote:

In D148876#4288774, @hoy wrote:

In D148876#4288764, @mingmingl wrote:

In D148876#4288686, @hoy wrote:

In D148876#4288504, @mingmingl wrote:

In D148876#4288286, @tejohnson wrote:

Would be good to explain motivation (maybe reference other patch?)

Makes sense, will do.

Also, the new tests only test the behavior with the internal option. It would be good to have tests with various pipelines to check the logic on when it gets cleared in those different cases simply based on the new pipeline logic.

I see your point. I'm going to write a test like https://github.com/llvm/llvm-project/blob/main/llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll to test pipeline logic.

In D148876#4288429, @wenlei wrote:

cc @hoy to make sure this works for csspgo.

thanks! The assumption here is that ICP runs once per pipeline and there are no other passes that uses value-profiles. If the assumption doesn't hold, I'm thinking about an explicit boolean in ICP pass constructor (which makes pipeline logic a little bit more complex)

Value profiles for indirect call sites can still be used by lld call graph sorting after ICP. Please see https://github.com/llvm/llvm-project/blob/26202a57e5e78b5473ef657737d181f0a68ed56d/llvm/lib/Transforms/Instrumentation/CGProfile.cpp#L81

oh thanks for this pointer! Clearing is not an option then.

CGProfile.cpp uses <counter, target-val> pairs but not 'total-count', I'm suggesting this option. Let me know your thoughts!

In this patch

1. extend its format from !1 = !{!"VP", IndirectCallKind , Sum, counter1, val1, counter2, val2} to !1 = !{!"VP", Kind , Sum, counter1, val1, counter2, val2, boolean/int 'indirect-call-promotion-all-done'}
2. indirect-call-promotion-all-done is initially 0, and set to 1 after the last ICP in a compiler pipeline

In D147954

3. simplify-cfg bails out from merge indirect-calls when seeing indirect-call-promotion-all-done as 0, could merge target value profiles after seeing indirect-call-promotion-all-done being set to 1, so that CGProfile.cpp could get the <counter, target-val> pairs.

Meanwhile, the current ICP should probably still preserve counts as opposed to clearing when all targets are used.

cc @tejohnson @davidxl

I guess I don't get the motivation of the original change. Can you please remind me why you'd like to cleanup VP metadata after ICP?

(D147954 is about the problem to solve and has more discussions) In brief, https://gcc.godbolt.org/z/Pxoh8PrGE shows a case where simplify-cfg aggressively drops profile metadata while it shouldn't. This would happen when prelink simplify-cfg enables instruction sink and postlink ICP couldn't see the value profiles.
The motivation of this patch and D147954 is to preserve value profiles attached on indirect calls for indirect-call-promotion to use.

Turning off simplify-cfg sinking/hoisting completely in prelink pipeline would be more invasive -> for 'llvm/test/Transforms/SimplifyCFG/merge-direct-call-branch-weights-in-hoist.ll' in D147954, it's desired to hoist address calculation instructions (from if.then and if.else in https://gcc.godbolt.org/z/b4Y9Mqrq1) in prelink stage.

Thanks for the explanation. IIUC, the fix for SimplyCFG dropping !VP is to either block the corresponding optimization or to merge the !VP data when merge callsites. What is the problem of always merging callsite !VP?

[mingmingl]

In D148876#4288873, @hoy wrote:

In D148876#4288824, @mingmingl wrote:

In D148876#4288774, @hoy wrote:

In D148876#4288764, @mingmingl wrote:

In D148876#4288686, @hoy wrote:

In D148876#4288504, @mingmingl wrote:

In D148876#4288286, @tejohnson wrote:

Would be good to explain motivation (maybe reference other patch?)

Makes sense, will do.

Also, the new tests only test the behavior with the internal option. It would be good to have tests with various pipelines to check the logic on when it gets cleared in those different cases simply based on the new pipeline logic.

I see your point. I'm going to write a test like https://github.com/llvm/llvm-project/blob/main/llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll to test pipeline logic.

In D148876#4288429, @wenlei wrote:

cc @hoy to make sure this works for csspgo.

thanks! The assumption here is that ICP runs once per pipeline and there are no other passes that uses value-profiles. If the assumption doesn't hold, I'm thinking about an explicit boolean in ICP pass constructor (which makes pipeline logic a little bit more complex)

Value profiles for indirect call sites can still be used by lld call graph sorting after ICP. Please see https://github.com/llvm/llvm-project/blob/26202a57e5e78b5473ef657737d181f0a68ed56d/llvm/lib/Transforms/Instrumentation/CGProfile.cpp#L81

oh thanks for this pointer! Clearing is not an option then.

CGProfile.cpp uses <counter, target-val> pairs but not 'total-count', I'm suggesting this option. Let me know your thoughts!

In this patch

1. extend its format from !1 = !{!"VP", IndirectCallKind , Sum, counter1, val1, counter2, val2} to !1 = !{!"VP", Kind , Sum, counter1, val1, counter2, val2, boolean/int 'indirect-call-promotion-all-done'}
2. indirect-call-promotion-all-done is initially 0, and set to 1 after the last ICP in a compiler pipeline

In D147954

3. simplify-cfg bails out from merge indirect-calls when seeing indirect-call-promotion-all-done as 0, could merge target value profiles after seeing indirect-call-promotion-all-done being set to 1, so that CGProfile.cpp could get the <counter, target-val> pairs.

Meanwhile, the current ICP should probably still preserve counts as opposed to clearing when all targets are used.

cc @tejohnson @davidxl

I guess I don't get the motivation of the original change. Can you please remind me why you'd like to cleanup VP metadata after ICP?

(D147954 is about the problem to solve and has more discussions) In brief, https://gcc.godbolt.org/z/Pxoh8PrGE shows a case where simplify-cfg aggressively drops profile metadata while it shouldn't. This would happen when prelink simplify-cfg enables instruction sink and postlink ICP couldn't see the value profiles.
The motivation of this patch and D147954 is to preserve value profiles attached on indirect calls for indirect-call-promotion to use.

Turning off simplify-cfg sinking/hoisting completely in prelink pipeline would be more invasive -> for 'llvm/test/Transforms/SimplifyCFG/merge-direct-call-branch-weights-in-hoist.ll' in D147954, it's desired to hoist address calculation instructions (from if.then and if.else in https://gcc.godbolt.org/z/b4Y9Mqrq1) in prelink stage.

Thanks for the explanation. IIUC, the fix for SimplyCFG dropping !VP is to either block the corresponding optimization or to merge the !VP data when merge callsites. What is the problem of always merging callsite !VP?

Take https://gcc.godbolt.org/z/o6G68rn3v as example, func1 and func2 are two different virtual methods so inherently has different target values.

Say each virtual callsite has a sum and a list of <counter, target-value> pairs, merging <counter, target-value> pairs is okay, either merging sum, or use max(sum1, sum2) changed the ratio, and could make a significant target value no longer profitable according to isPromotionProfitable

[wenlei]

Thanks for the explanation. IIUC, the fix for SimplyCFG dropping !VP is to either block the corresponding optimization or to merge the !VP data when merge callsites. What is the problem of always merging callsite !VP?

@hoy the problem is you lose flow-sensitivity for the VP after merge. And since ICP is important, it perhaps warrant special handling.

[hoy]

I see. Merging could cause the loss of flow-sensitivity for PGO, but not merging doesn't give optimal code quality.

Here are my thoughts about what could happen with @mingmingl's approach that allows merging only after ICP.

1. For the training build that does not use a profile, my intuition is that we will always merge the indirect callsites, which will lead to a merged !VP profile.
2. Then for the next build that uses the profile, the callsites will first be promoted in separate, and then the leftover unpromoted callsites will be merged. That merged callsite will lead to a merged !VP profile. The already promoted callsites, if not merged, will lead to separate !VP profiles.
3. For all subsequent builds, the separate !VP profiles will be dominant, the leftover merged profile shouldn't be important as they represent non-typical paths.

The questions is, in #2, whether promoted callsites can be merged or not. I'm not quite sure but since their promotion is based on the same profile from #1, the resulting direct calls will all look the same which might be good candidates to merge. If merged, the generated profile based on this build will still lose flow-sensitivity.

There's also another possibility which is no callsites can be merged after ICP, because of the conditional code introduced during ICP. If this is common, then it sounds like just letting indirect callsites be merged (#1) as is isn't a deal breaker.

Thoughts?

[davidxl]

In D148876#4289051, @hoy wrote:

I see. Merging could cause the loss of flow-sensitivity for PGO, but not merging doesn't give optimal code quality.

Here are my thoughts about what could happen with @mingmingl's approach that allows merging only after ICP.

For instrumentation PGO, the situation is much simpler -- just need to make sure the callsite merge does not happen before instrumentation. Your concern is on AutoFDO with iteration.

1. For the training build that does not use a profile, my intuition is that we will always merge the indirect callsites, which will lead to a merged !VP profile.

For initial build of the profiling binary, yes this is true.

2. Then for the next build that uses the profile, the callsites will first be promoted in separate, and then the leftover unpromoted callsites will be merged.

Depending the phase ordering of sample loading. The unpromoted callsites is not merged before sample loading, then I assume one of the callsites will be annotated with VP data (assuming that in the profile binary the merged callsite inherits one source's debug info). If that is the case, this patch will prevent those sites from being merged. And starting from this iteration, all future iterations should be fine (remain unmerged).

The promoted callsite should behave similarly as unpromoted callsite.

For instance,

// original code:

if (cond) {
    fp1->foo();   //line 3; targeting bar
} else  {
  fp2-> foo(); // line 5; targeting blah

}

// After merging in profile binary we get

fp = select cond, fp1, fp2;
fp->foo(); // line 3 targeting bar, blah

// After sampling, line 3 gets VP data 'bar', 'blah'

// In optimized build, assuming merge does not happen before sample loading, the following will happen:

 

-   fp1 gets VP data 'bar', and 'blah' and get promoted, 
-   fp2 at line 5 has no VP data
-   merging will be prevented
-   line 3 will be promoted (less precisedly)
-   line5 won't be promoted (no VP data)
-

// in the next round of profiling, both line 3 and line 5 have independent VP data, and things will start get stable from now on.

3. For all subsequent builds, the separate !VP profiles will be dominant, the leftover merged profile shouldn't be important as they represent non-typical paths.

The VP data from promoted branches will be mapped to the same line including the left over one.

For instance, the line 3 is promoted:

 if (fp1 == 'bar') 
     bar();
 else if (fp1 == 'blah')
     blah();            // no samples in this case
else
  fp1();

The resulting profile for line 3 will have 'bar' as dominating target.

The questions is, in #2, whether promoted callsites can be merged or not. I'm not quite sure but since their promotion is based on the same profile from #1, the resulting direct calls will all look the same which might be good candidates to merge. If merged, the generated profile based on this build will still lose flow-sensitivity.

Icall promotion itself will likely block any future merging.

There's also another possibility which is no callsites can be merged after ICP, because of the conditional code introduced during ICP. If this is common, then it sounds like just letting indirect callsites be merged (#1) as is isn't a deal breaker.

See above. This patch will help stablize promotion behavior after 2 iterations which is good for AFDO.

Thoughts?

[davidxl]

Note the format of my last reply is messed up a little.

[mingmingl]

In D148876#4289051, @hoy wrote:

I see. Merging could cause the loss of flow-sensitivity for PGO, but not merging doesn't give optimal code quality.

Here are my thoughts about what could happen with @mingmingl's approach that allows merging only after ICP.

1. For the training build that does not use a profile, my intuition is that we will always merge the indirect callsites, which will lead to a merged !VP profile.

For an instrumentation + thinlto build, value profiles of two different indirect callsites won't be merged

• instrumentation of target values happens before the simplify-cfg pass that enables hoist and sink.
• https://gcc.godbolt.org/z/MM976vn55 is an example -> xur@ pointed out the counter index is from %. = select i1 %cmp, i64 2, i64 1 , so each callsite is correctly instrumented (each index for one entry in the linked list>. Note simplify-cfg pass runs multiple times in a pipeline, configured with different options. SimplifyCFG pass before PGOInstrumentationGen doesn't sink or hoist as shown in the example.

For a SamplePGO + thinlto build,

1. if a function foo gets two indirect callsites c1 and c2 merged in the training build, hypothetically one debug location is kept in the merged callsite, or maybe both debug locations are dropped.
   • If both debug locations are dropped, the merged one callsite won't be annotated with value profiles.
   • If one debug location is kept, target values of both c1 and c2 are attributed to this debug location, and indirect callsite for this debug location is annotated.
2. When collected sample-pgo profiles have two value profiles in the first place, sample-pgo-use is more likely to see two indirect callsites, each annotated with different value profiles.

2. Then for the next build that uses the profile, the callsites will first be promoted in separate, and then the leftover unpromoted callsites will be merged. That merged callsite will lead to a merged !VP profile. The already promoted callsites, if not merged, will lead to separate !VP profiles.

#2 discusses what happens when value profiles are merged on two different callsites.

For Sample-PGO, when two indirect callsites are merged in the first place in the training build (where sample-pgo profiles are collected), the value profile could be associated with at most one debug location, so at most one of the two indirect callsites could be annotated.

3. For all subsequent builds, the separate !VP profiles will be dominant, the leftover merged profile shouldn't be important as they represent non-typical paths.

The questions is, in #2, whether promoted callsites can be merged or not. I'm not quite sure but since their promotion is based on the same profile from #1, the resulting direct calls will all look the same which might be good candidates to merge. If merged, the generated profile based on this build will still lose flow-sensitivity.

There's also another possibility which is no callsites can be merged after ICP, because of the conditional code introduced during ICP. If this is common, then it sounds like just letting indirect callsites be merged (#1) as is isn't a deal breaker.

It's likely that indirect callsites cannot be merged because of the introduced if-else branches; there is a trade-off to be made here (and already made when indirect calls are promoted today) -> for the significant indirect calls that show up enough number of times (per isPromotionProfitable), speculatively devirtualize it (possibly inline it in the later inliner pass) is overall good.

If extending value profiles to make it one-bit more stateful is ok (in terms of code quality, profile size, etc), it preserves the original per-callsite distribution compared with merging two different callsites.

[hoy]

Thanks for the illustration. For AutoFDO, I took a look at the debug info merger (https://github.com/llvm/llvm-project/blob/a17b71d17f853350dcd6c72ab141b196d7caec2a/llvm/lib/IR/DebugInfoMetadata.cpp#L114). Here is my understanding:

1. For two callsites that are originally from different lines of a function, the merged location will have a zero line number. This basically means the value profile for the merged callsite will be dropped by the sample loader.

2. For callsites originally from same location but ended up with different inline contexts in the caller function, the merger tries to reconcile their inline contexts by finding a common suffix of the contexts. If successful, the common suffix will be the debug location for the merged callsites. This may allow for the value profile to be shared by the two callsites (under a different inlining situation). E.g, for a callsite defined in function D at line 4, if somehow it gets inlined twice into function A:

A:1 @ B:2 @ C:3 @ D:4

A:2 @ B:3 @ C:3 @ D:4

Then my understanding is that the merged callsite will have C:3 @ D:4 as its debug string. If somehow in the next compilation A never inlines B but B inlines C twice, then the two callsites could end up sharing the same value profile.

#2 may sound uncommon, but #1 should be common. If that's the case, then we basically lose track of value profiles for merged callsites.

Please let me know if I miss anything. Despite AutoFDO, I think Mingming's solution that allows merging only after ICP should be good for Instrument PGO.

[davidxl]

In D148876#4289326, @hoy wrote:

Thanks for the illustration. For AutoFDO, I took a look at the debug info merger (https://github.com/llvm/llvm-project/blob/a17b71d17f853350dcd6c72ab141b196d7caec2a/llvm/lib/IR/DebugInfoMetadata.cpp#L114). Here is my understanding:

1. For two callsites that are originally from different lines of a function, the merged location will have a zero line number. This basically means the value profile for the merged callsite will be dropped by the sample loader.

Thanks for checking. I suspect bugs like this may exist, but it is orthogonal to the problem at hand.

2. For callsites originally from same location but ended up with different inline contexts in the caller function, the merger tries to reconcile their inline contexts by finding a common suffix of the contexts. If successful, the common suffix will be the debug location for the merged callsites. This may allow for the value profile to be shared by the two callsites (under a different inlining situation). E.g, for a callsite defined in function D at line 4, if somehow it gets inlined twice into function A:

A:1 @ B:2 @ C:3 @ D:4

A:2 @ B:3 @ C:3 @ D:4

Then my understanding is that the merged callsite will have C:3 @ D:4 as its debug string. If somehow in the next compilation A never inlines B but B inlines C twice, then the two callsites could end up sharing the same value profile.

True, but this happens today as well -- it is very unlikely such two callsites from inline instances can be merged in the first place.

#2 may sound uncommon,

yes. Besides sharing profile may still be better than dropping profile.

> but #1 should be common. If that's the case, then we basically lose track of value profiles for merged callsites.

Please let me know if I miss anything.

See my reply above.

Despite AutoFDO, I think Mingming's solution that allows merging only after ICP should be good for Instrument PGO.

[xur]

In D148876#4289423, @davidxl wrote:

In D148876#4289326, @hoy wrote:

Thanks for the illustration. For AutoFDO, I took a look at the debug info merger (https://github.com/llvm/llvm-project/blob/a17b71d17f853350dcd6c72ab141b196d7caec2a/llvm/lib/IR/DebugInfoMetadata.cpp#L114). Here is my understanding:

1. For two callsites that are originally from different lines of a function, the merged location will have a zero line number. This basically means the value profile for the merged callsite will be dropped by the sample loader.

Thanks for checking. I suspect bugs like this may exist, but it is orthogonal to the problem at hand.

I don't think this is a bug. This is the intended behavior -- if there is conflict debug info, just set to zero line number of remove the debug info.
This does create some problems for Sample FDO.

This is not a problem for pseudo-probe, right?

2. For callsites originally from same location but ended up with different inline contexts in the caller function, the merger tries to reconcile their inline contexts by finding a common suffix of the contexts. If successful, the common suffix will be the debug location for the merged callsites. This may allow for the value profile to be shared by the two callsites (under a different inlining situation). E.g, for a callsite defined in function D at line 4, if somehow it gets inlined twice into function A:

A:1 @ B:2 @ C:3 @ D:4

A:2 @ B:3 @ C:3 @ D:4

Then my understanding is that the merged callsite will have C:3 @ D:4 as its debug string. If somehow in the next compilation A never inlines B but B inlines C twice, then the two callsites could end up sharing the same value profile.

True, but this happens today as well -- it is very unlikely such two callsites from inline instances can be merged in the first place.

#2 may sound uncommon,

yes. Besides sharing profile may still be better than dropping profile.

> but #1 should be common. If that's the case, then we basically lose track of value profiles for merged callsites.

Please let me know if I miss anything.

See my reply above.

Despite AutoFDO, I think Mingming's solution that allows merging only after ICP should be good for Instrument PGO.

This does not make AutoFDO worse, right?
This seems also to benefit pseudo-probe based AutoFDO.

[hoy]

In D148876#4293319, @xur wrote:

In D148876#4289423, @davidxl wrote:

In D148876#4289326, @hoy wrote:

Thanks for the illustration. For AutoFDO, I took a look at the debug info merger (https://github.com/llvm/llvm-project/blob/a17b71d17f853350dcd6c72ab141b196d7caec2a/llvm/lib/IR/DebugInfoMetadata.cpp#L114). Here is my understanding:

1. For two callsites that are originally from different lines of a function, the merged location will have a zero line number. This basically means the value profile for the merged callsite will be dropped by the sample loader.

Thanks for checking. I suspect bugs like this may exist, but it is orthogonal to the problem at hand.

I don't think this is a bug. This is the intended behavior -- if there is conflict debug info, just set to zero line number of remove the debug info.
This does create some problems for Sample FDO.

This is not a problem for pseudo-probe, right?

Unfortunately it somehow still affects pseudo probe which uses !dbg info for callsites to store their probe information.

2. For callsites originally from same location but ended up with different inline contexts in the caller function, the merger tries to reconcile their inline contexts by finding a common suffix of the contexts. If successful, the common suffix will be the debug location for the merged callsites. This may allow for the value profile to be shared by the two callsites (under a different inlining situation). E.g, for a callsite defined in function D at line 4, if somehow it gets inlined twice into function A:

A:1 @ B:2 @ C:3 @ D:4

A:2 @ B:3 @ C:3 @ D:4

Then my understanding is that the merged callsite will have C:3 @ D:4 as its debug string. If somehow in the next compilation A never inlines B but B inlines C twice, then the two callsites could end up sharing the same value profile.

True, but this happens today as well -- it is very unlikely such two callsites from inline instances can be merged in the first place.

#2 may sound uncommon,

yes. Besides sharing profile may still be better than dropping profile.

> but #1 should be common. If that's the case, then we basically lose track of value profiles for merged callsites.

Please let me know if I miss anything.

See my reply above.

Despite AutoFDO, I think Mingming's solution that allows merging only after ICP should be good for Instrument PGO.

This does not make AutoFDO worse, right?
This seems also to benefit pseudo-probe based AutoFDO.

Right, it doesn't make AutoFDO worse. It should help pseudo probe in some cases where merging biases one callsite.

[wenlei]

In D148876#4288429, @wenlei wrote:

cc @hoy to make sure this works for csspgo.

Sorry for being vague. What I meant is to double check the pass ordering to make sure csspgo doesn't have extra use of VP after sample loader and ICP, so clearing is ok.

CGProfile is a good catch. Now if the final solution is to allow merging of VP after ICP is done, instead clearing all VP, I don't see any problem then.

[wenlei]

In D148876#4288504, @mingmingl wrote:

In D148876#4288286, @tejohnson wrote:

Would be good to explain motivation (maybe reference other patch?)

Makes sense, will do.

Can I piggyback on this to suggest a general feedback :) Reviewers would benefit from more explanation on why, in addition to what, and it applies to other patches as well. Without asking explicitly, I failed to grasp your intention of D147954 just by looking at the patch without asking questions. It somewhat applies to the profile reader changes too.

[mingmingl]

In D148876#4297923, @wenlei wrote:

In D148876#4288429, @wenlei wrote:

cc @hoy to make sure this works for csspgo.

Sorry for being vague. What I meant is to double check the pass ordering to make sure csspgo doesn't have extra use of VP after sample loader and ICP, so clearing is ok.

CGProfile is a good catch. Now if the final solution is to allow merging of VP after ICP is done, instead clearing all VP, I don't see any problem then.

Yep without the catch the clearing option would be bad. So indeed good catch!

As an update, I'll proceed to implement the idea of allow merging after ICP transformation, with slightly better implementations than extending "vp" format;

1. given allow-merging is either true or false for all value-profile instances depending on the execution stage of pipeline, use per module flag rather than per !prof boolean
2. If possible, introduce helper functions for individual transformation passes (simplifycfg or cse or whatever pass) to call, the helper function either returns 'wait, don't merge for now' (maybe fleshes out as nullptr) or the result of the merged instructions (with merged value profiles).

This way, 1) reduces repeated states to a per-module level, and 2) makes it easier to preserve metadata for other passes (not only simplify-cfg, might be early-cse pass, etc).

In D148876#4297926, @wenlei wrote:

In D148876#4288504, @mingmingl wrote:

In D148876#4288286, @tejohnson wrote:

Would be good to explain motivation (maybe reference other patch?)

Makes sense, will do.

Can I piggyback on this to suggest a general feedback :) Reviewers would benefit from more explanation on why, in addition to what, and it applies to other patches as well. Without asking explicitly, I failed to grasp your intention of D147954 just by looking at the patch without asking questions. It somewhat applies to the profile reader changes too.

Sure, I'll try to do better. My take is that natural language could be ambiguous compared with reduced godbolt examples (or diagrams to describe cfg, etc). Even with how-and-why explained, I could imagine questions are welcome rather than being frowned upon, like how existing questions inspires me to learn and find problems early.

[mingmingl]

resolve comments and absorbs D147954 to illustrate how module flag and helper functions are used.. Could split out to smaller patches as well.

• described the problem to solve in summary.
• modified existing tests for pass pipeline changes.
• (implementation) consider the possibility that an invoke-instruction could have both branch_weights'and value profiles.

[mingmingl]

llvm/include/llvm/IR/Instructions.h

4018 ↗

(On Diff #515602)

In the previous version this function was not called so should be deleted back then. Only getBranchWeightProfData was used. Now the change becomes obsolete..

llvm/lib/Transforms/Instrumentation/IndirectCallPromotion.cpp

307

Done in the sense that the code is obsolete

311

Good question. Only CallInst or InvokeInst should be relevant for function calls, since CallBrInst is [[ I see many isa<CallInst>(I) || isa<Invokeinst> in the codebase. | only used for inline assembly goto ]].

Added comment around helper function getFunctionCallInstr in llvm/lib/IR/Metadata.cpp for this.

llvm/include/llvm/IR/Metadata.h

 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/DenseMapInfo.h"
 #include "llvm/ADT/PointerUnion.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/ilist_node.h"
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/IR/Constant.h"
+#include "llvm/IR/Constants.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/Value.h"
 #include "llvm/Support/CBindingWrapping.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
 #include <iterator>
 #include <memory>
 #include <string>
 #include <type_traits>
+#include <unordered_map>
 #include <utility>
 
 namespace llvm {
 
 class Module;
 class ModuleSlotTracker;
 class raw_ostream;
 template <typename T> class StringMapEntry;
private:
   static void dispatchRecalculateHash(NodeTy *, std::false_type) {}
   template <class NodeTy>
   static void dispatchResetHash(NodeTy *N, std::true_type) {
     N->setHash(0);
   }
   template <class NodeTy>
   static void dispatchResetHash(NodeTy *, std::false_type) {}
 
+  // Given a value profile Node, gets the TotalCount and the list of
+  // <target-value, counter> pairs. Returns false if indirect-call
+  // sites value profile are not found.
+  static bool getIndirectCallValueProfileMetadata(
+      const MDNode *Node, uint64_t &Sum,
+      std::unordered_map<uint64_t, uint64_t> &ValCntMap);
+
+  // Similar to the function above but takes an instruction and !prof
+  // rather than known value profile.
+  // This is mostly a wrapper around the function above, and extracts
+  // value profile from invoke instructions.
+  static bool getIndirectCallValueProfileMetadata(
+      const Instruction *Inst, const MDNode *Node, uint64_t &Sum,
+      std::unordered_map<uint64_t, uint64_t> &ValCntMap);
+
+  /// Merge two indirect callsites value profiles by
+  /// 1) adding total count and each target value's counters
+  /// 2) sort <target-value, counter> pairs based on counter value descendingly.
+  static MDNode *getMergedIndirectCallProfMetadata(MDNode *A, MDNode *B,
+                                                   const Instruction *AInstr,
+                                                   const Instruction *BInstr);
+
   /// Merge branch weights from two direct callsites.
   static MDNode *mergeDirectCallProfMetadata(MDNode *A, MDNode *B,
                                              const Instruction *AInstr,
                                              const Instruction *BInstr);
 
 public:
   using op_iterator = const MDOperand *;
   using op_range = iterator_range<op_iterator>;
#include "llvm/IR/Metadata.def"
   /// Methods for metadata merging.
   static MDNode *concatenate(MDNode *A, MDNode *B);
   static MDNode *intersect(MDNode *A, MDNode *B);
   static MDNode *getMostGenericTBAA(MDNode *A, MDNode *B);
   static MDNode *getMostGenericFPMath(MDNode *A, MDNode *B);
   static MDNode *getMostGenericRange(MDNode *A, MDNode *B);
   static MDNode *getMostGenericAliasScope(MDNode *A, MDNode *B);
   static MDNode *getMostGenericAlignmentOrDereferenceable(MDNode *A, MDNode *B);
+
+  // Helper function to decide if it's profitable to merge indirect call profile
+  // metadata.
+  static bool isProfitableToMergeICallProfMetadata(MDNode *A, MDNode *B,
+                                                   const Module &M);
   /// Merge !prof metadata from two instructions.
   /// Currently only implemented with direct callsites with branch weights.
   static MDNode *getMergedProfMetadata(MDNode *A, MDNode *B,
                                        const Instruction *AInstr,
                                        const Instruction *BInstr);
 };
 
 /// Tuple of metadata.

llvm/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TRANSFORMS_INSTRUMENTATION_PGOINSTRUMENTATION_H
 #define LLVM_TRANSFORMS_INSTRUMENTATION_PGOINSTRUMENTATION_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/IR/PassManager.h"
+#include "llvm/Pass.h"
 #include <cstdint>
 #include <string>
 
 namespace llvm {
 
 class Function;
 class Instruction;
 class Module;
private:
   // If this is a context sensitive instrumentation.
   bool IsCS;
   IntrusiveRefCntPtr<vfs::FileSystem> FS;
 };
 
 /// The indirect function call promotion pass.
 class PGOIndirectCallPromotion : public PassInfoMixin<PGOIndirectCallPromotion> {
 public:
-  PGOIndirectCallPromotion(bool IsInLTO = false, bool SamplePGO = false)
-      : InLTO(IsInLTO), SamplePGO(SamplePGO) {}
+  PGOIndirectCallPromotion(
+      ThinOrFullLTOPhase Phase = ThinOrFullLTOPhase::ThinLTOPreLink,
+      bool SamplePGO = false)
+      : Phase(Phase), SamplePGO(SamplePGO) {}
 
   PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
 
 private:
-  bool InLTO;
+  ThinOrFullLTOPhase Phase;
   bool SamplePGO;
 };
 
 /// The profile size based optimization pass for memory intrinsics.
 class PGOMemOPSizeOpt : public PassInfoMixin<PGOMemOPSizeOpt> {
 public:
   PGOMemOPSizeOpt() = default;
 

llvm/lib/IR/Metadata.cpp

 #include "llvm/Support/Casting.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/MathExtras.h"
 #include <algorithm>
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
 #include <type_traits>
+#include <unordered_map>
 #include <utility>
 #include <vector>
 
 using namespace llvm;
 
 MetadataAsValue::MetadataAsValue(Type *Ty, Metadata *MD)
     : Value(Ty, MetadataAsValueVal), MD(MD) {
   track();
MDNode *MDNode::getMostGenericFPMath(MDNode *A, MDNode *B) {
 
   APFloat AVal = mdconst::extract<ConstantFP>(A->getOperand(0))->getValueAPF();
   APFloat BVal = mdconst::extract<ConstantFP>(B->getOperand(0))->getValueAPF();
   if (AVal < BVal)
     return A;
   return B;
 }
 
+bool MDNode::isProfitableToMergeICallProfMetadata(MDNode *A, MDNode *B,
+                                                  const Module &M) {
+  const ConstantInt *MF = mdconst::dyn_extract_or_null<ConstantInt>(
+      M.getModuleFlag("IndirectCallPromComplete"));
+
+  // Before PGOIndirectCallPromotion pass completes, merge value profiles
+  // only if at most indirect callsites have value profile metadata.
+  if (!MF || MF->isZero())
+    return !(A && B);
+
+  // If PGOIndirectCallPromotion pass completes, merging value profiles
+  // to preserve <target-value, counter> pairs for CGProfile pass.
+  // It's fine to sum the total count from two !prof and change distribution
+  // at this point since no other pass other than PGOIndirectCallPromotion
+  // uses total count.
+  return true;
+}
+
+bool MDNode::getIndirectCallValueProfileMetadata(
+    const Instruction *Inst, const MDNode *Node, uint64_t &Sum,
+    std::unordered_map<uint64_t, uint64_t> &ValCntMap) {
+  const unsigned NOps = Node->getNumOperands();
+  if (NOps < 3)
+    return false;
+
+  MDString *Tag = cast<MDString>(Node->getOperand(0));
+  if (Tag && Tag->getString().equals("VP"))
+    return getIndirectCallValueProfileMetadata(Node, Sum, ValCntMap);
+
+  auto isMDwithName = [](const MDNode *MD, const char *Str) {
+    auto *MDName = dyn_cast<MDString>(MD->getOperand(0));
+    if (!MDName)
+      return false;
+    return (MDName->getString().equals(Str));
+  };
+  if (NOps == 3 && isa<InvokeInst>(Inst)) {
+    for (unsigned I = 1; I < 3; I++) {
+      const MDNode *OperandNode = dyn_cast<const MDNode>(Node->getOperand(I));
+      if (OperandNode && isMDwithName(OperandNode, "VP"))
+        return getIndirectCallValueProfileMetadata(OperandNode, Sum, ValCntMap);
+    }
+  }
+  return false;
+}
+
+bool MDNode::getIndirectCallValueProfileMetadata(
+    const MDNode *Node, uint64_t &Sum,
+    std::unordered_map<uint64_t, uint64_t> &ValCntMap) {
+  const unsigned NOps = Node->getNumOperands();
+  if (NOps < 3)
+    return false;
+
+  // Operand 0 is the string "VP"
+  MDString *Tag = cast<MDString>(Node->getOperand(0));
+  if (!Tag || !Tag->getString().equals("VP"))
+    return false;
+
+  // Operand 1 is the value profile kind.
+  ConstantInt *KindInt = mdconst::dyn_extract<ConstantInt>(Node->getOperand(1));
+  if (!KindInt || !KindInt->getZExtValue() == 0)
+    return false;
+
+  ConstantInt *TotalCnt =
+      mdconst::dyn_extract<ConstantInt>(Node->getOperand(2));
+  if (!TotalCnt)
+    return false;
+
+  Sum = TotalCnt->getZExtValue();
+
+  for (unsigned I = 3; I < NOps; I += 2) {
+    ConstantInt *TargetValue =
+        mdconst::dyn_extract<ConstantInt>(Node->getOperand(I));
+    ConstantInt *Count =
+        mdconst::dyn_extract<ConstantInt>(Node->getOperand(I + 1));
+
+    if (!TargetValue || !Count)
+      return false;
+
+    const uint64_t ValInt = TargetValue->getZExtValue();
+
+    ValCntMap[ValInt] = SaturatingAdd(ValCntMap[ValInt], Count->getZExtValue());
+  }
+
+  return true;
+}
+
+MDNode *MDNode::getMergedIndirectCallProfMetadata(MDNode *A, MDNode *B,
+                                                  const Instruction *AInstr,
+                                                  const Instruction *BInstr) {
+  assert(A && B && "Caller guaranteed");
+  uint64_t ASum = 0;
+  uint64_t BSum = 0;
+  std::unordered_map<uint64_t, uint64_t> AValCntMap, BValCntMap;
+
+  // If one metadata is malformed, conservatively returns the other for now.
+  // FIXME: Note this shouldn't happen and IR verifier should verify the format
+  // of value profiles instead.
+  if (!getIndirectCallValueProfileMetadata(BInstr, B, BSum, BValCntMap))
+    return A;
+  if (!getIndirectCallValueProfileMetadata(BInstr, A, ASum, AValCntMap))
+    return B;
+
+  for (const auto &[Val, Cnt] : AValCntMap) {
+    BValCntMap[Val] = SaturatingAdd(Cnt, BValCntMap[Val]);
+  }
+
+  using ValCntPair = std::pair<uint64_t, uint64_t>;
+
+  SmallVector<ValCntPair, 12> MergedVals;
+  for (const auto &[Val, Cnt] : BValCntMap) {
+    MergedVals.push_back(std::make_pair(Val, Cnt));
+  }
+
+  // Sort target values by counters descendingly.
+  llvm::sort(MergedVals,
+             [](const ValCntPair &LHS, const ValCntPair &RHS) -> bool {
+               return LHS.second > RHS.second;
+             });
+
+  auto &Ctx = AInstr->getContext();
+  MDBuilder MDHelper(Ctx);
+  SmallVector<Metadata *, 8> Vals;
+  Vals.push_back(MDHelper.createString("VP"));
+  Vals.push_back(MDHelper.createConstant(
+      ConstantInt::get(Type::getInt32Ty(Ctx), 0 /* ValueKind */)));
+  Vals.push_back(MDHelper.createConstant(
+      ConstantInt::get(Type::getInt64Ty(Ctx), SaturatingAdd(ASum, BSum))));
+  for (const auto &[Val, Cnt] : MergedVals) {
+    Vals.push_back(
+        MDHelper.createConstant(ConstantInt::get(Type::getInt64Ty(Ctx), Val)));
+    Vals.push_back(
+        MDHelper.createConstant(ConstantInt::get(Type::getInt64Ty(Ctx), Cnt)));
+  }
+  return MDNode::get(Ctx, Vals);
+}
+
 // Call instructions with branch weights are only used in SamplePGO as
 // documented in
 /// https://llvm.org/docs/BranchWeightMetadata.html#callinst).
 MDNode *MDNode::mergeDirectCallProfMetadata(MDNode *A, MDNode *B,
                                             const Instruction *AInstr,
                                             const Instruction *BInstr) {
   assert(A && B && AInstr && BInstr && "Caller should guarantee");
   auto &Ctx = AInstr->getContext();
if (!(A && B)) {
     return A ? A : B;
   }
 
   assert(AInstr->getMetadata(LLVMContext::MD_prof) == A &&
          "Caller should guarantee");
   assert(BInstr->getMetadata(LLVMContext::MD_prof) == B &&
          "Caller should guarantee");
 
-  const CallInst *ACall = dyn_cast<CallInst>(AInstr);
-  const CallInst *BCall = dyn_cast<CallInst>(BInstr);
+  // Note a CallBase could be CallInst (e.g., direct or virtual member calls),
+  // InvokeInst (e.g., virtual destructors) or CallBrInst (inline assembly goto
+  // that are not relevant here).
+  auto getFunctionCallInstr = [](const Instruction *Instr) -> const CallBase * {
+    if (isa<CallInst>(Instr) || isa<InvokeInst>(Instr))
+      return dyn_cast<const CallBase>(Instr);
+    return nullptr;
+  };
+
+  const CallBase *ACall = getFunctionCallInstr(AInstr);
+  const CallBase *BCall = getFunctionCallInstr(BInstr);
 
   // Both ACall and BCall are direct callsites.
   if (ACall && BCall && ACall->getCalledFunction() &&
       BCall->getCalledFunction())
     return mergeDirectCallProfMetadata(A, B, AInstr, BInstr);
 
+  if (ACall && BCall && ACall->isIndirectCall() && BCall->isIndirectCall()) {
+    const Module *M = ACall->getModule();
+    assert(M == BCall->getModule() &&
+           "ACall and BCall should come from the same module");
+    if (!isProfitableToMergeICallProfMetadata(A, B, *M))
+      return A ? A : B;
+    return getMergedIndirectCallProfMetadata(A, B, AInstr, BInstr);
+  }
+
   // The rest of the cases are not implemented but could be added
   // when there are use cases.
   return nullptr;
 }
 
 static bool isContiguous(const ConstantRange &A, const ConstantRange &B) {
   return A.getUpper() == B.getLower() || A.getLower() == B.getUpper();
 }

llvm/lib/Passes/PassBuilderPipelines.cpp

PassBuilder::buildModuleSimplificationPipeline(OptimizationLevel Level,
   // passed to the compile) to the SamplePGO flag of ICP. This is used to
   // determine whether the new direct calls are annotated with prof metadata.
   // Ideally this should be determined from whether the IR is annotated with
   // sample profile, and not whether the a sample profile was provided on the
   // command line. E.g. for flattened profiles where we will not be reloading
   // the sample profile in the ThinLTO backend, we ideally shouldn't have to
   // provide the sample profile file.
   if (Phase == ThinOrFullLTOPhase::ThinLTOPostLink && !LoadSampleProfile)
-    MPM.addPass(PGOIndirectCallPromotion(true /* InLTO */, HasSampleProfile));
+    MPM.addPass(PGOIndirectCallPromotion(Phase, HasSampleProfile));
 
   // Do basic inference of function attributes from known properties of system
   // libraries and other oracles.
   MPM.addPass(InferFunctionAttrsPass());
   MPM.addPass(CoroEarlyPass());
 
   // Create an early function pass manager to cleanup the output of the
   // frontend.
if (LoadSampleProfile) {
     MPM.addPass(RequireAnalysisPass<ProfileSummaryAnalysis, Module>());
     // Do not invoke ICP in the LTOPrelink phase as it makes it hard
     // for the profile annotation to be accurate in the LTO backend.
     if (!isLTOPreLink(Phase))
       // We perform early indirect call promotion here, before globalopt.
       // This is important for the ThinLTO backend phase because otherwise
       // imported available_externally functions look unreferenced and are
       // removed.
-      MPM.addPass(
-          PGOIndirectCallPromotion(true /* IsInLTO */, true /* SamplePGO */));
+      MPM.addPass(PGOIndirectCallPromotion(Phase, true /* SamplePGO */));
   }
 
   // Try to perform OpenMP specific optimizations on the module. This is a
   // (quick!) no-op if there are no OpenMP runtime calls present in the module.
   MPM.addPass(OpenMPOptPass());
 
   if (AttributorRun & AttributorRunOption::MODULE)
     MPM.addPass(AttributorPass());
PassBuilder::buildModuleSimplificationPipeline(OptimizationLevel Level,
   // Add all the requested passes for instrumentation PGO, if requested.
   if (PGOOpt && Phase != ThinOrFullLTOPhase::ThinLTOPostLink &&
       (PGOOpt->Action == PGOOptions::IRInstr ||
        PGOOpt->Action == PGOOptions::IRUse)) {
     addPGOInstrPasses(MPM, Level,
                       /* RunProfileGen */ PGOOpt->Action == PGOOptions::IRInstr,
                       /* IsCS */ false, PGOOpt->ProfileFile,
                       PGOOpt->ProfileRemappingFile, Phase, PGOOpt->FS);
-    MPM.addPass(PGOIndirectCallPromotion(false, false));
+    MPM.addPass(PGOIndirectCallPromotion(Phase, false /* SamplePGO */));
   }
   if (PGOOpt && Phase != ThinOrFullLTOPhase::ThinLTOPostLink &&
       PGOOpt->CSAction == PGOOptions::CSIRInstr)
     MPM.addPass(PGOInstrumentationGenCreateVar(PGOOpt->CSProfileGenFile));
 
   // Synthesize function entry counts for non-PGO compilation.
   if (EnableSyntheticCounts && !PGOOpt)
     MPM.addPass(SyntheticCountsPropagation());
PassBuilder::buildLTODefaultPipeline(OptimizationLevel Level,
   if (Level.getSpeedupLevel() > 1) {
     MPM.addPass(createModuleToFunctionPassAdaptor(
         CallSiteSplittingPass(), PTO.EagerlyInvalidateAnalyses));
 
     // Indirect call promotion. This should promote all the targets that are
     // left by the earlier promotion pass that promotes intra-module targets.
     // This two-step promotion is to save the compile time. For LTO, it should
     // produce the same result as if we only do promotion here.
-    MPM.addPass(PGOIndirectCallPromotion(
-        true /* InLTO */, PGOOpt && PGOOpt->Action == PGOOptions::SampleUse));
+    MPM.addPass(PGOIndirectCallPromotion(ThinOrFullLTOPhase::FullLTOPostLink,
+                                         PGOOpt && PGOOpt->Action ==
+                                                       PGOOptions::SampleUse));
 
     // Propagate constants at call sites into the functions they call.  This
     // opens opportunities for globalopt (and inlining) by substituting function
     // pointers passed as arguments to direct uses of functions.
     MPM.addPass(IPSCCPPass(IPSCCPOptions(/*AllowFuncSpec=*/
                                          Level != OptimizationLevel::Os &&
                                          Level != OptimizationLevel::Oz)));
 

llvm/lib/Transforms/Instrumentation/IndirectCallPromotion.cpp

 //===- IndirectCallPromotion.cpp - Optimizations based on value profiling -===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the transformation that promotes indirect calls to
 // conditional direct calls when the indirect-call value profile metadata is
 // available.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Analysis/IndirectCallPromotionAnalysis.h"
 #include "llvm/Analysis/IndirectCallVisitor.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/Function.h"
static cl::opt<bool> ICPInvokeOnly("icp-invoke-only", cl::init(false),
                                             "invoke instruction only"));
 
 // Dump the function level IR if the transformation happened in this
 // function. For debug use only.
 static cl::opt<bool>
     ICPDUMPAFTER("icp-dumpafter", cl::init(false), cl::Hidden,
                  cl::desc("Dump IR after transformation happens"));
 
+static cl::opt<bool> MarkIndirectCallPromotionComplete(
+    "mark-icp-complete", cl::init(false), cl::Hidden,
+    cl::desc("If true, override the IndirectCallPromotionCompleteAfterPass. "
+             "For testing and debugging purpose only"));
+
 namespace {
 
 // The class for main data structure to promote indirect calls to conditional
 // direct calls.
 class ICallPromotionFunc {
 private:
   Function &F;
   Module *M;
 
   // Symtab that maps indirect call profile values to function names and
   // defines.
   InstrProfSymtab *Symtab;
 
   bool SamplePGO;
+  bool MarkIndirectCallsPromoted = false;
 
   OptimizationRemarkEmitter &ORE;
 
   // A struct that records the direct target and it's call count.
   struct PromotionCandidate {
     Function *TargetFunction;
     uint64_t Count;
 
uint32_t ICallPromotionFunc::tryToPromote(
   }
   return NumPromoted;
 }
 
 // Traverse all the indirect-call callsite and get the value profile
 // annotation to perform indirect-call promotion.
 bool ICallPromotionFunc::processFunction(ProfileSummaryInfo *PSI) {
   bool Changed = false;
   ICallPromotionAnalysis ICallAnalysis;
   for (auto *CB : findIndirectCalls(F)) {
     uint32_t NumVals, NumCandidates;
     uint64_t TotalCount;
     auto ICallProfDataRef = ICallAnalysis.getPromotionCandidatesForInstruction(
         CB, NumVals, TotalCount, NumCandidates);
     if (!NumCandidates ||
         (PSI && PSI->hasProfileSummary() && !PSI->isHotCount(TotalCount)))
       continue;
     auto PromotionCandidates = getPromotionCandidatesForCallSite(
         *CB, ICallProfDataRef, TotalCount, NumCandidates);
     uint32_t NumPromoted = tryToPromote(*CB, PromotionCandidates, TotalCount);
     if (NumPromoted == 0)
       continue;
 
     Changed = true;
     // Adjust the MD.prof metadata. First delete the old one.
     CB->setMetadata(LLVMContext::MD_prof, nullptr);
     // If all promoted, we don't need the MD.prof metadata.
     if (TotalCount == 0 || NumPromoted == NumVals)
       continue;
     // Otherwise we need update with the un-promoted records back.
     annotateValueSite(*M, *CB, ICallProfDataRef.slice(NumPromoted), TotalCount,
                       IPVK_IndirectCallTarget, NumCandidates);
   }
   return Changed;
 }
 
 // A wrapper function that does the actual work.
-static bool promoteIndirectCalls(Module &M, ProfileSummaryInfo *PSI,
-                                 bool InLTO, bool SamplePGO,
+static bool promoteIndirectCalls(Module &M, ProfileSummaryInfo *PSI, bool InLTO,
+                                 bool SamplePGO,
                                  ModuleAnalysisManager *AM = nullptr) {
   if (DisableICP)
     return false;
   InstrProfSymtab Symtab;
   if (Error E = Symtab.create(M, InLTO)) {
     std::string SymtabFailure = toString(std::move(E));
     M.getContext().emitError("Failed to create symtab: " + SymtabFailure);
     return false;
static bool promoteIndirectCalls(Module &M, ProfileSummaryInfo *PSI, bool InLTO,
   }
   return Changed;
 }
 
 PreservedAnalyses PGOIndirectCallPromotion::run(Module &M,
                                                 ModuleAnalysisManager &AM) {
   ProfileSummaryInfo *PSI = &AM.getResult<ProfileSummaryAnalysis>(M);
 
+  const bool InLTO = (Phase == ThinOrFullLTOPhase::ThinLTOPostLink ||
+                      Phase == ThinOrFullLTOPhase::FullLTOPostLink);
+
+  // If this ICP pass runs in a non-LTO pipeline or postlink pipeline of LTO
+  // build, it promotes all eligible indirect calls. After that it's performance
+  // neutral to merge indirect call value profiles.
+  bool IndirectCallPromotionCompleteAfterPass =
+      (Phase != ThinOrFullLTOPhase::ThinLTOPreLink &&
+       Phase != ThinOrFullLTOPhase::FullLTOPreLink);
+  if (MarkIndirectCallPromotionComplete.getNumOccurrences())
+    IndirectCallPromotionCompleteAfterPass = MarkIndirectCallPromotionComplete;
+
+  // Set module flag before pass completes. It's fine to set this flag at the
+  // beginning but then the name 'IndirectCallPromComplete' is confusing.
+  auto SetModuleFlagOnExit = llvm::make_scope_exit([&]() {
+    if (IndirectCallPromotionCompleteAfterPass)
+      M.setModuleFlag(Module::Error, "IndirectCallPromComplete",
+                      ConstantAsMetadata::get(ConstantInt::get(
+                          Type::getInt1Ty(M.getContext()), 1)));
+  });
+
   if (!promoteIndirectCalls(M, PSI, InLTO | ICPLTOMode,
                             SamplePGO | ICPSamplePGOMode, &AM))
     return PreservedAnalyses::all();
 
   return PreservedAnalyses::none();
 }

llvm/lib/Transforms/Utils/SimplifyCFG.cpp

 }
 
 static bool passingValueIsAlwaysUndefined(Value *V, Instruction *I, bool PtrValueMayBeModified = false);
 
 /// Helper function for HoistThenElseCodeToIf. Return true if identical
 /// instructions \p I1 and \p I2 can and should be hoisted.
 static bool shouldHoistCommonInstructions(Instruction *I1, Instruction *I2,
                                           const TargetTransformInfo &TTI) {
+
+  auto *C1 = dyn_cast<CallInst>(I1);
+  auto *C2 = dyn_cast<CallInst>(I2);
+  if (C1 && C2) {
     // If we're going to hoist a call, make sure that the two instructions
     // we're commoning/hoisting are both marked with musttail, or neither of
     // them is marked as such. Otherwise, we might end up in a situation where
     // we hoist from a block where the terminator is a `ret` to a block where
     // the terminator is a `br`, and `musttail` calls expect to be followed by
     // a return.
-  auto *C1 = dyn_cast<CallInst>(I1);
-  auto *C2 = dyn_cast<CallInst>(I2);
-  if (C1 && C2)
     if (C1->isMustTailCall() != C2->isMustTailCall())
       return false;
+    // Do not hoist two indirect calls which requires merging value profile
+    // metadata when it's not profitable to do so in a pass pipeline.
+    if (C1->isIndirectCall() && C2->isIndirectCall() &&
+        !MDNode::isProfitableToMergeICallProfMetadata(
+            C1->getMetadata(LLVMContext::MD_prof),
+            C2->getMetadata(LLVMContext::MD_prof), *C1->getModule()))
+      return false;
+  }
 
   if (!TTI.isProfitableToHoist(I1) || !TTI.isProfitableToHoist(I2))
     return false;
 
   // If any of the two call sites has nomerge or convergent attribute, stop
   // hoisting.
   if (const auto *CB1 = dyn_cast<CallBase>(I1))
     if (CB1->cannotMerge() || CB1->isConvergent())
if (isa<CallBase>(I0)) {
     auto IsIndirectCall = [](const Instruction *I) {
       return cast<CallBase>(I)->isIndirectCall();
     };
     bool HaveIndirectCalls = any_of(Insts, IsIndirectCall);
     bool AllCallsAreIndirect = all_of(Insts, IsIndirectCall);
     if (HaveIndirectCalls) {
       if (!AllCallsAreIndirect)
         return false;
+
+      const Module &M = *I0->getModule();
+      MDNode *I0Prof = I0->getMetadata(LLVMContext::MD_prof);
+      for (unsigned I = 1; I < Insts.size(); I++) {
+        if (!MDNode::isProfitableToMergeICallProfMetadata(
+                I0Prof, Insts[I]->getMetadata(LLVMContext::MD_prof), M))
+          return false;
+      }
     } else {
       // All callees must be identical.
       Value *Callee = nullptr;
       for (const Instruction *I : Insts) {
         Value *CurrCallee = cast<CallBase>(I)->getCalledOperand();
         if (!Callee)
           Callee = CurrCallee;
         else if (Callee != CurrCallee)

llvm/test/Other/new-pm-lto-defaults.ll

loop:
   %iv = phi i32 [ 0, %entry ], [ %iv.next, %loop ]
   %iv.next = add i32 %iv, 1
   tail call void @bar()
   %cmp = icmp eq i32 %iv, %n
   br i1 %cmp, label %exit, label %loop
 exit:
   ret void
 }
+
+; CHECK-O23SZ: !llvm.module.flags = !{[[ICP:![0-9]+]]}
+
+; CHECK-O23SZ: [[ICP]] = !{i32 1, !"IndirectCallPromComplete", i1 true}

llvm/test/Other/new-pm-pgo.ll

 ; RUN: opt -debug-pass-manager -passes='default<O2>' -pgo-kind=pgo-instr-gen-pipeline -profile-file='temp' %s 2>&1 |FileCheck %s --check-prefixes=GEN
 ; RUN: llvm-profdata merge %S/Inputs/new-pm-pgo.proftext -o %t.profdata
-; RUN: opt -debug-pass-manager -passes='default<O2>' -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' %s 2>&1 |FileCheck %s --check-prefixes=USE
-; RUN: opt -debug-pass-manager -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' %s 2>&1 |FileCheck %s --check-prefixes=USE
-; RUN: opt -debug-pass-manager -passes='thinlto<O2>' -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' %s 2>&1 |FileCheck %s --check-prefixes=USE_POST_LINK
-; RUN: opt -debug-pass-manager -passes='default<O2>' -hot-cold-split -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' %s 2>&1 |FileCheck %s --check-prefixes=USE --check-prefixes=SPLIT
-; RUN: opt -debug-pass-manager -passes='default<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-pgo.prof' %s 2>&1 \
-; RUN:     |FileCheck %s --check-prefixes=SAMPLE_USE,SAMPLE_USE_O
-; RUN: opt -debug-pass-manager -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-pgo.prof' %s 2>&1 \
-; RUN:     |FileCheck %s --check-prefixes=SAMPLE_USE,SAMPLE_USE_PRE_LINK
-; RUN: opt -debug-pass-manager -passes='thinlto<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-pgo.prof' %s 2>&1 \
-; RUN:     |FileCheck %s --check-prefixes=SAMPLE_USE,SAMPLE_USE_POST_LINK
+; RUN: opt -debug-pass-manager -passes='default<O2>' -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' -S %s 2>&1 |FileCheck %s --check-prefixes=USE,MOD_PS_ICP
+; RUN: opt -debug-pass-manager -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' -S %s 2>&1 |FileCheck %s --check-prefixes=USE,MOD_PS
+; RUN: opt -debug-pass-manager -passes='thinlto<O2>' -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' -S %s 2>&1 |FileCheck %s --check-prefixes=USE_POST_LINK,MOD_ICP
+; RUN: opt -debug-pass-manager -passes='default<O2>' -hot-cold-split -pgo-kind=pgo-instr-use-pipeline -profile-file='%t.profdata' -S %s 2>&1 |FileCheck %s --check-prefixes=USE,SPLIT,MOD_PS_ICP
+; RUN: opt -debug-pass-manager -passes='default<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-pgo.prof' -S %s 2>&1 \
+; RUN:     |FileCheck %s --check-prefixes=SAMPLE_USE,SAMPLE_USE_O,MOD_PS_ICP
+; RUN: opt -debug-pass-manager -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-pgo.prof' -S %s 2>&1 \
+; RUN:     |FileCheck %s --check-prefixes=SAMPLE_USE,SAMPLE_USE_PRE_LINK,MOD_PS
+; RUN: opt -debug-pass-manager -passes='thinlto<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file='%S/Inputs/new-pm-pgo.prof' -S %s 2>&1 \
+; RUN:     |FileCheck %s --check-prefixes=SAMPLE_USE,SAMPLE_USE_POST_LINK,MOD_PS_ICP
 ; RUN: opt -debug-pass-manager -passes='default<O2>' -debug-info-for-profiling %s 2>&1 |FileCheck %s --check-prefixes=SAMPLE_GEN
 ;
 ; GEN: Running pass: PGOInstrumentationGen
 ; USE: Running pass: PGOInstrumentationUse
 ; USE: Running pass: PGOIndirectCallPromotion
 ; USE: Running pass: PGOMemOPSizeOpt
 ; USE_POST_LINK: Running pass: PGOIndirectCallPromotion
 ; USE_POST_LINK: Running pass: PGOMemOPSizeOpt

 ; SAMPLE_USE_POST_LINK: Running pass: PGOIndirectCallPromotion
 ; SAMPLE_USE_PRE_LINK-NOT: Running pass: PGOIndirectCallPromotion
 ; SAMPLE_GEN: Running pass: AddDiscriminatorsPass
 ; SPLIT: Running pass: HotColdSplittingPass
 
 define void @foo() {
   ret void
 }
+
+
+; MOD_PS_ICP: !llvm.module.flags = !{[[PSMOD:![0-9]+]], [[ICPMOD:![0-9]+]]}
+; MOD_PS_ICP: [[PSMOD]] = !{i32 1, !"ProfileSummary", !1}
+; MOD_PS_ICP: [[ICPMOD]] = !{i32 1, !"IndirectCallPromComplete", i1 true}
+
+; MOD_PS: !llvm.module.flags = !{[[PSMOD:![0-9]+]]}
+; MOD_PS: [[PSMOD]] = !{i32 1, !"ProfileSummary", !1}
+
+; MOD_ICP: !llvm.module.flags = !{[[ICPMOD:![0-9]+]]}
+; MOD_ICP: [[ICPMOD]] = !{i32 1, !"IndirectCallPromComplete", i1 true}

llvm/test/Other/new-pm-thinlto-postlink-defaults.ll

loop:
   %iv = phi i32 [ 0, %entry ], [ %iv.next, %loop ]
   %iv.next = add i32 %iv, 1
   tail call void @bar()
   %cmp = icmp eq i32 %iv, %n
   br i1 %cmp, label %exit, label %loop
 exit:
   ret void
 }
+
+; CHECK-O: !llvm.module.flags = !{[[ICP:![0-9]+]]}
+
+; CHECK-0: [[ICP]] = !{i32 1, !"IndirectCallPromComplete", i1 true}

llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll

loop:
   %cmp = icmp eq i32 %iv, %n
   br i1 %cmp, label %exit, label %loop
 exit:
   ret void
 }
 
 !llvm.module.flags = !{!0}
 
+; CHECK-O: !llvm.module.flags = !{[[PS:![0-9]+]], [[ICP:![0-9]+]]}
+
 !0 = !{i32 1, !"ProfileSummary", !1}
 !1 = !{!2, !3, !4, !5, !6, !7, !8, !9}
 !2 = !{!"ProfileFormat", !"InstrProf"}
 !3 = !{!"TotalCount", i64 0}
 !4 = !{!"MaxCount", i64 0}
 !5 = !{!"MaxInternalCount", i64 0}
 !6 = !{!"MaxFunctionCount", i64 0}
 !7 = !{!"NumCounts", i64 0}

 !19 = !{i32 800000, i64 0, i32 0}
 !20 = !{i32 900000, i64 0, i32 0}
 !21 = !{i32 950000, i64 0, i32 0}
 !22 = !{i32 990000, i64 0, i32 0}
 !23 = !{i32 999000, i64 0, i32 0}
 !24 = !{i32 999900, i64 0, i32 0}
 !25 = !{i32 999990, i64 0, i32 0}
 !26 = !{i32 999999, i64 0, i32 0}
+; CHECK-O: [[ICP]] = !{i32 1, !"IndirectCallPromComplete", i1 true}

llvm/test/Other/new-pm-thinlto-postlink-samplepgo-defaults.ll

loop:
   %iv = phi i32 [ 0, %entry ], [ %iv.next, %loop ]
   %iv.next = add i32 %iv, 1
   tail call void @bar()
   %cmp = icmp eq i32 %iv, %n
   br i1 %cmp, label %exit, label %loop
 exit:
   ret void
 }
+
+; CHECK-O: !llvm.module.flags = !{[[PS:![0-9]+]], [[ICP:![0-9]+]]}
+
+; CHECK-O: [[PS]] = !{i32 1, !"ProfileSummary", !1}
+; CHECK-O: [[ICP]] = !{i32 1, !"IndirectCallPromComplete", i1 true}

llvm/test/Other/new-pm-thinlto-prelink-defaults.ll

loop:
   %iv = phi i32 [ 0, %entry ], [ %iv.next, %loop ]
   %iv.next = add i32 %iv, 1
   tail call void @bar()
   %cmp = icmp eq i32 %iv, %n
   br i1 %cmp, label %exit, label %loop
 exit:
   ret void
 }
+
+; CHECK-O-NOT: !llvm.module.flags

llvm/test/Other/new-pm-thinlto-prelink-pgo-defaults.ll

loop:
   %iv = phi i32 [ 0, %entry ], [ %iv.next, %loop ]
   %iv.next = add i32 %iv, 1
   tail call void @bar()
   %cmp = icmp eq i32 %iv, %n
   br i1 %cmp, label %exit, label %loop
 exit:
   ret void
 }
+
+; CHECK-O: !llvm.module.flags = !{[[PS:![0-9]+]]}
+
+; CHECK-O: [[PS]] = !{i32 1, !"ProfileSummary", !1}

llvm/test/Other/new-pm-thinlto-prelink-samplepgo-defaults.ll

loop:
   %iv = phi i32 [ 0, %entry ], [ %iv.next, %loop ]
   %iv.next = add i32 %iv, 1
   tail call void @bar()
   %cmp = icmp eq i32 %iv, %n
   br i1 %cmp, label %exit, label %loop
 exit:
   ret void
 }
+
+; CHECK-O: !llvm.module.flags = !{[[PS:![0-9]+]]}
+
+; CHECK-O: [[PS]] = !{i32 1, !"ProfileSummary", !1}

llvm/test/Transforms/PGOProfile/indirect_call_promotion.ll

-; RUN: opt < %s -passes=pgo-icall-prom -S -icp-total-percent-threshold=50 | FileCheck %s --check-prefix=ICALL-PROM
+; RUN: opt < %s -passes=pgo-icall-prom -S -icp-total-percent-threshold=50 | FileCheck %s --check-prefixes=ICALL-PROM,ICALL-PROM-NO-MODU
+; RUN: opt < %s -passes=pgo-icall-prom -S -icp-total-percent-threshold=50 -mark-icp-complete | FileCheck %s --check-prefixes=ICALL-PROM,ICALL-PROM-MODU
 ; RUN: opt < %s -passes=pgo-icall-prom -S -pass-remarks=pgo-icall-prom -icp-remaining-percent-threshold=0 -icp-total-percent-threshold=0 -icp-max-prom=4 2>&1 | FileCheck %s --check-prefix=PASS-REMARK
 ; RUN: opt < %s -passes=pgo-icall-prom -S -pass-remarks=pgo-icall-prom -icp-remaining-percent-threshold=0 -icp-total-percent-threshold=20 -icp-max-prom=4 2>&1 | FileCheck %s --check-prefix=PASS2-REMARK
 
 ; PASS-REMARK: remark: <unknown>:0:0: Promote indirect call to func4 with count 1030 out of 1600
 ; PASS-REMARK: remark: <unknown>:0:0: Promote indirect call to func2 with count 410 out of 570
 ; PASS-REMARK: remark: <unknown>:0:0: Promote indirect call to func3 with count 150 out of 160
 ; PASS-REMARK: remark: <unknown>:0:0: Promote indirect call to func1 with count 10 out of 10
 

 ; ICALL-PROM: if.false.orig_indirect:
 ; ICALL-PROM:   %call = call i32 %tmp(), !prof [[NEW_VP_METADATA:![0-9]+]]
   ret i32 %call
 ; ICALL-PROM: if.end.icp:
 ; ICALL-PROM:   [[PHI_RET:%[0-9]+]] = phi i32 [ %call, %if.false.orig_indirect ], [ [[DIRCALL_RET]], %if.true.direct_targ ]
 ; ICALL-PROM:   ret i32 [[PHI_RET]]
 }
 
+; ICALL-PROM-NO-MODU-NOT: !llvm.module.flags = !{[[MODULE_FLAG:![0-9]+]]}
+; ICALL-PROM-MODU: !llvm.module.flags = !{[[MODULE_FLAG:![0-9]+]]}
+
+; ICALL-PROM-MODULE: [[MODULE_FLAG]] = !{i32 1, !"IndirectCallPromComplete", i1 true}
+
 !1 = !{!"VP", i32 0, i64 1600, i64 7651369219802541373, i64 1030, i64 -4377547752858689819, i64 410, i64 -6929281286627296573, i64 150, i64 -2545542355363006406, i64 10}
 
 ; ICALL-PROM: [[BRANCH_WEIGHT]] = !{!"branch_weights", i32 1030, i32 570}
 ; ICALL-PROM: [[NEW_VP_METADATA]] = !{!"VP", i32 0, i64 570, i64 -4377547752858689819, i64 410}
 ; ICALL-PROM-SAMPLEPGO: [[CALL_METADATA]] = !{!"branch_weights", i32 1030}

llvm/test/Transforms/PGOProfile/indirect_call_promotion_musttail_typecheck.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; Mustcall needs stricter parameter type checks otherwise it will fail in verifier.
 
-; RUN: opt < %s -passes=pgo-icall-prom -S | FileCheck %s
+; RUN: opt < %s -passes=pgo-icall-prom -mark-icp-complete -S | FileCheck %s --check-prefixes=CHECK,CHECK-MODU
 
 ; Here we check there is no ICP due to parameter mismatch.
 define ptr @func(ptr %msg, ptr %ptr, ptr %ctx, i64 %data.coerce, ptr %table, i64 %hasbits) {
 ; CHECK-LABEL: @func(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr null, align 8
 ; CHECK-NEXT:    ret ptr null
 ; CHECK:       1:

 }
 
 define available_externally ptr @_ZN6proto28internal12ExtensionSet10ParseFieldEmPKcPKNS_7MessageEPNS0_16InternalMetadataEPNS0_12ParseContextE(ptr %this, i64 %tag, ptr %ptr, ptr %containing_type, ptr %metadata, ptr %ctx) {
 entry:
   ret ptr null
 }
 
 !0 = !{!"VP", i32 0, i64 2024, i64 -4843250054591211088, i64 -1, i64 1456131869974120143, i64 947, i64 -4941069334091589447, i64 18}
+
+; CHECK-MODU: !llvm.module.flags = !{[[MODULE_FLAG:![0-9]+]]}
+
+; CHECK-MODU: [[MODULE_FLAG]] = !{i32 1, !"IndirectCallPromComplete", i1 true}

llvm/test/Transforms/SimplifyCFG/merge-indirect-call-metadata.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --check-globals --version 2
+; RUN: opt < %s -passes='simplifycfg<no-sink-common-insts;hoist-common-insts>' -simplifycfg-require-and-preserve-domtree=1 -S | FileCheck %s --check-prefix=HOIST
+; RUN: opt < %s -passes='simplifycfg<sink-common-insts;no-hoist-common-insts>' -simplifycfg-require-and-preserve-domtree=1 -S | FileCheck %s --check-prefix=SINK
+
+; IR @indirect_call_value_profile_merged_by_hoist is generated based on following C++ with manually-annotated !prof
+; Tests that value profiles are merged with instruction hoisted if module flag 'IndirectCallPromComplete' is true.
+; class Base {
+;    public:
+;    virtual int gettype() = 0;
+;    virtual int func1(int a, int b) = 0;
+;};
+;
+; int func2(int x, int y);
+;
+; Base* createptr(int c);
+;
+; int func(int x, int a, int b, int c) {
+;    Base* d = createptr(c);
+;    if (d->gettype() % 5 == 0) {
+;      auto ret = d->func1(a, b);
+;      return ret + func2(b, a);
+;    }
+;    return d->func1(a, b);
+; }
+define i32 @indirect_call_value_profile_merged_by_hoist(i32 %x, i32 %a, i32 %b, i32 %c) {
+; HOIST-LABEL: define i32 @indirect_call_value_profile_merged_by_hoist
+; HOIST-SAME: (i32 [[X:%.*]], i32 [[A:%.*]], i32 [[B:%.*]], i32 [[C:%.*]]) {
+; HOIST-NEXT:  entry:
+; HOIST-NEXT:    [[CALL:%.*]] = tail call ptr @createptr(i32 [[C]])
+; HOIST-NEXT:    [[VTABLE:%.*]] = load ptr, ptr [[CALL]], align 8
+; HOIST-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[VTABLE]], align 8
+; HOIST-NEXT:    [[CALL1:%.*]] = tail call i32 [[TMP0]](ptr [[CALL]])
+; HOIST-NEXT:    [[REM:%.*]] = srem i32 [[CALL1]], 5
+; HOIST-NEXT:    [[CMP:%.*]] = icmp eq i32 [[REM]], 0
+; HOIST-NEXT:    [[VTABLE2:%.*]] = load ptr, ptr [[CALL]], align 8
+; HOIST-NEXT:    [[VFN3:%.*]] = getelementptr inbounds ptr, ptr [[VTABLE2]], i64 1
+; HOIST-NEXT:    [[TMP1:%.*]] = load ptr, ptr [[VFN3]], align 8
+; HOIST-NEXT:    [[CALL4:%.*]] = tail call i32 [[TMP1]](ptr [[CALL]], i32 [[A]], i32 [[B]]), !prof [[PROF1:![0-9]+]]
+; HOIST-NEXT:    br i1 [[CMP]], label [[IF_THEN:%.*]], label [[CLEANUP:%.*]]
+; HOIST:       if.then:
+; HOIST-NEXT:    [[CALL5:%.*]] = tail call i32 @func2(i32 [[B]], i32 [[A]])
+; HOIST-NEXT:    [[ADD:%.*]] = add nsw i32 [[CALL5]], [[CALL4]]
+; HOIST-NEXT:    br label [[CLEANUP]]
+; HOIST:       cleanup:
+; HOIST-NEXT:    [[RETVAL_0:%.*]] = phi i32 [ [[ADD]], [[IF_THEN]] ], [ [[CALL4]], [[ENTRY:%.*]] ]
+; HOIST-NEXT:    ret i32 [[RETVAL_0]]
+;
+
+entry:
+  %call = tail call ptr @createptr(i32 %c)
+  %vtable = load ptr, ptr %call, align 8
+  %0 = load ptr, ptr %vtable, align 8
+  %call1 = tail call i32 %0(ptr %call)
+  %rem = srem i32 %call1, 5
+  %cmp = icmp eq i32 %rem, 0
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %vtable2 = load ptr, ptr %call, align 8
+  %vfn3 = getelementptr inbounds ptr, ptr %vtable2, i64 1
+  %1 = load ptr, ptr %vfn3, align 8
+  %call4 = tail call i32 %1(ptr %call, i32 %a, i32 %b), !prof !1
+  %call5 = tail call i32 @func2(i32 %b, i32 %a)
+  %add = add nsw i32 %call5, %call4
+  br label %cleanup
+
+if.end:                                           ; preds = %entry
+  %vtable6 = load ptr, ptr %call, align 8
+  %vfn7 = getelementptr inbounds ptr, ptr %vtable6, i64 1
+  %2 = load ptr, ptr %vfn7, align 8
+  %call8 = tail call i32 %2(ptr %call, i32 %a, i32 %b), !prof !2
+  br label %cleanup
+
+cleanup:                                          ; preds = %if.end, %if.then
+  %retval.0 = phi i32 [ %add, %if.then ], [ %call8, %if.end ]
+  ret i32 %retval.0
+}
+
+; IR @call_not_sinked is generated based on the following C++ code, with manually annotated !prof
+; Tests that value profiles are merged with instruction sink if module flag 'IndirectCallPromComplete' is true.
+; class Base {
+;    public:
+;    virtual int func() = 0;
+;    virtual int func1(int a, int b) = 0;
+;    virtual int func2(int a, int b) = 0;
+; };
+;
+; Base* createptr(int c);
+;
+; int func(int x, int a, int b, int c) {
+;    Base* d = createptr(c);
+;    if (x % 1000 == 0)
+;      return d->func1(a, b);
+;    return d->func2(a, b);
+; }
+define i32 @indirect_call_value_profile_merged_by_sink(i32 %x, i32 %a, i32 %b, i32 %c) {
+; SINK-LABEL: define i32 @indirect_call_value_profile_merged_by_sink
+; SINK-SAME: (i32 [[X:%.*]], i32 [[A:%.*]], i32 [[B:%.*]], i32 [[C:%.*]]) {
+; SINK-NEXT:  entry:
+; SINK-NEXT:    [[CALL:%.*]] = tail call ptr @createptr(i32 [[C]])
+; SINK-NEXT:    [[REM:%.*]] = srem i32 [[X]], 1000
+; SINK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[REM]], 0
+; SINK-NEXT:    [[DOT:%.*]] = select i1 [[CMP]], i64 1, i64 2
+; SINK-NEXT:    [[VTABLE2:%.*]] = load ptr, ptr [[CALL]], align 8
+; SINK-NEXT:    [[VFN3:%.*]] = getelementptr inbounds ptr, ptr [[VTABLE2]], i64 [[DOT]]
+; SINK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[VFN3]], align 8
+; SINK-NEXT:    [[CALL4:%.*]] = tail call i32 [[TMP0]](ptr [[CALL]], i32 [[A]], i32 [[B]]), !prof [[PROF3:![0-9]+]]
+; SINK-NEXT:    ret i32 [[CALL4]]
+;
+entry:
+  %call = tail call ptr @createptr(i32 %c)
+  %rem = srem i32 %x, 1000
+  %cmp = icmp eq i32 %rem, 0
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %vtable = load ptr, ptr %call, align 8
+  %vfn = getelementptr inbounds ptr, ptr %vtable, i64 1
+  %0 = load ptr, ptr %vfn
+  %call1 = tail call i32 %0(ptr %call, i32 %a, i32 %b), !prof !3
+  br label %cleanup
+
+if.end:                                           ; preds = %entry
+  %vtable2 = load ptr, ptr %call, align 8
+  %vfn3 = getelementptr inbounds ptr, ptr %vtable2, i64 2
+  %1 = load ptr, ptr %vfn3
+  %call4 = tail call i32 %1(ptr %call, i32 %a, i32 %b), !prof !4
+  br label %cleanup
+
+cleanup:                                          ; preds = %if.end, %if.then
+  %retval.0 = phi i32 [ %call1, %if.then ], [ %call4, %if.end ]
+  ret i32 %retval.0
+}
+
+
+declare ptr @createptr(i32)
+declare i32 @func2(i32, i32)
+
+!llvm.module.flags = !{!0}
+
+!0 = !{i32 1, !"IndirectCallPromComplete", i1 true}
+!1 = !{!"VP", i32 0, i64 1600, i64 12345, i64 1030, i64 678, i64 410}
+!2 = !{!"VP", i32 0, i64 1601, i64 54321, i64 1030, i64 678, i64 410}
+!3 =!{!"VP", i32 0, i64 1600, i64 12345, i64 1030, i64 678, i64 410}
+!4 =!{!"VP", i32 0, i64 1601, i64 54321, i64 1030, i64 876, i64 410}
+;.
+; HOIST: [[META0:![0-9]+]] = !{i32 1, !"IndirectCallPromComplete", i1 true}
+; HOIST: [[PROF1]] = !{!"VP", i32 0, i64 3201, i64 12345, i64 1030, i64 54321, i64 1030, i64 678, i64 820}
+;.
+; SINK: [[META0:![0-9]+]] = !{i32 1, !"IndirectCallPromComplete", i1 true}
+; SINK: [[PROF3]] = !{!"VP", i32 0, i64 3201, i64 54321, i64 1030, i64 12345, i64 1030, i64 876, i64 410, i64 678, i64 410}
+;.

llvm/test/Transforms/SimplifyCFG/preserve-indirect-call-metadata.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --check-globals --version 2
+; RUN: opt < %s -passes='simplifycfg<no-sink-common-insts;hoist-common-insts>' -simplifycfg-require-and-preserve-domtree=1 -S | FileCheck %s --check-prefix=HOIST
+; RUN: opt < %s -passes='simplifycfg<sink-common-insts;no-hoist-common-insts>' -simplifycfg-require-and-preserve-domtree=1 -S | FileCheck %s --check-prefix=SINK
+
+; IR @indirect_call_value_profile_preserved_by_hoist is generated based on following C++ with manually-annotated !prof.
+; Tests that indirect callsites are preserved as opposed to sunk if module flag 'IndirectCallPromComplete' is not present.
+; Without preserving call instructions, `d->func1` is hoisted while it may not make sense
+; to do so. For example, the candidate calls are different based on derived type.
+; class Base {
+;    public:
+;    virtual int gettype() = 0;
+;    virtual int func1(int a, int b) = 0;
+;};
+;
+; int func2(int x, int y);
+;
+; Base* createptr(int c);
+;
+; int func(int x, int a, int b, int c) {
+;    Base* d = createptr(c);
+;    if (d->gettype() % 5 == 0) {
+;      auto ret = d->func1(a, b);
+;      return ret + func2(b, a);
+;    }
+;    return d->func1(a, b);
+; }
+define i32 @indirect_call_value_profile_preserved_by_hoist(i32 %x, i32 %a, i32 %b, i32 %c) {
+;
+; SINK-LABEL: define i32 @indirect_call_value_profile_preserved_by_hoist
+; SINK-SAME: (i32 [[X:%.*]], i32 [[A:%.*]], i32 [[B:%.*]], i32 [[C:%.*]]) {
+; SINK-NEXT:  entry:
+; SINK-NEXT:    [[CALL:%.*]] = tail call ptr @createptr(i32 [[C]])
+; SINK-NEXT:    [[VTABLE:%.*]] = load ptr, ptr [[CALL]], align 8
+; SINK-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[VTABLE]], align 8
+; SINK-NEXT:    [[CALL1:%.*]] = tail call i32 [[TMP0]](ptr [[CALL]])
+; SINK-NEXT:    [[REM:%.*]] = srem i32 [[CALL1]], 5
+; SINK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[REM]], 0
+; SINK-NEXT:    br i1 [[CMP]], label [[IF_THEN:%.*]], label [[IF_END:%.*]]
+; SINK:       if.then:
+; SINK-NEXT:    [[VTABLE2:%.*]] = load ptr, ptr [[CALL]], align 8
+; SINK-NEXT:    [[VFN3:%.*]] = getelementptr inbounds ptr, ptr [[VTABLE2]], i64 1
+; SINK-NEXT:    [[TMP1:%.*]] = load ptr, ptr [[VFN3]], align 8
+; SINK-NEXT:    [[CALL4:%.*]] = tail call i32 [[TMP1]](ptr [[CALL]], i32 [[A]], i32 [[B]]), !prof [[PROF0:![0-9]+]]
+; SINK-NEXT:    [[CALL5:%.*]] = tail call i32 @func2(i32 [[B]], i32 [[A]])
+; SINK-NEXT:    [[ADD:%.*]] = add nsw i32 [[CALL5]], [[CALL4]]
+; SINK-NEXT:    br label [[CLEANUP:%.*]]
+; SINK:       if.end:
+; SINK-NEXT:    [[VTABLE6:%.*]] = load ptr, ptr [[CALL]], align 8
+; SINK-NEXT:    [[VFN7:%.*]] = getelementptr inbounds ptr, ptr [[VTABLE6]], i64 1
+; SINK-NEXT:    [[TMP2:%.*]] = load ptr, ptr [[VFN7]], align 8
+; SINK-NEXT:    [[CALL8:%.*]] = tail call i32 [[TMP2]](ptr [[CALL]], i32 [[A]], i32 [[B]]), !prof [[PROF1:![0-9]+]]
+; SINK-NEXT:    br label [[CLEANUP]]
+; SINK:       cleanup:
+; SINK-NEXT:    [[RETVAL_0:%.*]] = phi i32 [ [[ADD]], [[IF_THEN]] ], [ [[CALL8]], [[IF_END]] ]
+; SINK-NEXT:    ret i32 [[RETVAL_0]]
+;
+
+entry:
+  %call = tail call ptr @createptr(i32 %c)
+  %vtable = load ptr, ptr %call, align 8
+  %0 = load ptr, ptr %vtable, align 8
+  %call1 = tail call i32 %0(ptr %call)
+  %rem = srem i32 %call1, 5
+  %cmp = icmp eq i32 %rem, 0
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %vtable2 = load ptr, ptr %call, align 8
+  %vfn3 = getelementptr inbounds ptr, ptr %vtable2, i64 1
+  %1 = load ptr, ptr %vfn3, align 8
+  %call4 = tail call i32 %1(ptr %call, i32 %a, i32 %b), !prof !1
+  %call5 = tail call i32 @func2(i32 %b, i32 %a)
+  %add = add nsw i32 %call5, %call4
+  br label %cleanup
+
+if.end:                                           ; preds = %entry
+  %vtable6 = load ptr, ptr %call, align 8
+  %vfn7 = getelementptr inbounds ptr, ptr %vtable6, i64 1
+  %2 = load ptr, ptr %vfn7, align 8
+  %call8 = tail call i32 %2(ptr %call, i32 %a, i32 %b), !prof !2
+  br label %cleanup
+
+cleanup:                                          ; preds = %if.end, %if.then
+  %retval.0 = phi i32 [ %add, %if.then ], [ %call8, %if.end ]
+  ret i32 %retval.0
+}
+
+; IR @indirect_call_value_profile_preserved_by_hoist is generated based on the following C++ code, with manually annotated !prof metadatas.
+; Note 'd->func1' and 'd->func2' are two virtual functions with different offsets so could have different value profiles.
+; Without preserving call instructions with !prof, call instructions are simplified to one with selected offsets.
+; class Base {
+;    public:
+;    virtual int func() = 0;
+;    virtual int func1(int a, int b) = 0;
+;    virtual int func2(int a, int b) = 0;
+; };
+;
+; Base* createptr(int c);
+;
+; int func(int x, int a, int b, int c) {
+;    Base* d = createptr(c);
+;    if (x % 1000 == 0)
+;      return d->func1(a, b);
+;    return d->func2(a, b);
+; }
+define i32 @indirect_call_value_profile_preserved_by_sink(i32 %x, i32 %a, i32 %b, i32 %c) {
+; HOIST-LABEL: define i32 @indirect_call_value_profile_preserved_by_sink
+; HOIST-SAME: (i32 [[X:%.*]], i32 [[A:%.*]], i32 [[B:%.*]], i32 [[C:%.*]]) {
+; HOIST-NEXT:  entry:
+; HOIST-NEXT:    [[CALL:%.*]] = tail call ptr @createptr(i32 [[C]])
+; HOIST-NEXT:    [[REM:%.*]] = srem i32 [[X]], 1000
+; HOIST-NEXT:    [[CMP:%.*]] = icmp eq i32 [[REM]], 0
+; HOIST-NEXT:    [[VTABLE:%.*]] = load ptr, ptr [[CALL]], align 8
+; HOIST-NEXT:    br i1 [[CMP]], label [[IF_THEN:%.*]], label [[IF_END:%.*]]
+; HOIST:       if.then:
+; HOIST-NEXT:    [[VFN:%.*]] = getelementptr inbounds ptr, ptr [[VTABLE]], i64 1
+; HOIST-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[VFN]], align 8
+; HOIST-NEXT:    [[CALL1:%.*]] = tail call i32 [[TMP0]](ptr [[CALL]], i32 [[A]], i32 [[B]]), !prof [[PROF2:![0-9]+]]
+; HOIST-NEXT:    br label [[CLEANUP:%.*]]
+; HOIST:       if.end:
+; HOIST-NEXT:    [[VFN3:%.*]] = getelementptr inbounds ptr, ptr [[VTABLE]], i64 2
+; HOIST-NEXT:    [[TMP1:%.*]] = load ptr, ptr [[VFN3]], align 8
+; HOIST-NEXT:    [[CALL4:%.*]] = tail call i32 [[TMP1]](ptr [[CALL]], i32 [[A]], i32 [[B]]), !prof [[PROF3:![0-9]+]]
+; HOIST-NEXT:    br label [[CLEANUP]]
+; HOIST:       cleanup:
+; HOIST-NEXT:    [[RETVAL_0:%.*]] = phi i32 [ [[CALL1]], [[IF_THEN]] ], [ [[CALL4]], [[IF_END]] ]
+; HOIST-NEXT:    ret i32 [[RETVAL_0]]
+
+entry:
+  %call = tail call ptr @createptr(i32 %c)
+  %rem = srem i32 %x, 1000
+  %cmp = icmp eq i32 %rem, 0
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %vtable = load ptr, ptr %call, align 8
+  %vfn = getelementptr inbounds ptr, ptr %vtable, i64 1
+  %0 = load ptr, ptr %vfn
+  %call1 = tail call i32 %0(ptr %call, i32 %a, i32 %b), !prof !3
+  br label %cleanup
+
+if.end:                                           ; preds = %entry
+  %vtable2 = load ptr, ptr %call, align 8
+  %vfn3 = getelementptr inbounds ptr, ptr %vtable2, i64 2
+  %1 = load ptr, ptr %vfn3
+  %call4 = tail call i32 %1(ptr %call, i32 %a, i32 %b), !prof !4
+  br label %cleanup
+
+cleanup:                                          ; preds = %if.end, %if.then
+  %retval.0 = phi i32 [ %call1, %if.then ], [ %call4, %if.end ]
+  ret i32 %retval.0
+}
+
+
+declare ptr @createptr(i32)
+declare i32 @func2(i32, i32)
+
+!1 = !{!"VP", i32 0, i64 1600, i64 12345, i64 1030, i64 678, i64 410}
+!2 = !{!"VP", i32 0, i64 1601, i64 54321, i64 1030, i64 678, i64 410}
+!3 =!{!"VP", i32 0, i64 1602, i64 12345, i64 1030, i64 678, i64 410}
+!4 =!{!"VP", i32 0, i64 1603, i64 54321, i64 1030, i64 876, i64 410}
+;.
+; HOIST: [[PROF2]] = !{!"VP", i32 0, i64 1602, i64 12345, i64 1030, i64 678, i64 410}
+; HOIST: [[PROF3]] = !{!"VP", i32 0, i64 1603, i64 54321, i64 1030, i64 876, i64 410}
+;.
+; SINK: [[PROF0]] = !{!"VP", i32 0, i64 1600, i64 12345, i64 1030, i64 678, i64 410}
+; SINK: [[PROF1]] = !{!"VP", i32 0, i64 1601, i64 54321, i64 1030, i64 678, i64 410}
+;.