[CSSPGO] Use nested context-sensitive profile.

[hoy]

Summary

CSSPGO currently employs a flat profile format for context-sensitive profiles. Such a flat profile allows for precisely manipulating contexts that is either inlined or not inlined. This is a benefit over the nested profile format used by non-CS AutoFDO. A downside of this is the longer build time due to parsing the indexing the full CS contexts.

For a CS flat profile, though only the context profiles relevant to a module are loaded when that module is compiled, the cost to figure out what profiles are relevant is noticeably high when there're many contexts, since the sample reader will need to scan all context strings anyway. On the contrary, a nested function profile has its related inline subcontexts isolated from other unrelated contexts. Therefore when compiling a set of functions, unrelated contexts will never need to be scanned.

In this change we are exploring using nested profile format for CSSPGO. This is expected to work based on an assumption that with a preinliner-computed profile all contexts are precomputed and expected to be inlined by the compiler. Contexts not expected to be inlined will be cut off and returned to corresponding base profiles (for top-level outlined functions). This naturally forms a nested profile where all nested contexts are expected to be inlined. The compiler will less likely optimize on derived contexts that are not precomputed.

A CS-nested profile will look exactly the same with regular nested profile except that each nested profile can come with an attributes. With pseudo probes, a nested profile shown as below can also have a CFG checksum.

main:1968679:12
 2: 24
 3: 28 _Z5funcAi:18
 3.1: 28 _Z5funcBi:30
 3: _Z5funcAi:1467398
  0: 10
  1: 10 _Z8funcLeafi:11
  3: 24
  1: _Z8funcLeafi:1467299
   0: 6
   1: 6
   3: 287884
   4: 287864 _Z3fibi:315608
   15: 23
    !CFGChecksum: 138828622701
   !Attributes: 2
  !CFGChecksum: 281479271677951
  !Attributes: 2

Specific work included in this change:

• A recursive profile converter to convert CS flat profile to nested profile.
• Extend function checksum and attribute metadata to be stored in nested way for text profile and extbinary profile.
• Unifiy sample loader inliner path for CS and preinlined nested profile.
  • Changes in the sample loader to support probe-based nested profile.

I've seen promising results regarding build time. A nested profile can result in a 20% shorter build time than a CS flat profile while keep an on-par performance. This is with -duplicate-contexts-into-base=1.

Test Plan:

[hoy]

Updating D115205: [CSSPGO] Use nested context-sensitive profile.

[hoy]

Moving profile conversion to be after CS profile merging/trimming.

[wenlei]

llvm/include/llvm/ProfileData/SampleProf.h

1183

High level comments.

1187

FunctionSamples already has tree structure, can we use FunctionSamples tree to hold the nesting profiles directly, instead of introducing an intermediate trie tree? (The intermediate trie tree also overlaps with ContextTrieNode..)

llvm/lib/ProfileData/SampleProf.cpp

39

nit: generate-merged-base-profiles - "When generating nested context-sensitive profiles, always generate extra base profile for function with all its context profiles merged into it."

534

When NodeProfile (parent profile) doesn't exist, would it be beneficial to create the NodeProfile to contain inlinee's profile? That would be a more faithful conversion from a flat profile.

537–543

IIUC, the per-context ContextShouldBeInlined now becomes a per-profile flag. However, shouldInlineCandidate used by the inliner still checks attribute ContextShouldBeInlined from sample's context. That was how we tell inliner to honor preinline decision. Now without that attribute, how does inliner know to honor preinline decision?

llvm/lib/ProfileData/SampleProfReader.cpp

249

update comment

346–347

Does this mean we only set function hash for top level profile? why skip setting hash for nested inlinees?

1092

This parameter is not used?

llvm/lib/Transforms/IPO/SampleProfile.cpp

1484

The definition for ProfileIsPreinlined isn't very clear. A CS profile can also be preinlined, but here it looks like you're using it specifically for flat profile that is preinlined. Suggest we adhere to the literal definition which would apply to both flat and nest profile for preinline.

The check here could be using whether context tracker is available, basically we're try to track context without context tracker (which is tied to flat CS profile).

1523

nit: promoteMergeNotInlinedContextSamples. this is effectively a "context tracker" for nest profile, so we can use similar name as context tracker.

1775

This also opens up priority inliner for non-CS (not even preinlined) profile. Some (manual) testing as sanity check would be good.

llvm/test/tools/llvm-profdata/Inputs/cs-sample-preinline.proftext

16

what triggered these attribute value changes?

llvm/test/tools/llvm-profdata/cs-sample-nested-profile.test

2

have a probe version test case too to cover read/write per-inlinee function hash?

llvm/tools/llvm-profgen/ProfileGenerator.cpp

757

The name ProfileIsPreinlined suggest it should be set for flat profile too when preinliner is on, but this seems to be only used for nested profile.

[hoy]

llvm/include/llvm/ProfileData/SampleProf.h

1187

Nodes in a context tree may not have a function profile. Right, the struct here basically does what ContextTrieNode do. I was thinking about making ContextTrieNode shared between ProfilingTools and IPO, and that would require moving the definition into SampleProf.h, which may not be what we want.

llvm/lib/ProfileData/SampleProf.cpp

39

Sounds good.

534

When NodeProfile doesn't exist, it means the parent function is cold, and its profile also doesn't exist in the flat profile. So we are honoring that by not creating such profile here.

537–543

The per-profile flag is only used to tell the compiler how to set up the inliner (priority vs non-prioirty) and mcf extra. When it comes to inlining a particular callsite, the inliner would not honor the inline decision based on the per-context attribute. We always encode the per-context attribute in the profile.

llvm/lib/ProfileData/SampleProfReader.cpp

249

Good catch, done.

346–347

We set function hash for all profiles. We just count top-level profiles by ProbeProfileCount for sanity check in the assert several lines below. I just renamed the variable to TopLevelProbeProfileCount.

1092

Good catch.

llvm/lib/Transforms/IPO/SampleProfile.cpp

1484

Yeah, ProfileIsNestedCS sounds better?

Makes sense to check against ContextTracker.

1523

Sounds good.

1775

Test added to llvm-project/llvm/test/Transforms/SampleProfile/inline-mergeprof.ll and inline.ll

llvm/test/tools/llvm-profdata/Inputs/cs-sample-preinline.proftext

16

It's due to the following change in SampleProfWriter.cpp:

if (S.getContext().getAllAttributes()) {
    OS.indent(Indent + 1);
    OS << "!Attributes: " << S.getContext().getAllAttributes() << "\n";

llvm/test/tools/llvm-profdata/cs-sample-nested-profile.test

2

test added.

llvm/tools/llvm-profgen/ProfileGenerator.cpp

757

The flag is supposed to be only used by the compiler to differentiate a nested CS profile from a nested non-CS profile. Maybe ProfileIsNestedCS sound better?

[hoy]

Addressing Wenlei's comments.

[wenlei]

llvm/lib/ProfileData/SampleProf.cpp

534

If we have preinliner on, cold parent should be removed from context already. If the parent exists, that means it's somewhat relevant for inlining.

537–543

When it comes to inlining a particular callsite, the inliner would not honor the inline decision based on the per-context attribute.

Did you mean would or would not? I expect the correct behavior to be that inliner will honor per-context ContextShouldBeInlined attribute, which is what happens with flat CS profile.

We always encode the per-context attribute in the profile.

Where is the per-context attribute transferred from flat profile to nest profile? This is the only place I see we check ContextShouldBeInlined, but it's transferred to global FunctionSamples::ProfileIsPreinlined, not per-context attribute.

llvm/lib/Transforms/IPO/SampleProfile.cpp

1484

Yeah, ProfileIsNestedCS sounds better?

That works. Though it doesn't make sense for ProfileIsNestedCS to be true while ProfileIsCS is false. So if we use ProfileIsNestedCS, we need to either 1) rename ProfileIsCS to be ProfileIsFlatCS, or 2) replace check on ProfileIsCS with ProfileIsCS & !ProfileIsNestedCS

llvm/tools/llvm-profdata/llvm-profdata.cpp

953

how about gen-nested-cs-profile to make it explicit and differentiate from non-CS profile, especially if we go with ProfileIsNestedCS global flag.

[wenlei]

llvm/test/tools/llvm-profdata/Inputs/cs-sample-preinline.proftext

16

I meant the value change from 0/1 to 2, not the ones that were 0.

[hoy]

llvm/lib/ProfileData/SampleProf.cpp

534

Right. Here we are not necessarily converting a preinlined flat profile. When the flat profile is preinlined, we'll set the ProfileIsNestedCS flag, otherwise nodes in the context trie could have no profiles.

537–543

Did you mean would or would not? I expect the correct behavior to be that inliner will honor per-context ContextShouldBeInlined attribute, which is what happens with flat CS profile.

I mean would. Sorry for the confusion.

Where is the per-context attribute transferred from flat profile to nest profile? This is the only place I see we check ContextShouldBeInlined, but it's transferred to global FunctionSamples::ProfileIsPreinlined, not per-context attribute.

The attribute that comes with a context in a flat profile will be encoded with the context into the nested profile as well. Note that we only reset the name of the original context but nothing else. The original attribute is still there.

// Reset the child context to be contextless.
    ChildProfile->getContext().setName(OrigChildContext.getName());

See the test llvm/test/tools/llvm-profdata/cs-sample-nested-profile.test .

llvm/lib/Transforms/IPO/SampleProfile.cpp

1484

ProfileIsFlatCS and ProfileIsNestedCS sounds good to me.

llvm/test/tools/llvm-profdata/Inputs/cs-sample-preinline.proftext

16

Ah, this is the input profile of a test. I made it to be preinliner-generated. It was copied from cs-sample.proftext.

llvm/tools/llvm-profdata/llvm-profdata.cpp

953

Sounds good.

[hoy]

Renaming ProfileIsCS to ProfileIsCSFlat, ProfileIsPreinlined to ProfileIsCSNested.

[hoy]

Rebasing

[wenlei]

llvm/lib/ProfileData/SampleProf.cpp

534

So at least in preinliner case, it could be better to keep parent profile?

537–543

Now I see how the attributed is transferred, thanks for clarification.

llvm/lib/ProfileData/SampleProfReader.cpp

249

I'm still seeing SeenMetadata in the comment?

llvm/lib/Transforms/IPO/SampleProfile.cpp

1484

nit: there's NestedCS and CSFlat (as opposed to FlatCS) in switch, section flags and variable names , which is a bit inconsistent.

[hoy]

Renaming gen-nested-cs-profile to gen-cs-nested-profile

llvm/lib/ProfileData/SampleProf.cpp

534

Yes, that's currently it is. With preinliner, every node in the context trie will come with a profile so NodeProfile will never be empty. Without preinliner, NodeProfile could be empty.

llvm/lib/ProfileData/SampleProfReader.cpp

249

Good catch, changed to DepthMetadata.

llvm/lib/Transforms/IPO/SampleProfile.cpp

1484

Yeah, we still use the switch gen-nested-cs-profile. Let me change it to gen-cs-nested-profile

[wenlei]

lgtm, thanks.

llvm/lib/ProfileData/SampleProf.cpp

534

ok, sounds good.

[uabelho]

Hi,

Is anyone else having problems with this commit?
We get:

/proj/flexasic/app/llvm/8.0/bin/clang++  -march=corei7  -DGTEST_HAS_RTTI=0 -D_DEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/ProfileData -I../lib/ProfileData -Iinclude -I../include -fPIC -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG    -fno-exceptions -fno-rtti -UNDEBUG -std=c++14 -MD -MT lib/ProfileData/CMakeFiles/LLVMProfileData.dir/SampleProf.cpp.o -MF lib/ProfileData/CMakeFiles/LLVMProfileData.dir/SampleProf.cpp.o.d -o lib/ProfileData/CMakeFiles/LLVMProfileData.dir/SampleProf.cpp.o -c ../lib/ProfileData/SampleProf.cpp
In file included from ../lib/ProfileData/SampleProf.cpp:14:
In file included from ../include/llvm/ProfileData/SampleProf.h:17:
In file included from ../include/llvm/ADT/DenseSet.h:16:
In file included from ../include/llvm/ADT/DenseMap.h:16:
In file included from ../include/llvm/ADT/DenseMapInfo.h:19:
In file included from /proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/tuple:39:
In file included from /proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/array:38:
In file included from /proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/stdexcept:39:
In file included from /proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/string:41:
In file included from /proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/allocator.h:46:
In file included from /proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/x86_64-unknown-linux-gnu/bits/c++allocator.h:33:
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/ext/new_allocator.h:120:23: error: no matching constructor for initialization of 'std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples>'
        { ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
                             ^   ~~~~~~~~~~~~~~~~~~~~~~~~~~~
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/alloc_traits.h:530:8: note: in instantiation of function template specialization '__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > >::construct<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples>, llvm::StringRef, llvm::sampleprof::FunctionSamples &>' requested here
        { __a.construct(__p, std::forward<_Args>(__args)...); }
              ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_tree.h:529:23: note: in instantiation of function template specialization 'std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > > >::construct<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples>, llvm::StringRef, llvm::sampleprof::FunctionSamples &>' requested here
              _Alloc_traits::construct(_M_get_Node_allocator(),
                             ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_tree.h:546:4: note: in instantiation of function template specialization 'std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples>, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> >, std::less<void>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > >::_M_construct_node<llvm::StringRef, llvm::sampleprof::FunctionSamples &>' requested here
          _M_construct_node(__tmp, std::forward<_Args>(__args)...);
          ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_tree.h:2123:19: note: in instantiation of function template specialization 'std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples>, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> >, std::less<void>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > >::_M_create_node<llvm::StringRef, llvm::sampleprof::FunctionSamples &>' requested here
        _Link_type __z = _M_create_node(std::forward<_Args>(__args)...);
                         ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_map.h:559:16: note: in instantiation of function template specialization 'std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples>, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> >, std::less<void>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > >::_M_emplace_unique<llvm::StringRef, llvm::sampleprof::FunctionSamples &>' requested here
        { return _M_t._M_emplace_unique(std::forward<_Args>(__args)...); }
                      ^
../lib/ProfileData/SampleProf.cpp:524:18: note: in instantiation of function template specialization 'std::map<std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples, std::less<void>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > >::emplace<llvm::StringRef, llvm::sampleprof::FunctionSamples &>' requested here
      SamplesMap.emplace(OrigChildContext.getName(), *ChildProfile);
                 ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:112:26: note: candidate constructor not viable: no known conversion from 'llvm::StringRef' to 'const const std::__cxx11::basic_string<char>' for 1st argument
      _GLIBCXX_CONSTEXPR pair(const _T1& __a, const _T2& __b)
                         ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:138:12: note: candidate constructor template not viable: no known conversion from 'llvm::StringRef' to 'const const std::__cxx11::basic_string<char>' for 1st argument
        constexpr pair(const _T1& __x, _U2&& __y)
                  ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:133:12: note: candidate template ignored: requirement 'is_convertible<llvm::StringRef, const std::__cxx11::basic_string<char> >::value' was not satisfied [with _U1 = llvm::StringRef]
        constexpr pair(_U1&& __x, const _T2& __y)
                  ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:144:12: note: candidate template ignored: requirement '__and_<std::is_convertible<llvm::StringRef, const std::__cxx11::basic_string<char> >, std::is_convertible<llvm::sampleprof::FunctionSamples &, llvm::sampleprof::FunctionSamples> >::value' was not satisfied [with _U1 = llvm::StringRef, _U2 = llvm::sampleprof::FunctionSamples &]
        constexpr pair(_U1&& __x, _U2&& __y)
                  ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:124:12: note: candidate constructor template not viable: requires single argument '__p', but 2 arguments were provided
        constexpr pair(const pair<_U1, _U2>& __p)
                  ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:150:12: note: candidate constructor template not viable: requires single argument '__p', but 2 arguments were provided
        constexpr pair(pair<_U1, _U2>&& __p)
                  ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:155:9: note: candidate constructor template not viable: requires 3 arguments, but 2 were provided
        pair(piecewise_construct_t, tuple<_Args1...>, tuple<_Args2...>);
        ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:206:9: note: candidate constructor template not viable: requires 4 arguments, but 2 were provided
        pair(tuple<_Args1...>&, tuple<_Args2...>&,
        ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:127:17: note: candidate constructor not viable: requires 1 argument, but 2 were provided
      constexpr pair(const pair&) = default;
                ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:128:17: note: candidate constructor not viable: requires 1 argument, but 2 were provided
      constexpr pair(pair&&) = default;
                ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:108:26: note: candidate constructor not viable: requires 0 arguments, but 2 were provided
      _GLIBCXX_CONSTEXPR pair()
                         ^
1 error generated.

[uabelho]

Looks like this bot fails the same way:
https://lab.llvm.org/buildbot/#/builders/118/builds/6994

[foad]

@uabelho it looks like this has been fixed by 156b1e6ba84887ba92688b3ad31e400c56f707b2.

[uabelho]

In D115205#3194804, @foad wrote:

@uabelho it looks like this has been fixed by 156b1e6ba84887ba92688b3ad31e400c56f707b2.

Oh, yes it has. Thanks!

llvm/include/llvm/ProfileData/SampleProf.h

enum class SecProfSummaryFlags : uint32_t {
   /// SecFlagFSDiscriminator means this profile uses flow-sensitive
   /// discriminators.
   SecFlagFSDiscriminator = (1 << 2)
 };
 
 enum class SecFuncMetadataFlags : uint32_t {
   SecFlagInvalid = 0,
   SecFlagIsProbeBased = (1 << 0),
-  SecFlagHasAttribute = (1 << 1)
+  SecFlagHasAttribute = (1 << 1),
+  SecFlagIsPreInlined = (1 << 2),
 };
 
 enum class SecFuncOffsetFlags : uint32_t {
   SecFlagInvalid = 0,
   // Store function offsets in an order of contexts. The order ensures that
   // callee contexts of a given context laid out next to it.
   SecFlagOrdered = (1 << 0),
 };
std::string toString() const {
     return getContextString(FullContext, false);
   }
 
   uint64_t getHashCode() const {
     return hasContext() ? hash_value(getContextFrames())
                         : hash_value(getName());
   }
 
-  /// Set the name of the function.
+  /// Set the name of the function and clear the current context.
   void setName(StringRef FunctionName) {
-    assert(FullContext.empty() &&
-           "setName should only be called for non-CS profile");
     Name = FunctionName;
+    FullContext = SampleContextFrames();
+    State = UnknownContext;
   }
 
   void setContext(SampleContextFrames Context,
                   ContextStateMask CState = RawContext) {
     assert(CState != UnknownContext);
     FullContext = Context;
     Name = Context.back().FuncName;
     State = CState;
void updateTotalSamples() {
     for (auto &I : CallsiteSamples) {
       for (auto &CS : I.second) {
         CS.second.updateTotalSamples();
         addTotalSamples(CS.second.getTotalSamples());
       }
     }
   }
 
+  // Set current context and all callee contexts to be synthetic.
+  void SetContextSynthetic() {
+    Context.setState(SyntheticContext);
+    for (auto &I : CallsiteSamples) {
+      for (auto &CS : I.second) {
+        CS.second.SetContextSynthetic();
+      }
+    }
+  }
+
   /// Return the number of samples collected at the given location.
   /// Each location is specified by \p LineOffset and \p Discriminator.
   /// If the location is not found in profile, return error.
   ErrorOr<uint64_t> findSamplesAt(uint32_t LineOffset,
                                   uint32_t Discriminator) const {
     const auto &ret = BodySamples.find(LineLocation(LineOffset, Discriminator));
     if (ret == BodySamples.end())
       return std::error_code();
public:
   /// instruction of the symbol. But as we directly get this info for raw
   /// profile without referring to potentially inaccurate debug info, this
   /// gives more accurate profile data and is preferred for standalone symbols.
   uint64_t getHeadSamples() const { return TotalHeadSamples; }
 
   /// Return the sample count of the first instruction of the function.
   /// The function can be either a standalone symbol or an inlined function.
   uint64_t getEntrySamples() const {
-    if (FunctionSamples::ProfileIsCS && getHeadSamples()) {
+    if (FunctionSamples::ProfileIsCSFlat && getHeadSamples()) {
       // For CS profile, if we already have more accurate head samples
       // counted by branch sample from caller, use them as entry samples.
       return getHeadSamples();
     }
     uint64_t Count = 0;
     // Use either BodySamples or CallsiteSamples which ever has the smaller
     // lineno.
     if (!BodySamples.empty() &&
public:
   /// Returns the line offset to the start line of the subprogram.
   /// We assume that a single function will not exceed 65535 LOC.
   static unsigned getOffset(const DILocation *DIL);
 
   /// Returns a unique call site identifier for a given debug location of a call
   /// instruction. This is wrapper of two scenarios, the probe-based profile and
   /// regular profile, to hide implementation details from the sample loader and
   /// the context tracker.
-  static LineLocation getCallSiteIdentifier(const DILocation *DIL);
+  static LineLocation getCallSiteIdentifier(const DILocation *DIL,
+                                            bool ProfileIsFS = false);
+
+  /// Returns a unique hash code for a combination of a callsite location and
+  /// the callee function name.
+  static uint64_t getCallSiteHash(StringRef CalleeName,
+                                  const LineLocation &Callsite);
 
   /// Get the FunctionSamples of the inline instance where DIL originates
   /// from.
   ///
   /// The FunctionSamples of the instruction (Machine or IR) associated to
   /// \p DIL is the inlined instance in which that instruction is coming from.
   /// We traverse the inline stack of that instruction, and match it with the
   /// tree nodes in the profile.
   ///
   /// \returns the FunctionSamples pointer to the inlined instance.
   /// If \p Remapper is not nullptr, it will be used to find matching
   /// FunctionSamples with not exactly the same but equivalent name.
   const FunctionSamples *findFunctionSamples(
       const DILocation *DIL,
       SampleProfileReaderItaniumRemapper *Remapper = nullptr) const;
 
   static bool ProfileIsProbeBased;
 
-  static bool ProfileIsCS;
+  static bool ProfileIsCSFlat;
+
+  static bool ProfileIsCSNested;
 
   SampleContext &getContext() const { return Context; }
 
   void setContext(const SampleContext &FContext) { Context = FContext; }
 
   static SampleProfileFormat Format;
 
   /// Whether the profile uses MD5 to represent string.
void trimAndMergeColdContextProfiles(uint64_t ColdCountThreshold,
                                        bool TrimBaseProfileOnly);
   // Canonicalize context profile name and attributes.
   void canonicalizeContextProfiles();
 
 private:
   SampleProfileMap &ProfileMap;
 };
 
+// CSProfileConverter converts a full context-sensitive flat sample profile into
+// a nested context-sensitive sample profile.
+class CSProfileConverter {
+public:
+  CSProfileConverter(SampleProfileMap &Profiles);
+  void convertProfiles();
+  struct FrameNode {
+    FrameNode(StringRef FName = StringRef(),
+              FunctionSamples *FSamples = nullptr,
+              LineLocation CallLoc = {0, 0})
+        : FuncName(FName), FuncSamples(FSamples), CallSiteLoc(CallLoc){};
+
+    // Map line+discriminator location to child frame
+    std::map<uint64_t, FrameNode> AllChildFrames;
+    // Function name for current frame
+    StringRef FuncName;
+    // Function Samples for current frame
+    FunctionSamples *FuncSamples;
+    // Callsite location in parent context
+    LineLocation CallSiteLoc;
+
+    FrameNode *getOrCreateChildFrame(const LineLocation &CallSite,
+                                     StringRef CalleeName);
+  };
+
+private:
+  // Nest all children profiles into the profile of Node.
+  void convertProfiles(FrameNode &Node);
+  FrameNode *getOrCreateContextPath(const SampleContext &Context);
+
+  SampleProfileMap &ProfileMap;
+  FrameNode RootFrame;
+};
+
 /// ProfileSymbolList records the list of function symbols shown up
 /// in the binary used to generate the profile. It is useful to
 /// to discriminate a function being so cold as not to shown up
 /// in the profile and a function newly added.
 class ProfileSymbolList {
 public:
   /// copy indicates whether we need to copy the underlying memory
   /// for the input Name.

llvm/include/llvm/ProfileData/SampleProfReader.h

public:
   MemoryBuffer *getBuffer() const { return Buffer.get(); }
 
   /// \brief Return the profile format.
   SampleProfileFormat getFormat() const { return Format; }
 
   /// Whether input profile is based on pseudo probes.
   bool profileIsProbeBased() const { return ProfileIsProbeBased; }
 
-  /// Whether input profile is fully context-sensitive
-  bool profileIsCS() const { return ProfileIsCS; }
+  /// Whether input profile is fully context-sensitive and flat.
+  bool profileIsCSFlat() const { return ProfileIsCSFlat; }
+
+  /// Whether input profile is fully context-sensitive and nested.
+  bool profileIsCSNested() const { return ProfileIsCSNested; }
 
   virtual std::unique_ptr<ProfileSymbolList> getProfileSymbolList() {
     return nullptr;
   };
 
   /// It includes all the names that have samples either in outline instance
   /// or inline instance.
   virtual std::vector<StringRef> *getNameTable() { return nullptr; }
protected:
   /// Compute summary for this profile.
   void computeSummary();
 
   std::unique_ptr<SampleProfileReaderItaniumRemapper> Remapper;
 
   /// \brief Whether samples are collected based on pseudo probes.
   bool ProfileIsProbeBased = false;
 
-  /// Whether function profiles are context-sensitive.
-  bool ProfileIsCS = false;
+  /// Whether function profiles are context-sensitive flat profiles.
+  bool ProfileIsCSFlat = false;
+
+  /// Whether function profiles are context-sensitive nested profiles.
+  bool ProfileIsCSNested = false;
 
   /// Number of context-sensitive profiles.
   uint32_t CSProfileCount = 0;
 
   /// Whether the function profiles use FS discriminators.
   bool ProfileIsFS = false;
 
   /// \brief The format of sample.
private:
   BumpPtrAllocator Allocator;
 
 protected:
   std::vector<SecHdrTableEntry> SecHdrTable;
   std::error_code readSecHdrTableEntry(uint32_t Idx);
   std::error_code readSecHdrTable();
 
   std::error_code readFuncMetadata(bool ProfileHasAttribute);
+  std::error_code readFuncMetadata(bool ProfileHasAttribute,
+                                   FunctionSamples *FProfile);
   std::error_code readFuncOffsetTable();
   std::error_code readFuncProfiles();
   std::error_code readMD5NameTable();
   std::error_code readNameTableSec(bool IsMD5);
   std::error_code readCSNameTableSec();
   std::error_code readProfileSymbolList();
 
   virtual std::error_code readHeader() override;

llvm/include/llvm/ProfileData/SampleProfWriter.h

protected:
   // Helper function to write name table.
   virtual std::error_code writeNameTable() override;
   virtual std::error_code
   writeContextIdx(const SampleContext &Context) override;
   std::error_code writeCSNameIdx(const SampleContext &Context);
   std::error_code writeCSNameTableSection();
 
   std::error_code writeFuncMetadata(const SampleProfileMap &Profiles);
+  std::error_code writeFuncMetadata(const FunctionSamples &Profile);
 
   // Functions to write various kinds of sections.
   std::error_code writeNameTableSection(const SampleProfileMap &ProfileMap);
   std::error_code writeFuncOffsetTable();
   std::error_code writeProfileSymbolListSection();
 
   SectionLayout SecLayout = DefaultLayout;
   // Specifiy the order of sections in section header table. Note

llvm/include/llvm/Transforms/IPO/ProfiledCallGraph.h

 };
 
 class ProfiledCallGraph {
 public:
   using iterator = std::set<ProfiledCallGraphEdge>::iterator;
 
   // Constructor for non-CS profile.
   ProfiledCallGraph(SampleProfileMap &ProfileMap) {
-    assert(!FunctionSamples::ProfileIsCS && "CS profile is not handled here");
+    assert(!FunctionSamples::ProfileIsCSFlat &&
+           "CS flat profile is not handled here");
     for (const auto &Samples : ProfileMap) {
       addProfiledCalls(Samples.second);
     }
   }
 
   // Constructor for CS profile.
   ProfiledCallGraph(SampleContextTracker &ContextTracker) {
     // BFS traverse the context profile trie to add call edges for calls shown

llvm/include/llvm/Transforms/IPO/SampleContextTracker.h

public:
   void addFunctionSize(uint32_t FSize);
   LineLocation getCallSiteLoc() const;
   ContextTrieNode *getParentContext() const;
   void setParentContext(ContextTrieNode *Parent);
   void dumpNode();
   void dumpTree();
 
 private:
-  static uint64_t nodeHash(StringRef ChildName, const LineLocation &Callsite);
-
   // Map line+discriminator location to child context
   std::map<uint64_t, ContextTrieNode> AllChildContext;
 
   // Link to parent context node
   ContextTrieNode *ParentContext;
 
   // Function name for current context
   StringRef FuncName;

llvm/lib/ProfileData/ProfileSummaryBuilder.cpp

SampleProfileSummaryBuilder::computeSummaryForProfiles(
   sampleprof::SampleProfileMap ContextLessProfiles;
   const sampleprof::SampleProfileMap *ProfilesToUse = &Profiles;
   // For CSSPGO, context-sensitive profile effectively split a function profile
   // into many copies each representing the CFG profile of a particular calling
   // context. That makes the count distribution looks more flat as we now have
   // more function profiles each with lower counts, which in turn leads to lower
   // hot thresholds. To compensate for that, by default we merge context
   // profiles before computing profile summary.
-  if (UseContextLessSummary || (sampleprof::FunctionSamples::ProfileIsCS &&
+  if (UseContextLessSummary || (sampleprof::FunctionSamples::ProfileIsCSFlat &&
                                 !UseContextLessSummary.getNumOccurrences())) {
     for (const auto &I : Profiles) {
       ContextLessProfiles[I.second.getName()].merge(I.second);
     }
     ProfilesToUse = &ContextLessProfiles;
   }
 
   for (const auto &I : *ProfilesToUse) {

llvm/lib/ProfileData/SampleProf.cpp

 using namespace llvm;
 using namespace sampleprof;
 
 static cl::opt<uint64_t> ProfileSymbolListCutOff(
     "profile-symbol-list-cutoff", cl::Hidden, cl::init(-1), cl::ZeroOrMore,
     cl::desc("Cutoff value about how many symbols in profile symbol list "
              "will be used. This is very useful for performance debugging"));
 
+cl::opt<bool> GenerateMergedBaseProfiles(
+    "generate-merged-base-profiles", cl::init(true), cl::ZeroOrMore,
+    cl::desc("When generating nested context-sensitive profiles, always "
+             "generate extra base profile for function with all its context "
+             "profiles merged into it."));
+
 namespace llvm {
 namespace sampleprof {
 SampleProfileFormat FunctionSamples::Format;
 bool FunctionSamples::ProfileIsProbeBased = false;
-bool FunctionSamples::ProfileIsCS = false;
+bool FunctionSamples::ProfileIsCSFlat = false;
+bool FunctionSamples::ProfileIsCSNested = false;
 bool FunctionSamples::UseMD5 = false;
 bool FunctionSamples::HasUniqSuffix = true;
 bool FunctionSamples::ProfileIsFS = false;
 } // namespace sampleprof
 } // namespace llvm
 
 namespace {
 
void sampleprof::sortFuncProfiles(
   });
 }
 
 unsigned FunctionSamples::getOffset(const DILocation *DIL) {
   return (DIL->getLine() - DIL->getScope()->getSubprogram()->getLine()) &
       0xffff;
 }
 
-LineLocation FunctionSamples::getCallSiteIdentifier(const DILocation *DIL) {
-  if (FunctionSamples::ProfileIsProbeBased)
+LineLocation FunctionSamples::getCallSiteIdentifier(const DILocation *DIL,
+                                                    bool ProfileIsFS) {
+  if (FunctionSamples::ProfileIsProbeBased) {
     // In a pseudo-probe based profile, a callsite is simply represented by the
     // ID of the probe associated with the call instruction. The probe ID is
     // encoded in the Discriminator field of the call instruction's debug
     // metadata.
     return LineLocation(PseudoProbeDwarfDiscriminator::extractProbeIndex(
                             DIL->getDiscriminator()),
                         0);
-  else
-    return LineLocation(FunctionSamples::getOffset(DIL),
-                        DIL->getBaseDiscriminator());
+  } else {
+    unsigned Discriminator =
+        ProfileIsFS ? DIL->getDiscriminator() : DIL->getBaseDiscriminator();
+    return LineLocation(FunctionSamples::getOffset(DIL), Discriminator);
+  }
+}
+
+uint64_t FunctionSamples::getCallSiteHash(StringRef CalleeName,
+                                          const LineLocation &Callsite) {
+  uint64_t NameHash = std::hash<std::string>{}(CalleeName.str());
+  uint64_t LocId =
+      (((uint64_t)Callsite.LineOffset) << 32) | Callsite.Discriminator;
+  return NameHash + (LocId << 5) + LocId;
 }
 
 const FunctionSamples *FunctionSamples::findFunctionSamples(
     const DILocation *DIL, SampleProfileReaderItaniumRemapper *Remapper) const {
   assert(DIL);
   SmallVector<std::pair<LineLocation, StringRef>, 10> S;
 
   const DILocation *PrevDIL = DIL;
   for (DIL = DIL->getInlinedAt(); DIL; DIL = DIL->getInlinedAt()) {
-    unsigned Discriminator;
-    if (ProfileIsFS)
-      Discriminator = DIL->getDiscriminator();
-    else
-      Discriminator = DIL->getBaseDiscriminator();
-
     // Use C++ linkage name if possible.
     StringRef Name = PrevDIL->getScope()->getSubprogram()->getLinkageName();
     if (Name.empty())
       Name = PrevDIL->getScope()->getSubprogram()->getName();
-
-    S.push_back(
-        std::make_pair(LineLocation(getOffset(DIL), Discriminator), Name));
+    S.emplace_back(FunctionSamples::getCallSiteIdentifier(
+                       DIL, FunctionSamples::ProfileIsFS),
+                   Name);
     PrevDIL = DIL;
   }
+
   if (S.size() == 0)
     return this;
   const FunctionSamples *FS = this;
   for (int i = S.size() - 1; i >= 0 && FS != nullptr; i--) {
     FS = FS->findFunctionSamplesAt(S[i].first, S[i].second, Remapper);
   }
   return FS;
 }

 void ProfileSymbolList::dump(raw_ostream &OS) const {
   OS << "======== Dump profile symbol list ========\n";
   std::vector<StringRef> SortedList(Syms.begin(), Syms.end());
   llvm::sort(SortedList);
 
   for (auto &Sym : SortedList)
     OS << Sym << "\n";
 }
+
+CSProfileConverter::FrameNode *
+CSProfileConverter::FrameNode::getOrCreateChildFrame(
+    const LineLocation &CallSite, StringRef CalleeName) {
+  uint64_t Hash = FunctionSamples::getCallSiteHash(CalleeName, CallSite);
+  auto It = AllChildFrames.find(Hash);
+  if (It != AllChildFrames.end()) {
+    assert(It->second.FuncName == CalleeName &&
+           "Hash collision for child context node");
+    return &It->second;
+  }
+
+  AllChildFrames[Hash] = FrameNode(CalleeName, nullptr, CallSite);
+  return &AllChildFrames[Hash];
+}
+
+CSProfileConverter::CSProfileConverter(SampleProfileMap &Profiles)
+    : ProfileMap(Profiles) {
+  for (auto &FuncSample : Profiles) {
+    FunctionSamples *FSamples = &FuncSample.second;
+    auto *NewNode = getOrCreateContextPath(FSamples->getContext());
+    assert(!NewNode->FuncSamples && "New node cannot have sample profile");
+    NewNode->FuncSamples = FSamples;
+  }
+}
+
+CSProfileConverter::FrameNode *
+CSProfileConverter::getOrCreateContextPath(const SampleContext &Context) {
+  auto Node = &RootFrame;
+  LineLocation CallSiteLoc(0, 0);
+  for (auto &Callsite : Context.getContextFrames()) {
+    Node = Node->getOrCreateChildFrame(CallSiteLoc, Callsite.FuncName);
+    CallSiteLoc = Callsite.Location;
+  }
+  return Node;
+}
+
+void CSProfileConverter::convertProfiles(CSProfileConverter::FrameNode &Node) {
+  // Process each child profile. Add each child profile to callsite profile map
+  // of the current node `Node` if `Node` comes with a profile. Otherwise
+  // promote the child profile to a standalone profile.
+  auto *NodeProfile = Node.FuncSamples;
+  for (auto &It : Node.AllChildFrames) {
+    auto &ChildNode = It.second;
+    convertProfiles(ChildNode);
+    auto *ChildProfile = ChildNode.FuncSamples;
+    if (!ChildProfile)
+      continue;
+    SampleContext OrigChildContext = ChildProfile->getContext();
+    // Reset the child context to be contextless.
+    ChildProfile->getContext().setName(OrigChildContext.getName());
+    if (NodeProfile) {
+      // Add child profile to the callsite profile map.
+      auto &SamplesMap = NodeProfile->functionSamplesAt(ChildNode.CallSiteLoc);
+      SamplesMap.emplace(OrigChildContext.getName(), *ChildProfile);
+      NodeProfile->addTotalSamples(ChildProfile->getTotalSamples());
+    }
+
+    // Separate child profile to be a standalone profile, if the current parent
+    // profile doesn't exist. This is a duplicating operation when the child
+    // profile is already incorporated into the parent which is still useful and
+    // thus done optionally. It is seen that duplicating context profiles into
+    // base profiles improves the code quality for thinlto build by allowing a
+    // profile in the prelink phase for to-be-fully-inlined functions.
+    if (!NodeProfile || GenerateMergedBaseProfiles)
+      ProfileMap[ChildProfile->getContext()].merge(*ChildProfile);
+
+    // Contexts coming with a `ContextShouldBeInlined` attribute indicate this
+    // is a preinliner-computed profile.
+    if (OrigChildContext.hasAttribute(ContextShouldBeInlined))
+      FunctionSamples::ProfileIsCSNested = true;
+
+    // Remove the original child profile.
+    ProfileMap.erase(OrigChildContext);
+  }
+}
+
+void CSProfileConverter::convertProfiles() { convertProfiles(RootFrame); }

llvm/lib/ProfileData/SampleProfReader.cpp

static bool ParseLine(const StringRef &Input, LineType &LineTy, uint32_t &Depth,
                       uint32_t &Discriminator, StringRef &CalleeName,
                       DenseMap<StringRef, uint64_t> &TargetCountMap,
                       uint64_t &FunctionHash, uint32_t &Attributes) {
   for (Depth = 0; Input[Depth] == ' '; Depth++)
     ;
   if (Depth == 0)
     return false;
 
-  if (Depth == 1 && Input[Depth] == '!') {
+  if (Input[Depth] == '!') {
     LineTy = LineType::Metadata;
     return parseMetadata(Input.substr(Depth), FunctionHash, Attributes);
   }
 
   size_t n1 = Input.find(':');
   StringRef Loc = Input.substr(Depth, n1 - Depth);
   size_t n2 = Loc.find('.');
   if (n2 == StringRef::npos) {

 /// the expected format.
 ///
 /// \returns true if the file was loaded successfully, false otherwise.
 std::error_code SampleProfileReaderText::readImpl() {
   line_iterator LineIt(*Buffer, /*SkipBlanks=*/true, '#');
   sampleprof_error Result = sampleprof_error::success;
 
   InlineCallStack InlineStack;
-  uint32_t ProbeProfileCount = 0;
+  uint32_t TopLevelProbeProfileCount = 0;
 
   // SeenMetadata tracks whether we have processed metadata for the current
-  // top-level function profile.
-  bool SeenMetadata = false;
+  // top-level or nested function profile.
+  uint32_t DepthMetadata = 0;
 
   ProfileIsFS = ProfileIsFSDisciminator;
   FunctionSamples::ProfileIsFS = ProfileIsFS;
   for (; !LineIt.is_at_eof(); ++LineIt) {
     if ((*LineIt)[(*LineIt).find_first_not_of(' ')] == '#')
       continue;
     // Read the header of each function.
     //
for (; !LineIt.is_at_eof(); ++LineIt) {
     if ((*LineIt)[0] != ' ') {
       uint64_t NumSamples, NumHeadSamples;
       StringRef FName;
       if (!ParseHead(*LineIt, FName, NumSamples, NumHeadSamples)) {
         reportError(LineIt.line_number(),
                     "Expected 'mangled_name:NUM:NUM', found " + *LineIt);
         return sampleprof_error::malformed;
       }
-      SeenMetadata = false;
+      DepthMetadata = 0;
       SampleContext FContext(FName, CSNameTable);
       if (FContext.hasContext())
         ++CSProfileCount;
       Profiles[FContext] = FunctionSamples();
       FunctionSamples &FProfile = Profiles[FContext];
       FProfile.setContext(FContext);
       MergeResult(Result, FProfile.addTotalSamples(NumSamples));
       MergeResult(Result, FProfile.addHeadSamples(NumHeadSamples));
if ((*LineIt)[0] != ' ') {
       if (!ParseLine(*LineIt, LineTy, Depth, NumSamples, LineOffset,
                      Discriminator, FName, TargetCountMap, FunctionHash,
                      Attributes)) {
         reportError(LineIt.line_number(),
                     "Expected 'NUM[.NUM]: NUM[ mangled_name:NUM]*', found " +
                         *LineIt);
         return sampleprof_error::malformed;
       }
-      if (SeenMetadata && LineTy != LineType::Metadata) {
+      if (LineTy != LineType::Metadata && Depth == DepthMetadata) {
         // Metadata must be put at the end of a function profile.
         reportError(LineIt.line_number(),
                     "Found non-metadata after metadata: " + *LineIt);
         return sampleprof_error::malformed;
       }
 
       // Here we handle FS discriminators.
       Discriminator &= getDiscriminatorMask();
 
       while (InlineStack.size() > Depth) {
         InlineStack.pop_back();
       }
       switch (LineTy) {
       case LineType::CallSiteProfile: {
         FunctionSamples &FSamples = InlineStack.back()->functionSamplesAt(
             LineLocation(LineOffset, Discriminator))[std::string(FName)];
         FSamples.setName(FName);
         MergeResult(Result, FSamples.addTotalSamples(NumSamples));
         InlineStack.push_back(&FSamples);
+        DepthMetadata = 0;
         break;
       }
       case LineType::BodyProfile: {
         while (InlineStack.size() > Depth) {
           InlineStack.pop_back();
         }
         FunctionSamples &FProfile = *InlineStack.back();
         for (const auto &name_count : TargetCountMap) {
           MergeResult(Result, FProfile.addCalledTargetSamples(
                                   LineOffset, Discriminator, name_count.first,
                                   name_count.second));
         }
         MergeResult(Result, FProfile.addBodySamples(LineOffset, Discriminator,
                                                     NumSamples));
         break;
       }
       case LineType::Metadata: {
         FunctionSamples &FProfile = *InlineStack.back();
         if (FunctionHash) {
           FProfile.setFunctionHash(FunctionHash);
-          ++ProbeProfileCount;
+          if (Depth == 1)
+            ++TopLevelProbeProfileCount;
         }
-        if (Attributes)
-          FProfile.getContext().setAllAttributes(Attributes);
-        SeenMetadata = true;
+        if (Attributes & (uint32_t)ContextShouldBeInlined)
+          ProfileIsCSNested = true;
+        DepthMetadata = Depth;
         break;
       }
       }
     }
   }
 
   assert((CSProfileCount == 0 || CSProfileCount == Profiles.size()) &&
          "Cannot have both context-sensitive and regular profile");
-  ProfileIsCS = (CSProfileCount > 0);
-  assert((ProbeProfileCount == 0 || ProbeProfileCount == Profiles.size()) &&
+  ProfileIsCSFlat = (CSProfileCount > 0);
+  assert((TopLevelProbeProfileCount == 0 ||
+          TopLevelProbeProfileCount == Profiles.size()) &&
          "Cannot have both probe-based profiles and regular profiles");
-  ProfileIsProbeBased = (ProbeProfileCount > 0);
+  ProfileIsProbeBased = (TopLevelProbeProfileCount > 0);
   FunctionSamples::ProfileIsProbeBased = ProfileIsProbeBased;
-  FunctionSamples::ProfileIsCS = ProfileIsCS;
+  FunctionSamples::ProfileIsCSFlat = ProfileIsCSFlat;
+  FunctionSamples::ProfileIsCSNested = ProfileIsCSNested;
 
   if (Result == sampleprof_error::success)
     computeSummary();
 
   return Result;
 }
 
 bool SampleProfileReaderText::hasFormat(const MemoryBuffer &Buffer) {
if (std::error_code EC = ContextIdx.getError())
     return EC;
   if (*ContextIdx >= CSNameTable->size())
     return sampleprof_error::truncated_name_table;
   return (*CSNameTable)[*ContextIdx];
 }
 
 ErrorOr<SampleContext>
 SampleProfileReaderExtBinaryBase::readSampleContextFromTable() {
-  if (ProfileIsCS) {
+  if (ProfileIsCSFlat) {
     auto FContext(readContextFromTable());
     if (std::error_code EC = FContext.getError())
       return EC;
     return SampleContext(*FContext);
   } else {
     auto FName(readStringFromTable());
     if (std::error_code EC = FName.getError())
       return EC;
     return SampleContext(*FName);
   }
 }
 
 std::error_code SampleProfileReaderExtBinaryBase::readOneSection(
     const uint8_t *Start, uint64_t Size, const SecHdrTableEntry &Entry) {
   Data = Start;
   End = Start + Size;
   switch (Entry.Type) {
   case SecProfSummary:
     if (std::error_code EC = readSummary())
       return EC;
     if (hasSecFlag(Entry, SecProfSummaryFlags::SecFlagPartial))
       Summary->setPartialProfile(true);
     if (hasSecFlag(Entry, SecProfSummaryFlags::SecFlagFullContext))
-      FunctionSamples::ProfileIsCS = ProfileIsCS = true;
+      FunctionSamples::ProfileIsCSFlat = ProfileIsCSFlat = true;
     if (hasSecFlag(Entry, SecProfSummaryFlags::SecFlagFSDiscriminator))
       FunctionSamples::ProfileIsFS = ProfileIsFS = true;
     break;
   case SecNameTable: {
     FixedLengthMD5 =
         hasSecFlag(Entry, SecNameTableFlags::SecFlagFixedLengthMD5);
     bool UseMD5 = hasSecFlag(Entry, SecNameTableFlags::SecFlagMD5Name);
     assert((!FixedLengthMD5 || UseMD5) &&
case SecFuncOffsetTable:
     FuncOffsetsOrdered = hasSecFlag(Entry, SecFuncOffsetFlags::SecFlagOrdered);
     if (std::error_code EC = readFuncOffsetTable())
       return EC;
     break;
   case SecFuncMetadata: {
     ProfileIsProbeBased =
         hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagIsProbeBased);
     FunctionSamples::ProfileIsProbeBased = ProfileIsProbeBased;
+    ProfileIsCSNested =
+        hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagIsPreInlined);
+    FunctionSamples::ProfileIsCSNested = ProfileIsCSNested;
     bool HasAttribute =
         hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagHasAttribute);
     if (std::error_code EC = readFuncMetadata(HasAttribute))
       return EC;
     break;
   }
   case SecProfileSymbolList:
     if (std::error_code EC = readProfileSymbolList())
std::error_code SampleProfileReaderExtBinaryBase::readFuncProfiles() {
   } else {
     // Load function profiles on demand.
     if (Remapper) {
       for (auto Name : FuncsToUse) {
         Remapper->insert(Name);
       }
     }
 
-    if (ProfileIsCS) {
+    if (ProfileIsCSFlat) {
       DenseSet<uint64_t> FuncGuidsToUse;
       if (useMD5()) {
         for (auto Name : FuncsToUse)
           FuncGuidsToUse.insert(Function::getGUID(Name));
       }
 
       // For each function in current module, load all context profiles for
       // the function as well as their callee contexts which can help profile
if (ProfileIsCSFlat) {
             return EC;
         }
       }
     }
     Data = End;
   }
   assert((CSProfileCount == 0 || CSProfileCount == Profiles.size()) &&
          "Cannot have both context-sensitive and regular profile");
-  assert((!CSProfileCount || ProfileIsCS) &&
+  assert((!CSProfileCount || ProfileIsCSFlat) &&
          "Section flag should be consistent with actual profile");
   return sampleprof_error::success;
 }
 
 std::error_code SampleProfileReaderExtBinaryBase::readProfileSymbolList() {
   if (!ProfSymList)
     ProfSymList = std::make_unique<ProfileSymbolList>();
 
std::error_code SampleProfileReaderExtBinaryBase::readCSNameTableSec() {
   }
 
   // From this point the underlying object of CSNameTable should be immutable.
   CSNameTable.reset(PNameVec);
   return sampleprof_error::success;
 }
 
 std::error_code
-SampleProfileReaderExtBinaryBase::readFuncMetadata(bool ProfileHasAttribute) {
-  while (Data < End) {
-    auto FContext(readSampleContextFromTable());
-    if (std::error_code EC = FContext.getError())
-      return EC;
 
-    bool ProfileInMap = Profiles.count(*FContext);
+SampleProfileReaderExtBinaryBase::readFuncMetadata(bool ProfileHasAttribute,
+                                                   FunctionSamples *FProfile) {
+  if (Data < End) {
     if (ProfileIsProbeBased) {
       auto Checksum = readNumber<uint64_t>();
       if (std::error_code EC = Checksum.getError())
         return EC;
-      if (ProfileInMap)
-        Profiles[*FContext].setFunctionHash(*Checksum);
+      if (FProfile)
+        FProfile->setFunctionHash(*Checksum);
     }
 
     if (ProfileHasAttribute) {
       auto Attributes = readNumber<uint32_t>();
       if (std::error_code EC = Attributes.getError())
         return EC;
-      if (ProfileInMap)
-        Profiles[*FContext].getContext().setAllAttributes(*Attributes);
+      if (FProfile)
+        FProfile->getContext().setAllAttributes(*Attributes);
     }
+
+    if (!ProfileIsCSFlat) {
+      // Read all the attributes for inlined function calls.
+      auto NumCallsites = readNumber<uint32_t>();
+      if (std::error_code EC = NumCallsites.getError())
+        return EC;
+
+      for (uint32_t J = 0; J < *NumCallsites; ++J) {
+        auto LineOffset = readNumber<uint64_t>();
+        if (std::error_code EC = LineOffset.getError())
+          return EC;
+
+        auto Discriminator = readNumber<uint64_t>();
+        if (std::error_code EC = Discriminator.getError())
+          return EC;
+
+        auto FContext(readSampleContextFromTable());
+        if (std::error_code EC = FContext.getError())
+          return EC;
+
+        FunctionSamples *CalleeProfile = nullptr;
+        if (FProfile) {
+          CalleeProfile = const_cast<FunctionSamples *>(
+              &FProfile->functionSamplesAt(LineLocation(
+                  *LineOffset,
+                  *Discriminator))[std::string(FContext.get().getName())]);
+        }
+        if (std::error_code EC =
+                readFuncMetadata(ProfileHasAttribute, CalleeProfile))
+          return EC;
+      }
+    }
+  }
+
+  return sampleprof_error::success;
+}
+
+std::error_code
+SampleProfileReaderExtBinaryBase::readFuncMetadata(bool ProfileHasAttribute) {
+  while (Data < End) {
+    auto FContext(readSampleContextFromTable());
+    if (std::error_code EC = FContext.getError())
+      return EC;
+    FunctionSamples *FProfile = nullptr;
+    auto It = Profiles.find(*FContext);
+    if (It != Profiles.end())
+      FProfile = &It->second;
+
+    if (std::error_code EC = readFuncMetadata(ProfileHasAttribute, FProfile))
+      return EC;
   }
 
   assert(Data == End && "More data is read than expected");
   return sampleprof_error::success;
 }
 
 std::error_code SampleProfileReaderCompactBinary::readNameTable() {
   auto Size = readNumber<uint64_t>();
case SecFuncOffsetTable:
     if (hasSecFlag(Entry, SecFuncOffsetFlags::SecFlagOrdered))
       Flags.append("ordered,");
     break;
   case SecFuncMetadata:
     if (hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagIsProbeBased))
       Flags.append("probe,");
     if (hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagHasAttribute))
       Flags.append("attr,");
+    if (hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagIsPreInlined))
+      Flags.append("preinlined,");
     break;
   default:
     break;
   }
   char &last = Flags.back();
   if (last == ',')
     last = '}';
   else

llvm/lib/ProfileData/SampleProfWriter.cpp

std::error_code SampleProfileWriterExtBinaryBase::writeFuncOffsetTable() {
   // Write out FuncOffsetTable.
   auto WriteItem = [&](const SampleContext &Context, uint64_t Offset) {
     if (std::error_code EC = writeContextIdx(Context))
       return EC;
     encodeULEB128(Offset, OS);
     return (std::error_code)sampleprof_error::success;
   };
 
-  if (FunctionSamples::ProfileIsCS) {
+  if (FunctionSamples::ProfileIsCSFlat) {
     // Sort the contexts before writing them out. This is to help fast load all
     // context profiles for a function as well as their callee contexts which
     // can help profile-guided importing for ThinLTO.
     std::map<SampleContext, uint64_t> OrderedFuncOffsetTable(
         FuncOffsetTable.begin(), FuncOffsetTable.end());
     for (const auto &Entry : OrderedFuncOffsetTable) {
       if (std::error_code EC = WriteItem(Entry.first, Entry.second))
         return EC;
     }
     addSectionFlag(SecFuncOffsetTable, SecFuncOffsetFlags::SecFlagOrdered);
   } else {
     for (const auto &Entry : FuncOffsetTable) {
       if (std::error_code EC = WriteItem(Entry.first, Entry.second))
         return EC;
     }
   }
 
   FuncOffsetTable.clear();
   return sampleprof_error::success;
 }
 
 std::error_code SampleProfileWriterExtBinaryBase::writeFuncMetadata(
+    const FunctionSamples &FunctionProfile) {
+  auto &OS = *OutputStream;
+  if (std::error_code EC = writeContextIdx(FunctionProfile.getContext()))
+    return EC;
+
+  if (FunctionSamples::ProfileIsProbeBased)
+    encodeULEB128(FunctionProfile.getFunctionHash(), OS);
+  if (FunctionSamples::ProfileIsCSFlat || FunctionSamples::ProfileIsCSNested) {
+    encodeULEB128(FunctionProfile.getContext().getAllAttributes(), OS);
+  }
+
+  if (!FunctionSamples::ProfileIsCSFlat) {
+    // Recursively emit attributes for all callee samples.
+    uint64_t NumCallsites = 0;
+    for (const auto &J : FunctionProfile.getCallsiteSamples())
+      NumCallsites += J.second.size();
+    encodeULEB128(NumCallsites, OS);
+    for (const auto &J : FunctionProfile.getCallsiteSamples()) {
+      for (const auto &FS : J.second) {
+        LineLocation Loc = J.first;
+        encodeULEB128(Loc.LineOffset, OS);
+        encodeULEB128(Loc.Discriminator, OS);
+        if (std::error_code EC = writeFuncMetadata(FS.second))
+          return EC;
+      }
+    }
+  }
+
+  return sampleprof_error::success;
+}
+
+std::error_code SampleProfileWriterExtBinaryBase::writeFuncMetadata(
     const SampleProfileMap &Profiles) {
-  if (!FunctionSamples::ProfileIsProbeBased && !FunctionSamples::ProfileIsCS)
+  if (!FunctionSamples::ProfileIsProbeBased &&
+      !FunctionSamples::ProfileIsCSFlat && !FunctionSamples::ProfileIsCSNested)
     return sampleprof_error::success;
-  auto &OS = *OutputStream;
   for (const auto &Entry : Profiles) {
-    if (std::error_code EC = writeContextIdx(Entry.second.getContext()))
+    if (std::error_code EC = writeFuncMetadata(Entry.second))
       return EC;
-    if (FunctionSamples::ProfileIsProbeBased)
-      encodeULEB128(Entry.second.getFunctionHash(), OS);
-    if (FunctionSamples::ProfileIsCS)
-      encodeULEB128(Entry.second.getContext().getAllAttributes(), OS);
   }
   return sampleprof_error::success;
 }
 
 std::error_code SampleProfileWriterExtBinaryBase::writeNameTable() {
   if (!UseMD5)
     return SampleProfileWriterBinary::writeNameTable();
 

 
 std::error_code SampleProfileWriterExtBinaryBase::writeOneSection(
     SecType Type, uint32_t LayoutIdx, const SampleProfileMap &ProfileMap) {
   // The setting of SecFlagCompress should happen before markSectionStart.
   if (Type == SecProfileSymbolList && ProfSymList && ProfSymList->toCompress())
     setToCompressSection(SecProfileSymbolList);
   if (Type == SecFuncMetadata && FunctionSamples::ProfileIsProbeBased)
     addSectionFlag(SecFuncMetadata, SecFuncMetadataFlags::SecFlagIsProbeBased);
-  if (Type == SecProfSummary && FunctionSamples::ProfileIsCS)
-    addSectionFlag(SecProfSummary, SecProfSummaryFlags::SecFlagFullContext);
-  if (Type == SecFuncMetadata && FunctionSamples::ProfileIsCS)
+  if (Type == SecFuncMetadata && FunctionSamples::ProfileIsCSNested)
+    addSectionFlag(SecFuncMetadata, SecFuncMetadataFlags::SecFlagIsPreInlined);
+  if (Type == SecFuncMetadata &&
+      (FunctionSamples::ProfileIsCSFlat || FunctionSamples::ProfileIsCSNested))
     addSectionFlag(SecFuncMetadata, SecFuncMetadataFlags::SecFlagHasAttribute);
+  if (Type == SecProfSummary && FunctionSamples::ProfileIsCSFlat)
+    addSectionFlag(SecProfSummary, SecProfSummaryFlags::SecFlagFullContext);
   if (Type == SecProfSummary && FunctionSamples::ProfileIsFS)
     addSectionFlag(SecProfSummary, SecProfSummaryFlags::SecFlagFSDiscriminator);
 
   uint64_t SectionStart = markSectionStart(Type, LayoutIdx);
   switch (Type) {
   case SecProfSummary:
     computeSummary(ProfileMap);
     if (auto EC = writeSummary())

 /// Note: it may be tempting to implement this in terms of
 /// FunctionSamples::print().  Please don't.  The dump functionality is intended
 /// for debugging and has no specified form.
 ///
 /// The format used here is more structured and deliberate because
 /// it needs to be parsed by the SampleProfileReaderText class.
 std::error_code SampleProfileWriterText::writeSample(const FunctionSamples &S) {
   auto &OS = *OutputStream;
-  if (FunctionSamples::ProfileIsCS)
+  if (FunctionSamples::ProfileIsCSFlat)
     OS << "[" << S.getContext().toString() << "]:" << S.getTotalSamples();
   else
     OS << S.getName() << ":" << S.getTotalSamples();
 
   if (Indent == 0)
     OS << ":" << S.getHeadSamples();
   OS << "\n";
 
for (const auto &FS : I->second) {
         OS << Loc.LineOffset << ": ";
       else
         OS << Loc.LineOffset << "." << Loc.Discriminator << ": ";
       if (std::error_code EC = writeSample(CalleeSamples))
         return EC;
     }
   Indent -= 1;
 
-  if (Indent == 0) {
-    if (FunctionSamples::ProfileIsProbeBased) {
-      OS.indent(Indent + 1);
-      OS << "!CFGChecksum: " << S.getFunctionHash() << "\n";
-    }
-    if (FunctionSamples::ProfileIsCS) {
-      OS.indent(Indent + 1);
-      OS << "!Attributes: " << S.getContext().getAllAttributes() << "\n";
-    }
-  }
+
+  if (S.getContext().getAllAttributes()) {
 
   return sampleprof_error::success;
 }
 
 std::error_code
 SampleProfileWriterBinary::writeContextIdx(const SampleContext &Context) {
   assert(!Context.hasContext() && "cs profile is not supported");
   return writeNameIdx(Context.getName());

 /// \returns an error code indicating the status of the created writer.
 ErrorOr<std::unique_ptr<SampleProfileWriter>>
 SampleProfileWriter::create(std::unique_ptr<raw_ostream> &OS,
                             SampleProfileFormat Format) {
   std::error_code EC;
   std::unique_ptr<SampleProfileWriter> Writer;
 
   // Currently only Text and Extended Binary format are supported for CSSPGO.
-  if ((FunctionSamples::ProfileIsCS || FunctionSamples::ProfileIsProbeBased) &&
+  if ((FunctionSamples::ProfileIsCSFlat ||
+       FunctionSamples::ProfileIsProbeBased) &&
       (Format == SPF_Binary || Format == SPF_Compact_Binary))
     return sampleprof_error::unsupported_writing_format;
 
   if (Format == SPF_Binary)
     Writer.reset(new SampleProfileWriterRawBinary(OS));
   else if (Format == SPF_Ext_Binary)
     Writer.reset(new SampleProfileWriterExtBinary(OS));
   else if (Format == SPF_Compact_Binary)

llvm/lib/Transforms/IPO/SampleContextTracker.cpp

 
 namespace llvm {
 
 ContextTrieNode *ContextTrieNode::getChildContext(const LineLocation &CallSite,
                                                   StringRef CalleeName) {
   if (CalleeName.empty())
     return getHottestChildContext(CallSite);
 
-  uint64_t Hash = nodeHash(CalleeName, CallSite);
+  uint64_t Hash = FunctionSamples::getCallSiteHash(CalleeName, CallSite);
   auto It = AllChildContext.find(Hash);
   if (It != AllChildContext.end())
     return &It->second;
   return nullptr;
 }
 
 ContextTrieNode *
 ContextTrieNode::getHottestChildContext(const LineLocation &CallSite) {
ContextTrieNode::getHottestChildContext(const LineLocation &CallSite) {
   }
 
   return ChildNodeRet;
 }
 
 ContextTrieNode &ContextTrieNode::moveToChildContext(
     const LineLocation &CallSite, ContextTrieNode &&NodeToMove,
     uint32_t ContextFramesToRemove, bool DeleteNode) {
-  uint64_t Hash = nodeHash(NodeToMove.getFuncName(), CallSite);
+  uint64_t Hash =
+      FunctionSamples::getCallSiteHash(NodeToMove.getFuncName(), CallSite);
   assert(!AllChildContext.count(Hash) && "Node to remove must exist");
   LineLocation OldCallSite = NodeToMove.CallSiteLoc;
   ContextTrieNode &OldParentContext = *NodeToMove.getParentContext();
   AllChildContext[Hash] = NodeToMove;
   ContextTrieNode &NewNode = AllChildContext[Hash];
   NewNode.CallSiteLoc = CallSite;
 
   // Walk through nodes in the moved the subtree, and update
ContextTrieNode &ContextTrieNode::moveToChildContext(
   if (DeleteNode)
     OldParentContext.removeChildContext(OldCallSite, NewNode.getFuncName());
 
   return NewNode;
 }
 
 void ContextTrieNode::removeChildContext(const LineLocation &CallSite,
                                          StringRef CalleeName) {
-  uint64_t Hash = nodeHash(CalleeName, CallSite);
+  uint64_t Hash = FunctionSamples::getCallSiteHash(CalleeName, CallSite);
   // Note this essentially calls dtor and destroys that child context
   AllChildContext.erase(Hash);
 }
 
 std::map<uint64_t, ContextTrieNode> &ContextTrieNode::getAllChildContext() {
   return AllChildContext;
 }
 
while (!NodeQueue.empty()) {
 
     for (auto &It : Node->getAllChildContext()) {
       ContextTrieNode *ChildNode = &It.second;
       NodeQueue.push(ChildNode);
     }
   }
 }
 
-uint64_t ContextTrieNode::nodeHash(StringRef ChildName,
-                                   const LineLocation &Callsite) {
-  // We still use child's name for child hash, this is
-  // because for children of root node, we don't have
-  // different line/discriminator, and we'll rely on name
-  // to differentiate children.
-  uint64_t NameHash = std::hash<std::string>{}(ChildName.str());
-  uint64_t LocId =
-      (((uint64_t)Callsite.LineOffset) << 32) | Callsite.Discriminator;
-  return NameHash + (LocId << 5) + LocId;
-}
-
 ContextTrieNode *ContextTrieNode::getOrCreateChildContext(
     const LineLocation &CallSite, StringRef CalleeName, bool AllowCreate) {
-  uint64_t Hash = nodeHash(CalleeName, CallSite);
+  uint64_t Hash = FunctionSamples::getCallSiteHash(CalleeName, CallSite);
   auto It = AllChildContext.find(Hash);
   if (It != AllChildContext.end()) {
     assert(It->second.getFuncName() == CalleeName &&
            "Hash collision for child context node");
     return &It->second;
   }
 
   if (!AllowCreate)

llvm/lib/Transforms/IPO/SampleProfile.cpp

protected:
   bool
   inlineHotFunctionsWithPriority(Function &F,
                                  DenseSet<GlobalValue::GUID> &InlinedGUIDs);
   // Inline cold/small functions in addition to hot ones
   bool shouldInlineColdCallee(CallBase &CallInst);
   void emitOptimizationRemarksForInlineCandidates(
       const SmallVectorImpl<CallBase *> &Candidates, const Function &F,
       bool Hot);
+  void promoteMergeNotInlinedContextSamples(
+      DenseMap<CallBase *, const FunctionSamples *> NonInlinedCallSites,
+      const Function &F);
   std::vector<Function *> buildFunctionOrder(Module &M, CallGraph *CG);
   std::unique_ptr<ProfiledCallGraph> buildProfiledCallGraph(CallGraph &CG);
   void generateMDProfMetadata(Function &F);
 
   /// Map from function name to Function *. Used to find the function from
   /// the function name. If the function name contains suffix, additional
   /// entry is added to map from the stripped name to the function if there
   /// is one-to-one mapping.
   StringMap<Function *> SymbolMap;
 
   std::function<AssumptionCache &(Function &)> GetAC;
   std::function<TargetTransformInfo &(Function &)> GetTTI;
   std::function<const TargetLibraryInfo &(Function &)> GetTLI;
 
   /// Profile tracker for different context.
   std::unique_ptr<SampleContextTracker> ContextTracker;
 
   /// Flag indicating whether input profile is context-sensitive
-  bool ProfileIsCS = false;
+  bool ProfileIsCSFlat = false;
 
   /// Flag indicating which LTO/ThinLTO phase the pass is invoked in.
   ///
   /// We need to know the LTO phase because for example in ThinLTOPrelink
   /// phase, in annotation, we should not promote indirect calls. Instead,
   /// we will mark GUIDs that needs to be annotated to the function.
   ThinOrFullLTOPhase LTOPhase;
 
if (isa<BranchInst>(Inst) || isa<IntrinsicInst>(Inst) || isa<PHINode>(Inst))
     return std::error_code();
 
   // For non-CS profile, if a direct call/invoke instruction is inlined in
   // profile (findCalleeFunctionSamples returns non-empty result), but not
   // inlined here, it means that the inlined callsite has no sample, thus the
   // call instruction should have 0 count.
   // For CS profile, the callsite count of previously inlined callees is
   // populated with the entry count of the callees.
-  if (!ProfileIsCS)
+  if (!ProfileIsCSFlat)
     if (const auto *CB = dyn_cast<CallBase>(&Inst))
       if (!CB->isIndirectCall() && findCalleeFunctionSamples(*CB))
         return 0;
 
   return getInstWeightImpl(Inst);
 }
 
 // Here use error_code to represent: 1) The dangling probe. 2) Ignore the weight
if (!FS)
     return 0;
 
   // For non-CS profile, If a direct call/invoke instruction is inlined in
   // profile (findCalleeFunctionSamples returns non-empty result), but not
   // inlined here, it means that the inlined callsite has no sample, thus the
   // call instruction should have 0 count.
   // For CS profile, the callsite count of previously inlined callees is
   // populated with the entry count of the callees.
-  if (!ProfileIsCS)
+  if (!ProfileIsCSFlat)
     if (const auto *CB = dyn_cast<CallBase>(&Inst))
       if (!CB->isIndirectCall() && findCalleeFunctionSamples(*CB))
         return 0;
 
   const ErrorOr<uint64_t> &R = FS->findSamplesAt(Probe->Id, 0);
   if (R) {
     uint64_t Samples = R.get() * Probe->Factor;
     bool FirstMark = CoverageTracker.markSamplesUsed(FS, Probe->Id, 0, Samples);
SampleProfileLoader::findCalleeFunctionSamples(const CallBase &Inst) const {
   if (!DIL) {
     return nullptr;
   }
 
   StringRef CalleeName;
   if (Function *Callee = Inst.getCalledFunction())
     CalleeName = Callee->getName();
 
-  if (ProfileIsCS)
+  if (ProfileIsCSFlat)
     return ContextTracker->getCalleeContextSamplesFor(Inst, CalleeName);
 
   const FunctionSamples *FS = findFunctionSamples(Inst);
   if (FS == nullptr)
     return nullptr;
 
   return FS->findFunctionSamplesAt(FunctionSamples::getCallSiteIdentifier(DIL),
                                    CalleeName, Reader->getRemapper());
SampleProfileLoader::findIndirectCallFunctionSamples(
   auto FSCompare = [](const FunctionSamples *L, const FunctionSamples *R) {
     assert(L && R && "Expect non-null FunctionSamples");
     if (L->getEntrySamples() != R->getEntrySamples())
       return L->getEntrySamples() > R->getEntrySamples();
     return FunctionSamples::getGUID(L->getName()) <
            FunctionSamples::getGUID(R->getName());
   };
 
-  if (ProfileIsCS) {
+  if (ProfileIsCSFlat) {
     auto CalleeSamples =
         ContextTracker->getIndirectCalleeContextSamplesFor(DIL);
     if (CalleeSamples.empty())
       return R;
 
     // For CSSPGO, we only use target context profile's entry count
     // as that already includes both inlined callee and non-inlined ones..
     Sum = 0;
SampleProfileLoader::findFunctionSamples(const Instruction &Inst) const {
   }
 
   const DILocation *DIL = Inst.getDebugLoc();
   if (!DIL)
     return Samples;
 
   auto it = DILocation2SampleMap.try_emplace(DIL,nullptr);
   if (it.second) {
-    if (ProfileIsCS)
+    if (ProfileIsCSFlat)
       it.first->second = ContextTracker->getContextSamplesFor(DIL);
     else
       it.first->second =
           Samples->findFunctionSamples(DIL, Reader->getRemapper());
   }
   return it.first->second;
 }
 
if (CB && getExternalInlineAdvisorShouldInline(*CB)) {
     // Otherwise, drop the threshold to import everything that we can
     Threshold = 0;
   }
 
   assert(Samples && "expect non-null caller profile");
 
   // For AutoFDO profile, retrieve candidate profiles by walking over
   // the nested inlinee profiles.
-  if (!ProfileIsCS) {
+  if (!ProfileIsCSFlat) {
     Samples->findInlinedFunctions(InlinedGUIDs, SymbolMap, Threshold);
     return;
   }
 
   ContextTrieNode *Caller =
       ContextTracker->getContextFor(Samples->getContext());
   std::queue<ContextTrieNode *> CalleeList;
   CalleeList.push(Caller);
for (auto &BB : F) {
       for (auto &I : BB.getInstList()) {
         const FunctionSamples *FS = nullptr;
         if (auto *CB = dyn_cast<CallBase>(&I)) {
           if (!isa<IntrinsicInst>(I)) {
             if ((FS = findCalleeFunctionSamples(*CB))) {
               assert((!FunctionSamples::UseMD5 || FS->GUIDToFuncNameMap) &&
                      "GUIDToFuncNameMap has to be populated");
               AllCandidates.push_back(CB);
-              if (FS->getEntrySamples() > 0 || ProfileIsCS)
+              if (FS->getEntrySamples() > 0 || ProfileIsCSFlat)
                 LocalNotInlinedCallSites.try_emplace(CB, FS);
               if (callsiteIsHot(FS, PSI, ProfAccForSymsInList))
                 Hot = true;
               else if (shouldInlineColdCallee(*CB))
                 ColdCandidates.push_back(CB);
             } else if (getExternalInlineAdvisorShouldInline(*CB)) {
               AllCandidates.push_back(CB);
             }
for (CallBase *I : CIS) {
                                     InlinedGUIDs, SymbolMap,
                                     PSI->getOrCompHotCountThreshold());
       }
     }
     Changed |= LocalChanged;
   }
 
   // For CS profile, profile for not inlined context will be merged when
-  // base profile is being trieved
-  if (ProfileIsCS)
-    return Changed;
-
-  // Accumulate not inlined callsite information into notInlinedSamples
-  for (const auto &Pair : LocalNotInlinedCallSites) {
-    CallBase *I = Pair.getFirst();
-    Function *Callee = I->getCalledFunction();
-    if (!Callee || Callee->isDeclaration())
-      continue;
-
-    ORE->emit(OptimizationRemarkAnalysis(CSINLINE_DEBUG, "NotInline",
-                                         I->getDebugLoc(), I->getParent())
-              << "previous inlining not repeated: '"
-              << ore::NV("Callee", Callee) << "' into '"
-              << ore::NV("Caller", &F) << "'");
-
-    ++NumCSNotInlined;
-    const FunctionSamples *FS = Pair.getSecond();
-    if (FS->getTotalSamples() == 0 && FS->getEntrySamples() == 0) {
-      continue;
-    }
-
-    if (ProfileMergeInlinee) {
-      // A function call can be replicated by optimizations like callsite
-      // splitting or jump threading and the replicates end up sharing the
-      // sample nested callee profile instead of slicing the original inlinee's
-      // profile. We want to do merge exactly once by filtering out callee
-      // profiles with a non-zero head sample count.
-      if (FS->getHeadSamples() == 0) {
-        // Use entry samples as head samples during the merge, as inlinees
-        // don't have head samples.
-        const_cast<FunctionSamples *>(FS)->addHeadSamples(
-            FS->getEntrySamples());
-
-        // Note that we have to do the merge right after processing function.
-        // This allows OutlineFS's profile to be used for annotation during
-        // top-down processing of functions' annotation.
-        FunctionSamples *OutlineFS = Reader->getOrCreateSamplesFor(*Callee);
-        OutlineFS->merge(*FS);
-      }
-    } else {
-      auto pair =
-          notInlinedCallInfo.try_emplace(Callee, NotInlinedProfileInfo{0});
-      pair.first->second.entryCount += FS->getEntrySamples();
-    }
-  }
+  // base profile is being retrieved.
+  if (!FunctionSamples::ProfileIsCSFlat)
+    promoteMergeNotInlinedContextSamples(LocalNotInlinedCallSites, F);
   return Changed;
 }
 
 bool SampleProfileLoader::tryInlineCandidate(
     InlineCandidate &Candidate, SmallVector<CallBase *, 8> *InlinedCallSites) {
 
   CallBase &CB = *Candidate.CallInstr;
   Function *CalledFunction = CB.getCalledFunction();
if (InlineFunction(CB, IFI).isSuccess()) {
 
     // Now populate the list of newly exposed call sites.
     if (InlinedCallSites) {
       InlinedCallSites->clear();
       for (auto &I : IFI.InlinedCallSites)
         InlinedCallSites->push_back(I);
     }
 
-    if (ProfileIsCS)
+    if (ProfileIsCSFlat)
       ContextTracker->markContextSamplesInlined(Candidate.CalleeSamples);
     ++NumCSInlined;
 
     // Prorate inlined probes for a duplicated inlining callsite which probably
     // has a distribution less than 100%. Samples for an inlinee should be
     // distributed among the copies of the original callsite based on each
     // callsite's distribution factor for counts accuracy. Note that an inlined
     // probe may come with its own distribution factor if it has been duplicated
SampleProfileLoader::shouldInlineCandidate(InlineCandidate &Candidate) {
 
   // Otherwise only use the cost from call analyzer, but overwite threshold with
   // Sample PGO threshold.
   return InlineCost::get(Cost.getCost(), SampleThreshold);
 }
 
 bool SampleProfileLoader::inlineHotFunctionsWithPriority(
     Function &F, DenseSet<GlobalValue::GUID> &InlinedGUIDs) {
-  assert(ProfileIsCS && "Prioritiy based inliner only works with CSSPGO now");
 
   // ProfAccForSymsInList is used in callsiteIsHot. The assertion makes sure
   // Profile symbol list is ignored when profile-sample-accurate is on.
   assert((!ProfAccForSymsInList ||
           (!ProfileSampleAccurate &&
            !F.hasFnAttribute("profile-sample-accurate"))) &&
          "ProfAccForSymsInList should be false when profile-sample-accurate "
          "is enabled");
assert(ProfileInlineLimitMax >= ProfileInlineLimitMin &&
          "Max inline size limit should not be smaller than min inline size "
          "limit.");
   unsigned SizeLimit = F.getInstructionCount() * ProfileInlineGrowthLimit;
   SizeLimit = std::min(SizeLimit, (unsigned)ProfileInlineLimitMax);
   SizeLimit = std::max(SizeLimit, (unsigned)ProfileInlineLimitMin);
   if (ExternalInlineAdvisor)
     SizeLimit = std::numeric_limits<unsigned>::max();
 
+  DenseMap<CallBase *, const FunctionSamples *> LocalNotInlinedCallSites;
+
   // Perform iterative BFS call site prioritized inlining
   bool Changed = false;
   while (!CQueue.empty() && F.getInstructionCount() < SizeLimit) {
     InlineCandidate Candidate = CQueue.top();
     CQueue.pop();
     CallBase *I = Candidate.CallInstr;
     Function *CalledFunction = I->getCalledFunction();
 
if (I->isIndirectCall()) {
         if (tryPromoteAndInlineCandidate(F, Candidate, SumOrigin, Sum,
                                          &InlinedCallSites)) {
           for (auto *CB : InlinedCallSites) {
             if (getInlineCandidate(&NewCandidate, CB))
               CQueue.emplace(NewCandidate);
           }
           ICPCount++;
           Changed = true;
+        } else if (!ContextTracker) {
+          LocalNotInlinedCallSites.try_emplace(I, FS);
         }
       }
     } else if (CalledFunction && CalledFunction->getSubprogram() &&
                !CalledFunction->isDeclaration()) {
       SmallVector<CallBase *, 8> InlinedCallSites;
       if (tryInlineCandidate(Candidate, &InlinedCallSites)) {
         for (auto *CB : InlinedCallSites) {
           if (getInlineCandidate(&NewCandidate, CB))
             CQueue.emplace(NewCandidate);
         }
         Changed = true;
+      } else if (!ContextTracker) {
+        LocalNotInlinedCallSites.try_emplace(I, Candidate.CalleeSamples);
       }
     } else if (LTOPhase == ThinOrFullLTOPhase::ThinLTOPreLink) {
       findExternalInlineCandidate(I, findCalleeFunctionSamples(*I),
                                   InlinedGUIDs, SymbolMap,
                                   PSI->getOrCompHotCountThreshold());
     }
   }
 
   if (!CQueue.empty()) {
     if (SizeLimit == (unsigned)ProfileInlineLimitMax)
       ++NumCSInlinedHitMaxLimit;
     else if (SizeLimit == (unsigned)ProfileInlineLimitMin)
       ++NumCSInlinedHitMinLimit;
     else
       ++NumCSInlinedHitGrowthLimit;
   }
 
+  // For CS profile, profile for not inlined context will be merged when
+  // base profile is being retrieved.
+  if (!FunctionSamples::ProfileIsCSFlat)
+    promoteMergeNotInlinedContextSamples(LocalNotInlinedCallSites, F);
   return Changed;
 }
 
+void SampleProfileLoader::promoteMergeNotInlinedContextSamples(
+    DenseMap<CallBase *, const FunctionSamples *> NonInlinedCallSites,
+    const Function &F) {
+  // Accumulate not inlined callsite information into notInlinedSamples
+  for (const auto &Pair : NonInlinedCallSites) {
+    CallBase *I = Pair.getFirst();
+    Function *Callee = I->getCalledFunction();
+    if (!Callee || Callee->isDeclaration())
+      continue;
+
+    ORE->emit(OptimizationRemarkAnalysis(CSINLINE_DEBUG, "NotInline",
+                                         I->getDebugLoc(), I->getParent())
+              << "previous inlining not repeated: '"
+              << ore::NV("Callee", Callee) << "' into '"
+              << ore::NV("Caller", &F) << "'");
+
+    ++NumCSNotInlined;
+    const FunctionSamples *FS = Pair.getSecond();
+    if (FS->getTotalSamples() == 0 && FS->getEntrySamples() == 0) {
+      continue;
+    }
+
+    if (ProfileMergeInlinee) {
+      // A function call can be replicated by optimizations like callsite
+      // splitting or jump threading and the replicates end up sharing the
+      // sample nested callee profile instead of slicing the original
+      // inlinee's profile. We want to do merge exactly once by filtering out
+      // callee profiles with a non-zero head sample count.
+      if (FS->getHeadSamples() == 0) {
+        // Use entry samples as head samples during the merge, as inlinees
+        // don't have head samples.
+        const_cast<FunctionSamples *>(FS)->addHeadSamples(
+            FS->getEntrySamples());
+
+        // Note that we have to do the merge right after processing function.
+        // This allows OutlineFS's profile to be used for annotation during
+        // top-down processing of functions' annotation.
+        FunctionSamples *OutlineFS = Reader->getOrCreateSamplesFor(*Callee);
+        OutlineFS->merge(*FS, 1);
+        // Set outlined profile to be synthetic to not bias the inliner.
+        OutlineFS->SetContextSynthetic();
+      }
+    } else {
+      auto pair =
+          notInlinedCallInfo.try_emplace(Callee, NotInlinedProfileInfo{0});
+      pair.first->second.entryCount += FS->getEntrySamples();
+    }
+  }
+}
+
 /// Returns the sorted CallTargetMap \p M by count in descending order.
 static SmallVector<InstrProfValueData, 2>
 GetSortedValueDataFromCallTargets(const SampleRecord::CallTargetMap &M) {
   SmallVector<InstrProfValueData, 2> R;
   for (const auto &I : SampleRecord::SortCallTargets(M)) {
     R.emplace_back(
         InstrProfValueData{FunctionSamples::getGUID(I.first), I.second});
   }
if (BlockWeights[BB]) {
           SmallVector<InstrProfValueData, 2> SortedCallTargets =
               GetSortedValueDataFromCallTargets(T.get());
           uint64_t Sum = 0;
           for (const auto &C : T.get())
             Sum += C.second;
           // With CSSPGO all indirect call targets are counted torwards the
           // original indirect call site in the profile, including both
           // inlined and non-inlined targets.
-          if (!FunctionSamples::ProfileIsCS) {
+          if (!FunctionSamples::ProfileIsCSFlat) {
             if (const FunctionSamplesMap *M =
                     FS->findFunctionSamplesMapAt(CallSite)) {
               for (const auto &NameFS : *M)
                 Sum += NameFS.second.getEntrySamples();
             }
           }
           if (Sum)
             updateIDTMetaData(I, SortedCallTargets, Sum);
if (FunctionSamples::ProfileIsProbeBased) {
     if (getFunctionLoc(F) == 0)
       return false;
 
     LLVM_DEBUG(dbgs() << "Line number for the first instruction in "
                       << F.getName() << ": " << getFunctionLoc(F) << "\n");
   }
 
   DenseSet<GlobalValue::GUID> InlinedGUIDs;
-  if (ProfileIsCS && CallsitePrioritizedInline)
+  if (CallsitePrioritizedInline)
     Changed |= inlineHotFunctionsWithPriority(F, InlinedGUIDs);
   else
     Changed |= inlineHotFunctions(F, InlinedGUIDs);
 
   Changed |= computeAndPropagateWeights(F, InlinedGUIDs);
 
   if (Changed)
     generateMDProfMetadata(F);

 INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(ProfileSummaryInfoWrapperPass)
 INITIALIZE_PASS_END(SampleProfileLoaderLegacyPass, "sample-profile",
                     "Sample Profile loader", false, false)
 
 std::unique_ptr<ProfiledCallGraph>
 SampleProfileLoader::buildProfiledCallGraph(CallGraph &CG) {
   std::unique_ptr<ProfiledCallGraph> ProfiledCG;
-  if (ProfileIsCS)
+  if (ProfileIsCSFlat)
     ProfiledCG = std::make_unique<ProfiledCallGraph>(*ContextTracker);
   else
     ProfiledCG = std::make_unique<ProfiledCallGraph>(Reader->getProfiles());
 
   // Add all functions into the profiled call graph even if they are not in
   // the profile. This makes sure functions missing from the profile still
   // gets a chance to be processed.
   for (auto &Node : CG) {
for (Function &F : M)
       if (!F.isDeclaration() && F.hasFnAttribute("use-sample-profile"))
         FunctionOrderList.push_back(&F);
     return FunctionOrderList;
   }
 
   assert(&CG->getModule() == &M);
 
   if (UseProfiledCallGraph ||
-      (ProfileIsCS && !UseProfiledCallGraph.getNumOccurrences())) {
+      (ProfileIsCSFlat && !UseProfiledCallGraph.getNumOccurrences())) {
     // Use profiled call edges to augment the top-down order. There are cases
     // that the top-down order computed based on the static call graph doesn't
     // reflect real execution order. For example
     //
     // 1. Incomplete static call graph due to unknown indirect call targets.
     //    Adjusting the order by considering indirect call edges from the
     //    profile can enable the inlining of indirect call targets by allowing
     //    the caller processed before them.
ExternalInlineAdvisor = getReplayInlineAdvisor(
         ReplayInlinerSettings{ProfileInlineReplayFile,
                               ProfileInlineReplayScope,
                               ProfileInlineReplayFallback,
                               {ProfileInlineReplayFormat}},
         /*EmitRemarks=*/false);
   }
 
   // Apply tweaks if context-sensitive profile is available.
-  if (Reader->profileIsCS()) {
-    ProfileIsCS = true;
-    FunctionSamples::ProfileIsCS = true;
-
+  if (Reader->profileIsCSFlat() || Reader->profileIsCSNested()) {
+    ProfileIsCSFlat = Reader->profileIsCSFlat();
     // Enable priority-base inliner and size inline by default for CSSPGO.
     if (!ProfileSizeInline.getNumOccurrences())
       ProfileSizeInline = true;
     if (!CallsitePrioritizedInline.getNumOccurrences())
       CallsitePrioritizedInline = true;
 
     // For CSSPGO, use preinliner decision by default when available.
     if (!UsePreInlinerDecision.getNumOccurrences())
       UsePreInlinerDecision = true;
 
     // For CSSPGO, we also allow recursive inline to best use context profile.
     if (!AllowRecursiveInline.getNumOccurrences())
       AllowRecursiveInline = true;
 
     // Enable iterative-BFI by default for CSSPGO.
     if (!UseIterativeBFIInference.getNumOccurrences())
       UseIterativeBFIInference = true;
     // Enable Profi by default for CSSPGO.
     if (!SampleProfileUseProfi.getNumOccurrences())
       SampleProfileUseProfi = true;
 
+    if (FunctionSamples::ProfileIsCSFlat) {
       // Tracker for profiles under different context
       ContextTracker = std::make_unique<SampleContextTracker>(
           Reader->getProfiles(), &GUIDToFuncNameMap);
     }
+  }
 
   // Load pseudo probe descriptors for probe-based function samples.
   if (Reader->profileIsProbeBased()) {
     ProbeManager = std::make_unique<PseudoProbeManager>(M);
     if (!ProbeManager->moduleIsProbed(M)) {
       const char *Msg =
           "Pseudo-probe-based profile requires SampleProfileProbePass";
       Ctx.diagnose(DiagnosticInfoSampleProfile(Filename, Msg));
bool SampleProfileLoader::runOnModule(Module &M, ModuleAnalysisManager *AM,
   bool retval = false;
   for (auto F : buildFunctionOrder(M, CG)) {
     assert(!F->isDeclaration());
     clearFunctionData();
     retval |= runOnFunction(*F, AM);
   }
 
   // Account for cold calls not inlined....
-  if (!ProfileIsCS)
+  if (!ProfileIsCSFlat)
     for (const std::pair<Function *, NotInlinedProfileInfo> &pair :
          notInlinedCallInfo)
       updateProfileCallee(pair.first, pair.second.entryCount);
 
   return retval;
 }
 
 bool SampleProfileLoaderLegacyPass::runOnModule(Module &M) {
auto &FAM =
         AM->getResult<FunctionAnalysisManagerModuleProxy>(*F.getParent())
             .getManager();
     ORE = &FAM.getResult<OptimizationRemarkEmitterAnalysis>(F);
   } else {
     OwnedORE = std::make_unique<OptimizationRemarkEmitter>(&F);
     ORE = OwnedORE.get();
   }
 
-  if (ProfileIsCS)
+  if (ProfileIsCSFlat)
     Samples = ContextTracker->getBaseSamplesFor(F);
   else
     Samples = Reader->getSamplesFor(F);
 
   if (Samples && !Samples->empty())
     return emitAnnotations(F);
   return false;
 }

llvm/test/Transforms/SampleProfile/Inputs/inline-priority.prof

+main:225715:0
+ 2.1: 5553
+ 3: 5391
+ 3.1: _Z3sumii:5860
+  0: 5300
+  1: 5279
+  2: 5279

llvm/test/Transforms/SampleProfile/csspgo-inline.ll

 ; Test for CSSPGO's new early inliner using priority queue
 
 ; Note that we need new pass manager to enable top-down processing for sample profile loader
 ; Test we inlined the following in top-down order with old inliner
 ;   main:3 @ _Z5funcAi
 ;   main:3 @ _Z5funcAi:1 @ _Z8funcLeafi
 ;   _Z5funcBi:1 @ _Z8funcLeafi
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -sample-profile-prioritized-inline=0 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
 
 ; RUN: llvm-profdata merge --sample --extbinary --use-md5 %S/Inputs/profile-context-tracker.prof -o %t.md5
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.md5 -sample-profile-inline-size -sample-profile-prioritized-inline=0 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
 
+; RUN: llvm-profdata merge --sample --text --gen-nested-cs-profile %S/Inputs/profile-context-tracker.prof -o %t.prof
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.prof -sample-profile-inline-size -sample-profile-prioritized-inline=0 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
+
 ; With new FDO early inliner, callee entry count is used to drive inlining instead of callee total samples, so we get less inlining for given profile
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-NEW
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.prof -sample-profile-prioritized-inline -sample-profile-inline-size -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-NEW
 ;
 ; With new FDO early inliner, callee entry count is used to drive inlining instead of callee total samples, tuning hot cutoff can get us the same inlining
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -profile-summary-cutoff-hot=999900 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
 ;
 ; With new FDO early inliner, callee entry count is used to drive inlining instead of callee total samples, tuning cold sample profile inline threshold can get us the same inlining
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -sample-profile-cold-inline-threshold=200 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
 ;
 ; With new FDO early inliner and tuned cutoff, we can control inlining through size growth tuning knob.

llvm/test/Transforms/SampleProfile/csspgo-use-preinliner.ll

 ; Test for CSSPGO's sample loader inlining to make sure it can use llvm-profgen preinliner's decision
 
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/csspgo-use-preinliner.prof -pass-remarks=inline -sample-profile-prioritized-inline -profile-sample-accurate -sample-profile-use-preinliner=0 -S 2>&1 | FileCheck %s --check-prefix=DEFAULT
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/csspgo-use-preinliner.prof -pass-remarks=inline -sample-profile-prioritized-inline -profile-sample-accurate -sample-profile-use-preinliner=1 -S 2>&1 | FileCheck %s --check-prefix=PREINLINE
 
+; RUN: llvm-profdata merge --sample --text --gen-nested-cs-profile -generate-merged-base-profiles=0 %S/Inputs/csspgo-use-preinliner.prof -o %t.prof
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.prof -pass-remarks=inline -sample-profile-prioritized-inline -profile-sample-accurate -sample-profile-use-preinliner=0 -S 2>&1 | FileCheck %s --check-prefix=DEFAULT
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.prof -pass-remarks=inline -sample-profile-prioritized-inline -profile-sample-accurate -sample-profile-use-preinliner=1 -S 2>&1 | FileCheck %s --check-prefix=PREINLINE
 
 ; DEFAULT: '_Z5funcAi' inlined into 'main'
 ; DEFAULT-NOT: inlined into
 
 ; PREINLINE-NOT: inlined into
 ; `[main:3 @ _Z5funcAi]` does not have preinline decision, so it's up for loader inliner to decided.
 ; PREINLINE: '_Z5funcAi' inlined into 'main'
 ; `[main:3 @ _Z5funcAi:1 @ _Z8funcLeafi]` is inlined according to preinline decision.

llvm/test/Transforms/SampleProfile/inline-mergeprof.ll

 ; Test we lose details of not inlined profile without '-sample-profile-merge-inlinee'
 ; RUN: opt < %s -sample-profile -sample-profile-file=%S/Inputs/inline-mergeprof.prof -sample-profile-merge-inlinee=false -use-profiled-call-graph=0 -enable-new-pm=0 -S | FileCheck -check-prefix=SCALE %s
 ; RUN: opt < %s -sample-profile -sample-profile-file=%S/Inputs/inline-mergeprof.prof -sample-profile-merge-inlinee=true -use-profiled-call-graph=0 -enable-new-pm=0 -S | FileCheck -check-prefix=SCALE %s
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-mergeprof.prof -sample-profile-merge-inlinee=false -use-profiled-call-graph=0 -S | FileCheck -check-prefix=SCALE %s
 
 ; Test we properly merge not inlined profile with '-sample-profile-merge-inlinee'
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-mergeprof.prof -sample-profile-merge-inlinee=true -use-profiled-call-graph=0 -S | FileCheck -check-prefix=MERGE  %s
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-mergeprof.prof -sample-profile-merge-inlinee=true -use-profiled-call-graph=0 -sample-profile-prioritized-inline=1 -S | FileCheck -check-prefix=MERGE  %s
 
 ; Test we properly merge not inlined profile with '-sample-profile-merge-inlinee'
 ; when the profile uses md5.
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-mergeprof.md5.prof -sample-profile-merge-inlinee=true -use-profiled-call-graph=0 -S | FileCheck -check-prefix=MERGE  %s
 
 @.str = private unnamed_addr constant [11 x i8] c"sum is %d\0A\00", align 1
 
 define i32 @main() #0 !dbg !6 {

llvm/test/Transforms/SampleProfile/inline.ll

 ; RUN: opt < %s -sample-profile -sample-profile-file=%S/Inputs/inline.prof -S | FileCheck %s
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline.prof -S | FileCheck %s
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-priority.prof -sample-profile-prioritized-inline=1 -S | FileCheck %s
 
 ; Original C++ test case
 ;
 ; #include <stdio.h>
 ;
 ; int sum(int x, int y) {
 ;   return x + y;
 ; }

llvm/test/tools/llvm-profdata/Inputs/cs-sample-preinline-probe.proftext

 [main:3 @ _Z5funcAi:1 @ _Z8funcLeafi]:1467299:11
  0: 6
  1: 6
  3: 287884
  4: 287864 _Z3fibi:315608
  15: 23
- !Attributes: 0
+ !CFGChecksum: 281479271677951
+ !Attributes: 2
 [main:3.1 @ _Z5funcBi:1 @ _Z8funcLeafi]:500853:20
  0: 15
  1: 15
  3: 74946
  4: 74941 _Z3fibi:82359
  10: 23324
  11: 23327 _Z3fibi:25228
  15: 11
- !Attributes: 1
+ !CFGChecksum: 281479271677951
+  !Attributes: 2
 [external:12 @ main]:154:12
  2: 12
  3: 10 _Z5funcAi:7
  3.1: 10 _Z5funcBi:11
- !Attributes: 0
+ !CFGChecksum: 563125815542069
 [main]:154:0
  2: 12
  3: 18 _Z5funcAi:11
  3.1: 18 _Z5funcBi:19
- !Attributes: 0
+ !CFGChecksum: 563125815542069
 [external:10 @ _Z5funcBi]:120:10
  0: 10
  1: 10
- !Attributes: 0
+ !CFGChecksum: 563022570642068
 [externalA:17 @ _Z5funcBi]:120:3
  0: 3
  1: 3
- !Attributes: 0
+ !CFGChecksum: 563022570642068
 [main:3.1 @ _Z5funcBi]:120:19
  0: 19
  1: 19 _Z8funcLeafi:20
  3: 12
- !Attributes: 1
+ !CFGChecksum: 563022570642068
+ !Attributes: 2
 [main:3 @ _Z5funcAi]:99:11
  0: 10
  1: 10 _Z8funcLeafi:11
  3: 24
- !Attributes: 0
+ !CFGChecksum: 844530426352218
+ !Attributes: 2

llvm/test/tools/llvm-profdata/Inputs/cs-sample-preinline.proftext

 [main:3 @ _Z5funcAi:1 @ _Z8funcLeafi]:1467299:11
  0: 6
  1: 6
  3: 287884
  4: 287864 _Z3fibi:315608
  15: 23
- !Attributes: 0
+ !Attributes: 2
 [main:3.1 @ _Z5funcBi:1 @ _Z8funcLeafi]:500853:20
  0: 15
  1: 15
  3: 74946
  4: 74941 _Z3fibi:82359
  10: 23324
  11: 23327 _Z3fibi:25228
  15: 11
- !Attributes: 1
+ !Attributes: 2
 [external:12 @ main]:154:12
  2: 12
  3: 10 _Z5funcAi:7
  3.1: 10 _Z5funcBi:11
- !Attributes: 0
 [main]:154:0
  2: 12
  3: 18 _Z5funcAi:11
  3.1: 18 _Z5funcBi:19
- !Attributes: 0
 [external:10 @ _Z5funcBi]:120:10
  0: 10
  1: 10
- !Attributes: 0
 [externalA:17 @ _Z5funcBi]:120:3
  0: 3
  1: 3
- !Attributes: 0
 [main:3.1 @ _Z5funcBi]:120:19
  0: 19
  1: 19 _Z8funcLeafi:20
  3: 12
- !Attributes: 1
+ !Attributes: 2
 [main:3 @ _Z5funcAi]:99:11
  0: 10
  1: 10 _Z8funcLeafi:11
  3: 24
- !Attributes: 0
+ !Attributes: 2

llvm/test/tools/llvm-profdata/Inputs/cs-sample.proftext

 [main:3 @ _Z5funcAi:1 @ _Z8funcLeafi]:1467299:11
  0: 6
  1: 6
  3: 287884
  4: 287864 _Z3fibi:315608
  15: 23
- !Attributes: 0
 [main:3.1 @ _Z5funcBi:1 @ _Z8funcLeafi]:500853:20
  0: 15
  1: 15
  3: 74946
  4: 74941 _Z3fibi:82359
  10: 23324
  11: 23327 _Z3fibi:25228
  15: 11
  !Attributes: 1
 [external:12 @ main]:154:12
  2: 12
  3: 10 _Z5funcAi:7
  3.1: 10 _Z5funcBi:11
- !Attributes: 0
 [main]:154:0
  2: 12
  3: 18 _Z5funcAi:11
  3.1: 18 _Z5funcBi:19
- !Attributes: 0
 [external:10 @ _Z5funcBi]:120:10
  0: 10
  1: 10
- !Attributes: 0
 [externalA:17 @ _Z5funcBi]:120:3
  0: 3
  1: 3
- !Attributes: 0
 [main:3.1 @ _Z5funcBi]:120:19
  0: 19
  1: 19 _Z8funcLeafi:20
  3: 12
  !Attributes: 1
 [main:3 @ _Z5funcAi]:99:11
  0: 10
  1: 10 _Z8funcLeafi:11
  3: 24
- !Attributes: 0

llvm/test/tools/llvm-profdata/cs-sample-nested-profile.test

+RUN: llvm-profdata merge --sample --text -output=%t.proftext %S/Inputs/cs-sample-preinline.proftext --gen-nested-cs-profile=1 -generate-merged-base-profiles=0
+RUN: FileCheck %s < %t.proftext --match-full-lines --strict-whitespace
+RUN: llvm-profdata merge --sample --text -output=%t.probe.proftext %S/Inputs/cs-sample-preinline-probe.proftext --gen-nested-cs-profile=1 -generate-merged-base-profiles=0
+RUN: FileCheck %s < %t.probe.proftext --match-full-lines --strict-whitespace -check-prefix=PROBE
+RUN: llvm-profdata merge --sample --extbinary -output=%t.profbin %S/Inputs/cs-sample-preinline.proftext --gen-nested-cs-profile=1 -generate-merged-base-profiles=0
+RUN: llvm-profdata merge --sample --text -output=%t2.proftext %t.profbin
+RUN: FileCheck %s < %t2.proftext --match-full-lines --strict-whitespace
+RUN: llvm-profdata show --sample -show-sec-info-only %t.profbin | FileCheck %s -check-prefix=PREINLINE
+RUN: llvm-profdata merge --sample --text -output=%t.proftext %S/Inputs/cs-sample-preinline.proftext --gen-nested-cs-profile=1 -generate-merged-base-profiles=1
+RUN: FileCheck %s < %t.proftext --match-full-lines --strict-whitespace -check-prefix=RECOUNT
+
+
+; CHECK:main:1968679:12
+; CHECK-NEXT: 2: 24
+; CHECK-NEXT: 3: 28 _Z5funcAi:18
+; CHECK-NEXT: 3.1: 28 _Z5funcBi:30
+; CHECK-NEXT: 3: _Z5funcAi:1467398
+; CHECK-NEXT:  0: 10
+; CHECK-NEXT:  1: 10 _Z8funcLeafi:11
+; CHECK-NEXT:  3: 24
+; CHECK-NEXT:  1: _Z8funcLeafi:1467299
+; CHECK-NEXT:   0: 6
+; CHECK-NEXT:   1: 6
+; CHECK-NEXT:   3: 287884
+; CHECK-NEXT:   4: 287864 _Z3fibi:315608
+; CHECK-NEXT:   15: 23
+; CHECK-NEXT:   !Attributes: 2
+; CHECK-NEXT:  !Attributes: 2
+; CHECK-NEXT: 3.1: _Z5funcBi:500973
+; CHECK-NEXT:  0: 19
+; CHECK-NEXT:  1: 19 _Z8funcLeafi:20
+; CHECK-NEXT:  3: 12
+; CHECK-NEXT:  1: _Z8funcLeafi:500853
+; CHECK-NEXT:   0: 15
+; CHECK-NEXT:   1: 15
+; CHECK-NEXT:   3: 74946
+; CHECK-NEXT:   4: 74941 _Z3fibi:82359
+; CHECK-NEXT:   10: 23324
+; CHECK-NEXT:   11: 23327 _Z3fibi:25228
+; CHECK-NEXT:   15: 11
+; CHECK-NEXT:   !Attributes: 2
+; CHECK-NEXT:  !Attributes: 2
+; CHECK-NEXT:_Z5funcBi:240:13
+; CHECK-NEXT: 0: 13
+; CHECK-NEXT: 1: 13
+
+
+
+; RECOUNT:main:1968679:12
+; RECOUNT-NEXT: 2: 24
+; RECOUNT-NEXT: 3: 28 _Z5funcAi:18
+; RECOUNT-NEXT: 3.1: 28 _Z5funcBi:30
+; RECOUNT-NEXT: 3: _Z5funcAi:1467398
+; RECOUNT-NEXT:  0: 10
+; RECOUNT-NEXT:  1: 10 _Z8funcLeafi:11
+; RECOUNT-NEXT:  3: 24
+; RECOUNT-NEXT:  1: _Z8funcLeafi:1467299
+; RECOUNT-NEXT:   0: 6
+; RECOUNT-NEXT:   1: 6
+; RECOUNT-NEXT:   3: 287884
+; RECOUNT-NEXT:   4: 287864 _Z3fibi:315608
+; RECOUNT-NEXT:   15: 23
+; RECOUNT-NEXT:   !Attributes: 2
+; RECOUNT-NEXT:  !Attributes: 2
+; RECOUNT-NEXT: 3.1: _Z5funcBi:500973
+; RECOUNT-NEXT:  0: 19
+; RECOUNT-NEXT:  1: 19 _Z8funcLeafi:20
+; RECOUNT-NEXT:  3: 12
+; RECOUNT-NEXT:  1: _Z8funcLeafi:500853
+; RECOUNT-NEXT:   0: 15
+; RECOUNT-NEXT:   1: 15
+; RECOUNT-NEXT:   3: 74946
+; RECOUNT-NEXT:   4: 74941 _Z3fibi:82359
+; RECOUNT-NEXT:   10: 23324
+; RECOUNT-NEXT:   11: 23327 _Z3fibi:25228
+; RECOUNT-NEXT:   15: 11
+; RECOUNT-NEXT:   !Attributes: 2
+; RECOUNT-NEXT:  !Attributes: 2
+; RECOUNT-NEXT:_Z8funcLeafi:1968152:31
+; RECOUNT-NEXT: 0: 21
+; RECOUNT-NEXT: 1: 21
+; RECOUNT-NEXT: 3: 362830
+; RECOUNT-NEXT: 4: 362805 _Z3fibi:397967
+; RECOUNT-NEXT: 10: 23324
+; RECOUNT-NEXT: 11: 23327 _Z3fibi:25228
+; RECOUNT-NEXT: 15: 34
+; RECOUNT-NEXT: !Attributes: 2
+; RECOUNT-NEXT:_Z5funcAi:1467398:11
+; RECOUNT-NEXT: 0: 10
+; RECOUNT-NEXT: 1: 10 _Z8funcLeafi:11
+; RECOUNT-NEXT: 3: 24
+; RECOUNT-NEXT: 1: _Z8funcLeafi:1467299
+; RECOUNT-NEXT:  0: 6
+; RECOUNT-NEXT:  1: 6
+; RECOUNT-NEXT:  3: 287884
+; RECOUNT-NEXT:  4: 287864 _Z3fibi:315608
+; RECOUNT-NEXT:  15: 23
+; RECOUNT-NEXT:  !Attributes: 2
+; RECOUNT-NEXT: !Attributes: 2
+; RECOUNT-NEXT:_Z5funcBi:501213:32
+; RECOUNT-NEXT: 0: 32
+; RECOUNT-NEXT: 1: 32 _Z8funcLeafi:20
+; RECOUNT-NEXT: 3: 12
+; RECOUNT-NEXT: 1: _Z8funcLeafi:500853
+; RECOUNT-NEXT:  0: 15
+; RECOUNT-NEXT:  1: 15
+; RECOUNT-NEXT:  3: 74946
+; RECOUNT-NEXT:  4: 74941 _Z3fibi:82359
+; RECOUNT-NEXT:  10: 23324
+; RECOUNT-NEXT:  11: 23327 _Z3fibi:25228
+; RECOUNT-NEXT:  15: 11
+; RECOUNT-NEXT:  !Attributes: 2
+
+; PROBE:main:1968679:12
+; PROBE-NEXT: 2: 24
+; PROBE-NEXT: 3: 28 _Z5funcAi:18
+; PROBE-NEXT: 3.1: 28 _Z5funcBi:30
+; PROBE-NEXT: 3: _Z5funcAi:1467398
+; PROBE-NEXT:  0: 10
+; PROBE-NEXT:  1: 10 _Z8funcLeafi:11
+; PROBE-NEXT:  3: 24
+; PROBE-NEXT:  1: _Z8funcLeafi:1467299
+; PROBE-NEXT:   0: 6
+; PROBE-NEXT:   1: 6
+; PROBE-NEXT:   3: 287884
+; PROBE-NEXT:   4: 287864 _Z3fibi:315608
+; PROBE-NEXT:   15: 23
+; PROBE-NEXT:   !CFGChecksum: 281479271677951
+; PROBE-NEXT:   !Attributes: 2
+; PROBE-NEXT:  !CFGChecksum: 844530426352218
+; PROBE-NEXT:  !Attributes: 2
+; PROBE-NEXT: 3.1: _Z5funcBi:500973
+; PROBE-NEXT:  0: 19
+; PROBE-NEXT:  1: 19 _Z8funcLeafi:20
+; PROBE-NEXT:  3: 12
+; PROBE-NEXT:  1: _Z8funcLeafi:500853
+; PROBE-NEXT:   0: 15
+; PROBE-NEXT:   1: 15
+; PROBE-NEXT:   3: 74946
+; PROBE-NEXT:   4: 74941 _Z3fibi:82359
+; PROBE-NEXT:   10: 23324
+; PROBE-NEXT:   11: 23327 _Z3fibi:25228
+; PROBE-NEXT:   15: 11
+; PROBE-NEXT:   !CFGChecksum: 281479271677951
+; PROBE-NEXT:   !Attributes: 2
+; PROBE-NEXT:  !CFGChecksum: 563022570642068
+; PROBE-NEXT:  !Attributes: 2
+; PROBE-NEXT: !CFGChecksum: 563125815542069
+; PROBE-NEXT:_Z5funcBi:240:13
+; PROBE-NEXT: 0: 13
+; PROBE-NEXT: 1: 13
+; PROBE-NEXT: !CFGChecksum: 563022570642068
+
+
+; PREINLINE: FunctionMetadata {{.*}} Flags: {attr,preinlined}
+
+

llvm/test/tools/llvm-profdata/cs-sample-trimmer.test

 RUN: llvm-profdata merge --sample --text -output=%t.proftext %S/Inputs/cs-sample.proftext
 RUN: diff -b %t.proftext %S/Inputs/cs-sample.proftext
 
 RUN: llvm-profdata merge --sample --text -output=%t.proftext %S/Inputs/cs-sample.proftext -sample-merge-cold-context -profile-summary-cold-count=500
 RUN: FileCheck %s --input-file %t.proftext --check-prefixes=CHECK-TRIM,CHECK-MERGE
 RUN: llvm-profdata merge --sample --text -output=%t.proftext %S/Inputs/cs-sample.proftext -sample-merge-cold-context -sample-trim-cold-context -profile-summary-cold-count=500
 RUN: FileCheck %s --input-file %t.proftext --check-prefixes=CHECK-TRIM,CHECK-END
 RUN: llvm-profdata merge --sample --text -output=%t.proftext %S/Inputs/cs-sample.proftext -sample-merge-cold-context -sample-trim-cold-context -profile-summary-cutoff-cold=990000
 RUN: FileCheck %s --input-file %t.proftext --check-prefixes=CHECK-TRIM,CHECK-END
 
 CHECK-TRIM:      [main:3 @ _Z5funcAi:1 @ _Z8funcLeafi]:1467299:11
 CHECK-TRIM-NEXT:  0: 6
 CHECK-TRIM-NEXT:  1: 6
 CHECK-TRIM-NEXT:  3: 287884
 CHECK-TRIM-NEXT:  4: 287864 _Z3fibi:315608
 CHECK-TRIM-NEXT:  15: 23
-CHECK-TRIM-NEXT:  !Attributes: 0
 CHECK-TRIM-NEXT: [main:3.1 @ _Z5funcBi:1 @ _Z8funcLeafi]:500853:20
 CHECK-TRIM-NEXT:  0: 15
 CHECK-TRIM-NEXT:  1: 15
 CHECK-TRIM-NEXT:  3: 74946
 CHECK-TRIM-NEXT:  4: 74941 _Z3fibi:82359
 CHECK-TRIM-NEXT:  10: 23324
 CHECK-TRIM-NEXT:  11: 23327 _Z3fibi:25228
 CHECK-TRIM-NEXT:  15: 11
 CHECK-TRIM-NEXT:  !Attributes: 1
 CHECK-END-NOT:   [
 CHECK-MERGE:     [_Z5funcBi]:360:32
 CHECK-MERGE-NEXT: 0: 32
 CHECK-MERGE-NEXT: 1: 32 _Z8funcLeafi:20
 CHECK-MERGE-NEXT: 3: 12
-CHECK-MERGE-NEXT: !Attributes: 0
 CHECK-MERGE-NEXT:[main]:308:12
 CHECK-MERGE-NEXT: 2: 24
 CHECK-MERGE-NEXT: 3: 28 _Z5funcAi:18
 CHECK-MERGE-NEXT: 3.1: 28 _Z5funcBi:30
-CHECK-MERGE-NEXT: !Attributes: 0
 CHECK-MERGE-NEXT:[_Z5funcAi]:99:11
 CHECK-MERGE-NEXT: 0: 10
 CHECK-MERGE-NEXT: 1: 10 _Z8funcLeafi:11
 CHECK-MERGE-NEXT: 3: 24
-CHECK-MERGE-NEXT: !Attributes: 0

llvm/test/tools/llvm-profgen/cs-preinline.test

 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-cs-noprobe.perfscript --binary=%S/Inputs/inline-cs-noprobe.perfbin --output=%t --csspgo-preinliner=1  -sample-profile-cold-inline-threshold=0
 ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-NO-PREINL
 
 ; Test cold profile trimming. Only base profiles should be dropped.
 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-cs-noprobe.perfscript --binary=%S/Inputs/inline-cs-noprobe.perfbin --output=%t --csspgo-preinliner=1  --csprof-trim-cold-context=1 --profile-summary-hot-count=250
 
 ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-TRIM
 
+; Test llvm-profgen with preinliner on will merge not inlinable profile into base profile.
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-cs-noprobe.perfscript --binary=%S/Inputs/inline-cs-noprobe.perfbin --output=%t --csspgo-preinliner=1 --gen-nested-cs-profile=1
+; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-PREINL-NEST
+
 ; CHECK-DEFAULT:     [main:1 @ foo]:44:0
 ; CHECK-DEFAULT-NEXT: 2.1: 14
 ; CHECK-DEFAULT-NEXT: 3: 15
 ; CHECK-DEFAULT-NEXT: 3.1: 14 bar:14
 ; CHECK-DEFAULT-NEXT: 3.2: 1
 ; CHECK-DEFAULT-NEXT: !Attributes: 1
 ; CHECK-DEFAULT-NEXT:[main:1 @ foo:3.1 @ bar]:14:0
 ; CHECK-DEFAULT-NEXT: 1: 14

 ; CHECK-NO-PREINL-NEXT:[bar]:14:0
 ; CHECK-NO-PREINL-NEXT: 1: 14
 ; CHECK-NO-PREINL-NEXT: !Attributes: 1
 
 ; CHECK-TRIM-NOT: [foo]
 ; CHECK-TRIM:[foo:3.1 @ bar]:14:0
 ; CHECK-TRIM-NEXT: 1: 14
 ; CHECK-TRIM-NEXT: !Attributes: 3
+
+; CHECK-PREINL-NEST:     foo:58:0
+; CHECK-PREINL-NEST-NEXT: 2.1: 14
+; CHECK-PREINL-NEST-NEXT: 3: 15
+; CHECK-PREINL-NEST-NEXT: 3.1: 14 bar:14
+; CHECK-PREINL-NEST-NEXT: 3.2: 1
+; CHECK-PREINL-NEST-NEXT: 3.1: bar:14
+; CHECK-PREINL-NEST-NEXT:  1: 14

llvm/test/tools/llvm-profgen/merge-cold-profile.test

 ; CHECK-NEXT: 1: 4
 ; CHECK-NEXT: 3: 4
 ; CHECK-NEXT: 4: 2
 ; CHECK-NEXT: 5: 1
 ; CHECK-NEXT: 6: 0
 ; CHECK-NEXT: 7: 2 fb:2
 ; CHECK-NEXT: 8: 1 fa:1
 ; CHECK-NEXT: !CFGChecksum: 563070469352221
-; CHECK-NEXT: !Attributes: 0
 ; CHECK-NEXT:[main:2 @ foo:5 @ fa:8 @ fa:7 @ fb:5 @ fb]:13:4
 ; CHECK-NEXT: 1: 4
 ; CHECK-NEXT: 2: 3
 ; CHECK-NEXT: 3: 1
 ; CHECK-NEXT: 4: 0
 ; CHECK-NEXT: 5: 4 fb:4
 ; CHECK-NEXT: 6: 1 fa:1
 ; CHECK-NEXT: !CFGChecksum: 563022570642068
 
 ; CHECK-KEEP-COLD:     [fb]:19:6
 ; CHECK-KEEP-COLD-NEXT: 1: 6
 ; CHECK-KEEP-COLD-NEXT: 2: 3
 ; CHECK-KEEP-COLD-NEXT: 3: 3
 ; CHECK-KEEP-COLD-NEXT: 4: 0
 ; CHECK-KEEP-COLD-NEXT: 5: 4 fb:4
 ; CHECK-KEEP-COLD-NEXT: 6: 3 fa:3
 ; CHECK-KEEP-COLD-NEXT: !CFGChecksum: 563022570642068
-; CHECK-KEEP-COLD-NEXT: !Attributes: 0
 ; CHECK-KEEP-COLD-NEXT:[fa]:14:4
 ; CHECK-KEEP-COLD-NEXT: 1: 4
 ; CHECK-KEEP-COLD-NEXT: 3: 4
 ; CHECK-KEEP-COLD-NEXT: 4: 2
 ; CHECK-KEEP-COLD-NEXT: 5: 1
 ; CHECK-KEEP-COLD-NEXT: 6: 0
 ; CHECK-KEEP-COLD-NEXT: 7: 2 fb:2
 ; CHECK-KEEP-COLD-NEXT: 8: 1 fa:1

 ; CHECK-COLD-CONTEXT-LENGTH:     [fb:5 @ fb]:13:4
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 1: 4
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 2: 3
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 3: 1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 4: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 5: 4 fb:4
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 6: 1 fa:1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: !CFGChecksum: 563022570642068
-; CHECK-COLD-CONTEXT-LENGTH-NEXT: !Attributes: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT:[fb:6 @ fa]:10:3
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 1: 3
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 3: 3
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 4: 1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 5: 1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 6: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 7: 1 fb:1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 8: 1 fa:1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: !CFGChecksum: 563070469352221
-; CHECK-COLD-CONTEXT-LENGTH-NEXT: !Attributes: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT:[fa:7 @ fb]:6:2
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 1: 2
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 2: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 3: 2
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 4: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 5: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 6: 2 fa:2
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: !CFGChecksum: 563022570642068
-; CHECK-COLD-CONTEXT-LENGTH-NEXT: !Attributes: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT:[fa:8 @ fa]:4:1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 1: 1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 3: 1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 4: 1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 5: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 6: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 7: 1 fb:1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 8: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: !CFGChecksum: 563070469352221
-; CHECK-COLD-CONTEXT-LENGTH-NEXT: !Attributes: 0
 
 ; clang -O3 -fexperimental-new-pass-manager -fuse-ld=lld -fpseudo-probe-for-profiling
 ; -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Xclang -mdisable-tail-calls
 ; -g test.c  -o a.out
 
 ; Copied from recursion-compression.test
 #include <stdio.h>
 

llvm/test/tools/llvm-profgen/truncated-pseudoprobe.test

 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/truncated-pseudoprobe.perfscript --binary=%S/Inputs/truncated-pseudoprobe.perfbin --output=%t
 ; RUN: FileCheck %s --input-file %t
 
 ; CHECK:     [foo]:75:0
 ; CHECK-NEXT:  1: 0
 ; CHECK-NEXT:  2: 15
 ; CHECK-NEXT:  3: 15
 ; CHECK-NEXT:  4: 15
 ; CHECK-NEXT:  5: 0
 ; CHECK-NEXT:  6: 15
 ; CHECK-NEXT:  7: 0
 ; CHECK-NEXT:  8: 15 bar:15
 ; CHECK-NEXT:  9: 0
 ; CHECK-NEXT:  !CFGChecksum: 563088904013236
-; CHECK-NEXT:  !Attributes: 0
 ; CHECK:     [foo:8 @ bar]:30:15
 ; CHECK-NEXT:  1: 15
 ; CHECK-NEXT:  4: 15
 ; CHECK-NEXT:  !CFGChecksum: 72617220756
 ; CHECK-NEXT:  !Attributes: 1
 
 ; truncated-pseudoprobe.perfbin is from the following compile commands:
 ; llc truncated-pseudoprobe.ll -filetype=obj -o truncated-pseudoprobe.o
 ; clang truncated-pseudoprobe.o -o truncated-pseudoprobe.perfbin

llvm/tools/llvm-profdata/llvm-profdata.cpp

else
       Writer.setPartialProfile();
   }
 }
 
 static void
 mergeSampleProfile(const WeightedFileVector &Inputs, SymbolRemapper *Remapper,
                    StringRef OutputFilename, ProfileFormat OutputFormat,
                    StringRef ProfileSymbolListFile, bool CompressAllSections,
-                   bool UseMD5, bool GenPartialProfile,
+                   bool UseMD5, bool GenPartialProfile, bool GenNestedCSProfile,
                    bool SampleMergeColdContext, bool SampleTrimColdContext,
                    bool SampleColdContextFrameDepth, FailureMode FailMode) {
   using namespace sampleprof;
   SampleProfileMap ProfileMap;
   SmallVector<std::unique_ptr<sampleprof::SampleProfileReader>, 5> Readers;
   LLVMContext Context;
   sampleprof::ProfileSymbolList WriterList;
   Optional<bool> ProfileIsProbeBased;
-  Optional<bool> ProfileIsCS;
+  Optional<bool> ProfileIsCSFlat;
   for (const auto &Input : Inputs) {
     auto ReaderOrErr = SampleProfileReader::create(Input.Filename, Context,
                                                    FSDiscriminatorPassOption);
     if (std::error_code EC = ReaderOrErr.getError()) {
       warnOrExitGivenError(FailMode, EC, Input.Filename);
       continue;
     }
 
for (const auto &Input : Inputs) {
     }
 
     SampleProfileMap &Profiles = Reader->getProfiles();
     if (ProfileIsProbeBased.hasValue() &&
         ProfileIsProbeBased != FunctionSamples::ProfileIsProbeBased)
       exitWithError(
           "cannot merge probe-based profile with non-probe-based profile");
     ProfileIsProbeBased = FunctionSamples::ProfileIsProbeBased;
-    if (ProfileIsCS.hasValue() && ProfileIsCS != FunctionSamples::ProfileIsCS)
+    if (ProfileIsCSFlat.hasValue() &&
+        ProfileIsCSFlat != FunctionSamples::ProfileIsCSFlat)
       exitWithError("cannot merge CS profile with non-CS profile");
-    ProfileIsCS = FunctionSamples::ProfileIsCS;
+    ProfileIsCSFlat = FunctionSamples::ProfileIsCSFlat;
     for (SampleProfileMap::iterator I = Profiles.begin(), E = Profiles.end();
          I != E; ++I) {
       sampleprof_error Result = sampleprof_error::success;
       FunctionSamples Remapped =
           Remapper ? remapSamples(I->second, *Remapper, Result)
                    : FunctionSamples();
       FunctionSamples &Samples = Remapper ? Remapped : I->second;
       SampleContext FContext = Samples.getContext();
       MergeResult(Result, ProfileMap[FContext].merge(Samples, Input.Weight));
       if (Result != sampleprof_error::success) {
         std::error_code EC = make_error_code(Result);
         handleMergeWriterError(errorCodeToError(EC), Input.Filename,
                                FContext.toString());
       }
     }
 
     std::unique_ptr<sampleprof::ProfileSymbolList> ReaderList =
         Reader->getProfileSymbolList();
     if (ReaderList)
       WriterList.merge(*ReaderList);
   }
 
-  if (ProfileIsCS && (SampleMergeColdContext || SampleTrimColdContext)) {
+  if (ProfileIsCSFlat && (SampleMergeColdContext || SampleTrimColdContext)) {
     // Use threshold calculated from profile summary unless specified.
     SampleProfileSummaryBuilder Builder(ProfileSummaryBuilder::DefaultCutoffs);
     auto Summary = Builder.computeSummaryForProfiles(ProfileMap);
     uint64_t SampleProfColdThreshold =
         ProfileSummaryBuilder::getColdCountThreshold(
             (Summary->getDetailedSummary()));
 
     // Trim and merge cold context profile using cold threshold above;
     SampleContextTrimmer(ProfileMap)
         .trimAndMergeColdContextProfiles(
             SampleProfColdThreshold, SampleTrimColdContext,
             SampleMergeColdContext, SampleColdContextFrameDepth, false);
   }
 
+  if (ProfileIsCSFlat && GenNestedCSProfile) {
+    CSProfileConverter CSConverter(ProfileMap);
+    CSConverter.convertProfiles();
+    ProfileIsCSFlat = FunctionSamples::ProfileIsCSFlat = false;
+  }
+
   auto WriterOrErr =
       SampleProfileWriter::create(OutputFilename, FormatMap[OutputFormat]);
   if (std::error_code EC = WriterOrErr.getError())
     exitWithErrorCode(EC, OutputFilename);
 
   auto Writer = std::move(WriterOrErr.get());
   // WriterList will have StringRef refering to string in Buffer.
   // Make sure Buffer lives as long as WriterList.
static int merge_main(int argc, const char *argv[]) {
   cl::opt<unsigned> SupplMinSizeThreshold(
       "suppl-min-size-threshold", cl::init(10), cl::Hidden,
       cl::desc("If the size of a function is smaller than the threshold, "
                "assume it can be inlined by PGO early inliner and it won't "
                "be adjusted based on sample profile."));
   cl::opt<unsigned> InstrProfColdThreshold(
       "instr-prof-cold-threshold", cl::init(0), cl::Hidden,
       cl::desc("User specified cold threshold for instr profile which will "
-               "override the cold threshold got from profile summary."));
+               "override the cold threshold got from profile summary. "));
+  cl::opt<bool> GenNestedCSProfile(
+      "gen-nested-cs-profile", cl::Hidden, cl::init(false),
+      cl::desc("Generate nested function profiles for CSSPGO"));
 
   cl::ParseCommandLineOptions(argc, argv, "LLVM profile data merger\n");
 
   WeightedFileVector WeightedInputs;
   for (StringRef Filename : InputFilenames)
     addWeightedInput(WeightedInputs, {std::string(Filename), 1});
   for (StringRef WeightedFilename : WeightedInputFilenames)
     addWeightedInput(WeightedInputs, parseWeightedFile(WeightedFilename));
static int merge_main(int argc, const char *argv[]) {
   }
 
   if (ProfileKind == instr)
     mergeInstrProfile(WeightedInputs, Remapper.get(), OutputFilename,
                       OutputFormat, OutputSparse, NumThreads, FailureMode);
   else
     mergeSampleProfile(WeightedInputs, Remapper.get(), OutputFilename,
                        OutputFormat, ProfileSymbolListFile, CompressAllSections,
-                       UseMD5, GenPartialProfile, SampleMergeColdContext,
-                       SampleTrimColdContext, SampleColdContextFrameDepth,
-                       FailureMode);
-
+                       UseMD5, GenPartialProfile, GenNestedCSProfile,
+                       SampleMergeColdContext, SampleTrimColdContext,
+                       SampleColdContextFrameDepth, FailureMode);
   return 0;
 }
 
 /// Computer the overlap b/w profile BaseFilename and profile TestFilename.
 static void overlapInstrProfile(const std::string &BaseFilename,
                                 const std::string &TestFilename,
                                 const OverlapFuncFilters &FuncFilter,
                                 raw_fd_ostream &OS, bool IsCS) {
std::error_code SampleOverlapAggregator::loadProfiles() {
 
   if (std::error_code EC = BaseReader->read())
     exitWithErrorCode(EC, BaseFilename);
   if (std::error_code EC = TestReader->read())
     exitWithErrorCode(EC, TestFilename);
   if (BaseReader->profileIsProbeBased() != TestReader->profileIsProbeBased())
     exitWithError(
         "cannot compare probe-based profile with non-probe-based profile");
-  if (BaseReader->profileIsCS() != TestReader->profileIsCS())
+  if (BaseReader->profileIsCSFlat() != TestReader->profileIsCSFlat())
     exitWithError("cannot compare CS profile with non-CS profile");
 
   // Load BaseHotThreshold and TestHotThreshold as 99-percentile threshold in
   // profile summary.
   ProfileSummary &BasePS = BaseReader->getSummary();
   ProfileSummary &TestPS = TestReader->getSummary();
   BaseHotThreshold =
       ProfileSummaryBuilder::getHotCountThreshold(BasePS.getDetailedSummary());

llvm/tools/llvm-profgen/PerfReader.h

public:
   static std::unique_ptr<PerfReaderBase> create(ProfiledBinary *Binary,
                                                 PerfInputFile &PerfInput);
 
   // Entry of the reader to parse multiple perf traces
   virtual void parsePerfTraces() = 0;
   const ContextSampleCounterMap &getSampleCounters() const {
     return SampleCounters;
   }
-  bool profileIsCS() { return ProfileIsCS; }
+  bool profileIsCSFlat() { return ProfileIsCSFlat; }
 
 protected:
   ProfiledBinary *Binary = nullptr;
   StringRef PerfTraceFile;
 
   ContextSampleCounterMap SampleCounters;
-  bool ProfileIsCS = false;
+  bool ProfileIsCSFlat = false;
 };
 
 // Read perf script to parse the events and samples.
 class PerfScriptReader : public PerfReaderBase {
 public:
   PerfScriptReader(ProfiledBinary *B, StringRef PerfTrace)
       : PerfReaderBase(B, PerfTrace){};
 

llvm/tools/llvm-profgen/PerfReader.cpp

for (auto &I : Counter) {
       OS.indent(Indent);
       OS << Twine::utohexstr(Start) << Separator << Twine::utohexstr(End) << ":"
          << I.second << "\n";
     }
   };
 
   for (auto &CI : OrderedCounters) {
     uint32_t Indent = 0;
-    if (ProfileIsCS) {
+    if (ProfileIsCSFlat) {
       // Context string key
       OS << "[" << CI.first << "]\n";
       Indent = 2;
     }
 
     SampleCounter &Counter = *CI.second;
     SCounterPrinter(Counter.RangeCounter, "-", Indent);
     SCounterPrinter(Counter.BranchCounter, "->", Indent);

 void UnsymbolizedProfileReader::readUnsymbolizedProfile(StringRef FileName) {
   TraceStream TraceIt(FileName);
   while (!TraceIt.isAtEoF()) {
     std::shared_ptr<StringBasedCtxKey> Key =
         std::make_shared<StringBasedCtxKey>();
     StringRef Line = TraceIt.getCurrentLine();
     // Read context stack for CS profile.
     if (Line.startswith("[")) {
-      ProfileIsCS = true;
+      ProfileIsCSFlat = true;
       auto I = ContextStrSet.insert(Line.str());
       SampleContext::createCtxVectorFromStr(*I.first, Key->Context);
       TraceIt.advance();
     }
     auto Ret =
         SampleCounters.emplace(Hashable<ContextKey>(Key), SampleCounter());
     readSampleCounters(TraceIt, Ret.first->second);
   }
while (!TraceIt.isAtEoF()) {
     }
   }
 
   exitWithError("Invalid perf script input!");
   return PerfContent::UnknownContent;
 }
 
 void HybridPerfReader::generateUnsymbolizedProfile() {
-  ProfileIsCS = !IgnoreStackSamples;
-  if (ProfileIsCS)
+  ProfileIsCSFlat = !IgnoreStackSamples;
+  if (ProfileIsCSFlat)
     unwindSamples();
   else
     PerfScriptReader::generateUnsymbolizedProfile();
 }
 
 void PerfScriptReader::warnTruncatedStack() {
   if (ShowDetailedWarning) {
     for (auto Address : InvalidReturnAddresses) {

llvm/tools/llvm-profgen/ProfileGenerator.h

 
 public:
   ProfileGeneratorBase(ProfiledBinary *Binary,
                        const ContextSampleCounterMap &Counters)
       : Binary(Binary), SampleCounters(Counters){};
   virtual ~ProfileGeneratorBase() = default;
   static std::unique_ptr<ProfileGeneratorBase>
   create(ProfiledBinary *Binary, const ContextSampleCounterMap &SampleCounters,
-         bool ProfileIsCS);
+         bool ProfileIsCSFlat);
   virtual void generateProfile() = 0;
   void write();
 
   static uint32_t
   getDuplicationFactor(unsigned Discriminator,
                        bool UseFSD = ProfileGeneratorBase::UseFSDiscriminator) {
     return UseFSD ? 1
                   : llvm::DILocation::getDuplicationFactorFromDiscriminator(

llvm/tools/llvm-profgen/ProfileGenerator.cpp

llvm::cl::desc(
         "specify density threshold for hot functions (default: 1000)"),
     llvm::cl::Optional);
 static cl::opt<bool> ShowDensity("show-density", llvm::cl::init(false),
                                  llvm::cl::desc("show profile density details"),
                                  llvm::cl::Optional);
 
 extern cl::opt<int> ProfileSummaryCutoffHot;
 
+static cl::opt<bool> GenNestedCSProfile(
+    "gen-nested-cs-profile", cl::Hidden, cl::init(false),
+    cl::desc("Generate nested function profiles for CSSPGO"));
+
 using namespace llvm;
 using namespace sampleprof;
 
 namespace llvm {
 namespace sampleprof {
 
 // Initialize the MaxCompressionSize to -1 which means no size limit
 int32_t CSProfileGenerator::MaxCompressionSize = -1;
 
 int CSProfileGenerator::MaxContextDepth = -1;
 
 bool ProfileGeneratorBase::UseFSDiscriminator = false;
 
 std::unique_ptr<ProfileGeneratorBase>
 ProfileGeneratorBase::create(ProfiledBinary *Binary,
                              const ContextSampleCounterMap &SampleCounters,
-                             bool ProfileIsCS) {
+                             bool ProfileIsCSFlat) {
   std::unique_ptr<ProfileGeneratorBase> Generator;
-  if (ProfileIsCS) {
+  if (ProfileIsCSFlat) {
     if (Binary->useFSDiscriminator())
       exitWithError("FS discriminator is not supported in CS profile.");
     Generator.reset(new CSProfileGenerator(Binary, SampleCounters));
   } else {
     Generator.reset(new ProfileGenerator(Binary, SampleCounters));
   }
   ProfileGeneratorBase::UseFSDiscriminator = Binary->useFSDiscriminator();
   FunctionSamples::ProfileIsFS = Binary->useFSDiscriminator();
if (I == ProfileMap.end()) {
     FunctionSamples &FProfile = Ret.first->second;
     FProfile.setContext(FContext);
     return Ret.first->second;
   }
   return I->second;
 }
 
 void CSProfileGenerator::generateProfile() {
-  FunctionSamples::ProfileIsCS = true;
+  FunctionSamples::ProfileIsCSFlat = true;
 
   if (Binary->getTrackFuncContextSize())
     computeSizeForProfiledFunctions();
 
   if (Binary->usePseudoProbes()) {
     // Enable pseudo probe functionalities in SampleProf
     FunctionSamples::ProfileIsProbeBased = true;
     generateProbeBasedProfile();
void CSProfileGenerator::postProcessProfiles() {
 
   // Merge function samples of CS profile to calculate profile density.
   sampleprof::SampleProfileMap ContextLessProfiles;
   for (const auto &I : ProfileMap) {
     ContextLessProfiles[I.second.getName()].merge(I.second);
   }
 
   calculateAndShowDensity(ContextLessProfiles);
+  if (GenNestedCSProfile) {
+    CSProfileConverter CSConverter(ProfileMap);
+    CSConverter.convertProfiles();
+    FunctionSamples::ProfileIsCSFlat = false;
+    FunctionSamples::ProfileIsCSNested = EnableCSPreInliner;
+  }
 }
 
 void ProfileGeneratorBase::computeSummaryAndThreshold() {
   SampleProfileSummaryBuilder Builder(ProfileSummaryBuilder::DefaultCutoffs);
   auto Summary = Builder.computeSummaryForProfiles(ProfileMap);
   HotCountThreshold = ProfileSummaryBuilder::getHotCountThreshold(
       (Summary->getDetailedSummary()));
   ColdCountThreshold = ProfileSummaryBuilder::getColdCountThreshold(

llvm/tools/llvm-profgen/llvm-profgen.cpp

int main(int argc, const char *argv[]) {
   // Parse perf events and samples
   Reader->parsePerfTraces();
 
   if (SkipSymbolization)
     return EXIT_SUCCESS;
 
   std::unique_ptr<ProfileGeneratorBase> Generator =
       ProfileGeneratorBase::create(Binary.get(), Reader->getSampleCounters(),
-                                   Reader->profileIsCS());
+                                   Reader->profileIsCSFlat());
   Generator->generateProfile();
   Generator->write();
 
   return EXIT_SUCCESS;
 }