[CSSPGO] Use nested context-sensitive profile.

[hoy]

Summary

CSSPGO currently employs a flat profile format for context-sensitive profiles. Such a flat profile allows for precisely manipulating contexts that is either inlined or not inlined. This is a benefit over the nested profile format used by non-CS AutoFDO. A downside of this is the longer build time due to parsing the indexing the full CS contexts.

For a CS flat profile, though only the context profiles relevant to a module are loaded when that module is compiled, the cost to figure out what profiles are relevant is noticeably high when there're many contexts, since the sample reader will need to scan all context strings anyway. On the contrary, a nested function profile has its related inline subcontexts isolated from other unrelated contexts. Therefore when compiling a set of functions, unrelated contexts will never need to be scanned.

In this change we are exploring using nested profile format for CSSPGO. This is expected to work based on an assumption that with a preinliner-computed profile all contexts are precomputed and expected to be inlined by the compiler. Contexts not expected to be inlined will be cut off and returned to corresponding base profiles (for top-level outlined functions). This naturally forms a nested profile where all nested contexts are expected to be inlined. The compiler will less likely optimize on derived contexts that are not precomputed.

A CS-nested profile will look exactly the same with regular nested profile except that each nested profile can come with an attributes. With pseudo probes, a nested profile shown as below can also have a CFG checksum.

main:1968679:12
 2: 24
 3: 28 _Z5funcAi:18
 3.1: 28 _Z5funcBi:30
 3: _Z5funcAi:1467398
  0: 10
  1: 10 _Z8funcLeafi:11
  3: 24
  1: _Z8funcLeafi:1467299
   0: 6
   1: 6
   3: 287884
   4: 287864 _Z3fibi:315608
   15: 23
    !CFGChecksum: 138828622701
   !Attributes: 2
  !CFGChecksum: 281479271677951
  !Attributes: 2

Specific work included in this change:

• A recursive profile converter to convert CS flat profile to nested profile.
• Extend function checksum and attribute metadata to be stored in nested way for text profile and extbinary profile.
• Unifiy sample loader inliner path for CS and preinlined nested profile.
  • Changes in the sample loader to support probe-based nested profile.

I've seen promising results regarding build time. A nested profile can result in a 20% shorter build time than a CS flat profile while keep an on-par performance. This is with -duplicate-contexts-into-base=1.

Test Plan:

[hoy]

Updating D115205: [CSSPGO] Use nested context-sensitive profile.

[hoy]

Moving profile conversion to be after CS profile merging/trimming.

[wenlei]

llvm/include/llvm/ProfileData/SampleProf.h

1182

High level comments.

1186

FunctionSamples already has tree structure, can we use FunctionSamples tree to hold the nesting profiles directly, instead of introducing an intermediate trie tree? (The intermediate trie tree also overlaps with ContextTrieNode..)

llvm/lib/ProfileData/SampleProf.cpp

39

nit: generate-merged-base-profiles - "When generating nested context-sensitive profiles, always generate extra base profile for function with all its context profiles merged into it."

534

When NodeProfile (parent profile) doesn't exist, would it be beneficial to create the NodeProfile to contain inlinee's profile? That would be a more faithful conversion from a flat profile.

537–543

IIUC, the per-context ContextShouldBeInlined now becomes a per-profile flag. However, shouldInlineCandidate used by the inliner still checks attribute ContextShouldBeInlined from sample's context. That was how we tell inliner to honor preinline decision. Now without that attribute, how does inliner know to honor preinline decision?

llvm/lib/ProfileData/SampleProfReader.cpp

249

update comment

346–347

Does this mean we only set function hash for top level profile? why skip setting hash for nested inlinees?

1091

This parameter is not used?

llvm/lib/Transforms/IPO/SampleProfile.cpp

1484

The definition for ProfileIsPreinlined isn't very clear. A CS profile can also be preinlined, but here it looks like you're using it specifically for flat profile that is preinlined. Suggest we adhere to the literal definition which would apply to both flat and nest profile for preinline.

The check here could be using whether context tracker is available, basically we're try to track context without context tracker (which is tied to flat CS profile).

1523

nit: promoteMergeNotInlinedContextSamples. this is effectively a "context tracker" for nest profile, so we can use similar name as context tracker.

1775

This also opens up priority inliner for non-CS (not even preinlined) profile. Some (manual) testing as sanity check would be good.

llvm/test/tools/llvm-profdata/Inputs/cs-sample-preinline.proftext

16

what triggered these attribute value changes?

llvm/test/tools/llvm-profdata/cs-sample-nested-profile.test

2

have a probe version test case too to cover read/write per-inlinee function hash?

llvm/tools/llvm-profgen/ProfileGenerator.cpp

757

The name ProfileIsPreinlined suggest it should be set for flat profile too when preinliner is on, but this seems to be only used for nested profile.

[hoy]

llvm/include/llvm/ProfileData/SampleProf.h

1186

Nodes in a context tree may not have a function profile. Right, the struct here basically does what ContextTrieNode do. I was thinking about making ContextTrieNode shared between ProfilingTools and IPO, and that would require moving the definition into SampleProf.h, which may not be what we want.

llvm/lib/ProfileData/SampleProf.cpp

39

Sounds good.

534

When NodeProfile doesn't exist, it means the parent function is cold, and its profile also doesn't exist in the flat profile. So we are honoring that by not creating such profile here.

537–543

The per-profile flag is only used to tell the compiler how to set up the inliner (priority vs non-prioirty) and mcf extra. When it comes to inlining a particular callsite, the inliner would not honor the inline decision based on the per-context attribute. We always encode the per-context attribute in the profile.

llvm/lib/ProfileData/SampleProfReader.cpp

249

Good catch, done.

346–347

We set function hash for all profiles. We just count top-level profiles by ProbeProfileCount for sanity check in the assert several lines below. I just renamed the variable to TopLevelProbeProfileCount.

1091

Good catch.

llvm/lib/Transforms/IPO/SampleProfile.cpp

1484

Yeah, ProfileIsNestedCS sounds better?

Makes sense to check against ContextTracker.

1523

Sounds good.

1775

Test added to llvm-project/llvm/test/Transforms/SampleProfile/inline-mergeprof.ll and inline.ll

llvm/test/tools/llvm-profdata/Inputs/cs-sample-preinline.proftext

16

It's due to the following change in SampleProfWriter.cpp:

if (S.getContext().getAllAttributes()) {
    OS.indent(Indent + 1);
    OS << "!Attributes: " << S.getContext().getAllAttributes() << "\n";

llvm/test/tools/llvm-profdata/cs-sample-nested-profile.test

2

test added.

llvm/tools/llvm-profgen/ProfileGenerator.cpp

757

The flag is supposed to be only used by the compiler to differentiate a nested CS profile from a nested non-CS profile. Maybe ProfileIsNestedCS sound better?

[hoy]

Addressing Wenlei's comments.

[wenlei]

llvm/lib/ProfileData/SampleProf.cpp

534

If we have preinliner on, cold parent should be removed from context already. If the parent exists, that means it's somewhat relevant for inlining.

537–543

When it comes to inlining a particular callsite, the inliner would not honor the inline decision based on the per-context attribute.

Did you mean would or would not? I expect the correct behavior to be that inliner will honor per-context ContextShouldBeInlined attribute, which is what happens with flat CS profile.

We always encode the per-context attribute in the profile.

Where is the per-context attribute transferred from flat profile to nest profile? This is the only place I see we check ContextShouldBeInlined, but it's transferred to global FunctionSamples::ProfileIsPreinlined, not per-context attribute.

llvm/lib/Transforms/IPO/SampleProfile.cpp

1484

Yeah, ProfileIsNestedCS sounds better?

That works. Though it doesn't make sense for ProfileIsNestedCS to be true while ProfileIsCS is false. So if we use ProfileIsNestedCS, we need to either 1) rename ProfileIsCS to be ProfileIsFlatCS, or 2) replace check on ProfileIsCS with ProfileIsCS & !ProfileIsNestedCS

llvm/tools/llvm-profdata/llvm-profdata.cpp

952

how about gen-nested-cs-profile to make it explicit and differentiate from non-CS profile, especially if we go with ProfileIsNestedCS global flag.

[wenlei]

llvm/test/tools/llvm-profdata/Inputs/cs-sample-preinline.proftext

16

I meant the value change from 0/1 to 2, not the ones that were 0.

[hoy]

llvm/lib/ProfileData/SampleProf.cpp

534

Right. Here we are not necessarily converting a preinlined flat profile. When the flat profile is preinlined, we'll set the ProfileIsNestedCS flag, otherwise nodes in the context trie could have no profiles.

537–543

Did you mean would or would not? I expect the correct behavior to be that inliner will honor per-context ContextShouldBeInlined attribute, which is what happens with flat CS profile.

I mean would. Sorry for the confusion.

Where is the per-context attribute transferred from flat profile to nest profile? This is the only place I see we check ContextShouldBeInlined, but it's transferred to global FunctionSamples::ProfileIsPreinlined, not per-context attribute.

The attribute that comes with a context in a flat profile will be encoded with the context into the nested profile as well. Note that we only reset the name of the original context but nothing else. The original attribute is still there.

// Reset the child context to be contextless.
    ChildProfile->getContext().setName(OrigChildContext.getName());

See the test llvm/test/tools/llvm-profdata/cs-sample-nested-profile.test .

llvm/lib/Transforms/IPO/SampleProfile.cpp

1484

ProfileIsFlatCS and ProfileIsNestedCS sounds good to me.

llvm/test/tools/llvm-profdata/Inputs/cs-sample-preinline.proftext

16

Ah, this is the input profile of a test. I made it to be preinliner-generated. It was copied from cs-sample.proftext.

llvm/tools/llvm-profdata/llvm-profdata.cpp

952

Sounds good.

[hoy]

Renaming ProfileIsCS to ProfileIsCSFlat, ProfileIsPreinlined to ProfileIsCSNested.

[hoy]

Rebasing

[wenlei]

llvm/lib/ProfileData/SampleProf.cpp

534

So at least in preinliner case, it could be better to keep parent profile?

537–543

Now I see how the attributed is transferred, thanks for clarification.

llvm/lib/ProfileData/SampleProfReader.cpp

249

I'm still seeing SeenMetadata in the comment?

llvm/lib/Transforms/IPO/SampleProfile.cpp

1484

nit: there's NestedCS and CSFlat (as opposed to FlatCS) in switch, section flags and variable names , which is a bit inconsistent.

[hoy]

Renaming gen-nested-cs-profile to gen-cs-nested-profile

llvm/lib/ProfileData/SampleProf.cpp

534

Yes, that's currently it is. With preinliner, every node in the context trie will come with a profile so NodeProfile will never be empty. Without preinliner, NodeProfile could be empty.

llvm/lib/ProfileData/SampleProfReader.cpp

249

Good catch, changed to DepthMetadata.

llvm/lib/Transforms/IPO/SampleProfile.cpp

1484

Yeah, we still use the switch gen-nested-cs-profile. Let me change it to gen-cs-nested-profile

[wenlei]

lgtm, thanks.

llvm/lib/ProfileData/SampleProf.cpp

534

ok, sounds good.

[uabelho]

Hi,

Is anyone else having problems with this commit?
We get:

/proj/flexasic/app/llvm/8.0/bin/clang++  -march=corei7  -DGTEST_HAS_RTTI=0 -D_DEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/ProfileData -I../lib/ProfileData -Iinclude -I../include -fPIC -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG    -fno-exceptions -fno-rtti -UNDEBUG -std=c++14 -MD -MT lib/ProfileData/CMakeFiles/LLVMProfileData.dir/SampleProf.cpp.o -MF lib/ProfileData/CMakeFiles/LLVMProfileData.dir/SampleProf.cpp.o.d -o lib/ProfileData/CMakeFiles/LLVMProfileData.dir/SampleProf.cpp.o -c ../lib/ProfileData/SampleProf.cpp
In file included from ../lib/ProfileData/SampleProf.cpp:14:
In file included from ../include/llvm/ProfileData/SampleProf.h:17:
In file included from ../include/llvm/ADT/DenseSet.h:16:
In file included from ../include/llvm/ADT/DenseMap.h:16:
In file included from ../include/llvm/ADT/DenseMapInfo.h:19:
In file included from /proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/tuple:39:
In file included from /proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/array:38:
In file included from /proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/stdexcept:39:
In file included from /proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/string:41:
In file included from /proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/allocator.h:46:
In file included from /proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/x86_64-unknown-linux-gnu/bits/c++allocator.h:33:
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/ext/new_allocator.h:120:23: error: no matching constructor for initialization of 'std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples>'
        { ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
                             ^   ~~~~~~~~~~~~~~~~~~~~~~~~~~~
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/alloc_traits.h:530:8: note: in instantiation of function template specialization '__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > >::construct<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples>, llvm::StringRef, llvm::sampleprof::FunctionSamples &>' requested here
        { __a.construct(__p, std::forward<_Args>(__args)...); }
              ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_tree.h:529:23: note: in instantiation of function template specialization 'std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > > >::construct<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples>, llvm::StringRef, llvm::sampleprof::FunctionSamples &>' requested here
              _Alloc_traits::construct(_M_get_Node_allocator(),
                             ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_tree.h:546:4: note: in instantiation of function template specialization 'std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples>, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> >, std::less<void>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > >::_M_construct_node<llvm::StringRef, llvm::sampleprof::FunctionSamples &>' requested here
          _M_construct_node(__tmp, std::forward<_Args>(__args)...);
          ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_tree.h:2123:19: note: in instantiation of function template specialization 'std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples>, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> >, std::less<void>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > >::_M_create_node<llvm::StringRef, llvm::sampleprof::FunctionSamples &>' requested here
        _Link_type __z = _M_create_node(std::forward<_Args>(__args)...);
                         ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_map.h:559:16: note: in instantiation of function template specialization 'std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples>, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> >, std::less<void>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > >::_M_emplace_unique<llvm::StringRef, llvm::sampleprof::FunctionSamples &>' requested here
        { return _M_t._M_emplace_unique(std::forward<_Args>(__args)...); }
                      ^
../lib/ProfileData/SampleProf.cpp:524:18: note: in instantiation of function template specialization 'std::map<std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples, std::less<void>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > >::emplace<llvm::StringRef, llvm::sampleprof::FunctionSamples &>' requested here
      SamplesMap.emplace(OrigChildContext.getName(), *ChildProfile);
                 ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:112:26: note: candidate constructor not viable: no known conversion from 'llvm::StringRef' to 'const const std::__cxx11::basic_string<char>' for 1st argument
      _GLIBCXX_CONSTEXPR pair(const _T1& __a, const _T2& __b)
                         ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:138:12: note: candidate constructor template not viable: no known conversion from 'llvm::StringRef' to 'const const std::__cxx11::basic_string<char>' for 1st argument
        constexpr pair(const _T1& __x, _U2&& __y)
                  ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:133:12: note: candidate template ignored: requirement 'is_convertible<llvm::StringRef, const std::__cxx11::basic_string<char> >::value' was not satisfied [with _U1 = llvm::StringRef]
        constexpr pair(_U1&& __x, const _T2& __y)
                  ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:144:12: note: candidate template ignored: requirement '__and_<std::is_convertible<llvm::StringRef, const std::__cxx11::basic_string<char> >, std::is_convertible<llvm::sampleprof::FunctionSamples &, llvm::sampleprof::FunctionSamples> >::value' was not satisfied [with _U1 = llvm::StringRef, _U2 = llvm::sampleprof::FunctionSamples &]
        constexpr pair(_U1&& __x, _U2&& __y)
                  ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:124:12: note: candidate constructor template not viable: requires single argument '__p', but 2 arguments were provided
        constexpr pair(const pair<_U1, _U2>& __p)
                  ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:150:12: note: candidate constructor template not viable: requires single argument '__p', but 2 arguments were provided
        constexpr pair(pair<_U1, _U2>&& __p)
                  ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:155:9: note: candidate constructor template not viable: requires 3 arguments, but 2 were provided
        pair(piecewise_construct_t, tuple<_Args1...>, tuple<_Args2...>);
        ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:206:9: note: candidate constructor template not viable: requires 4 arguments, but 2 were provided
        pair(tuple<_Args1...>&, tuple<_Args2...>&,
        ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:127:17: note: candidate constructor not viable: requires 1 argument, but 2 were provided
      constexpr pair(const pair&) = default;
                ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:128:17: note: candidate constructor not viable: requires 1 argument, but 2 were provided
      constexpr pair(pair&&) = default;
                ^
/proj/flexasic/app/llvm/8.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_pair.h:108:26: note: candidate constructor not viable: requires 0 arguments, but 2 were provided
      _GLIBCXX_CONSTEXPR pair()
                         ^
1 error generated.

[uabelho]

Looks like this bot fails the same way:
https://lab.llvm.org/buildbot/#/builders/118/builds/6994

[foad]

@uabelho it looks like this has been fixed by 156b1e6ba84887ba92688b3ad31e400c56f707b2.

[uabelho]

In D115205#3194804, @foad wrote:

@uabelho it looks like this has been fixed by 156b1e6ba84887ba92688b3ad31e400c56f707b2.

Oh, yes it has. Thanks!

llvm/include/llvm/ProfileData/SampleProf.h

enum class SecProfSummaryFlags : uint32_t {
   /// SecFlagFSDiscriminator means this profile uses flow-sensitive
   /// discriminators.
   SecFlagFSDiscriminator = (1 << 2)
 };
 
 enum class SecFuncMetadataFlags : uint32_t {
   SecFlagInvalid = 0,
   SecFlagIsProbeBased = (1 << 0),
-  SecFlagHasAttribute = (1 << 1)
+  SecFlagHasAttribute = (1 << 1),
+  SecFlagIsPreInlined = (2 << 1),
 };
 
 enum class SecFuncOffsetFlags : uint32_t {
   SecFlagInvalid = 0,
   // Store function offsets in an order of contexts. The order ensures that
   // callee contexts of a given context laid out next to it.
   SecFlagOrdered = (1 << 0),
 };
std::string toString() const {
     return getContextString(FullContext, false);
   }
 
   uint64_t getHashCode() const {
     return hasContext() ? hash_value(getContextFrames())
                         : hash_value(getName());
   }
 
-  /// Set the name of the function.
+  /// Set the name of the function and clear the current context.
   void setName(StringRef FunctionName) {
-    assert(FullContext.empty() &&
-           "setName should only be called for non-CS profile");
     Name = FunctionName;
+    FullContext = SampleContextFrames();
+    State = UnknownContext;
   }
 
   void setContext(SampleContextFrames Context,
                   ContextStateMask CState = RawContext) {
     assert(CState != UnknownContext);
     FullContext = Context;
     Name = Context.back().FuncName;
     State = CState;
void updateTotalSamples() {
     for (auto &I : CallsiteSamples) {
       for (auto &CS : I.second) {
         CS.second.updateTotalSamples();
         addTotalSamples(CS.second.getTotalSamples());
       }
     }
   }
 
+  // Set current context and all callee contexts to be synthetic.
+  void SetContextSynthetic() {
+    Context.setState(SyntheticContext);
+    for (auto &I : CallsiteSamples) {
+      for (auto &CS : I.second) {
+        CS.second.SetContextSynthetic();
+      }
+    }
+  }
+
   /// Return the number of samples collected at the given location.
   /// Each location is specified by \p LineOffset and \p Discriminator.
   /// If the location is not found in profile, return error.
   ErrorOr<uint64_t> findSamplesAt(uint32_t LineOffset,
                                   uint32_t Discriminator) const {
     const auto &ret = BodySamples.find(LineLocation(LineOffset, Discriminator));
     if (ret == BodySamples.end())
       return std::error_code();
public:
   static unsigned getOffset(const DILocation *DIL);
 
   /// Returns a unique call site identifier for a given debug location of a call
   /// instruction. This is wrapper of two scenarios, the probe-based profile and
   /// regular profile, to hide implementation details from the sample loader and
   /// the context tracker.
   static LineLocation getCallSiteIdentifier(const DILocation *DIL);
 
+  /// Returns a unique hash code for a combination of a callsite location and
+  /// the callee function name.
+  static uint64_t getCallSiteHash(StringRef CalleeName,
+                                  const LineLocation &Callsite);
+
   /// Get the FunctionSamples of the inline instance where DIL originates
   /// from.
   ///
   /// The FunctionSamples of the instruction (Machine or IR) associated to
   /// \p DIL is the inlined instance in which that instruction is coming from.
   /// We traverse the inline stack of that instruction, and match it with the
   /// tree nodes in the profile.
   ///
   /// \returns the FunctionSamples pointer to the inlined instance.
   /// If \p Remapper is not nullptr, it will be used to find matching
   /// FunctionSamples with not exactly the same but equivalent name.
   const FunctionSamples *findFunctionSamples(
       const DILocation *DIL,
       SampleProfileReaderItaniumRemapper *Remapper = nullptr) const;
 
   static bool ProfileIsProbeBased;
 
   static bool ProfileIsCS;
 
+  static bool ProfileIsPreinlined;
+
   SampleContext &getContext() const { return Context; }
 
   void setContext(const SampleContext &FContext) { Context = FContext; }
 
   static SampleProfileFormat Format;
 
   /// Whether the profile uses MD5 to represent string.
   static bool UseMD5;
void trimAndMergeColdContextProfiles(uint64_t ColdCountThreshold,
                                        bool TrimBaseProfileOnly);
   // Canonicalize context profile name and attributes.
   void canonicalizeContextProfiles();
 
 private:
   SampleProfileMap &ProfileMap;
 };
 
+class CSProfileConverter {
+public:
+  CSProfileConverter(SampleProfileMap &Profiles);
+  void convertProfiles();
+  struct FrameNode {
+    FrameNode(StringRef FName = StringRef(),
+              FunctionSamples *FSamples = nullptr,
+              LineLocation CallLoc = {0, 0})
+        : FuncName(FName), FuncSamples(FSamples), CallSiteLoc(CallLoc){};
+
+    // Map line+discriminator location to child frame
+    std::map<uint64_t, FrameNode> AllChildFrames;
+    // Function name for current frame
+    StringRef FuncName;
+    // Function Samples for current frame
+    FunctionSamples *FuncSamples;
+    // Callsite location in parent context
+    LineLocation CallSiteLoc;
+
+    FrameNode *getOrCreateChildFrame(const LineLocation &CallSite,
+                                     StringRef CalleeName);
+  };
+
+private:
+  // Nest all children profiles into the profile of Node.
+  void convertProfiles(FrameNode &Node);
+  FrameNode *getOrCreateContextPath(const SampleContext &Context);
+
+  SampleProfileMap &ProfileMap;
+  FrameNode RootFrame;
+};
+
 /// ProfileSymbolList records the list of function symbols shown up
 /// in the binary used to generate the profile. It is useful to
 /// to discriminate a function being so cold as not to shown up
 /// in the profile and a function newly added.
 class ProfileSymbolList {
 public:
   /// copy indicates whether we need to copy the underlying memory
   /// for the input Name.

llvm/include/llvm/ProfileData/SampleProfReader.h

public:
   SampleProfileFormat getFormat() const { return Format; }
 
   /// Whether input profile is based on pseudo probes.
   bool profileIsProbeBased() const { return ProfileIsProbeBased; }
 
   /// Whether input profile is fully context-sensitive
   bool profileIsCS() const { return ProfileIsCS; }
 
+  bool profileIsPreinlined() const { return ProfileIsPreinlined; }
+
   virtual std::unique_ptr<ProfileSymbolList> getProfileSymbolList() {
     return nullptr;
   };
 
   /// It includes all the names that have samples either in outline instance
   /// or inline instance.
   virtual std::vector<StringRef> *getNameTable() { return nullptr; }
   virtual bool dumpSectionInfo(raw_ostream &OS = dbgs()) { return false; };
protected:
   std::unique_ptr<SampleProfileReaderItaniumRemapper> Remapper;
 
   /// \brief Whether samples are collected based on pseudo probes.
   bool ProfileIsProbeBased = false;
 
   /// Whether function profiles are context-sensitive.
   bool ProfileIsCS = false;
 
+  /// Whether function profiles are generated by the CS preinliner with full
+  /// inline decisions made.
+  bool ProfileIsPreinlined = false;
+
   /// Number of context-sensitive profiles.
   uint32_t CSProfileCount = 0;
 
   /// Whether the function profiles use FS discriminators.
   bool ProfileIsFS = false;
 
   /// \brief The format of sample.
   SampleProfileFormat Format = SPF_None;
private:
   BumpPtrAllocator Allocator;
 
 protected:
   std::vector<SecHdrTableEntry> SecHdrTable;
   std::error_code readSecHdrTableEntry(uint32_t Idx);
   std::error_code readSecHdrTable();
 
   std::error_code readFuncMetadata(bool ProfileHasAttribute);
+  std::error_code readFuncMetadata(bool ProfileHasAttribute,
+                                   FunctionSamples *FProfile, bool Inlined);
   std::error_code readFuncOffsetTable();
   std::error_code readFuncProfiles();
   std::error_code readMD5NameTable();
   std::error_code readNameTableSec(bool IsMD5);
   std::error_code readCSNameTableSec();
   std::error_code readProfileSymbolList();
 
   virtual std::error_code readHeader() override;

llvm/include/llvm/ProfileData/SampleProfWriter.h

protected:
   // Helper function to write name table.
   virtual std::error_code writeNameTable() override;
   virtual std::error_code
   writeContextIdx(const SampleContext &Context) override;
   std::error_code writeCSNameIdx(const SampleContext &Context);
   std::error_code writeCSNameTableSection();
 
   std::error_code writeFuncMetadata(const SampleProfileMap &Profiles);
+  std::error_code writeFuncMetadata(const FunctionSamples &Profile);
 
   // Functions to write various kinds of sections.
   std::error_code writeNameTableSection(const SampleProfileMap &ProfileMap);
   std::error_code writeFuncOffsetTable();
   std::error_code writeProfileSymbolListSection();
 
   SectionLayout SecLayout = DefaultLayout;
   // Specifiy the order of sections in section header table. Note

llvm/include/llvm/Transforms/IPO/SampleContextTracker.h

public:
   void addFunctionSize(uint32_t FSize);
   LineLocation getCallSiteLoc() const;
   ContextTrieNode *getParentContext() const;
   void setParentContext(ContextTrieNode *Parent);
   void dumpNode();
   void dumpTree();
 
 private:
-  static uint64_t nodeHash(StringRef ChildName, const LineLocation &Callsite);
-
   // Map line+discriminator location to child context
   std::map<uint64_t, ContextTrieNode> AllChildContext;
 
   // Link to parent context node
   ContextTrieNode *ParentContext;
 
   // Function name for current context
   StringRef FuncName;

llvm/lib/ProfileData/SampleProf.cpp

 using namespace llvm;
 using namespace sampleprof;
 
 static cl::opt<uint64_t> ProfileSymbolListCutOff(
     "profile-symbol-list-cutoff", cl::Hidden, cl::init(-1), cl::ZeroOrMore,
     cl::desc("Cutoff value about how many symbols in profile symbol list "
              "will be used. This is very useful for performance debugging"));
 
+cl::opt<bool> DuplicateContextProfilesIntoBaseProfile(
+    "duplicate-contexts-into-base", cl::init(true), cl::ZeroOrMore,
+    cl::desc("When converting a CS flat profile into a nested profile, "
+             "duplicating all context profiles of a function into its base "
+             "profile"));
+
 namespace llvm {
 namespace sampleprof {
 SampleProfileFormat FunctionSamples::Format;
 bool FunctionSamples::ProfileIsProbeBased = false;
 bool FunctionSamples::ProfileIsCS = false;
+bool FunctionSamples::ProfileIsPreinlined = false;
 bool FunctionSamples::UseMD5 = false;
 bool FunctionSamples::HasUniqSuffix = true;
 bool FunctionSamples::ProfileIsFS = false;
 } // namespace sampleprof
 } // namespace llvm
 
 namespace {
 

 }
 
 unsigned FunctionSamples::getOffset(const DILocation *DIL) {
   return (DIL->getLine() - DIL->getScope()->getSubprogram()->getLine()) &
       0xffff;
 }
 
 LineLocation FunctionSamples::getCallSiteIdentifier(const DILocation *DIL) {
-  if (FunctionSamples::ProfileIsProbeBased)
+  if (FunctionSamples::ProfileIsProbeBased) {
     // In a pseudo-probe based profile, a callsite is simply represented by the
     // ID of the probe associated with the call instruction. The probe ID is
     // encoded in the Discriminator field of the call instruction's debug
     // metadata.
     return LineLocation(PseudoProbeDwarfDiscriminator::extractProbeIndex(
                             DIL->getDiscriminator()),
                         0);
+  } else {
+    unsigned Discriminator;
+    if (FunctionSamples::ProfileIsFS)
+      Discriminator = DIL->getDiscriminator();
     else
-    return LineLocation(FunctionSamples::getOffset(DIL),
-                        DIL->getBaseDiscriminator());
+      Discriminator = DIL->getBaseDiscriminator();
+    return LineLocation(FunctionSamples::getOffset(DIL), Discriminator);
+  }
+}
+
+uint64_t FunctionSamples::getCallSiteHash(StringRef CalleeName,
+                                          const LineLocation &Callsite) {
+  uint64_t NameHash = std::hash<std::string>{}(CalleeName.str());
+  uint64_t LocId =
+      (((uint64_t)Callsite.LineOffset) << 32) | Callsite.Discriminator;
+  return NameHash + (LocId << 5) + LocId;
 }
 
 const FunctionSamples *FunctionSamples::findFunctionSamples(
     const DILocation *DIL, SampleProfileReaderItaniumRemapper *Remapper) const {
   assert(DIL);
   SmallVector<std::pair<LineLocation, StringRef>, 10> S;
 
   const DILocation *PrevDIL = DIL;
   for (DIL = DIL->getInlinedAt(); DIL; DIL = DIL->getInlinedAt()) {
-    unsigned Discriminator;
-    if (ProfileIsFS)
-      Discriminator = DIL->getDiscriminator();
-    else
-      Discriminator = DIL->getBaseDiscriminator();
-
     // Use C++ linkage name if possible.
     StringRef Name = PrevDIL->getScope()->getSubprogram()->getLinkageName();
     if (Name.empty())
       Name = PrevDIL->getScope()->getSubprogram()->getName();
-
-    S.push_back(
-        std::make_pair(LineLocation(getOffset(DIL), Discriminator), Name));
+    S.emplace_back(FunctionSamples::getCallSiteIdentifier(DIL), Name);
     PrevDIL = DIL;
   }
+
   if (S.size() == 0)
     return this;
   const FunctionSamples *FS = this;
   for (int i = S.size() - 1; i >= 0 && FS != nullptr; i--) {
     FS = FS->findFunctionSamplesAt(S[i].first, S[i].second, Remapper);
   }
   return FS;
 }

 void ProfileSymbolList::dump(raw_ostream &OS) const {
   OS << "======== Dump profile symbol list ========\n";
   std::vector<StringRef> SortedList(Syms.begin(), Syms.end());
   llvm::sort(SortedList);
 
   for (auto &Sym : SortedList)
     OS << Sym << "\n";
 }
+
+CSProfileConverter::FrameNode *
+CSProfileConverter::FrameNode::getOrCreateChildFrame(
+    const LineLocation &CallSite, StringRef CalleeName) {
+  uint64_t Hash = FunctionSamples::getCallSiteHash(CalleeName, CallSite);
+  auto It = AllChildFrames.find(Hash);
+  if (It != AllChildFrames.end()) {
+    assert(It->second.FuncName == CalleeName &&
+           "Hash collision for child context node");
+    return &It->second;
+  }
+
+  AllChildFrames[Hash] = FrameNode(CalleeName, nullptr, CallSite);
+  return &AllChildFrames[Hash];
+}
+
+CSProfileConverter::CSProfileConverter(SampleProfileMap &Profiles)
+    : ProfileMap(Profiles) {
+  for (auto &FuncSample : Profiles) {
+    FunctionSamples *FSamples = &FuncSample.second;
+    auto *NewNode = getOrCreateContextPath(FSamples->getContext());
+    assert(!NewNode->FuncSamples && "New node cannot have sample profile");
+    NewNode->FuncSamples = FSamples;
+  }
+}
+
+CSProfileConverter::FrameNode *
+CSProfileConverter::getOrCreateContextPath(const SampleContext &Context) {
+  auto Node = &RootFrame;
+  LineLocation CallSiteLoc(0, 0);
+  for (auto &Callsite : Context.getContextFrames()) {
+    Node = Node->getOrCreateChildFrame(CallSiteLoc, Callsite.FuncName);
+    CallSiteLoc = Callsite.Location;
+  }
+  return Node;
+}
+
+void CSProfileConverter::convertProfiles(CSProfileConverter::FrameNode &Node) {
+  // Process each child profile. Add each child profile to callsite profile map
+  // of the current node `Node` if `Node` comes with a profile. Otherwise
+  // promote the child profile to a standalone profile.
+  auto *NodeProfile = Node.FuncSamples;
+  for (auto &It : Node.AllChildFrames) {
+    auto &ChildNode = It.second;
+    convertProfiles(ChildNode);
+    auto *ChildProfile = ChildNode.FuncSamples;
+    if (!ChildProfile)
+      continue;
+    SampleContext OrigChildContext = ChildProfile->getContext();
+    // Reset the child context to be contextless.
+    ChildProfile->getContext().setName(OrigChildContext.getName());
+    if (NodeProfile) {
+      // Add child profile to the callsite profile map.
+      auto &SamplesMap = NodeProfile->functionSamplesAt(ChildNode.CallSiteLoc);
+      SamplesMap.emplace(OrigChildContext.getName(), *ChildProfile);
+      NodeProfile->addTotalSamples(ChildProfile->getTotalSamples());
+    }
+
+    // Separate child profile to be a standalone profile, if the current parent
+    // profile doesn't exist. This is a duplicating operation when the child
+    // profile is already incorporated into the parent which is still useful and
+    // thus done optionally. It is seen that duplicating context profiles into
+    // base profiles improves the code quality for thinlto build by allowing a
+    // profile in the prelink phase for to-be-fully-inlined functions.
+    if (!NodeProfile || DuplicateContextProfilesIntoBaseProfile)
+      ProfileMap[ChildProfile->getContext()].merge(*ChildProfile);
+
+    // Contexts coming with a `ContextShouldBeInlined` attribute indicate this
+    // is a preinliner-computed profile.
+    if (OrigChildContext.hasAttribute(ContextShouldBeInlined))
+      FunctionSamples::ProfileIsPreinlined = true;
+
+    // Remove the original child profile.
+    ProfileMap.erase(OrigChildContext);
+  }
+}
+
+void CSProfileConverter::convertProfiles() { convertProfiles(RootFrame); }

llvm/lib/ProfileData/SampleProfReader.cpp

static bool ParseLine(const StringRef &Input, LineType &LineTy, uint32_t &Depth,
                       uint32_t &Discriminator, StringRef &CalleeName,
                       DenseMap<StringRef, uint64_t> &TargetCountMap,
                       uint64_t &FunctionHash, uint32_t &Attributes) {
   for (Depth = 0; Input[Depth] == ' '; Depth++)
     ;
   if (Depth == 0)
     return false;
 
-  if (Depth == 1 && Input[Depth] == '!') {
+  if (Input[Depth] == '!') {
     LineTy = LineType::Metadata;
     return parseMetadata(Input.substr(Depth), FunctionHash, Attributes);
   }
 
   size_t n1 = Input.find(':');
   StringRef Loc = Input.substr(Depth, n1 - Depth);
   size_t n2 = Loc.find('.');
   if (n2 == StringRef::npos) {

 /// \returns true if the file was loaded successfully, false otherwise.
 std::error_code SampleProfileReaderText::readImpl() {
   line_iterator LineIt(*Buffer, /*SkipBlanks=*/true, '#');
   sampleprof_error Result = sampleprof_error::success;
 
   InlineCallStack InlineStack;
   uint32_t ProbeProfileCount = 0;
 
   // SeenMetadata tracks whether we have processed metadata for the current
   // top-level function profile.
-  bool SeenMetadata = false;
+  uint32_t DepthMetadata = 0;
 
   ProfileIsFS = ProfileIsFSDisciminator;
   FunctionSamples::ProfileIsFS = ProfileIsFS;
   for (; !LineIt.is_at_eof(); ++LineIt) {
     if ((*LineIt)[(*LineIt).find_first_not_of(' ')] == '#')
       continue;
     // Read the header of each function.
     //
for (; !LineIt.is_at_eof(); ++LineIt) {
     if ((*LineIt)[0] != ' ') {
       uint64_t NumSamples, NumHeadSamples;
       StringRef FName;
       if (!ParseHead(*LineIt, FName, NumSamples, NumHeadSamples)) {
         reportError(LineIt.line_number(),
                     "Expected 'mangled_name:NUM:NUM', found " + *LineIt);
         return sampleprof_error::malformed;
       }
-      SeenMetadata = false;
+      DepthMetadata = 0;
       SampleContext FContext(FName, CSNameTable);
       if (FContext.hasContext())
         ++CSProfileCount;
       Profiles[FContext] = FunctionSamples();
       FunctionSamples &FProfile = Profiles[FContext];
       FProfile.setContext(FContext);
       MergeResult(Result, FProfile.addTotalSamples(NumSamples));
       MergeResult(Result, FProfile.addHeadSamples(NumHeadSamples));
if ((*LineIt)[0] != ' ') {
       if (!ParseLine(*LineIt, LineTy, Depth, NumSamples, LineOffset,
                      Discriminator, FName, TargetCountMap, FunctionHash,
                      Attributes)) {
         reportError(LineIt.line_number(),
                     "Expected 'NUM[.NUM]: NUM[ mangled_name:NUM]*', found " +
                         *LineIt);
         return sampleprof_error::malformed;
       }
-      if (SeenMetadata && LineTy != LineType::Metadata) {
+      if (LineTy != LineType::Metadata && Depth == DepthMetadata) {
         // Metadata must be put at the end of a function profile.
         reportError(LineIt.line_number(),
                     "Found non-metadata after metadata: " + *LineIt);
         return sampleprof_error::malformed;
       }
 
       // Here we handle FS discriminators.
       Discriminator &= getDiscriminatorMask();
 
       while (InlineStack.size() > Depth) {
         InlineStack.pop_back();
       }
       switch (LineTy) {
       case LineType::CallSiteProfile: {
         FunctionSamples &FSamples = InlineStack.back()->functionSamplesAt(
             LineLocation(LineOffset, Discriminator))[std::string(FName)];
         FSamples.setName(FName);
         MergeResult(Result, FSamples.addTotalSamples(NumSamples));
         InlineStack.push_back(&FSamples);
+        DepthMetadata = 0;
         break;
       }
       case LineType::BodyProfile: {
         while (InlineStack.size() > Depth) {
           InlineStack.pop_back();
         }
         FunctionSamples &FProfile = *InlineStack.back();
         for (const auto &name_count : TargetCountMap) {
           MergeResult(Result, FProfile.addCalledTargetSamples(
                                   LineOffset, Discriminator, name_count.first,
                                   name_count.second));
         }
         MergeResult(Result, FProfile.addBodySamples(LineOffset, Discriminator,
                                                     NumSamples));
         break;
       }
       case LineType::Metadata: {
         FunctionSamples &FProfile = *InlineStack.back();
         if (FunctionHash) {
           FProfile.setFunctionHash(FunctionHash);
+          if (Depth == 1)
             ++ProbeProfileCount;
         }
-        if (Attributes)
-          FProfile.getContext().setAllAttributes(Attributes);
-        SeenMetadata = true;
+        if (Attributes & (uint32_t)ContextShouldBeInlined)
+          ProfileIsPreinlined = true;
+        DepthMetadata = Depth;
         break;
       }
       }
     }
   }
 
   assert((CSProfileCount == 0 || CSProfileCount == Profiles.size()) &&
          "Cannot have both context-sensitive and regular profile");
   ProfileIsCS = (CSProfileCount > 0);
   assert((ProbeProfileCount == 0 || ProbeProfileCount == Profiles.size()) &&
          "Cannot have both probe-based profiles and regular profiles");
   ProfileIsProbeBased = (ProbeProfileCount > 0);
   FunctionSamples::ProfileIsProbeBased = ProfileIsProbeBased;
   FunctionSamples::ProfileIsCS = ProfileIsCS;
+  FunctionSamples::ProfileIsPreinlined = ProfileIsPreinlined;
 
   if (Result == sampleprof_error::success)
     computeSummary();
 
   return Result;
 }
 
 bool SampleProfileReaderText::hasFormat(const MemoryBuffer &Buffer) {
case SecFuncOffsetTable:
     FuncOffsetsOrdered = hasSecFlag(Entry, SecFuncOffsetFlags::SecFlagOrdered);
     if (std::error_code EC = readFuncOffsetTable())
       return EC;
     break;
   case SecFuncMetadata: {
     ProfileIsProbeBased =
         hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagIsProbeBased);
     FunctionSamples::ProfileIsProbeBased = ProfileIsProbeBased;
+    ProfileIsPreinlined =
+        hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagIsPreInlined);
+    FunctionSamples::ProfileIsPreinlined = ProfileIsPreinlined;
     bool HasAttribute =
         hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagHasAttribute);
     if (std::error_code EC = readFuncMetadata(HasAttribute))
       return EC;
     break;
   }
   case SecProfileSymbolList:
     if (std::error_code EC = readProfileSymbolList())
std::error_code SampleProfileReaderExtBinaryBase::readCSNameTableSec() {
   }
 
   // From this point the underlying object of CSNameTable should be immutable.
   CSNameTable.reset(PNameVec);
   return sampleprof_error::success;
 }
 
 std::error_code
-SampleProfileReaderExtBinaryBase::readFuncMetadata(bool ProfileHasAttribute) {
-  while (Data < End) {
-    auto FContext(readSampleContextFromTable());
-    if (std::error_code EC = FContext.getError())
-      return EC;
 
-    bool ProfileInMap = Profiles.count(*FContext);
+SampleProfileReaderExtBinaryBase::readFuncMetadata(bool ProfileHasAttribute,
+                                                   FunctionSamples *FProfile,
+                                                   bool Inlined) {
+  if (Data < End) {
     if (ProfileIsProbeBased) {
       auto Checksum = readNumber<uint64_t>();
       if (std::error_code EC = Checksum.getError())
         return EC;
-      if (ProfileInMap)
-        Profiles[*FContext].setFunctionHash(*Checksum);
+      if (FProfile)
+        FProfile->setFunctionHash(*Checksum);
     }
 
     if (ProfileHasAttribute) {
       auto Attributes = readNumber<uint32_t>();
       if (std::error_code EC = Attributes.getError())
         return EC;
-      if (ProfileInMap)
-        Profiles[*FContext].getContext().setAllAttributes(*Attributes);
+      if (FProfile)
+        FProfile->getContext().setAllAttributes(*Attributes);
     }
+
+    if (!ProfileIsCS) {
+      // Read all the attributes for inlined function calls.
+      auto NumCallsites = readNumber<uint32_t>();
+      if (std::error_code EC = NumCallsites.getError())
+        return EC;
+
+      for (uint32_t J = 0; J < *NumCallsites; ++J) {
+        auto LineOffset = readNumber<uint64_t>();
+        if (std::error_code EC = LineOffset.getError())
+          return EC;
+
+        auto Discriminator = readNumber<uint64_t>();
+        if (std::error_code EC = Discriminator.getError())
+          return EC;
+
+        auto FContext(readSampleContextFromTable());
+        if (std::error_code EC = FContext.getError())
+          return EC;
+
+        FunctionSamples *CalleeProfile = nullptr;
+        if (FProfile) {
+          CalleeProfile = const_cast<FunctionSamples *>(
+              &FProfile->functionSamplesAt(LineLocation(
+                  *LineOffset,
+                  *Discriminator))[std::string(FContext.get().getName())]);
+        }
+        if (std::error_code EC =
+                readFuncMetadata(ProfileHasAttribute, CalleeProfile, true))
+          return EC;
+      }
+    }
+  }
+
+  return sampleprof_error::success;
+}
+
+std::error_code
+SampleProfileReaderExtBinaryBase::readFuncMetadata(bool ProfileHasAttribute) {
+  while (Data < End) {
+    auto FContext(readSampleContextFromTable());
+    if (std::error_code EC = FContext.getError())
+      return EC;
+    FunctionSamples *FProfile = nullptr;
+    auto It = Profiles.find(*FContext);
+    if (It != Profiles.end())
+      FProfile = &It->second;
+
+    if (std::error_code EC =
+            readFuncMetadata(ProfileHasAttribute, FProfile, false))
+      return EC;
   }
 
   assert(Data == End && "More data is read than expected");
   return sampleprof_error::success;
 }
 
 std::error_code SampleProfileReaderCompactBinary::readNameTable() {
   auto Size = readNumber<uint64_t>();
case SecFuncOffsetTable:
     if (hasSecFlag(Entry, SecFuncOffsetFlags::SecFlagOrdered))
       Flags.append("ordered,");
     break;
   case SecFuncMetadata:
     if (hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagIsProbeBased))
       Flags.append("probe,");
     if (hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagHasAttribute))
       Flags.append("attr,");
+    if (hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagIsPreInlined))
+      Flags.append("preinlined,");
     break;
   default:
     break;
   }
   char &last = Flags.back();
   if (last == ',')
     last = '}';
   else

llvm/lib/ProfileData/SampleProfWriter.cpp

if (FunctionSamples::ProfileIsCS) {
     }
   }
 
   FuncOffsetTable.clear();
   return sampleprof_error::success;
 }
 
 std::error_code SampleProfileWriterExtBinaryBase::writeFuncMetadata(
+    const FunctionSamples &FunctionProfile) {
+  auto &OS = *OutputStream;
+  if (std::error_code EC = writeContextIdx(FunctionProfile.getContext()))
+    return EC;
+
+  if (FunctionSamples::ProfileIsProbeBased)
+    encodeULEB128(FunctionProfile.getFunctionHash(), OS);
+  if (FunctionSamples::ProfileIsCS || FunctionSamples::ProfileIsPreinlined) {
+    encodeULEB128(FunctionProfile.getContext().getAllAttributes(), OS);
+  }
+
+  if (!FunctionSamples::ProfileIsCS) {
+    // Recursively emit attributes for all callee samples.
+    uint64_t NumCallsites = 0;
+    for (const auto &J : FunctionProfile.getCallsiteSamples())
+      NumCallsites += J.second.size();
+    encodeULEB128(NumCallsites, OS);
+    for (const auto &J : FunctionProfile.getCallsiteSamples()) {
+      for (const auto &FS : J.second) {
+        LineLocation Loc = J.first;
+        encodeULEB128(Loc.LineOffset, OS);
+        encodeULEB128(Loc.Discriminator, OS);
+        if (std::error_code EC = writeFuncMetadata(FS.second))
+          return EC;
+      }
+    }
+  }
+
+  return sampleprof_error::success;
+}
+
+std::error_code SampleProfileWriterExtBinaryBase::writeFuncMetadata(
     const SampleProfileMap &Profiles) {
-  if (!FunctionSamples::ProfileIsProbeBased && !FunctionSamples::ProfileIsCS)
+  if (!FunctionSamples::ProfileIsProbeBased && !FunctionSamples::ProfileIsCS &&
+      !FunctionSamples::ProfileIsPreinlined)
     return sampleprof_error::success;
-  auto &OS = *OutputStream;
   for (const auto &Entry : Profiles) {
-    if (std::error_code EC = writeContextIdx(Entry.second.getContext()))
+    if (std::error_code EC = writeFuncMetadata(Entry.second))
       return EC;
-    if (FunctionSamples::ProfileIsProbeBased)
-      encodeULEB128(Entry.second.getFunctionHash(), OS);
-    if (FunctionSamples::ProfileIsCS)
-      encodeULEB128(Entry.second.getContext().getAllAttributes(), OS);
   }
   return sampleprof_error::success;
 }
 
 std::error_code SampleProfileWriterExtBinaryBase::writeNameTable() {
   if (!UseMD5)
     return SampleProfileWriterBinary::writeNameTable();
 

 
 std::error_code SampleProfileWriterExtBinaryBase::writeOneSection(
     SecType Type, uint32_t LayoutIdx, const SampleProfileMap &ProfileMap) {
   // The setting of SecFlagCompress should happen before markSectionStart.
   if (Type == SecProfileSymbolList && ProfSymList && ProfSymList->toCompress())
     setToCompressSection(SecProfileSymbolList);
   if (Type == SecFuncMetadata && FunctionSamples::ProfileIsProbeBased)
     addSectionFlag(SecFuncMetadata, SecFuncMetadataFlags::SecFlagIsProbeBased);
+  if (Type == SecFuncMetadata && FunctionSamples::ProfileIsPreinlined)
+    addSectionFlag(SecFuncMetadata, SecFuncMetadataFlags::SecFlagIsPreInlined);
+  if (Type == SecFuncMetadata &&
+      (FunctionSamples::ProfileIsCS || FunctionSamples::ProfileIsPreinlined))
+    addSectionFlag(SecFuncMetadata, SecFuncMetadataFlags::SecFlagHasAttribute);
   if (Type == SecProfSummary && FunctionSamples::ProfileIsCS)
     addSectionFlag(SecProfSummary, SecProfSummaryFlags::SecFlagFullContext);
-  if (Type == SecFuncMetadata && FunctionSamples::ProfileIsCS)
-    addSectionFlag(SecFuncMetadata, SecFuncMetadataFlags::SecFlagHasAttribute);
   if (Type == SecProfSummary && FunctionSamples::ProfileIsFS)
     addSectionFlag(SecProfSummary, SecProfSummaryFlags::SecFlagFSDiscriminator);
 
   uint64_t SectionStart = markSectionStart(Type, LayoutIdx);
   switch (Type) {
   case SecProfSummary:
     computeSummary(ProfileMap);
     if (auto EC = writeSummary())
for (const auto &FS : I->second) {
         OS << Loc.LineOffset << ": ";
       else
         OS << Loc.LineOffset << "." << Loc.Discriminator << ": ";
       if (std::error_code EC = writeSample(CalleeSamples))
         return EC;
     }
   Indent -= 1;
 
-  if (Indent == 0) {
-    if (FunctionSamples::ProfileIsProbeBased) {
-      OS.indent(Indent + 1);
-      OS << "!CFGChecksum: " << S.getFunctionHash() << "\n";
-    }
-    if (FunctionSamples::ProfileIsCS) {
-      OS.indent(Indent + 1);
-      OS << "!Attributes: " << S.getContext().getAllAttributes() << "\n";
-    }
-  }
+
+  if (S.getContext().getAllAttributes()) {
 
   return sampleprof_error::success;
 }
 
 std::error_code
 SampleProfileWriterBinary::writeContextIdx(const SampleContext &Context) {
   assert(!Context.hasContext() && "cs profile is not supported");
   return writeNameIdx(Context.getName());

llvm/lib/Transforms/IPO/SampleContextTracker.cpp

 
 namespace llvm {
 
 ContextTrieNode *ContextTrieNode::getChildContext(const LineLocation &CallSite,
                                                   StringRef CalleeName) {
   if (CalleeName.empty())
     return getHottestChildContext(CallSite);
 
-  uint64_t Hash = nodeHash(CalleeName, CallSite);
+  uint64_t Hash = FunctionSamples::getCallSiteHash(CalleeName, CallSite);
   auto It = AllChildContext.find(Hash);
   if (It != AllChildContext.end())
     return &It->second;
   return nullptr;
 }
 
 ContextTrieNode *
 ContextTrieNode::getHottestChildContext(const LineLocation &CallSite) {
ContextTrieNode::getHottestChildContext(const LineLocation &CallSite) {
   }
 
   return ChildNodeRet;
 }
 
 ContextTrieNode &ContextTrieNode::moveToChildContext(
     const LineLocation &CallSite, ContextTrieNode &&NodeToMove,
     uint32_t ContextFramesToRemove, bool DeleteNode) {
-  uint64_t Hash = nodeHash(NodeToMove.getFuncName(), CallSite);
+  uint64_t Hash =
+      FunctionSamples::getCallSiteHash(NodeToMove.getFuncName(), CallSite);
   assert(!AllChildContext.count(Hash) && "Node to remove must exist");
   LineLocation OldCallSite = NodeToMove.CallSiteLoc;
   ContextTrieNode &OldParentContext = *NodeToMove.getParentContext();
   AllChildContext[Hash] = NodeToMove;
   ContextTrieNode &NewNode = AllChildContext[Hash];
   NewNode.CallSiteLoc = CallSite;
 
   // Walk through nodes in the moved the subtree, and update
ContextTrieNode &ContextTrieNode::moveToChildContext(
   if (DeleteNode)
     OldParentContext.removeChildContext(OldCallSite, NewNode.getFuncName());
 
   return NewNode;
 }
 
 void ContextTrieNode::removeChildContext(const LineLocation &CallSite,
                                          StringRef CalleeName) {
-  uint64_t Hash = nodeHash(CalleeName, CallSite);
+  uint64_t Hash = FunctionSamples::getCallSiteHash(CalleeName, CallSite);
   // Note this essentially calls dtor and destroys that child context
   AllChildContext.erase(Hash);
 }
 
 std::map<uint64_t, ContextTrieNode> &ContextTrieNode::getAllChildContext() {
   return AllChildContext;
 }
 
while (!NodeQueue.empty()) {
 
     for (auto &It : Node->getAllChildContext()) {
       ContextTrieNode *ChildNode = &It.second;
       NodeQueue.push(ChildNode);
     }
   }
 }
 
-uint64_t ContextTrieNode::nodeHash(StringRef ChildName,
-                                   const LineLocation &Callsite) {
-  // We still use child's name for child hash, this is
-  // because for children of root node, we don't have
-  // different line/discriminator, and we'll rely on name
-  // to differentiate children.
-  uint64_t NameHash = std::hash<std::string>{}(ChildName.str());
-  uint64_t LocId =
-      (((uint64_t)Callsite.LineOffset) << 32) | Callsite.Discriminator;
-  return NameHash + (LocId << 5) + LocId;
-}
-
 ContextTrieNode *ContextTrieNode::getOrCreateChildContext(
     const LineLocation &CallSite, StringRef CalleeName, bool AllowCreate) {
-  uint64_t Hash = nodeHash(CalleeName, CallSite);
+  uint64_t Hash = FunctionSamples::getCallSiteHash(CalleeName, CallSite);
   auto It = AllChildContext.find(Hash);
   if (It != AllChildContext.end()) {
     assert(It->second.getFuncName() == CalleeName &&
            "Hash collision for child context node");
     return &It->second;
   }
 
   if (!AllowCreate)

llvm/lib/Transforms/IPO/SampleProfile.cpp

protected:
   bool
   inlineHotFunctionsWithPriority(Function &F,
                                  DenseSet<GlobalValue::GUID> &InlinedGUIDs);
   // Inline cold/small functions in addition to hot ones
   bool shouldInlineColdCallee(CallBase &CallInst);
   void emitOptimizationRemarksForInlineCandidates(
       const SmallVectorImpl<CallBase *> &Candidates, const Function &F,
       bool Hot);
+  void accumulateNonInlinedSamples(
+      DenseMap<CallBase *, const FunctionSamples *> NonInlinedCallSites,
+      const Function &F);
   std::vector<Function *> buildFunctionOrder(Module &M, CallGraph *CG);
   std::unique_ptr<ProfiledCallGraph> buildProfiledCallGraph(CallGraph &CG);
   void generateMDProfMetadata(Function &F);
 
   /// Map from function name to Function *. Used to find the function from
   /// the function name. If the function name contains suffix, additional
   /// entry is added to map from the stripped name to the function if there
   /// is one-to-one mapping.
for (CallBase *I : CIS) {
                                     InlinedGUIDs, SymbolMap,
                                     PSI->getOrCompHotCountThreshold());
       }
     }
     Changed |= LocalChanged;
   }
 
   // For CS profile, profile for not inlined context will be merged when
-  // base profile is being trieved
-  if (ProfileIsCS)
-    return Changed;
-
-  // Accumulate not inlined callsite information into notInlinedSamples
-  for (const auto &Pair : LocalNotInlinedCallSites) {
-    CallBase *I = Pair.getFirst();
-    Function *Callee = I->getCalledFunction();
-    if (!Callee || Callee->isDeclaration())
-      continue;
-
-    ORE->emit(OptimizationRemarkAnalysis(CSINLINE_DEBUG, "NotInline",
-                                         I->getDebugLoc(), I->getParent())
-              << "previous inlining not repeated: '"
-              << ore::NV("Callee", Callee) << "' into '"
-              << ore::NV("Caller", &F) << "'");
-
-    ++NumCSNotInlined;
-    const FunctionSamples *FS = Pair.getSecond();
-    if (FS->getTotalSamples() == 0 && FS->getEntrySamples() == 0) {
-      continue;
-    }
-
-    if (ProfileMergeInlinee) {
-      // A function call can be replicated by optimizations like callsite
-      // splitting or jump threading and the replicates end up sharing the
-      // sample nested callee profile instead of slicing the original inlinee's
-      // profile. We want to do merge exactly once by filtering out callee
-      // profiles with a non-zero head sample count.
-      if (FS->getHeadSamples() == 0) {
-        // Use entry samples as head samples during the merge, as inlinees
-        // don't have head samples.
-        const_cast<FunctionSamples *>(FS)->addHeadSamples(
-            FS->getEntrySamples());
-
-        // Note that we have to do the merge right after processing function.
-        // This allows OutlineFS's profile to be used for annotation during
-        // top-down processing of functions' annotation.
-        FunctionSamples *OutlineFS = Reader->getOrCreateSamplesFor(*Callee);
-        OutlineFS->merge(*FS);
-      }
-    } else {
-      auto pair =
-          notInlinedCallInfo.try_emplace(Callee, NotInlinedProfileInfo{0});
-      pair.first->second.entryCount += FS->getEntrySamples();
-    }
-  }
+  // base profile is being retrieved.
+  if (!FunctionSamples::ProfileIsCS)
+    accumulateNonInlinedSamples(LocalNotInlinedCallSites, F);
   return Changed;
 }
 
 bool SampleProfileLoader::tryInlineCandidate(
     InlineCandidate &Candidate, SmallVector<CallBase *, 8> *InlinedCallSites) {
 
   CallBase &CB = *Candidate.CallInstr;
   Function *CalledFunction = CB.getCalledFunction();
SampleProfileLoader::shouldInlineCandidate(InlineCandidate &Candidate) {
 
   // Otherwise only use the cost from call analyzer, but overwite threshold with
   // Sample PGO threshold.
   return InlineCost::get(Cost.getCost(), SampleThreshold);
 }
 
 bool SampleProfileLoader::inlineHotFunctionsWithPriority(
     Function &F, DenseSet<GlobalValue::GUID> &InlinedGUIDs) {
-  assert(ProfileIsCS && "Prioritiy based inliner only works with CSSPGO now");
 
   // ProfAccForSymsInList is used in callsiteIsHot. The assertion makes sure
   // Profile symbol list is ignored when profile-sample-accurate is on.
   assert((!ProfAccForSymsInList ||
           (!ProfileSampleAccurate &&
            !F.hasFnAttribute("profile-sample-accurate"))) &&
          "ProfAccForSymsInList should be false when profile-sample-accurate "
          "is enabled");
assert(ProfileInlineLimitMax >= ProfileInlineLimitMin &&
          "Max inline size limit should not be smaller than min inline size "
          "limit.");
   unsigned SizeLimit = F.getInstructionCount() * ProfileInlineGrowthLimit;
   SizeLimit = std::min(SizeLimit, (unsigned)ProfileInlineLimitMax);
   SizeLimit = std::max(SizeLimit, (unsigned)ProfileInlineLimitMin);
   if (ExternalInlineAdvisor)
     SizeLimit = std::numeric_limits<unsigned>::max();
 
+  DenseMap<CallBase *, const FunctionSamples *> LocalNotInlinedCallSites;
+
   // Perform iterative BFS call site prioritized inlining
   bool Changed = false;
   while (!CQueue.empty() && F.getInstructionCount() < SizeLimit) {
     InlineCandidate Candidate = CQueue.top();
     CQueue.pop();
     CallBase *I = Candidate.CallInstr;
     Function *CalledFunction = I->getCalledFunction();
 
if (I->isIndirectCall()) {
         if (tryPromoteAndInlineCandidate(F, Candidate, SumOrigin, Sum,
                                          &InlinedCallSites)) {
           for (auto *CB : InlinedCallSites) {
             if (getInlineCandidate(&NewCandidate, CB))
               CQueue.emplace(NewCandidate);
           }
           ICPCount++;
           Changed = true;
+        } else if (FunctionSamples::ProfileIsPreinlined) {
+          LocalNotInlinedCallSites.try_emplace(I, FS);
         }
       }
     } else if (CalledFunction && CalledFunction->getSubprogram() &&
                !CalledFunction->isDeclaration()) {
       SmallVector<CallBase *, 8> InlinedCallSites;
       if (tryInlineCandidate(Candidate, &InlinedCallSites)) {
         for (auto *CB : InlinedCallSites) {
           if (getInlineCandidate(&NewCandidate, CB))
             CQueue.emplace(NewCandidate);
         }
         Changed = true;
+      } else if (FunctionSamples::ProfileIsPreinlined) {
+        LocalNotInlinedCallSites.try_emplace(I, Candidate.CalleeSamples);
       }
     } else if (LTOPhase == ThinOrFullLTOPhase::ThinLTOPreLink) {
       findExternalInlineCandidate(I, findCalleeFunctionSamples(*I),
                                   InlinedGUIDs, SymbolMap,
                                   PSI->getOrCompHotCountThreshold());
     }
   }
 
   if (!CQueue.empty()) {
     if (SizeLimit == (unsigned)ProfileInlineLimitMax)
       ++NumCSInlinedHitMaxLimit;
     else if (SizeLimit == (unsigned)ProfileInlineLimitMin)
       ++NumCSInlinedHitMinLimit;
     else
       ++NumCSInlinedHitGrowthLimit;
   }
 
+  // For CS profile, profile for not inlined context will be merged when
+  // base profile is being retrieved.
+  if (!FunctionSamples::ProfileIsCS)
+    accumulateNonInlinedSamples(LocalNotInlinedCallSites, F);
   return Changed;
 }
 
+void SampleProfileLoader::accumulateNonInlinedSamples(
+    DenseMap<CallBase *, const FunctionSamples *> NonInlinedCallSites,
+    const Function &F) {
+  // Accumulate not inlined callsite information into notInlinedSamples
+  for (const auto &Pair : NonInlinedCallSites) {
+    CallBase *I = Pair.getFirst();
+    Function *Callee = I->getCalledFunction();
+    if (!Callee || Callee->isDeclaration())
+      continue;
+
+    ORE->emit(OptimizationRemarkAnalysis(CSINLINE_DEBUG, "NotInline",
+                                         I->getDebugLoc(), I->getParent())
+              << "previous inlining not repeated: '"
+              << ore::NV("Callee", Callee) << "' into '"
+              << ore::NV("Caller", &F) << "'");
+
+    ++NumCSNotInlined;
+    const FunctionSamples *FS = Pair.getSecond();
+    if (FS->getTotalSamples() == 0 && FS->getEntrySamples() == 0) {
+      continue;
+    }
+
+    if (ProfileMergeInlinee) {
+      // A function call can be replicated by optimizations like callsite
+      // splitting or jump threading and the replicates end up sharing the
+      // sample nested callee profile instead of slicing the original
+      // inlinee's profile. We want to do merge exactly once by filtering out
+      // callee profiles with a non-zero head sample count.
+      if (FS->getHeadSamples() == 0) {
+        // Use entry samples as head samples during the merge, as inlinees
+        // don't have head samples.
+        const_cast<FunctionSamples *>(FS)->addHeadSamples(
+            FS->getEntrySamples());
+
+        // Note that we have to do the merge right after processing function.
+        // This allows OutlineFS's profile to be used for annotation during
+        // top-down processing of functions' annotation.
+        FunctionSamples *OutlineFS = Reader->getOrCreateSamplesFor(*Callee);
+        OutlineFS->merge(*FS, 1);
+        // Set outlined profile to be synthetic to not bias the inliner.
+        OutlineFS->SetContextSynthetic();
+      }
+    } else {
+      auto pair =
+          notInlinedCallInfo.try_emplace(Callee, NotInlinedProfileInfo{0});
+      pair.first->second.entryCount += FS->getEntrySamples();
+    }
+  }
+}
+
 /// Returns the sorted CallTargetMap \p M by count in descending order.
 static SmallVector<InstrProfValueData, 2>
 GetSortedValueDataFromCallTargets(const SampleRecord::CallTargetMap &M) {
   SmallVector<InstrProfValueData, 2> R;
   for (const auto &I : SampleRecord::SortCallTargets(M)) {
     R.emplace_back(
         InstrProfValueData{FunctionSamples::getGUID(I.first), I.second});
   }
if (FunctionSamples::ProfileIsProbeBased) {
     if (getFunctionLoc(F) == 0)
       return false;
 
     LLVM_DEBUG(dbgs() << "Line number for the first instruction in "
                       << F.getName() << ": " << getFunctionLoc(F) << "\n");
   }
 
   DenseSet<GlobalValue::GUID> InlinedGUIDs;
-  if (ProfileIsCS && CallsitePrioritizedInline)
+  if (CallsitePrioritizedInline)
     Changed |= inlineHotFunctionsWithPriority(F, InlinedGUIDs);
   else
     Changed |= inlineHotFunctions(F, InlinedGUIDs);
 
   Changed |= computeAndPropagateWeights(F, InlinedGUIDs);
 
   if (Changed)
     generateMDProfMetadata(F);
ExternalInlineAdvisor = getReplayInlineAdvisor(
         ReplayInlinerSettings{ProfileInlineReplayFile,
                               ProfileInlineReplayScope,
                               ProfileInlineReplayFallback,
                               {ProfileInlineReplayFormat}},
         /*EmitRemarks=*/false);
   }
 
   // Apply tweaks if context-sensitive profile is available.
-  if (Reader->profileIsCS()) {
-    ProfileIsCS = true;
-    FunctionSamples::ProfileIsCS = true;
-
+  if (Reader->profileIsCS() || Reader->profileIsPreinlined()) {
+    ProfileIsCS = Reader->profileIsCS();
     // Enable priority-base inliner and size inline by default for CSSPGO.
     if (!ProfileSizeInline.getNumOccurrences())
       ProfileSizeInline = true;
     if (!CallsitePrioritizedInline.getNumOccurrences())
       CallsitePrioritizedInline = true;
 
     // For CSSPGO, use preinliner decision by default when available.
     if (!UsePreInlinerDecision.getNumOccurrences())
       UsePreInlinerDecision = true;
 
     // For CSSPGO, we also allow recursive inline to best use context profile.
     if (!AllowRecursiveInline.getNumOccurrences())
       AllowRecursiveInline = true;
 
     // Enable iterative-BFI by default for CSSPGO.
     if (!UseIterativeBFIInference.getNumOccurrences())
       UseIterativeBFIInference = true;
     // Enable Profi by default for CSSPGO.
     if (!SampleProfileUseProfi.getNumOccurrences())
       SampleProfileUseProfi = true;
 
+    if (FunctionSamples::ProfileIsCS) {
       // Tracker for profiles under different context
       ContextTracker = std::make_unique<SampleContextTracker>(
           Reader->getProfiles(), &GUIDToFuncNameMap);
     }
+  }
 
   // Load pseudo probe descriptors for probe-based function samples.
   if (Reader->profileIsProbeBased()) {
     ProbeManager = std::make_unique<PseudoProbeManager>(M);
     if (!ProbeManager->moduleIsProbed(M)) {
       const char *Msg =
           "Pseudo-probe-based profile requires SampleProfileProbePass";
       Ctx.diagnose(DiagnosticInfoSampleProfile(Filename, Msg));

llvm/test/Transforms/SampleProfile/csspgo-inline.ll

 ; Test for CSSPGO's new early inliner using priority queue
 
 ; Note that we need new pass manager to enable top-down processing for sample profile loader
 ; Test we inlined the following in top-down order with old inliner
 ;   main:3 @ _Z5funcAi
 ;   main:3 @ _Z5funcAi:1 @ _Z8funcLeafi
 ;   _Z5funcBi:1 @ _Z8funcLeafi
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -sample-profile-prioritized-inline=0 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
 
 ; RUN: llvm-profdata merge --sample --extbinary --use-md5 %S/Inputs/profile-context-tracker.prof -o %t.md5
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.md5 -sample-profile-inline-size -sample-profile-prioritized-inline=0 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
 
+; RUN: llvm-profdata merge --sample --text --gen-nested-profile %S/Inputs/profile-context-tracker.prof -o %t.prof
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.prof -sample-profile-inline-size -sample-profile-prioritized-inline=0 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
+
 ; With new FDO early inliner, callee entry count is used to drive inlining instead of callee total samples, so we get less inlining for given profile
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-NEW
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.prof -sample-profile-prioritized-inline -sample-profile-inline-size -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-NEW
 ;
 ; With new FDO early inliner, callee entry count is used to drive inlining instead of callee total samples, tuning hot cutoff can get us the same inlining
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -profile-summary-cutoff-hot=999900 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
 ;
 ; With new FDO early inliner, callee entry count is used to drive inlining instead of callee total samples, tuning cold sample profile inline threshold can get us the same inlining
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -sample-profile-cold-inline-threshold=200 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
 ;
 ; With new FDO early inliner and tuned cutoff, we can control inlining through size growth tuning knob.

llvm/test/Transforms/SampleProfile/csspgo-use-preinliner.ll

 ; Test for CSSPGO's sample loader inlining to make sure it can use llvm-profgen preinliner's decision
 
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/csspgo-use-preinliner.prof -pass-remarks=inline -sample-profile-prioritized-inline -profile-sample-accurate -sample-profile-use-preinliner=0 -S 2>&1 | FileCheck %s --check-prefix=DEFAULT
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/csspgo-use-preinliner.prof -pass-remarks=inline -sample-profile-prioritized-inline -profile-sample-accurate -sample-profile-use-preinliner=1 -S 2>&1 | FileCheck %s --check-prefix=PREINLINE
 
+; RUN: llvm-profdata merge --sample --text --gen-nested-profile -duplicate-contexts-into-base=0 %S/Inputs/csspgo-use-preinliner.prof -o %t.prof
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.prof -pass-remarks=inline -sample-profile-prioritized-inline -profile-sample-accurate -sample-profile-use-preinliner=0 -S 2>&1 | FileCheck %s --check-prefix=DEFAULT
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.prof -pass-remarks=inline -sample-profile-prioritized-inline -profile-sample-accurate -sample-profile-use-preinliner=1 -S 2>&1 | FileCheck %s --check-prefix=PREINLINE
 
 ; DEFAULT: '_Z5funcAi' inlined into 'main'
 ; DEFAULT-NOT: inlined into
 
 ; PREINLINE-NOT: inlined into
 ; `[main:3 @ _Z5funcAi]` does not have preinline decision, so it's up for loader inliner to decided.
 ; PREINLINE: '_Z5funcAi' inlined into 'main'
 ; `[main:3 @ _Z5funcAi:1 @ _Z8funcLeafi]` is inlined according to preinline decision.

llvm/test/tools/llvm-profdata/Inputs/cs-sample-preinline.proftext

 [main:3 @ _Z5funcAi:1 @ _Z8funcLeafi]:1467299:11
  0: 6
  1: 6
  3: 287884
  4: 287864 _Z3fibi:315608
  15: 23
- !Attributes: 0
+ !Attributes: 2
 [main:3.1 @ _Z5funcBi:1 @ _Z8funcLeafi]:500853:20
  0: 15
  1: 15
  3: 74946
  4: 74941 _Z3fibi:82359
  10: 23324
  11: 23327 _Z3fibi:25228
  15: 11
- !Attributes: 1
+ !Attributes: 2
 [external:12 @ main]:154:12
  2: 12
  3: 10 _Z5funcAi:7
  3.1: 10 _Z5funcBi:11
- !Attributes: 0
 [main]:154:0
  2: 12
  3: 18 _Z5funcAi:11
  3.1: 18 _Z5funcBi:19
- !Attributes: 0
 [external:10 @ _Z5funcBi]:120:10
  0: 10
  1: 10
- !Attributes: 0
 [externalA:17 @ _Z5funcBi]:120:3
  0: 3
  1: 3
- !Attributes: 0
 [main:3.1 @ _Z5funcBi]:120:19
  0: 19
  1: 19 _Z8funcLeafi:20
  3: 12
- !Attributes: 1
+ !Attributes: 2
 [main:3 @ _Z5funcAi]:99:11
  0: 10
  1: 10 _Z8funcLeafi:11
  3: 24
- !Attributes: 0
+ !Attributes: 2

llvm/test/tools/llvm-profdata/Inputs/cs-sample.proftext

 [main:3 @ _Z5funcAi:1 @ _Z8funcLeafi]:1467299:11
  0: 6
  1: 6
  3: 287884
  4: 287864 _Z3fibi:315608
  15: 23
- !Attributes: 0
 [main:3.1 @ _Z5funcBi:1 @ _Z8funcLeafi]:500853:20
  0: 15
  1: 15
  3: 74946
  4: 74941 _Z3fibi:82359
  10: 23324
  11: 23327 _Z3fibi:25228
  15: 11
  !Attributes: 1
 [external:12 @ main]:154:12
  2: 12
  3: 10 _Z5funcAi:7
  3.1: 10 _Z5funcBi:11
- !Attributes: 0
 [main]:154:0
  2: 12
  3: 18 _Z5funcAi:11
  3.1: 18 _Z5funcBi:19
- !Attributes: 0
 [external:10 @ _Z5funcBi]:120:10
  0: 10
  1: 10
- !Attributes: 0
 [externalA:17 @ _Z5funcBi]:120:3
  0: 3
  1: 3
- !Attributes: 0
 [main:3.1 @ _Z5funcBi]:120:19
  0: 19
  1: 19 _Z8funcLeafi:20
  3: 12
  !Attributes: 1
 [main:3 @ _Z5funcAi]:99:11
  0: 10
  1: 10 _Z8funcLeafi:11
  3: 24
- !Attributes: 0

llvm/test/tools/llvm-profdata/cs-sample-nested-profile.test

+RUN: llvm-profdata merge --sample --text -output=%t.proftext %S/Inputs/cs-sample-preinline.proftext --gen-nested-profile=1 -duplicate-contexts-into-base=0
+RUN: FileCheck %s < %t.proftext --match-full-lines --strict-whitespace
+RUN: llvm-profdata merge --sample --extbinary -output=%t.profbin %S/Inputs/cs-sample-preinline.proftext --gen-nested-profile=1 -duplicate-contexts-into-base=0
+RUN: llvm-profdata merge --sample --text -output=%t2.proftext %t.profbin
+RUN: FileCheck %s < %t2.proftext --match-full-lines --strict-whitespace
+RUN: llvm-profdata show --sample -show-sec-info-only %t.profbin | FileCheck %s -check-prefix=PREINLINE
+RUN: llvm-profdata merge --sample --text -output=%t.proftext %S/Inputs/cs-sample-preinline.proftext --gen-nested-profile=1 -duplicate-contexts-into-base=1
+RUN: FileCheck %s < %t.proftext --match-full-lines --strict-whitespace -check-prefix=RECOUNT
+
+
+; CHECK:main:1968679:12
+; CHECK-NEXT: 2: 24
+; CHECK-NEXT: 3: 28 _Z5funcAi:18
+; CHECK-NEXT: 3.1: 28 _Z5funcBi:30
+; CHECK-NEXT: 3: _Z5funcAi:1467398
+; CHECK-NEXT:  0: 10
+; CHECK-NEXT:  1: 10 _Z8funcLeafi:11
+; CHECK-NEXT:  3: 24
+; CHECK-NEXT:  1: _Z8funcLeafi:1467299
+; CHECK-NEXT:   0: 6
+; CHECK-NEXT:   1: 6
+; CHECK-NEXT:   3: 287884
+; CHECK-NEXT:   4: 287864 _Z3fibi:315608
+; CHECK-NEXT:   15: 23
+; CHECK-NEXT:   !Attributes: 2
+; CHECK-NEXT:  !Attributes: 2
+; CHECK-NEXT: 3.1: _Z5funcBi:500973
+; CHECK-NEXT:  0: 19
+; CHECK-NEXT:  1: 19 _Z8funcLeafi:20
+; CHECK-NEXT:  3: 12
+; CHECK-NEXT:  1: _Z8funcLeafi:500853
+; CHECK-NEXT:   0: 15
+; CHECK-NEXT:   1: 15
+; CHECK-NEXT:   3: 74946
+; CHECK-NEXT:   4: 74941 _Z3fibi:82359
+; CHECK-NEXT:   10: 23324
+; CHECK-NEXT:   11: 23327 _Z3fibi:25228
+; CHECK-NEXT:   15: 11
+; CHECK-NEXT:   !Attributes: 2
+; CHECK-NEXT:  !Attributes: 2
+; CHECK-NEXT:_Z5funcBi:240:13
+; CHECK-NEXT: 0: 13
+; CHECK-NEXT: 1: 13
+
+
+
+; RECOUNT:main:1968679:12
+; RECOUNT-NEXT: 2: 24
+; RECOUNT-NEXT: 3: 28 _Z5funcAi:18
+; RECOUNT-NEXT: 3.1: 28 _Z5funcBi:30
+; RECOUNT-NEXT: 3: _Z5funcAi:1467398
+; RECOUNT-NEXT:  0: 10
+; RECOUNT-NEXT:  1: 10 _Z8funcLeafi:11
+; RECOUNT-NEXT:  3: 24
+; RECOUNT-NEXT:  1: _Z8funcLeafi:1467299
+; RECOUNT-NEXT:   0: 6
+; RECOUNT-NEXT:   1: 6
+; RECOUNT-NEXT:   3: 287884
+; RECOUNT-NEXT:   4: 287864 _Z3fibi:315608
+; RECOUNT-NEXT:   15: 23
+; RECOUNT-NEXT:   !Attributes: 2
+; RECOUNT-NEXT:  !Attributes: 2
+; RECOUNT-NEXT: 3.1: _Z5funcBi:500973
+; RECOUNT-NEXT:  0: 19
+; RECOUNT-NEXT:  1: 19 _Z8funcLeafi:20
+; RECOUNT-NEXT:  3: 12
+; RECOUNT-NEXT:  1: _Z8funcLeafi:500853
+; RECOUNT-NEXT:   0: 15
+; RECOUNT-NEXT:   1: 15
+; RECOUNT-NEXT:   3: 74946
+; RECOUNT-NEXT:   4: 74941 _Z3fibi:82359
+; RECOUNT-NEXT:   10: 23324
+; RECOUNT-NEXT:   11: 23327 _Z3fibi:25228
+; RECOUNT-NEXT:   15: 11
+; RECOUNT-NEXT:   !Attributes: 2
+; RECOUNT-NEXT:  !Attributes: 2
+; RECOUNT-NEXT:_Z8funcLeafi:1968152:31
+; RECOUNT-NEXT: 0: 21
+; RECOUNT-NEXT: 1: 21
+; RECOUNT-NEXT: 3: 362830
+; RECOUNT-NEXT: 4: 362805 _Z3fibi:397967
+; RECOUNT-NEXT: 10: 23324
+; RECOUNT-NEXT: 11: 23327 _Z3fibi:25228
+; RECOUNT-NEXT: 15: 34
+; RECOUNT-NEXT: !Attributes: 2
+; RECOUNT-NEXT:_Z5funcAi:1467398:11
+; RECOUNT-NEXT: 0: 10
+; RECOUNT-NEXT: 1: 10 _Z8funcLeafi:11
+; RECOUNT-NEXT: 3: 24
+; RECOUNT-NEXT: 1: _Z8funcLeafi:1467299
+; RECOUNT-NEXT:  0: 6
+; RECOUNT-NEXT:  1: 6
+; RECOUNT-NEXT:  3: 287884
+; RECOUNT-NEXT:  4: 287864 _Z3fibi:315608
+; RECOUNT-NEXT:  15: 23
+; RECOUNT-NEXT:  !Attributes: 2
+; RECOUNT-NEXT: !Attributes: 2
+; RECOUNT-NEXT:_Z5funcBi:501213:32
+; RECOUNT-NEXT: 0: 32
+; RECOUNT-NEXT: 1: 32 _Z8funcLeafi:20
+; RECOUNT-NEXT: 3: 12
+; RECOUNT-NEXT: 1: _Z8funcLeafi:500853
+; RECOUNT-NEXT:  0: 15
+; RECOUNT-NEXT:  1: 15
+; RECOUNT-NEXT:  3: 74946
+; RECOUNT-NEXT:  4: 74941 _Z3fibi:82359
+; RECOUNT-NEXT:  10: 23324
+; RECOUNT-NEXT:  11: 23327 _Z3fibi:25228
+; RECOUNT-NEXT:  15: 11
+; RECOUNT-NEXT:  !Attributes: 2
+
+
+; PREINLINE: FunctionMetadata {{.*}} Flags: {attr,preinlined}

llvm/test/tools/llvm-profdata/cs-sample-trimmer.test

 RUN: llvm-profdata merge --sample --text -output=%t.proftext %S/Inputs/cs-sample.proftext
 RUN: diff -b %t.proftext %S/Inputs/cs-sample.proftext
 
 RUN: llvm-profdata merge --sample --text -output=%t.proftext %S/Inputs/cs-sample.proftext -sample-merge-cold-context -profile-summary-cold-count=500
 RUN: FileCheck %s --input-file %t.proftext --check-prefixes=CHECK-TRIM,CHECK-MERGE
 RUN: llvm-profdata merge --sample --text -output=%t.proftext %S/Inputs/cs-sample.proftext -sample-merge-cold-context -sample-trim-cold-context -profile-summary-cold-count=500
 RUN: FileCheck %s --input-file %t.proftext --check-prefixes=CHECK-TRIM,CHECK-END
 RUN: llvm-profdata merge --sample --text -output=%t.proftext %S/Inputs/cs-sample.proftext -sample-merge-cold-context -sample-trim-cold-context -profile-summary-cutoff-cold=990000
 RUN: FileCheck %s --input-file %t.proftext --check-prefixes=CHECK-TRIM,CHECK-END
 
 CHECK-TRIM:      [main:3 @ _Z5funcAi:1 @ _Z8funcLeafi]:1467299:11
 CHECK-TRIM-NEXT:  0: 6
 CHECK-TRIM-NEXT:  1: 6
 CHECK-TRIM-NEXT:  3: 287884
 CHECK-TRIM-NEXT:  4: 287864 _Z3fibi:315608
 CHECK-TRIM-NEXT:  15: 23
-CHECK-TRIM-NEXT:  !Attributes: 0
 CHECK-TRIM-NEXT: [main:3.1 @ _Z5funcBi:1 @ _Z8funcLeafi]:500853:20
 CHECK-TRIM-NEXT:  0: 15
 CHECK-TRIM-NEXT:  1: 15
 CHECK-TRIM-NEXT:  3: 74946
 CHECK-TRIM-NEXT:  4: 74941 _Z3fibi:82359
 CHECK-TRIM-NEXT:  10: 23324
 CHECK-TRIM-NEXT:  11: 23327 _Z3fibi:25228
 CHECK-TRIM-NEXT:  15: 11
 CHECK-TRIM-NEXT:  !Attributes: 1
 CHECK-END-NOT:   [
 CHECK-MERGE:     [_Z5funcBi]:360:32
 CHECK-MERGE-NEXT: 0: 32
 CHECK-MERGE-NEXT: 1: 32 _Z8funcLeafi:20
 CHECK-MERGE-NEXT: 3: 12
-CHECK-MERGE-NEXT: !Attributes: 0
 CHECK-MERGE-NEXT:[main]:308:12
 CHECK-MERGE-NEXT: 2: 24
 CHECK-MERGE-NEXT: 3: 28 _Z5funcAi:18
 CHECK-MERGE-NEXT: 3.1: 28 _Z5funcBi:30
-CHECK-MERGE-NEXT: !Attributes: 0
 CHECK-MERGE-NEXT:[_Z5funcAi]:99:11
 CHECK-MERGE-NEXT: 0: 10
 CHECK-MERGE-NEXT: 1: 10 _Z8funcLeafi:11
 CHECK-MERGE-NEXT: 3: 24
-CHECK-MERGE-NEXT: !Attributes: 0

llvm/test/tools/llvm-profgen/cs-preinline.test

 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-cs-noprobe.perfscript --binary=%S/Inputs/inline-cs-noprobe.perfbin --output=%t --csspgo-preinliner=1  -sample-profile-cold-inline-threshold=0
 ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-NO-PREINL
 
 ; Test cold profile trimming. Only base profiles should be dropped.
 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-cs-noprobe.perfscript --binary=%S/Inputs/inline-cs-noprobe.perfbin --output=%t --csspgo-preinliner=1  --csprof-trim-cold-context=1 --profile-summary-hot-count=250
 
 ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-TRIM
 
+; Test llvm-profgen with preinliner on will merge not inlinable profile into base profile.
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-cs-noprobe.perfscript --binary=%S/Inputs/inline-cs-noprobe.perfbin --output=%t --csspgo-preinliner=1 --gen-nested-profile=1
+; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-PREINL-NEST
+
 ; CHECK-DEFAULT:     [main:1 @ foo]:44:0
 ; CHECK-DEFAULT-NEXT: 2.1: 14
 ; CHECK-DEFAULT-NEXT: 3: 15
 ; CHECK-DEFAULT-NEXT: 3.1: 14 bar:14
 ; CHECK-DEFAULT-NEXT: 3.2: 1
 ; CHECK-DEFAULT-NEXT: !Attributes: 1
 ; CHECK-DEFAULT-NEXT:[main:1 @ foo:3.1 @ bar]:14:0
 ; CHECK-DEFAULT-NEXT: 1: 14

 ; CHECK-NO-PREINL-NEXT:[bar]:14:0
 ; CHECK-NO-PREINL-NEXT: 1: 14
 ; CHECK-NO-PREINL-NEXT: !Attributes: 1
 
 ; CHECK-TRIM-NOT: [foo]
 ; CHECK-TRIM:[foo:3.1 @ bar]:14:0
 ; CHECK-TRIM-NEXT: 1: 14
 ; CHECK-TRIM-NEXT: !Attributes: 3
+
+; CHECK-PREINL-NEST:     foo:58:0
+; CHECK-PREINL-NEST-NEXT: 2.1: 14
+; CHECK-PREINL-NEST-NEXT: 3: 15
+; CHECK-PREINL-NEST-NEXT: 3.1: 14 bar:14
+; CHECK-PREINL-NEST-NEXT: 3.2: 1
+; CHECK-PREINL-NEST-NEXT: 3.1: bar:14
+; CHECK-PREINL-NEST-NEXT:  1: 14

llvm/test/tools/llvm-profgen/merge-cold-profile.test

 ; CHECK-NEXT: 1: 4
 ; CHECK-NEXT: 3: 4
 ; CHECK-NEXT: 4: 2
 ; CHECK-NEXT: 5: 1
 ; CHECK-NEXT: 6: 0
 ; CHECK-NEXT: 7: 2 fb:2
 ; CHECK-NEXT: 8: 1 fa:1
 ; CHECK-NEXT: !CFGChecksum: 563070469352221
-; CHECK-NEXT: !Attributes: 0
 ; CHECK-NEXT:[main:2 @ foo:5 @ fa:8 @ fa:7 @ fb:5 @ fb]:13:4
 ; CHECK-NEXT: 1: 4
 ; CHECK-NEXT: 2: 3
 ; CHECK-NEXT: 3: 1
 ; CHECK-NEXT: 4: 0
 ; CHECK-NEXT: 5: 4 fb:4
 ; CHECK-NEXT: 6: 1 fa:1
 ; CHECK-NEXT: !CFGChecksum: 563022570642068
 
 ; CHECK-KEEP-COLD:     [fb]:19:6
 ; CHECK-KEEP-COLD-NEXT: 1: 6
 ; CHECK-KEEP-COLD-NEXT: 2: 3
 ; CHECK-KEEP-COLD-NEXT: 3: 3
 ; CHECK-KEEP-COLD-NEXT: 4: 0
 ; CHECK-KEEP-COLD-NEXT: 5: 4 fb:4
 ; CHECK-KEEP-COLD-NEXT: 6: 3 fa:3
 ; CHECK-KEEP-COLD-NEXT: !CFGChecksum: 563022570642068
-; CHECK-KEEP-COLD-NEXT: !Attributes: 0
 ; CHECK-KEEP-COLD-NEXT:[fa]:14:4
 ; CHECK-KEEP-COLD-NEXT: 1: 4
 ; CHECK-KEEP-COLD-NEXT: 3: 4
 ; CHECK-KEEP-COLD-NEXT: 4: 2
 ; CHECK-KEEP-COLD-NEXT: 5: 1
 ; CHECK-KEEP-COLD-NEXT: 6: 0
 ; CHECK-KEEP-COLD-NEXT: 7: 2 fb:2
 ; CHECK-KEEP-COLD-NEXT: 8: 1 fa:1

 ; CHECK-COLD-CONTEXT-LENGTH:     [fb:5 @ fb]:13:4
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 1: 4
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 2: 3
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 3: 1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 4: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 5: 4 fb:4
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 6: 1 fa:1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: !CFGChecksum: 563022570642068
-; CHECK-COLD-CONTEXT-LENGTH-NEXT: !Attributes: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT:[fb:6 @ fa]:10:3
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 1: 3
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 3: 3
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 4: 1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 5: 1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 6: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 7: 1 fb:1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 8: 1 fa:1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: !CFGChecksum: 563070469352221
-; CHECK-COLD-CONTEXT-LENGTH-NEXT: !Attributes: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT:[fa:7 @ fb]:6:2
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 1: 2
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 2: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 3: 2
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 4: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 5: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 6: 2 fa:2
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: !CFGChecksum: 563022570642068
-; CHECK-COLD-CONTEXT-LENGTH-NEXT: !Attributes: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT:[fa:8 @ fa]:4:1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 1: 1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 3: 1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 4: 1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 5: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 6: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 7: 1 fb:1
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: 8: 0
 ; CHECK-COLD-CONTEXT-LENGTH-NEXT: !CFGChecksum: 563070469352221
-; CHECK-COLD-CONTEXT-LENGTH-NEXT: !Attributes: 0
 
 ; clang -O3 -fexperimental-new-pass-manager -fuse-ld=lld -fpseudo-probe-for-profiling
 ; -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Xclang -mdisable-tail-calls
 ; -g test.c  -o a.out
 
 ; Copied from recursion-compression.test
 #include <stdio.h>
 

llvm/test/tools/llvm-profgen/truncated-pseudoprobe.test

 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/truncated-pseudoprobe.perfscript --binary=%S/Inputs/truncated-pseudoprobe.perfbin --output=%t
 ; RUN: FileCheck %s --input-file %t
 
 ; CHECK:     [foo]:75:0
 ; CHECK-NEXT:  1: 0
 ; CHECK-NEXT:  2: 15
 ; CHECK-NEXT:  3: 15
 ; CHECK-NEXT:  4: 15
 ; CHECK-NEXT:  5: 0
 ; CHECK-NEXT:  6: 15
 ; CHECK-NEXT:  7: 0
 ; CHECK-NEXT:  8: 15 bar:15
 ; CHECK-NEXT:  9: 0
 ; CHECK-NEXT:  !CFGChecksum: 563088904013236
-; CHECK-NEXT:  !Attributes: 0
 ; CHECK:     [foo:8 @ bar]:30:15
 ; CHECK-NEXT:  1: 15
 ; CHECK-NEXT:  4: 15
 ; CHECK-NEXT:  !CFGChecksum: 72617220756
 ; CHECK-NEXT:  !Attributes: 1
 
 ; truncated-pseudoprobe.perfbin is from the following compile commands:
 ; llc truncated-pseudoprobe.ll -filetype=obj -o truncated-pseudoprobe.o
 ; clang truncated-pseudoprobe.o -o truncated-pseudoprobe.perfbin

llvm/tools/llvm-profdata/llvm-profdata.cpp

else
       Writer.setPartialProfile();
   }
 }
 
 static void
 mergeSampleProfile(const WeightedFileVector &Inputs, SymbolRemapper *Remapper,
                    StringRef OutputFilename, ProfileFormat OutputFormat,
                    StringRef ProfileSymbolListFile, bool CompressAllSections,
-                   bool UseMD5, bool GenPartialProfile,
+                   bool UseMD5, bool GenPartialProfile, bool GenNestedProfile,
                    bool SampleMergeColdContext, bool SampleTrimColdContext,
                    bool SampleColdContextFrameDepth, FailureMode FailMode) {
   using namespace sampleprof;
   SampleProfileMap ProfileMap;
   SmallVector<std::unique_ptr<sampleprof::SampleProfileReader>, 5> Readers;
   LLVMContext Context;
   sampleprof::ProfileSymbolList WriterList;
   Optional<bool> ProfileIsProbeBased;
for (SampleProfileMap::iterator I = Profiles.begin(), E = Profiles.end();
       MergeResult(Result, ProfileMap[FContext].merge(Samples, Input.Weight));
       if (Result != sampleprof_error::success) {
         std::error_code EC = make_error_code(Result);
         handleMergeWriterError(errorCodeToError(EC), Input.Filename,
                                FContext.toString());
       }
     }
 
+    if (ProfileIsCS && GenNestedProfile) {
+      CSProfileConverter CSConverter(ProfileMap);
+      CSConverter.convertProfiles();
+      ProfileIsCS = FunctionSamples::ProfileIsCS = false;
+    }
+
     std::unique_ptr<sampleprof::ProfileSymbolList> ReaderList =
         Reader->getProfileSymbolList();
     if (ReaderList)
       WriterList.merge(*ReaderList);
   }
 
   if (ProfileIsCS && (SampleMergeColdContext || SampleTrimColdContext)) {
     // Use threshold calculated from profile summary unless specified.
static int merge_main(int argc, const char *argv[]) {
   cl::opt<unsigned> SupplMinSizeThreshold(
       "suppl-min-size-threshold", cl::init(10), cl::Hidden,
       cl::desc("If the size of a function is smaller than the threshold, "
                "assume it can be inlined by PGO early inliner and it won't "
                "be adjusted based on sample profile."));
   cl::opt<unsigned> InstrProfColdThreshold(
       "instr-prof-cold-threshold", cl::init(0), cl::Hidden,
       cl::desc("User specified cold threshold for instr profile which will "
-               "override the cold threshold got from profile summary."));
+               "override the cold threshold got from profile summary. "));
+  cl::opt<bool> GenNestedProfile(
+      "gen-nested-profile", cl::Hidden, cl::init(false),
+      cl::desc("Generate nested function profiles for CSSPGO"));
 
   cl::ParseCommandLineOptions(argc, argv, "LLVM profile data merger\n");
 
   WeightedFileVector WeightedInputs;
   for (StringRef Filename : InputFilenames)
     addWeightedInput(WeightedInputs, {std::string(Filename), 1});
   for (StringRef WeightedFilename : WeightedInputFilenames)
     addWeightedInput(WeightedInputs, parseWeightedFile(WeightedFilename));
static int merge_main(int argc, const char *argv[]) {
   }
 
   if (ProfileKind == instr)
     mergeInstrProfile(WeightedInputs, Remapper.get(), OutputFilename,
                       OutputFormat, OutputSparse, NumThreads, FailureMode);
   else
     mergeSampleProfile(WeightedInputs, Remapper.get(), OutputFilename,
                        OutputFormat, ProfileSymbolListFile, CompressAllSections,
-                       UseMD5, GenPartialProfile, SampleMergeColdContext,
-                       SampleTrimColdContext, SampleColdContextFrameDepth,
-                       FailureMode);
-
+                       UseMD5, GenPartialProfile, GenNestedProfile,
+                       SampleMergeColdContext, SampleTrimColdContext,
+                       SampleColdContextFrameDepth, FailureMode);
   return 0;
 }
 
 /// Computer the overlap b/w profile BaseFilename and profile TestFilename.
 static void overlapInstrProfile(const std::string &BaseFilename,
                                 const std::string &TestFilename,
                                 const OverlapFuncFilters &FuncFilter,
                                 raw_fd_ostream &OS, bool IsCS) {

llvm/tools/llvm-profgen/ProfileGenerator.cpp

llvm::cl::desc(
         "specify density threshold for hot functions (default: 1000)"),
     llvm::cl::Optional);
 static cl::opt<bool> ShowDensity("show-density", llvm::cl::init(false),
                                  llvm::cl::desc("show profile density details"),
                                  llvm::cl::Optional);
 
 extern cl::opt<int> ProfileSummaryCutoffHot;
 
+static cl::opt<bool>
+    GenNestedProfile("gen-nested-profile", cl::Hidden, cl::init(false),
+                     cl::desc("Generate nested function profiles for CSSPGO"));
+
 using namespace llvm;
 using namespace sampleprof;
 
 namespace llvm {
 namespace sampleprof {
 
 // Initialize the MaxCompressionSize to -1 which means no size limit
 int32_t CSProfileGenerator::MaxCompressionSize = -1;
void CSProfileGenerator::postProcessProfiles() {
 
   // Merge function samples of CS profile to calculate profile density.
   sampleprof::SampleProfileMap ContextLessProfiles;
   for (const auto &I : ProfileMap) {
     ContextLessProfiles[I.second.getName()].merge(I.second);
   }
 
   calculateAndShowDensity(ContextLessProfiles);
+  if (GenNestedProfile) {
+    CSProfileConverter CSConverter(ProfileMap);
+    CSConverter.convertProfiles();
+    FunctionSamples::ProfileIsCS = false;
+    FunctionSamples::ProfileIsPreinlined = EnableCSPreInliner;
+  }
 }
 
 void ProfileGeneratorBase::computeSummaryAndThreshold() {
   SampleProfileSummaryBuilder Builder(ProfileSummaryBuilder::DefaultCutoffs);
   auto Summary = Builder.computeSummaryForProfiles(ProfileMap);
   HotCountThreshold = ProfileSummaryBuilder::getHotCountThreshold(
       (Summary->getDetailedSummary()));
   ColdCountThreshold = ProfileSummaryBuilder::getColdCountThreshold(