[SampleFDO] Enhance profile remapping support for inline instance

[wmi]

Summary

Profile remapping is a feature to match a function in the module with its profile in sample profile if the function name and the name in profile look different but are equivalent using given remapping rules. This is a useful feature to keep the performance stable by specifying some remapping rules when sampleFDO targets are going through some large scale function signature change.

However, currently profile remapping support is only valid for outline function profile in SampleFDO. It cannot match a callee with an inline instance profile if they have different but equivalent names. We found that without the support for inline instance profile, remapping is less effective for some large scale change.

To add that support, the patch introduces a temporary remapper in addition to the current remapper used for outline function profile. The temporary remapper will be populated with remapping rules at the same time as the regular remapper. At a callsite when compiler tries to find an inline instance profile to match with the callee, it will be populated with all the inline instance profiles at the same location as the callsite first, then check whether the callee name exists in the remapper. After the matching is done, the remapper should be cleared except that the remapping rules should be kept, that is why it is called temporary remapper. The clearing is necessary to prevent the callee name from being mistakenly matched with an inline instance profile at a different location or with an outline profile.

In a planned large scale change of int64 type (long long) to int64_t (long), we found the performance of a google internal benchmark degraded by 2% if nothing was done. That was very significant for us. If current profile remapping was enabled, the performance degradation dropped to 1.2%, and it was still significant. If the profile remapping with the current patch was enabled, the performance degradation further dropped to 0.14%, and that was small enough to be acceptable.

[wmi]

Remove an unrelated file being accidently included.

[hoy]

Nice to see the perf regression drops to around 1% with this change!

llvm/include/llvm/ProfileData/SampleProf.h

438

Nit: Update comment with Remapper?

llvm/lib/ProfileData/SampleProf.cpp

223

Does this remove only the last callee name added to the the remapper? Maybe give a more explicit name for that?

[wmi]

Hongtao, thanks for the review!

llvm/include/llvm/ProfileData/SampleProf.h

438

Updated.

llvm/lib/ProfileData/SampleProf.cpp

223

Yes, it only removes the last callee name. I cannot find a good name so I just change the remapper to CalleeRemapper.

[wmi]

Address Hongtao's comment.

[wenlei]

llvm/lib/ProfileData/SampleProf.cpp

218–222

I guess I didn't fully understand the clearing part - if a function (inlinee) is renamed, why do we not want to apply the remapping globally? E.g. for A->B, and C->B inlined profile, if B is renamed to D, we would want to match D to B for both A->D profile and C->D path, right? Then a global remapper including all names should suffice.

Could you share a concrete example as to why each function needs its own temp remapper for all its inlinees?

[wmi]

llvm/lib/ProfileData/SampleProf.cpp

218–222

Right, given two names B and D, ideally we want the remapper to tell us whether they are identical by returning true or false for remapper.equivalent(B, D). However, remapper doesn't work like this. To check whether B and D are identical, remapper needs to insert B into a foldingset first by parsing it, then check whether D has already existed in it by parsing D and trying to insert D (but not truely insert it).

Similarly, if we want to check whether an equivalent name exists in a nameset, remapper need to insert alll the names in the nameset before it can answer the query. So to answer the query whether the callee's name matches an inline instance profile at a specific callsite, it needs to clear names in the remapper inserted at other callsites before it inserts the callee's name at the current callsite into remapper.

Existing remapper doesn't have to be cleared throughout the sample profile loading pass because it insert all the names of outline instance in the sample profile, and the question it needs to answer in the whole pass is whether a function name has a match in the outline instances. So because of the way how remapper works, we need different remappers to answer different types of remapping queries.

[wenlei]

llvm/lib/ProfileData/SampleProf.cpp

218–222

Maybe I'm still missing something, but the part that I don't understand is why would we need to have boundary for nameset and the queries? For mangle names, they all live in a single conceptual "namespace", regardless of whether it's name for inlinee or outlined function.

For inlinee's name, it's possible that we've inserted their names in global remapper when initializing with all outlined functions' names. If not, we could also insert it into global remapper right before checking for alternative names? To me the only difference for inlinee's name is that, we might discover those names a bit later as we may not see them when initializing the global remapper with outlined function names, but that doesn't seem to warrant creating boundaries for nameset (between outline vs inlinee, inlinees for one functions vs inlinees for another).

[wmi]

llvm/lib/ProfileData/SampleProf.cpp

218–222

why would we need to have boundary for nameset and the queries?

That is because the query we want to have is whether a callee matches any inline instance at a specific callsite location, not the inline stances at any other callsites. That is why we want boundary for nameset related with the current callsite, and need to clear the names related with other callsites.

For inlinee's name, it's possible that we've inserted their names in global remapper

Yes, many inline instances have already exist as outline instance in global remapper. If we use global remapper, we can know a callee match an inline/outline instance somewhere in the profile, but don't know whether the callee match an inline instance at the current callsite.

[wenlei]

llvm/lib/ProfileData/SampleProf.cpp

218–222

Ok, thanks for clarification. Looking closer at the code, I think I understand the reason now. You're iterating over the old names to see if anyone of them matches variant of new name (callee name at the call site), then you have to limit the temp remapper to contain only the new name of interest (callee name).

For outlined functions profiles, the remapper provides new_name -> old_profile mapping (SampleProfileReaderItaniumRemapper::getSamplesFor(StringRef Fname)). Dealing with name->profile mapping is tricky for inlinee's profiles. But would it be possible to tweak the remapper to support new_name->old_name lookup? With that, inlinee profile can be handled universally by retrieving old name for new callee name, then iterate the call site profiles to get inlinee's profile.

And to be consistent with SampleProfileReaderItaniumRemapper::getSamplesFor for outline profile, handling of call site profile can also be abstracted away into the remapper?

Specifically, I'm thinking about handling inlinees in SampleProfileReaderItaniumRemapper::applyRemapping too and keeping a Key to old_name map, instead of Key to samples map.

[wmi]

llvm/lib/ProfileData/SampleProf.cpp

218–222

That is a very good suggestion! Following the suggestion I get the same remapping result with simpler code, and I find I can also add support for indirect call with the simpler implementation and am working on it.

[wmi]

Address Wenlei's comment and add the support to search indirect call promotion candidate using remapper.

[wenlei]

Great - thanks for making the changes. Looks good with two minor comments.

llvm/lib/Transforms/IPO/SampleProfile.cpp

1066

Do we actually need this extra check? Renaming won't change wether a call is self-recursive, so the check on line 1059 should be good enough?

1934

hoist Remapper = Reader->getRemapper() out of the loop?

[wmi]

llvm/lib/Transforms/IPO/SampleProfile.cpp

1066

Line 1059 only checks whether an inline instance at the callsite has the identical name with the parent function F. It is possible that the inline instance has a different but equivalent name as F. In that case, SymbolMap may map the name of the inline instance back to the definition of F and use F as the indirect call promotion target.

I can add one more check "R->getValue()->getName() != F.getName()" to the if (R != SymbolMap.end() ...) below and simplify the code a little bit.

1934

Good point.

[wenlei]

llvm/lib/Transforms/IPO/SampleProfile.cpp

1066

I see. Or R->getValue() != &F would work too?

[wmi]

llvm/lib/Transforms/IPO/SampleProfile.cpp

1066

Yes, that is better.

[wmi]

Address Wenlei's comment.

[wenlei]

LGTM, thanks!

[wmi]

Thanks Wenlei for the review and the insightful suggestions!

[hoyFB]

Looks even better now!

llvm/include/llvm/ProfileData/SampleProf.h

 private:
   uint64_t NumSamples = 0;
   CallTargetMap CallTargets;
 };
 
 raw_ostream &operator<<(raw_ostream &OS, const SampleRecord &Sample);
 
 class FunctionSamples;
+class SampleProfileReaderItaniumRemapper;
 
 using BodySampleMap = std::map<LineLocation, SampleRecord>;
 // NOTE: Using a StringMap here makes parsed profiles consume around 17% more
 // memory, which is *very* significant for large profiles.
 using FunctionSamplesMap = std::map<std::string, FunctionSamples, std::less<>>;
 using CallsiteSampleMap = std::map<LineLocation, FunctionSamplesMap>;
 
 /// Representation of the samples collected for a function.
if (iter == CallsiteSamples.end())
       return nullptr;
     return &iter->second;
   }
 
   /// Returns a pointer to FunctionSamples at the given callsite location \p Loc
   /// with callee \p CalleeName. If no callsite can be found, relax the
   /// restriction to return the FunctionSamples at callsite location \p Loc
   /// with the maximum total sample count.
-  const FunctionSamples *findFunctionSamplesAt(const LineLocation &Loc,
-                                               StringRef CalleeName) const {
-    std::string CalleeGUID;
-    CalleeName = getRepInFormat(CalleeName, UseMD5, CalleeGUID);
-
-    auto iter = CallsiteSamples.find(Loc);
-    if (iter == CallsiteSamples.end())
-      return nullptr;
-    auto FS = iter->second.find(CalleeName);
-    if (FS != iter->second.end())
-      return &FS->second;
-    // If we cannot find exact match of the callee name, return the FS with
-    // the max total count. Only do this when CalleeName is not provided,
-    // i.e., only for indirect calls.
-    if (!CalleeName.empty())
-      return nullptr;
-    uint64_t MaxTotalSamples = 0;
-    const FunctionSamples *R = nullptr;
-    for (const auto &NameFS : iter->second)
-      if (NameFS.second.getTotalSamples() >= MaxTotalSamples) {
-        MaxTotalSamples = NameFS.second.getTotalSamples();
-        R = &NameFS.second;
-      }
-    return R;
-  }
+  const FunctionSamples *findFunctionSamplesAt(
+      const LineLocation &Loc, StringRef CalleeName,
+      SampleProfileReaderItaniumRemapper *Remapper = nullptr) const;
 
   bool empty() const { return TotalSamples == 0; }
 
   /// Return the total number of samples collected inside the function.
   uint64_t getTotalSamples() const { return TotalSamples; }
 
   /// Return the total number of branch samples that have the function as the
   /// branch target. This should be equivalent to the sample of the first
public:
   /// from.
   ///
   /// The FunctionSamples of the instruction (Machine or IR) associated to
   /// \p DIL is the inlined instance in which that instruction is coming from.
   /// We traverse the inline stack of that instruction, and match it with the
   /// tree nodes in the profile.
   ///
   /// \returns the FunctionSamples pointer to the inlined instance.
-  const FunctionSamples *findFunctionSamples(const DILocation *DIL) const;
+  const FunctionSamples *findFunctionSamples(
+      const DILocation *DIL,
+      SampleProfileReaderItaniumRemapper *Remapper = nullptr) const;
 
   static SampleProfileFormat Format;
 
   /// Whether the profile uses MD5 to represent string.
   static bool UseMD5;
 
   /// GUIDToFuncNameMap saves the mapping from GUID to the symbol name, for
   /// all the function symbols defined or declared in current module.

llvm/include/llvm/ProfileData/SampleProfReader.h

public:
   void insert(StringRef FunctionName) { Remappings->insert(FunctionName); }
 
   /// Query whether there is equivalent in the remapper which has been
   /// inserted.
   bool exist(StringRef FunctionName) {
     return Remappings->lookup(FunctionName);
   }
 
+  /// Clear function names inserted into remapper.
+  void clear() { Remappings->clear(); }
+
   /// Return the samples collected for function \p F if remapper knows
   /// it is present in SampleMap.
   FunctionSamples *getSamplesFor(StringRef FunctionName);
 
 private:
   // The buffer holding the content read from remapping file.
   std::unique_ptr<MemoryBuffer> Buffer;
   std::unique_ptr<SymbolRemappingReader> Remappings;
public:
   /// It includes all the names that have samples either in outline instance
   /// or inline instance.
   virtual std::vector<StringRef> *getNameTable() { return nullptr; }
   virtual bool dumpSectionInfo(raw_ostream &OS = dbgs()) { return false; };
 
   /// Return whether names in the profile are all MD5 numbers.
   virtual bool useMD5() { return false; }
 
+  /// Return temporary remapper used for name matching at individual callsite.
+  SampleProfileReaderItaniumRemapper *getTempRemapper() {
+    return TempRemapper.get();
+  }
+
 protected:
   /// Map every function to its associated profile.
   ///
   /// The profile of every function executed at runtime is collected
   /// in the structure FunctionSamples. This maps function objects
   /// to their corresponding profiles.
   StringMap<FunctionSamples> Profiles;
 
protected:
   static std::unique_ptr<ProfileSummary>
   takeSummary(SampleProfileReader &Reader) {
     return std::move(Reader.Summary);
   }
 
   /// Compute summary for this profile.
   void computeSummary();
 
+  /// Remapper records the names of all the outlined function in the profile.
+  /// It is used to check whether a function has outlined profile. Remapper
+  /// is populated after profile reading and is kept unchanged afterwards.
   std::unique_ptr<SampleProfileReaderItaniumRemapper> Remapper;
+  /// TempRemapper first records the names of inline instances at a location
+  /// in the profile, and then it will be used to check whether the name of
+  /// given callsite can be found in the names recorded in TempRemapper, so
+  /// TempRemapper will be populated and cleared repeatedly for different
+  /// callsites.
+  std::unique_ptr<SampleProfileReaderItaniumRemapper> TempRemapper;
 
   /// \brief The format of sample.
   SampleProfileFormat Format = SPF_None;
 };
 
 class SampleProfileReaderText : public SampleProfileReader {
 public:
   SampleProfileReaderText(std::unique_ptr<MemoryBuffer> B, LLVMContext &C)

llvm/include/llvm/Support/ItaniumManglingCanonicalizer.h

public:
   ///
   /// The string denoted by Mangling must live as long as the canonicalizer.
   Key canonicalize(StringRef Mangling);
 
   /// Find a canonical key for the specified mangling, if one has already been
   /// formed. Otherwise returns Key().
   Key lookup(StringRef Mangling);
 
+  /// Clear canonical keys. The equivalence remapping is kept.
+  void clearCanonicalKeys();
+
 private:
   struct Impl;
   Impl *P;
 };
 } // namespace llvm
 
 #endif // LLVM_SUPPORT_ITANIUMMANGLINGCANONICALIZER_H

llvm/include/llvm/Support/SymbolRemappingReader.h

public:
   ///
   /// The result will typically be Key() if no equivalent symbol has been
   /// inserted, but this is not guaranteed: a Key different from all keys ever
   /// returned by \c insert may be returned instead.
   Key lookup(StringRef FunctionName) {
     return Canonicalizer.lookup(FunctionName);
   }
 
+  /// Clear canonical keys. The equivalence remapping is kept.
+  void clear() { Canonicalizer.clearCanonicalKeys(); }
+
 private:
   ItaniumManglingCanonicalizer Canonicalizer;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_SUPPORT_SYMBOLREMAPPINGREADER_H

llvm/lib/ProfileData/SampleProf.cpp

 //=-- SampleProf.cpp - Sample profiling format support --------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file contains common definitions used in the reading and writing of
 // sample profile data.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ProfileData/SampleProf.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/DebugInfoMetadata.h"
+#include "llvm/ProfileData/SampleProfReader.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/LEB128.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/raw_ostream.h"
 #include <string>
raw_ostream &llvm::sampleprof::operator<<(raw_ostream &OS,
   return OS;
 }
 
 unsigned FunctionSamples::getOffset(const DILocation *DIL) {
   return (DIL->getLine() - DIL->getScope()->getSubprogram()->getLine()) &
       0xffff;
 }
 
-const FunctionSamples *
-FunctionSamples::findFunctionSamples(const DILocation *DIL) const {
+const FunctionSamples *FunctionSamples::findFunctionSamples(
+    const DILocation *DIL, SampleProfileReaderItaniumRemapper *Remapper) const {
   assert(DIL);
   SmallVector<std::pair<LineLocation, StringRef>, 10> S;
 
   const DILocation *PrevDIL = DIL;
   for (DIL = DIL->getInlinedAt(); DIL; DIL = DIL->getInlinedAt()) {
     S.push_back(std::make_pair(
         LineLocation(getOffset(DIL), DIL->getBaseDiscriminator()),
         PrevDIL->getScope()->getSubprogram()->getLinkageName()));
     PrevDIL = DIL;
   }
   if (S.size() == 0)
     return this;
   const FunctionSamples *FS = this;
   for (int i = S.size() - 1; i >= 0 && FS != nullptr; i--) {
-    FS = FS->findFunctionSamplesAt(S[i].first, S[i].second);
+    FS = FS->findFunctionSamplesAt(S[i].first, S[i].second, Remapper);
   }
   return FS;
 }
 
+/// Returns a pointer to FunctionSamples at the given callsite location \p Loc
+/// with callee \p CalleeName. If no callsite can be found, relax the
+/// restriction to return the FunctionSamples at callsite location \p Loc
+/// with the maximum total sample count.
+const FunctionSamples *FunctionSamples::findFunctionSamplesAt(
+    const LineLocation &Loc, StringRef CalleeName,
+    SampleProfileReaderItaniumRemapper *Remapper) const {
+  std::string CalleeGUID;
+  CalleeName = getRepInFormat(CalleeName, UseMD5, CalleeGUID);
+
+  auto iter = CallsiteSamples.find(Loc);
+  if (iter == CallsiteSamples.end())
+    return nullptr;
+  auto FS = iter->second.find(CalleeName);
+  if (FS != iter->second.end())
+    return &FS->second;
+
+  // If we cannot find FunctionSamples with exact CalleeName and if Remapper
+  // is given, try whether we can find FunctionSamples with equivalent
+  // CalleeName using remapping rules. To avoid mixing profile inline instances
+  // at one callsite with inline instances at other callsites, the Remapper
+  // has to be cleared everytime before it is used, however the remapping table
+  // containing the remapping rules in the Remapper will be kept in clearing.
+  if (Remapper) {
+    Remapper->clear();
+    Remapper->insert(CalleeName);
+    for (const auto &NameFS : iter->second) {
+      if (Remapper->exist(NameFS.first))
+        return &NameFS.second;
+    }
+  }
+  // If we cannot find exact match of the callee name, return the FS with
+  // the max total count. Only do this when CalleeName is not provided,
+  // i.e., only for indirect calls.
+  if (!CalleeName.empty())
+    return nullptr;
+  uint64_t MaxTotalSamples = 0;
+  const FunctionSamples *R = nullptr;
+  for (const auto &NameFS : iter->second)
+    if (NameFS.second.getTotalSamples() >= MaxTotalSamples) {
+      MaxTotalSamples = NameFS.second.getTotalSamples();
+      R = &NameFS.second;
+    }
+  return R;
+}
+
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 LLVM_DUMP_METHOD void FunctionSamples::dump() const { print(dbgs(), 0); }
 #endif
 
 std::error_code ProfileSymbolList::read(const uint8_t *Data,
                                         uint64_t ListSize) {
   const char *ListStart = reinterpret_cast<const char *>(Data);
   uint64_t Size = 0;

llvm/lib/ProfileData/SampleProfReader.cpp

if (!RemapFilename.empty()) {
     auto ReaderOrErr =
         SampleProfileReaderItaniumRemapper::create(RemapFilename, *Reader, C);
     if (std::error_code EC = ReaderOrErr.getError()) {
       std::string Msg = "Could not create remapper: " + EC.message();
       C.diagnose(DiagnosticInfoSampleProfile(RemapFilename, Msg));
       return EC;
     }
     Reader->Remapper = std::move(ReaderOrErr.get());
+
+    ReaderOrErr =
+        SampleProfileReaderItaniumRemapper::create(RemapFilename, *Reader, C);
+    if (std::error_code EC = ReaderOrErr.getError()) {
+      std::string Msg = "Could not create remapper: " + EC.message();
+      C.diagnose(DiagnosticInfoSampleProfile(RemapFilename, Msg));
+      return EC;
+    }
+    Reader->TempRemapper = std::move(ReaderOrErr.get());
   }
 
   FunctionSamples::Format = Reader->getFormat();
   if (std::error_code EC = Reader->readHeader()) {
     return EC;
   }
 
   return std::move(Reader);

llvm/lib/Support/ItaniumManglingCanonicalizer.cpp

 //===----------------- ItaniumManglingCanonicalizer.cpp -------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Support/ItaniumManglingCanonicalizer.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/FoldingSet.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Demangle/ItaniumDemangle.h"
 #include "llvm/Support/Allocator.h"
 
 using namespace llvm;
 using llvm::itanium_demangle::ForwardTemplateReference;
 using llvm::itanium_demangle::Node;
template<> void ProfileNode::operator()(const ForwardTemplateReference *N) {
   llvm_unreachable("should never canonicalize a ForwardTemplateReference");
 }
 
 void profileNode(llvm::FoldingSetNodeID &ID, const Node *N) {
   N->visit(ProfileNode{ID});
 }
 
 class FoldingNodeAllocator {
+  BumpPtrAllocator RawAlloc;
+
+protected:
   class alignas(alignof(Node *)) NodeHeader : public llvm::FoldingSetNode {
   public:
     // 'Node' in this context names the injected-class-name of the base class.
     itanium_demangle::Node *getNode() {
       return reinterpret_cast<itanium_demangle::Node *>(this + 1);
     }
     void Profile(llvm::FoldingSetNodeID &ID) { profileNode(ID, getNode()); }
   };
-
-  BumpPtrAllocator RawAlloc;
   llvm::FoldingSet<NodeHeader> Nodes;
 
 public:
   void reset() {}
 
   template <typename T, typename... Args>
   std::pair<Node *, bool> getOrCreateNode(bool CreateNewNodes, Args &&... As) {
     // FIXME: Don't canonicalize forward template references for now, because
public:
 
   bool isMostRecentlyCreated(Node *N) const { return MostRecentlyCreated == N; }
 
   void trackUsesOf(Node *N) {
     TrackedNode = N;
     TrackedNodeIsUsed = false;
   }
   bool trackedNodeIsUsed() const { return TrackedNodeIsUsed; }
+
+  void clearNodesNotRemapping();
 };
 
+/// Clear all the nodes unrelated with remapping rules.
+void CanonicalizerAllocator::clearNodesNotRemapping() {
+  // Collect Nodes in Remappings.
+  DenseSet<itanium_demangle::Node *> NodesInRemappings;
+  for (auto &NodePair : Remappings) {
+    NodesInRemappings.insert(NodePair.first);
+    NodesInRemappings.insert(NodePair.second);
+  }
+
+  // Collect all the NodeHeaders in Nodes but not in Remappings
+  // into NodeHeaderSet.
+  DenseSet<NodeHeader *> NodeHeaderSet;
+  for (auto &NH : Nodes) {
+    if (NodesInRemappings.contains(NH.getNode()))
+      continue;
+    NodeHeaderSet.insert(&NH);
+  }
+
+  // Clear all the NodeHeaders in NodeHeaderSet.
+  for (auto &NH : NodeHeaderSet)
+    Nodes.RemoveNode(NH);
+}
+
 /// Convert St3foo to NSt3fooE so that equivalences naming one also affect the
 /// other.
 template<>
 struct CanonicalizerAllocator::MakeNodeImpl<
            itanium_demangle::StdQualifiedName> {
   CanonicalizerAllocator &Self;
   Node *make(Node *Child) {
     Node *StdNamespace = Self.makeNode<itanium_demangle::NameType>("std");

 ItaniumManglingCanonicalizer::canonicalize(StringRef Mangling) {
   return parseMaybeMangledName(P->Demangler, Mangling, true);
 }
 
 ItaniumManglingCanonicalizer::Key
 ItaniumManglingCanonicalizer::lookup(StringRef Mangling) {
   return parseMaybeMangledName(P->Demangler, Mangling, false);
 }
+
+void ItaniumManglingCanonicalizer::clearCanonicalKeys() {
+  P->Demangler.ASTAllocator.clearNodesNotRemapping();
+}

llvm/lib/Transforms/IPO/SampleProfile.cpp

if (Function *Callee = Inst.getCalledFunction())
     CalleeName = Callee->getName();
 
   const FunctionSamples *FS = findFunctionSamples(Inst);
   if (FS == nullptr)
     return nullptr;
 
   return FS->findFunctionSamplesAt(LineLocation(FunctionSamples::getOffset(DIL),
                                                 DIL->getBaseDiscriminator()),
-                                   CalleeName);
+                                   CalleeName, Reader->getTempRemapper());
 }
 
 /// Returns a vector of FunctionSamples that are the indirect call targets
 /// of \p Inst. The vector is sorted by the total number of samples. Stores
 /// the total call count of the indirect call in \p Sum.
 std::vector<const FunctionSamples *>
 SampleProfileLoader::findIndirectCallFunctionSamples(
     const Instruction &Inst, uint64_t &Sum) const {

 const FunctionSamples *
 SampleProfileLoader::findFunctionSamples(const Instruction &Inst) const {
   const DILocation *DIL = Inst.getDebugLoc();
   if (!DIL)
     return Samples;
 
   auto it = DILocation2SampleMap.try_emplace(DIL,nullptr);
   if (it.second)
-    it.first->second = Samples->findFunctionSamples(DIL);
+    it.first->second =
+        Samples->findFunctionSamples(DIL, Reader->getTempRemapper());
   return it.first->second;
 }
 
 bool SampleProfileLoader::inlineCallInstruction(CallBase &CB) {
   if (ExternalInlineAdvisor) {
     auto Advice = ExternalInlineAdvisor->getAdvice(CB);
     if (!Advice->isInliningRecommended()) {
       Advice->recordUnattemptedInlining();
for (CallBase *I : CIS) {
           // recursive. As llvm does not inline recursive calls, we will
           // simply ignore it instead of handling it explicitly.
           if (CalleeFunctionName == F.getName())
             continue;
 
           if (!callsiteIsHot(FS, PSI))
             continue;
 
           const char *Reason = "Callee function not available";
           auto R = SymbolMap.find(CalleeFunctionName);
           if (R != SymbolMap.end() && R->getValue() &&
               !R->getValue()->isDeclaration() &&
               R->getValue()->getSubprogram() &&
               R->getValue()->hasFnAttribute("use-sample-profile") &&
               isLegalToPromote(*I, R->getValue(), &Reason)) {
             uint64_t C = FS->getEntrySamples();
             auto &DI =
if (pos != StringRef::npos) {
       // thus there are multiple functions that are mapped to the same
       // stripped name. In this case of name conflicting, set the value
       // to nullptr to avoid confusion.
       if (!r.second)
         r.first->second = nullptr;
     }
   }
 
   bool retval = false;
   for (auto F : buildFunctionOrder(M, CG)) {
     assert(!F->isDeclaration());
     clearFunctionData();
     retval |= runOnFunction(*F, AM);
   }
 
   // Account for cold calls not inlined....
   for (const std::pair<Function *, NotInlinedProfileInfo> &pair :

llvm/unittests/ProfileData/SampleProfTest.cpp

void verifyProfileSummary(ProfileSummary &Summary, Module &M,
                             const bool AddPartialField,
                             const bool AddPartialProfileRatioField) {
     LLVMContext &Context = M.getContext();
     const bool IsPartialProfile = Summary.isPartialProfile();
     const double PartialProfileRatio = Summary.getPartialProfileRatio();
     auto VerifySummary = [IsPartialProfile, PartialProfileRatio](
                              ProfileSummary &Summary) mutable {
       ASSERT_EQ(ProfileSummary::PSK_Sample, Summary.getKind());
-      ASSERT_EQ(137392u, Summary.getTotalCount());
-      ASSERT_EQ(8u, Summary.getNumCounts());
+      ASSERT_EQ(138211u, Summary.getTotalCount());
+      ASSERT_EQ(10u, Summary.getNumCounts());
       ASSERT_EQ(4u, Summary.getNumFunctions());
       ASSERT_EQ(1437u, Summary.getMaxFunctionCount());
       ASSERT_EQ(60351u, Summary.getMaxCount());
       ASSERT_EQ(IsPartialProfile, Summary.isPartialProfile());
       ASSERT_EQ(PartialProfileRatio, Summary.getPartialProfileRatio());
 
       uint32_t Cutoff = 800000;
       auto Predicate = [&Cutoff](const ProfileSummaryEntry &PE) {
         return PE.Cutoff == Cutoff;
       };
       std::vector<ProfileSummaryEntry> &Details = Summary.getDetailedSummary();
       auto EightyPerc = find_if(Details, Predicate);
       Cutoff = 900000;
       auto NinetyPerc = find_if(Details, Predicate);
       Cutoff = 950000;
       auto NinetyFivePerc = find_if(Details, Predicate);
       Cutoff = 990000;
       auto NinetyNinePerc = find_if(Details, Predicate);
       ASSERT_EQ(60000u, EightyPerc->MinCount);
       ASSERT_EQ(12557u, NinetyPerc->MinCount);
       ASSERT_EQ(12557u, NinetyFivePerc->MinCount);
-      ASSERT_EQ(610u, NinetyNinePerc->MinCount);
+      ASSERT_EQ(600u, NinetyNinePerc->MinCount);
     };
     VerifySummary(Summary);
 
     // Test that conversion of summary to and from Metadata works.
     Metadata *MD =
         Summary.getMD(Context, AddPartialField, AddPartialProfileRatioField);
     ASSERT_TRUE(MD);
     ProfileSummary *PS = ProfileSummary::getFromMD(MD);
void testRoundTrip(SampleProfileFormat Format, bool Remap, bool UseMD5) {
     FooSamples.addTotalSamples(7711);
     FooSamples.addHeadSamples(610);
     FooSamples.addBodySamples(1, 0, 610);
     FooSamples.addBodySamples(2, 0, 600);
     FooSamples.addBodySamples(4, 0, 60000);
     FooSamples.addBodySamples(8, 0, 60351);
     FooSamples.addBodySamples(10, 0, 605);
 
+    // Add inline instance with name "_Z3gooi".
+    StringRef GooName("_Z3gooi");
+    auto &GooSamples =
+        FooSamples.functionSamplesAt(LineLocation(7, 0))[GooName.str()];
+    GooSamples.setName(GooName);
+    GooSamples.addTotalSamples(502);
+    GooSamples.addBodySamples(3, 0, 502);
+
+    // Add inline instance with name "_Z3hooi".
+    StringRef HooName("_Z3hooi");
+    auto &HooSamples =
+        GooSamples.functionSamplesAt(LineLocation(9, 0))[HooName.str()];
+    HooSamples.setName(HooName);
+    HooSamples.addTotalSamples(317);
+    HooSamples.addBodySamples(4, 0, 317);
+
     StringRef BarName("_Z3bari");
     FunctionSamples BarSamples;
     BarSamples.setName(BarName);
     BarSamples.addTotalSamples(20301);
     BarSamples.addHeadSamples(1437);
     BarSamples.addBodySamples(1, 0, 1437);
     // Test how reader/writer handles unmangled names.
     StringRef MconstructName("_M_construct<char *>");
FunctionType *fn_type =
         FunctionType::get(Type::getVoidTy(Context), {}, false);
 
     SmallVector<char, 128> RemapPath;
     StringRef RemapFile;
     if (Remap) {
       createRemapFile(RemapPath, RemapFile);
       FooName = "_Z4fauxi";
       BarName = "_Z3barl";
+      GooName = "_Z3gool";
+      HooName = "_Z3hool";
     }
 
     M.getOrInsertFunction(FooName, fn_type);
     M.getOrInsertFunction(BarName, fn_type);
     M.getOrInsertFunction(BooName, fn_type);
 
     ProfileSymbolList List;
     if (Format == SampleProfileFormat::SPF_Ext_Binary) {
void testRoundTrip(SampleProfileFormat Format, bool Remap, bool UseMD5) {
     FunctionSamples *ReadFooSamples = Reader->getSamplesFor(FooName);
     ASSERT_TRUE(ReadFooSamples != nullptr);
     if (!UseMD5) {
       ASSERT_EQ("_Z3fooi", ReadFooSamples->getName());
     }
     ASSERT_EQ(7711u, ReadFooSamples->getTotalSamples());
     ASSERT_EQ(610u, ReadFooSamples->getHeadSamples());
 
+    // Try to find a FunctionSamples with GooName at given callsites containing
+    // inline instance for GooName. Test the correct FunctionSamples can be
+    // found with Remapper support.
+    const FunctionSamples *ReadGooSamples =
+        ReadFooSamples->findFunctionSamplesAt(LineLocation(7, 0), GooName,
+                                              Reader->getTempRemapper());
+    ASSERT_TRUE(ReadGooSamples != nullptr);
+    ASSERT_EQ(502u, ReadGooSamples->getTotalSamples());
+
+    // Try to find a FunctionSamples with GooName at given callsites containing
+    // no inline instance for GooName. Test no FunctionSamples will be
+    // found with Remapper support.
+    const FunctionSamples *ReadGooSamplesAgain =
+        ReadFooSamples->findFunctionSamplesAt(LineLocation(9, 0), GooName,
+                                              Reader->getTempRemapper());
+    ASSERT_TRUE(ReadGooSamplesAgain == nullptr);
+
+    // The inline instance of Hoo is inside of the inline instance of Goo.
+    // Try to find a FunctionSamples with HooName at given callsites containing
+    // inline instance for HooName. Test the correct FunctionSamples can be
+    // found with Remapper support.
+    const FunctionSamples *ReadHooSamples =
+        ReadGooSamples->findFunctionSamplesAt(LineLocation(9, 0), HooName,
+                                              Reader->getTempRemapper());
+    ASSERT_TRUE(ReadHooSamples != nullptr);
+    ASSERT_EQ(317u, ReadHooSamples->getTotalSamples());
+
     FunctionSamples *ReadBarSamples = Reader->getSamplesFor(BarName);
     ASSERT_TRUE(ReadBarSamples != nullptr);
     if (!UseMD5) {
       ASSERT_EQ("_Z3bari", ReadBarSamples->getName());
     }
     ASSERT_EQ(20301u, ReadBarSamples->getTotalSamples());
     ASSERT_EQ(1437u, ReadBarSamples->getHeadSamples());
     ErrorOr<SampleRecord::CallTargetMap> CTMap =