[llvm-profdata] Improve profile supplementation

[xur]

Summary

Current implementation promotes a non-cold function in the SampleFDO profile
into a hot function in the FDO profile. This is too aggressive. This patch
promotes a hot functions in the SampleFDO profile into a hot function, and a
warm function in SampleFDO into a warm function in FDO.

[davidxl]

llvm/lib/ProfileData/ProfileSummaryBuilder.cpp

227–228

probably worth defining symbolic constants for these values.

Also how can we prevent these values are accidentally used somewhere else?

llvm/tools/llvm-profdata/llvm-profdata.cpp

506

comment mismatches.

508–515

why using coldthreshold?

[xur]

llvm/lib/ProfileData/ProfileSummaryBuilder.cpp

227–228

Symbolic constants seem reasonable.

These values should not be encountered in real profiles. 2^64 is too large for any counter values to come close.
But we do need to special handling of these values.
It's possible to misuse -- I think overlap function does not handle this well.
There are not too many places to use the counters directly. We just need to check all of them.

There might be other ways to implement -- like extending the profile format. But this is the simplest way.

llvm/tools/llvm-profdata/llvm-profdata.cpp

508–515

We want to set the entry count so that the function will be warm

The entry count needs to be greater than ColdThreshold but less than HotThreshold.

I think I need to add code to make sure it's less than HotThreshold.

[davidxl]

llvm/lib/ProfileData/ProfileSummaryBuilder.cpp

227–228

profile counter reader should do something when reading those values into instrProfRecord so that the special values are not leaked.

llvm/tools/llvm-profdata/llvm-profdata.cpp

519

are the counts all -1 or -2 in this case? what is the point of scaling?

[xur]

llvm/lib/ProfileData/ProfileSummaryBuilder.cpp

227–228

Agreed. We can have some APIs to InstrProfRecord to encapsulate the values.

The other alternative is to extend the profile format to have a information section for each function. Right now we are overload some information in the function hash function. I think have an optional information section for each function would be useful.

llvm/tools/llvm-profdata/llvm-profdata.cpp

506

will fix

519

If we assign -1 or -2, we will not do scaling, it will early return on line 504.
(that happens all zero counters or zero counters percentage exceed the threshold.)

This part is doing the scaling on the original counters.

[xur]

This patches adds the support for mapping to warm function, just following the existing practice (where it maps to hot functions).
The constant of -1 is already in llvm.

Since we are talking about hiding these values within InstrProfRecord, I would also change the way how the information pass to to down streams: instead of using entry counts,
For a function with InstrProfRecord value of -1, we would set the function attribute hot,
For a function with InstrProfRecord value of -2, we would remove function attribute cold if it has, otherwise do nothing.

For warm functions, we could also delete the function profile in the supplementation step. But it's less optimal:
(1) we will have a no-profile-found warning (it is not a big problem, as the default is off)
(2) we will not be able to fix the incorrect user annotation (where the user sets the function to cold).

[xur]

This is more complex change that hide the pseudo count values in InstroProfRecord.
Needs to touch more files. But I think this improves the original patch.

[davidxl]

llvm/include/llvm/ProfileData/InstrProf.h

844

pass CountPseudoKind

llvm/lib/ProfileData/InstrProf.cpp

721

typo: proifiles

722

typo peseudo

725

We should probably assume/require both profiles are (or not) supplemented, and always take the non pseudo one.

Or do not support merge with supplemented profile (i.e. only supplement after merging).

[xur]

llvm/include/llvm/ProfileData/InstrProf.h

844

Ack.

llvm/lib/ProfileData/InstrProf.cpp

721

Ack.

722

Ack.

725

Supplementing can produce a normal profile (only scaling), or a profile with pseudo counts.
So it's hard to force not merging supplemented profiles.

Probably the fist suggestion is the way to go: give a hard error when merging profile with pseudo counts and without pseudo counts.

[davidxl]

emit an error for merging profiles of different kind is ok.

[xur]

Integrated David's most recent review comments.

[davidxl]

llvm/lib/ProfileData/InstrProf.cpp

736

should this loop be skipped for pseudo kind?

llvm/lib/ProfileData/ProfileSummaryBuilder.cpp

234

add assert on count value (not pseudo values).

[xur]

llvm/lib/ProfileData/InstrProf.cpp

736

Yes. We can skip this loop for pesudo kind as the value will not be used.

llvm/lib/ProfileData/ProfileSummaryBuilder.cpp

234

Ack.

[xur]

Integrated David's review comment.
For the merge profile merge, also skip the value profile merge if the count is pseudo kind.

Also added some tests to test pseudo profile merge.

[davidxl]

lgtm

llvm/include/llvm/ProfileData/InstrProf.h

 class MDNode;
 class Module;
 
 enum InstrProfSectKind {
 #define INSTR_PROF_SECT_ENTRY(Kind, SectNameCommon, SectNameCoff, Prefix) Kind,
 #include "llvm/ProfileData/InstrProfData.inc"
 };
 
+/// Return the max count value. We reserver a few large values for special use.
+inline uint64_t getInstrMaxCountValue() {
+  return std::numeric_limits<uint64_t>::max() - 2;
+}
+
 /// Return the name of the profile section corresponding to \p IPSK.
 ///
 /// The name of the section depends on the object format type \p OF. If
 /// \p AddSegmentInfo is true, a segment prefix and additional linker hints may
 /// be added to the section name (this is the default).
 std::string getInstrProfSectionName(InstrProfSectKind IPSK,
                                     Triple::ObjectFormatType OF,
                                     bool AddSegmentInfo = true);
struct InstrProfRecord {
   void overlap(InstrProfRecord &Other, OverlapStats &Overlap,
                OverlapStats &FuncLevelOverlap, uint64_t ValueCutoff);
 
   /// Compute the overlap of value profile counts.
   void overlapValueProfData(uint32_t ValueKind, InstrProfRecord &Src,
                             OverlapStats &Overlap,
                             OverlapStats &FuncLevelOverlap);
 
+  enum CountPseudoKind {
+    NotPseudo = 0,
+    PseudoHot,
+    PseudoWarm,
+  };
+  enum PseudoCountVal {
+    HotFunctionVal = -1,
+    WarmFunctionVal = -2,
+  };
+  CountPseudoKind getCountPseudoKind() const {
+    uint64_t FirstCount = Counts[0];
+    if (FirstCount == (uint64_t)HotFunctionVal)
+      return PseudoHot;
+    if (FirstCount == (uint64_t)WarmFunctionVal)
+      return PseudoWarm;
+    return NotPseudo;
+  }
+  void setPseudoCount(bool IsHot) {
+    uint64_t Val = (uint64_t)(IsHot ? HotFunctionVal : WarmFunctionVal);
+    Counts[0] = Val;
+  }
+
 private:
   struct ValueProfData {
     std::vector<InstrProfValueSiteRecord> IndirectCallSites;
     std::vector<InstrProfValueSiteRecord> MemOPSizes;
   };
   std::unique_ptr<ValueProfData> ValueData;
 
   MutableArrayRef<InstrProfValueSiteRecord>

llvm/lib/ProfileData/InstrProf.cpp

void InstrProfRecord::merge(InstrProfRecord &Other, uint64_t Weight,
                             function_ref<void(instrprof_error)> Warn) {
   // If the number of counters doesn't match we either have bad data
   // or a hash collision.
   if (Counts.size() != Other.Counts.size()) {
     Warn(instrprof_error::count_mismatch);
     return;
   }
 
-  for (size_t I = 0, E = Other.Counts.size(); I < E; ++I) {
+  // Special handling of first count for PesudoCount
+  int Start = 0;
+  CountPseudoKind OtherKind = Other.getCountPseudoKind();
+  CountPseudoKind ThisKind = getCountPseudoKind();
+  // This is less optimial as one of the proifles might not be
+  // a peseudo one and might have better information. But we don't
+  // have summary information here. We cannot tell if the non-pseudo profile
+  // is better. We just use pesudo profile here.
+  if (OtherKind != NotPseudo || ThisKind != NotPseudo) {
+    bool IsHot = (OtherKind == PseudoHot || ThisKind == PseudoHot);
+    setPseudoCount(IsHot);
+    Start = 1;
+  }
+
+  for (size_t I = Start, E = Other.Counts.size(); I < E; ++I) {
     bool Overflowed;
-    Counts[I] =
+    uint64_t Value =
         SaturatingMultiplyAdd(Other.Counts[I], Weight, Counts[I], &Overflowed);
+    if (Value > getInstrMaxCountValue()) {
+      Value = getInstrMaxCountValue();
+      Overflowed = true;
+    }
+    Counts[I] = Value;
     if (Overflowed)
       Warn(instrprof_error::counter_overflow);
   }
 
   for (uint32_t Kind = IPVK_First; Kind <= IPVK_Last; ++Kind)
     mergeValueProfData(Kind, Other, Weight, Warn);
 }
 
 void InstrProfRecord::scaleValueProfData(
     uint32_t ValueKind, uint64_t N, uint64_t D,
     function_ref<void(instrprof_error)> Warn) {
   for (auto &R : getValueSitesForKind(ValueKind))
     R.scale(N, D, Warn);
 }
 
 void InstrProfRecord::scale(uint64_t N, uint64_t D,
                             function_ref<void(instrprof_error)> Warn) {
   assert(D != 0 && "D cannot be 0");
   for (auto &Count : this->Counts) {
     bool Overflowed;
     Count = SaturatingMultiply(Count, N, &Overflowed) / D;
+    if (Count > getInstrMaxCountValue()) {
+      Count = getInstrMaxCountValue();
+      Overflowed = true;
+    }
     if (Overflowed)
       Warn(instrprof_error::counter_overflow);
   }
   for (uint32_t Kind = IPVK_First; Kind <= IPVK_Last; ++Kind)
     scaleValueProfData(Kind, N, D, Warn);
 }
 
 // Map indirect call target name hash to name string.

llvm/lib/ProfileData/ProfileSummaryBuilder.cpp

ProfileSummaryBuilder::getEntryForPercentile(const SummaryEntryVector &DS,
   return *It;
 }
 
 void InstrProfSummaryBuilder::addRecord(const InstrProfRecord &R) {
   // The first counter is not necessarily an entry count for IR
   // instrumentation profiles.
   // Eventually MaxFunctionCount will become obsolete and this can be
   // removed.
+
+  if (R.getCountPseudoKind() != InstrProfRecord::NotPseudo)
+    return;
+
   addEntryCount(R.Counts[0]);
   for (size_t I = 1, E = R.Counts.size(); I < E; ++I)
     addInternalCount(R.Counts[I]);
 }
 
 // To compute the detailed summary, we consider each line containing samples as
 // equivalent to a block with a count in the instrumented profile.
 void SampleProfileSummaryBuilder::addRecord(

 std::unique_ptr<ProfileSummary> InstrProfSummaryBuilder::getSummary() {
   computeDetailedSummary();
   return std::make_unique<ProfileSummary>(
       ProfileSummary::PSK_Instr, DetailedSummary, TotalCount, MaxCount,
       MaxInternalBlockCount, MaxFunctionCount, NumCounts, NumFunctions);
 }
 
 void InstrProfSummaryBuilder::addEntryCount(uint64_t Count) {
   NumFunctions++;
-
-  // Skip invalid count.
-  if (Count == (uint64_t)-1)
-    return;
-
   addCount(Count);
   if (Count > MaxFunctionCount)
     MaxFunctionCount = Count;
 }
 
 void InstrProfSummaryBuilder::addInternalCount(uint64_t Count) {
-  // Skip invalid count.
-  if (Count == (uint64_t)-1)
-    return;
-
   addCount(Count);
   if (Count > MaxInternalBlockCount)
     MaxInternalBlockCount = Count;
 }

llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp

PGOUseFunc(Function &Func, Module *Modu, TargetLibraryInfo &TLI,
              ProfileSummaryInfo *PSI, bool IsCS, bool InstrumentFuncEntry)
       : F(Func), M(Modu), BFI(BFIin), PSI(PSI),
         FuncInfo(Func, TLI, ComdatMembers, false, BPI, BFIin, IsCS,
                  InstrumentFuncEntry),
         FreqAttr(FFA_Normal), IsCS(IsCS) {}
 
   // Read counts for the instrumented BB from profile.
   bool readCounters(IndexedInstrProfReader *PGOReader, bool &AllZeros,
-                    bool &AllMinusOnes);
+                    InstrProfRecord::CountPseudoKind &PseudoKind);
 
   // Populate the counts for all BBs.
   void populateCounters();
 
   // Set the branch weights based on the count values.
   void setBranchWeights();
 
   // Annotate the value profile call sites for all value kind.
static void annotateFunctionWithHashMismatch(Function &F,
   MDNode *MD = MDTuple::get(ctx, Names);
   F.setMetadata(LLVMContext::MD_annotation, MD);
 }
 
 // Read the profile from ProfileFileName and assign the value to the
 // instrumented BB and the edges. This function also updates ProgramMaxCount.
 // Return true if the profile are successfully read, and false on errors.
 bool PGOUseFunc::readCounters(IndexedInstrProfReader *PGOReader, bool &AllZeros,
-                              bool &AllMinusOnes) {
+                              InstrProfRecord::CountPseudoKind &PseudoKind) {
   auto &Ctx = M->getContext();
   uint64_t MismatchedFuncSum = 0;
   Expected<InstrProfRecord> Result = PGOReader->getInstrProfRecord(
       FuncInfo.FuncName, FuncInfo.FunctionHash, &MismatchedFuncSum);
   if (Error E = Result.takeError()) {
     handleAllErrors(std::move(E), [&](const InstrProfError &IPE) {
       auto Err = IPE.get();
       bool SkipWarning = false;
handleAllErrors(std::move(E), [&](const InstrProfError &IPE) {
           std::string(" count discarded");
 
       Ctx.diagnose(
           DiagnosticInfoPGOProfile(M->getName().data(), Msg, DS_Warning));
     });
     return false;
   }
   ProfileRecord = std::move(Result.get());
+  PseudoKind = ProfileRecord.getCountPseudoKind();
+  if (PseudoKind != InstrProfRecord::NotPseudo) {
+    return true;
+  }
   std::vector<uint64_t> &CountFromProfile = ProfileRecord.Counts;
 
   IsCS ? NumOfCSPGOFunc++ : NumOfPGOFunc++;
   LLVM_DEBUG(dbgs() << CountFromProfile.size() << " counts\n");
-  AllMinusOnes = (CountFromProfile.size() > 0);
+
   uint64_t ValueSum = 0;
   for (unsigned I = 0, S = CountFromProfile.size(); I < S; I++) {
     LLVM_DEBUG(dbgs() << "  " << I << ": " << CountFromProfile[I] << "\n");
     ValueSum += CountFromProfile[I];
-    if (CountFromProfile[I] != (uint64_t)-1)
-      AllMinusOnes = false;
   }
   AllZeros = (ValueSum == 0);
 
   LLVM_DEBUG(dbgs() << "SUM =  " << ValueSum << "\n");
 
   getBBInfo(nullptr).UnknownCountOutEdge = 2;
   getBBInfo(nullptr).UnknownCountInEdge = 2;
 
for (auto &F : M) {
     auto &TLI = LookupTLI(F);
     auto *BPI = LookupBPI(F);
     auto *BFI = LookupBFI(F);
     // Split indirectbr critical edges here before computing the MST rather than
     // later in getInstrBB() to avoid invalidating it.
     SplitIndirectBrCriticalEdges(F, /*IgnoreBlocksWithoutPHI=*/false, BPI, BFI);
     PGOUseFunc Func(F, &M, TLI, ComdatMembers, BPI, BFI, PSI, IsCS,
                     InstrumentFuncEntry);
-    // When AllMinusOnes is true, it means the profile for the function
-    // is unrepresentative and this function is actually hot. Set the
+    // When AllMinus is not zero, it means the profile for the function
+    // is unrepresentative and this function is actually hot / warm. Set the
     // entry count of the function to be multiple times of hot threshold
     // and drop all its internal counters.
-    bool AllMinusOnes = false;
+    InstrProfRecord::CountPseudoKind PseudoKind = InstrProfRecord::NotPseudo;
     bool AllZeros = false;
-    if (!Func.readCounters(PGOReader.get(), AllZeros, AllMinusOnes))
+    if (!Func.readCounters(PGOReader.get(), AllZeros, PseudoKind))
       continue;
     if (AllZeros) {
       F.setEntryCount(ProfileCount(0, Function::PCT_Real));
       if (Func.getProgramMaxCount() != 0)
         ColdFunctions.push_back(&F);
       continue;
     }
-    const unsigned MultiplyFactor = 3;
-    if (AllMinusOnes) {
-      uint64_t HotThreshold = PSI->getHotCountThreshold();
-      if (HotThreshold)
-        F.setEntryCount(
-            ProfileCount(HotThreshold * MultiplyFactor, Function::PCT_Real));
-      HotFunctions.push_back(&F);
+    if (PseudoKind != InstrProfRecord::NotPseudo) {
+      // Clear Function attribute cold.
+      if (F.hasFnAttribute(Attribute::Cold))
+        F.removeFnAttr(Attribute::Cold);
+      // Set Function attribute as hot.
+      if (PseudoKind == InstrProfRecord::PseudoHot)
+        F.addFnAttr(Attribute::Hot);
       continue;
     }
     Func.populateCounters();
     Func.setBranchWeights();
     Func.annotateValueSites();
     Func.annotateIrrLoopHeaderWeights();
     PGOUseFunc::FuncFreqAttr FreqAttr = Func.getFuncFreqAttr();
     if (FreqAttr == PGOUseFunc::FFA_Cold)

llvm/test/Transforms/PGOProfile/Inputs/sample-profile-warm.proftext

+test_simple_for:40:40
+ 1: 10
+ 2: 10
+ 3: 10
+ 4: 10
+
+moo:1000:1000
+ 1: 2000
+ 2: 2000
+ 3: 2000
+ 4: 2000
+ 5: 2000
+

llvm/test/Transforms/PGOProfile/Inputs/suppl-profile.proftext

 # :ir is the flag to indicate this is IR level profile.
 :ir
 test_simple_for
 1063705162469825436
 2
 0
 0
 
 foo
 2582734
 4
 1000
 270
 180
 760
+
+boo
+2582734
+4
+100
+27
+10
+70
+

llvm/test/Transforms/PGOProfile/suppl-profile.ll

 ; Supplement instr profile suppl-profile.proftext with sample profile
 ; sample-profile.proftext.
+; For hot functions:
 ; RUN: llvm-profdata merge -instr -suppl-min-size-threshold=0 \
-; RUN:   -supplement-instr-with-sample=%p/Inputs/sample-profile.proftext \
+; RUN:   -supplement-instr-with-sample=%p/Inputs/sample-profile-hot.proftext \
 ; RUN:   %S/Inputs/suppl-profile.proftext -o %t.profdata
-; RUN: opt < %s -passes=pgo-instr-use -pgo-test-profile-file=%t.profdata -S | FileCheck %s
+; RUN: opt < %s -passes=pgo-instr-use -pgo-test-profile-file=%t.profdata -S | FileCheck %s --check-prefix=HOT
+; For warm functions:
+; RUN: llvm-profdata merge -instr -suppl-min-size-threshold=0 \
+; RUN:   -supplement-instr-with-sample=%p/Inputs/sample-profile-warm.proftext \
+; RUN:   %S/Inputs/suppl-profile.proftext -o %t1.profdata
+; RUN: opt < %s -passes=pgo-instr-use -pgo-test-profile-file=%t1.profdata -S | FileCheck %s --check-prefix=WARM
 
 target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
-; Check test_simple_for has a non-zero entry count and doesn't have any other
-; prof metadata.
-; CHECK: @test_simple_for(i32 %n) {{.*}} !prof ![[ENTRY_COUNT:[0-9]+]]
-; CHECK-NOT: !prof !
-; CHECK: ![[ENTRY_COUNT]] = !{!"function_entry_count", i64 540}
-define i32 @test_simple_for(i32 %n) {
+; Check test_simple_for has proper hot/cold attribute and no profile counts.
+; HOT: @test_simple_for(i32 %n)
+; HOT-SAME: #[[ATTRIBTE:[0-9]*]]
+; HOT-NOT: !prof !{{.*}}
+; HOT-SAME: {
+; HOT: attributes #[[ATTRIBTE]] = { hot }
+; WARM: @test_simple_for(i32 %n)
+; WARM-NOT: #{{.*}}
+; WARM-NOT: !prof !{{.*}}
+; WARM-SAME: {
+define i32 @test_simple_for(i32 %n) #0 {
 entry:
   br label %for.cond
 
 for.cond:
   %i = phi i32 [ 0, %entry ], [ %inc1, %for.inc ]
   %sum = phi i32 [ 1, %entry ], [ %inc, %for.inc ]
   %cmp = icmp slt i32 %i, %n
   br i1 %cmp, label %for.body, label %for.end
 
 for.body:
   %inc = add nsw i32 %sum, 1
   br label %for.inc
 
 for.inc:
   %inc1 = add nsw i32 %i, 1
   br label %for.cond
 
 for.end:
   ret i32 %sum
 }
+
+attributes #0 = { cold }

llvm/test/tools/llvm-profdata/overflow-instr.test

 Tests for overflow when merging instrumented profiles.
 
 1- Merge profile having maximum counts with itself and verify overflow detected and saturation occurred
 RUN: llvm-profdata merge -instr %p/Inputs/overflow-instr.proftext %p/Inputs/overflow-instr.proftext -o %t.out 2>&1 | FileCheck %s -check-prefix=MERGE_OVERFLOW
 RUN: llvm-profdata show -instr -all-functions -counts %t.out | FileCheck %s --check-prefix=SHOW_OVERFLOW
 MERGE_OVERFLOW: {{.*}}.proftext: overflow: counter overflow
-SHOW_OVERFLOW: Function count: 18446744073709551615
-SHOW_OVERFLOW-NEXT: Block counts: [18446744073709551615, 18446744073709551615]
+SHOW_OVERFLOW: Counters: 3    <PseudoHot>
+SHOW_OVERFLOW-NEXT: Instrumentation level: Front-end
 
 2- Merge profile having maximum counts by itself and verify no overflow
 RUN: llvm-profdata merge -instr %p/Inputs/overflow-instr.proftext -o %t.out 2>&1 | FileCheck %s -check-prefix=MERGE_NO_OVERFLOW -allow-empty
 RUN: llvm-profdata show -instr -all-functions -counts %t.out | FileCheck %s --check-prefix=SHOW_NO_OVERFLOW
 MERGE_NO_OVERFLOW-NOT: {{.*}}.proftext: overflow: counter overflow
-SHOW_NO_OVERFLOW: Function count: 18446744073709551615
-SHOW_NO_OVERFLOW-NEXT: Block counts: [9223372036854775808, 18446744073709551615]
+SHOW_NO_OVERFLOW: Counters: 3    <PseudoHot>
+SHOW_NO_OVERFLOW-NEXT: Instrumentation level: Front-end

llvm/test/tools/llvm-profdata/suppl-instr-with-sample.test

 Some basic tests for supplementing instrumentation profile with sample profile.
 
-Test all of goo's counters will be set to -1.
+Test all of goo's counters will be set to PesudoHot.
 RUN: llvm-profdata merge \
 RUN:     -supplement-instr-with-sample=%p/Inputs/mix_sample.proftext \
 RUN:     -suppl-min-size-threshold=0 %p/Inputs/mix_instr.proftext -o %t
 RUN: llvm-profdata show %t -all-functions -counts | FileCheck %s --check-prefix=MIX1
 
 MIX1: foo:
 MIX1-NEXT: Hash: 0x0000000000000007
 MIX1-NEXT: Counters: 5
 MIX1-NEXT: Block counts: [12, 13, 0, 0, 0]
 MIX1: goo:
 MIX1-NEXT: Hash: 0x0000000000000005
 MIX1-NEXT: Counters: 3
-MIX1-NEXT: Block counts: [18446744073709551615, 18446744073709551615, 18446744073709551615]
+MIX1-NOT: Block counts:
+MIX1-SAME: <PseudoHot>
 MIX1: moo:
 MIX1-NEXT: Hash: 0x0000000000000009
 MIX1-NEXT: Counters: 4
 MIX1-NEXT: Block counts: [3000, 1000, 2000, 500]
 
 Test when the zero counter ratio of foo is higher than zero-counter-threshold.
 RUN: llvm-profdata merge \
 RUN:     -supplement-instr-with-sample=%p/Inputs/mix_sample.proftext \
 RUN:     -suppl-min-size-threshold=0 -zero-counter-threshold=0.5 \
 RUN:     -instr-prof-cold-threshold=30 %p/Inputs/mix_instr.proftext -o %t
 RUN: llvm-profdata show %t -all-functions -counts | FileCheck %s --check-prefix=MIX2
 
 MIX2: foo:
 MIX2-NEXT: Hash: 0x0000000000000007
 MIX2-NEXT: Counters: 5
-MIX2-NEXT: Block counts: [18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615]
+MIX2-NOT: Block counts:
+MIX2-SAME: <PseudoHot>
 MIX2: goo:
 MIX2-NEXT: Hash: 0x0000000000000005
 MIX2-NEXT: Counters: 3
-MIX2-NEXT: Block counts: [18446744073709551615, 18446744073709551615, 18446744073709551615]
+MIX2-NOT: Block counts:
+MIX2-SAME: <PseudoHot>
 MIX2: moo:
 MIX2-NEXT: Hash: 0x0000000000000009
 MIX2-NEXT: Counters: 4
 MIX2-NEXT: Block counts: [3000, 1000, 2000, 500]
 
 Test when the zero counter ratio of foo is lower than zero-counter-threshold.
 RUN: llvm-profdata merge \
 RUN:     -supplement-instr-with-sample=%p/Inputs/mix_sample.proftext \
 RUN:     -suppl-min-size-threshold=0 -zero-counter-threshold=0.7 \
 RUN:     -instr-prof-cold-threshold=30 %p/Inputs/mix_instr.proftext -o %t
 RUN: llvm-profdata show %t -all-functions -counts | FileCheck %s --check-prefix=MIX3
 
 MIX3: foo:
 MIX3-NEXT: Hash: 0x0000000000000007
 MIX3-NEXT: Counters: 5
 MIX3-NEXT: Block counts: [1384, 1500, 0, 0, 0]
 MIX3: goo:
 MIX3-NEXT: Hash: 0x0000000000000005
 MIX3-NEXT: Counters: 3
-MIX3-NEXT: Block counts: [18446744073709551615, 18446744073709551615, 18446744073709551615]
+MIX3-NOT: Block counts:
+MIX3-SAME: <PseudoHot>
 MIX3: moo:
 MIX3-NEXT: Hash: 0x0000000000000009
 MIX3-NEXT: Counters: 4
 MIX3-NEXT: Block counts: [3000, 1000, 2000, 500]
 
 Test foo's profile won't be adjusted because its size is smaller
 than suppl-min-size-threshold.
 RUN: llvm-profdata merge \
 RUN:     -supplement-instr-with-sample=%p/Inputs/mix_sample.proftext \
 RUN:     -suppl-min-size-threshold=2 -zero-counter-threshold=0.7 \
 RUN:     -instr-prof-cold-threshold=30 %p/Inputs/mix_instr.proftext -o %t
 RUN: llvm-profdata show %t -all-functions -counts | FileCheck %s --check-prefix=MIX4
 
 MIX4: foo:
 MIX4-NEXT: Hash: 0x0000000000000007
 MIX4-NEXT: Counters: 5
 MIX4-NEXT: Block counts: [12, 13, 0, 0, 0]
 MIX4: goo:
 MIX4-NEXT: Hash: 0x0000000000000005
 MIX4-NEXT: Counters: 3
-MIX4-NEXT: Block counts: [18446744073709551615, 18446744073709551615, 18446744073709551615]
+MIX4-NOT: Block counts:
+MIX4-SAME: <PseudoHot>
 MIX4: moo:
 MIX4-NEXT: Hash: 0x0000000000000009
 MIX4-NEXT: Counters: 4
 MIX4-NEXT: Block counts: [3000, 1000, 2000, 500]
 
-Test profile summary won't be affected by -1 counter.
+Test profile summary won't be affected by pseudo counters.
 RUN: llvm-profdata merge \
 RUN:     -supplement-instr-with-sample=%p/Inputs/mix_sample.proftext \
 RUN:     -suppl-min-size-threshold=0 %p/Inputs/mix_instr.proftext -o %t
 RUN: llvm-profdata show %t -detailed-summary | FileCheck %s --check-prefix=MIX5
 
 MIX5: Instrumentation level: IR
-MIX5-NEXT: Total functions: 3
+MIX5-NEXT: Total functions: 2
 MIX5-NEXT: Maximum function count: 3000
 MIX5-NEXT: Maximum internal block count: 2000
 MIX5-NEXT: Total number of blocks: 9
 MIX5-NEXT: Total count: 6525
 MIX5-NEXT: Detailed summary:
 MIX5-NEXT: 1 blocks with count >= 3000 account for 1 percentage of the total counts.
 MIX5-NEXT: 1 blocks with count >= 3000 account for 10 percentage of the total counts.
 MIX5-NEXT: 1 blocks with count >= 3000 account for 20 percentage of the total counts.

llvm/tools/llvm-profdata/llvm-profdata.cpp

InstrProfileEntry::InstrProfileEntry(InstrProfRecord *Record) {
   uint64_t ZeroCntNum = 0;
   for (size_t I = 0; I < CntNum; ++I) {
     MaxCount = std::max(MaxCount, Record->Counts[I]);
     ZeroCntNum += !Record->Counts[I];
   }
   ZeroCounterRatio = (float)ZeroCntNum / CntNum;
 }
 
-/// Either set all the counters in the instr profile entry \p IFE to -1
-/// in order to drop the profile or scale up the counters in \p IFP to
-/// be above hot threshold. We use the ratio of zero counters in the
-/// profile of a function to decide the profile is helpful or harmful
-/// for performance, and to choose whether to scale up or drop it.
-static void updateInstrProfileEntry(InstrProfileEntry &IFE,
+/// Either set all the counters in the instr profile entry \p IFE to
+/// -1 / -2 /in order to drop the profile or scale up the
+/// counters in \p IFP to be above hot / cold threshold. We use
+/// the ratio of zero counters in the profile of a function to
+/// decide the profile is helpful or harmful for performance,
+/// and to choose whether to scale up or drop it.
+static void updateInstrProfileEntry(InstrProfileEntry &IFE, bool SetToHot,
                                     uint64_t HotInstrThreshold,
+                                    uint64_t ColdInstrThreshold,
                                     float ZeroCounterThreshold) {
   InstrProfRecord *ProfRecord = IFE.ProfRecord;
   if (!IFE.MaxCount || IFE.ZeroCounterRatio > ZeroCounterThreshold) {
     // If all or most of the counters of the function are zero, the
     // profile is unaccountable and shuld be dropped. Reset all the
-    // counters to be -1 and PGO profile-use will drop the profile.
+    // counters to be -1 / -2 and PGO profile-use will drop the profile.
     // All counters being -1 also implies that the function is hot so
     // PGO profile-use will also set the entry count metadata to be
     // above hot threshold.
-    for (size_t I = 0; I < ProfRecord->Counts.size(); ++I)
-      ProfRecord->Counts[I] = -1;
+    // All counters being -2 implies that the function is warm so
+    // PGO profile-use will also set the entry count metadata to be
+    // above cold threshold.
+    ProfRecord->setPseudoCount(SetToHot);
     return;
   }
 
-  // Scale up the MaxCount to be multiple times above hot threshold.
+  // Scale up the MaxCount to be multiple times above hot / cold threshold.
   const unsigned MultiplyFactor = 3;
-  uint64_t Numerator = HotInstrThreshold * MultiplyFactor;
+  uint64_t Threshold = (SetToHot ? HotInstrThreshold : ColdInstrThreshold);
+  uint64_t Numerator = Threshold * MultiplyFactor;
+
+  // Make sure Threshold for warm not reacch HotInstrThreshold.
+  if (!SetToHot && Threshold >= HotInstrThreshold) {
+    Threshold = (HotInstrThreshold + ColdInstrThreshold) / 2;
+  }
+
   uint64_t Denominator = IFE.MaxCount;
+  if (Numerator <= Denominator)
+    return;
   ProfRecord->scale(Numerator, Denominator, [&](instrprof_error E) {
     warn(toString(make_error<InstrProfError>(E)));
   });
 }
 
 const uint64_t ColdPercentileIdx = 15;
 const uint64_t HotPercentileIdx = 11;
 
 using sampleprof::FSDiscriminatorPass;
for (auto &PD : WC->Writer.getProfileData()) {
     InstrProfRecord *R = &PD.getValue().begin()->second;
     InstrProfileMap[PD.getKey()] = InstrProfileEntry(R);
   }
 
   ProfileSummary InstrPS = *IPBuilder.getSummary();
   ProfileSummary SamplePS = Reader->getSummary();
 
   // Compute cold thresholds for instr profile and sample profile.
+  uint64_t HotSampleThreshold =
+      ProfileSummaryBuilder::getEntryForPercentile(
+          SamplePS.getDetailedSummary(),
+          ProfileSummaryBuilder::DefaultCutoffs[HotPercentileIdx])
+          .MinCount;
   uint64_t ColdSampleThreshold =
       ProfileSummaryBuilder::getEntryForPercentile(
           SamplePS.getDetailedSummary(),
           ProfileSummaryBuilder::DefaultCutoffs[ColdPercentileIdx])
           .MinCount;
   uint64_t HotInstrThreshold =
       ProfileSummaryBuilder::getEntryForPercentile(
           InstrPS.getDetailedSummary(),
           ProfileSummaryBuilder::DefaultCutoffs[HotPercentileIdx])
           .MinCount;
   uint64_t ColdInstrThreshold =
       InstrProfColdThreshold
           ? InstrProfColdThreshold
           : ProfileSummaryBuilder::getEntryForPercentile(
                 InstrPS.getDetailedSummary(),
                 ProfileSummaryBuilder::DefaultCutoffs[ColdPercentileIdx])
                 .MinCount;
 
   // Find hot/warm functions in sample profile which is cold in instr profile
   // and adjust the profiles of those functions in the instr profile.
   for (const auto &PD : Reader->getProfiles()) {
     auto &FContext = PD.first;
     const sampleprof::FunctionSamples &FS = PD.second;
     auto It = InstrProfileMap.find(FContext.toString());
-    if (FS.getHeadSamples() > ColdSampleThreshold &&
-        It != InstrProfileMap.end() &&
+
+    uint64_t SampleMaxCount = FS.getMaxCountInside();
+    if (SampleMaxCount < ColdSampleThreshold)
+      continue;
+
+    if (It != InstrProfileMap.end() &&
         It->second.MaxCount <= ColdInstrThreshold &&
         FS.getBodySamples().size() >= SupplMinSizeThreshold) {
-      updateInstrProfileEntry(It->second, HotInstrThreshold,
-                              ZeroCounterThreshold);
+      bool SetToHot = SampleMaxCount >= HotSampleThreshold;
+      updateInstrProfileEntry(It->second, SetToHot, HotInstrThreshold,
+                              ColdInstrThreshold, ZeroCounterThreshold);
     }
   }
 }
 
 /// The main function to supplement instr profile with sample profile.
 /// \Inputs contains the instr profile. \p SampleFilename specifies the
 /// sample profile. \p OutputFilename specifies the output profile name.
 /// \p OutputFormat specifies the output profile format. \p OutputSparse
for (const auto &Func : *Reader) {
     if (ShowCovered) {
       if (llvm::any_of(Func.Counts, [](uint64_t C) { return C; }))
         OS << Func.Name << "\n";
       continue;
     }
 
     uint64_t FuncMax = 0;
     uint64_t FuncSum = 0;
-    for (size_t I = 0, E = Func.Counts.size(); I < E; ++I) {
-      if (Func.Counts[I] == (uint64_t)-1)
-        continue;
+
+    auto PseudoKind = Func.getCountPseudoKind();
+    if (PseudoKind != InstrProfRecord::NotPseudo) {
+      if (Show) {
+        if (!ShownFunctions)
+          OS << "Counters:\n";
+        ++ShownFunctions;
+        OS << "  " << Func.Name << ":\n"
+           << "    Hash: " << format("0x%016" PRIx64, Func.Hash) << "\n"
+           << "    Counters: " << Func.Counts.size();
+        if (PseudoKind == InstrProfRecord::PseudoHot)
+          OS << "    <PseudoHot>\n";
+        else if (PseudoKind == InstrProfRecord::PseudoWarm)
+          OS << "    <PseudoWarm>\n";
+        else
+          llvm_unreachable("Unknown PseudoKind");
+      }
+    }
+
+    for (size_t I = 0, E = Func.Counts.size(); I < E; ++I) {
       FuncMax = std::max(FuncMax, Func.Counts[I]);
       FuncSum += Func.Counts[I];
     }
 
     if (FuncMax < ValueCutoff) {
       ++BelowCutoffFunctions;
       if (OnlyListBelow) {
         OS << "  " << Func.Name << ": (Max = " << FuncMax

llvm/unittests/ProfileData/InstrProfTest.cpp

TEST_P(MaybeSparseInstrProfTest, get_icall_data_merge1) {
   ASSERT_EQ(4U, VD_4[1].Count);
   ASSERT_EQ(StringRef((const char *)VD_4[2].Value, 7), StringRef("callee1"));
   ASSERT_EQ(2U, VD_4[2].Count);
 }
 
 TEST_P(MaybeSparseInstrProfTest, get_icall_data_merge1_saturation) {
   static const char bar[] = "bar";
 
-  const uint64_t Max = std::numeric_limits<uint64_t>::max();
+  const uint64_t MaxValCount = std::numeric_limits<uint64_t>::max();
+  const uint64_t MaxEdgeCount = getInstrMaxCountValue();
 
   instrprof_error Result;
   auto Err = [&](Error E) { Result = InstrProfError::take(std::move(E)); };
   Result = instrprof_error::success;
   Writer.addRecord({"foo", 0x1234, {1}}, Err);
   ASSERT_EQ(Result, instrprof_error::success);
 
   // Verify counter overflow.
   Result = instrprof_error::success;
-  Writer.addRecord({"foo", 0x1234, {Max}}, Err);
+  Writer.addRecord({"foo", 0x1234, {MaxEdgeCount}}, Err);
   ASSERT_EQ(Result, instrprof_error::counter_overflow);
 
   Result = instrprof_error::success;
   Writer.addRecord({bar, 0x9012, {8}}, Err);
   ASSERT_EQ(Result, instrprof_error::success);
 
   NamedInstrProfRecord Record4("baz", 0x5678, {3, 4});
   Record4.reserveSites(IPVK_IndirectCallTarget, 1);
   InstrProfValueData VD4[] = {{uint64_t(bar), 1}};
   Record4.addValueData(IPVK_IndirectCallTarget, 0, VD4, 1, nullptr);
   Result = instrprof_error::success;
   Writer.addRecord(std::move(Record4), Err);
   ASSERT_EQ(Result, instrprof_error::success);
 
   // Verify value data counter overflow.
   NamedInstrProfRecord Record5("baz", 0x5678, {5, 6});
   Record5.reserveSites(IPVK_IndirectCallTarget, 1);
-  InstrProfValueData VD5[] = {{uint64_t(bar), Max}};
+  InstrProfValueData VD5[] = {{uint64_t(bar), MaxValCount}};
   Record5.addValueData(IPVK_IndirectCallTarget, 0, VD5, 1, nullptr);
   Result = instrprof_error::success;
   Writer.addRecord(std::move(Record5), Err);
   ASSERT_EQ(Result, instrprof_error::counter_overflow);
 
   auto Profile = Writer.writeBuffer();
   readProfile(std::move(Profile));
 
   // Verify saturation of counts.
   Expected<InstrProfRecord> ReadRecord1 =
       Reader->getInstrProfRecord("foo", 0x1234);
   EXPECT_THAT_ERROR(ReadRecord1.takeError(), Succeeded());
-  ASSERT_EQ(Max, ReadRecord1->Counts[0]);
+  ASSERT_EQ(MaxEdgeCount, ReadRecord1->Counts[0]);
 
   Expected<InstrProfRecord> ReadRecord2 =
       Reader->getInstrProfRecord("baz", 0x5678);
   ASSERT_TRUE(bool(ReadRecord2));
   ASSERT_EQ(1U, ReadRecord2->getNumValueSites(IPVK_IndirectCallTarget));
   std::unique_ptr<InstrProfValueData[]> VD =
       ReadRecord2->getValueForSite(IPVK_IndirectCallTarget, 0);
   ASSERT_EQ(StringRef("bar"), StringRef((const char *)VD[0].Value, 3));
-  ASSERT_EQ(Max, VD[0].Count);
+  ASSERT_EQ(MaxValCount, VD[0].Count);
 }
 
 // This test tests that when there are too many values
 // for a given site, the merged results are properly
 // truncated.
 TEST_P(MaybeSparseInstrProfTest, get_icall_data_merge_site_trunc) {
   static const char caller[] = "caller";