[IR] Add hot to function attributes and use hot/cold attribute in function section prefix/suffix

[xur]

Summary

Clang FE currently has hot/cold function attribute. But we only have cold
function attribute in LLVM IR.

This patch adds support of hot function attribute to LLVM IR.
This attribute will be used in setting function section prefix/suffix.
Currently .hot and .unlikely suffix only are added in PGO (Sample PGO)
compilation (through isFunctionHotInCallGraph and isFunctionColdInCallGraph).

This patch changes the behavior. The new behavior is:
(1) If the user annotates a function as hot or isFunctionHotInCallGraph
is true, this function will be marked as hot.
(2) If the user annotates a function as cold or isFunctionColdInCallGraph
is true, this function will be marked as cold.

The changes are:
(1) user annotated function will used in setting function section prefix/suffix.
(2) hot attribute overwrites profile count based hotness.
(3) profile count based hotness overwrite user annotated cold attribute.

The intention for these changes is to provide the user a way to mark
certain function as hot in cases where training input is hard to cover all
the hot functions.

One thing that we might also need to change is we set cold attribute
in PGO instrumentation. This code is before isFunctionColdInCallGraph and
not use PSI. With this patch, we may mark more cold functions.
I did not change this part because cold attribute is used many downstream
optimizations other than setting function section prefix/suffix.

[davidxl]

For documentation, please update docs/LangRef.rst.

[davidxl]

Since this is a LangRef change, I suggest sending the summary of the patch as a RFC to llvm-dev list first to hear opinions from the community first.

[arsenm]

This is missing all of the asm parser/printer and bitcode tests

[xur]

Updated LangRef and added test cases per review comments.

[davidxl]

llvm/docs/LangRef.rst

1506 ↗

(On Diff #311731)

typo: information.

llvm/lib/CodeGen/CodeGenPrepare.cpp

476

document that this is a conservative behavior.

480–481

For conservative behavior, this should be &&

llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp

1758

Perhaps emit a warning here?

[MaskRay]

clang/lib/CodeGen/CodeGenModule.cpp

1742–1744

Omit braces

https://llvm.org/docs/CodingStandards.html#don-t-use-braces-on-simple-single-statement-bodies-of-if-else-loop-statements

clang/test/CodeGen/attributes.c

66

Nit: use [[#HOTDEF:]] (which is shorthand for [[#%u,HOTDEF:]]

[xur]

clang/test/CodeGen/attributes.c

66

Thanks. Good to know this format.

llvm/docs/LangRef.rst

1506 ↗

(On Diff #311731)

Fixed

llvm/lib/CodeGen/CodeGenPrepare.cpp

480–481

Should not be && -- this would require all cold functions be marked as cold by the users to place into unlikely section. I think it's too much.

I used '||" for the following reasons:
(1) If a function is not annotated as "cold", use PSI to determine if to place into unlikely section. This is no change from current behavior.
(2) If the function is marked as code
(2.1) if PSI shows the function is HOT, place into 'hot' section
(2.2) if PSI shows cold, place into 'unlikely' section
(2.3) if PSI shows neither code nor hot. This is a warm function. We honor the annotation, and place into 'unlikely" section.

llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp

1758

Added.

[xur]

Integrated comments from David and MaskRay.

[davidxl]

lgtm

[xur]

clang/test/CodeGen/attributes.c

66

@MaskRay: This does not work for me. #HOTDEF only matches numbers. I put ##HOTDEF here and add # to the uses later. But the match still failed.

I reverted to old format (also match the existing code. I plan to commit as it is. Feel free to simplify this after my commit. Thanks!

clang/lib/CodeGen/CGCall.cpp

if (TargetDecl) {
     if (TargetDecl->hasAttr<ReturnsTwiceAttr>())
       FuncAttrs.addAttribute(llvm::Attribute::ReturnsTwice);
     if (TargetDecl->hasAttr<NoThrowAttr>())
       FuncAttrs.addAttribute(llvm::Attribute::NoUnwind);
     if (TargetDecl->hasAttr<NoReturnAttr>())
       FuncAttrs.addAttribute(llvm::Attribute::NoReturn);
     if (TargetDecl->hasAttr<ColdAttr>())
       FuncAttrs.addAttribute(llvm::Attribute::Cold);
+    if (TargetDecl->hasAttr<HotAttr>())
+      FuncAttrs.addAttribute(llvm::Attribute::Hot);
     if (TargetDecl->hasAttr<NoDuplicateAttr>())
       FuncAttrs.addAttribute(llvm::Attribute::NoDuplicate);
     if (TargetDecl->hasAttr<ConvergentAttr>())
       FuncAttrs.addAttribute(llvm::Attribute::Convergent);
 
     if (const FunctionDecl *Fn = dyn_cast<FunctionDecl>(TargetDecl)) {
       AddAttributesFromFunctionProtoType(
           getContext(), FuncAttrs, Fn->getType()->getAs<FunctionProtoType>());

clang/lib/CodeGen/CodeGenModule.cpp

void CodeGenModule::SetLLVMFunctionAttributesForDefinition(const Decl *D,
   // Add other optimization related attributes if we are optimizing this
   // function.
   if (!D->hasAttr<OptimizeNoneAttr>()) {
     if (D->hasAttr<ColdAttr>()) {
       if (!ShouldAddOptNone)
         B.addAttribute(llvm::Attribute::OptimizeForSize);
       B.addAttribute(llvm::Attribute::Cold);
     }
-
+    if (D->hasAttr<HotAttr>()) {
+      B.addAttribute(llvm::Attribute::Hot);
+    }
     if (D->hasAttr<MinSizeAttr>())
       B.addAttribute(llvm::Attribute::MinSize);
   }
 
   F->addAttributes(llvm::AttributeList::FunctionIndex, B);
 
   unsigned alignment = D->getMaxAlignment() / Context.getCharWidth();
   if (alignment)

clang/test/CodeGen/attributes.c

 
 // CHECK: define void @t72() [[COLDDEF:#[0-9]+]] {
 void t71(void) __attribute__((cold));
 void t72() __attribute__((cold));
 void t72() { t71(); }
 // CHECK: call void @t71() [[COLDSITE:#[0-9]+]]
 // CHECK: declare void @t71() [[COLDDECL:#[0-9]+]]
 
+// CHECK: define void @t82() [[HOTDEF:#[0-9]+]] {
+void t81(void) __attribute__((hot));
+void t82() __attribute__((hot));
+void t82() { t81(); }
+// CHECK: call void @t81() [[HOTSITE:#[0-9]+]]
+// CHECK: declare void @t81() [[HOTDECL:#[0-9]+]]
+
 // CHECK: define void @t10() [[NUW]] section "SECT" {
 void t10(void) __attribute__((section("SECT")));
 void t10(void) {}
 // CHECK: define void @t11() [[NUW]] section "SECT" {
 void __attribute__((section("SECT"))) t11(void) {}
 
 // CHECK: define i32 @t19() [[NUW]] {
 extern int t19(void) __attribute__((weak_import));
void t24(f_t f1) {
   __attribute__((nocf_check)) f_t p = f1;
   (*p)();
 }
 
 // CHECK: attributes [[NUW]] = { noinline nounwind{{.*}} }
 // CHECK: attributes [[NR]] = { noinline noreturn nounwind{{.*}} }
 // CHECK: attributes [[COLDDEF]] = { cold {{.*}}}
 // CHECK: attributes [[COLDDECL]] = { cold {{.*}}}
+// CHECK: attributes [[HOTDEF]] = { hot {{.*}}}
+// CHECK: attributes [[HOTDECL]] = { hot {{.*}}}
 // CHECK: attributes [[NOCF_CHECK_FUNC]] = { nocf_check {{.*}}}
 // CHECK: attributes [[COLDSITE]] = { cold {{.*}}}
+// CHECK: attributes [[HOTSITE]] = { hot {{.*}}}
 // CHECK: attributes [[NOCF_CHECK_CALL]] = { nocf_check }

llvm/include/llvm/Bitcode/LLVMBitCodes.h

enum AttributeKindCodes {
   ATTR_KIND_NOSYNC = 63,
   ATTR_KIND_SANITIZE_MEMTAG = 64,
   ATTR_KIND_PREALLOCATED = 65,
   ATTR_KIND_NO_MERGE = 66,
   ATTR_KIND_NULL_POINTER_IS_VALID = 67,
   ATTR_KIND_NOUNDEF = 68,
   ATTR_KIND_BYREF = 69,
   ATTR_KIND_MUSTPROGRESS = 70,
+  ATTR_KIND_HOT = 71,
 };
 
 enum ComdatSelectionKindCodes {
   COMDAT_SELECTION_KIND_ANY = 1,
   COMDAT_SELECTION_KIND_EXACT_MATCH = 2,
   COMDAT_SELECTION_KIND_LARGEST = 3,
   COMDAT_SELECTION_KIND_NO_DUPLICATES = 4,
   COMDAT_SELECTION_KIND_SAME_SIZE = 5,

llvm/include/llvm/IR/Attributes.td

 def NoUndef : EnumAttr<"noundef">;
 
 /// Marks function as being in a cold path.
 def Cold : EnumAttr<"cold">;
 
 /// Can only be moved to control-equivalent blocks.
 def Convergent : EnumAttr<"convergent">;
 
+/// Marks function as being in a hot path and frequently called.
+def Hot: EnumAttr<"hot">;
+
 /// Pointer is known to be dereferenceable.
 def Dereferenceable : IntAttr<"dereferenceable">;
 
 /// Pointer is either null or dereferenceable.
 def DereferenceableOrNull : IntAttr<"dereferenceable_or_null">;
 
 /// Function may only access memory that is inaccessible from IR.
 def InaccessibleMemOnly : EnumAttr<"inaccessiblememonly">;

llvm/lib/AsmParser/LLParser.cpp

case lltok::kw_allocsize: {
         return true;
       B.addAllocSizeAttr(ElemSizeArg, NumElemsArg);
       continue;
     }
     case lltok::kw_alwaysinline: B.addAttribute(Attribute::AlwaysInline); break;
     case lltok::kw_argmemonly: B.addAttribute(Attribute::ArgMemOnly); break;
     case lltok::kw_builtin: B.addAttribute(Attribute::Builtin); break;
     case lltok::kw_cold: B.addAttribute(Attribute::Cold); break;
+    case lltok::kw_hot: B.addAttribute(Attribute::Hot); break;
     case lltok::kw_convergent: B.addAttribute(Attribute::Convergent); break;
     case lltok::kw_inaccessiblememonly:
       B.addAttribute(Attribute::InaccessibleMemOnly); break;
     case lltok::kw_inaccessiblemem_or_argmemonly:
       B.addAttribute(Attribute::InaccessibleMemOrArgMemOnly); break;
     case lltok::kw_inlinehint: B.addAttribute(Attribute::InlineHint); break;
     case lltok::kw_jumptable: B.addAttribute(Attribute::JumpTable); break;
     case lltok::kw_minsize: B.addAttribute(Attribute::MinSize); break;

llvm/lib/Bitcode/Reader/BitcodeReader.cpp

static Attribute::AttrKind getAttrFromCode(uint64_t Code) {
   case bitc::ATTR_KIND_PREALLOCATED:
     return Attribute::Preallocated;
   case bitc::ATTR_KIND_NOUNDEF:
     return Attribute::NoUndef;
   case bitc::ATTR_KIND_BYREF:
     return Attribute::ByRef;
   case bitc::ATTR_KIND_MUSTPROGRESS:
     return Attribute::MustProgress;
+  case bitc::ATTR_KIND_HOT:
+    return Attribute::Hot;
   }
 }
 
 Error BitcodeReader::parseAlignmentValue(uint64_t Exponent,
                                          MaybeAlign &Alignment) {
   // Note: Alignment in bitcode files is incremented by 1, so that zero
   // can be used for default alignment.
   if (Exponent > Value::MaxAlignmentExponent + 1)

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

static uint64_t getAttrKindEncoding(Attribute::AttrKind Kind) {
   case Attribute::ByVal:
     return bitc::ATTR_KIND_BY_VAL;
   case Attribute::Convergent:
     return bitc::ATTR_KIND_CONVERGENT;
   case Attribute::InAlloca:
     return bitc::ATTR_KIND_IN_ALLOCA;
   case Attribute::Cold:
     return bitc::ATTR_KIND_COLD;
+  case Attribute::Hot:
+    return bitc::ATTR_KIND_HOT;
   case Attribute::InaccessibleMemOnly:
     return bitc::ATTR_KIND_INACCESSIBLEMEM_ONLY;
   case Attribute::InaccessibleMemOrArgMemOnly:
     return bitc::ATTR_KIND_INACCESSIBLEMEM_OR_ARGMEMONLY;
   case Attribute::InlineHint:
     return bitc::ATTR_KIND_INLINE_HINT;
   case Attribute::InReg:
     return bitc::ATTR_KIND_IN_REG;

llvm/lib/CodeGen/CodeGenPrepare.cpp

bool CodeGenPrepare::runOnFunction(Function &F) {
   TLInfo = &getAnalysis<TargetLibraryInfoWrapperPass>().getTLI(F);
   TTI = &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
   LI = &getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
   BPI.reset(new BranchProbabilityInfo(F, *LI));
   BFI.reset(new BlockFrequencyInfo(F, *BPI, *LI));
   PSI = &getAnalysis<ProfileSummaryInfoWrapperPass>().getPSI();
   OptSize = F.hasOptSize();
   if (ProfileGuidedSectionPrefix) {
-    if (PSI->isFunctionHotInCallGraph(&F, *BFI))
+    // The hot attribute overwrites profile count based hotness while profile
+    // counts based hotness overwrite the cold attribute.
+    if (F.hasFnAttribute(Attribute::Hot) ||
+        PSI->isFunctionHotInCallGraph(&F, *BFI))
       F.setSectionPrefix(".hot");
-    else if (PSI->isFunctionColdInCallGraph(&F, *BFI))
+    else if (PSI->isFunctionColdInCallGraph(&F, *BFI) ||
+             F.hasFnAttribute(Attribute::Cold))
       F.setSectionPrefix(".unlikely");
     else if (ProfileUnknownInSpecialSection && PSI->hasPartialSampleProfile() &&
              PSI->isFunctionHotnessUnknown(F))
       F.setSectionPrefix(".unknown");
   }
 
   /// This optimization identifies DIV instructions that can be
   /// profitably bypassed and carried out with a shorter, faster divide.

llvm/lib/IR/Attributes.cpp

std::string Attribute::getAsString(bool InAttrGrp) const {
   if (hasAttribute(Attribute::SanitizeMemory))
     return "sanitize_memory";
   if (hasAttribute(Attribute::UWTable))
     return "uwtable";
   if (hasAttribute(Attribute::ZExt))
     return "zeroext";
   if (hasAttribute(Attribute::Cold))
     return "cold";
+  if (hasAttribute(Attribute::Hot))
+    return "hot";
   if (hasAttribute(Attribute::ImmArg))
     return "immarg";
   if (hasAttribute(Attribute::NoUndef))
     return "noundef";
   if (hasAttribute(Attribute::MustProgress))
     return "mustprogress";
 
   const bool IsByVal = hasAttribute(Attribute::ByVal);

llvm/lib/IR/Verifier.cpp

static bool isFuncOnlyAttr(Attribute::AttrKind Kind) {
   case Attribute::SanitizeMemTag:
   case Attribute::SanitizeThread:
   case Attribute::SanitizeMemory:
   case Attribute::MinSize:
   case Attribute::NoDuplicate:
   case Attribute::Builtin:
   case Attribute::NoBuiltin:
   case Attribute::Cold:
+  case Attribute::Hot:
   case Attribute::OptForFuzzing:
   case Attribute::OptimizeNone:
   case Attribute::JumpTable:
   case Attribute::Convergent:
   case Attribute::ArgMemOnly:
   case Attribute::NoRecurse:
   case Attribute::InaccessibleMemOnly:
   case Attribute::InaccessibleMemOrArgMemOnly:

llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp

static bool annotateAllFunctions(
   // can affect the BranchProbabilityInfo of any callers, resulting in an
   // inconsistent MST between prof-gen and prof-use.
   for (auto &F : HotFunctions) {
     F->addFnAttr(Attribute::InlineHint);
     LLVM_DEBUG(dbgs() << "Set inline attribute to function: " << F->getName()
                       << "\n");
   }
   for (auto &F : ColdFunctions) {
+    // Only set when there is no Attribute::Hot set by the user. For Hot
+    // attribute, user's annotation has the precedence over the profile.
+    if (F->hasFnAttribute(Attribute::Hot))
+      continue;
     F->addFnAttr(Attribute::Cold);
     LLVM_DEBUG(dbgs() << "Set cold attribute to function: " << F->getName()
                       << "\n");
   }
   return true;
 }
 
 PGOInstrumentationUse::PGOInstrumentationUse(std::string Filename,

llvm/test/CodeGen/X86/hot-unlikely-section-prefix.ll

+; Test hot or unlikely section postfix based on profile and user annotation.
+; RUN: llc < %s | FileCheck %s
+target triple = "x86_64-unknown-linux-gnu"
+
+; Function Attrs: inlinehint norecurse nounwind readnone uwtable
+define dso_local i32 @hot1() #0 !prof !31 {
+entry:
+  ret i32 1
+}
+; CHECK: .section        .text.hot.,"ax",@progbits
+; CHECK: .globl  hot1
+
+; Function Attrs: cold norecurse nounwind readnone uwtable
+define dso_local i32 @cold1() #1 !prof !32 {
+entry:
+  ret i32 1
+}
+; CHECK: .section        .text.unlikely.,"ax",@progbits
+; CHECK: .globl  cold1
+
+; Function Attrs: cold inlinehint noinline norecurse nounwind optsize readnone uwtable
+define dso_local i32 @hot2() #2 !prof !31 {
+entry:
+  ret i32 1
+}
+; CHECK: .section        .text.hot.,"ax",@progbits
+; CHECK: .globl  hot2
+
+define dso_local i32 @normal() {
+entry:
+  ret i32 1
+}
+; CHECK: text
+; CHECK: .globl  normal
+
+; Function Attrs: hot noinline norecurse nounwind readnone uwtable
+define dso_local i32 @hot3() #3 !prof !32 {
+entry:
+  ret i32 1
+}
+; CHECK: .section        .text.hot.,"ax",@progbits
+; CHECK: .globl  hot3
+
+; Function Attrs: cold noinline norecurse nounwind optsize readnone uwtable
+define dso_local i32 @cold2() #4 {
+entry:
+  ret i32 1
+}
+; CHECK: .section        .text.unlikely.,"ax",@progbits
+; CHECK: .globl  cold2
+
+; Function Attrs: hot noinline norecurse nounwind readnone uwtable
+define dso_local i32 @hot4() #3 {
+entry:
+  ret i32 1
+}
+; CHECK: .section        .text.hot.,"ax",@progbits
+; CHECK: .globl  hot4
+
+attributes #0 = { inlinehint norecurse nounwind readnone uwtable }
+attributes #1 = { cold norecurse nounwind readnone uwtable }
+attributes #2 = { cold inlinehint noinline norecurse nounwind optsize readnone uwtable }
+attributes #3 = { hot noinline norecurse nounwind readnone uwtable }
+attributes #4 = { cold noinline norecurse nounwind optsize readnone uwtable }
+
+!llvm.module.flags = !{!0, !1}
+!llvm.ident = !{!30}
+
+!0 = !{i32 1, !"wchar_size", i32 4}
+!1 = !{i32 1, !"ProfileSummary", !2}
+!2 = !{!3, !4, !5, !6, !7, !8, !9, !10, !11, !12}
+!3 = !{!"ProfileFormat", !"InstrProf"}
+!4 = !{!"TotalCount", i64 402020}
+!5 = !{!"MaxCount", i64 200000}
+!6 = !{!"MaxInternalCount", i64 2000}
+!7 = !{!"MaxFunctionCount", i64 200000}
+!8 = !{!"NumCounts", i64 7}
+!9 = !{!"NumFunctions", i64 5}
+!10 = !{!"IsPartialProfile", i64 0}
+!11 = !{!"PartialProfileRatio", double 0.000000e+00}
+!12 = !{!"DetailedSummary", !13}
+!13 = !{!14, !15, !16, !17, !18, !19, !20, !21, !22, !23, !24, !25, !26, !27, !28, !29}
+!14 = !{i32 10000, i64 200000, i32 1}
+!15 = !{i32 100000, i64 200000, i32 1}
+!16 = !{i32 200000, i64 200000, i32 1}
+!17 = !{i32 300000, i64 200000, i32 1}
+!18 = !{i32 400000, i64 200000, i32 1}
+!19 = !{i32 500000, i64 100000, i32 3}
+!20 = !{i32 600000, i64 100000, i32 3}
+!21 = !{i32 700000, i64 100000, i32 3}
+!22 = !{i32 800000, i64 100000, i32 3}
+!23 = !{i32 900000, i64 100000, i32 3}
+!24 = !{i32 950000, i64 100000, i32 3}
+!25 = !{i32 990000, i64 100000, i32 3}
+!26 = !{i32 999000, i64 2000, i32 4}
+!27 = !{i32 999900, i64 2000, i32 4}
+!28 = !{i32 999990, i64 10, i32 6}
+!29 = !{i32 999999, i64 10, i32 6}
+!30 = !{!"clang version 12.0.0 (https://github.com/llvm/llvm-project.git 53c5fdd59a5cf7fbb4dcb7a7e84c9c4a40d32a84)"}
+!31 = !{!"function_entry_count", i64 100000}
+!32 = !{!"function_entry_count", i64 10}