[InstrProfiling] Add -runtime-counter-relocation=function mode

[mcgrathr]

Summary

The -runtime-counter-relocation switch is changed from a Boolean to a
three-way switch: none, variable, or function. The previous
-runtime-counter-relocation=true behavior is now selected with
-runtime-counter-relocation=variable. In the new ...=function mode,
instead of loading the global variable __llvm_profile_counter_bias,
instrumented code calls the function __llvm_profile_get_counter_bias
with no arguments to return the bias. This can be a runtime function or
it can be defined in the translation unit to be inlined.

[mcgrathr]

Use a DenseMap rather than instruction name to match bias load.

[mcgrathr]

second copy of InstrProfData.inc

[phosek]

An alternative I thought of would be to always use a call and generate a (weak) implementation of __llvm_profile_counter_dynamic_bias inside each translation unit. The default generated function would simply return the variable. I'm not sure what the impact on performance would be though.

compiler-rt/include/profile/InstrProfData.inc

668

It's not clear to me from this name that it refers to a function that returns the bias. How about __llvm_profile_get_counter_bias? That would match symbols like __llvm_profile_get_magic or __llvm_profile_get_version.

[mcgrathr]

In D114383#3160468, @phosek wrote:

An alternative I thought of would be to always use a call and generate a (weak) implementation of __llvm_profile_counter_dynamic_bias inside each translation unit. The default generated function would simply return the variable. I'm not sure what the impact on performance would be though.

I think the impact is likely to be significant in any place that the function is not inlined. It's not just the call/return overhead per se (which might well be in the noise), but it also requires immediately spilling all the argument registers in the prologue because the calling convention for the bias fetch will clobber all those registers. Without LTO, I don't think there's any possibility for a "weak inlined" definition, since inlining needs to happen early in compilation and weak vs strong definitions are not resolved one way or another until link time.

[mcgrathr]

renamed callback

[phosek]

LGTM

compiler-rt/include/profile/InstrProfData.inc

668

I'd rename the macro to match the variable as well.

llvm/test/Instrumentation/InstrProfiling/runtime-counter-relocation.ll

17

30

[davidxl]

I might have missed the discussion somewhere. What the use case for the dynamic bias?

compiler-rt/include/profile/InstrProfData.inc

 #define VARIANT_MASKS_ALL 0xff00000000000000ULL
 #define GET_VERSION(V) ((V) & ~VARIANT_MASKS_ALL)
 #define VARIANT_MASK_IR_PROF (0x1ULL << 56)
 #define VARIANT_MASK_CSIR_PROF (0x1ULL << 57)
 #define VARIANT_MASK_INSTR_ENTRY (0x1ULL << 58)
 #define INSTR_PROF_RAW_VERSION_VAR __llvm_profile_raw_version
 #define INSTR_PROF_PROFILE_RUNTIME_VAR __llvm_profile_runtime
 #define INSTR_PROF_PROFILE_COUNTER_BIAS_VAR __llvm_profile_counter_bias
+#define INSTR_PROF_PROFILE_COUNTER_DYNAMIC_BIAS_FUNC __llvm_profile_get_counter_bias
 
 /* The variable that holds the name of the profile data
  * specified via command line. */
 #define INSTR_PROF_PROFILE_NAME_VAR __llvm_profile_filename
 
 /* section name strings common to all targets other
    than WIN32 */
 #define INSTR_PROF_DATA_COMMON __llvm_prf_data

llvm/include/llvm/ProfileData/InstrProf.h

 inline StringRef getInstrProfRuntimeHookVarUseFuncName() {
   return "__llvm_profile_runtime_user";
 }
 
 inline StringRef getInstrProfCounterBiasVarName() {
   return INSTR_PROF_QUOTE(INSTR_PROF_PROFILE_COUNTER_BIAS_VAR);
 }
 
+inline StringRef getInstrProfCounterDynamicBiasFuncName() {
+  return INSTR_PROF_QUOTE(INSTR_PROF_PROFILE_COUNTER_DYNAMIC_BIAS_FUNC);
+}
+
 /// Return the marker used to separate PGO names during serialization.
 inline StringRef getInstrProfNameSeparator() { return "\01"; }
 
 /// Return the modified name for function \c F suitable to be
 /// used the key for profile lookup. Variable \c InLTO indicates if this
 /// is called in LTO optimization passes.
 std::string getPGOFuncName(const Function &F, bool InLTO = false,
                            uint64_t Version = INSTR_PROF_INDEX_VERSION);

llvm/include/llvm/ProfileData/InstrProfData.inc

 #define VARIANT_MASKS_ALL 0xff00000000000000ULL
 #define GET_VERSION(V) ((V) & ~VARIANT_MASKS_ALL)
 #define VARIANT_MASK_IR_PROF (0x1ULL << 56)
 #define VARIANT_MASK_CSIR_PROF (0x1ULL << 57)
 #define VARIANT_MASK_INSTR_ENTRY (0x1ULL << 58)
 #define INSTR_PROF_RAW_VERSION_VAR __llvm_profile_raw_version
 #define INSTR_PROF_PROFILE_RUNTIME_VAR __llvm_profile_runtime
 #define INSTR_PROF_PROFILE_COUNTER_BIAS_VAR __llvm_profile_counter_bias
+#define INSTR_PROF_PROFILE_COUNTER_DYNAMIC_BIAS_FUNC __llvm_profile_get_counter_bias
 
 /* The variable that holds the name of the profile data
  * specified via command line. */
 #define INSTR_PROF_PROFILE_NAME_VAR __llvm_profile_filename
 
 /* section name strings common to all targets other
    than WIN32 */
 #define INSTR_PROF_DATA_COMMON __llvm_prf_data

llvm/include/llvm/Transforms/Instrumentation/InstrProfiling.h

struct PerFunctionProfileData {
     GlobalVariable *RegionCounters = nullptr;
     GlobalVariable *DataVar = nullptr;
 
     PerFunctionProfileData() {
       memset(NumValueSites, 0, sizeof(uint32_t) * (IPVK_Last + 1));
     }
   };
   DenseMap<GlobalVariable *, PerFunctionProfileData> ProfileDataMap;
+  // If runtime relocation is enabled, this maps functions to the instruction
+  // that produces the profile relocation bias.
+  DenseMap<const Function *, Instruction *> FunctionToProfileBiasMap;
   std::vector<GlobalValue *> CompilerUsedVars;
   std::vector<GlobalValue *> UsedVars;
   std::vector<GlobalVariable *> ReferencedNames;
   GlobalVariable *NamesVar;
   size_t NamesSize;
 
   // Is this lowering for the context-sensitive instrumentation.
   bool IsCS;

llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp

 
 namespace {
 
 cl::opt<bool> DoHashBasedCounterSplit(
     "hash-based-counter-split",
     cl::desc("Rename counter variable of a comdat function based on cfg hash"),
     cl::init(true));
 
-cl::opt<bool>
-    RuntimeCounterRelocation("runtime-counter-relocation",
-                             cl::desc("Enable relocating counters at runtime."),
-                             cl::init(false));
+enum class RuntimeCounterRelocationTy {
+  kNone,
+  kVariable,
+  kFunction,
+};
+
+cl::opt<RuntimeCounterRelocationTy> RuntimeCounterRelocation(
+    "runtime-counter-relocation",
+    cl::values(clEnumValN(RuntimeCounterRelocationTy::kNone, "none",
+                          "Non-relocatable counters."),
+               clEnumValN(RuntimeCounterRelocationTy::kVariable, "variable",
+                          "Counters relocation bias from a global variable."),
+               clEnumValN(RuntimeCounterRelocationTy::kFunction, "function",
+                          "Counters relocation bias from a function call.")),
+    cl::init(RuntimeCounterRelocationTy::kNone));
 
 cl::opt<bool> ValueProfileStaticAlloc(
     "vp-static-alloc",
     cl::desc("Do static counter allocation for value profiler"),
     cl::init(true));
 
 cl::opt<double> NumCountersPerValueSite(
     "vp-counters-per-site",

 }
 
 bool InstrProfiling::isRuntimeCounterRelocationEnabled() const {
   // Mach-O don't support weak external references.
   if (TT.isOSBinFormatMachO())
     return false;
 
   if (RuntimeCounterRelocation.getNumOccurrences() > 0)
-    return RuntimeCounterRelocation;
+    return RuntimeCounterRelocation != RuntimeCounterRelocationTy::kNone;
 
   // Fuchsia uses runtime counter relocation by default.
   return TT.isOSFuchsia();
 }
 
 bool InstrProfiling::isCounterPromotionEnabled() const {
   if (DoCounterPromotion.getNumOccurrences() > 0)
     return DoCounterPromotion;
void InstrProfiling::lowerIncrement(InstrProfIncrementInst *Inc) {
   uint64_t Index = Inc->getIndex()->getZExtValue();
   Value *Addr = Builder.CreateConstInBoundsGEP2_32(Counters->getValueType(),
                                                    Counters, 0, Index);
 
   if (isRuntimeCounterRelocationEnabled()) {
     Type *Int64Ty = Type::getInt64Ty(M->getContext());
     Type *Int64PtrTy = Type::getInt64PtrTy(M->getContext());
     Function *Fn = Inc->getParent()->getParent();
-    Instruction &I = Fn->getEntryBlock().front();
-    LoadInst *LI = dyn_cast<LoadInst>(&I);
-    if (!LI) {
-      IRBuilder<> Builder(&I);
+    Instruction *&BiasInst = FunctionToProfileBiasMap[Fn];
+    if (!BiasInst) {
+      IRBuilder<> Builder(&Fn->getEntryBlock().front());
+      if (RuntimeCounterRelocation == RuntimeCounterRelocationTy::kFunction) {
+        BiasInst = Builder.CreateCall(
+            M->getOrInsertFunction(getInstrProfCounterDynamicBiasFuncName(),
+                                   Int64Ty),
+            None, "prof_counters_bias");
+      } else {
         GlobalVariable *Bias =
             M->getGlobalVariable(getInstrProfCounterBiasVarName());
         if (!Bias) {
           // Compiler must define this variable when runtime counter relocation
           // is being used. Runtime has a weak external reference that is used
           // to check whether that's the case or not.
-        Bias = new GlobalVariable(
-            *M, Int64Ty, false, GlobalValue::LinkOnceODRLinkage,
-            Constant::getNullValue(Int64Ty), getInstrProfCounterBiasVarName());
+          Bias = new GlobalVariable(*M, Int64Ty, false,
+                                    GlobalValue::LinkOnceODRLinkage,
+                                    Constant::getNullValue(Int64Ty),
+                                    getInstrProfCounterBiasVarName());
           Bias->setVisibility(GlobalVariable::HiddenVisibility);
           // A definition that's weak (linkonce_odr) without being in a COMDAT
           // section wouldn't lead to link errors, but it would lead to a dead
-        // data word from every TU but one. Putting it in COMDAT ensures there
-        // will be exactly one data slot in the link.
+          // data word from every TU but one. Putting it in COMDAT ensures
+          // there will be exactly one data slot in the link.
           if (TT.supportsCOMDAT())
             Bias->setComdat(M->getOrInsertComdat(Bias->getName()));
         }
-      LI = Builder.CreateLoad(Int64Ty, Bias);
+        BiasInst = Builder.CreateLoad(Int64Ty, Bias, "prof_counters_bias");
+      }
     }
-    auto *Add = Builder.CreateAdd(Builder.CreatePtrToInt(Addr, Int64Ty), LI);
+    auto *Add =
+        Builder.CreateAdd(Builder.CreatePtrToInt(Addr, Int64Ty), BiasInst);
     Addr = Builder.CreateIntToPtr(Add, Int64PtrTy);
   }
 
   if (Options.Atomic || AtomicCounterUpdateAll ||
       (Index == 0 && AtomicFirstCounter)) {
     Builder.CreateAtomicRMW(AtomicRMWInst::Add, Addr, Inc->getStep(),
                             MaybeAlign(), AtomicOrdering::Monotonic);
   } else {

llvm/test/Instrumentation/InstrProfiling/runtime-counter-relocation.ll

 ; RUN: opt < %s -S -instrprof | FileCheck %s
-; RUN: opt < %s -S -instrprof -runtime-counter-relocation | FileCheck -check-prefixes=RELOC %s
+; RUN: opt < %s -S -instrprof -runtime-counter-relocation=variable | FileCheck -check-prefixes=RELOC,RELOCVAR %s
+; RUN: opt < %s -S -instrprof -runtime-counter-relocation=function | FileCheck -check-prefixes=RELOC,RELOCFUNC %s
 
 target triple = "x86_64-unknown-linux-gnu"
 
 @__profn_foo = private constant [3 x i8] c"foo"
-; RELOC: $__llvm_profile_counter_bias = comdat any
-; RELOC: @__llvm_profile_counter_bias = linkonce_odr hidden global i64 0, comdat
+; RELOCVAR: $__llvm_profile_counter_bias = comdat any
+; RELOCVAR: @__llvm_profile_counter_bias = linkonce_odr hidden global i64 0, comdat
 
 ; CHECK-LABEL: define void @foo
 ; CHECK-NEXT: %pgocount = load i64, i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc_foo, i32 0, i32 0)
 ; CHECK-NEXT: %1 = add i64 %pgocount, 1
 ; CHECK-NEXT: store i64 %1, i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc_foo, i32 0, i32 0)
 ; RELOC-LABEL: define void @foo
-; RELOC-NEXT: %1 = load i64, i64* @__llvm_profile_counter_bias
-; RELOC-NEXT: %2 = add i64 ptrtoint ([1 x i64]* @__profc_foo to i64), %1
-; RELOC-NEXT: %3 = inttoptr i64 %2 to i64*
-; RELOC-NEXT: %pgocount = load i64, i64* %3
-; RELOC-NEXT: %4 = add i64 %pgocount, 1
-; RELOC-NEXT: store i64 %4, i64* %3
+; RELOCVAR-NEXT: %prof_counters_bias = load i64, i64* @__llvm_profile_counter_bias
+; RELOCFUNC-NEXT: %prof_counters_bias = call i64 @__llvm_profile_counter_dynamic_bias()
+; RELOC-NEXT: %1 = add i64 ptrtoint ([1 x i64]* @__profc_foo to i64), %prof_counters_bias
+; RELOC-NEXT: %2 = inttoptr i64 %1 to i64*
+; RELOC-NEXT: %pgocount = load i64, i64* %2
+; RELOC-NEXT: %3 = add i64 %pgocount, 1
+; RELOC-NEXT: store i64 %3, i64* %2
 define void @foo() {
   call void @llvm.instrprof.increment(i8* getelementptr inbounds ([3 x i8], [3 x i8]* @__profn_foo, i32 0, i32 0), i64 0, i32 1, i32 0)
   ret void
 }
 
 declare void @llvm.instrprof.increment(i8*, i64, i32, i32)
+
+; RELOCFUNC: declare i64 @__llvm_profile_counter_dynamic_bias()