Add a new flavor "BBLabels" to LLVM_BUILD_INSTRUMENTED

[shenhan]

Summary

This CL adds "BBLabels" support for LLVM_BUILD_INSTRUMENTED.

"BBLabels" adds -fbasic-block-sections=labels to C/CXX flags as well as LD flags when doing ThinLTO. With this option, CMAKE builds binaries with auxiliary basic block meta data that later guide propeller[1] profile generation.

With this patch (and its to-be-uploaded child patch that adds support for passing propeller profiles using LLVM_PROFDATA_FILE), it enables people to build propeller optimized llvm tools in a formalized way.

Here is the proposed workflow to build a propeller optimized clang binary (similar to PGO build).

1. Build instrumented clang binary:

cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_BUILD_INSTRUMENTED=BBLabels -DLLVM_ENABLE_PROJECTS="clang" llvm-project/llvm
ninja build

1. Using the above built binary in day-to-day routine and collect perf.data file and then convert it to propeller profiles.
2. Build the optimized clang binary:

cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_PROPELLER_PROFILES=<profile_name> -DLLVM_ENABLE_PROJECTS="clang" llvm-project/llvm

The experiment data show propeller optimized clang has a 5% compiler time improvement over PGO build.

[1]. Propeller, submitted in 2021.

[shenhan]

Here is the proposed workflow to build a propeller optimized clang binary (similar to PGO build).

1. Build instrumented clang binary:

cmake -G Ninja ... -DLLVM_BUILD_INSTRUMENTED=BBLabels ... llvm-project/llvm
ninja build

2. Using the above built binary in day-to-day routine and collect perf.data file and then convert it to propeller profiles.

3. Build the optimized clang binary:

cmake -G Ninja ... -DLLVM_PROPELLER_PROFILES=<profile> ... llvm-project/llvm

The experiment data show propeller optimized clang has a 5% compiler time improvement over PGO build.

[tmsriram]

In D125426#3509699, @shenhan wrote:

Here is the proposed workflow to build a propeller optimized clang binary (similar to PGO build).

1. Build instrumented clang binary:

cmake -G Ninja ... -DLLVM_BUILD_INSTRUMENTED=BBLabels ... llvm-project/llvm
ninja build

2. Using the above built binary in day-to-day routine and collect perf.data file and then convert it to propeller profiles.

3. Build the optimized clang binary:

cmake -G Ninja ... -DLLVM_PROPELLER_PROFILES=<profile> ... llvm-project/llvm

The experiment data show propeller optimized clang has a 5% compiler time improvement over PGO build.

Thanks! Please move this to the description.

llvm/cmake/modules/HandleLLVMOptions.cmake

981

Suggestion: Maybe say "Use BBLabels for Propeller instrumentation via basic block labels"

[shenhan]

In D125426#3509715, @tmsriram wrote:

In D125426#3509699, @shenhan wrote:

Here is the proposed workflow to build a propeller optimized clang binary (similar to PGO build).

1. Build instrumented clang binary:

cmake -G Ninja ... -DLLVM_BUILD_INSTRUMENTED=BBLabels ... llvm-project/llvm
ninja build

2. Using the above built binary in day-to-day routine and collect perf.data file and then convert it to propeller profiles.

3. Build the optimized clang binary:

cmake -G Ninja ... -DLLVM_PROPELLER_PROFILES=<profile> ... llvm-project/llvm

The experiment data show propeller optimized clang has a 5% compiler time improvement over PGO build.

Thanks! Please move this to the description.

Done.

[hans]

Nice!

Please update llvm/docs/CMake.rst to document the different options of LLVM_BUILD_INSTRUMENTED.

Thanks for including the workflow for building a propeller-optimized Clang. Perhaps this should be added to llvm/docs/HowToBuildWithPGO.rst? Or could it be added to Propeller's documentation (where is that?). Also, could the workflow include to how apply this on top of ThinLTO and PGO for peak performance?

[tmsriram]

This LGTM, please address hans' comments.

llvm/cmake/modules/HandleLLVMOptions.cmake

 option(LLVM_ENABLE_RTTI "Enable run time type information" OFF)
 if(LLVM_ENABLE_EH AND NOT LLVM_ENABLE_RTTI)
   message(FATAL_ERROR "Exception handling requires RTTI. You must set LLVM_ENABLE_RTTI to ON")
 endif()
 
 option(LLVM_ENABLE_IR_PGO "Build LLVM and tools with IR PGO instrumentation (deprecated)" Off)
 mark_as_advanced(LLVM_ENABLE_IR_PGO)
 
-set(LLVM_BUILD_INSTRUMENTED OFF CACHE STRING "Build LLVM and tools with PGO instrumentation. May be specified as IR or Frontend")
+set(LLVM_BUILD_INSTRUMENTED OFF CACHE STRING "Build LLVM and tools with instrumentation. Use IR, CSIR or Frontend for PGO. Use BBLabels for basic block sections instrumentation")
 set(LLVM_VP_COUNTERS_PER_SITE "1.5" CACHE STRING "Value profile counters to use per site for IR PGO with Clang")
 mark_as_advanced(LLVM_BUILD_INSTRUMENTED LLVM_VP_COUNTERS_PER_SITE)
 string(TOUPPER "${LLVM_BUILD_INSTRUMENTED}" uppercase_LLVM_BUILD_INSTRUMENTED)
 
 if (LLVM_BUILD_INSTRUMENTED)
   if (LLVM_ENABLE_IR_PGO OR uppercase_LLVM_BUILD_INSTRUMENTED STREQUAL "IR")
     append("-fprofile-generate=\"${LLVM_PROFILE_DATA_DIR}\""
       CMAKE_CXX_FLAGS
elseif(uppercase_LLVM_BUILD_INSTRUMENTED STREQUAL "CSIR")
     append("-fcs-profile-generate=\"${LLVM_CSPROFILE_DATA_DIR}\""
       CMAKE_CXX_FLAGS
       CMAKE_C_FLAGS)
     if(NOT LINKER_IS_LLD_LINK)
       append("-fcs-profile-generate=\"${LLVM_CSPROFILE_DATA_DIR}\""
         CMAKE_EXE_LINKER_FLAGS
         CMAKE_SHARED_LINKER_FLAGS)
     endif()
+  elseif(uppercase_LLVM_BUILD_INSTRUMENTED STREQUAL "BBLABELS")
+    if (CMAKE_CXX_COMPILER_ID MATCHES "Clang" AND
+        CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 11)
+      append("-fbasic-block-sections=labels" CMAKE_CXX_FLAGS CMAKE_C_FLAGS)
+    else()
+      message(FATAL_ERROR
+        "LLVM_BUILD_INSTRUMENTED=BBLabels is only available when building with clang")
+    endif()
+    if(uppercase_LLVM_ENABLE_LTO STREQUAL "THIN")
+      append("-Wl,--lto-basic-block-sections=labels" CMAKE_EXE_LINKER_FLAGS)
+    endif()
   else()
     append("-fprofile-instr-generate=\"${LLVM_PROFILE_FILE_PATTERN}\""
       CMAKE_CXX_FLAGS
       CMAKE_C_FLAGS)
     if(NOT LINKER_IS_LLD_LINK)
       append("-fprofile-instr-generate=\"${LLVM_PROFILE_FILE_PATTERN}\""
         CMAKE_EXE_LINKER_FLAGS
         CMAKE_SHARED_LINKER_FLAGS)