[test-suite] Add cmake option to use IR PGO (TEST_SUITE_USE_IR_PGO)

[vsk]

Summary

This allows building programs in the test suite with LLVM's IR PGO
feature. Support for re-building programs with the collected profdata is
already present.

E.g, to build for training:

$ cmake -G Ninja -DCMAKE_C_COMPILER=$CC -C ~/src/llvm-test-suite/cmake/caches/target-arm64-iphoneos-internal.cmake -C ~/src/llvm-test-suite/cmake/caches/ReleaseLTO.cmake ~/src/llvm-test-suite -DTEST_SUITE_REMOTE_HOST=device -DTEST_SUITE_PROFILE_GENERATE=On -DTEST_SUITE_USE_IR_PGO=On -DTEST_SUITE_BENCHMARKING_ONLY=On -DTEST_SUITE_RUN_TYPE=train

To collect profile data: ninja && ninja rsync && llvm-lit .

To re-build with training data present, just re-configure and re-run ninja:

$ cmake -G Ninja -DCMAKE_C_COMPILER=$CC -C ~/src/llvm-test-suite/cmake/caches/target-arm64-iphoneos-internal.cmake -C ~/src/llvm-test-suite/cmake/caches/ReleaseLTO.cmake ~/src/llvm-test-suite -DTEST_SUITE_REMOTE_HOST=device -DTEST_SUITE_PROFILE_GENERATE=Off -DTEST_SUITE_BENCHMARKING_ONLY=On -DTEST_SUITE_RUN_TYPE=ref -DTEST_SUITE_PROFILE_USE=On

[anemet]

LGTM

CMakeLists.txt

find_program(TEST_SUITE_LLVM_SIZE NAMES "llvm-size"
                HINTS ${CMAKE_C_COMPILER_DIRECTORY})
   mark_as_advanced(TEST_SUITE_LLVM_SIZE)
   if(TEST_SUITE_LLVM_SIZE STREQUAL "TEST_SUITE_LLVM_SIZE-NOTFOUND")
     message(FATAL_ERROR "llvm-size not found.
 Make sure it is in your path or set TEST_SUITE_COLLECT_CODE_SIZE to OFF")
   endif()
 endif()
 
+option(TEST_SUITE_USE_IR_PGO
+  "Use IR PGO instrumentation (requires TEST_SUITE_PROFILE_GENERATE)" OFF)
 
 # Enable profile generate mode in lit. Note that this does not automatically
 # add something like -fprofile-instr-generate to the compiler flags.
 option(TEST_SUITE_PROFILE_GENERATE "Enable lit profile generate mode" OFF)
 # Set value to python style True/False
 if(TEST_SUITE_PROFILE_GENERATE)
   find_program(TEST_SUITE_LLVM_PROFDATA NAMES "llvm-profdata"
                HINTS ${CMAKE_C_COMPILER_DIRECTORY})
   mark_as_advanced(TEST_SUITE_LLVM_PROFDATA)
   if(TEST_SUITE_LLVM_PROFDATA STREQUAL "TEST_SUITE_LLVM_PROFDATA-NOTFOUND")
     message(FATAL_ERROR "llvm-profdata not found.
 Make sure it is in your path or set TEST_SUITE_PROFILE_GENERATE to OFF")
   endif()
 
   set(TEST_SUITE_PROFILE_GENERATE "True")
-  list(APPEND CFLAGS -fprofile-instr-generate)
-  list(APPEND CXXFLAGS -fprofile-instr-generate)
-  list(APPEND LDFLAGS -fprofile-instr-generate)
+
+  set(profile_instrumentation_flags -fprofile-instr-generate)
+  if(TEST_SUITE_USE_IR_PGO)
+    set(profile_instrumentation_flags -fprofile-generate)
+  endif()
+
+  list(APPEND CFLAGS   ${profile_instrumentation_flags})
+  list(APPEND CXXFLAGS ${profile_instrumentation_flags})
+  list(APPEND LDFLAGS  ${profile_instrumentation_flags})
 else()
   set(TEST_SUITE_PROFILE_GENERATE "False")
 endif()
 
 option(TEST_SUITE_PROFILE_USE
       "Add apropriate -fprofile-instr-use to CFLAGS/CXXFLAGS for each benchmark"
       OFF)
 

litsupport/README.md

- `cmake -DTEST_SUITE_RUN_BENCHMARKS=Off` removes the `run` module, so no
   benchmarks are actually run; this is useful as code size data, compiletime, or
   compilation statistics can be collected anyway.
 - `cmake -DTEST_SUITE_RUN_UNDER=qemu` enable the `run_under` module and will
   prefix all benchmark invocations with the specified command (here: `qemu`).
 - `cmake -DTEST_SUITE_REMOTE_HOST=xxx` enabled the `remote` module that uses
   ssh to run benchmarks on a remote device (assuming shared file systems).
 - `cmake -DTEST_SUITE_PROFILE_GENERATE` compiles benchmark with
   `-fprofile-instr-generate` and enables the `profilegen` module that runs
-  `llvm-profdata` after running the benchmarks.
+  `llvm-profdata` after running the benchmarks. To use LLVM IR PGO instead of
+  the clang frontend's PGO feature, set `-DTEST_SUITE_USE_IR_PGO=On`.
 
 Available modules are found in the `litsupport/modules` directory.
 
 Developing New Modules
 ----------------------
 
 Testing modules consist of a python module with a `mutatePlan(context, plan)`
 function. Modules can: