[CMake] Add support for generating profdata for clang from training files

[beanz]

Summary

This patch adds support for using LIT to drive generating PGO profile data for clang.

This first pass implementation should work on Linux and Unix based platforms. If you build clang using CMake with LLVM_BUILD_INSTRUMENTED=On the CMake build generates a generate-profdata target that will use the just-built clang to build any test files (see hello_world.cpp as an example). Each test compile will generate profraw files for each clang process. After all tests have run CMake will merge the profraw files using llvm-profdata.

Future opportunities for extension:

• Support for Build->Profile->Build bootstrapping
• Support for linker order file generation using a similar mechanism and the same training data
• Support for Windows

[silvas]

Can you elaborate on why this patch is restricted to unix?

Also, this probably requires some documentation in
http://llvm.org/docs/CMake.html
(we don't really have an analogous page just for clang currently, so the llvm one is probably the best place right now)

utils/perf-training/CMakeLists.txt

21 ↗

(On Diff #42561)

Can you write a tiny pure-Python helper script for clear-profraw and another for generate-profdata? That will be more portable.

If this is the only thing blocking windows support, I think that shooting for windows support on the initial patch is worth a shot.

[vsk]

Thanks! Interesting approach. I think @cmatthews would appreciate a generate-profdata-from-lnt target if you're up for it :).

Comments inline --

utils/perf-training/CMakeLists.txt

2 ↗

(On Diff #42561)

Afaict, lines 2-8 look like they're provided by configure_lit_site_cfg (AddLLVM.cmake).

8 ↗

(On Diff #42561)

Is it possible for this line to interact poorly with the one in clang/test/CMakeLists.txt? E.g something weird like if CMAKE_CFG_INTDIR is a substring of LLVM_BUILD_MODE?

17 ↗

(On Diff #42561)

Can we build llvm-profdata too?

25 ↗

(On Diff #42561)

We should use a llvm-profdata built out of the current source. Does the llvm_find_program function help with this?

33 ↗

(On Diff #42561)

Sean's comment about relying on find applies here too.

utils/perf-training/lit.cfg

7 ↗

(On Diff #42561)

This duplicates code in tools/clang/test/lit.cfg. Can we lift it into a shared module?

utils/perf-training/lit.site.cfg.in

3 ↗

(On Diff #42561)

I don't know too much about what's going on here. It seems weird to check in auto generated files.. why do we need to do that?

[beanz]

Sean,

The reason for restricting to Unix is two fold. (1) the shell script goop, which I can replace with python and (2) I don't have a windows box to test on, so I didn't want people to think it worked.

If I replace the shell goop with Python this will probably "Just Work" everywhere, so I can do that.

I also agree about your documentation point. That will need to be a separate patch.

More comments inline.

utils/perf-training/CMakeLists.txt

2 ↗

(On Diff #42561)

configure_lit_site_cfg doesn't set CLANG_TOOLS_DIR, which needs to be set and uses this code. We should probably clean this all up so we don't need this duplication, but that is a separate problem.

8 ↗

(On Diff #42561)

It won't interact with the other directory because CMake does level-based scoping. The tests are processed before this code, and the directory pops back out to the root before traversing this.

17 ↗

(On Diff #42561)

You actually don't want that. You want your llvm-profdata to match the profile runtime you're building with. Since this builds with the profile runtime coming with the compiler you're using (which is usually the system compiler) you'll want a system profdata tool.

I do plan to support bootstrap PGO generation where if you do a 3-stage build. In that world stage1 will be your toolchain to use to build the instrumented compiler and stage1 would provide llvm-profdata.

21 ↗

(On Diff #42561)

I'll give it a go today.

25 ↗

(On Diff #42561)

No, we really don't want to do that. See my comment above (Note: I had implemented that and it doesn't work).

utils/perf-training/lit.cfg

7 ↗

(On Diff #42561)

This code probably isn't needed. Since it should always use just-built clang, I can probably just use lit.util.which directly instead of this wrapper. I'll make that change.

utils/perf-training/lit.site.cfg.in

3 ↗

(On Diff #42561)

This file is actually the input to the auto-generation. It needs to be checked in. That comment is all over the lit.*.in files so that people know the outputted files are auto-generated.

This probably ins't super important anymore because we don't support in-source builds, but back in the day it was because you'd end up with all these auto-generated lit.site.cfg files all over you source directory.

[beanz]

Updates based on feedback from Sean and Vedant.

• Should work on Windows now
• Cleaned up lit config code

[silvas]

In D15462#309889, @beanz wrote:

Sean,

The reason for restricting to Unix is two fold. (1) the shell script goop, which I can replace with python and (2) I don't have a windows box to test on, so I didn't want people to think it worked.

If I replace the shell goop with Python this will probably "Just Work" everywhere, so I can do that.

The lack of a windows box is totally understandable. Should Work is fine. I'll probably give this a shot on windows some time and we'll catch anything then.

utils/perf-training/perf-helper.py

14 ↗

(On Diff #42761)

typo: Proofraw

28 ↗

(On Diff #42761)

The explicit return isn't needed if you aren't returning anything (but I think you want to return 0).

33 ↗

(On Diff #42761)

You seem to be using the return value of this function (and clean) as though they returned a value, but they don't seem to. Did you mean return 1 or whatever for these return's?

36 ↗

(On Diff #42761)

I think you can use subprocess.call or subprocess.check_call here (they are just sugar API around Popen, but they cover the common cases like this case).

40 ↗

(On Diff #42761)

I don't think you need this variable. Everywhere you set it you could just do sys.exit(<the thing you were setting return_code to>)

44 ↗

(On Diff #42761)

Cute eval, but probably better to just use an explicit dictionary. Personally, I don't think we need the error handling (this script is never invoked by hand in regular duty, I don't think).

Something like this:

def main():
    COMMANDS = {'merge': merge, 'clean': clean}
    f = COMMANDS[sys.argv[1]]
    sys.exit(f(sys.argv[2:]))

That is one of the beauties (IMO) of python. The native safety of the language and "sufficiently readable" behavior when an error occurs (e.g. sys.argv[1] doesn't exist, invalid command is specified, etc.; you will get a stack trace showing the line where the error occurred and a readable error message) means that you can actually go quite far for scripts that are not meant to be explicitly and frequently "human-invoked" as their primary purpose.

I don't think it's worth going overboard for a quick script like this.

[beanz]

Cleaning up the python helper as per Sean's suggestions.

[silvas]

Thanks. This LGTM but I'd wait for Duncan or Justin to sign off on it; they're likely to have more higher-level thoughts.

utils/perf-training/perf-helper.py

36 ↗

(On Diff #42891)

I don't think you need the PIPE arguments (or maybe look into check_output if you want the output)

cfe/trunk/CMakeLists.txt

if(CLANG_BUILT_STANDALONE)
     add_lit_target(check-all
       "Running all regression tests"
       ${LLVM_LIT_TESTSUITES}
       PARAMS ${LLVM_LIT_PARAMS}
       DEPENDS ${LLVM_LIT_DEPENDS}
       ARGS ${LLVM_LIT_EXTRA_ARGS}
       )
   endif()
+  add_subdirectory(utils/perf-training)
 endif()
 
 option(CLANG_INCLUDE_DOCS "Generate build targets for the Clang docs."
   ${LLVM_INCLUDE_DOCS})
 if( CLANG_INCLUDE_DOCS )
   add_subdirectory(docs)
 endif()
 

cfe/trunk/utils/perf-training/CMakeLists.txt

+if(LLVM_BUILD_INSTRUMENTED)
+  if (CMAKE_CFG_INTDIR STREQUAL ".")
+    set(LLVM_BUILD_MODE ".")
+  else ()
+    set(LLVM_BUILD_MODE "%(build_mode)s")
+  endif ()
+
+  string(REPLACE ${CMAKE_CFG_INTDIR} ${LLVM_BUILD_MODE} CLANG_TOOLS_DIR ${LLVM_RUNTIME_OUTPUT_INTDIR})
+
+  configure_lit_site_cfg(
+    ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
+    ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
+    )
+
+  add_lit_testsuite(generate-profraw "Generating clang PGO data"
+    ${CMAKE_CURRENT_BINARY_DIR}
+    DEPENDS clang clear-profraw
+    )
+
+  add_custom_target(clear-profraw
+    COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/perf-helper.py clean ${CMAKE_CURRENT_BINARY_DIR}
+    COMMENT "Clearing old profraw data")
+
+  if(NOT LLVM_PROFDATA)
+    find_program(LLVM_PROFDATA llvm-profdata)
+  endif()
+
+  if(NOT LLVM_PROFDATA)
+    message(FATAL_ERROR "Must set LLVM_PROFDATA to point to llvm-profdata to use for merging PGO data")
+  endif()
+
+  add_custom_target(generate-profdata
+    COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/perf-helper.py merge ${LLVM_PROFDATA} ${CMAKE_CURRENT_BINARY_DIR}/clang.profdata ${CMAKE_CURRENT_BINARY_DIR}
+    COMMENT "Merging profdata"
+    DEPENDS generate-profraw)
+endif()

cfe/trunk/utils/perf-training/README.txt

+==========================
+ Performance Training Data
+==========================
+
+This directory contains simple source files for use as training data for
+generating PGO data and linker order files for clang.

cfe/trunk/utils/perf-training/cxx/hello_world.cpp

+// RUN: %clang_cpp -c %s
+#include <iostream>
+
+int main(int, char**) {
+  std::cout << "Hello, World!";
+  return 0;
+}

cfe/trunk/utils/perf-training/lit.cfg

+# -*- Python -*-
+
+from lit import Test
+import lit.formats
+import lit.util
+
+def getSysrootFlagsOnDarwin(config, lit_config):
+    # On Darwin, support relocatable SDKs by providing Clang with a
+    # default system root path.
+    if 'darwin' in config.target_triple:
+        try:
+            out = lit.util.capture(['xcrun', '--show-sdk-path']).strip()
+            res = 0
+        except OSError:
+            res = -1
+        if res == 0 and out:
+            sdk_path = out
+            lit_config.note('using SDKROOT: %r' % sdk_path)
+            return '-isysroot %s' % sdk_path
+    return ''
+
+sysroot_flags = getSysrootFlagsOnDarwin(config, lit_config)
+
+config.clang = lit.util.which('clang', config.clang_tools_dir).replace('\\', '/')
+
+config.name = 'Clang Perf Training'
+config.suffixes = ['.c', '.cpp', '.m', '.mm', '.cu', '.ll', '.cl', '.s', '.S', '.modulemap']
+
+use_lit_shell = os.environ.get("LIT_USE_INTERNAL_SHELL")
+config.test_format = lit.formats.ShTest(use_lit_shell == "0")
+config.substitutions.append( ('%clang_cpp', ' %s --driver-mode=cpp %s ' % (config.clang, sysroot_flags)))
+config.substitutions.append( ('%clang', ' %s %s ' % (config.clang, sysroot_flags) ) )
+
+config.environment['LLVM_PROFILE_FILE'] = 'perf-training-%p.profraw'
+

cfe/trunk/utils/perf-training/lit.site.cfg.in

+import sys
+
+## Autogenerated by LLVM/Clang configuration.
+# Do not edit!
+config.clang_tools_dir = "@CLANG_TOOLS_DIR@"
+config.test_exec_root = "@CMAKE_CURRENT_BINARY_DIR@"
+config.test_source_root = "@CMAKE_CURRENT_SOURCE_DIR@"
+config.target_triple = "@TARGET_TRIPLE@"
+
+# Support substitution of the tools and libs dirs with user parameters. This is
+# used when we can't determine the tool dir at configuration time.
+try:
+    config.clang_tools_dir = config.clang_tools_dir % lit_config.params
+except KeyError:
+    e = sys.exc_info()[1]
+    key, = e.args
+    lit_config.fatal("unable to find %r parameter, use '--param=%s=VALUE'" % (key,key))
+
+# Let the main config do the real work.
+lit_config.load_config(config, "@CLANG_SOURCE_DIR@/utils/perf-training/lit.cfg")

cfe/trunk/utils/perf-training/perf-helper.py

+#===- perf-helper.py - Clang Python Bindings -----------------*- python -*--===#
+#
+#                     The LLVM Compiler Infrastructure
+#
+# This file is distributed under the University of Illinois Open Source
+# License. See LICENSE.TXT for details.
+#
+#===------------------------------------------------------------------------===#
+
+import sys
+import os
+import subprocess
+
+def findProfrawFiles(path):
+  profraw_files = []
+  for root, dirs, files in os.walk(path): 
+    for filename in files:
+      if filename.endswith(".profraw"):
+        profraw_files.append(os.path.join(root, filename))
+  return profraw_files
+
+def clean(args):
+  if len(args) != 1:
+    print 'Usage: %s clean <path>\n\tRemoves all *.profraw files from <path>.' % __file__
+    return 1
+  for profraw in findProfrawFiles(args[0]):
+    os.remove(profraw)
+  return 0
+
+def merge(args):
+  if len(args) != 3:
+    print 'Usage: %s clean <llvm-profdata> <output> <path>\n\tMerges all profraw files from path into output.' % __file__
+    return 1
+  cmd = [args[0], 'merge', '-o', args[1]]
+  cmd.extend(findProfrawFiles(args[2]))
+  subprocess.check_call(cmd)
+  return 0
+
+commands = {'clean' : clean, 'merge' : merge}
+
+def main():
+  f = commands[sys.argv[1]]
+  sys.exit(f(sys.argv[2:]))
+
+if __name__ == '__main__':
+  main()