Add streamexecutor-config

[jhen]

Summary

Similar to llvm-config, gets command-line flags that are needed to build
applications linking against StreamExecutor.

[jlebar]

streamexecutor/tools/streamexecutor-config/streamexecutor-config.in

62 ↗

(On Diff #70558)

Can we scope the try/except around tokens.index?

87 ↗

(On Diff #70558)

Consider description=__doc__ and then just updating this file's docstring.

129 ↗

(On Diff #70558)

Nit, write to stderr (print('foo', file=sys.stderr))? (Actually not quite a nit because of how this program is going to be used -- ideally we wouldn't print anything other than flags to stdout.)

130 ↗

(On Diff #70558)

Exit with an error code? Same elsewhere.

161 ↗

(On Diff #70558)

Should we do something to tokenize the flags? (So that e.g. -Idir with spaces shows up correctly?)

[jhen]

• Respond to jlebar's comments

streamexecutor/tools/streamexecutor-config/streamexecutor-config.in

129 ↗

(On Diff #70558)

Since I'm using sys.exit here now, I'm just passing the error message to sys.exit instead and that will print it to stderr as well as returning error code 1.

161 ↗

(On Diff #70558)

Great catch! I added explicit quotes around @CMAKE_INSTALL_PREFIX@ everywhere it is used. This made it slightly more complicated to check whether a token being added by SE was already present in the LLVM output, so I added a new function has_token to do the token escaping before doing the search.

As for the output of llvm-config, I'll just trust that llvm-config quotes things correctly.

[jlebar]

streamexecutor/tools/streamexecutor-config/streamexecutor-config.in

155 ↗

(On Diff #70579)

How does @CMAKE_INSTALL_PREFIX@ work? That is, who's responsible for replacing it with an actual path? From the example in the docstring, it looks like nobody is?

[jhen]

streamexecutor/tools/streamexecutor-config/streamexecutor-config.in

155 ↗

(On Diff #70579)

This is a cmake built-in variable that is replaced at build time when this file is configured.

The code in this patch that sets up this configuration is the cmake configure_file command in line 2 in streamexecutor/tools/streamexecutor-config/CMakeLists.txt.

[jlebar]

streamexecutor/tools/streamexecutor-config/streamexecutor-config.in

155 ↗

(On Diff #70579)

This is a cmake built-in variable that is replaced at build time when this file is configured.

...wow. Okay.

What happens if the install prefix contains the ' character? Do we just not care?

[jprice]

streamexecutor/tools/streamexecutor-config/streamexecutor-config.in

148 ↗

(On Diff #70579)

I wonder if it's worth automatically picking out the path to llvm-config using @LLVM_BINARY_DIR@ instead of relying on it being on PATH. Makes things slightly easier for the user, and reduces the risk that they have a different LLVM on their PATH to the one that SE was built against.

[jhen]

• ESCAPE_QUOTES for cmake configure_file

streamexecutor/tools/streamexecutor-config/streamexecutor-config.in

155 ↗

(On Diff #70579)

Oh, thanks for reminding me. I added the ESCAPE_QUOTES option to the cmake configure_file command. That should make sure any quotes in the variable are properly escaped.

[jlebar]

OK. The python code '\"' is equivalent to the string containing one character, double-quote. So I think this is good so long as ESCAPE_QUOTES also escapes backslashes as \\ (which we care about because Windows filenames contain backslashes).

[jhen]

• Use LLVM_BINARY_DIR to find llvm-config

[jhen]

In D24302#536251, @jlebar wrote:

OK. The python code '\"' is equivalent to the string containing one character, double-quote. So I think this is good so long as ESCAPE_QUOTES also escapes backslashes as \\ (which we care about because Windows filenames contain backslashes).

It turns out that ESCAPE_QUOTES doesn't do the right thing for Python because it is only meant for C string literals. So, for example, it doesn't escape single quotes. So I removed the ESCAPE_QUOTES option for cmake configure_file and instead wrapped the substitution in triple double quotes. This will still break if the path contains triple double quotes, but I'm willing to let that happen.

Between this change and the change to using LLVM_BINARY_DIRECTORY for the llvm-config path, jlebar, I think you might want to take another quick glance to make sure things still look OK.

streamexecutor/tools/streamexecutor-config/streamexecutor-config.in

149 ↗

(On Diff #70582)

I think that's a good idea. I made that change and added another option to let the user override the llvm-config path if the configuration somehow goes wrong.

[jprice]

Maybe I'm doing something wrong, but the single-quotes wrapping the include/lib paths given by --cppflags and --ldflags are not working for me. The compiler fails to find the StreamExecutor headers and libraries, and warns that the library directory doesn't exist. It's almost as if the quotes are being taken as part of the paths.

If I manually remove those quotes from the streamexecutor-config script then it works fine. It also works if I manually copy the output into the compiler command-line, so maybe there's some weird issue with how the quotes are handled by the shell when substituting command output with $()?

Tried on both OS X and Ubuntu, with a trivial program that just includes StreamExecutor.h.

[jhen]

In D24302#536448, @jprice wrote:

Maybe I'm doing something wrong, but the single-quotes wrapping the include/lib paths given by --cppflags and --ldflags are not working for me. The compiler fails to find the StreamExecutor headers and libraries, and warns that the library directory doesn't exist. It's almost as if the quotes are being taken as part of the paths.

If I manually remove those quotes from the streamexecutor-config script then it works fine. It also works if I manually copy the output into the compiler command-line, so maybe there's some weird issue with how the quotes are handled by the shell when substituting command output with $()?

Tried on both OS X and Ubuntu, with a trivial program that just includes StreamExecutor.h.

I can verify that the quotes are breaking it, and reading through the shell docs shows that the shell doesn't process quotes after command substitution, so I guess the quotes are being passed as literal characters in those arguments. I have an idea of how to fix this, and I should have a patch posted soon.

[jhen]

Well, it turns out that the current llvm-config does not handle directories with special characters. I built llvm with -DCMAKE_INSTALL_PREFIX="$HOME/private/dir 'with' spaces" and installed it there. Then when I ran llvm-config --cppflags I got the following output:

-I/usr/local/google/home/jhen/private/dir 'with' spaces/include   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS

where we can see that the spaces and single quotes have not been escaped.

I think we should mimic the llvm-config behavior because otherwise there will be two separate ways that paths are formatted in the output, and if anyone ever does make a workaround for llvm-config, it won't work for streamexecutor-config.

So I would say that even though it's kind of pathetic, I think we should not try to escape special characters in our output either.

[jhen]

• Remove quotes around CMAKE_INSTALL_PREFIX

[jlebar]

I think we should mimic the llvm-config behavior because otherwise there will be two separate ways that paths are formatted in the output, and if anyone ever does make a workaround for llvm-config, it won't work for streamexecutor-config.

Sounds good to me.

So I would say that even though it's kind of pathetic [...]

You can always put a passive-aggressive comment in our source. :)

parallel-libs/trunk/streamexecutor/CMakeLists.txt

 cmake_minimum_required(VERSION 3.1)
 
 option(STREAM_EXECUTOR_UNIT_TESTS "enable unit tests" ON)
 option(STREAM_EXECUTOR_ENABLE_DOXYGEN "enable StreamExecutor doxygen" ON)
+option(STREAM_EXECUTOR_ENABLE_CONFIG_TOOL "enable building streamexecutor-config tool" ON)
 
 # First find includes relative to the streamexecutor top-level source path.
 include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR}/include)
 
 # If we are not building as part of LLVM, build StreamExecutor as a standalone
 # project using LLVM as an external library:
 string(
     COMPARE

 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
 # Add warning flags.
 set(CMAKE_CXX_FLAGS  "${CMAKE_CXX_FLAGS} -Wall -Wextra -Wno-unused-parameter")
 
 add_subdirectory(lib)
 add_subdirectory(examples)
 
+if(STREAM_EXECUTOR_ENABLE_CONFIG_TOOL )
+    add_subdirectory(tools/streamexecutor-config)
+endif(STREAM_EXECUTOR_ENABLE_CONFIG_TOOL )
+
 install(DIRECTORY include/ DESTINATION include)
 
 if (STREAM_EXECUTOR_ENABLE_DOXYGEN)
     find_package(Doxygen REQUIRED)
     configure_file(Doxyfile.in ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile @ONLY)
     add_custom_target(
         doc
         ${DOXYGEN_EXECUTABLE}
         ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile
         WORKING_DIRECTORY
         ${CMAKE_CURRENT_BINARY_DIR}
         COMMENT
         "Generating API documentation with Doxygen"
         VERBATIM)
 endif(STREAM_EXECUTOR_ENABLE_DOXYGEN)

parallel-libs/trunk/streamexecutor/Doxyfile.in

 
 # If the value of the INPUT tag contains directories, you can use the
 # EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude
 # certain files from those directories.
 #
 # Note that the wildcards are matched against the file with absolute path, so to
 # exclude all test directories for example use the pattern */test/*
 
-EXCLUDE_PATTERNS       = */examples/* */unittests/*
+EXCLUDE_PATTERNS       = */examples/* */tools/* */unittests/*
 
 # The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names
 # (namespaces, classes, functions, etc.) that should be excluded from the
 # output. The symbol name can be a fully qualified name, a word, or if the
 # wildcard * is used, a substring. Examples: ANamespace, AClass,
 # AClass::ANamespace, ANamespace::*Test
 #
 # Note that the wildcards are matched against the file with absolute path, so to

parallel-libs/trunk/streamexecutor/tools/streamexecutor-config/CMakeLists.txt

+find_package(PythonInterp REQUIRED)
+configure_file(streamexecutor-config.in streamexecutor-config)
+install(PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/streamexecutor-config DESTINATION bin)

parallel-libs/trunk/streamexecutor/tools/streamexecutor-config/streamexecutor-config.in

+#!@PYTHON_EXECUTABLE@
+#
+#===- streamexecutor-config - Build config script for SE -----*- python -*--===#
+#
+#                     The LLVM Compiler Infrastructure
+#
+# This file is distributed under the University of Illinois Open Source
+# License. See LICENSE.TXT for details.
+#
+#===------------------------------------------------------------------------===#
+
+r"""
+Get configuration info needed to compile programs which use StreamExecutor.
+
+Runs llvm-config and adds StreamExecutor-specific flags to the output. Supports
+only the subset of llvm-config flags that are relevant for applications
+compiling against StreamExecutor.
+
+This utility will typically be used to construct a compile command line for an
+application which depends on the StreamExecutor library.
+
+For example:
+  c++ example.cpp -o example \
+      $(streamexecutor-config \
+          --cppflags --cxxflags --ldflags --libs --system-libs)
+"""
+
+import argparse
+import errno
+import os
+import shlex
+import subprocess
+import sys
+
+def get_llvm_config_dir():
+  """Gets the path to the llvm-config executable.
+
+  Surrounding the cmake replacement with triple quotes should allow for paths
+  that contain quotes."""
+  return r"""@LLVM_BINARY_DIR@/bin"""
+
+def get_cmake_install_prefix():
+  """Gets the value of the cmake variable CMAKE_INSTALL_PREFIX.
+
+  Surrounding the cmake replacement with triple quotes should allow for paths
+  that contain quotes."""
+  return r"""@CMAKE_INSTALL_PREFIX@"""
+
+def cuddle_flag(flag, tokens):
+  """If flag appears by itself in tokens, combines it with the next token.
+
+  >>> tokens = ['-I', '/usr/include']
+  >>> cuddle_flag('-I', tokens)
+  >>> tokens
+  ['-I/usr/include']
+
+  >>> tokens = ['-L', '/usr/lib']
+  >>> cuddle_flag('-L', tokens)
+  >>> tokens
+  ['-L/usr/lib']
+
+  >>> tokens = ['-I']
+  >>> cuddle_flag('-I', tokens)
+  >>> tokens
+  ['-I']
+
+  >>> tokens = ['-I', '/usr/include', '-I', '/usr/local/include']
+  >>> cuddle_flag('-I', tokens)
+  >>> tokens
+  ['-I/usr/include', '-I/usr/local/include']
+  """
+  start = 0
+  while True:
+    try:
+      index = tokens.index(flag, start)
+    except ValueError:
+      return
+    if index + 1 < len(tokens):
+      follower = tokens.pop(index + 1)
+      tokens[index] = flag + follower
+    start = index + 1
+
+def get_llvm_config_output_for_dir(llvm_config_dir, flags_string):
+  """Calls llvm-config at the given path and returns the output with -I and -L
+  flags cuddled."""
+  output = subprocess.check_output(
+      ['%s/llvm-config' % llvm_config_dir] + flags_string.split()).strip()
+  tokens = shlex.split(output)
+  cuddle_flag('-I', tokens)
+  cuddle_flag('-L', tokens)
+  return ' '.join(tokens)
+
+def has_token(token, string):
+  """Checks if the given token appears in the string.
+
+  The token argument must be a single shell token.
+
+  >>> string = '-I/usr/include -L"/usr/lib"'
+  >>> has_token('-I/usr/include', string)
+  True
+  >>> has_token('-I/usr/local/include', string)
+  False
+  >>> has_token('-I"/usr/include"', string)
+  True
+  >>> has_token('-L"/usr/lib"', string)
+  True
+  >>> has_token('-L/usr/lib', string)
+  True
+  """
+  split_token = shlex.split(token)
+  if len(split_token) > 1:
+    raise ValueError('has_token called with a multi-token token: ' + token)
+  escaped_token = split_token[0]
+  return escaped_token in shlex.split(string)
+
+def main():
+  parser = argparse.ArgumentParser(
+      prog='streamexecutor-config',
+      formatter_class=argparse.RawDescriptionHelpFormatter,
+      description=__doc__)
+
+  parser.add_argument(
+      '--cppflags',
+      action='store_true',
+      help=
+        'C preprocessor flags for files that include StreamExecutor headers.')
+
+  parser.add_argument(
+      '--cxxflags',
+      action='store_true',
+      help='C++ compiler flags for files that include StreamExecutor headers.')
+
+  parser.add_argument(
+      '--ldflags',
+      action='store_true',
+      help='Print linker flags.')
+
+  parser.add_argument(
+      '--libs',
+      action='store_true',
+      help='Libraries needed to link against StreamExecutor.')
+
+  parser.add_argument(
+      '--system-libs',
+      action='store_true',
+      help='System libraries needed to link against StreamExecutor.')
+
+  parser.add_argument(
+      '--llvm-config-dir',
+      default=get_llvm_config_dir(),
+      help='Directory containing the llvm-config executable. '\
+          'If not specified, defaults to the cmake-configured location')
+
+  args = parser.parse_args()
+
+  # Print the help message if the user did not pass any flag arguments.
+  if not any(
+      getattr(args, flag)
+        for flag in ('cppflags', 'cxxflags', 'ldflags', 'libs', 'system_libs')):
+    parser.print_help()
+    sys.exit(1)
+
+  # Check for the presence of the llvm-config executable.
+  if not os.path.isfile('%s/llvm-config' % args.llvm_config_dir):
+    sys.exit('llvm-config not found in: ' + args.llvm_config_dir)
+  if not os.access('%s/llvm-config' % args.llvm_config_dir, os.X_OK):
+    sys.exit('llvm-config not executable in: ' + args.llvm_config_dir)
+
+  # We will always use args.llvm_config_dir as the second argument to
+  # get_llvm_config_output_for_path.
+  get_llvm_config_output = lambda flags : get_llvm_config_output_for_dir(
+      args.llvm_config_dir, flags)
+
+  all_flags = []
+
+  if args.cppflags:
+    llvm_flags = get_llvm_config_output('--cppflags')
+    all_flags.append(llvm_flags)
+    se_flag = "-I%s/include" % get_cmake_install_prefix()
+    if not has_token(token=se_flag, string=llvm_flags):
+      all_flags.append(se_flag)
+
+  if args.cxxflags:
+    all_flags.append(get_llvm_config_output('--cxxflags'))
+
+  if args.ldflags:
+    llvm_flags = get_llvm_config_output('--ldflags')
+    all_flags.append(llvm_flags)
+    se_flag = "-L%s/lib" % get_cmake_install_prefix()
+    if not has_token(token=se_flag, string=llvm_flags):
+      all_flags.append(se_flag)
+
+  if args.libs:
+    llvm_flags = get_llvm_config_output('--libs support symbolize')
+    se_flag = '-lstreamexecutor'
+    if not has_token(token=se_flag, string=llvm_flags):
+      all_flags.append(se_flag)
+    all_flags.append(llvm_flags)
+
+  if args.system_libs:
+    all_flags.append(get_llvm_config_output('--system-libs'))
+
+  print(' '.join(all_flags))
+
+if __name__ == '__main__':
+  main()