Replace gen_dynamic_list.py with a portable shell script

[krytarowski]

Summary

Reimplement the Python script with a shell one, as
Python is not available in all environments.

This is especially needed in the integration process
of compiler-rt sanitizers into the NetBSD distribution,
where Python is absent and must not be used as a tool.

New script is also designed to be applicable for
cross-building to a different CPU architecture.
It checks the target CPU with file(1) rather than
relying on host's uname value.

No functional change for existing users. The shell
script generates exactly the same result as the
original Python file for all the .syms files on NetBSD.

[mgorny]

Also, note that technically all commands in shell may fail, and especially commands operating on external files, so you may consider adding some more error handling. You may get very weird results e.g. if you ran out of space in the middle of the script.

lib/sanitizer_common/scripts/gen_dynamic_list.sh

19

You still have .py here.

33

You can fold the command inside, i.e. if ! $my_nm ...; then

34

Why are you using printf when the same result could be achieved by simpler echo?

42

By convention, global variables in shell script should be uppercase.

45

You probably want to respect $TMPDIR here. Also, $(...) is strongly preferred over backticks which are unreadable and non-nesting friendly. Quote the filenames, in case TMPDIR contained spaces.

And, well, speaking of portability mktemp(1) is not POSIX.

48

Quote the file paths inside too.

50

Why not

: "${NM:=nm}"

?

61

Don't use the x hack, it is really, really obsolete and even autotools discourage that by now.

63

echo

Also, don't you want to exit 1 or like on error?

177

I think the standard awk way would be to use ~ regex match here.

199

Quote file paths here and below.

Why are you passing $tmpfile2 twice?

[krytarowski]

Also, note that technically all commands in shell may fail, and especially commands operating on external files, so you may consider adding some more error handling. You may get very weird results e.g. if you ran out of space in the middle of the script.

Is it fine to go here for set -e?

lib/sanitizer_common/scripts/gen_dynamic_list.sh

33

Right, but it's easier for debugging this code with unfolded commands.

34

No reason, I can switch to echo.

42

Style in LLVM? It depends whom you ask. Uppercase symbols are for global variables pulled from environment. If this is the style of LLVM, I will align to it.

45

Right with $TMPDIR.

Is there a better replacement for mktemp(1)? It's available in all modern BSDs, Darwin, GNU and commercial Unices (Solaris, HP/UX). The API behind the command is POSIX though (mktemp(3)).

And if I would need to invent mktemp(1) with a handwritten random number generator, it's tricky to get such number as well.

backticks vs $() is a matter of taste here, if LLVM recommends one or the other, I will switch it (or keep it).

61

It's not a hack, but portability idiom. There might be issues with evaluating empty strings "" or using -z.

63

Right, missing exit.

177

For some reason ^_interceptor_ didn't work as expected and there is need to skip namespaced (and mangled) symbols. We always know the length/position here so it's a straightforward solution.

199

There are two passes NR==FNR { to catch the first one and the other branch for the 2nd one.

Do we support building in paths with spaces in names of directories? It doesn't cost much to add quotes so I will add them.

[krytarowski]

Does this work and produce exactly the same result on Linux (.syms content) with the original .py script and the .sh version?

lib/sanitizer_common/scripts/gen_dynamic_list.sh

50

:= would be needed if nm could be expanded to something, we probably don't need it property.

[krytarowski]

lib/sanitizer_common/scripts/gen_dynamic_list.sh

19

It probably overly closely emulated the original version.. I will change it.

[compnerd]

Shell is not available on Windows, python however is considered a portable shell within the project. There are a number of tools within the LLVM project that require python, I'm not sure that removing the dependency is the right thing to do here.

[mgorny]

Alternatively, I think this could be implemented in CMake script.

[krytarowski]

In D56180#1343094, @compnerd wrote:

Shell is not available on Windows, python however is considered a portable shell within the project. There are a number of tools within the LLVM project that require python, I'm not sure that removing the dependency is the right thing to do here.

It's not portable enough to be kept as a build dependency (it's fine for tests and auxiliary infrastructure). I need to replace it with something else.

Python is not a base of any BSD and every project integrating gen_dynamic_list will need to reinvent/replace this Python script. It's the current case with NetBSD.

As an alternative to shell, I can write it fully with awk(1) (+ nm(1), file(1)) but it will probably face the same concern as sh(1).

In D56180#1343123, @mgorny wrote:

Alternatively, I think this could be implemented in CMake script.

In the end I'm replacing CMake with BSD Makefiles too. I'm aware that CMake isn't used by at least RTEMS too.
Not sure if I can express this script with CMake rules (at least few years ago it used to be at least impracticably difficult to script with CMake, if possible at all).

A hybrid solution of keeping this .sh file around in upstream repo is fine too.

I can write a C program that is built as a tool and generates these .syms files (and calls with system(3) (C89 call) and assume that there is available nm(1) and file(1)).

If there is no interest upstream I can keep the shell script downstream.

[mgorny]

Hmm, a C++ program using LLVMSupport, alike tablegen, would probably work for everyone, wouldn't it?

[krytarowski]

Using LLVM libraries is fine, a little bit of dependency as we need to get LLVM libraries/headers to craft tools.

We have currently a similar issue with tablegen used as a tool.

I will keep using downstream this .sh for now. If someone intends to pick this .sh (and even integrate with LLVM) feel free to do so.

cmake/Modules/SanitizerUtils.cmake

 include(CompilerRTUtils)
 
 set(SANITIZER_GEN_DYNAMIC_LIST
-  ${COMPILER_RT_SOURCE_DIR}/lib/sanitizer_common/scripts/gen_dynamic_list.py)
+  ${COMPILER_RT_SOURCE_DIR}/lib/sanitizer_common/scripts/gen_dynamic_list.sh)
 
 set(SANITIZER_LINT_SCRIPT
   ${COMPILER_RT_SOURCE_DIR}/lib/sanitizer_common/scripts/check_lint.sh)
 
 # Create a target "<name>-<arch>-symbols" that would generate the list of
 # symbols that need to be exported from sanitizer runtime "<name>". Function
 # interceptors are exported automatically, user can also provide files with
 # symbol names that should be exported as well.
macro(add_sanitizer_rt_symbols name)
   foreach(arch ${ARG_ARCHS})
     set(target_name ${name}-${arch})
     set(stamp ${CMAKE_CURRENT_BINARY_DIR}/${target_name}.syms-stamp)
     set(extra_args)
     foreach(arg ${ARG_EXTRA})
       list(APPEND extra_args "--extra" ${arg})
     endforeach()
     add_custom_command(OUTPUT ${stamp}
-      COMMAND ${PYTHON_EXECUTABLE}
+      COMMAND
         ${SANITIZER_GEN_DYNAMIC_LIST} ${extra_args} $<TARGET_FILE:${target_name}>
         > $<TARGET_FILE:${target_name}>.syms
       COMMAND ${CMAKE_COMMAND} -E touch ${stamp}
       DEPENDS ${target_name} ${SANITIZER_GEN_DYNAMIC_LIST} ${ARG_EXTRA}
       WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
       COMMENT "Generating exported symbols for ${target_name}"
       VERBATIM)
     add_custom_target(${target_name}-symbols ALL
macro(add_sanitizer_rt_version_list name)
   set(args)
   foreach(arg ${ARG_EXTRA})
     list(APPEND args "--extra" ${arg})
   endforeach()
   foreach(arg ${ARG_LIBS})
     list(APPEND args "$<TARGET_FILE:${arg}>")
   endforeach()
   add_custom_command(OUTPUT ${vers}
-    COMMAND ${PYTHON_EXECUTABLE}
+    COMMAND
       ${SANITIZER_GEN_DYNAMIC_LIST} --version-list ${args}
       > ${vers}
     DEPENDS ${SANITIZER_GEN_DYNAMIC_LIST} ${ARG_EXTRA} ${ARG_LIBS}
     WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
     COMMENT "Generating version list for ${name}"
     VERBATIM)
 
   add_custom_target(${name}-version-list ALL

lib/interception/interception.h

# define DEFINE_REAL(ret_type, func, ...) \
     }
 #else
 # define DEFINE_REAL(ret_type, func, ...)
 #endif
 
 #if SANITIZER_FUCHSIA
 
 // We need to define the __interceptor_func name just to get
-// sanitizer_common/scripts/gen_dynamic_list.py to export func.
+// sanitizer_common/scripts/gen_dynamic_list.sh to export func.
 // But we don't need to export __interceptor_func to get that.
 #define INTERCEPTOR(ret_type, func, ...)                                \
   extern "C"[[ gnu::alias(#func), gnu::visibility("hidden") ]] ret_type \
       __interceptor_##func(__VA_ARGS__);                                \
   extern "C" INTERCEPTOR_ATTRIBUTE ret_type func(__VA_ARGS__)
 
 #elif !SANITIZER_MAC
 

lib/sanitizer_common/sanitizer_common_interceptors.inc

 // We need a hidden symbol aliasing the above, so that we can jump
 // directly to it from the assembly below.
 extern "C" __attribute__((alias("__interceptor___tls_get_addr_internal"),
                           visibility("hidden")))
 uptr __tls_get_addr_hidden(void *arg);
 // Now carefully intercept __tls_get_offset.
 asm(
   ".text\n"
-// The __intercept_ version has to exist, so that gen_dynamic_list.py
+// The __intercept_ version has to exist, so that gen_dynamic_list.sh
 // exports our symbol.
   ".weak __tls_get_offset\n"
   ".type __tls_get_offset, @function\n"
   "__tls_get_offset:\n"
   ".global __interceptor___tls_get_offset\n"
   ".type __interceptor___tls_get_offset, @function\n"
   "__interceptor___tls_get_offset:\n"
 #ifdef __s390x__

lib/sanitizer_common/scripts/gen_dynamic_list.py

-#!/usr/bin/env python
-#===- lib/sanitizer_common/scripts/gen_dynamic_list.py ---------------------===#
-#
-#                     The LLVM Compiler Infrastructure
-#
-# This file is distributed under the University of Illinois Open Source
-# License. See LICENSE.TXT for details.
-#
-#===------------------------------------------------------------------------===#
-#
-# Generates the list of functions that should be exported from sanitizer
-# runtimes. The output format is recognized by --dynamic-list linker option.
-# Usage:
-#   gen_dynamic_list.py libclang_rt.*san*.a [ files ... ]
-#
-#===------------------------------------------------------------------------===#
-import argparse
-import os
-import re
-import subprocess
-import sys
-import platform
-
-new_delete = set([
-                  '_Znam', '_ZnamRKSt9nothrow_t',    # operator new[](unsigned long)
-                  '_Znwm', '_ZnwmRKSt9nothrow_t',    # operator new(unsigned long)
-                  '_Znaj', '_ZnajRKSt9nothrow_t',    # operator new[](unsigned int)
-                  '_Znwj', '_ZnwjRKSt9nothrow_t',    # operator new(unsigned int)
-                  # operator new(unsigned long, std::align_val_t)
-                  '_ZnwmSt11align_val_t', '_ZnwmSt11align_val_tRKSt9nothrow_t',
-                  # operator new(unsigned int, std::align_val_t)
-                  '_ZnwjSt11align_val_t', '_ZnwjSt11align_val_tRKSt9nothrow_t',
-                  # operator new[](unsigned long, std::align_val_t)
-                  '_ZnamSt11align_val_t', '_ZnamSt11align_val_tRKSt9nothrow_t',
-                  # operator new[](unsigned int, std::align_val_t)
-                  '_ZnajSt11align_val_t', '_ZnajSt11align_val_tRKSt9nothrow_t',
-                  '_ZdaPv', '_ZdaPvRKSt9nothrow_t',  # operator delete[](void *)
-                  '_ZdlPv', '_ZdlPvRKSt9nothrow_t',  # operator delete(void *)
-                  '_ZdaPvm',                         # operator delete[](void*, unsigned long)
-                  '_ZdlPvm',                         # operator delete(void*, unsigned long)
-                  '_ZdaPvj',                         # operator delete[](void*, unsigned int)
-                  '_ZdlPvj',                         # operator delete(void*, unsigned int)
-                  # operator delete(void*, std::align_val_t)
-                  '_ZdlPvSt11align_val_t', '_ZdlPvSt11align_val_tRKSt9nothrow_t',
-                  # operator delete[](void*, std::align_val_t)
-                  '_ZdaPvSt11align_val_t', '_ZdaPvSt11align_val_tRKSt9nothrow_t',
-                  # operator delete(void*, unsigned long,  std::align_val_t)
-                  '_ZdlPvmSt11align_val_t',
-                  # operator delete[](void*, unsigned long, std::align_val_t)
-                  '_ZdaPvmSt11align_val_t',
-                  # operator delete(void*, unsigned int,  std::align_val_t)
-                  '_ZdlPvjSt11align_val_t',
-                  # operator delete[](void*, unsigned int, std::align_val_t)
-                  '_ZdaPvjSt11align_val_t',
-                  ])
-
-versioned_functions = set(['memcpy', 'pthread_attr_getaffinity_np',
-                           'pthread_cond_broadcast',
-                           'pthread_cond_destroy', 'pthread_cond_init',
-                           'pthread_cond_signal', 'pthread_cond_timedwait',
-                           'pthread_cond_wait', 'realpath',
-                           'sched_getaffinity'])
-
-def get_global_functions(library):
-  functions = []
-  nm = os.environ.get('NM', 'nm')
-  nm_proc = subprocess.Popen([nm, library], stdout=subprocess.PIPE,
-                             stderr=subprocess.PIPE)
-  nm_out = nm_proc.communicate()[0].decode().split('\n')
-  if nm_proc.returncode != 0:
-    raise subprocess.CalledProcessError(nm_proc.returncode, nm)
-  func_symbols = ['T', 'W']
-  # On PowerPC, nm prints function descriptors from .data section.
-  if platform.uname()[4] in ["powerpc", "ppc64"]:
-    func_symbols += ['D']
-  for line in nm_out:
-    cols = line.split(' ')
-    if len(cols) == 3 and cols[1] in func_symbols :
-      functions.append(cols[2])
-  return functions
-
-def main(argv):
-  parser = argparse.ArgumentParser()
-  parser.add_argument('--version-list', action='store_true')
-  parser.add_argument('--extra', default=[], action='append')
-  parser.add_argument('libraries', default=[], nargs='+')
-  args = parser.parse_args()
-
-  result = []
-
-  all_functions = []
-  for library in args.libraries:
-    all_functions.extend(get_global_functions(library))
-  function_set = set(all_functions)
-  for func in all_functions:
-    # Export new/delete operators.
-    if func in new_delete:
-      result.append(func)
-      continue
-    # Export interceptors.
-    match = re.match('__interceptor_(.*)', func)
-    if match:
-      result.append(func)
-      # We have to avoid exporting the interceptors for versioned library
-      # functions due to gold internal error.
-      orig_name = match.group(1)
-      if orig_name in function_set and (args.version_list or orig_name not in versioned_functions):
-        result.append(orig_name)
-      continue
-    # Export sanitizer interface functions.
-    if re.match('__sanitizer_(.*)', func):
-      result.append(func)
-
-  # Additional exported functions from files.
-  for fname in args.extra:
-    f = open(fname, 'r')
-    for line in f:
-      result.append(line.rstrip())
-  # Print the resulting list in the format recognized by ld.
-  print('{')
-  if args.version_list:
-    print('global:')
-  result.sort()
-  for f in result:
-    print(u'  %s;' % f)
-  if args.version_list:
-    print('local:')
-    print('  *;')
-  print('};')
-
-if __name__ == '__main__':
-  main(sys.argv)

lib/sanitizer_common/scripts/gen_dynamic_list.sh

+#!/bin/sh
+#===- lib/sanitizer_common/scripts/gen_dynamic_list.sh ---------------------===#
+#
+#                     The LLVM Compiler Infrastructure
+#
+# This file is distributed under the University of Illinois Open Source
+# License. See LICENSE.TXT for details.
+#
+#===------------------------------------------------------------------------===#
+#
+# Generates the list of functions that should be exported from sanitizer
+# runtimes. The output format is recognized by --dynamic-list linker option.
+# Usage:
+#   gen_dynamic_list.sh libclang_rt.*san*.a [ files ... ]
+#
+#===------------------------------------------------------------------------===#
+
+usage() {
+  echo "usage: gen_dynamic_list.py [-h] [--version-list] [--extra EXTRA]"
+  echo "                           libraries [libraries ...]"
+}
+
+get_global_functions() {
+  # On PowerPC, nm prints function descriptors from .data section.
+  $my_file $1 | grep -q "PowerPC"
+  if [ $? -eq 0 ]; then
+    export func_symbols="T|W|D"
+  else
+    export func_symbols="T|W"
+  fi
+
+  $my_nm $1 > $tmpfile1
+  if [ $? -ne 0 ]; then
+    printf 'Error calling: %s %s... bailing out\n' "$my_nm" "$1"
+    exit 1
+  fi
+
+  $my_awk '{if(NF == 3 && $2 ~ ENVIRON["func_symbols"]){print $3}}' \
+    $tmpfile1 >> $tmpfile2
+}
+
+version_list=
+extra=
+
+tmpfile1=`mktemp /tmp/${0##*/}.XXXXXX` || exit 1
+tmpfile2=`mktemp /tmp/${0##*/}.XXXXXX` || exit 1
+
+trap "rm -f $tmpfile1 $tmpfile2" EXIT
+
+my_nm=${NM:-nm}
+my_awk=${AWK:-awk}
+my_file=${FILE:-file}
+
+while :; do
+  case $1 in
+    -h|--help)
+      usage
+      exit
+      ;;
+    --extra)
+      if [ "x$2" = "x" ]; then
+        usage
+        printf 'Error missing argument for the option\n'
+      else
+        extra=$2
+        shift
+      fi
+      ;;
+    --extra=?*)
+      extra=${1#*=}
+      ;;
+    --extra=)
+      usage
+      printf 'Error missing argument: %s\n' "$1"
+      ;;
+    --version-list)
+      version_list=1
+      ;;
+    --)
+      shift
+      break
+      ;;
+    -?*)
+      usage
+      printf 'Error parsing argument: %s\n' "$1"
+      ;;
+    *)
+      if [ "x$1" = "x" ]; then
+        break
+      else
+        get_global_functions "$1"
+      fi
+      ;;
+  esac
+  shift
+done
+
+export version_list
+
+$my_awk "
+BEGIN {
+  # operator new[](unsigned long)
+  new_delete[\"_Znam\"] = 1
+  new_delete[\"_ZnamRKSt9nothrow_t\"] = 1
+  # operator new(unsigned long)
+  new_delete[\"_Znwm\"] = 1
+  new_delete[\"_ZnwmRKSt9nothrow_t\"] = 1
+  # operator new[](unsigned int)
+  new_delete[\"_Znaj\"] = 1
+  new_delete[\"_ZnajRKSt9nothrow_t\"] = 1
+  # operator new(unsigned int)
+  new_delete[\"_Znwj\"] = 1
+  new_delete[\"_ZnwjRKSt9nothrow_t\"] = 1
+  # operator new(unsigned long, std::align_val_t)
+  new_delete[\"_ZnwmSt11align_val_t\"] = 1
+  new_delete[\"_ZnwmSt11align_val_tRKSt9nothrow_t\"] = 1
+  # operator new(unsigned int, std::align_val_t)
+  new_delete[\"_ZnwjSt11align_val_t\"] = 1
+  new_delete[\"_ZnwjSt11align_val_tRKSt9nothrow_t\"] = 1
+  # operator new[](unsigned long, std::align_val_t)
+  new_delete[\"_ZnamSt11align_val_t\"] = 1
+  new_delete[\"_ZnamSt11align_val_tRKSt9nothrow_t\"] = 1
+  # operator new[](unsigned int, std::align_val_t)
+  new_delete[\"_ZnajSt11align_val_t\"] = 1
+  new_delete[\"_ZnajSt11align_val_tRKSt9nothrow_t\"] = 1
+  # operator delete[](void *)
+  new_delete[\"_ZdaPv\"] = 1
+  new_delete[\"_ZdaPvRKSt9nothrow_t\"] = 1
+  # operator delete(void *)
+  new_delete[\"_ZdlPv\"] = 1
+  new_delete[\"_ZdlPvRKSt9nothrow_t\"] = 1
+  # operator delete[](void*, unsigned long)
+  new_delete[\"_ZdaPvm\"] = 1
+  # operator delete(void*, unsigned long)
+  new_delete[\"_ZdlPvm\"] = 1
+  # operator delete[](void*, unsigned int)
+  new_delete[\"_ZdaPvj\"] = 1
+  # operator delete(void*, unsigned int)
+  new_delete[\"_ZdlPvj\"] = 1
+  # operator delete(void*, std::align_val_t)
+  new_delete[\"_ZdlPvSt11align_val_t\"] = 1
+  new_delete[\"_ZdlPvSt11align_val_tRKSt9nothrow_t\"] = 1
+  # operator delete[](void*, std::align_val_t)
+  new_delete[\"_ZdaPvSt11align_val_t\"] = 1
+  new_delete[\"_ZdaPvSt11align_val_tRKSt9nothrow_t\"] = 1
+  # operator delete(void*, unsigned long,  std::align_val_t)
+  new_delete[\"_ZdlPvmSt11align_val_t\"] = 1
+  # operator delete[](void*, unsigned long, std::align_val_t)
+  new_delete[\"_ZdaPvmSt11align_val_t\"] = 1
+  # operator delete(void*, unsigned int,  std::align_val_t)
+  new_delete[\"_ZdlPvjSt11align_val_t\"] = 1
+  # operator delete[](void*, unsigned int, std::align_val_t)
+  new_delete[\"_ZdaPvjSt11align_val_t\"] = 1
+
+  versioned_functions[\"memcpy\"] = 1
+  versioned_functions[\"pthread_attr_getaffinity_np\"] = 1
+  versioned_functions[\"pthread_cond_broadcast\"] = 1
+  versioned_functions[\"pthread_cond_destroy\"] = 1
+  versioned_functions[\"pthread_cond_init\"] = 1
+  versioned_functions[\"pthread_cond_signal\"] = 1
+  versioned_functions[\"pthread_cond_timedwait\"] = 1
+  versioned_functions[\"pthread_cond_wait\"] = 1
+  versioned_functions[\"realpath\"] = 1
+  versioned_functions[\"sched_getaffinity\"] = 1
+
+  len = 0
+}
+NR==FNR {
+  function_set[\$0] = 1
+  next
+}
+{
+  if (\$0 in new_delete) {
+    result[len++] = \$0
+    next
+  }
+  if (substr(\$0, 1, 14) == \"__interceptor_\") {
+    result[len++] = \$0
+    # We have to avoid exporting the interceptors for versioned library
+    # functions due to gold internal error.
+    orig_name = substr(\$0, 15)
+    if ((orig_name in function_set) &&
+        (ENVIRON["version_list"] == "1" ||
+        !(orig_name in versioned_functions))) {
+      result[len++] = orig_name
+    }
+    next
+  }
+  if (substr(\$0, 1, 12) == \"__sanitizer_\") {
+    result[len++] = \$0
+    next
+  }
+}
+END {
+  for (i in result) {
+    print result[i]
+  }
+}
+" $tmpfile2 $tmpfile2 > $tmpfile1
+
+if [ "x$extra" != "x" ]; then
+  cat $extra >> $tmpfile1
+fi
+
+sort $tmpfile1 -o $tmpfile2
+
+echo "{"
+if [ "x$version_list" != "x" ]; then
+  echo "global:"
+fi
+$my_awk '{print "  " $0 ";"}' $tmpfile2
+if [ "x$version_list" != "x" ]; then
+  echo "local:"
+  echo "  *;"
+fi
+echo "};"