Use response file when generating LLVM-C.dll

[Wallbraker]

Summary

As discovered in D56774 the command line gets to long, so use a response file to give the script the libs. This change has been tested and is confirmed working for me.

Cheers, Jakob.

[serge-sans-paille]

tools/llvm-shlib/CMakeLists.txt

140

You could do the merge in the end to keep the formating stuff in a single place:

string(REPLACE ";" "\n" FILE_CONTENT "${FULL_LIB_NAMES}")

147

filename missing here

tools/llvm-shlib/gen-msvc-exports.py

87

bad indent.

[serge-sans-paille]

tools/llvm-shlib/gen-msvc-exports.py

107

useless parenthesis spotted.

You should probably add the libs from the file to the one from the CLI instead of (siltently) erasing the one from CLI.

[hans]

I actually tried the patch and got

CMake Error at tools/llvm-shlib/CMakeLists.txt:144 (file):
  file failed to open for writing (Invalid argument):

    C:/src/llvm/build.shlib/$(Configuration)/lib/LLVMDemangle.lib

since the whole purpose is to create llvm-c.dll for Windows, I image you have a Windows box somewhere to test this with :-)

[Wallbraker]

Updated changes with comments from @serge-sans-paille, thanks!

[Wallbraker]

In D56781#1359729, @hans wrote:

I actually tried the patch and got

Thanks for doing that and your patience!

CMake Error at tools/llvm-shlib/CMakeLists.txt:144 (file):
  file failed to open for writing (Invalid argument):

    C:/src/llvm/build.shlib/$(Configuration)/lib/LLVMDemangle.lib

Well that error message is because of me missing the file argument, now I am quite worried about $(Configuration) tho where is that coming from? :(

since the whole purpose is to create llvm-c.dll for Windows, I image you have a Windows box somewhere to test this with :-)

Ugh, I guess I have to set a windows box up. I'm doing that so that I and any of the users of my projects that uses LLVM don't have to go through the quite painful procedure of building LLVM on Windows. But yeah I was hoping nothing had broken since I last tested this.

[Wallbraker]

Not tested, will address last comment and then setup something I can test this on.

tools/llvm-shlib/gen-msvc-exports.py

107

I am unashamedly hard cargo-culting all cmake and python code here as I have almost zero proficiency in both. (Insert i_have_no_idea_what_i_am_doing.jpg )

Ok will change soon

[Wallbraker]

Should address all of the comments, haven't done anything about the $(Configuration) @hans saw in the error message. Will setup something to test this.

[Wallbraker]

And remove the parentheses.

[Wallbraker]

Have now tested this patch and confirmed that it is working for me at least. Needed to make the libs argument optional or the python script would error out and correct splitLines to splitlines. I did not see any sign of $(Configuration) in my scripts.

[Wallbraker]

@hans @serge-sans-paille So I built the package and has verified that the LLVM-C.dll is included in the package but I could not find the LLVM-C.lib file tho. After some hours of digging through the multitude of *.cmake files I have to admit defeat on how to get that in there, any pointers would be very much welcome.

[serge-sans-paille]

tools/llvm-shlib/CMakeLists.txt

149

if FILE_CONTENT contains space, they are going to be suppressed by file(...) unless you quote the variable.

tools/llvm-shlib/gen-msvc-exports.py

110

avoid creating an extra copy using libs.extend(f.read)...)

[Wallbraker]

Thanks, this should fix all concerns raised, I tested this as well and seems to work.

Still need help with shipping the LLVM-C.lib in the windows installer, while that isn't super necessary it will help people using it.

[serge-sans-paille]

LGTM, thanks

[serge-sans-paille]

@Wallbraker if you don't have commit rights I can commit this one for you.

[Wallbraker]

@serge-sans-paille I do not, so that would be greatly appreciated.

Cheers, Jakob.

[cloudhan]

I encounter similar issue, a weird $(Configuration) appears in my build folder. The source is clone from https://github.com/llvm/llvm-project

Sorry for the Chinese error message. But all you should be aware are "警告" is "Warning", "错误" is "Error" and "失败" is "Failed"

    28>PrepareForBuild:
         正在创建目录“x64\RelWithDebInfo\LLVM-C_exports\”。
         正在创建目录“x64\RelWithDebInfo\LLVM-C_exports\LLVM-C_exports.tlog\”。
       InitializeBuildStatus:
         正在创建“x64\RelWithDebInfo\LLVM-C_exports\LLVM-C_exports.tlog\unsuccessfulbuild”，因为已指定“AlwaysCreate”。
       CustomBuild:
         Generating export list for LLVM-C
         Traceback (most recent call last):
           File "D:/workspaces/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 116, in <module>
             main()
           File "D:/workspaces/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 109, in main
             with open(ns.libsfile) as f:
         FileNotFoundError: [Errno 2] No such file or directory: 'D:/workspaces/llvm-project/build/llvm/RelWithDebInfo/
         libllvm-c.args'
    28>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targe
       ts(209,5): error MSB6006: “cmd.exe”已退出，代码为 1。 [D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C_exp
       orts.vcxproj]
    28>已完成生成项目“D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C_exports.vcxproj”(默认目标)的操作 - 失败。
     8>已完成生成项目“D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C.vcxproj”(默认目标)的操作 - 失败。
     1>已完成生成项目“D:\workspaces\llvm-project\build\llvm\ALL_BUILD.vcxproj”(默认目标)的操作 - 失败。
生成失败。

       “D:\workspaces\llvm-project\build\llvm\ALL_BUILD.vcxproj”(默认目标) (1) ->
       “D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C.vcxproj”(默认目标) (8) ->
       “D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C_exports.vcxproj”(默认目标) (28) ->
       “D:\workspaces\llvm-project\build\llvm\lib\DebugInfo\DWARF\LLVMDebugInfoDWARF.vcxproj”(默认目标) (56) ->
       (ClCompile 目标) ->
         D:\workspaces\llvm-project\llvm\lib\DebugInfo\DWARF\DWARFVerifier.cpp(834): warning C4819: The file contains a
        character that cannot be represented in the current code page (936). Save the file in Unicode format to prevent
        data loss [D:\workspaces\llvm-project\build\llvm\lib\DebugInfo\DWARF\LLVMDebugInfoDWARF.vcxproj]


       “D:\workspaces\llvm-project\build\llvm\ALL_BUILD.vcxproj”(默认目标) (1) ->
       “D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C.vcxproj”(默认目标) (8) ->
       “D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C_exports.vcxproj”(默认目标) (28) ->
       (CustomBuild 目标) ->
         C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.tar
       gets(209,5): error MSB6006: “cmd.exe”已退出，代码为 1。 [D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C_e
       xports.vcxproj]

    1 个警告
    1 个错误

已用时间 00:56:59.07

This line should be highlighted:

FileNotFoundError: [Errno 2] No such file or directory: 'D:/workspaces/llvm-project/build/llvm/RelWithDebInfo/libllvm-c.args'

The file libllvm-c.args lives in D:\workspaces\llvm-project\build\llvm\$(Configuration)\libllvm-c.args

After manually copying the file to where it should be and rebuild with cmake --build . --config RelWithDebInfo -- /m

    19>D:/workspaces/llvm-project/build/llvm/RelWithDebInfo/bin\llvm-nm.exe : error : D:/workspaces/llvm-project/build/
       llvm/$(Configuration)/lib/LLVMDemangle.lib: no such file or directory. [D:\workspaces\llvm-project\build\llvm\to
       ols\llvm-shlib\LLVM-C_exports.vcxproj]
       FinalizeBuildStatus:
         正在删除文件“opt.dir\RelWithDebInfo\opt.tlog\unsuccessfulbuild”。
         正在对“opt.dir\RelWithDebInfo\opt.tlog\opt.lastbuildstate”执行 Touch 任务。
   259>已完成生成项目“D:\workspaces\llvm-project\build\llvm\tools\opt\opt.vcxproj”(默认目标)的操作。
    19>CustomBuild:
         Traceback (most recent call last):
           File "D:/workspaces/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 116, in <module>
             main()
           File "D:/workspaces/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 112, in main
             gen_llvm_c_export(ns.output, ns.underscore, libs, ns.nm)
           File "D:/workspaces/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 72, in gen_llvm_c_export
             check_call([nm, '-g', lib], stdout=dumpout_f)
           File "C:\Users\cloud\Miniconda3\lib\subprocess.py", line 328, in check_call
             raise CalledProcessError(retcode, cmd)
         subprocess.CalledProcessError: Command '['D:/workspaces/llvm-project/build/llvm/RelWithDebInfo/bin/llvm-nm', '
         -g', 'D:/workspaces/llvm-project/build/llvm/$(Configuration)/lib/LLVMDemangle.lib']' returned non-zero exit st
         atus 1.
失败。
     7>已完成生成项目“D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C.vcxproj”(默认目标)的操作 - 失败。
   256>Link:
         所有输出均为最新。
         llvm-xray.vcxproj -> D:\workspaces\llvm-project\build\llvm\RelWithDebInfo\bin\llvm-xray.exe
       FinalizeBuildStatus:
         正在删除文件“llvm-xray.dir\RelWithDebInfo\llvm-xray.tlog\unsuccessfulbuild”。
         正在对“llvm-xray.dir\RelWithDebInfo\llvm-xray.tlog\llvm-xray.lastbuildstate”执行 Touch 任务。
   256>已完成生成项目“D:\workspaces\llvm-project\build\llvm\tools\llvm-xray\llvm-xray.vcxproj”(默认目标)的操作。
     1>已完成生成项目“D:\workspaces\llvm-project\build\llvm\ALL_BUILD.vcxproj”(默认目标)的操作 - 失败。

生成失败。
       “D:\workspaces\llvm-project\build\llvm\ALL_BUILD.vcxproj”(默认目标) (1) ->
       “D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C.vcxproj”(默认目标) (7) ->
       “D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C_exports.vcxproj”(默认目标) (19) ->
       (CustomBuild 目标) ->
         D:/workspaces/llvm-project/build/llvm/RelWithDebInfo/bin\llvm-nm.exe : error : D:/workspaces/llvm-project/buil
       d/llvm/$(Configuration)/lib/LLVMDemangle.lib: no such file or directory. [D:\workspaces\llvm-project\build\llvm\
       tools\llvm-shlib\LLVM-C_exports.vcxproj]

    0 个警告
    1 个错误

已用时间 00:00:40.21

I think this should be a related issue ,

[stella.stamenova]

This change is causing build failures on Windows when using Visual Studio as the generator. It would also cause the same failure on for any other generator that supports multiple configurations.

The problem is that CMAKE_CFG_INTDIR changes value based on the tool (https://cmake.org/cmake/help/v3.13/variable/CMAKE_CFG_INTDIR.html) AND in the case of a generator that supports multiple configurations, it has a place-holder value (such as $(Configuration)) since the configuration is not known when CMake is invoked - the configuration is determined at build time.

You can look at other places in LLVM (or LLDB) where CMAKE_CFG_INTDIR is used to see a couple of ways that it can be handled. For example in llvm-lit:

if(NOT "${CMAKE_CFG_INTDIR}" STREQUAL ".")
  foreach(BUILD_MODE ${CMAKE_CONFIGURATION_TYPES})
    string(REPLACE ${CMAKE_CFG_INTDIR} ${BUILD_MODE} bi ${LIT_BASE_DIR})
    set(bi "${bi}/${LIT_FILE_NAME}")
    configure_file(
      llvm-lit.in
      ${bi}
      )
  endforeach()
else()
  set(BUILD_MODE .)
  configure_file(
    llvm-lit.in
    ${LIT_BASE_DIR}/${LIT_FILE_NAME}
    )
endif()

I can help test a fix when you have one - in the mean time, this change should probably be reverted since multiple people are seeing the issue.

[hans]

In D56781#1374777, @stella.stamenova wrote:

This change is causing build failures on Windows when using Visual Studio as the generator. It would also cause the same failure on for any other generator that supports multiple configurations.

I've reverted this and r352250 in r352492.

[Wallbraker]

Okay, I have updated the patch now with the suggestions from @stella.stamenova, thanks! Please try it out, I'm currently running a build myself using ninja to make sure that it still works.

[stella.stamenova]

tools/llvm-shlib/CMakeLists.txt

143

Maybe there's something wrong with my view of this change, but the way I am seeing it, tools/llvm-shlib/CMakeLists.txt appears twice and one of the versions has LIBSFILE and the other one doesn't anymore.

The way LIBSFILE is defined, it still depends on CMAKE_CFG_INTDIR, so this won't work on Windows with VS.

Could you check whether you see the change the same way (with two tools/llvm-shlib/CMakeLists.txt) and update it, so that it's clear what the final change is?

145

There lots of duplication between the two cases, I think you can simplify this.

[Wallbraker]

Refactor loop

[Wallbraker]

This should fix it, doing a build now.

tools/llvm-shlib/CMakeLists.txt

143

I'm only see one, sorry, not sure what I can do to fix it. My Phabricator-fu is weak.

If I understand CMAKE_CFG_INTDIR correctly it will work, as it's only used in the single config build and in the multi-config build the builder should replace it in the call to python (modulo the one big in this current version.

145

Ah I think I see how to do it, I think, I'm not very good at CMake.

[Wallbraker]

Ugh, I see some types, let me respin.

[Wallbraker]

Typo fixed, I only see one llvm-shlib/CMakeLists.txt file.

[stella.stamenova]

Build passes with VS now.

[Wallbraker]

Thank you for testing and your feedback!

CMakeLists.txt

 # shared library by setting LLVM_DYLIB_COMPONENTS to a semi-colon delimited
 # list of LLVM components. All component names handled by llvm-config are valid.
 if(NOT DEFINED LLVM_DYLIB_COMPONENTS)
   set(LLVM_DYLIB_COMPONENTS "all" CACHE STRING
     "Semicolon-separated list of components to include in libLLVM, or \"all\".")
 endif()
 option(LLVM_LINK_LLVM_DYLIB "Link tools against the libllvm dynamic library" OFF)
 if(MSVC)
-  option(LLVM_BUILD_LLVM_C_DYLIB "Build LLVM-C.dll (Windows only)" OFF)
+  option(LLVM_BUILD_LLVM_C_DYLIB "Build LLVM-C.dll (Windows only)" ON)
 else()
   option(LLVM_BUILD_LLVM_C_DYLIB "Build libllvm-c re-export library (Darwin only)" OFF)
 endif()
 set(LLVM_BUILD_LLVM_DYLIB_default OFF)
 if(LLVM_LINK_LLVM_DYLIB OR (LLVM_BUILD_LLVM_C_DYLIB AND NOT MSVC))
   set(LLVM_BUILD_LLVM_DYLIB_default ON)
 endif()
 option(LLVM_BUILD_LLVM_DYLIB "Build libllvm dynamic library" ${LLVM_BUILD_LLVM_DYLIB_default})

tools/llvm-shlib/CMakeLists.txt

if(MSVC)
 
   # The python script needs to know whether symbols are prefixed with underscores or not.
   if(LLVM_HOST_TRIPLE STREQUAL "i686-pc-win32")
     set(GEN_UNDERSCORE "--underscore")
   else()
     set(GEN_UNDERSCORE "")
   endif()
 
+  # Set this name here, not used in multi conf loop,
+  # but add script will pick the right one.
+  set(LIBSFILE ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/libllvm-c.args)
+
   # Get the full name to the libs so the python script understands them.
   foreach(lib ${LIB_NAMES})
-    list(APPEND FULL_LIB_NAMES ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/lib/${lib}.lib)
+     list(APPEND FULL_LIB_NAMES ${CMAKE_BINARY_DIR}/{CMAKE_CFG_INTDIR}/lib/${lib}.lib)
+  endforeach()
+
+  # Need to seperate lib names with newlines.
+  string(REPLACE ";" "\n" FILE_CONTENT "${FULL_LIB_NAMES}")
+
+  if(NOT "${CMAKE_CFG_INTDIR}" STREQUAL ".")
+    foreach(BUILD_MODE ${CMAKE_CONFIGURATION_TYPES})
+      # Replace the special string with a per config directory.
+      string(REPLACE ${CMAKE_CFG_INTDIR} ${BUILD_MODE} PER_CONF_CONTENT "${FILE_CONTENT}")
+
+      # Write out the full lib names into file to be read by the python script.
+      # One libsfile per build, the add_custom_command should expand
+      # ${CMAKE_CFG_INTDIR} correctly and select the right one.
+      file(WRITE ${CMAKE_BINARY_DIR}/${BUILD_MODE}/libllvm-c.args "${PER_CONF_CONTENT}")
     endforeach()
+  else()
+    # Write out the full lib names into file to be read by the python script.
+    file(WRITE ${LIBSFILE} "${FILE_CONTENT}")
+  endif()
 
   # Generate the exports file dynamically.
   set(GEN_SCRIPT ${CMAKE_CURRENT_SOURCE_DIR}/gen-msvc-exports.py)
 
   set(LLVM_EXPORTED_SYMBOL_FILE ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/libllvm-c.exports)
 
   add_custom_command(OUTPUT ${LLVM_EXPORTED_SYMBOL_FILE}
-    COMMAND ${PYTHON_EXECUTABLE} ${GEN_SCRIPT} ${FULL_LIB_NAMES} ${GEN_UNDERSCORE} --nm ${LLVM_TOOLS_BINARY_DIR}/llvm-nm -o ${LLVM_EXPORTED_SYMBOL_FILE}
+    COMMAND ${PYTHON_EXECUTABLE} ${GEN_SCRIPT} --libsfile ${LIBSFILE} ${GEN_UNDERSCORE} --nm ${LLVM_TOOLS_BINARY_DIR}/llvm-nm -o ${LLVM_EXPORTED_SYMBOL_FILE}
     DEPENDS ${LIB_NAMES} llvm-nm
     COMMENT "Generating export list for LLVM-C"
     VERBATIM )
 
   # Finally link the target.
   add_llvm_library(LLVM-C SHARED ${SOURCES} DEPENDS intrinsics_gen)
 
 endif()

tools/llvm-shlib/gen-msvc-exports.py

with removing(touch_tempfile(prefix='dumpout', suffix='.txt')) as dumpout:
                         if m is not None:
                             output_f.write(m.group(1) + '\n')
 
 
 def main():
     parser = argparse.ArgumentParser('gen-msvc-exports')
 
     parser.add_argument(
+        '-i', '--libsfile', help='file with list of libs, new line separated',
+        action='store', default=None
+    )
+    parser.add_argument(
         '-o', '--output', help='output filename', default='LLVM-C.exports'
     )
     parser.add_argument('-u', '--underscore',
         help='labels are prefixed with an underscore (use for 32 bit DLLs)',
         action='store_true'
     )
     parser.add_argument(
         '--nm', help='path to the llvm-nm executable', default='llvm-nm'
     )
     parser.add_argument(
-        'libs', metavar='LIBS', nargs='+', help='list of libraries to generate export from'
+        'libs', metavar='LIBS', nargs='*', help='list of libraries to generate export from'
     )
 
     ns = parser.parse_args()
 
-    gen_llvm_c_export(ns.output, ns.underscore, ns.libs, ns.nm)
+    libs = ns.libs
+
+    # Add if we where given a libsfile add it to the libs.
+    if ns.libsfile:
+        with open(ns.libsfile) as f:
+            libs.extend(f.read().splitlines())
+
+    gen_llvm_c_export(ns.output, ns.underscore, libs, ns.nm)
 
 
 if __name__ == '__main__':
     main()

utils/release/build_llvm_package.bat

 svn.exe export -r %revision% http://llvm.org/svn/llvm-project/clang-tools-extra/%branch% llvm/tools/clang/tools/extra || exit /b
 svn.exe export -r %revision% http://llvm.org/svn/llvm-project/lld/%branch% llvm/tools/lld || exit /b
 svn.exe export -r %revision% http://llvm.org/svn/llvm-project/compiler-rt/%branch% llvm/projects/compiler-rt || exit /b
 svn.exe export -r %revision% http://llvm.org/svn/llvm-project/openmp/%branch% llvm/projects/openmp || exit /b
 svn.exe export -r %revision% http://llvm.org/svn/llvm-project/lldb/%branch% llvm/tools/lldb || exit /b
 
 
 REM Setting CMAKE_CL_SHOWINCLUDES_PREFIX to work around PR27226.
-set cmake_flags=-DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON -DCMAKE_INSTALL_UCRT_LIBRARIES=ON -DCLANG_FORMAT_VS_VERSION=%clang_format_vs_version% -DPACKAGE_VERSION=%package_version% -DLLDB_RELOCATABLE_PYTHON=1 -DLLDB_TEST_COMPILER=%cd%\build32_stage0\bin\clang.exe -DCMAKE_CL_SHOWINCLUDES_PREFIX="Note: including file: "
+set cmake_flags=^
+  -DCMAKE_BUILD_TYPE=Release ^
+  -DLLVM_ENABLE_ASSERTIONS=ON ^
+  -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON ^
+  -DLLVM_BUILD_LLVM_C_DYLIB=ON ^
+  -DCMAKE_INSTALL_UCRT_LIBRARIES=ON ^
+  -DCLANG_FORMAT_VS_VERSION=%clang_format_vs_version% ^
+  -DPACKAGE_VERSION=%package_version% ^
+  -DLLDB_RELOCATABLE_PYTHON=1 ^
+  -DLLDB_TEST_COMPILER=%cd%\build32_stage0\bin\clang.exe ^
+  -DCMAKE_CL_SHOWINCLUDES_PREFIX="Note: including file: "
 
 REM TODO: Run the "check-all" tests.
 
 set "VSCMD_START_DIR=%CD%"
 call "%vsdevcmd%" -arch=x86
 set CC=
 set CXX=
 mkdir build32_stage0