Page MenuHomePhabricator

Use response file when generating LLVM-C.dll
ClosedPublic

Authored by Wallbraker on Jan 16 2019, 5:49 AM.

Details

Summary

As discovered in D56774 the command line gets to long, so use a response file to give the script the libs. This change has been tested and is confirmed working for me.

Cheers, Jakob.

Diff Detail

Event Timeline

Wallbraker created this revision.Jan 16 2019, 5:49 AM
tools/llvm-shlib/CMakeLists.txt
137

You could do the merge in the end to keep the formating stuff in a single place:

string(REPLACE ";" "\n" FILE_CONTENT "${FULL_LIB_NAMES}")

144

filename missing here

tools/llvm-shlib/gen-msvc-exports.py
88

bad indent.

serge-sans-paille requested changes to this revision.Jan 16 2019, 6:21 AM
This revision now requires changes to proceed.Jan 16 2019, 6:21 AM
tools/llvm-shlib/gen-msvc-exports.py
108

useless parenthesis spotted.

You should probably add the libs from the file to the one from the CLI instead of (siltently) erasing the one from CLI.

hans added a comment.Jan 16 2019, 6:24 AM

I actually tried the patch and got

CMake Error at tools/llvm-shlib/CMakeLists.txt:144 (file):
  file failed to open for writing (Invalid argument):

    C:/src/llvm/build.shlib/$(Configuration)/lib/LLVMDemangle.lib

since the whole purpose is to create llvm-c.dll for Windows, I image you have a Windows box somewhere to test this with :-)

Updated changes with comments from @serge-sans-paille, thanks!

Wallbraker marked 3 inline comments as done.Jan 16 2019, 6:27 AM

I actually tried the patch and got

Thanks for doing that and your patience!

CMake Error at tools/llvm-shlib/CMakeLists.txt:144 (file):
  file failed to open for writing (Invalid argument):

    C:/src/llvm/build.shlib/$(Configuration)/lib/LLVMDemangle.lib

Well that error message is because of me missing the file argument, now I am quite worried about $(Configuration) tho where is that coming from? :(

since the whole purpose is to create llvm-c.dll for Windows, I image you have a Windows box somewhere to test this with :-)

Ugh, I guess I have to set a windows box up. I'm doing that so that I and any of the users of my projects that uses LLVM don't have to go through the quite painful procedure of building LLVM on Windows. But yeah I was hoping nothing had broken since I last tested this.

Wallbraker marked an inline comment as done.Jan 16 2019, 6:39 AM

Not tested, will address last comment and then setup something I can test this on.

tools/llvm-shlib/gen-msvc-exports.py
108

I am unashamedly hard cargo-culting all cmake and python code here as I have almost zero proficiency in both. (Insert i_have_no_idea_what_i_am_doing.jpg )

Ok will change soon

Wallbraker updated this revision to Diff 182034.EditedJan 16 2019, 6:43 AM

Should address all of the comments, haven't done anything about the $(Configuration) @hans saw in the error message. Will setup something to test this.

And remove the parentheses.

Wallbraker marked an inline comment as done.Jan 16 2019, 6:44 AM
Wallbraker updated this revision to Diff 182044.EditedJan 16 2019, 7:36 AM

Have now tested this patch and confirmed that it is working for me at least. Needed to make the libs argument optional or the python script would error out and correct splitLines to splitlines. I did not see any sign of $(Configuration) in my scripts.

@hans @serge-sans-paille So I built the package and has verified that the LLVM-C.dll is included in the package but I could not find the LLVM-C.lib file tho. After some hours of digging through the multitude of *.cmake files I have to admit defeat on how to get that in there, any pointers would be very much welcome.

tools/llvm-shlib/CMakeLists.txt
146

if FILE_CONTENT contains space, they are going to be suppressed by file(...) unless you quote the variable.

tools/llvm-shlib/gen-msvc-exports.py
111

avoid creating an extra copy using libs.extend(f.read)...)

Wallbraker edited the summary of this revision. (Show Details)

Thanks, this should fix all concerns raised, I tested this as well and seems to work.

Still need help with shipping the LLVM-C.lib in the windows installer, while that isn't super necessary it will help people using it.

Wallbraker marked 2 inline comments as done.Jan 17 2019, 2:40 AM

LGTM, thanks

This revision is now accepted and ready to land.Jan 17 2019, 2:49 AM

@Wallbraker if you don't have commit rights I can commit this one for you.

@serge-sans-paille I do not, so that would be greatly appreciated.

Cheers, Jakob.

This revision was automatically updated to reflect the committed changes.

I encounter similar issue, a weird $(Configuration) appears in my build folder. The source is clone from https://github.com/llvm/llvm-project

Sorry for the Chinese error message. But all you should be aware are "警告" is "Warning", "错误" is "Error" and "失败" is "Failed"

    28>PrepareForBuild:
         正在创建目录“x64\RelWithDebInfo\LLVM-C_exports\”。
         正在创建目录“x64\RelWithDebInfo\LLVM-C_exports\LLVM-C_exports.tlog\”。
       InitializeBuildStatus:
         正在创建“x64\RelWithDebInfo\LLVM-C_exports\LLVM-C_exports.tlog\unsuccessfulbuild”,因为已指定“AlwaysCreate”。
       CustomBuild:
         Generating export list for LLVM-C
         Traceback (most recent call last):
           File "D:/workspaces/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 116, in <module>
             main()
           File "D:/workspaces/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 109, in main
             with open(ns.libsfile) as f:
         FileNotFoundError: [Errno 2] No such file or directory: 'D:/workspaces/llvm-project/build/llvm/RelWithDebInfo/
         libllvm-c.args'
    28>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targe
       ts(209,5): error MSB6006: “cmd.exe”已退出,代码为 1。 [D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C_exp
       orts.vcxproj]
    28>已完成生成项目“D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C_exports.vcxproj”(默认目标)的操作 - 失败。
     8>已完成生成项目“D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C.vcxproj”(默认目标)的操作 - 失败。
     1>已完成生成项目“D:\workspaces\llvm-project\build\llvm\ALL_BUILD.vcxproj”(默认目标)的操作 - 失败。
生成失败。

       “D:\workspaces\llvm-project\build\llvm\ALL_BUILD.vcxproj”(默认目标) (1) ->
       “D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C.vcxproj”(默认目标) (8) ->
       “D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C_exports.vcxproj”(默认目标) (28) ->
       “D:\workspaces\llvm-project\build\llvm\lib\DebugInfo\DWARF\LLVMDebugInfoDWARF.vcxproj”(默认目标) (56) ->
       (ClCompile 目标) ->
         D:\workspaces\llvm-project\llvm\lib\DebugInfo\DWARF\DWARFVerifier.cpp(834): warning C4819: The file contains a
        character that cannot be represented in the current code page (936). Save the file in Unicode format to prevent
        data loss [D:\workspaces\llvm-project\build\llvm\lib\DebugInfo\DWARF\LLVMDebugInfoDWARF.vcxproj]


       “D:\workspaces\llvm-project\build\llvm\ALL_BUILD.vcxproj”(默认目标) (1) ->
       “D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C.vcxproj”(默认目标) (8) ->
       “D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C_exports.vcxproj”(默认目标) (28) ->
       (CustomBuild 目标) ->
         C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.tar
       gets(209,5): error MSB6006: “cmd.exe”已退出,代码为 1。 [D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C_e
       xports.vcxproj]

    1 个警告
    1 个错误

已用时间 00:56:59.07

This line should be highlighted:

FileNotFoundError: [Errno 2] No such file or directory: 'D:/workspaces/llvm-project/build/llvm/RelWithDebInfo/libllvm-c.args'

The file libllvm-c.args lives in D:\workspaces\llvm-project\build\llvm\$(Configuration)\libllvm-c.args

After manually copying the file to where it should be and rebuild with cmake --build . --config RelWithDebInfo -- /m

    19>D:/workspaces/llvm-project/build/llvm/RelWithDebInfo/bin\llvm-nm.exe : error : D:/workspaces/llvm-project/build/
       llvm/$(Configuration)/lib/LLVMDemangle.lib: no such file or directory. [D:\workspaces\llvm-project\build\llvm\to
       ols\llvm-shlib\LLVM-C_exports.vcxproj]
       FinalizeBuildStatus:
         正在删除文件“opt.dir\RelWithDebInfo\opt.tlog\unsuccessfulbuild”。
         正在对“opt.dir\RelWithDebInfo\opt.tlog\opt.lastbuildstate”执行 Touch 任务。
   259>已完成生成项目“D:\workspaces\llvm-project\build\llvm\tools\opt\opt.vcxproj”(默认目标)的操作。
    19>CustomBuild:
         Traceback (most recent call last):
           File "D:/workspaces/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 116, in <module>
             main()
           File "D:/workspaces/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 112, in main
             gen_llvm_c_export(ns.output, ns.underscore, libs, ns.nm)
           File "D:/workspaces/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 72, in gen_llvm_c_export
             check_call([nm, '-g', lib], stdout=dumpout_f)
           File "C:\Users\cloud\Miniconda3\lib\subprocess.py", line 328, in check_call
             raise CalledProcessError(retcode, cmd)
         subprocess.CalledProcessError: Command '['D:/workspaces/llvm-project/build/llvm/RelWithDebInfo/bin/llvm-nm', '
         -g', 'D:/workspaces/llvm-project/build/llvm/$(Configuration)/lib/LLVMDemangle.lib']' returned non-zero exit st
         atus 1.
失败。
     7>已完成生成项目“D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C.vcxproj”(默认目标)的操作 - 失败。
   256>Link:
         所有输出均为最新。
         llvm-xray.vcxproj -> D:\workspaces\llvm-project\build\llvm\RelWithDebInfo\bin\llvm-xray.exe
       FinalizeBuildStatus:
         正在删除文件“llvm-xray.dir\RelWithDebInfo\llvm-xray.tlog\unsuccessfulbuild”。
         正在对“llvm-xray.dir\RelWithDebInfo\llvm-xray.tlog\llvm-xray.lastbuildstate”执行 Touch 任务。
   256>已完成生成项目“D:\workspaces\llvm-project\build\llvm\tools\llvm-xray\llvm-xray.vcxproj”(默认目标)的操作。
     1>已完成生成项目“D:\workspaces\llvm-project\build\llvm\ALL_BUILD.vcxproj”(默认目标)的操作 - 失败。

生成失败。
       “D:\workspaces\llvm-project\build\llvm\ALL_BUILD.vcxproj”(默认目标) (1) ->
       “D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C.vcxproj”(默认目标) (7) ->
       “D:\workspaces\llvm-project\build\llvm\tools\llvm-shlib\LLVM-C_exports.vcxproj”(默认目标) (19) ->
       (CustomBuild 目标) ->
         D:/workspaces/llvm-project/build/llvm/RelWithDebInfo/bin\llvm-nm.exe : error : D:/workspaces/llvm-project/buil
       d/llvm/$(Configuration)/lib/LLVMDemangle.lib: no such file or directory. [D:\workspaces\llvm-project\build\llvm\
       tools\llvm-shlib\LLVM-C_exports.vcxproj]

    0 个警告
    1 个错误

已用时间 00:00:40.21

I think this should be a related issue ,

This change is causing build failures on Windows when using Visual Studio as the generator. It would also cause the same failure on for any other generator that supports multiple configurations.

The problem is that CMAKE_CFG_INTDIR changes value based on the tool (https://cmake.org/cmake/help/v3.13/variable/CMAKE_CFG_INTDIR.html) AND in the case of a generator that supports multiple configurations, it has a place-holder value (such as $(Configuration)) since the configuration is not known when CMake is invoked - the configuration is determined at build time.

You can look at other places in LLVM (or LLDB) where CMAKE_CFG_INTDIR is used to see a couple of ways that it can be handled. For example in llvm-lit:

if(NOT "${CMAKE_CFG_INTDIR}" STREQUAL ".")
  foreach(BUILD_MODE ${CMAKE_CONFIGURATION_TYPES})
    string(REPLACE ${CMAKE_CFG_INTDIR} ${BUILD_MODE} bi ${LIT_BASE_DIR})
    set(bi "${bi}/${LIT_FILE_NAME}")
    configure_file(
      llvm-lit.in
      ${bi}
      )
  endforeach()
else()
  set(BUILD_MODE .)
  configure_file(
    llvm-lit.in
    ${LIT_BASE_DIR}/${LIT_FILE_NAME}
    )
endif()

I can help test a fix when you have one - in the mean time, this change should probably be reverted since multiple people are seeing the issue.

hans added a comment.Jan 29 2019, 5:43 AM

This change is causing build failures on Windows when using Visual Studio as the generator. It would also cause the same failure on for any other generator that supports multiple configurations.

I've reverted this and r352250 in r352492.

Wallbraker edited the summary of this revision. (Show Details)

Okay, I have updated the patch now with the suggestions from @stella.stamenova, thanks! Please try it out, I'm currently running a build myself using ninja to make sure that it still works.

Wallbraker reopened this revision.Jan 29 2019, 8:37 AM
This revision is now accepted and ready to land.Jan 29 2019, 8:37 AM
stella.stamenova requested changes to this revision.Jan 29 2019, 10:18 AM
stella.stamenova added inline comments.
tools/llvm-shlib/CMakeLists.txt
140

Maybe there's something wrong with my view of this change, but the way I am seeing it, tools/llvm-shlib/CMakeLists.txt appears twice and one of the versions has LIBSFILE and the other one doesn't anymore.

The way LIBSFILE is defined, it still depends on CMAKE_CFG_INTDIR, so this won't work on Windows with VS.

Could you check whether you see the change the same way (with two tools/llvm-shlib/CMakeLists.txt) and update it, so that it's clear what the final change is?

142

There lots of duplication between the two cases, I think you can simplify this.

This revision now requires changes to proceed.Jan 29 2019, 10:18 AM
Wallbraker marked 2 inline comments as done.

Refactor loop

This should fix it, doing a build now.

tools/llvm-shlib/CMakeLists.txt
140

I'm only see one, sorry, not sure what I can do to fix it. My Phabricator-fu is weak.

If I understand CMAKE_CFG_INTDIR correctly it will work, as it's only used in the single config build and in the multi-config build the builder should replace it in the call to python (modulo the one big in this current version.

142

Ah I think I see how to do it, I think, I'm not very good at CMake.

Ugh, I see some types, let me respin.

Typo fixed, I only see one llvm-shlib/CMakeLists.txt file.

stella.stamenova accepted this revision.Jan 29 2019, 2:38 PM

Build passes with VS now.

This revision is now accepted and ready to land.Jan 29 2019, 2:38 PM
Wallbraker marked 2 inline comments as done.Jan 29 2019, 3:09 PM

Thank you for testing and your feedback!

This revision was automatically updated to reflect the committed changes.
Herald added a project: Restricted Project. · View Herald TranscriptMar 19 2019, 2:14 AM
cloudhan removed a subscriber: cloudhan.Jan 12 2021, 7:25 AM