Add script to download prerequisites for Windows build

[pbo-linaro]

Summary

llvm/utils/release/download_prerequisites.bat
usage: <out_folder>

This script downloads needed dependencies in <out_folder> and create an
activate.bat script to load them on PATH.

In build_llvm_release.bat, add an option --download-prerequisites that
calls this script before doing the build.

Example: build_llvm_release.bat --version 15.0.0 --x64 --download-prerequisites

[pbo-linaro]

rebased commit

[tschuett]

Would a README.md help to understand what the requirements are without having to read batch scripts?

[pbo-linaro]

In D135353#3840491, @tschuett wrote:

Would a README.md help to understand what the requirements are without having to read batch scripts?

No problem to add one, but it's redundant information. Will future commiters think to update both? I tried to make the beginning of the script readable, so people can see directly what is done.

[hans]

llvm/utils/release/download_prerequisites.bat

17

Whoa, I had not realized pushd/popd is a thing on Windows.

37

I suppose VS is tricky, but for Python and Perl, it would be *really* nice if they could also be handled automatically. Don't they have any official zip files we could use?

93

copy-paste-o from get_7z?

101

Clever :)

128

I'll be very happy not to have to do this manually ever again :-)

151

I'm not actually sure we need nuget at the moment? It was used when we built the visual studio clang-format plugin, but that was dropped a while ago.

[pbo-linaro]

llvm/utils/release/download_prerequisites.bat

37

Python and Perl have portable version, but as discussed here, the python used for build will be a dependency at runtime for lldb. Since that involves to packages python *with* llvm release, there is more work involved.

For Perl, I don't know exactly how it's used, but the comment "OpenMP runtime" made me adopt a prudent approach.

93

Good catch, thanks

128

yes. The worse part in this script was finding how to escape url. Batch is really... exotic, to stay polite.

151

Ok, I can probably remove this then. Just included that based on an original comment.

[hans]

llvm/utils/release/download_prerequisites.bat

37

The current Python version used for the win releases is 3.10.5. Since we're already using it to build, we might as well hard-code it here. If there is a way to download and extract that without running commands manually it would be very handy.

llvm/utils/release/build_llvm_release.bat

 @echo off
 setlocal enabledelayedexpansion
 
 goto begin
 
 :usage
 echo Script for building the LLVM installer on Windows,
 echo used for the releases at https://github.com/llvm/llvm-project/releases
 echo.
 echo Usage: build_llvm_release.bat --version ^<version^> [--x86,--x64]
 echo.
 echo Options:
 echo --version: [required] version to build
 echo --help: display this help
 echo --x86: build and test x86 variant
 echo --x64: build and test x64 variant
+echo --download-prerequisites: automatically download prerequisites needed
 echo.
 echo Note: At least one variant to build is required.
+echo Note: For prerequisites, VS, python and perl must be installed manually.
 echo.
 echo Example: build_llvm_release.bat --version 15.0.0 --x86 --x64
 exit /b 1
 
 :begin
 
 ::==============================================================================
 :: parse args
 set version=
 set help=
 set x86=
 set x64=
+set download-prerequisites=
 call :parse_args %*
 
 if "%help%" NEQ "" goto usage
 
 if "%version%" == "" (
     echo --version option is required
     echo =============================
     goto usage
 )
 
 if "%x64%" == "" if "%x86%" == "" (
     echo nothing to build!
     echo choose one or several variants from: --x86 --x64
     exit /b 1
 )
 
 ::==============================================================================
-:: check prerequisites
+:: check prerequisites (See download_prerequisites.bat script for details)
 REM Note:
 REM   7zip versions 21.x and higher will try to extract the symlinks in
 REM   llvm's git archive, which requires running as administrator.
 
 REM Check 7-zip version and/or administrator permissions.
 for /f "delims=" %%i in ('7z.exe ^| findstr /r "2[1-9].[0-9][0-9]"') do set version_7z=%%i
 if not "%version_7z%"=="" (
   REM Unique temporary filename to use by the 'mklink' command.
   set "link_name=%temp%\%username%_%random%_%random%.tmp"
 
   REM As the 'mklink' requires elevated permissions, the symbolic link
   REM creation will fail if the script is not running as administrator.
   mklink /d "!link_name!" . 1>nul 2>nul
   if errorlevel 1 (
     echo.
     echo Script requires administrator permissions, or a 7-zip version 20.x or older.
     echo Current version is "%version_7z%"
     exit /b 1
   ) else (
     REM Remove the temporary symbolic link.
     rd "!link_name!"
   )
 )
 
-REM Prerequisites:
-REM
-REM   Visual Studio 2019, CMake, Ninja, GNUWin32, SWIG, Python 3,
-REM   NSIS with the strlen_8192 patch,
-REM   Visual Studio 2019 SDK and Nuget (for the clang-format plugin),
-REM   Perl (for the OpenMP run-time).
-REM
-REM
-REM   For LLDB, SWIG version <= 3.0.8 needs to be used to work around
-REM   https://github.com/swig/swig/issues/769
-REM
-
 :: Detect Visual Studio
 set vsdevcmd=
 set vs_2019_prefix=C:\Program Files (x86)\Microsoft Visual Studio\2019
 :: try potential activated visual studio, then 2019, with different editions
 call :find_visual_studio "%VSINSTALLDIR%"
 call :find_visual_studio "%vs_2019_prefix%\Enterprise"
 call :find_visual_studio "%vs_2019_prefix%\Professional"
 call :find_visual_studio "%vs_2019_prefix%\Community"
 call :find_visual_studio "%vs_2019_prefix%\BuildTools"
 if not exist "%vsdevcmd%" (
   echo Can't find any installation of Visual Studio
   exit /b 1
 )
 echo Using VS devcmd: %vsdevcmd%
 
+if "%download-prerequisites%" == "true" (
+    call %~dp0\download_prerequisites.bat prerequisites || exit /b 1
+    call prerequisites/activate.bat || exit /b 1
+)
+
 ::==============================================================================
 :: start echoing what we do
 @echo on
 
 set python32_dir=C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python310-32
 set python64_dir=C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python310
 
 set revision=llvmorg-%version%

 
 REM Stage0 binaries directory; used in stage1.
 set "stage0_bin_dir=%build_dir%/build32_stage0/bin"
 set cmake_flags=^
   %common_cmake_flags% ^
   -DLLDB_TEST_COMPILER=%stage0_bin_dir%/clang.exe ^
   -DPYTHON_HOME=%PYTHONHOME% ^
   -DPython3_ROOT_DIR=%PYTHONHOME% ^
-  -DLIBXML2_INCLUDE_DIRS=%libxmldir%/include/libxml2 ^
+  -DLIBXML2_INCLUDE_DIR=%libxmldir%/include/libxml2 ^
   -DLIBXML2_LIBRARIES=%libxmldir%/lib/libxml2s.lib
 
 cmake -GNinja %cmake_flags% ..\llvm-project\llvm || exit /b 1
 ninja || ninja || ninja || exit /b 1
 REM ninja check-llvm || ninja check-llvm || ninja check-llvm || exit /b 1
 REM ninja check-clang || ninja check-clang || ninja check-clang || exit /b 1
 ninja check-lld || ninja check-lld || ninja check-lld || exit /b 1
 ninja check-sanitizer || ninja check-sanitizer || ninja check-sanitizer || exit /b 1

 
 REM Stage0 binaries directory; used in stage1.
 set "stage0_bin_dir=%build_dir%/build64_stage0/bin"
 set cmake_flags=^
   %common_cmake_flags% ^
   -DLLDB_TEST_COMPILER=%stage0_bin_dir%/clang.exe ^
   -DPYTHON_HOME=%PYTHONHOME% ^
   -DPython3_ROOT_DIR=%PYTHONHOME% ^
-  -DLIBXML2_INCLUDE_DIRS=%libxmldir%/include/libxml2 ^
+  -DLIBXML2_INCLUDE_DIR=%libxmldir%/include/libxml2 ^
   -DLIBXML2_LIBRARIES=%libxmldir%/lib/libxml2s.lib
 
 cmake -GNinja %cmake_flags% ..\llvm-project\llvm || exit /b 1
 ninja || ninja || ninja || exit /b 1
 ninja check-llvm || ninja check-llvm || ninja check-llvm || exit /b 1
 ninja check-clang || ninja check-clang || ninja check-clang || exit /b 1
 ninja check-lld || ninja check-lld || ninja check-lld || exit /b 1
 ninja check-sanitizer || ninja check-sanitizer || ninja check-sanitizer || exit /b 1

llvm/utils/release/download_prerequisites.bat

+@echo off
+
+if "%1" == "" (
+    echo usage: ^<out_folder^>
+    echo.
+    echo This script downloads prerequisites needed to build LLVM.
+    echo In ^<out_folder^>, an activate.bat script will be generated.
+    echo It can be called to override PATH.
+    echo
+    echo VS, Python and Perl installers are downloaded, but it's up to
+    echo you to install them.
+    exit /b 1
+)
+
+set out_folder=%1
+mkdir %out_folder%
+pushd %out_folder%
+
+echo ===========================================================
+call :get_msys2 || exit /b 1
+echo ===========================================================
+call :get_7z || exit /b 1
+echo ===========================================================
+call :get_ninja || exit /b 1
+echo ===========================================================
+call :get_nsis || exit /b 1
+echo ===========================================================
+call :get_swig || exit /b 1
+echo ===========================================================
+:: Visual Studio 2019 SDK and Nuget (for the clang-format plugin)
+call :get_nuget || exit /b 1
+echo ===========================================================
+call :get_cmake || exit /b 1
+echo ===========================================================
+
+echo Those dependencies are just downloaded, and you can install them
+echo system-wide manually
+set url_vs2019=https://aka.ms/vs/16/release/vs_community.exe
+call :download "VS2019 installer" %url_vs2019% vs_community_2019.exe || exit /b 1
+set url_vs2022=https://aka.ms/vs/17/release/vs_community.exe
+call :download "VS2022 installer" %url_vs2022% vs_community_2022.exe || exit /b 1
+set url_pythonx86=https://www.python.org/ftp/python/3.10.5/python-3.10.5.exe
+call :download "Python x86" %url_pythonx86% python310_x86.exe || exit /b 1
+set url_pythonx64=https://www.python.org/ftp/python/3.10.5/python-3.10.5.amd64.exe
+call :download "Python x64" %url_pythonx64% python310_x64.exe || exit /b 1
+:: Perl (for the OpenMP run-time)
+set url_perl=https://strawberryperl.com/download/5.32.1.1/strawberry-perl-5.32.1.1-64bit.msi
+call :download "Perl" %url_perl% perl_setup.exe || exit /b 1
+
+echo ===========================================================
+echo To import dependencies in your PATH, call %cd%\activate.bat
+popd
+exit /b 0
+
+::===============================================================
+:prepend_path
+set added_relative_path=%1
+if not exist %added_relative_path% (
+    echo %added_relative_path% can't be found under %out_folder%
+    exit /b 1
+)
+:: update current PATH
+set "PATH=%cd%/%added_relative_path%;%PATH%"
+:: set activate.bat
+echo set PATH=%%~dp0/%added_relative_path%;%%PATH%% >> activate.bat
+exit /b 0
+
+::===============================================================
+:download
+set description=%~1
+set url=%2
+set out_file=%3
+echo Download %description% from %url% in %out_file%
+curl -L %url% -o %out_file% || exit /b 1
+exit /b 0
+
+::===============================================================
+:get_msys2
+mkdir msys2 && pushd msys2
+set url_msys2=https://github.com/msys2/msys2-installer/releases/download/2022-09-04/msys2-base-x86_64-20220904.sfx.exe
+set msys2_packages=p7zip unzip diffutils
+call :download "Msys2 self extracting installer" %url_msys2% msys2_setup.exe || exit /b 1
+msys2_setup.exe || exit /b 1
+popd
+call :prepend_path msys2/msys64/usr/bin || exit /b 1
+:: initialize
+bash.exe -c "pacman-key --init && pacman-key --populate msys2 && pacman -Sy" || exit /b 1
+bash.exe -c "echo 'SigLevel = Never' >> /etc/pacman.conf" || exit /b 1
+:: install some needed packages
+bash.exe -c "yes | pacman -S %msys2_packages%" || exit /b 1
+:: may leave gpg-agent behind
+taskkill /IM gpg-agent.exe /f
+where 7z || exit /b 1
+exit /b 0
+
+::===============================================================
+:get_7z
+mkdir 7zip && pushd 7zip
+set url_7zip=https://www.7-zip.org/a/7z2201-x64.exe
+call :download "7zip" %url_7zip% 7zip_setup.exe || exit /b 1
+:: extract using msys2 7z
+7z x 7zip_setup.exe || exit /b 1
+popd
+call :prepend_path 7zip || exit /b 1
+where 7z || exit /b 1
+exit /b 0
+
+::===============================================================
+:get_ninja
+mkdir ninja && pushd ninja
+set url_ninja=https://github.com/ninja-build/ninja/releases/download/v1.11.1/ninja-win.zip
+call :download "ninja" %url_ninja% ninja.zip || exit /b 1
+unzip ninja.zip || exit /b 1
+popd
+call :prepend_path ninja || exit /b 1
+where ninja || exit /b 1
+exit /b 0
+
+::===============================================================
+:get_nsis
+mkdir nsis && pushd nsis
+:: you can appreciate the double escape to allow %20 in url
+set url_nsis=https://freefr.dl.sourceforge.net/project/nsis/NSIS%%%%203/3.08/nsis-3.08-setup.exe
+set url_nsis_patch=https://freefr.dl.sourceforge.net/project/nsis/NSIS%%%%203/3.08/nsis-3.08-strlen_8192.zip
+call :download "NSIS" "%url_nsis%" nsis_setup.exe || exit /b 1
+7z x nsis_setup.exe || exit /b 1
+call :download "NSIS" "%url_nsis_patch%" nsis_patch.zip || exit /b 1
+REM overwrite some files with patch
+unzip -o nsis_patch.zip || exit /b 1
+popd
+call :prepend_path nsis || exit /b 1
+where NSIS || exit /b 1
+exit /b 0
+
+::===============================================================
+:get_swig
+mkdir swig && pushd swig
+REM For LLDB, SWIG version <= 3.0.8 or >= 4.0 needs to be used to work around
+REM https://github.com/swig/swig/issues/769
+set swig_version=4.0.2
+set url_swig=https://freefr.dl.sourceforge.net/project/swig/swigwin/swigwin-%swig_version%/swigwin-%swig_version%.zip
+call :download "SWIG" %url_swig% swig.zip || exit /b 1
+unzip -q swig.zip || exit /b 1
+mv swigwin-%swig_version%/* . || exit /b 1
+popd
+call :prepend_path swig || exit /b 1
+where swig || exit /b 1
+exit /b 0
+
+::===============================================================
+:get_nuget
+mkdir nuget && pushd nuget
+set url_nuget=https://dist.nuget.org/win-x86-commandline/v6.1.0/nuget.exe
+call :download "nuget" %url_nuget% nuget.exe || exit /b 1
+popd
+call :prepend_path nuget || exit /b 1
+where nuget || exit /b 1
+exit /b 0
+
+::===============================================================
+:get_cmake
+mkdir cmake && pushd cmake
+set cmake_version=3.24.2
+set url_cmake=https://github.com/Kitware/CMake/releases/download/v%cmake_version%/cmake-%cmake_version%-windows-x86_64.zip
+call :download "cmake" %url_cmake% cmake.zip || exit /b 1
+unzip -q cmake.zip || exit /b 1
+mv cmake-*/* . || exit /b 1
+popd
+call :prepend_path cmake/bin || exit /b 1
+where cmake || exit /b 1
+exit /b 0