This is an archive of the discontinued LLVM Phabricator instance.

[docs] Add new document on building distributions
ClosedPublic

Authored by beanz on May 16 2019, 3:37 PM.

Details

Summary

This document is an attempt to provide a guide for best practices for using the LLVM build system to generate distributable LLVM-based tools.

Most of the document is geared toward distributions of LLVM-based toolchains, but much of it also applies to distributing other LLVM-based tools and libraries.

Diff Detail

Repository
rL LLVM

Event Timeline

beanz created this revision.May 16 2019, 3:37 PM
Herald added a project: Restricted Project. · View Herald TranscriptMay 16 2019, 3:37 PM
Herald added a subscriber: arphaman. · View Herald Transcript
tstellar accepted this revision.May 16 2019, 3:42 PM

This is very useful, thanks for writing this up.

This revision is now accepted and ready to land.May 16 2019, 3:42 PM

This is great! Might it be worth mentioning LLVM_INSTALL_TOOLCHAIN_ONLY somewhere?

beanz added a comment.May 16 2019, 3:56 PM

This is great! Might it be worth mentioning LLVM_INSTALL_TOOLCHAIN_ONLY somewhere?

I kinda have mixed feelings about that. I want to discourage people from using the install target when building distributions because I think that if you are building a distribution you should be conscious of exactly what goes into it. The install-distribution target does that in a much cleaner interface than the install target provides. That said, it is probably worth a mention, and I should explain the differences in expected behavior between install and `install-distribution.

beanz updated this revision to Diff 199927.May 16 2019, 4:07 PM

Adding information about distribution targets and LLVM_INSTALL_TOOLCHAIN_ONLY.

A few suggestions to consider, but this is GREAT, thanks!

llvm/docs/BuildingADistribution.rst
53 ↗(On Diff #199927)

"Due to design and implemenation decisions in LLVM relies on" can be improved, maybe remove "in" and add a comma; "Due to design and implemenation decisions, LLVM relies on ..."

86 ↗(On Diff #199927)

Is distributation target an internal target that shouldn't be used or a convenience target and might be used by the people building the distribution ? If the former consider saying "internal build target".

108 ↗(On Diff #199927)

Consider adding' with "BOOTSTRAP_*, see :doc:AdvancedBuilds.

112 ↗(On Diff #199927)

I suggest simplifying 110..112 to something like:

"The first and simplest way to control the optimization is using CMAKE_BUILD_TYPE. The main ..."

123 ↗(On Diff #199927)

Consider changing "includes static archives" to "includes static libraries" as archives isn't used anywhere else.

155 ↗(On Diff #199927)

Maybe add "see :doc:CMake"

188 ↗(On Diff #199927)

Consider moving to ":doc:CMake" and then added to the "already documented" list above?

beanz marked 3 inline comments as done.May 16 2019, 8:40 PM
beanz added inline comments.
llvm/docs/BuildingADistribution.rst
86 ↗(On Diff #199927)

If you want to install a distribution onto your local system into a path that requires sudo, and you don’t want to build everything, you need to use it. It is also a handy from a workflow perspective. If it were only for internal build system purposes I would likely not have made it a target.

123 ↗(On Diff #199927)

I’m not sure what you mean by the term “archive” not being used elsewhere. We use the term “static archive” and “static library” pretty interchangeably around LLVM. It is really a matter of preference and often whether you come from a Unix background where archives are created with an archiver (ar).

188 ↗(On Diff #199927)

This is an uncommon setting to alter, and we really don’t want that document to have an exhaustive list of our options.

beanz updated this revision to Diff 199971.May 16 2019, 8:41 PM

Updates based on @winksaville's feedback

hans added a comment.May 17 2019, 12:22 AM

Interesting! I didn't know about most of this.

I want to discourage people from using the install target when building distributions because I think that if you are building a distribution you should be conscious of exactly what goes into it. The install-distribution target does that in a much cleaner interface than the install target provides.

But the install target is kind of a standard cmake thing; wouldn't it be better if we made that target good? That's also what the cmake "package" target uses, which is what we use to build the Windows installer.

llvm/docs/BuildingADistribution.rst
202 ↗(On Diff #199971)

I thought it defaults to OFF?

llvm/CMakeLists.txt:221:option(LLVM_INSTALL_TOOLCHAIN_ONLY "Only include toolchain files in the 'install' target." OFF)

beanz added a comment.EditedMay 17 2019, 9:24 AM

But the install target is kind of a standard cmake thing; wouldn't it be better if we made that target good? That's also what the cmake "package" target uses, which is what we use to build the Windows installer.

It has been a while since I introduced this stuff (3 years now), but the rationale was that the install target and the all target aren't really well exposed in CMake. One of the goals of this was to reduce build time for generating distributions, and there is no way in CMake to "build only the things that get installed" without running the `install` target.

I should look at how the package target works. I suspect there may be a way to make a package-distribution target.

beanz marked an inline comment as done.May 17 2019, 9:24 AM
beanz added inline comments.
llvm/docs/BuildingADistribution.rst
202 ↗(On Diff #199971)

You're right. Good catch.

beanz updated this revision to Diff 200058.May 17 2019, 9:32 AM

Fixing the wording around LLVM_INSTALL_TOOLCHAIN_ONLY

@hans I have some ideas on how to make package work. There is a simple solution to make it work as long as you're not using any runtime components, but I have an idea on how to make it work with runtime components.

hans added a comment.May 17 2019, 10:35 AM

@hans I have some ideas on how to make package work. There is a simple solution to make it work as long as you're not using any runtime components, but I have an idea on how to make it work with runtime components.

The Windows packages currently includes at least the asan runtime (and a bunch of other stuff like openmp that people ask for but I'm never quite sure if it works).

The Windows packages currently includes at least the asan runtime (and a bunch of other stuff like openmp that people ask for but I'm never quite sure if it works).

But I assume you're not building them using the just-built clang (LLVM_ENABLE_RUNTIMES). I'm not sure how any of our current mechanisms for installing those would work with cpack.

hans added a comment.May 17 2019, 10:45 AM

The Windows packages currently includes at least the asan runtime (and a bunch of other stuff like openmp that people ask for but I'm never quite sure if it works).

But I assume you're not building them using the just-built clang (LLVM_ENABLE_RUNTIMES). I'm not sure how any of our current mechanisms for installing those would work with cpack.

Hmm, I've never set LLVM_ENABLE_RUNTIMES, I just build with a previously built clang. The script lives in utils/release/build_llvm_package.bat

Hmm, I've never set LLVM_ENABLE_RUNTIMES, I just build with a previously built clang. The script lives in utils/release/build_llvm_package.bat

Given that there is a one-line fix to make the package target install the distribution list of components. I'm working on making this work for the runtimes build too.

smeenai added inline comments.
llvm/docs/BuildingADistribution.rst
128 ↗(On Diff #200058)

@beanz While you're in a documentation writing mood, I think you'd mentioned wanting to write up tips on creating good PGO sample files and getting good profiling data? :D

129 ↗(On Diff #200058)

I think "but" would read better than "however" here.

203 ↗(On Diff #200058)

Nit: semicolon instead of comma here.

LLVM_INSTALL_TOOLCHAIN_ONLY also prevents libraries from being installed, right?

Is it worth mentioning LLVM_TOOLCHAIN_TOOLS, which you need to use to get things to actually install if you're using LLVM_INSTALL_TOOLCHAIN_ONLY?

I'm trying to test the distribution commands from lines 65-67, I first do the cmake:

$ cd ~/prgs/llvm/llvm-project
$ mkdir build
$ cd build
$ cmake -G Ninja -C ../clang/cmake/caches/DistributionExample.cmake ../llvm
loading initial cache file ../clang/cmake/caches/DistributionExample.cmake
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
...
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Configuring done
-- Generating done
-- Build files have been written to: /home/wink/prgs/llvm/llvm-project/build

That succeeded, then I try the nina command but that fails:

$ ninja stage2-distribution
ninja: error: unknown target 'stage2-distribution'

I then check the ninja targets but there are no stage or distribution targets:

$ ninja -t targets | grep stage
$ ninja -t targets | grep distribution
$ 

What have I done wrong?

winksaville added a comment.EditedMay 19 2019, 2:24 PM

What have I done wrong?

One thing I apparently did wrong was not supply LLVM_ENABLE_PROJECTS so
I added that and CMAKE_INSTALL_PREFIX:

$ cmake ../llvm -G Ninja -C ../clang/cmake/caches/DistributionExample.cmake -DLLVM_ENABLE_PROJECTS=clang -DCMAKE_INSTALL_PREFIX=/home/wink/local-dist

With there is now a stage2-distribution and it started compiling but eventually failed:

++ ninja stage2-distribution -j11 -v
[1/1999] /usr/bin/c++  -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Support -I/home/wink/prgs/llvm/llvm-project-2/llvm/lib/Support -I/usr/include/libxml2 -Iinclude -I/home/wink/prgs/llvm/llvm-project-2/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG    -fno-exceptions -fno-rtti -MD -MT lib/Support/CMakeFiles/LLVMSupport.dir/Atomic.cpp.o -MF lib/Support/CMakeFiles/LLVMSupport.dir/Atomic.cpp.o.d -o lib/Support/CMakeFiles/LLVMSupport.dir/Atomic.cpp.o -c /home/wink/prgs/llvm/llvm-project-2/llvm/lib/Support/Atomic.cpp
...

[1997/1999] cd /home/wink/prgs/llvm/llvm-project-2/build-dist/tools/clang/stage2-bins && /usr/bin/cmake -DCMAKE_INSTALL_PREFIX=/home/wink/local-dist -C /home/wink/prgs/llvm/llvm-project-2/clang/cmake/caches/DistributionExample-stage2.cmake -DLLVM_ENABLE_LTO=ON -DLLVM_EXTERNAL_CLANG_SOURCE_DIR=/home/wink/prgs/llvm/llvm-project-2/llvm/../clang -DPACKAGE_VERSION=9.0.0svn -DPACKAGE_VENDOR=LLVM.org -DLLVM_VERSION_MAJOR=9 -DLLVM_VERSION_MINOR=0 -DLLVM_VERSION_PATCH=0 -DCLANG_VERSION_MAJOR=9 -DCLANG_VERSION_MINOR=0 -DCLANG_VERSION_PATCHLEVEL=0 -DLLVM_VERSION_SUFFIX=svn -DLLVM_BINUTILS_INCDIR= -DCLANG_REPOSITORY_STRING= -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -DLLVM_ENABLE_PROJECTS=clang -DCLANG_STAGE=stage2 -DCMAKE_CXX_COMPILER=/home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/clang++ -DCMAKE_C_COMPILER=/home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/clang -DCMAKE_ASM_COMPILER=/home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/clang -DCMAKE_ASM_COMPILER_ID=Clang -DCMAKE_AR=/home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/llvm-ar -DCMAKE_RANLIB=/home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/llvm-ranlib -GNinja /home/wink/prgs/llvm/llvm-project-2/llvm && /usr/bin/cmake -E touch /home/wink/prgs/llvm/llvm-project-2/build-dist/tools/clang/stage2-stamps//stage2-configure
loading initial cache file /home/wink/prgs/llvm/llvm-project-2/clang/cmake/caches/DistributionExample-stage2.cmake
-- The C compiler identification is Clang 9.0.0
-- The CXX compiler identification is Clang 9.0.0
-- The ASM compiler identification is Clang
-- Found assembler: /home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/clang
-- Check for working C compiler: /home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/clang
-- Check for working C compiler: /home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/clang -- works
...
CMake Error at CMakeLists.txt:1108 (message):
  Specified distribution component 'builtins' doesn't have a target


CMake Error at CMakeLists.txt:1114 (message):
  Specified distribution component 'builtins' doesn't have an install target


CMake Error at CMakeLists.txt:1120 (message):
  Specified distribution component 'builtins' doesn't have an
  install-stripped target.  Its installation target creation should be
  changed to use add_llvm_install_targets, or you should manually create the
  'install-builtins-stripped' target.


CMake Error at CMakeLists.txt:1108 (message):
  Specified distribution component 'runtimes' doesn't have a target


CMake Error at CMakeLists.txt:1114 (message):
  Specified distribution component 'runtimes' doesn't have an install target


CMake Error at CMakeLists.txt:1120 (message):
  Specified distribution component 'runtimes' doesn't have an
  install-stripped target.  Its installation target creation should be
  changed to use add_llvm_install_targets, or you should manually create the
  'install-runtimes-stripped' target.


-- Failed to find LLVM FileCheck
...
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- failed to compile
-- Configuring incomplete, errors occurred!
See also "/home/wink/prgs/llvm/llvm-project-2/build-dist/tools/clang/stage2-bins/CMakeFiles/CMakeOutput.log".
See also "/home/wink/prgs/llvm/llvm-project-2/build-dist/tools/clang/stage2-bins/CMakeFiles/CMakeError.log".
FAILED: tools/clang/stage2-stamps/stage2-configure 
cd /home/wink/prgs/llvm/llvm-project-2/build-dist/tools/clang/stage2-bins && /usr/bin/cmake -DCMAKE_INSTALL_PREFIX=/home/wink/local-dist -C /home/wink/prgs/llvm/llvm-project-2/clang/cmake/caches/DistributionExample-stage2.cmake -DLLVM_ENABLE_LTO=ON -DLLVM_EXTERNAL_CLANG_SOURCE_DIR=/home/wink/prgs/llvm/llvm-project-2/llvm/../clang -DPACKAGE_VERSION=9.0.0svn -DPACKAGE_VENDOR=LLVM.org -DLLVM_VERSION_MAJOR=9 -DLLVM_VERSION_MINOR=0 -DLLVM_VERSION_PATCH=0 -DCLANG_VERSION_MAJOR=9 -DCLANG_VERSION_MINOR=0 -DCLANG_VERSION_PATCHLEVEL=0 -DLLVM_VERSION_SUFFIX=svn -DLLVM_BINUTILS_INCDIR= -DCLANG_REPOSITORY_STRING= -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -DLLVM_ENABLE_PROJECTS=clang -DCLANG_STAGE=stage2 -DCMAKE_CXX_COMPILER=/home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/clang++ -DCMAKE_C_COMPILER=/home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/clang -DCMAKE_ASM_COMPILER=/home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/clang -DCMAKE_ASM_COMPILER_ID=Clang -DCMAKE_AR=/home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/llvm-ar -DCMAKE_RANLIB=/home/wink/prgs/llvm/llvm-project-2/build-dist/./bin/llvm-ranlib -GNinja /home/wink/prgs/llvm/llvm-project-2/llvm && /usr/bin/cmake -E touch /home/wink/prgs/llvm/llvm-project-2/build-dist/tools/clang/stage2-stamps//stage2-configure
ninja: build stopped: subcommand failed.G

So I'm guessing I need to "enable more projects" trying "clang-tools-extra-compiler-rt;libclc;libcxx;libcxxabi;libunwind" now.

beanz added a comment.May 19 2019, 2:45 PM

@winksaville, sorry those cache files pre-date the monorepo, and they haven’t been updated to support it. I will try and get a patch to update them today.

@winksaville, sorry those cache files pre-date the monorepo, and they haven’t been updated to support it. I will try and get a patch to update them today.

Thanks!

@winksaville I just committed an update to the DistributionExamples a minute ago that should get them working as long as you aren't using a Darwin host. If you are using a Darwin host you need D62155 as well.

@winksaville I just committed an update to the DistributionExamples a minute ago that should get them working as long as you aren't using a Darwin host. If you are using a Darwin host you need D62155 as well.

Txs, trying it now.

beanz updated this revision to Diff 200348.May 20 2019, 12:48 PM

Updates based on feedback from @smeenai

beanz added a comment.May 21 2019, 9:25 AM

This has had plenty of time to percolate, so I'm going to land it shorty. Any additional feedback we can incorporate in follow-up patches.

This revision was automatically updated to reflect the committed changes.