[lldb][docs] Update documentation for monorepo and CMake caches

[sgraenitz]

Summary

The lldb build system made good progress in the last months, but documentation was still lacking behind. Here's a patch to catch up. It would be nice to get an update also to the release/9.x branch. What do you think?

[teemperor]

LGTM (beside the standalone build section which I have no experience with).

[sgraenitz]

View rendered HTML output here: http://weliveindetail.github.io/blog/res/lldb-docs/resources/build.html

[labath]

Looks good to me.

[jryans]

In general, this looks great overall! I added few comments on things that would be good to clarify.

lldb/docs/resources/build.rst

108

Nit: It would be nice to have this link to the CMake Options paragraph.

119

Is the casing of Clang_DIR important here? (In most cases, CMake defines are all upper case, but perhaps this one is special?) If the casing is important, I would suggest noting that here. If it's not important, it's probably simpler go all upper case to match other usages.

270

It's unclear from reading the paragraph if using these caches is recommended for most users. If it is, it would helpful to state that. In general, some kind of statement that answers "who should use this" would help.

288

Nit: Standalone -> standalone perhaps?

[sgraenitz]

Add links to sections

[sgraenitz]

Polish section Standalone builds

[sgraenitz]

Polish section CMake caches

[jryans]

Great! 😁 Thanks for working on this!

[sgraenitz]

Updated rendered HTML: http://weliveindetail.github.io/blog/res/lldb-docs/resources/build.html

[sgraenitz]

Since Visual Studio is the only section in Building LLDB with CMake and Other Generators now, I would like to:

• move it merge it with another one, preferably Common CMake options > Windows
• rename Building LLDB with CMake & Ninja into Building LLDB with CMake (analog and with a link to http://llvm.org/docs/CMake.html)

Would that be acceptable?

[jryans]

In D65330#1602583, @sgraenitz wrote:

Since Visual Studio is the only section in Building LLDB with CMake and Other Generators now, I would like to:

• move it merge it with another one, preferably Common CMake options > Windows
• rename Building LLDB with CMake & Ninja into Building LLDB with CMake (analog and with a link to http://llvm.org/docs/CMake.html)

Would that be acceptable?

Yes, this sounds like a good plan.

[sgraenitz]

Polishing

[sgraenitz]

Merge Visual Studio build instructions into Common CMake options > Windows

[sgraenitz]

Since Visual Studio is the only section in Building LLDB with CMake and Other Generators now, ...

Yes, this sounds like a good plan.

Cool, patch Diff 211951 has the change. Here's the difference in generated output:
https://weliveindetail.github.io/blog/res/lldb-docs/resources/build.html
https://weliveindetail.github.io/blog/res/lldb-docs/resources/build-merged-vs.html

[sgraenitz]

I added a few links to LLVM documentation here and wondered what to do with them, if this gets cherry-picked to release/9.x, e.g. https://llvm.org/docs/CMake.html Ideally they would point to their 9.x counterparts, e.g. https://releases.llvm.org/9.0.0/docs/CMake.html and with 9.0.1 it should become https://releases.llvm.org/9.0.1/docs/CMake.html

llvm/docs/AdvancedBuilds.rst links within one build directory using ":doc:CMake", but apparently this doesn't work cross-project?

I could imaging to patch it once for the release and use configure_file from CMake to import the current version, but I am no sure if it's worth the effort =)

Is there a simple way to do this in Sphinx? Ideas / opinions?

[jryans]

In D65330#1602715, @sgraenitz wrote:

Cool, patch Diff 211951 has the change. Here's the difference in generated output:
https://weliveindetail.github.io/blog/res/lldb-docs/resources/build.html
https://weliveindetail.github.io/blog/res/lldb-docs/resources/build-merged-vs.html

Thanks, this looks good to me.

[jryans]

In D65330#1602737, @sgraenitz wrote:

I added a few links to LLVM documentation here and wondered what to do with them, if this gets cherry-picked to release/9.x, e.g. https://llvm.org/docs/CMake.html Ideally they would point to their 9.x counterparts, e.g. https://releases.llvm.org/9.0.0/docs/CMake.html and with 9.0.1 it should become https://releases.llvm.org/9.0.1/docs/CMake.html

llvm/docs/AdvancedBuilds.rst links within one build directory using ":doc:CMake", but apparently this doesn't work cross-project?

I could imaging to patch it once for the release and use configure_file from CMake to import the current version, but I am no sure if it's worth the effort =)

Is there a simple way to do this in Sphinx? Ideas / opinions?

Hmm, are there the LLDB docs archived per release? For LLVM overall, there's https://releases.llvm.org/ with links to say https://releases.llvm.org/8.0.0/docs/index.html, but LLDB doesn't seem to appear in sub-project list of versioned docs... I would say it's fine to not worry about this for now, and we can improve in a separate patch if needed.

[amccarth]

Looks good. Thanks for doing these updates!

lldb/docs/resources/build.rst

85

"Check out" should be two words here.

[stella.stamenova]

lldb/docs/resources/build.rst

94

Is Ninja the recommended build tool on Windows as well?

105

I thought some of the tests need lld as well

237

We retired VS2015 support

[sgraenitz]

In D65330#1602756, @jryans wrote:

Hmm, are there the LLDB docs archived per release? [...] LLDB doesn't seem to appear in sub-project list of versioned docs

Right, Clang docs are here: https://releases.llvm.org/8.0.0/tools/clang/docs/index.html, but it's not the same as http://clang.llvm.org/
Apparently, LLDB docs aren't archived. Where did you find the sub-project list of versioned docs?

[jryans]

In D65330#1602779, @sgraenitz wrote:

Where did you find the sub-project list of versioned docs?

On http://releases.llvm.org/, for some recent versions like 8.0.0, the "Documentation" column has separate versioned links for "llvm clang lld clang-extra libc++ polly", but lldb doesn't appear there.

[sgraenitz]

In D65330#1602786, @jryans wrote:

On http://releases.llvm.org/, for some recent versions like 8.0.0, the "Documentation" column has separate versioned links for "llvm clang lld clang-extra libc++ polly", but lldb doesn't appear there.

Interesting. Thks!

lldb/docs/resources/build.rst

94

Well it's recommended for LLVM and I used it for LLDB on Windows in the past.
Maybe @zturner knows more about it?

105

So far, the documentation didn't mention lld. I would recommend using lld to build LLDB on Linux, but AFAIK it's not required to build lld along with LLDB.

237

What generator string would you like to see here instead? (So far I just kept what was there.)

[stella.stamenova]

lldb/docs/resources/build.rst

94

LLDB does build with both Ninja and VS on Windows. I don't know that either one is "recommended" though and we usually build with VS.

105

For example, lldb\lit\SymbolFile\DWARF\debug-types-basic.test contains this:

REQUIRES: lld

There are other tests as well that require lld.

237

cmake -G "Visual Studio 15 2017 Win64" -Thost=x64

[amccarth]

lldb/docs/resources/build.rst

94

I use Ninja exclusively for both LLVM in general and LLDB in particular, regardless of whether I'm building with MSVC or bootstrapping with Clang.

Having a VS solution is great for finding your way around the repo and debugging. But Ninja seems to build much faster.

[clayborg]

Can we add a section for "Building LLDB with Xcode"? We had a shell script patch that was going to be checked in, not sure if it made it. But it did a two part thing where it built LLVM and clang and then generated an Xcode project. I would rather just produce the Xcode project using cmake, very similar to other instructions. My mac cmake line has:

cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;libcxx;lldb" -DCMAKE_BUILD_TYPE:STRING=Debug -DLLVM_ENABLE_ASSERTIONS:BOOL=TRUE -DLLDB_BUILD_FRAMEWORK:BOOL=TRUE -DLLDB_USE_SYSTEM_DEBUGSERVER=ON /path/to/llvm-project/llvm

Important bits are the projects being set to "clang;libcxx;lldb" (include libcxx) and building the framework and using the system debugserver

[sgraenitz]

Address recent feedback

[sgraenitz]

Hi Greg, thanks for your feedback.

Can we add a section for "Building LLDB with Xcode"? We had a shell script patch that was going to be checked in, not sure if it made it. But it did a two part thing where it built LLVM and clang and then generated an Xcode project.

"Build LLDB standalone for development with Xcode" describes it, see http://tiny.cc/i5ncaz#common-configurations-on-macos

I would rather just produce the Xcode project using cmake, very similar to other instructions.

This means changing the generator from Ninja to Xcode. I added a short note that other generators like Xcode or Visual Studio may be used as well (maybe we can avoid calling it "support", because in fact multi-config generators cause a lot of trouble).

My mac cmake line has:

cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;libcxx;lldb" -DCMAKE_BUILD_TYPE:STRING=Debug -DLLVM_ENABLE_ASSERTIONS:BOOL=TRUE -DLLDB_BUILD_FRAMEWORK:BOOL=TRUE -DLLDB_USE_SYSTEM_DEBUGSERVER=ON /path/to/llvm-project/llvm

LLVM_ENABLE_ASSERTIONS and many other important flags are documented in LLVM (linked from http://tiny.cc/i5ncaz#building-lldb-with-cmake).
CMAKE_BUILD_TYPE=Debug is the default
LLDB_BUILD_FRAMEWORK and LLDB_USE_SYSTEM_DEBUGSERVER are explained in the macOS section, see http://tiny.cc/i5ncaz#id2.

Important bits are the projects being set to "clang;libcxx;lldb" (include libcxx) and building the framework and using the system debugserver

Added an explanation for the test suite libcxx dependency to the macOS section too.

lldb/docs/resources/build.rst

94

I added a short note that other generators like Xcode or Visual Studio may be used as well (please see my answer to Greg).

[sgraenitz]

Feedback and polishing; latest rendered output here: http://tiny.cc/i5ncaz

[jryans]

Current version looks great to me! 😁

lldb/docs/resources/build.rst

90

This could be done in a separate patch, but I happened to notice that Get Involved section on the LLDB main page currently suggests Subversion at the moment. It would be nice to have this updated to Git as well.

[labath]

If we're going to be mentioning test dependencies, then it probably makes sense to mention lld too. AFAIK, lld is required for running the (dotest) test suite on windows. On other platforms it is not required, but there are tests which will only get run if lld is enabled.

[sgraenitz]

Thanks for your feedback.

In D65330#1604355, @labath wrote:

lld is required for running the (dotest) test suite on windows. On other platforms it is not required, but there are tests which will only get run if lld is enabled.

I mentioned this in Regular in-tree builds as: Parts of the LLDB test suite require lld. Add it to the list in order to run all tests.

lldb/docs/resources/build.rst

90

+1 (in a separate patch)

[labath]

Thanks for putting this together.

[stella.stamenova]

I think you need one small change for lld still - to make it clear that it is required on Windows in the Windows section, but it looks good otherwise. Please fix that before committing.

lldb/docs/resources/build.rst

192

The windows section should include a similar note to the macOS section for libcxx - but for lld.

[sgraenitz]

Add lld note to Windows section

[sgraenitz]

lldb/docs/resources/build.rst

90

D65437

lldb/docs/resources/build.rst

 macOS
 *****
 
 * To use the in-tree debug server on macOS, lldb needs to be code signed. For
   more information see :ref:`CodeSigning` below.
 * If you are building both Clang and LLDB together, be sure to also check out
   libc++, which is a required for testing on macOS.
 
-Building LLDB with CMake & Ninja
---------------------------------
+Building LLDB with CMake
+------------------------
+
+The LLVM project is migrating to a single monolithic respository for LLVM and
+its subprojects. This is the recommended way to build LLDB. Check out the
+source-tree with git:
+
+::
+
+  > git clone https://github.com/llvm/llvm-project.git
 
 CMake is a cross-platform build-generator tool. CMake does not build the
-project, it generates the files needed by your build tool. Assuming you're
-using Ninja, the invocation looks like this:
+project, it generates the files needed by your build tool. The recommended
+build tool for LLVM is Ninja, but other generators like Xcode or Visual Studio
+may be used as well. Please also read `Building LLVM with CMake
+<http://llvm.org/docs/CMake.html>`_.
+
+Regular in-tree builds
+**********************
+
+Create a new directory for your build-tree. From there run CMake and point it
+to the ``llvm`` directory in the source-tree:
 
 ::
 
-  > cmake -G Ninja <cmake variables> <path to root of llvm source tree>
+  > cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;lldb" [<cmake options>] path/to/llvm-project/llvm
 
-Once CMake has configured your build, you can run ``ninja`` to build LLDB.
+We used the ``LLVM_ENABLE_PROJECTS`` option here to tell the build-system which
+subprojects to build in addition to LLVM (for more options see
+:ref:`CommonCMakeOptions` and :ref:`CMakeCaches`). Parts of the LLDB test suite
+require ``lld``. Add it to the list in order to run all tests. Once CMake is done,
+run ninja to perform the actual build. We pass ``lldb`` here as the target, so
+it only builds what is necessary to run the lldb driver:
 
 ::
 
   > ninja lldb
 
+Standalone builds
+*****************
+
+This is another way to build LLDB. We can use the same source-tree as we
+checked out above, but now we will have two build-trees:
+
+* the main build-tree for LLDB in ``/path/to/lldb-build``
+* a provided build-tree for LLVM and Clang in ``/path/to/llvm-build``
+
+Run CMake with ``-B`` pointing to a new directory for the provided build-tree
+and the positional argument pointing to the ``llvm`` directory in the
+source-tree. Note that we leave out LLDB here and only include Clang.
+Then we build the ``ALL`` target with ninja:
+
+::
+
+  > cmake -B /path/to/llvm-build -G Ninja \
+          -DLLVM_ENABLE_PROJECTS=clang \
+          [<more cmake options>] /path/to/llvm-project/llvm
+  > ninja
+
+Now run CMake a second time with ``-B`` pointing to a new directory for the
+main build-tree and the positional argument pointing to the ``lldb`` directory
+in the source-tree. In order to find the provided build-tree, the build-system
+needs the options ``LLVM_DIR`` and ``Clang_DIR`` (CMake variables are
+case-sensitive!):
+
+::
+
+  > cmake -B /path/to/lldb-build -G Ninja \
+          -DLLVM_DIR=/path/to/llvm-build/lib/cmake/llvm \
+          -DClang_DIR=/path/to/llvm-build/lib/cmake/clang \
+          [<more cmake options>] /path/to/llvm-project/lldb
+  > ninja lldb
+
+
+.. _CommonCMakeOptions:
+
+Common CMake options
+********************
+
 Following is a description of some of the most important CMake variables which
 you are likely to encounter. A variable FOO is set by adding ``-DFOO=value`` to
 the CMake command line.
 
 If you want to debug the lldb that you're building -- that is, build it with
 debug info enabled -- pass two additional arguments to cmake before running
 ninja:
 
> cmake -G Ninja \
       -DLLDB_TEST_USE_CUSTOM_CXX_COMPILER=On \
       -DLLDB_TEST_C_COMPILER=<path to C compiler> \
       -DLLDB_TEST_CXX_COMPILER=<path to C++ compiler> \
       <path to root of llvm source tree>
 
 It is strongly recommend to use a release build for the compiler to speed up
 test execution.
 
 Windows
-*******
+^^^^^^^
 
 Although the following CMake variables are by no means Windows specific, they
 are commonly used on Windows.
 
 * ``LLDB_TEST_DEBUG_TEST_CRASHES`` (Default=0): If set to 1, will cause Windows
   to generate a crash dialog whenever lldb.exe or the python extension module
   crashes while running the test suite. If set to 0, LLDB will silently crash.
   Setting to 1 allows a developer to attach a JIT debugger at the time of a
::
 
   > cmake -G Ninja^
       -DLLDB_TEST_DEBUG_TEST_CRASHES=1^
       -DPYTHON_HOME=C:\Python35^
       -DLLDB_TEST_USE_CUSTOM_C_COMPILER=ON^
       -DLLDB_TEST_C_COMPILER=d:\src\llvmbuild\ninja_release\bin\clang.exe^
       <path to root of llvm source tree>
 
+
+Building with ninja is both faster and simpler than building with Visual Studio,
+but chances are you still want to debug LLDB with an IDE. One solution is to run
+cmake twice and generate the output into two different folders. One for
+compiling (the ninja folder), and one for editing, browsing and debugging.
+
+Follow the previous instructions in one directory, and generate a Visual Studio
+project in another directory.
+
+::
+
+  > cmake -G "Visual Studio 15 2017 Win64" -Thost=x64 <cmake variables> <path to root of llvm source tree>
+
+Then you can open the .sln file in Visual Studio, set lldb as the startup
+project, and use F5 to run it. You need only edit the project settings to set
+the executable and the working directory to point to binaries inside of the
+ninja tree.
+
+
 NetBSD
-******
+^^^^^^
 
 Current stable NetBSD release doesn't ship with libpanel(3), therefore it's
 required to disable curses(3) support with the
 ``-DLLDB_DISABLE_CURSES:BOOL=TRUE`` option. To make sure check if
 ``/usr/include/panel.h`` exists in your system.
 
 macOS
-*****
+^^^^^
 
-Here are some commonly used LLDB-specific CMake variables on macOS.
+On macOS the LLDB test suite requires libc++. Either add ``libcxx`` to
+``LLVM_ENABLE_PROJECTS`` or disable the test suite with
+``LLDB_ENABLE_TESTS=OFF``. Further useful options:
 
-* ``LLDB_BUILD_FRAMEWORK:BOOL`` : Builds the LLDB.framework.
-* ``LLDB_CODESIGN_IDENTITY:STRING`` : Determines the codesign identity to use.
-  An empty string means skip building debugserver to avoid codesigning.
+* ``LLDB_BUILD_FRAMEWORK:BOOL``: Builds the LLDB.framework.
+* ``LLDB_CODESIGN_IDENTITY:STRING``: Set the identity to use for code-signing
+  all executables. If not explicitly specified, only ``debugserver`` will be
+  code-signed with identity ``lldb_codesign`` (see :ref:`CodeSigning`).
+* ``LLDB_USE_SYSTEM_DEBUGSERVER:BOOL``: Use the system's debugserver, so lldb is
+  functional without setting up code-signing.
 
-Building LLDB with CMake and Other Generators
----------------------------------------------
 
-Compiling with ninja is both faster and simpler than compiling with MSVC or
-Xcode, but chances are you still want to debug LLDB with those IDEs. One
-solution to this is to run cmake twice and generate the output into two
-different folders. One for compiling (the ninja folder), and one for editing,
-browsing and debugging.
+.. _CMakeCaches:
 
+CMake caches
+************
 
-Visual Studio
-*************
+CMake caches allow to store common sets of configuration options in the form of
+CMake scripts and can be useful to reproduce builds for particular use-cases
+(see by analogy `usage in LLVM and Clang <http://llvm.org/docs/AdvancedBuilds.html>`_).
+A cache is passed to CMake with the ``-C`` flag, following the absolute path to
+the file on disk. Subsequent ``-D`` options are still allowed. Please find the
+currently available caches in the `lldb/cmake/caches/
+<https://github.com/llvm/llvm-project/tree/master/lldb/cmake/caches>`_
+directory.
 
-Follow the previous instructions in one directory, and generate a Visual Studio
-project in another directory.
+Common configurations on macOS
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Build, test and install a distribution of LLDB from the `monorepo
+<https://github.com/llvm/llvm-project>`_ (see also `Building a Distribution of
+LLVM <http://llvm.org/docs/BuildingADistribution.html>`_):
 
 ::
 
-  > cmake -G "Visual Studio 14 2015" <cmake variables> <path to root of llvm source tree>
+  > git clone https://github.com/llvm/llvm-project
 
-Then you can open the .sln file in Visual Studio, set lldb as the startup
-project, and use F5 to run it. You need only edit the project settings to set
-the executable and the working directory to point to binaries inside of the
-ninja tree.
+  > cmake -B /path/to/lldb-build -G Ninja \
+          -C /path/to/llvm-project/lldb/cmake/caches/Apple-lldb-macOS.cmake \
+          -DLLVM_ENABLE_PROJECTS="clang;libcxx;lldb" \
+          llvm-project/llvm
 
-Xcode
-*****
+  > DESTDIR=/path/to/lldb-install ninja -C /path/to/lldb-build check-lldb install-distribution
 
-Follow the previous instructions in one directory, and generate an Xcode
-project in another directory.
+Build LLDB standalone for development with Xcode:
 
 ::
 
-  > cmake -G Xcode <cmake variables> <path to root of llvm source tree>
+  > git clone https://github.com/llvm/llvm-project
+
+  > cmake -B /path/to/llvm-build -G Ninja \
+          -C /path/to/llvm-project/lldb/cmake/caches/Apple-lldb-base.cmake \
+          -DLLVM_ENABLE_PROJECTS="clang;libcxx" \
+          llvm-project/llvm
+  > ninja -C /path/to/llvm-build
+
+  > cmake -B /path/to/lldb-build -G Xcode \
+          -C /path/to/llvm-project/lldb/cmake/caches/Apple-lldb-Xcode.cmake \
+          -DLLVM_DIR=/path/to/llvm-build/lib/cmake/llvm \
+          -DClang_DIR=/path/to/llvm-build/lib/cmake/clang \
+          llvm-project/lldb
+  > open lldb.xcodeproj
+  > cmake --build /path/to/lldb-build --target check-lldb
 
 
 Building The Documentation
 --------------------------
 
 If you wish to build the optional (reference) documentation, additional
 dependencies are required:
 
 * Sphinx (for the website)
 * Graphviz (for the 'dot' tool)
 * doxygen (if you wish to build the C++ API reference)
 * epydoc (if you wish to build the Python API reference)
 
 To install the prerequisites for building the documentation (on Debian/Ubuntu)
 do:
 
 ::
 
   > sudo apt-get install doxygen graphviz python3-sphinx
   > sudo pip install epydoc
 
-To build the documentation, build the desired target(s).
+To build the documentation, configure with ``LLVM_ENABLE_SPHINX=ON`` and build the desired target(s).
 
 ::
 
   > ninja docs-lldb-html
   > ninja lldb-cpp-doc
   > ninja lldb-python-doc
 
 Cross-compiling LLDB

 
 To use the in-tree debug server on macOS, lldb needs to be code signed. The
 Debug, DebugClang and Release builds are set to code sign using a code signing
 certificate named ``lldb_codesign``. This document explains how to set up the
 signing certificate.
 
 Note that it's possible to build and use lldb on macOS without setting up code
 signing by using the system's debug server. To configure lldb in this way with
-cmake, specify ``-DLLDB_CODESIGN_IDENTITY=''``.
+cmake, specify ``-DLLDB_USE_SYSTEM_DEBUGSERVER=ON``.
 
 If you have re-installed a new OS, please delete all old ``lldb_codesign`` items
 from your keychain. There will be a code signing certification and a public
 and private key. Reboot after deleting them. You will also need to delete and
 build folders that contained old signed items. The darwin kernel will cache
 code signing using the executable's file system node, so you will need to
 delete the file so the kernel clears its cache.