[docs] Build and publish documentation for the runtimes using LLVM_ENABLE_RUNTIMES

[ldionne]

Summary

Since LLVM_ENABLE_PROJECTS=libcxx;libcxxabi;libunwind are not supported
anymore, the correct way to build the documentation for the runtimes is
to use a CMake invocation rooted at <monorepo>/runtimes instead of using
a bootstrapping build. This is somewhat annoying because it requires the
addition of another top-level job for publishing the docs, however this
seems to be the cleanest way to achieve this.

Based on the discussion falling out of https://reviews.llvm.org/D132480.

[ldionne]

I *think* this should resolve the issues we saw with https://reviews.llvm.org/D132480. However, I have no idea how to test this, so I'm not sure what's the best way to proceed here.

[aaron.ballman]

In D134356#3805676, @ldionne wrote:

I *think* this should resolve the issues we saw with https://reviews.llvm.org/D132480. However, I have no idea how to test this, so I'm not sure what's the best way to proceed here.

If @andreil99 is able to speculatively try this patch out on the impacted documentation bots, that will hopefully tell us if the changes work?

[gkistanova]

The patch as is wouldn’t work.

UnifiedTreeBuilder.getCmakeBuildFactory does not support src_to_build_dir as this patch expects.
The path to built documentation is different.
And there is a few other small things here and there.

Andrei made needed changes and the builders are staged. Here is the runtime one - https://lab.llvm.org/staging/#/builders/229.

[ldionne]

In D134356#3814442, @gkistanova wrote:

The patch as is wouldn’t work.

UnifiedTreeBuilder.getCmakeBuildFactory does not support src_to_build_dir as this patch expects.
The path to built documentation is different.
And there is a few other small things here and there.

Andrei made needed changes and the builders are staged. Here is the runtime one - https://lab.llvm.org/staging/#/builders/229.

Awesome, thanks! I'm a bit confused though, I'm not seeing any changes pushed to llvm-zorg? Is that what you mean by staging? In any case, please feel free to make the necessary changes, but please let me know so I can abandon this revision in case it is superseded!

[andreil99]

Don't worry, I'll take it from here.

If you could check the build steps and check the published libcxx and libunwind documentation to make sure everything works as expected, that would help.
https://lab.llvm.org/staging/#/builders/229

[ldionne]

In D134356#3820900, @andreil99 wrote:

Don't worry, I'll take it from here.

If you could check the build steps and check the published libcxx and libunwind documentation to make sure everything works as expected, that would help.
https://lab.llvm.org/staging/#/builders/229

Sounds great, thanks! I am checking the logs at https://lab.llvm.org/staging/#/builders/229/builds/13 right now:

• The CMake configuration seems OK, but -DLLVM_ENABLE_ASSERTIONS=OFF is being passed and it's superfluous. It doesn't hurt, though, if it's simpler for you to keep it.
• The ninja command to build the docs look OK.
• I wasn't able to find where the staging docs are published, so I couldn't check the output. I'll do that if you let me know where they are!

[gkistanova]

The docs are published to the normal places: https://libcxx.llvm.org/ and https://libcxxabi.llvm.org/ respectively.

[ldionne]

In D134356#3824388, @gkistanova wrote:

The docs are published to the normal places: https://libcxx.llvm.org/ and https://libcxxabi.llvm.org/ respectively.

Great, everything looks correct then!

buildbot/osuosl/master/config/builders.py

# Builders responsible building Sphinx documentation.
     'workernames' : ["gribozavr3"],
     'builddir': "lld-sphinx-docs",
     'factory' : SphinxDocsBuilder.getSphinxDocsBuildFactory(lld_html=True)},
 
     {'name':"libunwind-sphinx-docs",
     'tags'  : ["libunwind", "doc"],
     'workernames':["gribozavr3"],
     'builddir':"libunwind-sphinx-docs",
-    'factory': SphinxDocsBuilder.getSphinxDocsBuildFactory(libunwind_html=True)},
+    'factory': SphinxDocsBuilder.getSphinxRuntimesDocsBuildFactory(libunwind_html=True)},
 
     # Sphinx doc Publisher
     {'name' : "publish-sphinx-docs",
     'tags'  : ["doc"],
     'workernames' : ["as-worker-4"],
     'builddir': "publish-sphinx-docs",
     'factory' : SphinxDocsBuilder.getLLVMDocsBuildFactory(clean=True)},
 
+    {'name' : "publish-runtimes-sphinx-docs",
+    'tags'  : ["doc"],
+    'workernames' : ["as-worker-4"],
+    'builddir': "publish-runtimes-sphinx-docs",
+    'factory' : SphinxDocsBuilder.getLLVMRuntimesDocsBuildFactory(clean=True)},
+
     {'name' : "publish-lnt-sphinx-docs",
     'tags'  : ["doc"],
     'workernames' : ["as-worker-4"],
     'builddir': "publish-lnt-sphinx-docs",
     'factory' : HtmlDocsBuilder.getHtmlDocsBuildFactory()},
 
     {'name' : "publish-doxygen-docs",
     'tags'  : ["doc"],

zorg/buildbot/builders/SphinxDocsBuilder.py

 from collections import OrderedDict
+import os
 
 from buildbot.steps.shell import ShellCommand
 
 from zorg.buildbot.builders import UnifiedTreeBuilder
 from zorg.buildbot.commands.CmakeCommand import CmakeCommand
 from zorg.buildbot.commands.NinjaCommand import NinjaCommand
 
 llvm_docs = OrderedDict([

 
 
 def getSphinxDocsBuildFactory(
         llvm_html         = False, # Build LLVM HTML documentation
         llvm_man          = False, # Build LLVM man pages
         clang_html        = False, # Build Clang HTML documentation
         clang_tools_html  = False, # Build Clang Extra Tools HTML documentation
         lld_html          = False, # Build LLD HTML documentation
-        libcxx_html       = False, # Build Libc++ HTML documentation
-        libunwind_html    = False, # Build libunwind HTML documentation
         lldb_html         = False, # Build LLDB HTML documentation
         polly_html        = False, # Build Polly HTML documentation
         extra_configure_args = None,
         **kwargs):
 
     if extra_configure_args:
         cmake_args = extra_configure_args[:]
     else:
def getSphinxDocsBuildFactory(
     if clang_html or clang_tools_html or lldb_html:
         depends_on_projects.append('clang')
     if clang_tools_html:
         depends_on_projects.append('clang-tools-extra')
     if lld_html:
         depends_on_projects.append('lld')
     if lldb_html:
         depends_on_projects.append('lldb')
-    if libcxx_html:
-        depends_on_projects.append('libcxx')
-        depends_on_projects.append('libcxxabi')
-    if libunwind_html:
-        depends_on_projects.append('libunwind')
     if polly_html:
         depends_on_projects.append('polly')
 
+
     f = UnifiedTreeBuilder.getCmakeBuildFactory(
             depends_on_projects=depends_on_projects,
-            enable_runtimes=[], # Docs don't support runtimes build yet.
             llvm_srcdir=llvm_srcdir,
             obj_dir=llvm_objdir,
             extra_configure_args=cmake_args,
             **kwargs) # Pass through all the extra arguments.
 
     if llvm_html:
         f.addStep(NinjaCommand(name="docs-llvm-html",
                                haltOnFailure=True,
def getSphinxDocsBuildFactory(
     if lldb_html:
         f.addStep(NinjaCommand(name="docs-lldb-html",
                                haltOnFailure=True,
                                description=["Build LLDB Sphinx HTML documentation"],
                                workdir=llvm_objdir,
                                targets=['docs-lldb-html']
                               ))
 
+    if polly_html:
+        f.addStep(NinjaCommand(name="docs-polly-html",
+                               haltOnFailure=True,
+                               description=["Build Polly Sphinx HTML documentation"],
+                               workdir=llvm_objdir,
+                               targets=['docs-polly-html']
+                              ))
+
+    return f
+
+def getSphinxRuntimesDocsBuildFactory(
+        libcxx_html       = False, # Build Libc++ HTML documentation
+        libunwind_html    = False, # Build libunwind HTML documentation
+        extra_configure_args = None,
+        **kwargs):
+
+    if extra_configure_args:
+        cmake_args = extra_configure_args[:]
+    else:
+        cmake_args = list()
+
+    # Set proper defaults for the config flags.
+    CmakeCommand.applyDefaultOptions(cmake_args, [
+      ('-G',                    'Ninja'),
+      ('-DLLVM_ENABLE_SPHINX=', 'ON'),
+      ('-DSPHINX_OUTPUT_HTML=', 'ON'),
+      ('-DSPHINX_OUTPUT_MAN=',  'ON'),
+      ('-DLLDB_INCLUDE_TESTS=', 'OFF'),
+      ('-DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=', 'ON'),
+      ('-DLLVM_ENABLE_ASSERTIONS=',  'OFF'),
+      ])
+
+    llvm_srcdir = 'llvm/src'
+    llvm_objdir = 'llvm/build'
+
+    depends_on_runtimes = []
+    if libcxx_html:
+        depends_on_runtimes.append('libcxx')
+        depends_on_runtimes.append('libcxxabi')
+    if libunwind_html:
+        depends_on_runtimes.append('libunwind')
+
+
+    f = UnifiedTreeBuilder.getCmakeBuildFactory(
+            enable_runtimes=depends_on_runtimes,
+            llvm_srcdir=llvm_srcdir,
+            src_to_build_dir=os.path.join(llvm_srcdir, 'runtimes'),
+            obj_dir=llvm_objdir,
+            extra_configure_args=cmake_args,
+            **kwargs) # Pass through all the extra arguments.
+
     if libcxx_html:
         f.addStep(NinjaCommand(name="docs-libcxx-html",
                                haltOnFailure=True,
                                description=["Build Libc++ Sphinx HTML documentation"],
                                workdir=llvm_objdir,
                                targets=['docs-libcxx-html']
                               ))
 
     if libunwind_html:
         f.addStep(NinjaCommand(name="docs-libunwind-html",
                                haltOnFailure=True,
                                description=["Build libunwind Sphinx HTML documentation"],
                                workdir=llvm_objdir,
                                targets=['docs-libunwind-html']
                               ))
 
-    if polly_html:
-        f.addStep(NinjaCommand(name="docs-polly-html",
-                               haltOnFailure=True,
-                               description=["Build Polly Sphinx HTML documentation"],
-                               workdir=llvm_objdir,
-                               targets=['docs-polly-html']
-                              ))
-
     return f
 
 
 def getLLVMDocsBuildFactory(
         clean = False,
         depends_on_projects = None,
         extra_configure_args = None,
         env = None,
         **kwargs):
 
     if depends_on_projects is None:
         # All the projects by default.
         _depends_on_projects=[
             "llvm",
             "clang",
             "clang-tools-extra",
             "libc",
-            "libcxx",
-            "libcxxabi",
-            "libunwind",
             "lld",
             "lldb",
             "flang",
             "openmp",
             "polly",
         ]
     else:
         # Make a local copy of depends_on_projects, as we are going to modify
         # that.
         _depends_on_projects=depends_on_projects[:]
         # Some projects are interdependent for the purpose of documentation.
         # Enforce the dependencies.
         if ("clang-tools-extra" in _depends_on_projects or \
             "lldb" in _depends_on_projects
            ) and "clang" not in _depends_on_projects:
             _depends_on_projects.append("clang")
-        if "libcxx" in _depends_on_projects and \
-           "libcxxabi" not in _depends_on_projects:
-            _depends_on_projects.append("libcxxabi")
-        if "libcxxabi" in _depends_on_projects and \
-           "libcxx" not in _depends_on_projects:
-            _depends_on_projects.append("libcxx")
 
     # Make a local copy of the configure args, as we are going to modify that.
     if extra_configure_args:
         cmake_args = extra_configure_args[:]
     else:
         cmake_args = list()
 
     # Prepare environmental variables. Set here all env we want everywhere.
def getLLVMDocsBuildFactory(
     docs = [
         llvm_docs[project] for project in llvm_docs.keys()
         if project in _depends_on_projects
     ]
 
     f = UnifiedTreeBuilder.getCmakeBuildFactory(
             clean=clean,
             depends_on_projects=_depends_on_projects,
-            enable_runtimes=[], # Docs don't support runtimes build yet.
+            extra_configure_args=cmake_args,
+            env=merged_env,
+            **kwargs) # Pass through all the extra arguments.
+
+    UnifiedTreeBuilder.addNinjaSteps(
+        f,
+        targets=[d[0] for d in docs],
+        checks=[],
+        env=merged_env,
+        **kwargs)
+
+    # Publish just built documentation
+    for target, local_path, remote_path in docs:
+        f.addStep(
+            ShellCommand(
+                name="Publish {}".format(target),
+                description=[
+                    "Publish", "just", "built", "documentation", "for",
+                    "{}".format(target),
+                    ],
+                command=[
+                    'rsync',
+                    '-vrl',
+                    '--delete', '--force', '--delay-updates', '--delete-delay',
+                    '--ignore-times',
+                    '--checksum',
+                    '-p', '--chmod=Du=rwx,Dg=rwx,Do=rx,Fu=rw,Fg=rw,Fo=r',
+                    "{}".format(local_path),
+                    "lists.llvm.org:web/{}".format(remote_path),
+                    ],
+                env=merged_env,
+            )
+        )
+
+    return f
+
+def getLLVMRuntimesDocsBuildFactory(
+        clean = False,
+        depends_on_runtimes = None,
+        extra_configure_args = None,
+        env = None,
+        **kwargs):
+
+    if depends_on_runtimes is None:
+        # All the projects by default.
+        _depends_on_runtimes=[
+            "libcxx",
+            "libcxxabi",
+            "libunwind",
+        ]
+    else:
+        # Make a local copy of depends_on_runtimes, as we are going to modify
+        # that.
+        _depends_on_runtimes=depends_on_runtimes[:]
+        # Some runtimes are interdependent for the purpose of documentation.
+        # Enforce the dependencies.
+        if "libcxx" in _depends_on_runtimes and \
+           "libcxxabi" not in _depends_on_runtimes:
+            _depends_on_runtimes.append("libcxxabi")
+        if "libcxxabi" in _depends_on_runtimes and \
+           "libcxx" not in _depends_on_runtimes:
+            _depends_on_runtimes.append("libcxx")
+
+    # Make a local copy of the configure args, as we are going to modify that.
+    if extra_configure_args:
+        cmake_args = extra_configure_args[:]
+    else:
+        cmake_args = list()
+
+    # Prepare environmental variables. Set here all env we want everywhere.
+    merged_env = {
+        'TERM' : 'dumb' # Be cautious and disable color output from all tools.
+    }
+    if env is not None:
+        # Overwrite pre-set items with the given ones, so user can set anything.
+        merged_env.update(env)
+
+    CmakeCommand.applyDefaultOptions(cmake_args, [
+        ("-G",                           "Ninja"),
+        ("-DLLVM_ENABLE_SPHINX=",        "ON"),
+        ("-DSPHINX_OUTPUT_HTML=",        "ON"),
+        ("-DSPHINX_OUTPUT_MAN=",         "OFF"),
+        ("-DSPHINX_WARNINGS_AS_ERRORS=", "OFF"),
+        ("-DLLVM_ENABLE_ASSERTIONS=",    "OFF"),
+        ("-DCMAKE_BUILD_TYPE=",          "Release"),
+        ])
+
+    # Build docs for each of the runtimes this builder depends on
+    docs = [
+        llvm_docs[project] for project in llvm_docs.keys()
+        if project in _depends_on_runtimes
+    ]
+
+    f = UnifiedTreeBuilder.getCmakeBuildFactory(
+            clean=clean,
+            depends_on_projects=[],
+            enable_runtimes=_depends_on_runtimes,
+            src_to_build_dir=os.path.join(llvm_srcdir, 'runtimes'),
             extra_configure_args=cmake_args,
             env=merged_env,
             **kwargs) # Pass through all the extra arguments.
 
     UnifiedTreeBuilder.addNinjaSteps(
         f,
         targets=[d[0] for d in docs],
         checks=[],