[Flang] Generate documentation for compiler flags

[DylanFleming-arm]

Summary

This patch aims to create a webpage to document
Flang's command line options on https://flang.llvm.org/docs/
in a similar way to Clang's
https://clang.llvm.org/docs/ClangCommandLineReference.html

This is done by using clang_tablegen to generate an .rst
file from Options.td (which is current shared with Clang)
For this to work, ClangOptionDocEmitter.cpp was updated
to allow specific Flang flags to be included,
rather than bulk excluding clang flags.

Note:
Some headings in the generated documentation will incorrectly
contain references to Clang, e.g.
"Flags controlling the behaviour of Clang during compilation"
This is because Options.td (Which is shared between both Clang and Flang)
contains hard-coded DocBrief sections. I couldn't find a non-intrusive way
to make this target-dependant, as such I've left this as is, and it will need revisiting later.

[awarzynski]

Makes sense, thanks for working on this! Some minor comments below and inline.

From your summary:

This is done by using clang tablegen

What do you mean by "clang tablegen"? Is it Clang's clang_tablegen CMake function?

from options.td

Options.td ;-)

shared with clang

Most people associate "clang" with the executable. You may want to use "Clang" to avoid confusion (IIUC, you are referring to the sub-project rather than the executable here).

specific flang flags

Flang :) (same rationale as above)

clang/utils/TableGen/ClangOptionDocEmitter.cpp

328

I think that the way this is written at the moment is a bit counterintuitive (i.e. "if excluded or not included"). This is really down to how these include/exclude flags are used in various other places :( (i.e. there's little that we can do about it).

I think that it would make more sense to follow the style from OptTable.cpp. In particular (pseudo code):

if (excluded)
  return;
if (included_not_empty && !included)
  return;

This way it becomes clear that IncludedFlags (from "FlangOptionsDocs.td") is only taken into account when not empty. I would also add an assert in isIncluded:

assert(DocInfo->getValue("IncludedFlags") && Missing includeFlags);

WDYT?

[DylanFleming-arm]

Edited summary to be more clear.

I've also changed the format of the if(!included) as you suggested (and did the same for the call to isGroupIncluded)

I also added the assert you asked for, and then since there's an if statement checking for the opposite condition, I removed that as it's now unreachable.

[awarzynski]

The change in ClangOptionDocEmitter.cpp is required for Flang as it heavily relies on these "include" flags defined in Options.td. This won't affect Clang as it does not use IncludeFlags in ClangOptionsDocs.td. If Clang devs were to use them in "ClangOptionsDocs.td", the required hooks in ClangOptionDocEmitter.cpp will already be there :)

LGTM, thanks for working on this! Give it another day before merging - in case Clang folks would like to chime in.

[DylanFleming-arm]

After pushing to main, this patch cause a buildbot failure as CLANG_TABLEGEN_EXE could not be found.

I've updated flang/docs/CMakeLists.txt to set the parameter before making a call to clang_tablegen

[awarzynski]

LGTM, thanks!

(feel free to address my [nit] when merging or ignore altogether)

flang/docs/CMakeLists.txt

128

[nit] This is a bit out of place without a comment :) Could you mention that this CMake variable is required in clang_tablegen?

clang/utils/TableGen/ClangOptionDocEmitter.cpp

 //===- ClangOptionDocEmitter.cpp - Documentation for command line flags ---===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 // FIXME: Once this has stabilized, consider moving it to LLVM.
 //

 
 bool hasFlag(const Record *OptionOrGroup, StringRef OptionFlag) {
   for (const Record *Flag : OptionOrGroup->getValueAsListOfDefs("Flags"))
     if (Flag->getName() == OptionFlag)
       return true;
   return false;
 }
 
+bool isIncluded(const Record *OptionOrGroup, const Record *DocInfo) {
+  assert(DocInfo->getValue("IncludedFlags") && "Missing includeFlags");
+  for (StringRef Inclusion : DocInfo->getValueAsListOfStrings("IncludedFlags"))
+    if (hasFlag(OptionOrGroup, Inclusion))
+      return true;
+  return false;
+}
+
+bool isGroupIncluded(const DocumentedGroup &Group, const Record *DocInfo) {
+  if (isIncluded(Group.Group, DocInfo))
+    return true;
+  for (auto &O : Group.Options)
+    if (isIncluded(O.Option, DocInfo))
+      return true;
+  for (auto &G : Group.Groups) {
+    if (isIncluded(G.Group, DocInfo))
+      return true;
+    if (isGroupIncluded(G, DocInfo))
+      return true;
+  }
+  return false;
+}
+
 bool isExcluded(const Record *OptionOrGroup, const Record *DocInfo) {
   // FIXME: Provide a flag to specify the set of exclusions.
   for (StringRef Exclusion : DocInfo->getValueAsListOfStrings("ExcludedFlags"))
     if (hasFlag(OptionOrGroup, Exclusion))
       return true;
   return false;
 }
 
void forEachOptionName(const DocumentedOption &Option, const Record *DocInfo,
 
   for (auto *Alias : Option.Aliases)
     if (!isExcluded(Alias, DocInfo) && canSphinxCopeWithOption(Option.Option))
       F(Alias);
 }
 
 void emitOption(const DocumentedOption &Option, const Record *DocInfo,
                 raw_ostream &OS) {
   if (isExcluded(Option.Option, DocInfo))
     return;
+  if (DocInfo->getValue("IncludedFlags") && !isIncluded(Option.Option, DocInfo))
+    return;
   if (Option.Option->getValueAsDef("Kind")->getName() == "KIND_UNKNOWN" ||
       Option.Option->getValueAsDef("Kind")->getName() == "KIND_INPUT")
     return;
   if (!canSphinxCopeWithOption(Option.Option))
     return;
 
   // HACK: Emit a different program name with each option to work around
   // sphinx's inability to cope with options that differ only by punctuation

 void emitDocumentation(int Depth, const Documentation &Doc,
                        const Record *DocInfo, raw_ostream &OS);
 
 void emitGroup(int Depth, const DocumentedGroup &Group, const Record *DocInfo,
                raw_ostream &OS) {
   if (isExcluded(Group.Group, DocInfo))
     return;
 
+  if (DocInfo->getValue("IncludedFlags") && !isGroupIncluded(Group, DocInfo))
+    return;
+
   emitHeading(Depth,
               getRSTStringWithTextFallback(Group.Group, "DocName", "Name"), OS);
 
   // Emit the description, if we have one.
   std::string Description =
       getRSTStringWithTextFallback(Group.Group, "DocBrief", "HelpText");
   if (!Description.empty())
     OS << Description << "\n\n";

flang/docs/CMakeLists.txt

if (LLVM_ENABLE_DOXYGEN)
 
   if (NOT LLVM_INSTALL_TOOLCHAIN_ONLY AND LLVM_BUILD_DOCS)
     install(DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/doxygen/html
       DESTINATION docs/html)
   endif()
 endif()
 endif()
 
+function (gen_rst_file_from_td output_file td_option source docs_target)
+  if (NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/${source}")
+    message(FATAL_ERROR "Cannot find source file: ${source} in ${CMAKE_CURRENT_SOURCE_DIR}")
+  endif()
+  get_filename_component(TABLEGEN_INCLUDE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/${source}" DIRECTORY)
+  list(APPEND LLVM_TABLEGEN_FLAGS "-I${TABLEGEN_INCLUDE_DIR}")
+  clang_tablegen(Source/${output_file} ${td_option} SOURCE ${source} TARGET "gen-${output_file}")
+  add_dependencies(${docs_target} "gen-${output_file}")
+endfunction()
+
 if (LLVM_ENABLE_SPHINX)
   include(AddSphinxTarget)
   if (SPHINX_FOUND)
     if (${SPHINX_OUTPUT_HTML})
       add_sphinx_target(html flang SOURCE_DIR "${CMAKE_CURRENT_BINARY_DIR}/Source")
 
       add_dependencies(docs-flang-html copy-flang-src-docs)
 
       # Copy the flang/docs directory and the generated FIRLangRef.md file to a place in the binary directory.
       # Having all the files in a single directory makes it possible for Sphinx to process them together.
       # Add a dependency to the flang-doc target to ensure that the FIRLangRef.md file is generated before the copying happens.
       add_custom_target(copy-flang-src-docs
         COMMAND "${CMAKE_COMMAND}" -E copy_directory
         "${CMAKE_CURRENT_SOURCE_DIR}"
         "${CMAKE_CURRENT_BINARY_DIR}/Source"
         DEPENDS flang-doc)
 
-        # Runs a python script prior to HTML generation to prepend a header to FIRLangRef,
-        # Without the header, the page is incorrectly formatted, as it assumes the first entry is the page title.
-        add_custom_command(TARGET copy-flang-src-docs
-          COMMAND "${Python3_EXECUTABLE}"
-          ARGS ${CMAKE_CURRENT_BINARY_DIR}/Source/FIR/CreateFIRLangRef.py)
 
+
+      set(CLANG_TABLEGEN_EXE clang-tblgen)
+      gen_rst_file_from_td(FlangCommandLineReference.rst -gen-opt-docs ../include/flang/FlangOptionsDocs.td docs-flang-html)
     endif()
     if (${SPHINX_OUTPUT_MAN})
       add_sphinx_target(man flang)
     endif()
   endif()
 endif()
 

flang/docs/index.md

.. toctree::
    BijectiveInternalNameUniquing
    Calls
    Character
    ControlFlowGraph
    Directives
    DoConcurrent
    Extensions
    FIRLangRef
+   FlangCommandLineReference
    FlangDriver
    FortranIR
    FortranLLVMTestSuite
    IORuntimeInternals
    Intrinsics
    IntrinsicTypes
    LabelResolution
    ModFiles

flang/include/flang/FlangOptionsDocs.td

+//==--- FlangOptionDocs.td - Option documentation -------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+def GlobalDocumentation {
+  code Intro =[{..
+  -------------------------------------------------------------------
+  NOTE: This file is automatically generated by running clang-tblgen
+  -gen-opt-docs. Do not edit this file by hand!!
+  -------------------------------------------------------------------
+
+=====================================
+Flang command line argument reference
+=====================================
+.. contents::
+   :local:
+
+Introduction
+============
+
+}];
+
+  string Program = "flang";
+
+  list<string> ExcludedFlags = [];
+  list<string> IncludedFlags = ["FlangOption"];
+
+}
+
+
+include "../../../clang/include/clang/Driver/Options.td"