[FLANG] Pick `.md` files when building sphinx documentation.

[sameeranjoshi]

Summary

Need to build sphinx using below flags to Cmake
-DLLVM_ENABLE_SPHINX=ON -DSPHINX_WARNINGS_AS_ERRORS=OFF.
Generate html docs using cmake target
docs-flang-html
Generated html files should be at build/tools/flang/docs/html.
Patch in series from the dicussion on review
https://reviews.llvm.org/D85828

After this patch the markdown docmentation must be written using guide in-
llvm/docs/MarkdownQuickstartTemplate.md

[sscalpone]

Please add a note to the overview about this requirement:

After this patch the markdown docmentation must be written using guide in-
llvm/docs/MarkdownQuickstartTemplate.md

Please update the instructions for generating documentation in README.md.

flang/docs/conf.py

255–256

Maybe update the line "One line description of project" to be something like "A Fortran front end for LLVM"

[sameeranjoshi]

Address review comments.
For viewing on github
https://github.com/Sameeranjoshi/llvm-project/tree/arcpatch-D86131/flang

[kiranchandramohan]

flang/README.md

225

Nit: Suggestion: preferably be written in markdown(.md) documentation format -> preferably be written in markdown(.md) syntax
Nit: space between format and (.

flang/docs/conf.py

31

Is this over-riding the assignment to extensions above? If so can the one above be removed?

[sameeranjoshi]

Thanks @kiranchandramohan.
Updated diff.

[sscalpone]

The install target required me to build all of the documentation. I do not think that is related to this change. Otherwise, the web pages rendered nicely.

[sameeranjoshi]

Thanks @sscalpone, this issue was mentioned long back during the doxygen related patches.
llvm, mlir and might be other projects in tree seem to show the same issue.
https://reviews.llvm.org/D78136#1987218

[hans]

I tried building the Flang docs with this applied, and they build fine, but show up empty in my web browser.

Looking at the generated Overview.html, it seems to be because of the comment header at the top, which looks like this in HTML:

  <!--===- docs/Overview.md <p>Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
See https://llvm.org/LICENSE.txt for license information.
SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception</p>
<p>–&gt;</p>

It appears that the HTML comment is never terminated, because the --> has been turned into –> by the tool.

I would suggest dropping the comment headers from the docs; other LLVM projects don't seem to have them. Or if they're considered important, maybe try one of the other techniques in https://stackoverflow.com/questions/4823468/comments-in-markdown

[sameeranjoshi]

Hi @hans
The problem here is the version of recommonmark.
We strictly need version 0.5.0 or above.
See the note at https://www.sphinx-doc.org/en/master/usage/markdown.html#configuration
and https://github.com/llvm/llvm-project/blob/0fd425af071a9bc5c0891a4db09f4d9a466b7be9/flang/docs/conf.py#L44

I tried with the mentioned version and I can see the comments in the HTML files.
@sscalpone as well has verified it with recommonmark 0.6.0 version

Hopefully, @hans if you can verify it with a sample comment on your machine then I can revert the old patch D86875 and bring back licence text.

Thanks.

[hans]

In D86131#2263071, @sameeranjoshi wrote:

Hi @hans
The problem here is the version of recommonmark.
We strictly need version 0.5.0 or above.
See the note at https://www.sphinx-doc.org/en/master/usage/markdown.html#configuration
and https://github.com/llvm/llvm-project/blob/0fd425af071a9bc5c0891a4db09f4d9a466b7be9/flang/docs/conf.py#L44

I tried with the mentioned version and I can see the comments in the HTML files.
@sscalpone as well has verified it with recommonmark 0.6.0 version

Hopefully, @hans if you can verify it with a sample comment on your machine then I can revert the old patch D86875 and bring back licence text.

Thanks.

I don't have version 0.5.0 or above (I use the Debian package), but put the comment back in and I'll deal with it locally in my build.

flang/README.md

 To understand how a flang program communicates with libraries at runtime,
 see the discussion of [runtime descriptors](docs/RuntimeDescriptor.md).
 
 If you're interested in contributing to the compiler,
 read the [style guide](docs/C++style.md)
 and
 also review [how flang uses modern C++ features](docs/C++17.md).
 
+If you are interested in writing new documentation, follow 
+[markdown style guide from LLVM](https://github.com/llvm/llvm-project/blob/master/llvm/docs/MarkdownQuickstartTemplate.md).
+
 ## Supported C++ compilers
 
 Flang is written in C++17.
 
 The code has been compiled and tested with
 GCC versions from 7.2.0 to 9.3.0.
 
 The code has been compiled and tested with

 cd ~/llvm-project/build
 cmake -DLLVM_ENABLE_DOXYGEN=ON -DFLANG_INCLUDE_DOCS=ON ../llvm
 make doxygen-flang
 
 It will generate html in
 
     <build-dir>/tools/flang/docs/doxygen/html # for flang docs
 ```
+## Generate Sphinx-based Documentation
+<!TODO: Add webpage once we have a website.
+!>
+Flang documentation should preferably be written in `markdown(.md)` syntax (they can be in `reStructuredText(.rst)` format as well but markdown is recommended in first place), it
+is mostly meant to be processed by the Sphinx documentation generation
+system to create HTML pages which would be hosted on the webpage of flang and
+updated periodically.
+
+If you would like to generate and view the HTML locally, install
+Sphinx <http://sphinx-doc.org/> and then:
+
+- Pass `-DLLVM_ENABLE_SPHINX=ON -DSPHINX_WARNINGS_AS_ERRORS=OFF` to the cmake command.
+
+```
+cd ~/llvm-project/build
+cmake -DLLVM_ENABLE_SPHINX=ON -DSPHINX_WARNINGS_AS_ERRORS=OFF ../llvm
+make docs-flang-html
+
+It will generate html in
+
+   $BROWSER <build-dir>/tools/flang/docs/html/
+```

flang/docs/conf.py

 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #sys.path.insert(0, os.path.abspath('.'))
 
 # -- General configuration -----------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
 #needs_sphinx = '1.0'
-
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = ['sphinx.ext.todo', 'sphinx.ext.mathjax', 'sphinx.ext.intersphinx']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
 # The suffix of source filenames.
-source_suffix = '.rst'
+source_suffix = {
+    '.rst': 'restructuredtext',
+}
+try:
+  import recommonmark
+except ImportError:
+  # manpages do not use any .md sources
+  if not tags.has('builder-man'):
+    raise
+else:
+  import sphinx
+  if sphinx.version_info >= (3, 0):
+    # This requires 0.5 or later.
+    extensions.append('recommonmark')
+  else:
+    source_parsers = {'.md': 'recommonmark.parser.CommonMarkParser'}
+  source_suffix['.md'] = 'markdown'
 
 # The encoding of source files.
 #source_encoding = 'utf-8-sig'
 
 # The master toctree document.
-master_doc = 'ReleaseNotes'
+master_doc = 'Overview'
 
 # General information about the project.
 project = u'Flang'
 copyright = u'2017-%d, The Flang Team' % date.today().year
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents. These are currently set to zero because we don't use them.

 
 # Additional stuff for the LaTeX preamble.
 #'preamble': '',
 }
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title, author, documentclass [howto/manual]).
 latex_documents = [
-  ('ReleaseNotes', 'Flang.tex', u'Flang Documentation',
+  ('Overview', 'Flang.tex', u'Flang Documentation',
    u'The Flang Team', 'manual'),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of
 # the title page.
 #latex_logo = None
 
 # For "manual" documents, if this is true, then toplevel headings are parts,

 
 
 # -- Options for Texinfo output ------------------------------------------------
 
 # Grouping the document tree into Texinfo files. List of tuples
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-  ('ReleaseNotes', 'Flang', u'Flang Documentation',
-   u'The Flang Team', 'Flang', 'One line description of project.',
+  ('Overview', 'Flang', u'Flang Documentation',
+   u'The Flang Team', 'Flang', 'A Fortran front end for LLVM.',
    'Miscellaneous'),
 ]
 
 # Documents to append as an appendix to all manuals.
 #texinfo_appendices = []
 
 # If false, no module index is generated.
 #texinfo_domain_indices = True
 
 # How to display URL addresses: 'footnote', 'no', or 'inline'.
 #texinfo_show_urls = 'footnote'