Page MenuHomePhabricator

[Flang] Move mark down documentation(md) files to reStructuredText(rst) file format.
AbandonedPublic

Authored by sameeranjoshi on Aug 12 2020, 4:51 AM.

Details

Summary

Note: .md to .rst files were created using m2r
The tool was used by 1600+ users.
https://github.com/miyakogi/m2r

copy files to docs/ folder, delete /documentation folder.

Internal changes in files to modify bad occurances of ".md".

Remove licence headers.

Diff Detail

Unit TestsFailed

TimeTest
350 mslinux > LLVM.CodeGen/AMDGPU/GlobalISel::andn2.ll
Script: -- : 'RUN: at line 2'; /mnt/disks/ssd0/agent/llvm-project/build/bin/llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=tahiti < /mnt/disks/ssd0/agent/llvm-project/llvm/test/CodeGen/AMDGPU/GlobalISel/andn2.ll | /mnt/disks/ssd0/agent/llvm-project/build/bin/FileCheck -check-prefixes=GCN,GFX6 /mnt/disks/ssd0/agent/llvm-project/llvm/test/CodeGen/AMDGPU/GlobalISel/andn2.ll
330 mslinux > LLVM.CodeGen/AMDGPU/GlobalISel::orn2.ll
Script: -- : 'RUN: at line 2'; /mnt/disks/ssd0/agent/llvm-project/build/bin/llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=tahiti < /mnt/disks/ssd0/agent/llvm-project/llvm/test/CodeGen/AMDGPU/GlobalISel/orn2.ll | /mnt/disks/ssd0/agent/llvm-project/build/bin/FileCheck -check-prefixes=GCN,GFX6 /mnt/disks/ssd0/agent/llvm-project/llvm/test/CodeGen/AMDGPU/GlobalISel/orn2.ll
460 mslinux > LLVM.CodeGen/AMDGPU/GlobalISel::xnor.ll
Script: -- : 'RUN: at line 2'; /mnt/disks/ssd0/agent/llvm-project/build/bin/llc -global-isel -march=amdgcn -mcpu=gfx700 -verify-machineinstrs < /mnt/disks/ssd0/agent/llvm-project/llvm/test/CodeGen/AMDGPU/GlobalISel/xnor.ll | /mnt/disks/ssd0/agent/llvm-project/build/bin/FileCheck -check-prefixes=GCN,GFX7 /mnt/disks/ssd0/agent/llvm-project/llvm/test/CodeGen/AMDGPU/GlobalISel/xnor.ll
400 mswindows > LLVM.CodeGen/AMDGPU/GlobalISel::andn2.ll
Script: -- : 'RUN: at line 2'; c:\ws\w16c2-1\llvm-project\premerge-checks\build\bin\llc.exe -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=tahiti < C:\ws\w16c2-1\llvm-project\premerge-checks\llvm\test\CodeGen\AMDGPU\GlobalISel\andn2.ll | c:\ws\w16c2-1\llvm-project\premerge-checks\build\bin\filecheck.exe -check-prefixes=GCN,GFX6 C:\ws\w16c2-1\llvm-project\premerge-checks\llvm\test\CodeGen\AMDGPU\GlobalISel\andn2.ll
360 mswindows > LLVM.CodeGen/AMDGPU/GlobalISel::orn2.ll
Script: -- : 'RUN: at line 2'; c:\ws\w16c2-1\llvm-project\premerge-checks\build\bin\llc.exe -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=tahiti < C:\ws\w16c2-1\llvm-project\premerge-checks\llvm\test\CodeGen\AMDGPU\GlobalISel\orn2.ll | c:\ws\w16c2-1\llvm-project\premerge-checks\build\bin\filecheck.exe -check-prefixes=GCN,GFX6 C:\ws\w16c2-1\llvm-project\premerge-checks\llvm\test\CodeGen\AMDGPU\GlobalISel\orn2.ll
View Full Test Results (6 Failed)

Event Timeline

sameeranjoshi created this revision.Aug 12 2020, 4:51 AM
Herald added a project: Restricted Project. · View Herald TranscriptAug 12 2020, 4:51 AM
sameeranjoshi requested review of this revision.Aug 12 2020, 4:51 AM

To ease in reviewing the output of files when merged in llvm-project.
I have pushed in my local fork of llvm-project.
https://github.com/Sameeranjoshi/llvm-project/blob/sam-docu-rst/flang/

DavidTruby accepted this revision.Aug 12 2020, 5:00 AM

LGTM and they seem to render fine on github too, thanks for sharing that link so we can check that easily!

This revision is now accepted and ready to land.Aug 12 2020, 5:00 AM
richard.barton.arm requested changes to this revision.EditedAug 12 2020, 6:14 AM

I still see a few mentions of the old documentation dir location in the codebase:

llvm-project/flang$ git grep 'documentation/' .
README.md:Start with the [compiler overview](documentation/Overview.md).
README.md:read [Fortran For C Programmers](documentation/FortranForCProgrammers.md)
README.md:flang's specifications of the [Fortran grammar](documentation/f2018-grammar.txt)
README.md:the [OpenMP grammar](documentation/OpenMP-4.5-grammar.txt).
README.md:in [this document](documentation/Extensions.md).
README.md:see the [discussion of intrinsics](documentation/Intrinsics.md).
README.md:see the discussion of [runtime descriptors](documentation/RuntimeDescriptor.md).
README.md:read the [style guide](documentation/C++style.md)
README.md:also review [how flang uses modern C++ features](documentation/C++17.md).
docs/Character.rst:   <!--===- documentation/Character.rst
docs/OpenMP-4.5-grammar.txt:#===-- documentation/OpenMP-4.5-grammar.txt --------------------------------===#
docs/f2018-grammar.txt:#===-- documentation/f2018-grammar.txt -------------------------------------===#
docs/flang-c-style.el:;;===-- documentation/flang-c-style.el ------------------------------------===;;
lib/Evaluate/intrinsics.cpp:// documentation/intrinsics.md and section 16 of the Fortran 2018 standard
This revision now requires changes to proceed.Aug 12 2020, 6:14 AM

Could you please update the summary to indicate why you are making this change?

flang/docs/ArrayComposition.rst
11

What is the purpose of the backslash followed by space? It seems to occur in a lot of places.

DavidTruby added inline comments.Aug 12 2020, 7:52 AM
flang/docs/ArrayComposition.rst
11

I believe "extra" spaces are removed by rst when rendering to HTML, so the \ will preserve the space

klausler requested changes to this revision.Aug 12 2020, 7:56 AM

Why are you doing this? *.rst files are ugly and harder than *.md to edit correctly. I don't see how this is any kind of improvement.

sscalpone requested changes to this revision.Aug 12 2020, 8:02 AM

I think md is the wave of the future. Other llvm subproject use md. Let's stick with what we have unless there's a compelling advantage to using rst or another format.

I think md is the wave of the future. Other llvm subproject use md. Let's stick with what we have unless there's a compelling advantage to using rst or another format.

It's not really correct to say that other llvm subprojects use md: with the exception of MLIR every LLVM subproject uses rst, including LLVM itself.

MLIR hosts a separate website elsewhere, which is why they aren't tied to the same format as everyone else. If we want to do that and render the .md files to html instead then we can, but if we want to be on the same website as the rest of LLVM (except MLIR) we need to use rst as the text format.

That is, at least, my understanding for the motivation behind the change. @sameeranjoshi can probably provide a better explanation as it's his change.

+1 @DavidTruby - For pointing the reason for MLIR to use .md files.
It's correct that LLVM uses Sphinx infrastructure to build docs for any sub-project except MLIR.

" If we want to do that and render the .md files to html instead then we can, but if we want to be on the same website as the rest of LLVM (except MLIR) we need to use rst as the text format."

I remember we had a light on this issue long back(initial doxygen support) given that we want something like llvm.org/flang or more technically similar flang.llvm.org we would need to move to .rst from .md.
https://github.com/flang-compiler/f18/pull/1065#issuecomment-606721800
https://github.com/flang-compiler/f18/pull/1065#issuecomment-606532420

Here's one such reference recently
https://reviews.llvm.org/D85470#2207363

I tried hard to find any reference from the developers policy to use .rst as documentation formats, with no exact mentioning I found below page
http://llvm.org/docs/SphinxQuickstartTemplate.html
mentioning a few points as

  1. Sphinx to be more powerful than markdown file formats.
  2. Sending the .rst file to llvm-commits for review.

Neither I am against using .md files nor I have much details on .rst files, but I doubt Flang would render using their own website as MLIR, hence going with the option with LLVM(sphinx build and .rst files) seemed reasonable to be to come up with this patch.

This patch has basically 2 changes :

  1. Move files to docs/ folder from documentation/
  2. Change them to .rst

And I assume the first set of changes people agree upon & are the discussion seems to be around second point.
Thanks for reviewing all :)

sameeranjoshi added a comment.EditedAug 12 2020, 11:48 AM

I still see a few mentions of the old documentation dir location in the codebase:

llvm-project/flang$ git grep 'documentation/' .
README.md:Start with the [compiler overview](documentation/Overview.md).
README.md:read [Fortran For C Programmers](documentation/FortranForCProgrammers.md)
README.md:flang's specifications of the [Fortran grammar](documentation/f2018-grammar.txt)
README.md:the [OpenMP grammar](documentation/OpenMP-4.5-grammar.txt).
README.md:in [this document](documentation/Extensions.md).
README.md:see the [discussion of intrinsics](documentation/Intrinsics.md).
README.md:see the discussion of [runtime descriptors](documentation/RuntimeDescriptor.md).
README.md:read the [style guide](documentation/C++style.md)
README.md:also review [how flang uses modern C++ features](documentation/C++17.md).
docs/Character.rst:   <!--===- documentation/Character.rst
docs/OpenMP-4.5-grammar.txt:#===-- documentation/OpenMP-4.5-grammar.txt --------------------------------===#
docs/f2018-grammar.txt:#===-- documentation/f2018-grammar.txt -------------------------------------===#
docs/flang-c-style.el:;;===-- documentation/flang-c-style.el ------------------------------------===;;
lib/Evaluate/intrinsics.cpp:// documentation/intrinsics.md and section 16 of the Fortran 2018 standard

Good catch. Thanks I was grepping in a sub directory which was removed later by me.

+1 @DavidTruby - For pointing the reason for MLIR to use .md files.
It's correct that LLVM uses Sphinx infrastructure to build docs for any sub-project except MLIR.

Maybe @mehdi_amini can shed some light on the rationale for the divergence. (The distinction between the links http://clang.llvm.org and http://mlir.llvm.org doesn't seem that large.)

I think md is the wave of the future. Other llvm subproject use md. Let's stick with what we have unless there's a compelling advantage to using rst or another format.

It's not really correct to say that other llvm subprojects use md: with the exception of MLIR every LLVM subproject uses rst, including LLVM itself.

MLIR hosts a separate website elsewhere, which is why they aren't tied to the same format as everyone else. If we want to do that and render the .md files to html instead then we can, but if we want to be on the same website as the rest of LLVM (except MLIR) we need to use rst as the text format.

That is, at least, my understanding for the motivation behind the change. @sameeranjoshi can probably provide a better explanation as it's his change.

Both website are serving HTML: as long as you generate HTML you can serve easily either way.

LLVM is using Sphinx to turn its documentation from RST to HTML and copy this to the webserver.
MLIR is using Hugo (see mlir-www) to turn its .md into HTML and serve it directly from GitHub pages.

There aren't any real underlying technology from the website point of view, MLIR could be hosted with LLVM as well, and use the same theme, etc.

The use of Markdown in MLIR is mostly historical I believe (Google uses heavily markdown for its doc and MLIR started at Google).

The use of GitHub pages instead of llvm.org is because the admins told me back in the fall when I was setting up the website that "it would be nice to migrate llvm.org to GitHub pages".

Thanks for detailed information @mehdi_amini and all folks.
TL;DR version:
There is a sweet spot on this issue.
Sphinx supports now .md files.
LLVM has as well added changes in config files to pick up .md files, needs to extend this to flang.

Long version:

Sphinx has a support for markdown format files, but it supports commonmark based markdown flavour[1].
As there are tons of implementations of markdown format, commonmark is a project which standardizes the way we write markdown files.
Github has commonmark flavour supported with some extra specifications/features which they call GFM(Github Flavored Markdown[2])
Github's way of writing markdown files is a subset of commonmark.

So I think that should not be an issue for rendering the current .md files unless people have unknowingly referred GFM and written some parts of the documentation, in that case some changes in documentation would be needed.

Unfortunately, there was a commit[3] in llvm-core long time back which supported this commonmark based format(to move towards .md files) into llvm sphinx builds and buildbots and I guess no sub-project ever used it except to some extent LLVM.

I have fragmented this patch into 2 patches:

  1. To move files to docs/ folder - https://reviews.llvm.org/D85884
  2. I plan to add support for .md files in sphinx using the way LLVM recommends[4] in a later patch.

[1] https://www.sphinx-doc.org/en/master/usage/markdown.html
[2] https://guides.github.com/features/mastering-markdown/
[3] https://reviews.llvm.org/D44910
[4] https://github.com/llvm/llvm-project/commits/master/llvm/docs/MarkdownQuickstartTemplate.md

sameeranjoshi abandoned this revision.Aug 18 2020, 3:31 AM

Sphinx support for .md files.
https://reviews.llvm.org/D86131