Add clang-doc readme

[brettw]

Summary

Adds a readme to clang-doc with an overview of the structure of the code.

[paulkirth]

I'm a bit tied up until next week, and looped in some other reviewers. I can take a look on Monday if needed.

[paulkirth]

First, I don't think this is a good fit for a README.txt based on the content of other clang-tool README.txt files. Most of them only outline project scope or point to formal documentation on llvm.org. Other README files also tend to only provide build or dependency information. The one example I can find that does more is clangd/quality/README.md, which provides details on a number of internal data structures. Its preferable to document project internals in the .rst files as part of the developer facing documentation on llvm.org, so I think that's where something like this would ultimately belong.

I think there are some other issues with the content of this patch.

• I think there is some miscommunication about clang-doc's design. Clang-doc follows LLVM's architecture to a large extent by translating the AST into its own intermediate representation, and then letting custom backends generate the appropriate output. Further, this design allows the clang-doc IR to be serialized(though I understand this is not yet implemented), which is important for testing and scalability. The initial clang-doc RFC and discussion with other clang-tool developers pointed to this design as the best practice for scaling up to large organizations. I understand the tool is not meeting expectations in that regard, but it doesn't mean the design is the incorrect choice. While you're certainly free to disagree with that approach, this document isn't the place to codify that opinion. You're also free to submit patches to refactor clang-doc in the way you think it should be architected. Contributions, as always are welcome. 😄
• Call outs to "watch out" don't seem appropriate for formal documentation. A "note" or some other descriptor will work just as well without using language that makes the tool sound dangerous.
• The introduction is a coarse approximation of how libTooling operates, you can just say that it's build on libTooling and point to its documentation. I think that will 1) be more concise, and 2) allow you to focus on the details of clang-doc without worrying about describing the infrastructure its built upon. If you still feel that its central to this document, you may wish to quote it directly.
• Describing how to add a new field is mostly fine. The code should probably cary a reference to this documentation (or should generate the documentation).

My recommendation here is that you should move this into an rst file under doc and revise this document to more closely model the existing documentation practices in LLVM. I've left some suggestions inline that you may find useful. How you revise the document is up to you, and I won't' be a stickler if you would like to word something differently than I've suggested.

clang-tools-extra/clang-doc/README.txt

5–10

the libtooling documentation covers this, can we just link to that?

12–16

18–20

22–27

I'd drop this paragraph altogether.

30–31

Maybe something along these lines?

37–38

I'd add something about the various backends, or at least mention that the output is determined by a backend implementation.

[brettw]

I definitely wasn't intending to write "formal documentation" for clang-doc. I was trying to write some notes for a future contributor as I'm winding down my work here, and it seems nobody else is working on it or has any knowledge about it. Is there a place in LLVM for this type of documentation? I can rename the file to "notes" or something if you think that would help make things more clear.

[paulkirth]

In D136894#3905638, @brettw wrote:

I definitely wasn't intending to write "formal documentation" for clang-doc. I was trying to write some notes for a future contributor as I'm winding down my work here, and it seems nobody else is working on it or has any knowledge about it. Is there a place in LLVM for this type of documentation?

Most of LLVM's documentation is either in the formal documentation. Many things are also only documented in the code, though. We're trying to be better about that these days, but manyt things are only described in the comments of a particular .cpp file.

I can rename the file to "notes" or something if you think that would help make things more clear.

I can't think of anything like that elsewhere in our code base. If you don't want formal documentation, you can take the parts you think are appropriate and add them as comments to the relevant files. For things you think should be changed, an issue on github is probably the best way to make sure that gets addressed, and reference that somewhere in clang-doc's code.

[brettw]

Ok thanks.

clang-tools-extra/clang-doc/README.txt

+==============================
+How clang-doc works internally
+==============================
+
+Clang-doc uses the "tooling" library which can run the compiler. It can take the files directly or
+it can extract the file list and the commands corresponding to each from a compile_commands.json
+file.
+
+The tooling library spins up threads and parses the compilation units in parallel. Clang-doc
+registers a callback to run on the AST of each unit.
+
+When the AST is known, the MapASTVisitor in Mapper.cpp is run on the AST. It has callbacks for the
+main AST nodes that clang-doc cares about. This is a very simple object that mostly calls into
+Serialize.cpp to generate the "representation" of the code. These are the various *Info structures
+like FunctionInfo and NamespaceInfo defined in Representation.h that correspond to each element of
+the code that might be documented.
+
+The representation from each execution thread is serialized to bitcode using BitcodeWriter.cpp. This
+is a custom bitcode defined in BitcodeWriter.h and is NOT regular LLVM bitcode IR. Watch out: the
+"Serialize.cpp" file is not related to this step though the name may imply it.
+
+These bitcode representations are then passed back to the main thread and deserialized back to a
+new copy of the representation in BitcodeReader.cpp. This round-trip through bitcode is used only to
+get the data out of the AST visitor (which is expected to return a byte stream) and is never saved
+or used for any other purpose. This round-trip adds significant complexity and we should consider
+passing the Representation object hierarchy back to the main thread out-of-band without
+serialization and deleting the bitcode representation.
+
+After deserialization, the various representation objects from each thread are merged/reduced into a
+single structure. This is necessary both to collect everything and to merge the declarations of
+things (of which there may be many) with the actual implementation. Many fields on a record can't be
+merged in the abstract (for example, a boolean field on two structures with two different values has
+no clear merged result). But this question can be resolved by thinking about the items as
+definitions and declarations and picking the value which would be present on the definition (usually
+the more specific or non-default value).
+
+After merging the final representation is passed to the output generator which writes the final
+files.
+
+==================
+To add a new field
+==================
+
+To add a new bit of information to the documentation that you extract from the AST:
+
+ 1. Add it to the appropriate *Info structure in Representation.h.
+
+ 2. Extract the information from the AST by adding to the code in Serialize.cpp and possibly
+    Mapper.cpp. Save it to the Info structure used above.
+
+ 3. Write bitcode serialization code in BitcodeWriter.cpp for the new field.
+
+ 4. Write bitcode deserialization code in BitcodeReader.cpp.
+
+ 5. Add merging code in the merge() function for your Info structure (see above for advice).
+
+ 6. Update the backend(s) to use the new data.
+
+Note! Steps 3, 4, and especially 5 are easy to forget. Without all of these, the code may look like
+it works but you will only get default values out at the end.