[docs] Introduce clangd part in StandardCPlusPlusModules

[ChuanqiXu]

Summary

Introduce clangd part in to the document to tell end users about the status quo. It is pretty important since nowadays it is painful to write codes without the help form the code intelligence.

And my feeling in the recent days is that: the current status is obviously very far from good but it is not useless completely. As long as we can get a compile_commands.json, the clangd can start to work in some degree. And it should be easy for the build systems to generate the compile_commands.json. So I think it is meaningful to document the status quo.

[sammccall]

The status quo for modules on clangd is all flavors are completely unsupported: https://github.com/clangd/clangd/issues/1293.
I don't think we should encourage people to set up workflows that are known not to work properly and will probably break in future.

It's true that some parts work by coincidence (if you embed clang, you get all the features by default), but it's broken both in principle and in practice.

• most users don't have a version-locked clang and clangd, so relying on loading clang-produced BMI files from disk is the wrong design
• modules that get loaded somewhere inside clang never get indexed, which affects most features (e.g. code completion))
• various clangd features (include insertion, code completion ranking, unused include detection, many others) rely on the assumption that we have recorded the complete #include tree.
• in practice we know it's easy to get this to crash, and that users who follow instructions and get crashes expect support

To get to a better state someone needs to design and build support for modules in clangd. This is a major project. (Based on the discussions we've had so far, I *don't* expect the solution to look like "adopt clang's support" but rather require significant new infrastructure).

It's also one that doesn't currently have an owner. Unfortuntately (which has funded much of clangd's development) it's not likely to be a priority for $EMPLOYER unless/until we're using C++20 modules ourselves. (And we have less time to work on clangd at all, these days). We're open to someone else driving it (I know some folks from Meta were looking at this) as long as we head towards a design that's likely to be widely useful (e.g. no version lock, plan to work with common build systems).

This situation is not at all obvious to users, and I'm impressed you managed to figure out the bits that work! I think we should clarify the situation by making clangd emit an error when modules are enabled (we can disable this with a flag like --experimental-modules for people that want to work on modules support/test it once someone works on it).

[ChuanqiXu]

@sammccall Thanks for you professional and valuable input. It is very helpful. From the perspective of clangd, it looks not good indeed that the user can't get a stable support for modules.

This situation is not at all obvious to users, and I'm impressed you managed to figure out the bits that work! I think we should clarify the situation by making clangd emit an error when modules are enabled (we can disable this with a flag like --experimental-modules for people that want to work on modules support/test it once someone works on it).

Could I interpret your words as the following?

• We should and could emit an error in clangd if it detects we're using modules.
• We can add a flag --experimental-modules to skip such errors.

If yes, I think I know how to implement this. We can add a PPCallbacks, which implement the moduleImport callbacks so that we can know it is importing a module (or declare a module). This should be the proper way. And we can use (and document) the experimental-modules flag to get the facility today (and better in future). Or in another word, the tests in D137693 can't be passed. But if could be if we add the experimental-modules flag to the clangd. Do I understand correctly?

---

The following off is the plan in the longer term. From my point of view, your word mentions 2 different concern:

• inner concern: we need new design in the clangd to make clangd understand modules really.
• outer concern: we need an protocal between clangd and the build system to allow the clangd to inform the build system to build modules. And we need the compiler (clang) to parse the module file from different versions.

From my point of view, both concerns are hard to address. While I don't know a lot about the implementation details about clangd, both of the two outer concern are not easy to address. (Although they are absolutely the correct direction, of course). I'm afraid that I can take it. But I'll try to look at the implementation of clangd more (I've tried to read it for a few days and it is not easy indeed) to have a strong feelings.

[nridge]

In D137770#3919327, @ChuanqiXu wrote:

And we need the compiler (clang) to parse the module file from different versions.

I think it's more likely that we'll need to get clangd to build the modules itself (or by using a clang binary whose version matches clangd).

In scenarios where the build compiler is clang and its version matches clangd, we could have a mechanism for clangd to locate and reuse the modules built by the build system.

[spartacoos]

Hi @nridge @ChuanqiXu, I've been dealing with some issues trying to get modules working with clangd only to end up hacking some flimsy solutions which inevitably breakdown under some edge cases, those problems led me to this thread where I realised this is an active issue that's not being worked on. So, I want to put myself forward to help implement this new module system for clangd once you have settled on a design/approach. I'm not a C++ expert nor familiar with the clangd codebase, but I think I could help out if given some concrete steps. I'll try to familiarise myself with the codebase and join the discord. Are there any other resources where I can learn more about clangd development? e.g how do I go about testing ideas in clangd without breaking everything?

[ChuanqiXu]

In D137770#3945095, @spartacoos wrote:

Hi @nridge @ChuanqiXu, I've been dealing with some issues trying to get modules working with clangd only to end up hacking some flimsy solutions which inevitably breakdown under some edge cases, those problems led me to this thread where I realised this is an active issue that's not being worked on. So, I want to put myself forward to help implement this new module system for clangd once you have settled on a design/approach. I'm not a C++ expert nor familiar with the clangd codebase, but I think I could help out if given some concrete steps. I'll try to familiarise myself with the codebase and join the discord. Are there any other resources where I can learn more about clangd development? e.g how do I go about testing ideas in clangd without breaking everything?

Sorry I'm not familiar with the implementation for clangd too. Maybe we can CC @sammccall here or create another new thread to discuss the details. I can help with module related things (clang implementation and language) only (which seems not so helpful in this case). The only new information I know is that we can focus on named modules now. And for the header units, we can treat them in the similar way like normal headers.

[nridge]

In D137770#3945095, @spartacoos wrote:

I realised this is an active issue that's not being worked on. So, I want to put myself forward to help implement this new module system for clangd once you have settled on a design/approach.

Well, an issue here is that a design isn't actively being worked on either :) So it's probably best to start there.

I think a good starting point would be Sam's list of "issues that need to be addressed" in this comment.

A concrete way to make progress here might be:

• Identify scenarios that we explicitly do not want to support. An example might be "consuming binary module artifacts produced by a non-clang compiler".
• Of the scenarios that we do want to support, identify a subset to focus on implementing support for in an initial implementation ("support now" vs. "support later"). For example, we could start by limiting support to cases where clangd builds all modules from source itself and does not try to consume binary module artifacts built by anything other than itself. (Alternatively, we could start by limiting support only to cases where we assume the build compiler is version-locked to clangd and _only_ consume binary module artifacts built by the build compiler.) Since you're looking to work on this, the choice of subset can of course be informed by what scenarios you care about.
• Go through the "list of issues", and propose answers that are suitable for the "support now" scenarios, while still leaving open a path to bring the "support later" scenarios onboard.

We can then discuss the proposed approach in the issue and iterate on it.

Are there any other resources where I can learn more about clangd development?

The clangd website and particularly the design section are probably helpful reading, especially to understand some of the concepts like "preamble" discussed in the above list.

[ChuanqiXu]

@sammccall @nridge in a chat, @v.g.vassilev mentioned that it may be possible to make clangd for modules a GSoC project. I feel it is pretty good idea. How do you think about it?

clang/docs/StandardCPlusPlusModules.rst

 Another reason is that there are proposals to introduce module mappers to the C++ standard
 (for example, https://wg21.link/p1184r2).
 If we decide to reuse Clang's modulemap, we may get in trouble once we need to introduce another module mapper.
 
 So the final answer for why we don't reuse the interface of Clang modules for header units is that
 there are some differences between header units and Clang modules and that ignoring those
 differences now would likely become a problem in the future.
 
+Interoperability with tool chains
+=================================
+
+The modules change the traditional header based C++ project structures drastically.
+So the interoperability of many tool chains is broken. Here we track the status quo
+and the interoperability with clang for different tools.
+
+clangd
+======
+
+The clangd is a language server that can work with many editors via a plugin.
+The clangd can show diagnostics of errors and warnings in-place, suggest fixes,
+show extra hints about code problems and suggest code completion.
+
+Since clangd will invoke clang to parse the files according to flags from the
+`compilation database <https://clang.llvm.org/docs/JSONCompilationDatabase.html>`_,
+so as long as the clang can compile the files by these flags in the compilation database,
+the clangd should be able to handle the files in some level.
+
+So given clang can compile some modules codes now,
+the clangd can work for modules in some degree. The following is an simple example to show
+how to use clangd for modules:
+
+.. code-block:: c++
+
+  // A.cppm
+  export module A;
+  export void printA() {}
+
+  // Use.cpp
+  import A;
+  void foo() {
+  }
+
+  // compile_commands.json
+  [
+    {
+      "directory": "DIR",
+      "command": "clang++  -fprebuilt-module-path=DIR -std=c++20 -o DIR/main.cpp.o -c DIR/Use.cpp",
+      "file": "DIR/Use.cpp"
+    }
+  ]
+
+Now the important part is that we need to compile module A to BMI first:
+
+.. code-block:: console
+  
+  clang++  -fprebuilt-module-path=DIR -std=c++20 -o DIR/main.cpp.o -c DIR/Use.cpp
+
+Then in ``Use.cpp``, when we type `pr`, we can find the suggestion to `printA`.
+And if we require to find the definition for `printA`, we can go to the definition corretly.
+
+However, since modules bring completely new relationship to the C++ projects and
+clangd doesn't do special corresponding changes. It is natural that the interactions
+are not friendly, there are a lot of defects and even crashes. We can track the newest status
+in https://github.com/clangd/clangd/issues/1293.
+
 Possible Questions
 ==================
 
 How modules speed up compilation
 --------------------------------
 
 A classic theory for the reason why modules speed up the compilation is:
 if there are ``n`` headers and ``m`` source files and each header is included by each source file,