[C++20] [Modules] [ClangScanDeps] Add ClangScanDeps support for C++20 Named Modules in P1689 format (2/4)

[ChuanqiXu]

Summary

Close https://github.com/llvm/llvm-project/issues/51792
Close https://github.com/llvm/llvm-project/issues/56770

This patch adds ClangScanDeps support for C++20 Named Modules in P1689 format. We can find the P1689 format at: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p1689r5.html. As the demo (https://github.com/ChuanqiXu9/llvm-project/tree/MyP1689-ClangScanDeps) shows, with the patch and the one-phase compilation patch (https://reviews.llvm.org/D137059), we're able to compile C++20 Named Modules with CMake! And although P1689 is written by kitware people, other build systems should be able to use the format to compile C++20 Named Modules too.

TODO: Support header units in P1689 Format.
TODO2: Support C++20 Modules in the full dependency format of ClangScanDeps. We also want to support C++20 Modules and clang modules together according to https://discourse.llvm.org/t/how-should-we-support-dependency-scanner-for-c-20-modules/66027. But P1689 format cares about C++20 Modules only for now. So let's focus on C++ Modules and P1689 format. And look at the full dependency format later.

I'll add the ReleaseNotes and Documentations after the patch get landed.

[ChuanqiXu]

Fix a typo

[ChuanqiXu]

clang/include/clang/Frontend/DependencyOutputOptions.h

21

I thought to add the P1689 format here. But many places assume that there're only two DependencyOutputFormat. And the P1689 is different from the Make and NMake since it only cares about the modules. So I decided to add the StdC++ModuleDependencyFormat separate.

clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h

43

I thought to change the name at start. But then I felt this is the most clear name.

clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp

124–133

We can remove these functions if we don't declare them as pure virtual functions.

clang/tools/clang-scan-deps/ClangScanDeps.cpp

623–625

I tried to add a base dependency class and make P1689Deps and FullDeps as the derived class. But later the codes became harder to read and should be harder to maintain too. So I feel like it may be an overkill here to merge them together. The current style is easier to read. Specially we only have two Deps here.

[ben.boeckel]

clang/include/clang/Frontend/DependencyOutputOptions.h

21

This is fine because one technically wants -MM output for scanning rules as well so that if any headers change, the scanning can be performed again.

33

I don't think clang-scan-deps cares, but ideally P1689 would be used for Fortran module dependencies in the future.

clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h

43

You might want to clarify that it is R5 of the paper. Eventually it will be included in the P2222 TR.

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h

70

How is this expected to be represented in the output?

[ChuanqiXu]

Address comments.

[ChuanqiXu]

clang/include/clang/Frontend/DependencyOutputOptions.h

33

I add StdCXX prefix in the name. So it shouldn't be confusing.

clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h

43

I specify it in the comment. It should be fine since we should only support one P1689 format otherwise it would be pretty confusing.

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h

70

This is not expected to be represented in the output for P1689. So the type of ClangModuleMapModule would be simply ignored in P1689 format mode. I add it here for compatibility since ModuleID is originally implemented for Clang Modules.

[ChuanqiXu]

clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h

43

BTW, where is the P2222? I can't find it.

[ben.boeckel]

clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h

43

It's the reserved number for the SG15 Tooling Technical Report. It hasn't had any drafts yet. It'll contain P1689, but it'll be a plenary-blessed document. Making them aliases for each other should be sufficient when it comes out.

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h

70

Ok, sounds good.

[ben.boeckel]

Just to be clear, there will still be the clang mechanism, not just clang-scan-deps.

[tschuett]

In D137527#3915252, @ben.boeckel wrote:

Just to be clear, there will still be the clang mechanism, not just clang-scan-deps.

The clang mechanism sounds to me like Makefiles. As a side-effect clang/gcc drops dependency information on the disk. clang-scan-deps is for me ok let's analyse all of LLVM in one go.

[ben.boeckel]

In D137527#3915267, @tschuett wrote:

The clang mechanism sounds to me like Makefiles. As a side-effect clang/gcc drops dependency information on the disk. clang-scan-deps is for me ok let's analyse all of LLVM in one go.

No, it is P1689 still. Note that one-shot scanning does not work if there are generated sources as you cannot scan everything as they do not exist yet (and cannot if a source is generated by some tool that is being built in the same graph). Other problems:

• if any compile rule changes, everything needs rescanned (as compile_commands.json is a dependency of the scan); and
• in an explicit build, there are @rspfile arguments to the command lines which will not exist until after the scan is performed, so a *different* compile database *just for scanning* is required.

FWIW, CMake would likely use per-target compdbs and scan at that granularity.

[tschuett]

In D137527#3915316, @ben.boeckel wrote:

In D137527#3915267, @tschuett wrote:

The clang mechanism sounds to me like Makefiles. As a side-effect clang/gcc drops dependency information on the disk. clang-scan-deps is for me ok let's analyse all of LLVM in one go.

No, it is P1689 still. Note that one-shot scanning does not work if there are generated sources as you cannot scan everything as they do not exist yet (and cannot if a source is generated by some tool that is being built in the same graph). Other problems:

• if any compile rule changes, everything needs rescanned (as compile_commands.json is a dependency of the scan); and
• in an explicit build, there are @rspfile arguments to the command lines which will not exist until after the scan is performed, so a *different* compile database *just for scanning* is required.

FWIW, CMake would likely use per-target compdbs and scan at that granularity.

Totally agreed. Apple is using clang-scan-deps in production. How do you find new files?

[ben.boeckel]

In D137527#3915357, @tschuett wrote:

Totally agreed. Apple is using clang-scan-deps in production. How do you find new files?

When they get added to a target's source list in the CMake code. Whether explicit or through some (ick) file(GLOB) machinery. There's no other way for the file to show up in CMake's compile_commands.json anyways, so maybe you're referring to something else?

[ChuanqiXu]

In D137527#3915316, @ben.boeckel wrote:

In D137527#3915267, @tschuett wrote:

The clang mechanism sounds to me like Makefiles. As a side-effect clang/gcc drops dependency information on the disk. clang-scan-deps is for me ok let's analyse all of LLVM in one go.

No, it is P1689 still. Note that one-shot scanning does not work if there are generated sources as you cannot scan everything as they do not exist yet (and cannot if a source is generated by some tool that is being built in the same graph). Other problems:

• if any compile rule changes, everything needs rescanned (as compile_commands.json is a dependency of the scan); and
• in an explicit build, there are @rspfile arguments to the command lines which will not exist until after the scan is performed, so a *different* compile database *just for scanning* is required.

FWIW, CMake would likely use per-target compdbs and scan at that granularity.

The per-target scanning is implemented in D137534.

Just to be clear, there will still be the clang mechanism, not just clang-scan-deps.

What's the meaning? What is the clang mechanism about?

[ben.boeckel]

In D137527#3916383, @ChuanqiXu wrote:

What's the meaning? What is the clang mechanism about?

A non-clang-scan-deps mechanism to scan. I'll have to implement per-target compdb bits (while dropping the response file as well). Note that these compdb files are only useful for scanning, not anything else; they'll be stuffed somewhere under CMakeFiles/target.dir/$<CONFIG> or the like.

[ChuanqiXu]

In D137527#3916459, @ben.boeckel wrote:

In D137527#3916383, @ChuanqiXu wrote:

What's the meaning? What is the clang mechanism about?

A non-clang-scan-deps mechanism to scan. I'll have to implement per-target compdb bits (while dropping the response file as well). Note that these compdb files are only useful for scanning, not anything else; they'll be stuffed somewhere under CMakeFiles/target.dir/$<CONFIG> or the like.

Is D137534 sufficient for the per-target strategy?

[aaronmondal]

@ChuanqiXu I think I only now realized the potential impact of this.

Does this mean that if we have support for these depfiles we can just make clang find deps once, let it generate a depfile and then pass that depfile to downstream compilations so that we don't need to search for transitive deps (like headers and module BMIs) again? Does this already work for "regular" headers?

If I understand correctly, this could make builds, especially incremental builds MUCH faster, right?

[ChuanqiXu]

In D137527#3983165, @aaronmondal wrote:

@ChuanqiXu I think I only now realized the potential impact of this.

Does this mean that if we have support for these depfiles we can just make clang find deps once, let it generate a depfile and then pass that depfile to downstream compilations so that we don't need to search for transitive deps (like headers and module BMIs) again?

@aaronmondal

No. It won't handle transitive deps. It just produces the direct deps for each certain source file(s) standalone. See the Checks.cpp under P1689.cppm for the expected output. It is the responsibility of build systems to read and construct the full dependency graph.

For example of the P1689.cppm, now when you see a source file called User.cpp, and its output is User.o (The name is set by the build system by the compilation database. You can call it User.pcm if you support it.) Then the build system can lookup User.o in the output of the scan for the primary-output entry. Then the result told you there are 2 directly dependencies: M and third_party_module. And module M has the corresponding source path so M is a in-project module so we can build it. And we can lookup again for the corresponding source file. And for third_party_module, it doesn't have the source file so it is a third-party module. So the build system needs only to find the built third_party_module (find the third_party_module.pcm in the given options) instead of building it.\

Note that the above paragraph only describes a usage. It doesn't mean the build system must follow the suggestion. The patch itself (or the series patch itself) only provides the scanning results in a certain format. And it is free for the consumers for how to use it. For example, cmake uses a different strategy to query the dependencies.

You can look at https://github.com/ChuanqiXu9/llvm-project/tree/MyP1689-ClangScanDeps for an example. In the example, cmake requires the user to specify the set of module units. But it is not necessary. (maybe it would be more expensive to build the dependency graph if the user doesn't provides the set of module units.) And for the cmake codes, you can look at: https://gitlab.kitware.com/cmake/cmake/-/merge_requests/7978.

Does this already work for "regular" headers?

No. P1689 format only handles module units.

If I understand correctly, this could make builds, especially incremental builds MUCH faster, right?

No. It wouldn't make the builds faster in any sense. (Maybe it would make the build slower a little bit since it requires the build system to construct the build graph by it self). But it will make it easier for the users to describe the dependencies in the build scripts. For example, if I remember correctly, rules_ll requires the user to specify the module dependency of each file explicitly. I believe the syntax will be potentially simpler with the enhanced clang-scan-deps.

BTW, with the patch, we can handle more complicated dependency case:

#ifdef USE_A_MODULE
import a;
#elif USE_B_MODULE
import b;
#elif ...
...

With the patch, it will be much more simpler to handle the case precisely.

clang/test/ClangScanDeps/P1689.cppm

82

Here

[ben.boeckel]

Does this already work for "regular" headers?

P1689R0 and P1689R1 had a replacement for -MF bits, but it was dropped in order to simplify and focus what the format is actually for (it even had a place to indicate MSVC's #pragma comment(lib) "autolinking"…behavior, but was dropped in the hopes that autolinking is niche enough to not cater to in such a format. However, the versioning of the format is sufficient for these to come back if it is ever deemed important enough.

[aaronmondal]

Thanks a lot for the clarification.

It wouldn't make the builds faster in any sense. (Maybe it would make the build slower a little bit since it requires the build system to construct the build graph by it self). But it will make it easier for the users to describe the dependencies in the build scripts.

To me this is more than enough to make this feature very interesting 😃. Not having to maintain both module names and filenames for module builds will vastly outweigh the time needed to generate depfiles.

P1689R0 and P1689R1 had a replacement for -MF bits, but it was dropped in order to simplify and focus what the format is actually for

Seems like a good thing to me. I initially thought that there may be a way pass the paths to the headers already found by one clang invocation to another one so that -I was ignored for all headers that were already included by a previous target. On second thought, this was probably too naive and I wonder if the header file search actually makes up a relevant portion of a clang invocation. Probably not 😅

[ChuanqiXu]

In D137527#3985552, @aaronmondal wrote:

Thanks a lot for the clarification.

It wouldn't make the builds faster in any sense. (Maybe it would make the build slower a little bit since it requires the build system to construct the build graph by it self). But it will make it easier for the users to describe the dependencies in the build scripts.

To me this is more than enough to make this feature very interesting 😃. Not having to maintain both module names and filenames for module builds will vastly outweigh the time needed to generate depfiles.

P1689R0 and P1689R1 had a replacement for -MF bits, but it was dropped in order to simplify and focus what the format is actually for

Seems like a good thing to me. I initially thought that there may be a way pass the paths to the headers already found by one clang invocation to another one so that -I was ignored for all headers that were already included by a previous target. On second thought, this was probably too naive and I wonder if the header file search actually makes up a relevant portion of a clang invocation. Probably not 😅

https://reviews.llvm.org/D139168 tried to get the necessary headers within same invocation to avoid duplicate scanning. Although it can find the headers, it requires the user/build systems to pass -I in the compilation databases ahead of time. So -I couldn't be ignored.

[ben.boeckel]

clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h

98

This needs to use std::nullopt on the latest main. Perhaps these should be rebased?

[ChuanqiXu]

Rebase on main.

[ChuanqiXu]

clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h

98

Yeah, done.

[jansvoboda11]

clang/include/clang/Frontend/DependencyOutputOptions.h

84

Why does this live in DependencyOutputOptions? The only place you set and read this is the dependency scanner. Can't you propagate the clang-scan-deps flag to the collector without leaking this into Clang proper?

[ChuanqiXu]

Address comments to propagate the StdModuleDepFormat option from DependencyOutputOptions to ModuleDepCollector.

[ChuanqiXu]

clang/include/clang/Frontend/DependencyOutputOptions.h

84

I felt the semantics will be better sine StdModuleDepFormat is about the dependency output and DependencyOutputOptions controls the dependency output. I followed your suggestion since it makes sense too. And it won't be too late if we need to add it back to DependencyOutputOptions someday.

[jansvoboda11]

To be clear, my suggestion was not to propagate the option into the collector through DependencyOutputOptions.

I think having this more isolated and not leaking into Clang (where it's not needed) is cleaner, so I suggest leaving DependencyOutputOptions unchanged and propagating the option from clang-scan-deps directly into the collector.

clang/include/clang/Frontend/DependencyOutputOptions.h

84

I see, your reasoning makes sense. My understanding is that DependencyOutputOptions primarily control the .d files that may be emitted during compilation. We do take these options into account in the clang-scan-deps --format make mode, but that looks like an implementation shortcut that could be improved/eliminated, rather than mimicked.

[ChuanqiXu]

In D137527#4024055, @jansvoboda11 wrote:

To be clear, my suggestion was not to propagate the option into the collector through DependencyOutputOptions.

I think having this more isolated and not leaking into Clang (where it's not needed) is cleaner, so I suggest leaving DependencyOutputOptions unchanged and propagating the option from clang-scan-deps directly into the collector.

Oh, sorry. My previous comments are confusing. In the current revision, we don't touch DependencyOutputOptions and we propagating the option from clang-scan-deps directly into the collector indeed.

[jansvoboda11]

In D137527#4024695, @ChuanqiXu wrote:

In D137527#4024055, @jansvoboda11 wrote:

To be clear, my suggestion was not to propagate the option into the collector through DependencyOutputOptions.

I think having this more isolated and not leaking into Clang (where it's not needed) is cleaner, so I suggest leaving DependencyOutputOptions unchanged and propagating the option from clang-scan-deps directly into the collector.

Oh, sorry. My previous comments are confusing. In the current revision, we don't touch DependencyOutputOptions and we propagating the option from clang-scan-deps directly into the collector indeed.

I'm sorry, Phabricator showed me a stale revision that still modified DependencyOutputOptions. This is resolved now.

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h

60

Is this necessary to uniquely identify a standard C++ module? I.e. can we have two modules with the same name and context hash that have different source paths?

63

Same question here.

72

Same question here.

[ben.boeckel]

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h

60

It's possible:

cmake
add_library(foo)
target_sources(foo PUBLIC FILE_SET CXX_MODULES FILES
  $<$<CONFIG:Debug>:module_m_debug.cpp>
  $<$<NOT:$<CONFIG:Debug>>:module_m.cpp>)

where both provide export module M;, just with different bits due to debug (or any other reason, really).

[jansvoboda11]

clang/tools/clang-scan-deps/ClangScanDeps.cpp

671

How is this API expected to work if you provide a module name?

[jansvoboda11]

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h

60

That would be reflected in the context hash. (Or rather hash of the command line the scanner generates.)

[ChuanqiXu]

Address comments:

• Add struct P1689ModuleInfo to avoid pollute ModuleID.
• Remove meaningless parameter ModuleName from getP1689ModuleDependencyFile

[ChuanqiXu]

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h

60

You're right. It looks not good to add these things to ModuleID. Previously I thought it may be better to reuse the ModuleName in ModuleID. But it may be better to not touch ModuleID in P1689 now.

clang/tools/clang-scan-deps/ClangScanDeps.cpp

671

Oh, my bad. We should remove it. Thanks for noting this!

[ChuanqiXu]

Forgot to submit the inline comments. My bad.

[jansvoboda11]

LGTM provided my comments get resolved, but I want @Bigcheese to look over this patch too.

clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h

71

AFAIK we're moving off of llvm::Optional. It's probably better to use std::optional in new code. Applies to the rest of the patch.

95

Does this need to be fully qualified by clang::tooling::dependencies::? We're already in that namespace. Applies to the rest of the patch.

[ChuanqiXu]

Address comments.

[ChuanqiXu]

In D137527#4069750, @jansvoboda11 wrote:

LGTM provided my comments get resolved, but I want @Bigcheese to look over this patch too.

Thanks for reviewing this! I want to land this in 2.10 if no big problem shows up. Since 16.x is branched and 16.0 is going to be released in the early March. I want to have more time to backport, write documents and more tests.

[ChuanqiXu]

I am going to land this since many c++ users want to use cmake with modules for clang16. And it is not good for clang to be slow in this direction.

[uabelho]

Hi,

There seems to be something funny going on with the

ClangScanDeps/P1689.cppm

testcase added in this patch.

It fails randomly for me on top of tree (4ad8f7a189570).
Roughly 1 in 10 runs fail for me.
E.g. with a crash like

Exit Code: 2

Command Output (stderr):
--
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
#0 0x00000000004ef8b8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ef8b8)
#1 0x00000000004ed5ae llvm::sys::RunSignalHandlers() (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ed5ae)
#2 0x00000000004f0096 SignalHandler(int) Signals.cpp:0:0
#3 0x00007f6518263630 __restore_rt sigaction.c:0:0
#4 0x0000000000474f40 P1689Deps::addSourcePathsToRequires() crtstuff.c:0:0
#5 0x00000000004666d1 P1689Deps::printDependencies(llvm::raw_ostream&) crtstuff.c:0:0
#6 0x000000000046204a main (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x46204a)
#7 0x00007f6515996555 __libc_start_main (/lib64/libc.so.6+0x22555)
#8 0x000000000045fcad _start (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x45fcad)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /repo/uabelho/main-github/llvm/build-all/bin/FileCheck /repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp/Checks.cpp -DPREFIX=/repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp

--

********************
********************
Failed Tests (1):
  Clang :: ClangScanDeps/P1689.cppm

or

Exit Code: 2

Command Output (stderr):
--
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
 #0 0x00000000004ef8b8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ef8b8)
 #1 0x00000000004ed5ae llvm::sys::RunSignalHandlers() (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ed5ae)
 #2 0x00000000004f0096 SignalHandler(int) Signals.cpp:0:0
 #3 0x00007f422b33e630 __restore_rt sigaction.c:0:0
 #4 0x00007f4228a85387 raise (/lib64/libc.so.6+0x36387)
 #5 0x00007f4228a86a78 abort (/lib64/libc.so.6+0x37a78)
 #6 0x00007f4228ac7f67 __libc_message (/lib64/libc.so.6+0x78f67)
 #7 0x00007f4228ad0329 _int_free malloc.c:0:0
 #8 0x000000000047e7ef void std::vector<clang::tooling::dependencies::P1689Rule, std::allocator<clang::tooling::dependencies::P1689Rule>>::_M_realloc_insert<clang::tooling::dependencies::P1689Rule const&>(__gnu_cxx::__normal_iterator<clang::tooling::dependencies::P1689Rule*, std::vector<clang::tooling::dependencies::P1689Rule, std::allocator<clang::tooling::dependencies::P1689Rule>>>, clang::tooling::dependencies::P1689Rule const&) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x47e7ef)
 #9 0x000000000047cea1 std::_Function_handler<void (), main::$_1>::_M_invoke(std::_Any_data const&) ClangScanDeps.cpp:0:0
#10 0x000000000047ba3c std::_Function_handler<void (), llvm::ThreadPool::createTaskAndFuture(std::function<void ()>)::'lambda'()>::_M_invoke(std::_Any_data const&) crtstuff.c:0:0
#11 0x00000000004b73e9 llvm::ThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4b73e9)
#12 0x00000000004b8358 void* llvm::thread::ThreadProxy<std::tuple<llvm::ThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0
#13 0x00007f422b336ea5 start_thread pthread_create.c:0:0
#14 0x00007f4228b4db0d clone (/lib64/libc.so.6+0xfeb0d)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /repo/uabelho/main-github/llvm/build-all/bin/FileCheck /repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp/Checks.cpp -DPREFIX=/repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp

--

********************
********************
Failed Tests (1):
  Clang :: ClangScanDeps/P1689.cppm

but I've also seen things like

*** Error in `/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps': corrupted size vs. prev_size: 0x00007faf74000fc0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f474)[0x7fafa1ffa474]
/lib64/libc.so.6(+0x8156b)[0x7fafa1ffc56b]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x47e7ef]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x47cea1]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x47ba3c]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x4b73e9]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x4b8358]
/lib64/libpthread.so.0(+0x7ea5)[0x7fafa4862ea5]
/lib64/libc.so.6(clone+0x6d)[0x7fafa2079b0d]
======= Memory map: ========
00400000-08ffa000 r-xp 00000000 fd:01 63551                              /repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps
08ffa000-09b44000 rw-p 08bf9000 fd:01 63551                              /repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps
09b44000-09bbf000 rw-p 00000000 00:00 0 
0a86a000-0a8f3000 rw-p 00000000 00:00 0                                  [heap]
7faf64000000-7faf64021000 rw-p 00000000 00:00 0 
7faf64021000-7faf68000000 ---p 00000000 00:00 0 
[...]

[thakis]

Either this or D139168 is breaking tests on Windows: http://45.33.8.238/win/74780/step_7.txt

Please take a look and revert for now if it takes a while to fix.

[TWeaver]

@thakis @ChuanqiXu I have a revision up for review here https://reviews.llvm.org/D143749 that adds back the UNSUPPORTED: system-windows that was reverted in https://reviews.llvm.org/D139168

[ChuanqiXu]

In D137527#4117675, @uabelho wrote:

Hi,

There seems to be something funny going on with the

ClangScanDeps/P1689.cppm

testcase added in this patch.

It fails randomly for me on top of tree (4ad8f7a189570).
Roughly 1 in 10 runs fail for me.
E.g. with a crash like

Exit Code: 2

Command Output (stderr):
--
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
#0 0x00000000004ef8b8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ef8b8)
#1 0x00000000004ed5ae llvm::sys::RunSignalHandlers() (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ed5ae)
#2 0x00000000004f0096 SignalHandler(int) Signals.cpp:0:0
#3 0x00007f6518263630 __restore_rt sigaction.c:0:0
#4 0x0000000000474f40 P1689Deps::addSourcePathsToRequires() crtstuff.c:0:0
#5 0x00000000004666d1 P1689Deps::printDependencies(llvm::raw_ostream&) crtstuff.c:0:0
#6 0x000000000046204a main (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x46204a)
#7 0x00007f6515996555 __libc_start_main (/lib64/libc.so.6+0x22555)
#8 0x000000000045fcad _start (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x45fcad)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /repo/uabelho/main-github/llvm/build-all/bin/FileCheck /repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp/Checks.cpp -DPREFIX=/repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp

--

********************
********************
Failed Tests (1):
  Clang :: ClangScanDeps/P1689.cppm

or

Exit Code: 2

Command Output (stderr):
--
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
 #0 0x00000000004ef8b8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ef8b8)
 #1 0x00000000004ed5ae llvm::sys::RunSignalHandlers() (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ed5ae)
 #2 0x00000000004f0096 SignalHandler(int) Signals.cpp:0:0
 #3 0x00007f422b33e630 __restore_rt sigaction.c:0:0
 #4 0x00007f4228a85387 raise (/lib64/libc.so.6+0x36387)
 #5 0x00007f4228a86a78 abort (/lib64/libc.so.6+0x37a78)
 #6 0x00007f4228ac7f67 __libc_message (/lib64/libc.so.6+0x78f67)
 #7 0x00007f4228ad0329 _int_free malloc.c:0:0
 #8 0x000000000047e7ef void std::vector<clang::tooling::dependencies::P1689Rule, std::allocator<clang::tooling::dependencies::P1689Rule>>::_M_realloc_insert<clang::tooling::dependencies::P1689Rule const&>(__gnu_cxx::__normal_iterator<clang::tooling::dependencies::P1689Rule*, std::vector<clang::tooling::dependencies::P1689Rule, std::allocator<clang::tooling::dependencies::P1689Rule>>>, clang::tooling::dependencies::P1689Rule const&) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x47e7ef)
 #9 0x000000000047cea1 std::_Function_handler<void (), main::$_1>::_M_invoke(std::_Any_data const&) ClangScanDeps.cpp:0:0
#10 0x000000000047ba3c std::_Function_handler<void (), llvm::ThreadPool::createTaskAndFuture(std::function<void ()>)::'lambda'()>::_M_invoke(std::_Any_data const&) crtstuff.c:0:0
#11 0x00000000004b73e9 llvm::ThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4b73e9)
#12 0x00000000004b8358 void* llvm::thread::ThreadProxy<std::tuple<llvm::ThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0
#13 0x00007f422b336ea5 start_thread pthread_create.c:0:0
#14 0x00007f4228b4db0d clone (/lib64/libc.so.6+0xfeb0d)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /repo/uabelho/main-github/llvm/build-all/bin/FileCheck /repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp/Checks.cpp -DPREFIX=/repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp

--

********************
********************
Failed Tests (1):
  Clang :: ClangScanDeps/P1689.cppm

but I've also seen things like

*** Error in `/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps': corrupted size vs. prev_size: 0x00007faf74000fc0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f474)[0x7fafa1ffa474]
/lib64/libc.so.6(+0x8156b)[0x7fafa1ffc56b]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x47e7ef]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x47cea1]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x47ba3c]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x4b73e9]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x4b8358]
/lib64/libpthread.so.0(+0x7ea5)[0x7fafa4862ea5]
/lib64/libc.so.6(clone+0x6d)[0x7fafa2079b0d]
======= Memory map: ========
00400000-08ffa000 r-xp 00000000 fd:01 63551                              /repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps
08ffa000-09b44000 rw-p 08bf9000 fd:01 63551                              /repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps
09b44000-09bbf000 rw-p 00000000 00:00 0 
0a86a000-0a8f3000 rw-p 00000000 00:00 0                                  [heap]
7faf64000000-7faf64021000 rw-p 00000000 00:00 0 
7faf64021000-7faf68000000 ---p 00000000 00:00 0 
[...]

This should be caused by https://reviews.llvm.org/D139168 and I've reverted it. And the windows problem should be addressed by https://reviews.llvm.org/D143749. Thank you!

[dyung]

I'm still seeing random crashes today on a linux buildbot in the test you added P1689.cppm. Can you either fix or revert?

• https://lab.llvm.org/buildbot/#/builders/139/builds/35899
• https://lab.llvm.org/buildbot/#/builders/139/builds/35886

corrupted size vs. prev_size while consolidating
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
 #0 0x000055baf810283f llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/bin/clang-scan-deps+0x88883f)
 #1 0x000055baf8100214 SignalHandler(int) Signals.cpp:0:0
 #2 0x00007fa226358420 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x14420)
 #3 0x00007fa225e2500b raise /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/raise.c:51:1
 #4 0x00007fa225e04859 abort /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:81:7
 #5 0x00007fa225e6f26e __libc_message /build/glibc-SzIz7B/glibc-2.31/libio/../sysdeps/posix/libc_fatal.c:155:5
 #6 0x00007fa225e772fc /build/glibc-SzIz7B/glibc-2.31/malloc/malloc.c:5348:3
 #7 0x00007fa225e7904e _int_free /build/glibc-SzIz7B/glibc-2.31/malloc/malloc.c:4245:6
 #8 0x00007fa225e7c88d tcache_thread_shutdown /build/glibc-SzIz7B/glibc-2.31/malloc/malloc.c:2960:33
 #9 0x00007fa225e7c88d __malloc_arena_thread_freeres /build/glibc-SzIz7B/glibc-2.31/malloc/arena.c:951:3
#10 0x00007fa22634c62f start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:496:7
#11 0x00007fa225f01133 __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:97:0

[chapuni]

Excuse me, I have reverted 79a3803bb2cc, D137534 and this.
See also https://lab.llvm.org/buildbot/#/builders/239/builds/922

I guess P1689Deps would not be thread-safe.

[ChuanqiXu]

While I failed to reproduce the failure in my local environment, I found a thread-safe defect in my code. I forgot to add a Lock for P1689::addRules. I'd like to recommit this one and see if the bot would still complain. Thank you for reporting and reverting this.

[ChuanqiXu]

Add unsupported: windows.

clang/include/clang/Frontend/DependencyOutputOptions.h

 #include <vector>
 
 namespace clang {
 
 /// ShowIncludesDestination - Destination for /showIncludes output.
 enum class ShowIncludesDestination { None, Stdout, Stderr };
 
 /// DependencyOutputFormat - Format for the compiler dependency file.
 enum class DependencyOutputFormat { Make, NMake };
 
 /// ExtraDepKind - The kind of extra dependency file.
 enum ExtraDepKind {
   EDK_SanitizeIgnorelist,
   EDK_ProfileList,
   EDK_ModuleFile,
   EDK_DepFileEntry,
 };
 
+/// StdCXXModuleDependencyFormat - Format for the std cxx module dependency
+/// file.
+enum class StdCXXModuleDependencyFormat { None, P1689 };
+
 /// DependencyOutputOptions - Options for controlling the compiler dependency
 /// file generation.
 class DependencyOutputOptions {
 public:
   unsigned IncludeSystemHeaders : 1; ///< Include system header dependencies.
   unsigned ShowHeaderIncludes : 1;   ///< Show header inclusions (-H).
   unsigned UsePhonyTargets : 1;      ///< Include phony targets for each
                                      /// dependency, which can avoid some 'make'
public:
   std::string ShowIncludesPretendHeader;
 
   /// The file to write GraphViz-formatted header dependencies to.
   std::string DOTOutputFile;
 
   /// The directory to copy module dependencies to when collecting them.
   std::string ModuleDependencyOutputDir;
 
+  /// The format for the module dependency file.
+  StdCXXModuleDependencyFormat StdModuleDepFormat =
+      StdCXXModuleDependencyFormat::None;
+
 public:
   DependencyOutputOptions()
       : IncludeSystemHeaders(0), ShowHeaderIncludes(0), UsePhonyTargets(0),
         AddMissingHeaderDeps(0), IncludeModuleFiles(0),
         ShowSkippedHeaderIncludes(0) {}
 };
 
 }  // end namespace clang
 
 #endif

clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h

 
 /// The format that is output by the dependency scanner.
 enum class ScanningOutputFormat {
   /// This is the Makefile compatible dep format. This will include all of the
   /// deps necessary for an implicit modules build, but won't include any
   /// intermodule dependency information.
   Make,
 
-  /// This outputs the full module dependency graph suitable for use for
+  /// This outputs the full clang module dependency graph suitable for use for
   /// explicitly building modules.
   Full,
+
+  /// This outputs the dependency graph for standard c++ modules in P1689R5
+  /// format.
+  P1689,
 };
 
 /// The dependency scanning service contains shared configuration and state that
 /// is used by the individual dependency scanning workers.
 class DependencyScanningService {
 public:
   DependencyScanningService(ScanningMode Mode, ScanningOutputFormat Format,
                             bool OptimizeArgs = false,

clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h

struct FullDependencies {
   std::vector<std::string> DriverCommandLine;
 };
 
 struct FullDependenciesResult {
   FullDependencies FullDeps;
   std::vector<ModuleDeps> DiscoveredModules;
 };
 
+struct P1689Rule {
+  std::string PrimaryOutput;
+  llvm::Optional<ModuleID> Provides;
+  std::vector<ModuleID> Requires;
+};
+
 /// The high-level implementation of the dependency discovery tool that runs on
 /// an individual worker thread.
 class DependencyScanningTool {
 public:
   /// Construct a dependency scanning tool.
   DependencyScanningTool(DependencyScanningService &Service,
                          llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> FS =
                              llvm::vfs::createPhysicalFileSystem());
 
   /// Print out the dependency information into a string using the dependency
   /// file format that is specified in the options (-MD is the default) and
   /// return it. If \p ModuleName isn't empty, this function returns the
   /// dependency information of module \p ModuleName.
   ///
   /// \returns A \c StringError with the diagnostic output if clang errors
   /// occurred, dependency file contents otherwise.
   llvm::Expected<std::string>
   getDependencyFile(const std::vector<std::string> &CommandLine, StringRef CWD,
                     llvm::Optional<StringRef> ModuleName = None);
 
+  llvm::Expected<clang::tooling::dependencies::P1689Rule>
+  getP1689ModuleDependencyFile(const clang::tooling::CompileCommand &Command,
+                               StringRef CWD,
+                               llvm::Optional<StringRef> ModuleName = None);
+
   /// Collect the full module dependency graph for the input, ignoring any
   /// modules which have already been seen. If \p ModuleName isn't empty, this
   /// function returns the full dependency information of module \p ModuleName.
   ///
   /// \param AlreadySeen This stores modules which have previously been
   ///                    reported. Use the same instance for all calls to this
   ///                    function for a single \c DependencyScanningTool in a
   ///                    single build. Use a different one for different tools,

clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h

struct Command {
   std::string Executable;
   std::vector<std::string> Arguments;
 };
 
 class DependencyConsumer {
 public:
   virtual ~DependencyConsumer() {}
 
-  virtual void handleBuildCommand(Command Cmd) = 0;
+  virtual void
+  handleProvidedAndRequiredStdCXXModules(llvm::Optional<ModuleID> Provided,
+                                         std::vector<ModuleID> Requires) {}
+
+  virtual void handleBuildCommand(Command Cmd) {}
 
   virtual void
   handleDependencyOutputOpts(const DependencyOutputOptions &Opts) = 0;
 
   virtual void handleFileDependency(StringRef Filename) = 0;
 
   virtual void handlePrebuiltModuleDependency(PrebuiltModuleDep PMD) = 0;
 

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h

struct ModuleID {
   ///
   /// Modules with the same name and ContextHash but different invocations could
   /// cause non-deterministic build results.
   ///
   /// Modules with the same name but a different \c ContextHash should be
   /// treated as separate modules for the purpose of a build.
   std::string ContextHash;
 
+  /// Optional. The source path to the module.
+  std::string SourcePath;
+
+  /// If this module is a standard c++ interface unit.
+  bool IsStdCXXModuleInterface = true;
+
+  enum class ModuleType {
+    NamedCXXModule,
+    // To be supported
+    // AngleHeaderUnit,
+    // QuoteHeaderUnit,
+    ClangModuleMapModule
+  };
+  ModuleType Type = ModuleType::ClangModuleMapModule;
+
   bool operator==(const ModuleID &Other) const {
     return ModuleName == Other.ModuleName && ContextHash == Other.ContextHash;
   }
 };
 
 /// An output from a module compilation, such as the path of the module file.
 enum class ModuleOutputKind {
   /// The module file (.pcm). Required.
private:
   std::unique_ptr<DependencyOutputOptions> Opts;
   /// The original Clang invocation passed to dependency scanner.
   CompilerInvocation OriginalInvocation;
   /// Whether to optimize the modules' command-line arguments.
   bool OptimizeArgs;
   /// Whether to set up command-lines to load PCM files eagerly.
   bool EagerLoadModules;
 
+  llvm::Optional<ModuleID> ProvidedStdCXXModule;
+  std::vector<ModuleID> RequiredStdCXXModules;
+
+  bool isP1689Format() const;
   /// Checks whether the module is known as being prebuilt.
   bool isPrebuiltModule(const Module *M);
 
   /// Adds \p Path to \c FileDeps, making it absolute if necessary.
   void addFileDep(StringRef Path);
   /// Adds \p Path to \c MD.FileDeps, making it absolute if necessary.
   void addFileDep(ModuleDeps &MD, StringRef Path);
 

clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp

auto Result =
       Worker.computeDependencies(CWD, CommandLine, Consumer, ModuleName);
   if (Result)
     return std::move(Result);
   std::string Output;
   Consumer.printDependencies(Output);
   return Output;
 }
 
+llvm::Expected<P1689Rule> DependencyScanningTool::getP1689ModuleDependencyFile(
+    const clang::tooling::CompileCommand &Command, StringRef CWD,
+    llvm::Optional<StringRef> ModuleName) {
+  class P1689ModuleDependencyPrinterConsumer : public DependencyConsumer {
+  public:
+    P1689ModuleDependencyPrinterConsumer(P1689Rule &Rule,
+                                         const CompileCommand &Command)
+        : Filename(Command.Filename), Rule(Rule) {
+      Rule.PrimaryOutput = Command.Output;
+    }
+
+    void
+    handleDependencyOutputOpts(const DependencyOutputOptions &Opts) override {}
+    void handleFileDependency(StringRef File) override {}
+    void handlePrebuiltModuleDependency(PrebuiltModuleDep PMD) override {}
+    void handleModuleDependency(ModuleDeps MD) override {}
+    void handleContextHash(std::string Hash) override {}
+    std::string lookupModuleOutput(const ModuleID &ID,
+                                   ModuleOutputKind Kind) override {
+      llvm::report_fatal_error("unexpected call to lookupModuleOutput");
+    }
+
+    void handleProvidedAndRequiredStdCXXModules(
+        llvm::Optional<ModuleID> Provided,
+        std::vector<ModuleID> Requires) override {
+      Rule.Provides = Provided;
+      if (Rule.Provides)
+        Rule.Provides->SourcePath = Filename.str();
+      Rule.Requires = Requires;
+    }
+
+  private:
+    StringRef Filename;
+    clang::tooling::dependencies::P1689Rule &Rule;
+  };
+
+  P1689Rule Rule;
+  P1689ModuleDependencyPrinterConsumer Consumer(Rule, Command);
+  auto Result = Worker.computeDependencies(CWD, Command.CommandLine, Consumer,
+                                           ModuleName);
+  if (Result)
+    return std::move(Result);
+  return Rule;
+}
+
 llvm::Expected<FullDependenciesResult>
 DependencyScanningTool::getFullDependencies(
     const std::vector<std::string> &CommandLine, StringRef CWD,
     const llvm::StringSet<> &AlreadySeen,
     LookupModuleOutputCallback LookupModuleOutput,
     llvm::Optional<StringRef> ModuleName) {
   FullDependencyConsumer Consumer(AlreadySeen, LookupModuleOutput,
                                   Worker.shouldEagerLoadModules());

clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp

bool runInvocation(std::shared_ptr<CompilerInvocation> Invocation,
     Opts->IncludeSystemHeaders = true;
 
     switch (Format) {
     case ScanningOutputFormat::Make:
       ScanInstance.addDependencyCollector(
           std::make_shared<DependencyConsumerForwarder>(
               std::move(Opts), WorkingDirectory, Consumer));
       break;
+    case ScanningOutputFormat::P1689:
+      Opts->StdModuleDepFormat = StdCXXModuleDependencyFormat::P1689;
+      [[fallthrough]];
     case ScanningOutputFormat::Full:
       MDC = std::make_shared<ModuleDepCollector>(
           std::move(Opts), ScanInstance, Consumer, OriginalInvocation,
           OptimizeArgs, EagerLoadModules);
       ScanInstance.addDependencyCollector(MDC);
       break;
     }
 

clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp

if (!File && !Imported) {
     MDC.addFileDep(FileName);
   }
   handleImport(Imported);
 }
 
 void ModuleDepCollectorPP::moduleImport(SourceLocation ImportLoc,
                                         ModuleIdPath Path,
                                         const Module *Imported) {
+  if (MDC.ScanInstance.getPreprocessor().isImportingCXXNamedModules()) {
+    ModuleID RequiredModule;
+    RequiredModule.ModuleName = Path[0].first->getName().str();
+    RequiredModule.Type = ModuleID::ModuleType::NamedCXXModule;
+    MDC.RequiredStdCXXModules.push_back(RequiredModule);
+    return;
+  }
+
   handleImport(Imported);
 }
 
 void ModuleDepCollectorPP::handleImport(const Module *Imported) {
   if (!Imported)
     return;
 
   const Module *TopLevelModule = Imported->getTopLevelModule();
 
   if (MDC.isPrebuiltModule(TopLevelModule))
     MDC.DirectPrebuiltModularDeps.insert(
         {TopLevelModule, PrebuiltModuleDep{TopLevelModule}});
   else
     DirectModularDeps.insert(TopLevelModule);
 }
 
 void ModuleDepCollectorPP::EndOfMainFile() {
   FileID MainFileID = MDC.ScanInstance.getSourceManager().getMainFileID();
   MDC.MainFile = std::string(MDC.ScanInstance.getSourceManager()
                                  .getFileEntryForID(MainFileID)
                                  ->getName());
 
+  auto &PP = MDC.ScanInstance.getPreprocessor();
+  if (PP.isNamedModule()) {
+    ModuleID ProvidedModule;
+    ProvidedModule.ModuleName = PP.getNamedModuleName();
+    ProvidedModule.Type = ModuleID::ModuleType::NamedCXXModule;
+    ProvidedModule.IsStdCXXModuleInterface = PP.isNamedInterfaceUnit();
+    // Don't put implementation (non partition) unit as Provide.
+    // Put the module as required instead. Since the implementation
+    // unit will import the primary module implicitly.
+    if (PP.isImplementationUnit())
+      MDC.RequiredStdCXXModules.push_back(ProvidedModule);
+    else
+      MDC.ProvidedStdCXXModule = ProvidedModule;
+  }
+
   if (!MDC.ScanInstance.getPreprocessorOpts().ImplicitPCHInclude.empty())
     MDC.addFileDep(MDC.ScanInstance.getPreprocessorOpts().ImplicitPCHInclude);
 
   for (const Module *M :
        MDC.ScanInstance.getPreprocessor().getAffectingClangModules())
     if (!MDC.isPrebuiltModule(M))
       DirectModularDeps.insert(M);
 
   for (const Module *M : DirectModularDeps) {
     // A top-level module might not be actually imported as a module when
     // -fmodule-name is used to compile a translation unit that imports this
     // module. In that case it can be skipped. The appropriate header
     // dependencies will still be reported as expected.
     if (!M->getASTFile())
       continue;
     handleTopLevelModule(M);
   }
 
   MDC.Consumer.handleDependencyOutputOpts(*MDC.Opts);
 
+  if (MDC.isP1689Format())
+    MDC.Consumer.handleProvidedAndRequiredStdCXXModules(
+        MDC.ProvidedStdCXXModule, MDC.RequiredStdCXXModules);
+
   for (auto &&I : MDC.ModularDeps)
     MDC.Consumer.handleModuleDependency(*I.second);
 
   for (auto &&I : MDC.FileDeps)
     MDC.Consumer.handleFileDependency(I);
 
   for (auto &&I : MDC.DirectPrebuiltModularDeps)
     MDC.Consumer.handlePrebuiltModuleDependency(I.second);
void ModuleDepCollector::addFileDep(StringRef Path) {
   FileDeps.push_back(std::string(Path));
 }
 
 void ModuleDepCollector::addFileDep(ModuleDeps &MD, StringRef Path) {
   llvm::SmallString<256> Storage;
   Path = makeAbsolute(ScanInstance, Path, Storage);
   MD.FileDeps.insert(Path);
 }
+
+bool ModuleDepCollector::isP1689Format() const {
+  return Opts->StdModuleDepFormat == StdCXXModuleDependencyFormat::P1689;
+}

clang/test/ClangScanDeps/P1689.cppm

+// RUN: rm -fr %t
+// RUN: mkdir -p %t
+// RUN: split-file %s %t
+//
+// RUN: sed "s|DIR|%/t|g" %t/P1689.json.in > %t/P1689.json
+// RUN: clang-scan-deps -compilation-database %t/P1689.json -format=p1689 | FileCheck %t/Checks.cpp -DPREFIX=%/t
+// RUN: clang-scan-deps --mode=preprocess-dependency-directives -compilation-database %t/P1689.json -format=p1689 | FileCheck %t/Checks.cpp -DPREFIX=%/t
+
+//--- P1689.json.in
+[
+{
+  "directory": "DIR",
+  "command": "clang++ -std=c++20 DIR/M.cppm -c -o DIR/M.o",
+  "file": "DIR/M.cppm",
+  "output": "DIR/M.o"
+},
+{
+  "directory": "DIR",
+  "command": "clang++ -std=c++20 DIR/Impl.cpp -c -o DIR/Impl.o",
+  "file": "DIR/Impl.cpp",
+  "output": "DIR/Impl.o"
+},
+{
+  "directory": "DIR",
+  "command": "clang++ -std=c++20 DIR/impl_part.cppm -c -o DIR/impl_part.o",
+  "file": "DIR/impl_part.cppm",
+  "output": "DIR/impl_part.o"
+},
+{
+  "directory": "DIR",
+  "command": "clang++ -std=c++20 DIR/interface_part.cppm -c -o DIR/interface_part.o",
+  "file": "DIR/interface_part.cppm",
+  "output": "DIR/interface_part.o"
+},
+{
+  "directory": "DIR",
+  "command": "clang++ -std=c++20 DIR/User.cpp -c -o DIR/User.o",
+  "file": "DIR/User.cpp",
+  "output": "DIR/User.o"
+}
+]
+
+
+//--- M.cppm
+export module M;
+export import :interface_part;
+import :impl_part;
+export void Hello();
+
+//--- Impl.cpp
+module;
+#include "header.mock"
+module M;
+void Hello() {
+    std::cout << "Hello ";
+}
+
+//--- impl_part.cppm
+module;
+#include "header.mock"
+module M:impl_part;
+import :interface_part;
+
+std::string W = "World.";
+void World() {
+    std::cout << W << std::endl;
+}
+
+//--- interface_part.cppm
+export module M:interface_part;
+export void World();
+
+//--- User.cpp
+import M;
+import third_party_module;
+int main() {
+    Hello();
+    World();
+    return 0;
+}
+
+//--- Checks.cpp
+// CHECK: {
+// CHECK-NEXT:   "revision": 0,
+// CHECK-NEXT:   "rules": [
+// CHECK-NEXT:     {
+// CHECK-NEXT:       "primary-output": "[[PREFIX]]/Impl.o",
+// CHECK-NEXT:       "requires": [
+// CHECK-NEXT:         {
+// CHECK-NEXT:           "logical-name": "M",
+// CHECK-NEXT:           "source-path": "[[PREFIX]]/M.cppm"
+// CHECK-NEXT:         }
+// CHECK-NEXT:       ]
+// CHECK-NEXT:     },
+// CHECK-NEXT:     {
+// CHECK-NEXT:       "primary-output": "[[PREFIX]]/M.o",
+// CHECK-NEXT:       "provides": [
+// CHECK-NEXT:         {
+// CHECK-NEXT:           "is-interface": true,
+// CHECK-NEXT:           "logical-name": "M",
+// CHECK-NEXT:           "source-path": "[[PREFIX]]/M.cppm"
+// CHECK-NEXT:         }
+// CHECK-NEXT:       ],
+// CHECK-NEXT:       "requires": [
+// CHECK-NEXT:         {
+// CHECK-NEXT:           "logical-name": "M:interface_part",
+// CHECK-NEXT:           "source-path": "[[PREFIX]]/interface_part.cppm"
+// CHECK-NEXT:         },
+// CHECK-NEXT:         {
+// CHECK-NEXT:           "logical-name": "M:impl_part",
+// CHECK-NEXT:           "source-path": "[[PREFIX]]/impl_part.cppm"
+// CHECK-NEXT:         }
+// CHECK-NEXT:       ]
+// CHECK-NEXT:     },
+// CHECK-NEXT:     {
+// CHECK-NEXT:       "primary-output": "[[PREFIX]]/User.o",
+// CHECK-NEXT:       "requires": [
+// CHECK-NEXT:         {
+// CHECK-NEXT:           "logical-name": "M",
+// CHECK-NEXT:           "source-path": "[[PREFIX]]/M.cppm"
+// CHECK-NEXT:         },
+// CHECK-NEXT:         {
+// CHECK-NEXT:           "logical-name": "third_party_module"
+// CHECK-NEXT:         }
+// CHECK-NEXT:       ]
+// CHECK-NEXT:     },
+// CHECK-NEXT:     {
+// CHECK-NEXT:       "primary-output": "[[PREFIX]]/impl_part.o",
+// CHECK-NEXT:       "provides": [
+// CHECK-NEXT:         {
+// CHECK-NEXT:           "is-interface": false,
+// CHECK-NEXT:           "logical-name": "M:impl_part",
+// CHECK-NEXT:           "source-path": "[[PREFIX]]/impl_part.cppm"
+// CHECK-NEXT:         }
+// CHECK-NEXT:       ],
+// CHECK-NEXT:       "requires": [
+// CHECK-NEXT:         {
+// CHECK-NEXT:           "logical-name": "M:interface_part",
+// CHECK-NEXT:           "source-path": "[[PREFIX]]/interface_part.cppm"
+// CHECK-NEXT:         }
+// CHECK-NEXT:       ]
+// CHECK-NEXT:     },
+// CHECK-NEXT:     {
+// CHECK-NEXT:       "primary-output": "[[PREFIX]]/interface_part.o",
+// CHECK-NEXT:       "provides": [
+// CHECK-NEXT:         {
+// CHECK-NEXT:           "is-interface": true,
+// CHECK-NEXT:           "logical-name": "M:interface_part",
+// CHECK-NEXT:           "source-path": "[[PREFIX]]/interface_part.cppm"
+// CHECK-NEXT:         }
+// CHECK-NEXT:       ]
+// CHECK-NEXT:     }
+// CHECK-NEXT:   ],
+// CHECK-NEXT:   "version": 1
+// CHECK-NEXT: }
+
+//--- header.mock

clang/tools/clang-scan-deps/ClangScanDeps.cpp

llvm::cl::values(
         clEnumValN(ScanningMode::CanonicalPreprocessing, "preprocess",
                    "The set of dependencies is computed by preprocessing the "
                    "source files")),
     llvm::cl::init(ScanningMode::DependencyDirectivesScan),
     llvm::cl::cat(DependencyScannerCategory));
 
 static llvm::cl::opt<ScanningOutputFormat> Format(
     "format", llvm::cl::desc("The output format for the dependencies"),
-    llvm::cl::values(clEnumValN(ScanningOutputFormat::Make, "make",
-                                "Makefile compatible dep file"),
-                     clEnumValN(ScanningOutputFormat::Full, "experimental-full",
-                                "Full dependency graph suitable"
-                                " for explicitly building modules. This format "
-                                "is experimental and will change.")),
+    llvm::cl::values(
+        clEnumValN(ScanningOutputFormat::Make, "make",
+        clEnumValN(ScanningOutputFormat::P1689, "p1689",
+                   "Generate standard c++ modules dependency P1689 format"),
     llvm::cl::init(ScanningOutputFormat::Make),
     llvm::cl::cat(DependencyScannerCategory));
 
 static llvm::cl::opt<std::string> ModuleFilesDir(
     "module-files-dir",
     llvm::cl::desc(
         "The build directory for modules. Defaults to the value of "
         "'-fmodules-cache-path=' from command lines for implicit modules."),
llvm::handleAllErrors(
           });
         });
     return true;
   }
   FD.mergeDeps(Input, std::move(*MaybeFullDeps), InputIndex);
   return false;
 }
 
+class P1689Deps {
+public:
+  void printDependencies(raw_ostream &OS) {
+    addSourcePathsToRequires();
+    // Sort the modules by name to get a deterministic order.
+    llvm::sort(Rules, [](const P1689Rule &A, const P1689Rule &B) {
+      return A.PrimaryOutput < B.PrimaryOutput;
+    });
+
+    using namespace llvm::json;
+    Array OutputRules;
+    for (const P1689Rule &R : Rules) {
+      Object O{{"primary-output", R.PrimaryOutput}};
+
+      if (R.Provides) {
+        Array Provides;
+        Object Provided{{"logical-name", R.Provides->ModuleName},
+                        {"source-path", R.Provides->SourcePath},
+                        {"is-interface", R.Provides->IsStdCXXModuleInterface}};
+        Provides.push_back(std::move(Provided));
+        O.insert({"provides", std::move(Provides)});
+      }
+
+      Array Requires;
+      for (const ModuleID &Info : R.Requires) {
+        Object RequiredInfo{{"logical-name", Info.ModuleName}};
+        if (!Info.SourcePath.empty())
+          RequiredInfo.insert({"source-path", Info.SourcePath});
+        Requires.push_back(std::move(RequiredInfo));
+      }
+
+      if (!Requires.empty())
+        O.insert({"requires", std::move(Requires)});
+
+      OutputRules.push_back(std::move(O));
+    }
+
+    Object Output{
+        {"version", 1}, {"revision", 0}, {"rules", std::move(OutputRules)}};
+
+    OS << llvm::formatv("{0:2}\n", Value(std::move(Output)));
+  }
+
+  void addRules(P1689Rule &Rule) { Rules.push_back(Rule); }
+
+private:
+  void addSourcePathsToRequires() {
+    llvm::DenseMap<StringRef, StringRef> ModuleSourceMapper;
+    for (const P1689Rule &R : Rules)
+      if (R.Provides && !R.Provides->SourcePath.empty())
+        ModuleSourceMapper[R.Provides->ModuleName] = R.Provides->SourcePath;
+
+    for (P1689Rule &R : Rules) {
+      for (ModuleID &Info : R.Requires) {
+        auto Iter = ModuleSourceMapper.find(Info.ModuleName);
+        if (Iter != ModuleSourceMapper.end())
+          Info.SourcePath = Iter->second;
+      }
+    }
+  }
+
+  std::vector<P1689Rule> Rules;
+};
+
+static bool
+handleP1689DependencyToolResult(const std::string &Input,
+                                llvm::Expected<P1689Rule> &MaybeRule,
+                                P1689Deps &PD, SharedStream &Errs) {
+  if (!MaybeRule) {
+    llvm::handleAllErrors(
+        MaybeRule.takeError(), [&Input, &Errs](llvm::StringError &Err) {
+          Errs.applyLocked([&](raw_ostream &OS) {
+            OS << "Error while scanning dependencies for " << Input << ":\n";
+            OS << Err.getMessage();
+          });
+        });
+    return true;
+  }
+  PD.addRules(*MaybeRule);
+  return false;
+}
+
 /// Construct a path for the explicitly built PCM.
 static std::string constructPCMPath(ModuleID MID, StringRef OutputDir) {
   SmallString<256> ExplicitPCMPath(OutputDir);
   llvm::sys::path::append(ExplicitPCMPath, MID.ContextHash,
                           MID.ModuleName + "-" + MID.ContextHash + ".pcm");
   return std::string(ExplicitPCMPath);
 }
 
int main(int argc, const char **argv) {
   std::vector<std::unique_ptr<DependencyScanningTool>> WorkerTools;
   for (unsigned I = 0; I < Pool.getThreadCount(); ++I)
     WorkerTools.push_back(std::make_unique<DependencyScanningTool>(Service));
 
   std::vector<tooling::CompileCommand> Inputs =
       AdjustingCompilations->getAllCompileCommands();
 
   std::atomic<bool> HadErrors(false);
   FullDeps FD;
+  P1689Deps PD;
   std::mutex Lock;
   size_t Index = 0;
 
   if (Verbose) {
     llvm::outs() << "Running clang-scan-deps on " << Inputs.size()
                  << " files using " << Pool.getThreadCount() << " workers\n";
   }
   for (unsigned I = 0; I < Pool.getThreadCount(); ++I) {
-    Pool.async([I, &Lock, &Index, &Inputs, &HadErrors, &FD, &WorkerTools,
+    Pool.async([I, &Lock, &Index, &Inputs, &HadErrors, &FD, &PD, &WorkerTools,
                 &DependencyOS, &Errs]() {
       llvm::StringSet<> AlreadySeenModules;
       while (true) {
         const tooling::CompileCommand *Input;
         std::string Filename;
         std::string CWD;
         size_t LocalIndex;
         // Take the next input.
Pool.async([I, &Lock, &Index, &Inputs, &HadErrors, &FD, &PD, &WorkerTools,
 
         // Run the tool on it.
         if (Format == ScanningOutputFormat::Make) {
           auto MaybeFile = WorkerTools[I]->getDependencyFile(
               Input->CommandLine, CWD, MaybeModuleName);
           if (handleMakeDependencyToolResult(Filename, MaybeFile, DependencyOS,
                                              Errs))
             HadErrors = true;
+        } else if (Format == ScanningOutputFormat::P1689) {
+          auto MaybeRule = WorkerTools[I]->getP1689ModuleDependencyFile(
+              *Input, CWD, MaybeModuleName);
+          if (handleP1689DependencyToolResult(Filename, MaybeRule, PD, Errs))
+            HadErrors = true;
         } else if (DeprecatedDriverCommand) {
           auto MaybeFullDeps =
               WorkerTools[I]->getFullDependenciesLegacyDriverCommand(
                   Input->CommandLine, CWD, AlreadySeenModules, LookupOutput,
                   MaybeModuleName);
           if (handleFullDependencyToolResult(Filename, MaybeFullDeps, FD,
                                              LocalIndex, DependencyOS, Errs))
             HadErrors = true;
         } else {
           auto MaybeFullDeps = WorkerTools[I]->getFullDependencies(
               Input->CommandLine, CWD, AlreadySeenModules, LookupOutput,
               MaybeModuleName);
           if (handleFullDependencyToolResult(Filename, MaybeFullDeps, FD,
                                              LocalIndex, DependencyOS, Errs))
             HadErrors = true;
         }
       }
     });
   }
   Pool.wait();
 
   if (Format == ScanningOutputFormat::Full)
     FD.printFullOutput(llvm::outs());
+  else if (Format == ScanningOutputFormat::P1689)
+    PD.printDependencies(llvm::outs());
 
   return HadErrors;
 }