This is an archive of the discontinued LLVM Phabricator instance.

[C++20] [Modules] [ClangScanDeps] Add ClangScanDeps support for C++20 Named Modules in P1689 format (2/4)
ClosedPublic

Authored by ChuanqiXu on Nov 6 2022, 10:07 PM.

Details

Summary

Close https://github.com/llvm/llvm-project/issues/51792
Close https://github.com/llvm/llvm-project/issues/56770

This patch adds ClangScanDeps support for C++20 Named Modules in P1689 format. We can find the P1689 format at: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p1689r5.html. As the demo (https://github.com/ChuanqiXu9/llvm-project/tree/MyP1689-ClangScanDeps) shows, with the patch and the one-phase compilation patch (https://reviews.llvm.org/D137059), we're able to compile C++20 Named Modules with CMake! And although P1689 is written by kitware people, other build systems should be able to use the format to compile C++20 Named Modules too.

TODO: Support header units in P1689 Format.
TODO2: Support C++20 Modules in the full dependency format of ClangScanDeps. We also want to support C++20 Modules and clang modules together according to https://discourse.llvm.org/t/how-should-we-support-dependency-scanner-for-c-20-modules/66027. But P1689 format cares about C++20 Modules only for now. So let's focus on C++ Modules and P1689 format. And look at the full dependency format later.

I'll add the ReleaseNotes and Documentations after the patch get landed.

Diff Detail

Event Timeline

ChuanqiXu created this revision.Nov 6 2022, 10:07 PM
Herald added a project: Restricted Project. · View Herald TranscriptNov 6 2022, 10:07 PM
Herald added a subscriber: mgrang. · View Herald Transcript
ChuanqiXu requested review of this revision.Nov 6 2022, 10:07 PM
ChuanqiXu retitled this revision from [C++20] [Modules] [ClangScanDeps] Add ClangScanDeps support for C++20 Named Modules to [C++20] [Modules] [ClangScanDeps] Add ClangScanDeps support for C++20 Named Modules in P1689 format.Nov 6 2022, 10:17 PM
ChuanqiXu edited the summary of this revision. (Show Details)
ChuanqiXu updated this revision to Diff 473563.Nov 6 2022, 10:31 PM

Fix a typo

ChuanqiXu added inline comments.Nov 6 2022, 10:37 PM
clang/include/clang/Frontend/DependencyOutputOptions.h
21

I thought to add the P1689 format here. But many places assume that there're only two DependencyOutputFormat. And the P1689 is different from the Make and NMake since it only cares about the modules. So I decided to add the StdC++ModuleDependencyFormat separate.

clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
43

I thought to change the name at start. But then I felt this is the most clear name.

clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
124–133

We can remove these functions if we don't declare them as pure virtual functions.

clang/tools/clang-scan-deps/ClangScanDeps.cpp
623–625

I tried to add a base dependency class and make P1689Deps and FullDeps as the derived class. But later the codes became harder to read and should be harder to maintain too. So I feel like it may be an overkill here to merge them together. The current style is easier to read. Specially we only have two Deps here.

ChuanqiXu edited the summary of this revision. (Show Details)Nov 6 2022, 10:59 PM
ChuanqiXu retitled this revision from [C++20] [Modules] [ClangScanDeps] Add ClangScanDeps support for C++20 Named Modules in P1689 format to [C++20] [Modules] [ClangScanDeps] Add ClangScanDeps support for C++20 Named Modules in P1689 format (2/3).
ben.boeckel added inline comments.Nov 7 2022, 8:34 AM
clang/include/clang/Frontend/DependencyOutputOptions.h
21

This is fine because one technically wants -MM output for scanning rules as well so that if any headers change, the scanning can be performed again.

33

I don't think clang-scan-deps cares, but ideally P1689 would be used for Fortran module dependencies in the future.

clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
43

You might want to clarify that it is R5 of the paper. Eventually it will be included in the P2222 TR.

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
70

How is this expected to be represented in the output?

ChuanqiXu updated this revision to Diff 473845.Nov 7 2022, 6:17 PM
ChuanqiXu marked 3 inline comments as done.

Address comments.

ChuanqiXu marked an inline comment as done.Nov 7 2022, 6:18 PM
ChuanqiXu added inline comments.
clang/include/clang/Frontend/DependencyOutputOptions.h
33

I add StdCXX prefix in the name. So it shouldn't be confusing.

clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
43

I specify it in the comment. It should be fine since we should only support one P1689 format otherwise it would be pretty confusing.

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
70

This is not expected to be represented in the output for P1689. So the type of ClangModuleMapModule would be simply ignored in P1689 format mode. I add it here for compatibility since ModuleID is originally implemented for Clang Modules.

ChuanqiXu edited the summary of this revision. (Show Details)Nov 7 2022, 6:19 PM
ChuanqiXu marked an inline comment as done.Nov 7 2022, 6:35 PM
ChuanqiXu added inline comments.
clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
43

BTW, where is the P2222? I can't find it.

ben.boeckel added inline comments.Nov 8 2022, 5:11 AM
clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
43

It's the reserved number for the SG15 Tooling Technical Report. It hasn't had any drafts yet. It'll contain P1689, but it'll be a plenary-blessed document. Making them aliases for each other should be sufficient when it comes out.

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
70

Ok, sounds good.

Just to be clear, there will still be the clang mechanism, not just clang-scan-deps.

Just to be clear, there will still be the clang mechanism, not just clang-scan-deps.

The clang mechanism sounds to me like Makefiles. As a side-effect clang/gcc drops dependency information on the disk. clang-scan-deps is for me ok let's analyse all of LLVM in one go.

The clang mechanism sounds to me like Makefiles. As a side-effect clang/gcc drops dependency information on the disk. clang-scan-deps is for me ok let's analyse all of LLVM in one go.

No, it is P1689 still. Note that one-shot scanning does not work if there are generated sources as you cannot scan everything as they do not exist yet (and cannot if a source is generated by some tool that is being built in the same graph). Other problems:

  • if any compile rule changes, everything needs rescanned (as compile_commands.json is a dependency of the scan); and
  • in an explicit build, there are @rspfile arguments to the command lines which will not exist until after the scan is performed, so a *different* compile database *just for scanning* is required.

FWIW, CMake would likely use per-target compdbs and scan at that granularity.

The clang mechanism sounds to me like Makefiles. As a side-effect clang/gcc drops dependency information on the disk. clang-scan-deps is for me ok let's analyse all of LLVM in one go.

No, it is P1689 still. Note that one-shot scanning does not work if there are generated sources as you cannot scan everything as they do not exist yet (and cannot if a source is generated by some tool that is being built in the same graph). Other problems:

  • if any compile rule changes, everything needs rescanned (as compile_commands.json is a dependency of the scan); and
  • in an explicit build, there are @rspfile arguments to the command lines which will not exist until after the scan is performed, so a *different* compile database *just for scanning* is required.

FWIW, CMake would likely use per-target compdbs and scan at that granularity.

Totally agreed. Apple is using clang-scan-deps in production. How do you find new files?

Totally agreed. Apple is using clang-scan-deps in production. How do you find new files?

When they get added to a target's source list in the CMake code. Whether explicit or through some (ick) file(GLOB) machinery. There's no other way for the file to show up in CMake's compile_commands.json anyways, so maybe you're referring to something else?

The clang mechanism sounds to me like Makefiles. As a side-effect clang/gcc drops dependency information on the disk. clang-scan-deps is for me ok let's analyse all of LLVM in one go.

No, it is P1689 still. Note that one-shot scanning does not work if there are generated sources as you cannot scan everything as they do not exist yet (and cannot if a source is generated by some tool that is being built in the same graph). Other problems:

  • if any compile rule changes, everything needs rescanned (as compile_commands.json is a dependency of the scan); and
  • in an explicit build, there are @rspfile arguments to the command lines which will not exist until after the scan is performed, so a *different* compile database *just for scanning* is required.

FWIW, CMake would likely use per-target compdbs and scan at that granularity.

The per-target scanning is implemented in D137534.

Just to be clear, there will still be the clang mechanism, not just clang-scan-deps.

What's the meaning? What is the clang mechanism about?

What's the meaning? What is the clang mechanism about?

A non-clang-scan-deps mechanism to scan. I'll have to implement per-target compdb bits (while dropping the response file as well). Note that these compdb files are only useful for scanning, not anything else; they'll be stuffed somewhere under CMakeFiles/target.dir/$<CONFIG> or the like.

What's the meaning? What is the clang mechanism about?

A non-clang-scan-deps mechanism to scan. I'll have to implement per-target compdb bits (while dropping the response file as well). Note that these compdb files are only useful for scanning, not anything else; they'll be stuffed somewhere under CMakeFiles/target.dir/$<CONFIG> or the like.

Is D137534 sufficient for the per-target strategy?

ChuanqiXu retitled this revision from [C++20] [Modules] [ClangScanDeps] Add ClangScanDeps support for C++20 Named Modules in P1689 format (2/3) to [C++20] [Modules] [ClangScanDeps] Add ClangScanDeps support for C++20 Named Modules in P1689 format (2/4).Dec 4 2022, 7:19 PM

@ChuanqiXu I think I only now realized the potential impact of this.

Does this mean that if we have support for these depfiles we can just make clang find deps once, let it generate a depfile and then pass that depfile to downstream compilations so that we don't need to search for transitive deps (like headers and module BMIs) again? Does this already work for "regular" headers?

If I understand correctly, this could make builds, especially incremental builds MUCH faster, right?

@ChuanqiXu I think I only now realized the potential impact of this.

Does this mean that if we have support for these depfiles we can just make clang find deps once, let it generate a depfile and then pass that depfile to downstream compilations so that we don't need to search for transitive deps (like headers and module BMIs) again?

@aaronmondal

No. It won't handle transitive deps. It just produces the direct deps for each certain source file(s) standalone. See the Checks.cpp under P1689.cppm for the expected output. It is the responsibility of build systems to read and construct the full dependency graph.

For example of the P1689.cppm, now when you see a source file called User.cpp, and its output is User.o (The name is set by the build system by the compilation database. You can call it User.pcm if you support it.) Then the build system can lookup User.o in the output of the scan for the primary-output entry. Then the result told you there are 2 directly dependencies: M and third_party_module. And module M has the corresponding source path so M is a in-project module so we can build it. And we can lookup again for the corresponding source file. And for third_party_module, it doesn't have the source file so it is a third-party module. So the build system needs only to find the built third_party_module (find the third_party_module.pcm in the given options) instead of building it.\

Note that the above paragraph only describes a usage. It doesn't mean the build system must follow the suggestion. The patch itself (or the series patch itself) only provides the scanning results in a certain format. And it is free for the consumers for how to use it. For example, cmake uses a different strategy to query the dependencies.

You can look at https://github.com/ChuanqiXu9/llvm-project/tree/MyP1689-ClangScanDeps for an example. In the example, cmake requires the user to specify the set of module units. But it is not necessary. (maybe it would be more expensive to build the dependency graph if the user doesn't provides the set of module units.) And for the cmake codes, you can look at: https://gitlab.kitware.com/cmake/cmake/-/merge_requests/7978.

Does this already work for "regular" headers?

No. P1689 format only handles module units.

If I understand correctly, this could make builds, especially incremental builds MUCH faster, right?

No. It wouldn't make the builds faster in any sense. (Maybe it would make the build slower a little bit since it requires the build system to construct the build graph by it self). But it will make it easier for the users to describe the dependencies in the build scripts. For example, if I remember correctly, rules_ll requires the user to specify the module dependency of each file explicitly. I believe the syntax will be potentially simpler with the enhanced clang-scan-deps.

BTW, with the patch, we can handle more complicated dependency case:

#ifdef USE_A_MODULE
import a;
#elif USE_B_MODULE
import b;
#elif ...
...

With the patch, it will be much more simpler to handle the case precisely.

clang/test/ClangScanDeps/P1689.cppm
82

Here

Does this already work for "regular" headers?

P1689R0 and P1689R1 had a replacement for -MF bits, but it was dropped in order to simplify and focus what the format is actually for (it even had a place to indicate MSVC's #pragma comment(lib) "autolinking"…behavior, but was dropped in the hopes that autolinking is niche enough to not cater to in such a format. However, the versioning of the format is sufficient for these to come back if it is ever deemed important enough.

Thanks a lot for the clarification.

It wouldn't make the builds faster in any sense. (Maybe it would make the build slower a little bit since it requires the build system to construct the build graph by it self). But it will make it easier for the users to describe the dependencies in the build scripts.

To me this is more than enough to make this feature very interesting 😃. Not having to maintain both module names and filenames for module builds will vastly outweigh the time needed to generate depfiles.

P1689R0 and P1689R1 had a replacement for -MF bits, but it was dropped in order to simplify and focus what the format is actually for

Seems like a good thing to me. I initially thought that there may be a way pass the paths to the headers already found by one clang invocation to another one so that -I was ignored for all headers that were already included by a previous target. On second thought, this was probably too naive and I wonder if the header file search actually makes up a relevant portion of a clang invocation. Probably not 😅

chfast added a subscriber: chfast.Dec 11 2022, 5:54 AM

Thanks a lot for the clarification.

It wouldn't make the builds faster in any sense. (Maybe it would make the build slower a little bit since it requires the build system to construct the build graph by it self). But it will make it easier for the users to describe the dependencies in the build scripts.

To me this is more than enough to make this feature very interesting 😃. Not having to maintain both module names and filenames for module builds will vastly outweigh the time needed to generate depfiles.

P1689R0 and P1689R1 had a replacement for -MF bits, but it was dropped in order to simplify and focus what the format is actually for

Seems like a good thing to me. I initially thought that there may be a way pass the paths to the headers already found by one clang invocation to another one so that -I was ignored for all headers that were already included by a previous target. On second thought, this was probably too naive and I wonder if the header file search actually makes up a relevant portion of a clang invocation. Probably not 😅

https://reviews.llvm.org/D139168 tried to get the necessary headers within same invocation to avoid duplicate scanning. Although it can find the headers, it requires the user/build systems to pass -I in the compilation databases ahead of time. So -I couldn't be ignored.

ben.boeckel added inline comments.Dec 14 2022, 3:02 PM
clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
98

This needs to use std::nullopt on the latest main. Perhaps these should be rebased?

Rebase on main.

ChuanqiXu marked an inline comment as done.Dec 14 2022, 10:37 PM
ChuanqiXu added inline comments.
clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
98

Yeah, done.

jansvoboda11 added inline comments.Dec 22 2022, 10:26 AM
clang/include/clang/Frontend/DependencyOutputOptions.h
84

Why does this live in DependencyOutputOptions? The only place you set and read this is the dependency scanner. Can't you propagate the clang-scan-deps flag to the collector without leaking this into Clang proper?

ChuanqiXu updated this revision to Diff 485017.Dec 22 2022, 7:18 PM
ChuanqiXu marked an inline comment as done.

Address comments to propagate the StdModuleDepFormat option from DependencyOutputOptions to ModuleDepCollector.

ChuanqiXu added inline comments.Dec 22 2022, 7:20 PM
clang/include/clang/Frontend/DependencyOutputOptions.h
84

I felt the semantics will be better sine StdModuleDepFormat is about the dependency output and DependencyOutputOptions controls the dependency output. I followed your suggestion since it makes sense too. And it won't be too late if we need to add it back to DependencyOutputOptions someday.

jansvoboda11 requested changes to this revision.Jan 3 2023, 12:55 PM

To be clear, my suggestion was not to propagate the option into the collector through DependencyOutputOptions.

I think having this more isolated and not leaking into Clang (where it's not needed) is cleaner, so I suggest leaving DependencyOutputOptions unchanged and propagating the option from clang-scan-deps directly into the collector.

clang/include/clang/Frontend/DependencyOutputOptions.h
84

I see, your reasoning makes sense. My understanding is that DependencyOutputOptions primarily control the .d files that may be emitted during compilation. We do take these options into account in the clang-scan-deps --format make mode, but that looks like an implementation shortcut that could be improved/eliminated, rather than mimicked.

This revision now requires changes to proceed.Jan 3 2023, 12:55 PM
ChuanqiXu requested review of this revision.Jan 3 2023, 6:18 PM

To be clear, my suggestion was not to propagate the option into the collector through DependencyOutputOptions.

I think having this more isolated and not leaking into Clang (where it's not needed) is cleaner, so I suggest leaving DependencyOutputOptions unchanged and propagating the option from clang-scan-deps directly into the collector.

Oh, sorry. My previous comments are confusing. In the current revision, we don't touch DependencyOutputOptions and we propagating the option from clang-scan-deps directly into the collector indeed.

To be clear, my suggestion was not to propagate the option into the collector through DependencyOutputOptions.

I think having this more isolated and not leaking into Clang (where it's not needed) is cleaner, so I suggest leaving DependencyOutputOptions unchanged and propagating the option from clang-scan-deps directly into the collector.

Oh, sorry. My previous comments are confusing. In the current revision, we don't touch DependencyOutputOptions and we propagating the option from clang-scan-deps directly into the collector indeed.

I'm sorry, Phabricator showed me a stale revision that still modified DependencyOutputOptions. This is resolved now.

clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
60

Is this necessary to uniquely identify a standard C++ module? I.e. can we have two modules with the same name and context hash that have different source paths?

63

Same question here.

72

Same question here.

ben.boeckel added inline comments.Jan 9 2023, 10:23 AM
clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
60

It's possible:

cmake
add_library(foo)
target_sources(foo PUBLIC FILE_SET CXX_MODULES FILES
  $<$<CONFIG:Debug>:module_m_debug.cpp>
  $<$<NOT:$<CONFIG:Debug>>:module_m.cpp>)

where both provide export module M;, just with different bits due to debug (or any other reason, really).

jansvoboda11 added inline comments.Jan 9 2023, 10:24 AM
clang/tools/clang-scan-deps/ClangScanDeps.cpp
671

How is this API expected to work if you provide a module name?

jansvoboda11 added inline comments.Jan 9 2023, 10:29 AM
clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
60

That would be reflected in the context hash. (Or rather hash of the command line the scanner generates.)

ChuanqiXu updated this revision to Diff 487691.Jan 9 2023, 11:00 PM

Address comments:

  • Add struct P1689ModuleInfo to avoid pollute ModuleID.
  • Remove meaningless parameter ModuleName from getP1689ModuleDependencyFile
ChuanqiXu marked 6 inline comments as done.Jan 9 2023, 11:03 PM
ChuanqiXu added inline comments.
clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
60

You're right. It looks not good to add these things to ModuleID. Previously I thought it may be better to reuse the ModuleName in ModuleID. But it may be better to not touch ModuleID in P1689 now.

clang/tools/clang-scan-deps/ClangScanDeps.cpp
671

Oh, my bad. We should remove it. Thanks for noting this!

ChuanqiXu marked 2 inline comments as done.Jan 15 2023, 6:46 PM

Forgot to submit the inline comments. My bad.

jansvoboda11 accepted this revision.Jan 20 2023, 11:19 AM
jansvoboda11 added 1 blocking reviewer(s): Bigcheese.

LGTM provided my comments get resolved, but I want @Bigcheese to look over this patch too.

clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
71

AFAIK we're moving off of llvm::Optional. It's probably better to use std::optional in new code. Applies to the rest of the patch.

95

Does this need to be fully qualified by clang::tooling::dependencies::? We're already in that namespace. Applies to the rest of the patch.

ChuanqiXu updated this revision to Diff 493475.Jan 30 2023, 7:35 PM

Address comments.

ChuanqiXu marked 2 inline comments as done.Jan 30 2023, 7:41 PM

LGTM provided my comments get resolved, but I want @Bigcheese to look over this patch too.

Thanks for reviewing this! I want to land this in 2.10 if no big problem shows up. Since 16.x is branched and 16.0 is going to be released in the early March. I want to have more time to backport, write documents and more tests.

ChuanqiXu accepted this revision.Feb 9 2023, 6:00 PM

I am going to land this since many c++ users want to use cmake with modules for clang16. And it is not good for clang to be slow in this direction.

This revision was not accepted when it landed; it landed in state Needs Review.Feb 9 2023, 6:31 PM
This revision was landed with ongoing or failed builds.
This revision was automatically updated to reflect the committed changes.
Herald added a project: Restricted Project. · View Herald TranscriptFeb 9 2023, 6:31 PM
Herald added a subscriber: cfe-commits. · View Herald Transcript
uabelho added a subscriber: uabelho.EditedFeb 10 2023, 2:15 AM

Hi,

There seems to be something funny going on with the

ClangScanDeps/P1689.cppm

testcase added in this patch.

It fails randomly for me on top of tree (4ad8f7a189570).
Roughly 1 in 10 runs fail for me.
E.g. with a crash like

Exit Code: 2

Command Output (stderr):
--
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
#0 0x00000000004ef8b8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ef8b8)
#1 0x00000000004ed5ae llvm::sys::RunSignalHandlers() (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ed5ae)
#2 0x00000000004f0096 SignalHandler(int) Signals.cpp:0:0
#3 0x00007f6518263630 __restore_rt sigaction.c:0:0
#4 0x0000000000474f40 P1689Deps::addSourcePathsToRequires() crtstuff.c:0:0
#5 0x00000000004666d1 P1689Deps::printDependencies(llvm::raw_ostream&) crtstuff.c:0:0
#6 0x000000000046204a main (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x46204a)
#7 0x00007f6515996555 __libc_start_main (/lib64/libc.so.6+0x22555)
#8 0x000000000045fcad _start (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x45fcad)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /repo/uabelho/main-github/llvm/build-all/bin/FileCheck /repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp/Checks.cpp -DPREFIX=/repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp

--

********************
********************
Failed Tests (1):
  Clang :: ClangScanDeps/P1689.cppm

or

Exit Code: 2

Command Output (stderr):
--
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
 #0 0x00000000004ef8b8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ef8b8)
 #1 0x00000000004ed5ae llvm::sys::RunSignalHandlers() (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ed5ae)
 #2 0x00000000004f0096 SignalHandler(int) Signals.cpp:0:0
 #3 0x00007f422b33e630 __restore_rt sigaction.c:0:0
 #4 0x00007f4228a85387 raise (/lib64/libc.so.6+0x36387)
 #5 0x00007f4228a86a78 abort (/lib64/libc.so.6+0x37a78)
 #6 0x00007f4228ac7f67 __libc_message (/lib64/libc.so.6+0x78f67)
 #7 0x00007f4228ad0329 _int_free malloc.c:0:0
 #8 0x000000000047e7ef void std::vector<clang::tooling::dependencies::P1689Rule, std::allocator<clang::tooling::dependencies::P1689Rule>>::_M_realloc_insert<clang::tooling::dependencies::P1689Rule const&>(__gnu_cxx::__normal_iterator<clang::tooling::dependencies::P1689Rule*, std::vector<clang::tooling::dependencies::P1689Rule, std::allocator<clang::tooling::dependencies::P1689Rule>>>, clang::tooling::dependencies::P1689Rule const&) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x47e7ef)
 #9 0x000000000047cea1 std::_Function_handler<void (), main::$_1>::_M_invoke(std::_Any_data const&) ClangScanDeps.cpp:0:0
#10 0x000000000047ba3c std::_Function_handler<void (), llvm::ThreadPool::createTaskAndFuture(std::function<void ()>)::'lambda'()>::_M_invoke(std::_Any_data const&) crtstuff.c:0:0
#11 0x00000000004b73e9 llvm::ThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4b73e9)
#12 0x00000000004b8358 void* llvm::thread::ThreadProxy<std::tuple<llvm::ThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0
#13 0x00007f422b336ea5 start_thread pthread_create.c:0:0
#14 0x00007f4228b4db0d clone (/lib64/libc.so.6+0xfeb0d)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /repo/uabelho/main-github/llvm/build-all/bin/FileCheck /repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp/Checks.cpp -DPREFIX=/repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp

--

********************
********************
Failed Tests (1):
  Clang :: ClangScanDeps/P1689.cppm

but I've also seen things like

*** Error in `/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps': corrupted size vs. prev_size: 0x00007faf74000fc0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f474)[0x7fafa1ffa474]
/lib64/libc.so.6(+0x8156b)[0x7fafa1ffc56b]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x47e7ef]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x47cea1]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x47ba3c]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x4b73e9]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x4b8358]
/lib64/libpthread.so.0(+0x7ea5)[0x7fafa4862ea5]
/lib64/libc.so.6(clone+0x6d)[0x7fafa2079b0d]
======= Memory map: ========
00400000-08ffa000 r-xp 00000000 fd:01 63551                              /repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps
08ffa000-09b44000 rw-p 08bf9000 fd:01 63551                              /repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps
09b44000-09bbf000 rw-p 00000000 00:00 0 
0a86a000-0a8f3000 rw-p 00000000 00:00 0                                  [heap]
7faf64000000-7faf64021000 rw-p 00000000 00:00 0 
7faf64021000-7faf68000000 ---p 00000000 00:00 0 
[...]
thakis added a subscriber: thakis.Feb 10 2023, 6:33 AM

Either this or D139168 is breaking tests on Windows: http://45.33.8.238/win/74780/step_7.txt

Please take a look and revert for now if it takes a while to fix.

TWeaver added a subscriber: TWeaver.

@thakis @ChuanqiXu I have a revision up for review here https://reviews.llvm.org/D143749 that adds back the UNSUPPORTED: system-windows that was reverted in https://reviews.llvm.org/D139168

Hi,

There seems to be something funny going on with the

ClangScanDeps/P1689.cppm

testcase added in this patch.

It fails randomly for me on top of tree (4ad8f7a189570).
Roughly 1 in 10 runs fail for me.
E.g. with a crash like

Exit Code: 2

Command Output (stderr):
--
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
#0 0x00000000004ef8b8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ef8b8)
#1 0x00000000004ed5ae llvm::sys::RunSignalHandlers() (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ed5ae)
#2 0x00000000004f0096 SignalHandler(int) Signals.cpp:0:0
#3 0x00007f6518263630 __restore_rt sigaction.c:0:0
#4 0x0000000000474f40 P1689Deps::addSourcePathsToRequires() crtstuff.c:0:0
#5 0x00000000004666d1 P1689Deps::printDependencies(llvm::raw_ostream&) crtstuff.c:0:0
#6 0x000000000046204a main (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x46204a)
#7 0x00007f6515996555 __libc_start_main (/lib64/libc.so.6+0x22555)
#8 0x000000000045fcad _start (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x45fcad)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /repo/uabelho/main-github/llvm/build-all/bin/FileCheck /repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp/Checks.cpp -DPREFIX=/repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp

--

********************
********************
Failed Tests (1):
  Clang :: ClangScanDeps/P1689.cppm

or

Exit Code: 2

Command Output (stderr):
--
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
 #0 0x00000000004ef8b8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ef8b8)
 #1 0x00000000004ed5ae llvm::sys::RunSignalHandlers() (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4ed5ae)
 #2 0x00000000004f0096 SignalHandler(int) Signals.cpp:0:0
 #3 0x00007f422b33e630 __restore_rt sigaction.c:0:0
 #4 0x00007f4228a85387 raise (/lib64/libc.so.6+0x36387)
 #5 0x00007f4228a86a78 abort (/lib64/libc.so.6+0x37a78)
 #6 0x00007f4228ac7f67 __libc_message (/lib64/libc.so.6+0x78f67)
 #7 0x00007f4228ad0329 _int_free malloc.c:0:0
 #8 0x000000000047e7ef void std::vector<clang::tooling::dependencies::P1689Rule, std::allocator<clang::tooling::dependencies::P1689Rule>>::_M_realloc_insert<clang::tooling::dependencies::P1689Rule const&>(__gnu_cxx::__normal_iterator<clang::tooling::dependencies::P1689Rule*, std::vector<clang::tooling::dependencies::P1689Rule, std::allocator<clang::tooling::dependencies::P1689Rule>>>, clang::tooling::dependencies::P1689Rule const&) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x47e7ef)
 #9 0x000000000047cea1 std::_Function_handler<void (), main::$_1>::_M_invoke(std::_Any_data const&) ClangScanDeps.cpp:0:0
#10 0x000000000047ba3c std::_Function_handler<void (), llvm::ThreadPool::createTaskAndFuture(std::function<void ()>)::'lambda'()>::_M_invoke(std::_Any_data const&) crtstuff.c:0:0
#11 0x00000000004b73e9 llvm::ThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps+0x4b73e9)
#12 0x00000000004b8358 void* llvm::thread::ThreadProxy<std::tuple<llvm::ThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0
#13 0x00007f422b336ea5 start_thread pthread_create.c:0:0
#14 0x00007f4228b4db0d clone (/lib64/libc.so.6+0xfeb0d)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /repo/uabelho/main-github/llvm/build-all/bin/FileCheck /repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp/Checks.cpp -DPREFIX=/repo/uabelho/main-github/llvm/build-all/tools/clang/test/ClangScanDeps/Output/P1689.cppm.tmp

--

********************
********************
Failed Tests (1):
  Clang :: ClangScanDeps/P1689.cppm

but I've also seen things like

*** Error in `/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps': corrupted size vs. prev_size: 0x00007faf74000fc0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f474)[0x7fafa1ffa474]
/lib64/libc.so.6(+0x8156b)[0x7fafa1ffc56b]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x47e7ef]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x47cea1]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x47ba3c]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x4b73e9]
/repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps[0x4b8358]
/lib64/libpthread.so.0(+0x7ea5)[0x7fafa4862ea5]
/lib64/libc.so.6(clone+0x6d)[0x7fafa2079b0d]
======= Memory map: ========
00400000-08ffa000 r-xp 00000000 fd:01 63551                              /repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps
08ffa000-09b44000 rw-p 08bf9000 fd:01 63551                              /repo/uabelho/main-github/llvm/build-all/bin/clang-scan-deps
09b44000-09bbf000 rw-p 00000000 00:00 0 
0a86a000-0a8f3000 rw-p 00000000 00:00 0                                  [heap]
7faf64000000-7faf64021000 rw-p 00000000 00:00 0 
7faf64021000-7faf68000000 ---p 00000000 00:00 0 
[...]

This should be caused by https://reviews.llvm.org/D139168 and I've reverted it. And the windows problem should be addressed by https://reviews.llvm.org/D143749. Thank you!

dyung added a subscriber: dyung.Feb 11 2023, 11:42 AM

I'm still seeing random crashes today on a linux buildbot in the test you added P1689.cppm. Can you either fix or revert?

corrupted size vs. prev_size while consolidating
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
 #0 0x000055baf810283f llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/bin/clang-scan-deps+0x88883f)
 #1 0x000055baf8100214 SignalHandler(int) Signals.cpp:0:0
 #2 0x00007fa226358420 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x14420)
 #3 0x00007fa225e2500b raise /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/raise.c:51:1
 #4 0x00007fa225e04859 abort /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:81:7
 #5 0x00007fa225e6f26e __libc_message /build/glibc-SzIz7B/glibc-2.31/libio/../sysdeps/posix/libc_fatal.c:155:5
 #6 0x00007fa225e772fc /build/glibc-SzIz7B/glibc-2.31/malloc/malloc.c:5348:3
 #7 0x00007fa225e7904e _int_free /build/glibc-SzIz7B/glibc-2.31/malloc/malloc.c:4245:6
 #8 0x00007fa225e7c88d tcache_thread_shutdown /build/glibc-SzIz7B/glibc-2.31/malloc/malloc.c:2960:33
 #9 0x00007fa225e7c88d __malloc_arena_thread_freeres /build/glibc-SzIz7B/glibc-2.31/malloc/arena.c:951:3
#10 0x00007fa22634c62f start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:496:7
#11 0x00007fa225f01133 __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:97:0

Excuse me, I have reverted 79a3803bb2cc, D137534 and this.
See also https://lab.llvm.org/buildbot/#/builders/239/builds/922

I guess P1689Deps would not be thread-safe.

ChuanqiXu updated this revision to Diff 496821.Feb 12 2023, 6:40 PM

While I failed to reproduce the failure in my local environment, I found a thread-safe defect in my code. I forgot to add a Lock for P1689::addRules. I'd like to recommit this one and see if the bot would still complain. Thank you for reporting and reverting this.

ChuanqiXu updated this revision to Diff 496823.Feb 12 2023, 6:42 PM

Add unsupported: windows.