This is an archive of the discontinued LLVM Phabricator instance.

[clang-scan-deps] Introduce the DependencyScanning library with the thread worker code and better error handling
ClosedPublic

Authored by arphaman on Jun 21 2019, 5:02 PM.

Details

Summary

This patch extracts out the code that will powers the fast scanning worker into a new file in a new DependencyScanning library. The error and output handling is improved so that the clients can gather errors/results from the worker directly.

Diff Detail

Repository
rC Clang

Event Timeline

arphaman created this revision.Jun 21 2019, 5:02 PM
Herald added a project: Restricted Project. · View Herald TranscriptJun 21 2019, 5:02 PM

LGTM!

Some quick stats on our end (running Windows 10 on a Intel W-2135, 6-core, 3.7 GHz, NVMe SSD): on a large .SLN compiling approx. 16,000 .CPP files through 600 unity .CPPs and 23,000 .H files, out of 86 secs spent in ClangScanDeps, about 32 secs are spent in DirectoryLookup::LookUpFile.

clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
29

How do you actually select between the fast mode or the regular preprocess?

aganea accepted this revision.Jun 25 2019, 5:49 PM
This revision is now accepted and ready to land.Jun 25 2019, 5:49 PM
aganea added a subscriber: rnk.EditedJun 26 2019, 11:57 AM

A bit more detail on what we're seeing on our end (specs in the post above). The 'Count' column represents the number of 1ms samples taken in that function. The 'Weight' column is cumulated times for all cores, for a given process, in ms.

Image below shows most of the time is spent in clang-scan-deps.dll and indirectly, in ntoskrnl.exe. The graph at the bottom also shows that preprocessing the ~600 unity .CPP files in the project issues 16 million IOPS. The graphs on the right show no bottlenecks on the disk I/O on the OS I/O-side.

Top functions in ntoskrnl.exe, most callstacks end up in clang-scan-deps.dll:

A good chunk of all this is caused by llvm::sys::status(), even though is goes through the FileSystemStatCache, like I was suggesting previously (ping @rnk)
(beware, the callstacks are reversed in the image below)

I can take a look at llvm::sys::status() after the vacations (somewhere in August)

All this was implemented & investigated by @SamChaps

arphaman marked an inline comment as done.Jun 26 2019, 2:06 PM
arphaman added inline comments.
clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
29

You can't select it yet, as it's not yet implemented :)
A couple more follow-up patches, and it will be usable.

@aganea The FS status and most of file reads will be cached with the shared FS. Hopefully I can put up a patch for it this week.

This revision was automatically updated to reflect the committed changes.
Herald added a project: Restricted Project. · View Herald TranscriptJun 26 2019, 2:12 PM