[clang-tidy] Store checks profiling info as JSON files

[lebedev.ri]

Summary

Continuation of D46504.

Example output:

$ clang-tidy -enable-check-profile -store-check-profile=. -checks=-*,readability-function-size source.cpp
$ # Note that there won't be timings table printed to the console.
$ cat *.json
{
"file": "/path/to/source.cpp",
"timestamp": "2018-05-16 16:13:18.717446360",
"profile": {
  "time.clang-tidy.readability-function-size.wall": 1.0421266555786133e+00,
  "time.clang-tidy.readability-function-size.user": 9.2088400000005421e-01,
  "time.clang-tidy.readability-function-size.sys": 1.2418899999999974e-01
}
}

There are two arguments that control profile storage:

• -store-check-profile=<prefix>
  
  By default reports are printed in tabulated format to stderr. When this option is passed, these per-TU profiles are instead stored as JSON. If the prefix is not an absolute path, it is considered to be relative to the directory from where you have run :program:clang-tidy. All . and .. patterns in the path are collapsed, and symlinks are resolved.
  
  Example: Let's suppose you have a source file named example.cpp, located in /source directory.
  • If you specify -store-check-profile=/tmp, then the profile will be saved to /tmp/<timestamp>-example.cpp.json
  • If you run :program:clang-tidy from within /foo directory, and specify -store-check-profile=., then the profile will still be saved to /foo/<timestamp>-example.cpp.json

[Eugene.Zelenko]

I think will be good idea to store data in JSON format too.

docs/ReleaseNotes.rst

60

May be Profile information could be stored in SSV format. sounds better?

docs/clang-tidy/index.rst

756

Please use ` for command line options and other non-C/C++ constructs.

[lebedev.ri]

In D46602#1092084, @Eugene.Zelenko wrote:

I think will be good idea to store data in JSON format too.

Yeah, i have thought about it, and i'm not sure.
The output is so dumb so there isn't even much point in using anything more advanced than CSV.

[rja]

+1 for JSON

[lebedev.ri]

docs/clang-tidy/index.rst

756

`` is used for command line options elsewhere in this document.
Is there a reference somewhere?

[lebedev.ri]

• Deduplicate code by switching to already-existing YAML printer infrastructure from TimerGroup
• Switch to YAML. It's kinda ugly, but maybe better than having to manually construct CSV.. :/
• When the output prefix does not yet exist, still store the profile as YAML, don't fail at make_real
• When storing profile to file, don't print it to screen. I have tried it, and it is just too noisy for no apparent benefit.

[lebedev.ri]

• Produce valid-er YAML.

[lebedev.ri]

docs/clang-tidy/index.rst

779

Hmm, thinking about it a bit more, i think it should also contain the "file",

{
  "file": "/source/example.cpp",
  "profile": {
    "time.clang-tidy.readability-function-size.wall": 9.5367431640625000e-05,
    "time.clang-tidy.readability-function-size.user": 7.2000000000002617e-05,
    "time.clang-tidy.readability-function-size.sys": 1.8999999999999920e-05
  }
}

[alexfh]

In D46602#1092111, @rja wrote:

+1 for JSON

Could you explain why would you use YAML or JSON for this? How are you going to use/process this data?

[alexfh]

Roman, it looks to me that a simpler storage scheme would be sufficient. For example, YYYYMMDDhhmmss-InputFileName.cpp.csv. Main things are: 1. include a timestamp, so there's no need to overwrite old results, 2. include just the name of the file without any parent directories, 3. put all outputs into the same directory. This way we wouldn't have to create a directory structure and think about stripping a certain prefix (btw, utilities like patch just specify the number of path components to remove from the start, not the actual substring). WDYT?

[lebedev.ri]

In D46602#1092883, @alexfh wrote:

In D46602#1092111, @rja wrote:

+1 for JSON

Could you explain why would you use YAML or JSON for this? How are you going to use/process this data?

I personally don't have a preference here.
The "YAML" may be better because that is what already supported by TimerGroup, which allowed to drop the table printer, too.

The next step will be a python script:

1. Takes either a file names (these generated .yaml), or a prefix, If it is a prefix, it GLOBS all the .yaml's in there.
2. Load all the files from step 1.
3. Print global report from data collected (reduce(+) after grouping by check name) from all the files (what D45931 did)
4. Maybe print report about outliers - without "reduce(+) after grouping by check name", with percentages to the total time spent on TU. This is where the filename (and YAML) would be helpful.
5. ???
6. Run on codebases, find performance problems, contribute fixes
7. Profit!

[lebedev.ri]

In D46602#1092890, @alexfh wrote:

Roman, it looks to me that a simpler storage scheme would be sufficient. For example, YYYYMMDDhhmmss-InputFileName.cpp.csv.
Main things are:

1. include a timestamp, so there's no need to overwrite old results,

Of the input source file?

I don't like this, because when working on trying to improve the performance of the check,
one wouldn't touch the source file, only the clang-tidy sources. So either you would
have to touch sources (and if they are not writable?), or remove .csv beforehand,
or not output to file, but redirect output.

Neither of these possibilities sound great to me.

2. include just the name of the file without any parent directories,

That won't work, there are duplicate filenames even in LLVM.

$ find -iname Path.cpp
./lib/Support/Path.cpp
./unittests/Support/Path.cpp

3. put all outputs into the same directory. This way we wouldn't have to create a directory structure and think about stripping a certain prefix (btw, utilities like patch just specify the number of path components to remove from the start, not the actual substring). WDYT?

I'm not particularly looking forward to having being forced to have n thousands of reports in a single directory :)

[lebedev.ri]

In D46602#1092902, @lebedev.ri wrote:

In D46602#1092890, @alexfh wrote:

Roman, it looks to me that a simpler storage scheme would be sufficient. For example, YYYYMMDDhhmmss-InputFileName.cpp.csv.
Main things are:

1. include a timestamp, so there's no need to overwrite old results,

Of the input source file?

I don't like this, because when working on trying to improve the performance of the check,
one wouldn't touch the source file, only the clang-tidy sources. So either you would
have to touch sources (and if they are not writable?), or remove .csv beforehand,
or not output to file, but redirect output.

... also, a new report with a new name will be created each time the filetime changes, so not
only will it be fun from tooling point of view, but it will also leave old reports in-place..

Neither of these possibilities sound great to me.

2. include just the name of the file without any parent directories,

That won't work, there are duplicate filenames even in LLVM.

$ find -iname Path.cpp
./lib/Support/Path.cpp
./unittests/Support/Path.cpp

3. put all outputs into the same directory. This way we wouldn't have to create a directory structure and think about stripping a certain prefix (btw, utilities like patch just specify the number of path components to remove from the start, not the actual substring). WDYT?

I'm not particularly looking forward to having being forced to have n thousands of reports in a single directory :)

[lebedev.ri]

• Make json less flat, store source filename in it.

[alexfh]

In D46602#1092902, @lebedev.ri wrote:

In D46602#1092890, @alexfh wrote:

Roman, it looks to me that a simpler storage scheme would be sufficient. For example, YYYYMMDDhhmmss-InputFileName.cpp.csv.
Main things are:

1. include a timestamp, so there's no need to overwrite old results,

Of the input source file?

No, current timestamp, when each profile gets dumped.

2. include just the name of the file without any parent directories,

That won't work, there are duplicate filenames even in LLVM.

That's why I suggested to include a timestamp. Each result file will get a unique timestamp as a part of its name (unless the resolution of the timestamp is too coarse - compared to clang-tidy run time - to allow collisions).

$ find -iname Path.cpp
./lib/Support/Path.cpp
./unittests/Support/Path.cpp

> 3. put all outputs into the same directory. This way we wouldn't have to create a directory structure and think about stripping a certain prefix (btw, utilities like patch just specify the number of path components to remove from the start, not the actual substring). WDYT?

I'm not particularly looking forward to having being forced to have n thousands of reports in a single directory :)

What kind of problems do you expect to have with this? Are you going to look at the result files manually or use a script to aggregate them?

[lebedev.ri]

In D46602#1095960, @alexfh wrote:

In D46602#1092902, @lebedev.ri wrote:

In D46602#1092890, @alexfh wrote:

Roman, it looks to me that a simpler storage scheme would be sufficient. For example, YYYYMMDDhhmmss-InputFileName.cpp.csv.
Main things are:

1. include a timestamp, so there's no need to overwrite old results,

Of the input source file?

No, current timestamp, when each profile gets dumped.

Ah.
Then i don't understand this at all. Why would we want to do that, exactly?
Just so that we can avoid creating directory structure? Why do we want to avoid that?

2. include just the name of the file without any parent directories,

That won't work, there are duplicate filenames even in LLVM.

That's why I suggested to include a timestamp. Each result file will get a unique timestamp as a part of its name (unless the resolution of the timestamp is too coarse - compared to clang-tidy run time - to allow collisions).

$ find -iname Path.cpp
./lib/Support/Path.cpp
./unittests/Support/Path.cpp

> 3. put all outputs into the same directory. This way we wouldn't have to create a directory structure and think about stripping a certain prefix (btw, utilities like patch just specify the number of path components to remove from the start, not the actual substring). WDYT?

I'm not particularly looking forward to having being forced to have n thousands of reports in a single directory :)

What kind of problems do you expect to have with this? Are you going to look at the result files manually or use a script to aggregate them?

[alexfh]

In D46602#1095980, @lebedev.ri wrote:

In D46602#1095960, @alexfh wrote:

In D46602#1092902, @lebedev.ri wrote:

In D46602#1092890, @alexfh wrote:

Roman, it looks to me that a simpler storage scheme would be sufficient. For example, YYYYMMDDhhmmss-InputFileName.cpp.csv.
Main things are:

1. include a timestamp, so there's no need to overwrite old results,

Of the input source file?

No, current timestamp, when each profile gets dumped.

Ah.
Then i don't understand this at all. Why would we want to do that, exactly?
Just so that we can avoid creating directory structure? Why do we want to avoid that?

If clang-tidy is invoked manually, a simpler naming scheme with less configuration options would be easier to use, in particular:

1. no the need to specify the -store-check-profile-elide-prefix= option;
2. it's easier to see all results (no need to use find, just ls /output/directory or ls /output/directory/20180514* to see today's results, for example);
3. no chance of filename collision, and thus no chance of losing older results by just running clang-tidy again.

If clang-tidy is started by a script (e.g. run-clang-tidy.py or a modification of it specialized on profiling), then a specific naming scheme doesn't matter much, since the script can create a separate directory for each run or for each shard. I don't yet see how repeating the directory structure of the input files would make profiling results easier or more convenient to use. A relation between the input file and the result file can still be maintained in some other form (for example, by specifying the whole invocation inside the results file - if we want to use YAML).

[lebedev.ri]

In D46602#1097902, @alexfh wrote:

In D46602#1095980, @lebedev.ri wrote:

In D46602#1095960, @alexfh wrote:

In D46602#1092902, @lebedev.ri wrote:

In D46602#1092890, @alexfh wrote:

Roman, it looks to me that a simpler storage scheme would be sufficient. For example, YYYYMMDDhhmmss-InputFileName.cpp.csv.
Main things are:

1. include a timestamp, so there's no need to overwrite old results,

Of the input source file?

No, current timestamp, when each profile gets dumped.

Ah.
Then i don't understand this at all. Why would we want to do that, exactly?
Just so that we can avoid creating directory structure? Why do we want to avoid that?

If clang-tidy is invoked manually, a simpler naming scheme with less configuration options would be easier to use, in particular:

1. no the need to specify the -store-check-profile-elide-prefix= option;
2. it's easier to see all results (no need to use find, just ls /output/directory or ls /output/directory/20180514* to see today's results, for example);
3. no chance of filename collision, and thus no chance of losing older results by just running clang-tidy again.

How do i reflect that in tests? The output name will basically be random.

If clang-tidy is started by a script (e.g. run-clang-tidy.py or a modification of it specialized on profiling), then a specific naming scheme doesn't matter much, since the script can create a separate directory for each run or for each shard. I don't yet see how repeating the directory structure of the input files would make profiling results easier or more convenient to use. A relation between the input file and the result file can still be maintained in some other form (for example, by specifying the whole invocation inside the results file - if we want to use YAML).

[alexfh]

In D46602#1097954, @lebedev.ri wrote:

In D46602#1097902, @alexfh wrote:

In D46602#1095980, @lebedev.ri wrote:

In D46602#1095960, @alexfh wrote:

In D46602#1092902, @lebedev.ri wrote:

In D46602#1092890, @alexfh wrote:

Roman, it looks to me that a simpler storage scheme would be sufficient. For example, YYYYMMDDhhmmss-InputFileName.cpp.csv.
Main things are:

1. include a timestamp, so there's no need to overwrite old results,

Of the input source file?

No, current timestamp, when each profile gets dumped.

Ah.
Then i don't understand this at all. Why would we want to do that, exactly?
Just so that we can avoid creating directory structure? Why do we want to avoid that?

If clang-tidy is invoked manually, a simpler naming scheme with less configuration options would be easier to use, in particular:

1. no the need to specify the -store-check-profile-elide-prefix= option;
2. it's easier to see all results (no need to use find, just ls /output/directory or ls /output/directory/20180514* to see today's results, for example);
3. no chance of filename collision, and thus no chance of losing older results by just running clang-tidy again.

How do i reflect that in tests? The output name will basically be random.

Why random? It will follow a certain rule that can be verified. If we decide to name the file YYYYMMDD_InputFileName.cpp.yaml, for example, then the test can verify the file matching ????????_InputFileName.cpp.yaml exists and contains whatever it should contain. If we want the test to be stricter, it could verify that the ???????? part matches a certain pattern (say, 20\d\d[0-1][0-9][0-3][0-9]). Additionally, it could verify that after a short sleep a file with a different ???????? part is generated.

[lebedev.ri]

• Get rid of 'prefix elision'
• Don't use full source file name, only the filename
• Prefix that filename with ISO8601-like timestamp of the profile creation.

I think this is worse than before, horrible and ugly, but hey, i'm not the code owner.

[lebedev.ri]

In D46602#1098092, @alexfh wrote:

In D46602#1097954, @lebedev.ri wrote:

How do i reflect that in tests? The output name will basically be random.

Why random?

I was obviously talking about the FileCheck stuff, how to specify such a dynamic filename there..

[lebedev.ri]

• Rebased
• Slightly refactor how the profile storage params are computed, move that into ClangTidyProfiling::StorageParams helper struct.
• Store timestamp in the json too, since it's already in the filename. May be useful for garbage collection and alike.

[alexfh]

A couple of comments from a cursory look. I'll try to look closer later this week.

clang-tidy/ClangTidyDiagnosticConsumer.h

174

The name is misleading. I'd suggest using a name that unambiguously states that we're setting a prefix for profiling output files. Some ideas: setProfilePathPrefix, setStoredProfilePathPrefix, setProfileStorePathPrefix, setProfileResultsPathPrefix.

175

getProfileStorageParams?

[lebedev.ri]

Rename getter/setter functions.

Thank you for taking a look!

[• Quuxplusone]

clang-tidy/tool/ClangTidyMain.cpp

188

Drive-by nit: Each other help string ends with a newline ()"), on a line by itself). This one presumably should too.

[lebedev.ri]

clang-tidy/tool/ClangTidyMain.cpp

188

Hmm, yeah, at least for consistency.

[lebedev.ri]

Ping.

[george.karpenkov]

LGTM with a nit to me, but maybe clang-tidy code owners would need to sign that off as well.

clang-tidy/ClangTidy.h

238

doxygen comment for new param?

[lebedev.ri]

In D46602#1116973, @george.karpenkov wrote:

LGTM with a nit to me

Thank you!

I suspect that at least temporarily we will end up with two different tooling sets
to further post-process these results, since i really love to write bicycles and
then hopefully throw them away :)

In D46602#1116973, @george.karpenkov wrote:

but maybe clang-tidy code owners would need to sign that off as well.

Yeah, that is time-consuming as usual.

[aaron.ballman]

clang-tidy/ClangTidyProfiling.cpp

72

Missing a whitespace before the quote for the file name.

clang-tidy/tool/ClangTidyMain.cpp

343

The identifiers here don't match the coding convention (should be ProcessPrefix and Input by convention).

351–356

This creates a TOCTOU bug; can you call real_path() without checking for existence and instead check the error code to decide whether to spit out an error or return AbsolutePath?

[lebedev.ri]

Rebased.
Addressed review notes.

clang-tidy/tool/ClangTidyMain.cpp

351–356

Hmm, i think this can just go away.

[aaron.ballman]

clang-tidy/ClangTidy.h

230–232

Minor wording nits to fix grammar, how about: "...the profile will not be output to stderr, but will instead be stored as a JSON file in the specified directory."

clang-tidy/ClangTidyProfiling.cpp

47

I'm not keen that we call this printAsJSON() when the docs talk about writing out YAML. While all JSON is valid YAML, that feels like trivia we shouldn't encode in the function name. What do you think to renaming to printAsYAML() instead?

77

I'm not keen that we call printAsJSON when the docs talk about writing out YAML. While all JSON is valid YAML, that feels like trivia we shouldn't encode in the function name. What do you think to renaming to printAsYAML() instead?

docs/clang-tidy/index.rst

257

Spurious change?

756

use -> use the

757

will be output to stderr

801

in -> in the

[lebedev.ri]

Rebase ontop of rL333994, address review notes, canonicalize to JSON.

[lebedev.ri]

clang-tidy/ClangTidyProfiling.cpp

47

But the actual printer is called TG->printJSONValues()..
Let's go the other way around, and canonicalize to JSON.

[aaron.ballman]

LGTM!

clang-tidy/ClangTidyProfiling.cpp

47

Even better!

[lebedev.ri]

In D46602#1122194, @aaron.ballman wrote:

LGTM!

Thank you for the review.

Waiting on @alexfh ...

[alexfh]

LG

[lebedev.ri]

In D46602#1123631, @alexfh wrote:

LG

Thank you for the review!

clang-tidy/ClangTidy.h

 ClangTidyOptions::OptionMap
 getCheckOptions(const ClangTidyOptions &Options,
                 bool AllowEnablingAnalyzerAlphaCheckers);
 
 /// \brief Run a set of clang-tidy checks on a set of files.
 ///
 /// \param EnableCheckProfile If provided, it enables check profile collection
 /// in MatchFinder, and will contain the result of the profile.
+/// \param StoreCheckProfile If provided, and EnableCheckProfile is true,
+/// the profile will not be output to stderr, but will instead be stored
+/// as a JSON file in the specified directory.
 void runClangTidy(clang::tidy::ClangTidyContext &Context,
                   const tooling::CompilationDatabase &Compilations,
                   ArrayRef<std::string> InputFiles,
                   llvm::IntrusiveRefCntPtr<vfs::FileSystem> BaseFS,
-                  bool EnableCheckProfile = false);
+                  bool EnableCheckProfile = false,
+                  llvm::StringRef StoreCheckProfile = StringRef());
 
 // FIXME: This interface will need to be significantly extended to be useful.
 // FIXME: Implement confidence levels for displaying/fixing errors.
 //
 /// \brief Displays the found \p Errors to the users. If \p Fix is true, \p
 /// Errors containing fixes are automatically applied and reformatted. If no
 /// clang-format configuration file is found, the given \P FormatStyle is used.
 void handleErrors(ClangTidyContext &Context, bool Fix,

clang-tidy/ClangTidy.cpp

ClangTidyASTConsumerFactory::CreateASTConsumer(
 
   std::vector<std::unique_ptr<ClangTidyCheck>> Checks;
   CheckFactories->createChecks(&Context, Checks);
 
   ast_matchers::MatchFinder::MatchFinderOptions FinderOptions;
 
   std::unique_ptr<ClangTidyProfiling> Profiling;
   if (Context.getEnableProfiling()) {
-    Profiling = llvm::make_unique<ClangTidyProfiling>();
+    Profiling = llvm::make_unique<ClangTidyProfiling>(
+        Context.getProfileStorageParams());
     FinderOptions.CheckProfiling.emplace(Profiling->Records);
   }
 
   std::unique_ptr<ast_matchers::MatchFinder> Finder(
       new ast_matchers::MatchFinder(std::move(FinderOptions)));
 
   for (auto &Check : Checks) {
     Check->registerMatchers(&*Finder);
getCheckOptions(const ClangTidyOptions &Options,
   ClangTidyASTConsumerFactory Factory(Context);
   return Factory.getCheckOptions();
 }
 
 void runClangTidy(clang::tidy::ClangTidyContext &Context,
                   const CompilationDatabase &Compilations,
                   ArrayRef<std::string> InputFiles,
                   llvm::IntrusiveRefCntPtr<vfs::FileSystem> BaseFS,
-                  bool EnableCheckProfile) {
+                  bool EnableCheckProfile, llvm::StringRef StoreCheckProfile) {
   ClangTool Tool(Compilations, InputFiles,
                  std::make_shared<PCHContainerOperations>(), BaseFS);
 
   // Add extra arguments passed by the clang-tidy command-line.
   ArgumentsAdjuster PerFileExtraArgumentsInserter =
       [&Context](const CommandLineArguments &Args, StringRef Filename) {
         ClangTidyOptions Opts = Context.getOptionsForFile(Filename);
         CommandLineArguments AdjustedArgs = Args;
ArgumentsAdjuster PluginArgumentsRemover =
             AdjustedArgs.push_back(Args[I]);
         }
         return AdjustedArgs;
       };
 
   Tool.appendArgumentsAdjuster(PerFileExtraArgumentsInserter);
   Tool.appendArgumentsAdjuster(PluginArgumentsRemover);
   Context.setEnableProfiling(EnableCheckProfile);
+  Context.setProfileStoragePrefix(StoreCheckProfile);
 
   ClangTidyDiagnosticConsumer DiagConsumer(Context);
 
   Tool.setDiagnosticConsumer(&DiagConsumer);
 
   class ActionFactory : public FrontendActionFactory {
   public:
     ActionFactory(ClangTidyContext &Context) : ConsumerFactory(Context) {}

clang-tidy/ClangTidyDiagnosticConsumer.h

 //===--- ClangTidyDiagnosticConsumer.h - clang-tidy -------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANG_TIDY_CLANGTIDYDIAGNOSTICCONSUMER_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANG_TIDY_CLANGTIDYDIAGNOSTICCONSUMER_H
 
 #include "ClangTidyOptions.h"
+#include "ClangTidyProfiling.h"
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Tooling/Core/Diagnostic.h"
 #include "clang/Tooling/Refactoring.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/Support/Regex.h"
 #include "llvm/Support/Timer.h"
public:
 
   /// \brief Clears collected errors.
   void clearErrors() { Errors.clear(); }
 
   /// \brief Control profile collection in clang-tidy.
   void setEnableProfiling(bool Profile);
   bool getEnableProfiling() const { return Profile; }
 
+  /// \brief Control storage of profile date.
+  void setProfileStoragePrefix(StringRef ProfilePrefix);
+  llvm::Optional<ClangTidyProfiling::StorageParams>
+  getProfileStorageParams() const;
+
   /// \brief Should be called when starting to process new translation unit.
   void setCurrentBuildDirectory(StringRef BuildDirectory) {
     CurrentBuildDirectory = BuildDirectory;
   }
 
   /// \brief Returns build directory of the current translation unit.
   const std::string &getCurrentBuildDirectory() {
     return CurrentBuildDirectory;
private:
 
   ClangTidyStats Stats;
 
   std::string CurrentBuildDirectory;
 
   llvm::DenseMap<unsigned, std::string> CheckNamesByDiagnosticID;
 
   bool Profile;
+  std::string ProfilePrefix;
 
   bool AllowEnablingAnalyzerAlphaCheckers;
 };
 
 /// \brief A diagnostic consumer that turns each \c Diagnostic into a
 /// \c SourceManager-independent \c ClangTidyError.
 //
 // FIXME: If we move away from unit-tests, this can be moved to a private

clang-tidy/ClangTidyDiagnosticConsumer.cpp

ClangTidyOptions ClangTidyContext::getOptionsForFile(StringRef File) const {
   // Merge options on top of getDefaults() as a safeguard against options with
   // unset values.
   return ClangTidyOptions::getDefaults().mergeWith(
       OptionsProvider->getOptions(File));
 }
 
 void ClangTidyContext::setEnableProfiling(bool P) { Profile = P; }
 
+void ClangTidyContext::setProfileStoragePrefix(StringRef Prefix) {
+  ProfilePrefix = Prefix;
+}
+
+llvm::Optional<ClangTidyProfiling::StorageParams>
+ClangTidyContext::getProfileStorageParams() const {
+  if (ProfilePrefix.empty())
+    return llvm::None;
+
+  return ClangTidyProfiling::StorageParams(ProfilePrefix, CurrentFile);
+}
+
 bool ClangTidyContext::isCheckEnabled(StringRef CheckName) const {
   assert(CheckFilter != nullptr);
   return CheckFilter->contains(CheckName);
 }
 
 bool ClangTidyContext::treatAsError(StringRef CheckName) const {
   assert(WarningAsErrorFilter != nullptr);
   return WarningAsErrorFilter->contains(CheckName);

clang-tidy/ClangTidyProfiling.h

 //===--- ClangTidyProfiling.h - clang-tidy ----------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANG_TIDY_CLANGTIDYPROFILING_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANG_TIDY_CLANGTIDYPROFILING_H
 
+#include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringMap.h"
-#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Chrono.h"
 #include "llvm/Support/Timer.h"
 #include "llvm/Support/raw_ostream.h"
+#include <string>
 #include <utility>
 #include <vector>
 
 namespace clang {
 namespace tidy {
 
 class ClangTidyProfiling {
-  // Time is first to allow for sorting by it.
-  std::vector<std::pair<llvm::TimeRecord, llvm::StringRef>> Timers;
-  llvm::TimeRecord Total;
+public:
+  struct StorageParams {
+    llvm::sys::TimePoint<> Timestamp;
+    std::string SourceFilename;
+    std::string StoreFilename;
+
+    StorageParams() = default;
+
+    StorageParams(llvm::StringRef ProfilePrefix, llvm::StringRef SourceFile);
+  };
 
-  void preprocess();
+private:
+  llvm::Optional<llvm::TimerGroup> TG;
 
-  void printProfileData(llvm::raw_ostream &OS) const;
+  llvm::Optional<StorageParams> Storage;
+
+  void printUserFriendlyTable(llvm::raw_ostream &OS);
+  void printAsJSON(llvm::raw_ostream &OS);
+
+  void storeProfileData();
 
 public:
   llvm::StringMap<llvm::TimeRecord> Records;
 
+  ClangTidyProfiling() = default;
+
+  ClangTidyProfiling(llvm::Optional<StorageParams> Storage);
+
   ~ClangTidyProfiling();
 };
 
 } // end namespace tidy
 } // end namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANG_TIDY_CLANGTIDYPROFILING_H

clang-tidy/ClangTidyProfiling.cpp

 //===--- ClangTidyProfiling.cpp - clang-tidy --------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "ClangTidyProfiling.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/YAMLTraits.h"
+#include "llvm/Support/raw_ostream.h"
+#include <system_error>
+#include <utility>
 
 #define DEBUG_TYPE "clang-tidy-profiling"
 
 namespace clang {
 namespace tidy {
 
-void ClangTidyProfiling::preprocess() {
-  // Convert from a insertion-friendly map to sort-friendly vector.
-  Timers.clear();
-  Timers.reserve(Records.size());
-  for (const auto &P : Records) {
-    Timers.emplace_back(P.getValue(), P.getKey());
-    Total += P.getValue();
-  }
-  assert(Timers.size() == Records.size() && "Size mismatch after processing");
-
-  // We want the measurements to be sorted by decreasing time spent.
-  llvm::sort(Timers.begin(), Timers.end());
-}
-
-void ClangTidyProfiling::printProfileData(llvm::raw_ostream &OS) const {
-  std::string Line = "===" + std::string(73, '-') + "===\n";
-  OS << Line;
-
-  if (Total.getUserTime())
-    OS << "   ---User Time---";
-  if (Total.getSystemTime())
-    OS << "   --System Time--";
-  if (Total.getProcessTime())
-    OS << "   --User+System--";
-  OS << "   ---Wall Time---";
-  if (Total.getMemUsed())
-    OS << "  ---Mem---";
-  OS << "  --- Name ---\n";
-
-  // Loop through all of the timing data, printing it out.
-  for (auto I = Timers.rbegin(), E = Timers.rend(); I != E; ++I) {
-    I->first.print(Total, OS);
-    OS << I->second << '\n';
-  }
-
-  Total.print(Total, OS);
-  OS << "Total\n";
-  OS << Line << "\n";
+ClangTidyProfiling::StorageParams::StorageParams(llvm::StringRef ProfilePrefix,
+                                                 llvm::StringRef SourceFile)
+    : Timestamp(std::chrono::system_clock::now()), SourceFilename(SourceFile) {
+  llvm::SmallString<32> TimestampStr;
+  llvm::raw_svector_ostream OS(TimestampStr);
+  llvm::format_provider<decltype(Timestamp)>::format(Timestamp, OS,
+                                                     "%Y%m%d%H%M%S%N");
+
+  llvm::SmallString<256> FinalPrefix(ProfilePrefix);
+  llvm::sys::path::append(FinalPrefix, TimestampStr);
+
+  // So the full output name is: /ProfilePrefix/timestamp-inputfilename.json
+  StoreFilename = llvm::Twine(FinalPrefix + "-" +
+                              llvm::sys::path::filename(SourceFile) + ".json")
+                      .str();
+}
+
+void ClangTidyProfiling::printUserFriendlyTable(llvm::raw_ostream &OS) {
+  TG->print(OS);
   OS.flush();
 }
 
+void ClangTidyProfiling::printAsJSON(llvm::raw_ostream &OS) {
+  OS << "{\n";
+  OS << "\"file\": \"" << Storage->SourceFilename << "\",\n";
+  OS << "\"timestamp\": \"" << Storage->Timestamp << "\",\n";
+  OS << "\"profile\": {\n";
+  TG->printJSONValues(OS, "");
+  OS << "\n}\n";
+  OS << "}\n";
+  OS.flush();
+}
+
+void ClangTidyProfiling::storeProfileData() {
+  assert(Storage.hasValue() && "We should have a filename.");
+
+  llvm::SmallString<256> OutputDirectory(Storage->StoreFilename);
+  llvm::sys::path::remove_filename(OutputDirectory);
+  if (std::error_code EC = llvm::sys::fs::create_directories(OutputDirectory)) {
+    llvm::errs() << "Unable to create output directory '" << OutputDirectory
+                 << "': " << EC.message() << "\n";
+    return;
+  }
+
+  std::error_code EC;
+  llvm::raw_fd_ostream OS(Storage->StoreFilename, EC, llvm::sys::fs::F_None);
+  if (EC) {
+    llvm::errs() << "Error opening output file '" << Storage->StoreFilename
+                 << "': " << EC.message() << "\n";
+    return;
+  }
+
+  printAsJSON(OS);
+}
+
+ClangTidyProfiling::ClangTidyProfiling(llvm::Optional<StorageParams> Storage)
+    : Storage(std::move(Storage)) {}
+
 ClangTidyProfiling::~ClangTidyProfiling() {
-  preprocess();
-  printProfileData(llvm::errs());
+  TG.emplace("clang-tidy", "clang-tidy checks profiling", Records);
+
+  if (!Storage.hasValue())
+    printUserFriendlyTable(llvm::errs());
+  else
+    storeProfileData();
 }
 
 } // namespace tidy
 } // namespace clang

clang-tidy/tool/ClangTidyMain.cpp

 
 static cl::opt<bool> EnableCheckProfile("enable-check-profile", cl::desc(R"(
 Enable per-check timing profiles, and print a
 report to stderr.
 )"),
                                         cl::init(false),
                                         cl::cat(ClangTidyCategory));
 
+static cl::opt<std::string> StoreCheckProfile("store-check-profile",
+                                              cl::desc(R"(
+By default reports are printed in tabulated
+format to stderr. When this option is passed,
+these per-TU profiles are instead stored as JSON.
+)"),
+                                              cl::value_desc("prefix"),
+                                              cl::cat(ClangTidyCategory));
+
 /// This option allows enabling the experimental alpha checkers from the static
 /// analyzer. This option is set to false and not visible in help, because it is
 /// highly not recommended for users.
 static cl::opt<bool>
     AllowEnablingAnalyzerAlphaCheckers("allow-enabling-analyzer-alpha-checkers",
                                        cl::init(false), cl::Hidden,
                                        cl::cat(ClangTidyCategory));
 
static int clangTidyMain(int argc, const char **argv) {
   if (!BaseFS)
     return 1;
 
   auto OwningOptionsProvider = createOptionsProvider(BaseFS);
   auto *OptionsProvider = OwningOptionsProvider.get();
   if (!OptionsProvider)
     return 1;
 
+  auto MakeAbsolute = [](const std::string &Input) -> SmallString<256> {
+    if (Input.empty())
+      return {};
+    SmallString<256> AbsolutePath(Input);
+    if (std::error_code EC = llvm::sys::fs::make_absolute(AbsolutePath)) {
+      llvm::errs() << "Can't make absolute path from " << Input << ": "
+                   << EC.message() << "\n";
+    }
+    return AbsolutePath;
+  };
+
+  SmallString<256> ProfilePrefix = MakeAbsolute(StoreCheckProfile);
+
   StringRef FileName("dummy");
   auto PathList = OptionsParser.getSourcePathList();
   if (!PathList.empty()) {
     FileName = PathList.front();
   }
 
-  SmallString<256> FilePath(FileName);
-  if (std::error_code EC = llvm::sys::fs::make_absolute(FilePath)) {
-    llvm::errs() << "Can't make absolute path from " << FileName << ": "
-                 << EC.message() << "\n";
-  }
+  SmallString<256> FilePath = MakeAbsolute(FileName);
+
   ClangTidyOptions EffectiveOptions = OptionsProvider->getOptions(FilePath);
   std::vector<std::string> EnabledChecks =
       getCheckNames(EffectiveOptions, AllowEnablingAnalyzerAlphaCheckers);
 
   if (ExplainConfig) {
     // FIXME: Show other ClangTidyOptions' fields, like ExtraArg.
     std::vector<clang::tidy::ClangTidyOptionsProvider::OptionsSource>
         RawOptions = OptionsProvider->getRawOptions(FilePath);
static int clangTidyMain(int argc, const char **argv) {
 
   llvm::InitializeAllTargetInfos();
   llvm::InitializeAllTargetMCs();
   llvm::InitializeAllAsmParsers();
 
   ClangTidyContext Context(std::move(OwningOptionsProvider),
                            AllowEnablingAnalyzerAlphaCheckers);
   runClangTidy(Context, OptionsParser.getCompilations(), PathList, BaseFS,
-               EnableCheckProfile);
+               EnableCheckProfile, ProfilePrefix);
   ArrayRef<ClangTidyError> Errors = Context.getErrors();
   bool FoundErrors = llvm::find_if(Errors, [](const ClangTidyError &E) {
                        return E.DiagLevel == ClangTidyError::Error;
                      }) != Errors.end();
 
   const bool DisableFixes = Fix && FoundErrors && !FixErrors;
 
   unsigned WErrorCount = 0;

docs/ReleaseNotes.rst

 Improvements to clang-rename
 ----------------------------
 
 The improvements are...
 
 Improvements to clang-tidy
 --------------------------
 
+- The checks profiling info can now be stored as JSON files for futher
+  post-processing and analysis.
+
 - New module `abseil` for checks related to the `Abseil <https://abseil.io>`_
   library.
 
 - New module ``portability``.
 
 - New module ``zircon`` for checks related to Fuchsia's Zircon kernel.
 
 - New :doc:`android-comparison-in-temp-failure-retry

docs/clang-tidy/index.rst

 
 The ``-fix`` flag instructs :program:`clang-tidy` to fix found errors if
 supported by corresponding checks.
 
 An overview of all the command-line options:
 
 .. code-block:: console
 
-  $ clang-tidy -help
+  $ clang-tidy --help
   USAGE: clang-tidy [options] <source0> [... <sourceN>]
 
   OPTIONS:
 
   Generic Options:
 
-    -help                        - Display available options (-help-hidden for more)
-    -help-list                   - Display list of available options (-help-list-hidden for more)
-    -version                     - Display the version of this program
+    -help                         - Display available options (-help-hidden for more)
+    -help-list                    - Display list of available options (-help-list-hidden for more)
+    -version                      - Display the version of this program
 
   clang-tidy options:
 
-    -checks=<string>             -
+    -checks=<string>              -
                                     Comma-separated list of globs with optional '-'
                                     prefix. Globs are processed in order of
                                     appearance in the list. Globs without '-'
                                     prefix add checks with matching names to the
                                     set, globs with the '-' prefix remove checks
                                     with matching names from the set of enabled
                                     checks. This option's value is appended to the
                                     value of the 'Checks' option in .clang-tidy
                                     file, if any.
-    -config=<string>             -
+    -config=<string>              -
                                     Specifies a configuration in YAML/JSON format:
                                       -config="{Checks: '*',
                                                 CheckOptions: [{key: x,
                                                                 value: y}]}"
                                     When the value is empty, clang-tidy will
                                     attempt to find a file named .clang-tidy for
                                     each source file in its parent directories.
-    -dump-config                 -
+    -dump-config                  -
                                     Dumps configuration in the YAML format to
                                     stdout. This option can be used along with a
                                     file name (and '--' if the file is outside of a
                                     project with configured compilation database).
                                     The configuration used for this file will be
                                     printed.
                                     Use along with -checks=* to include
                                     configuration of all checks.
-    -enable-check-profile        -
+    -enable-check-profile         -
                                     Enable per-check timing profiles, and print a
                                     report to stderr.
-    -explain-config              -
+    -explain-config               -
                                     For each enabled check explains, where it is
                                     enabled, i.e. in clang-tidy binary, command
                                     line or a specific configuration file.
-    -export-fixes=<filename>     -
+    -export-fixes=<filename>      -
                                     YAML file to store suggested fixes in. The
                                     stored fixes can be applied to the input source
                                     code with clang-apply-replacements.
-    -extra-arg=<string>          - Additional argument to append to the compiler command line
-    -extra-arg-before=<string>   - Additional argument to prepend to the compiler command line
-    -fix                         -
+    -extra-arg=<string>           - Additional argument to append to the compiler command line
+    -extra-arg-before=<string>    - Additional argument to prepend to the compiler command line
+    -fix                          -
                                     Apply suggested fixes. Without -fix-errors
                                     clang-tidy will bail out if any compilation
                                     errors were found.
-    -fix-errors                  -
+    -fix-errors                   -
                                     Apply suggested fixes even if compilation
                                     errors were found. If compiler errors have
                                     attached fix-its, clang-tidy will apply them as
                                     well.
-    -format-style=<string>       -
+    -format-style=<string>        -
                                     Style for formatting code around applied fixes:
                                       - 'none' (default) turns off formatting
                                       - 'file' (literally 'file', not a placeholder)
                                         uses .clang-format file in the closest parent
                                         directory
                                       - '{ <json> }' specifies options inline, e.g.
                                         -format-style='{BasedOnStyle: llvm, IndentWidth: 8}'
                                       - 'llvm', 'google', 'webkit', 'mozilla'
                                     See clang-format documentation for the up-to-date
                                     information about formatting styles and options.
                                     This option overrides the 'FormatStyle` option in
                                     .clang-tidy file, if any.
-    -header-filter=<string>      -
+    -header-filter=<string>       -
                                     Regular expression matching the names of the
                                     headers to output diagnostics from. Diagnostics
                                     from the main file of each translation unit are
                                     always displayed.
                                     Can be used together with -line-filter.
                                     This option overrides the 'HeaderFilter' option
                                     in .clang-tidy file, if any.
-    -line-filter=<string>        -
+    -line-filter=<string>         -
                                     List of files with line ranges to filter the
                                     warnings. Can be used together with
                                     -header-filter. The format of the list is a
                                     JSON array of objects:
                                       [
                                         {"name":"file1.cpp","lines":[[1,3],[5,7]]},
                                         {"name":"file2.h"}
                                       ]
-    -list-checks                 -
+    -list-checks                  -
                                     List all enabled checks and exit. Use with
                                     -checks=* to list all available checks.
-    -p=<string>                  - Build path
-    -quiet                       -
+    -p=<string>                   - Build path
+    -quiet                        -
                                     Run clang-tidy in quiet mode. This suppresses
                                     printing statistics about ignored warnings and
                                     warnings treated as errors if the respective
                                     options are specified.
-    -system-headers              - Display the errors from system headers.
-    -warnings-as-errors=<string> -
+    -store-check-profile=<prefix> -
+                                    By default reports are printed in tabulated
+                                    format to stderr. When this option is passed,
+                                    these per-TU profiles are instead stored as JSON.
+    -system-headers               - Display the errors from system headers.
+    -vfsoverlay=<filename>        -
+                                    Overlay the virtual filesystem described by file
+                                    over the real file system.
+    -warnings-as-errors=<string>  -
                                     Upgrades warnings to errors. Same format as
                                     '-checks'.
                                     This option's value is appended to the value of
                                     the 'WarningsAsErrors' option in .clang-tidy
                                     file, if any.
 
   -p <build-path> is used to read a compile command database.
 
           For example, it can be a CMake build directory in which a file named
           compile_commands.json exists (use -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
           CMake option to get this output). When no build path is specified,
           a search for compile_commands.json will be attempted through all
           parent paths of the first input file . See:
configuration can be inspected using -dump-config:
       HeaderFilterRegex: ''
       FormatStyle:     none
       User:            user
       CheckOptions:
         - key:             some-check.SomeOption
           value:           'some value'
       ...
 
 :program:`clang-tidy` diagnostics are intended to call out code that does
 not adhere to a coding standard, or is otherwise problematic in some way.
 However, if it is known that the code is correct, the check-specific ways
 to silence the diagnostics could be used, if they are available (e.g. 
 bugprone-use-after-move can be silenced by re-initializing the variable after 
 it has been moved out, misc-string-integer-assignment can be suppressed by 
 explicitly casting the integer to char, readability-implicit-bool-conversion
 can also be suppressed by using explicit casts, etc.). If they are not 
 available or if changing the semantics of the code is not desired, 
* To restrict the files examined you can provide one or more regex arguments
   checks. It may also be necessary to restrict the header files warnings are
   displayed from using the ``-header-filter`` flag. It has the same behavior
   as the corresponding :program:`clang-tidy` flag.
 
 * To apply suggested fixes ``-fix`` can be passed as an argument. This gathers
   all changes in a temporary directory and applies them. Passing ``-format``
   will run clang-format over changed lines.
 
+
+On checks profiling
+-------------------
+
+:program:`clang-tidy` can collect per-check profiling info, and output it
+for each processed source file (translation unit).
+
+To enable profiling info collection, use the ``-enable-check-profile`` argument.
+The timings will be output to ``stderr`` as a table. Example output:
+
+.. code-block:: console
+
+  $ clang-tidy -enable-check-profile -checks=-*,readability-function-size source.cpp
+  ===-------------------------------------------------------------------------===
+                            clang-tidy checks profiling
+  ===-------------------------------------------------------------------------===
+    Total Execution Time: 1.0282 seconds (1.0258 wall clock)
+
+     ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
+     0.9136 (100.0%)   0.1146 (100.0%)   1.0282 (100.0%)   1.0258 (100.0%)  readability-function-size
+     0.9136 (100.0%)   0.1146 (100.0%)   1.0282 (100.0%)   1.0258 (100.0%)  Total
+
+It can also store that data as JSON files for further processing. Example output:
+
+.. code-block:: console
+
+  $ clang-tidy -enable-check-profile -store-check-profile=.  -checks=-*,readability-function-size source.cpp
+  $ # Note that there won't be timings table printed to the console.
+  $ ls /tmp/out/
+  20180516161318717446360-source.cpp.json
+  $ cat 20180516161318717446360-source.cpp.json
+  {
+  "file": "/path/to/source.cpp",
+  "timestamp": "2018-05-16 16:13:18.717446360",
+  "profile": {
+    "time.clang-tidy.readability-function-size.wall": 1.0421266555786133e+00,
+    "time.clang-tidy.readability-function-size.user": 9.2088400000005421e-01,
+    "time.clang-tidy.readability-function-size.sys": 1.2418899999999974e-01
+  }
+  }
+
+There is only one argument that controls profile storage:
+
+* ``-store-check-profile=<prefix>``
+
+  By default reports are printed in tabulated format to stderr. When this option
+  is passed, these per-TU profiles are instead stored as JSON.
+  If the prefix is not an absolute path, it is considered to be relative to the
+  directory from where you have run :program:`clang-tidy`. All ``.`` and ``..``
+  patterns in the path are collapsed, and symlinks are resolved.
+
+  Example:
+  Let's suppose you have a source file named ``example.cpp``, located in the
+  ``/source`` directory. Only the input filename is used, not the full path
+  to the source file. Additionally, it is prefixed with the current timestamp.
+
+  * If you specify ``-store-check-profile=/tmp``, then the profile will be saved
+    to ``/tmp/<ISO8601-like timestamp>-example.cpp.json``
+
+  * If you run :program:`clang-tidy` from within ``/foo`` directory, and specify
+    ``-store-check-profile=.``, then the profile will still be saved to
+    ``/foo/<ISO8601-like timestamp>-example.cpp.json``

test/clang-tidy/clang-tidy-enable-check-profile-one-tu.cpp

 // RUN: clang-tidy -enable-check-profile -checks='-*,readability-function-size' %s -- 2>&1 | FileCheck --match-full-lines -implicit-check-not='{{warning:|error:}}' %s
 
 // CHECK: ===-------------------------------------------------------------------------===
-// CHECK-NEXT: {{.*}}  --- Name ---
+// CHECK-NEXT:                          clang-tidy checks profiling
+// CHECK-NEXT: ===-------------------------------------------------------------------------===
+// CHECK-NEXT: Total Execution Time: {{.*}} seconds ({{.*}} wall clock)
+
+// CHECK: {{.*}}  --- Name ---
 // CHECK-NEXT: {{.*}}  readability-function-size
 // CHECK-NEXT: {{.*}}  Total
-// CHECK-NEXT: ===-------------------------------------------------------------------------===
 
 // CHECK-NOT: ===-------------------------------------------------------------------------===
+// CHECK-NOT:                          clang-tidy checks profiling
+// CHECK-NOT: ===-------------------------------------------------------------------------===
+// CHECK-NOT: Total Execution Time: {{.*}} seconds ({{.*}} wall clock)
+
 // CHECK-NOT: {{.*}}  --- Name ---
 // CHECK-NOT: {{.*}}  readability-function-size
 // CHECK-NOT: {{.*}}  Total
-// CHECK-NOT: ===-------------------------------------------------------------------------===
 
 class A {
   A() {}
   ~A() {}
 };

test/clang-tidy/clang-tidy-enable-check-profile-two-tu.cpp

 // RUN: clang-tidy -enable-check-profile -checks='-*,readability-function-size' %s %s -- 2>&1 | FileCheck --match-full-lines -implicit-check-not='{{warning:|error:}}' %s
 
 // CHECK: ===-------------------------------------------------------------------------===
-// CHECK-NEXT: {{.*}}  --- Name ---
+// CHECK-NEXT:                          clang-tidy checks profiling
+// CHECK-NEXT: ===-------------------------------------------------------------------------===
+// CHECK-NEXT: Total Execution Time: {{.*}} seconds ({{.*}} wall clock)
+
+// CHECK: {{.*}}  --- Name ---
 // CHECK-NEXT: {{.*}}  readability-function-size
 // CHECK-NEXT: {{.*}}  Total
-// CHECK-NEXT: ===-------------------------------------------------------------------------===
 
 // CHECK: ===-------------------------------------------------------------------------===
-// CHECK-NEXT: {{.*}}  --- Name ---
+// CHECK-NEXT:                          clang-tidy checks profiling
+// CHECK-NEXT: ===-------------------------------------------------------------------------===
+// CHECK-NEXT: Total Execution Time: {{.*}} seconds ({{.*}} wall clock)
+
+// CHECK: {{.*}}  --- Name ---
 // CHECK-NEXT: {{.*}}  readability-function-size
 // CHECK-NEXT: {{.*}}  Total
-// CHECK-NEXT: ===-------------------------------------------------------------------------===
 
 // CHECK-NOT: ===-------------------------------------------------------------------------===
+// CHECK-NOT:                          clang-tidy checks profiling
+// CHECK-NOT: ===-------------------------------------------------------------------------===
+// CHECK-NOT: Total Execution Time: {{.*}} seconds ({{.*}} wall clock)
+
 // CHECK-NOT: {{.*}}  --- Name ---
 // CHECK-NOT: {{.*}}  readability-function-size
 // CHECK-NOT: {{.*}}  Total
-// CHECK-NOT: ===-------------------------------------------------------------------------===
 
 class A {
   A() {}
   ~A() {}
 };

test/clang-tidy/clang-tidy-store-check-profile-one-tu.cpp

+// RUN: rm -rf %T/out
+// RUN: clang-tidy -enable-check-profile -checks='-*,readability-function-size' -store-check-profile=%T/out %s -- 2>&1 | not FileCheck --match-full-lines -implicit-check-not='{{warning:|error:}}' -check-prefix=CHECK-CONSOLE %s
+// RUN: cat %T/out/*-clang-tidy-store-check-profile-one-tu.cpp.json | FileCheck --match-full-lines -implicit-check-not='{{warning:|error:}}' -check-prefix=CHECK-FILE %s
+// RUN: rm -rf %T/out
+// RUN: clang-tidy -enable-check-profile -checks='-*,readability-function-size' -store-check-profile=%T/out %s -- 2>&1
+// RUN: cat %T/out/*-clang-tidy-store-check-profile-one-tu.cpp.json | FileCheck --match-full-lines -implicit-check-not='{{warning:|error:}}' -check-prefix=CHECK-FILE %s
+
+// CHECK-CONSOLE-NOT: ===-------------------------------------------------------------------------===
+// CHECK-CONSOLE-NOT: {{.*}}  --- Name ---
+// CHECK-CONSOLE-NOT: {{.*}}  readability-function-size
+// CHECK-CONSOLE-NOT: {{.*}}  Total
+// CHECK-CONSOLE-NOT: ===-------------------------------------------------------------------------===
+
+// CHECK-FILE: {
+// CHECK-FILE-NEXT:"file": "{{.*}}clang-tidy-store-check-profile-one-tu.cpp",
+// CHECK-FILE-NEXT:"timestamp": "{{[0-9]+}}-{{[0-9]+}}-{{[0-9]+}} {{[0-9]+}}:{{[0-9]+}}:{{[0-9]+}}.{{[0-9]+}}",
+// CHECK-FILE-NEXT:"profile": {
+// CHECK-FILE-NEXT: "time.clang-tidy.readability-function-size.wall": {{.*}}{{[0-9]}}.{{[0-9]+}}e{{[-+]}}{{[0-9]}}{{[0-9]}},
+// CHECK-FILE-NEXT: "time.clang-tidy.readability-function-size.user": {{.*}}{{[0-9]}}.{{[0-9]+}}e{{[-+]}}{{[0-9]}}{{[0-9]}},
+// CHECK-FILE-NEXT: "time.clang-tidy.readability-function-size.sys": {{.*}}{{[0-9]}}.{{[0-9]+}}e{{[-+]}}{{[0-9]}}{{[0-9]}}
+// CHECK-FILE-NEXT: }
+// CHECK-FILE-NEXT: }
+
+// CHECK-FILE-NOT: {
+// CHECK-FILE-NOT: "file": {{.*}}clang-tidy-store-check-profile-one-tu.cpp{{.*}},
+// CHECK-FILE-NOT: "timestamp": "{{[0-9]+}}-{{[0-9]+}}-{{[0-9]+}} {{[0-9]+}}:{{[0-9]+}}:{{[0-9]+}}.{{[0-9]+}}",
+// CHECK-FILE-NOT: "profile": {
+// CHECK-FILE-NOT:  "time.clang-tidy.readability-function-size.wall": {{.*}}{{[0-9]}}.{{[0-9]+}}e{{[-+]}}{{[0-9]}}{{[0-9]}},
+// CHECK-FILE-NOT:  "time.clang-tidy.readability-function-size.user": {{.*}}{{[0-9]}}.{{[0-9]+}}e{{[-+]}}{{[0-9]}}{{[0-9]}},
+// CHECK-FILE-NOT:  "time.clang-tidy.readability-function-size.sys": {{.*}}{{[0-9]}}.{{[0-9]+}}e{{[-+]}}{{[0-9]}}{{[0-9]}}
+// CHECK-FILE-NOT: }
+// CHECK-FILE-NOT: }
+
+class A {
+  A() {}
+  ~A() {}
+};