Add support for a backdoor driver option that enables emitting header usage information in JSON to a file

[ahatanak]

Summary

Each line in the file is a JSON object that has the name of the main source file followed by the list of system header files included directly or indirectly from that file.

For example:

{"source":"/tmp/foo.c", "includes":["/usr/include/stdio.h", "/usr/include/stdlib.h"]}

To reduce the amount of data written to the file, only the system headers that are directly included from a file that isn't in the system directory are recorded.

[jansvoboda11]

Mostly looks good. Is this expected to work with Clang modules at all?

clang/include/clang/Driver/Driver.h

248

Please document this.

clang/include/clang/Driver/Options.td

5659

Could this be an enum of the desired format instead? It would default to textual and be overwritten with something like -header-include-format=json? I think it would result in simpler interface if in the future we decide to add another format.

clang/lib/Frontend/HeaderIncludeGen.cpp

68

Do we really want this to be a set? I can imagine clients wanting the include order to be preserved and with potential duplicates (for textual headers without guards or #pragma once).

If the set semantics are indeed what we want, I'd recommend using llvm::StringSet<> instead.

[ahatanak]

Address review comments.

[ahatanak]

In D137996#3927947, @jansvoboda11 wrote:

Mostly looks good. Is this expected to work with Clang modules at all?

My use case only has to keep track of textual headers.

[jansvoboda11]

clang/include/clang/Driver/Driver.h

240

Can we make this HeaderIncludeFormatEnum instead of std::string?

clang/include/clang/Frontend/DependencyOutputOptions.h

49

I'd either drop the Enum suffix or replace it with Kind. But up to you.

clang/lib/Frontend/HeaderIncludeGen.cpp

258

This can be StringRef I think.

[ahatanak]

Address review comments. Also, change check string in test case to fix pre-merge checks that were failing.

clang/lib/Frontend/HeaderIncludeGen.cpp

258

I simplified this part by removing the code that checks whether MainID is valid. It's set when BeginSourceFile is called, so it should be valid.

[jansvoboda11]

clang/lib/Frontend/HeaderIncludeGen.cpp

294–295

Hmm, this new logic seems orthogonal to the output format. Changing the set of headers only based on the desired format seems counter-intuitive. I think we should either allow specifying the desired logic via a separate command-line flag, or decouple the new feature from the -header-include-file flag entirely. What are your thoughts?

(Sorry I didn't notice this earlier.)

[ahatanak]

Add another environment variable CC_PRINT_HEADERS_FILTERING, which can be used to specify whether header information should be filtered or not.

Currently, CC_PRINT_HEADERS_FORMAT=json, CC_PRINT_HEADERS_FILTERING=on and CC_PRINT_HEADERS_FORMAT=textual, CC_PRINT_HEADERS_FILTERING=off are allowed. CC_PRINT_HEADERS=1 gets translated to CC_PRINT_HEADERS_FORMAT=textual, CC_PRINT_HEADERS_FILTERING=off.

[jansvoboda11]

clang/lib/Driver/ToolChains/Clang.cpp

5660

Let's change "on" to something that won't interfere with potential future filters. How about "no-transitive-system"? (or "only-direct-system" if that better fits the semantics)

Same for the command-line counterpart.

clang/lib/Frontend/HeaderIncludeGen.cpp

161

I don't think asserting on invalid user input is a good strategy. Let's emit a diagnostic and exit gracefully instead.

clang/tools/driver/driver.cpp

278

This should probably be a proper diagnostic.

[ahatanak]

clang/lib/Frontend/HeaderIncludeGen.cpp

161

The combination of (textual,filtering) is rejected by the driver, so the assertion should always succeed if the user invoked the driver and set the environment variables. Do we need a diagnostic for the case where -header-include-format and -header-include-filtering are passed on the CC1 command line by the user?

[jansvoboda11]

clang/lib/Frontend/HeaderIncludeGen.cpp

161

Ah, you're right about the driver. Yes. I think we should emit a diagnostic when this is passed to cc1 on the command line. Good place for this might be ParseDependencyOutputArgs in CompilerInvocation.cpp, after the marshalling layer populates the DependencyOutputOptions fields.

[ahatanak]

Address review comments.

[jansvoboda11]

LGTM, thank you for working through this!

[thakis]

Looks like this breaks tests on windows: http://45.33.8.238/win/71451/step_7.txt

Please take a look and revert for now if it takes a while to fix.

[ahatanak]

abc87ad4ed6cc5e6568751f375d1c646edb502f7 should fix the failing test.

[SeanP]

clang/tools/driver/driver.cpp

250

This change is not POSIX compliant. If T is char *, the next call to getenv() on line 253 invalidates the value saved into OptVal. getenv() is allowed to use the same buffer for the return value. That means the value OptVal is pointing at will be overwritten by the next call.

When T is bool you need the !! operators to get the char * converted to bool correctly.

I suggest leaving this function as it was and calling getenv() directly for the scenario involving CC_PRINT_HEADERS_FORMAT. Make sure you save the getenv() results into std::string so the value isn't overridden.

[ahatanak]

clang/tools/driver/driver.cpp

250

Thank you for notifying me of the bug. It should be fixed in https://github.com/llvm/llvm-project/commit/34aa2e24c89ae39c0db4254d8aafcae0285dbe34

clang/include/clang/Basic/DiagnosticDriverKinds.td

 def err_drv_negative_columns : Error<
   "invalid value '%1' in '%0', value must be 'none' or a positive integer">;
 def err_drv_small_columns : Error<
   "invalid value '%1' in '%0', value must be '%2' or greater">;
 
 def err_drv_invalid_malign_branch_EQ : Error<
   "invalid argument '%0' to -malign-branch=; each element must be one of: %1">;
 
+def err_drv_print_header_env_var : Error<
+  "environment variable CC_PRINT_HEADERS_%select{FORMAT|FILTERING}0 has invalid value %1">;
+def err_drv_print_header_env_var_combination : Error<
+  "unsupported combination: CC_PRINT_HEADERS_FORMAT=%0 and CC_PRINT_HEADERS_FILTERING=%1">;
+def err_drv_print_header_env_var_combination_cc1 : Error<
+  "unsupported combination: -header-include-format=%0 and -header-include-filtering=%1">;
+
 def warn_O4_is_O3 : Warning<"-O4 is equivalent to -O3">, InGroup<Deprecated>;
 def warn_drv_optimization_value : Warning<"optimization level '%0' is not supported; using '%1%2' instead">,
   InGroup<InvalidCommandLineArgument>;
 def warn_ignored_gcc_optimization : Warning<"optimization flag '%0' is not supported">,
   InGroup<IgnoredOptimizationArgument>;
 def warn_ignored_clang_option : Warning<"the flag '%0' has been deprecated and will be ignored">,
   InGroup<UnusedCommandLineArgument>;
 def warn_drv_unsupported_opt_for_target : Warning<

clang/include/clang/Basic/HeaderInclude.h

+//===--- HeaderInclude.h - Header Include -----------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Defines enums used when emitting included header information.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_BASIC_HEADERINCLUDEFORMATKIND_H
+#define LLVM_CLANG_BASIC_HEADERINCLUDEFORMATKIND_H
+#include "llvm/ADT/StringSwitch.h"
+#include "llvm/Support/ErrorHandling.h"
+#include <utility>
+
+namespace clang {
+/// The format in which header information is emitted.
+enum HeaderIncludeFormatKind { HIFMT_None, HIFMT_Textual, HIFMT_JSON };
+
+/// Whether header information is filtered or not. If HIFIL_Only_Direct_System
+/// is used, only information on system headers directly included from
+/// non-system headers is emitted.
+enum HeaderIncludeFilteringKind { HIFIL_None, HIFIL_Only_Direct_System };
+
+inline HeaderIncludeFormatKind
+stringToHeaderIncludeFormatKind(const char *Str) {
+  return llvm::StringSwitch<HeaderIncludeFormatKind>(Str)
+      .Case("textual", HIFMT_Textual)
+      .Case("json", HIFMT_JSON)
+      .Default(HIFMT_None);
+}
+
+inline bool stringToHeaderIncludeFiltering(const char *Str,
+                                           HeaderIncludeFilteringKind &Kind) {
+  std::pair<bool, HeaderIncludeFilteringKind> P =
+      llvm::StringSwitch<std::pair<bool, HeaderIncludeFilteringKind>>(Str)
+          .Case("none", {true, HIFIL_None})
+          .Case("only-direct-system", {true, HIFIL_Only_Direct_System})
+          .Default({false, HIFIL_None});
+  Kind = P.second;
+  return P.first;
+}
+
+inline const char *headerIncludeFormatKindToString(HeaderIncludeFormatKind K) {
+  switch (K) {
+  case HIFMT_None:
+    llvm_unreachable("unexpected format kind");
+  case HIFMT_Textual:
+    return "textual";
+  case HIFMT_JSON:
+    return "json";
+  }
+}
+
+inline const char *
+headerIncludeFilteringKindToString(HeaderIncludeFilteringKind K) {
+  switch (K) {
+  case HIFIL_None:
+    return "none";
+  case HIFIL_Only_Direct_System:
+    return "only-direct-system";
+  }
+}
+
+} // end namespace clang
+
+#endif // LLVM_CLANG_BASIC_HEADERINCLUDEFORMATKIND_H

clang/include/clang/Driver/Driver.h

 //===--- Driver.h - Clang GCC Compatible Driver -----------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_DRIVER_DRIVER_H
 #define LLVM_CLANG_DRIVER_DRIVER_H
 
 #include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/HeaderInclude.h"
 #include "clang/Basic/LLVM.h"
 #include "clang/Driver/Action.h"
 #include "clang/Driver/DriverDiagnostic.h"
 #include "clang/Driver/InputInfo.h"
 #include "clang/Driver/Options.h"
 #include "clang/Driver/Phases.h"
 #include "clang/Driver/ToolChain.h"
 #include "clang/Driver/Types.h"
public:
 
   /// Only print tool bindings, don't build any jobs.
   unsigned CCCPrintBindings : 1;
 
   /// Set CC_PRINT_OPTIONS mode, which is like -v but logs the commands to
   /// CCPrintOptionsFilename or to stderr.
   unsigned CCPrintOptions : 1;
 
-  /// Set CC_PRINT_HEADERS mode, which causes the frontend to log header include
-  /// information to CCPrintHeadersFilename or to stderr.
-  unsigned CCPrintHeaders : 1;
+  /// The format of the header information that is emitted. If CC_PRINT_HEADERS
+  /// is set, the format is textual. Otherwise, the format is determined by the
+  /// enviroment variable CC_PRINT_HEADERS_FORMAT.
+  HeaderIncludeFormatKind CCPrintHeadersFormat = HIFMT_None;
+
+  /// This flag determines whether clang should filter the header information
+  /// that is emitted. If enviroment variable CC_PRINT_HEADERS_FILTERING is set
+  /// to "only-direct-system", only system headers that are directly included
+  /// from non-system headers are emitted.
+  HeaderIncludeFilteringKind CCPrintHeadersFiltering = HIFIL_None;
 
   /// Set CC_LOG_DIAGNOSTICS mode, which causes the frontend to log diagnostics
   /// to CCLogDiagnosticsFilename or to stderr, in a stable machine readable
   /// format.
   unsigned CCLogDiagnostics : 1;
 
   /// Whether the driver is generating diagnostics for debugging purposes.
   unsigned CCGenDiagnostics : 1;
 
   /// Set CC_PRINT_PROC_STAT mode, which causes the driver to dump

clang/include/clang/Driver/Options.td

def sys_header_deps : Flag<["-"], "sys-header-deps">,
   HelpText<"Include system headers in dependency output">,
   MarshallingInfoFlag<DependencyOutputOpts<"IncludeSystemHeaders">>;
 def module_file_deps : Flag<["-"], "module-file-deps">,
   HelpText<"Include module files in dependency output">,
   MarshallingInfoFlag<DependencyOutputOpts<"IncludeModuleFiles">>;
 def header_include_file : Separate<["-"], "header-include-file">,
   HelpText<"Filename (or -) to write header include output to">,
   MarshallingInfoString<DependencyOutputOpts<"HeaderIncludeOutputFile">>;
+def header_include_format_EQ : Joined<["-"], "header-include-format=">,
+  HelpText<"set format in which header info is emitted">,
+  Values<"textual,json">, NormalizedValues<["HIFMT_Textual", "HIFMT_JSON"]>,
+  MarshallingInfoEnum<DependencyOutputOpts<"HeaderIncludeFormat">, "HIFMT_Textual">;
+def header_include_filtering_EQ : Joined<["-"], "header-include-filtering=">,
+  HelpText<"set the flag that enables filtering header information">,
+  Values<"none,only-direct-system">, NormalizedValues<["HIFIL_None", "HIFIL_Only_Direct_System"]>,
+  MarshallingInfoEnum<DependencyOutputOpts<"HeaderIncludeFiltering">, "HIFIL_None">;
 def show_includes : Flag<["--"], "show-includes">,
   HelpText<"Print cl.exe style /showIncludes to stdout">;
 
 } // let Flags = [CC1Option, NoDriverOption]
 
 //===----------------------------------------------------------------------===//
 // Diagnostic Options
 //===----------------------------------------------------------------------===//

clang/include/clang/Frontend/DependencyOutputOptions.h

 //===--- DependencyOutputOptions.h ------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_FRONTEND_DEPENDENCYOUTPUTOPTIONS_H
 #define LLVM_CLANG_FRONTEND_DEPENDENCYOUTPUTOPTIONS_H
 
+#include "clang/Basic/HeaderInclude.h"
 #include <string>
 #include <vector>
 
 namespace clang {
 
 /// ShowIncludesDestination - Destination for /showIncludes output.
 enum class ShowIncludesDestination { None, Stdout, Stderr };
 
unsigned UsePhonyTargets : 1;      ///< Include phony targets for each
                                      /// problems.
   unsigned AddMissingHeaderDeps : 1; ///< Add missing headers to dependency list
   unsigned IncludeModuleFiles : 1; ///< Include module file dependencies.
   unsigned ShowSkippedHeaderIncludes : 1; ///< With ShowHeaderIncludes, show
                                           /// also includes that were skipped
                                           /// due to the "include guard
                                           /// optimization" or #pragma once.
 
+  /// The format of header information.
+  HeaderIncludeFormatKind HeaderIncludeFormat = HIFMT_Textual;
+
+  /// Determine whether header information should be filtered.
+  HeaderIncludeFilteringKind HeaderIncludeFiltering = HIFIL_None;
+
   /// Destination of cl.exe style /showIncludes info.
   ShowIncludesDestination ShowIncludesDest = ShowIncludesDestination::None;
 
   /// The format for the dependency file.
   DependencyOutputFormat OutputFormat = DependencyOutputFormat::Make;
 
   /// The file to write dependency output to.
   std::string OutputFile;
public:
 
   /// The directory to copy module dependencies to when collecting them.
   std::string ModuleDependencyOutputDir;
 
 public:
   DependencyOutputOptions()
       : IncludeSystemHeaders(0), ShowHeaderIncludes(0), UsePhonyTargets(0),
         AddMissingHeaderDeps(0), IncludeModuleFiles(0),
-        ShowSkippedHeaderIncludes(0) {}
+        ShowSkippedHeaderIncludes(0), HeaderIncludeFormat(HIFMT_Textual),
+        HeaderIncludeFiltering(HIFIL_None) {}
 };
 
 }  // end namespace clang
 
 #endif

clang/lib/Driver/Driver.cpp

Driver::Driver(StringRef ClangExecutable, StringRef TargetTriple,
                DiagnosticsEngine &Diags, std::string Title,
                IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS)
     : Diags(Diags), VFS(std::move(VFS)), Mode(GCCMode),
       SaveTemps(SaveTempsNone), BitcodeEmbed(EmbedNone),
       Offload(OffloadHostDevice), CXX20HeaderType(HeaderMode_None),
       ModulesModeCXX20(false), LTOMode(LTOK_None),
       ClangExecutable(ClangExecutable), SysRoot(DEFAULT_SYSROOT),
       DriverTitle(Title), CCCPrintBindings(false), CCPrintOptions(false),
-      CCPrintHeaders(false), CCLogDiagnostics(false), CCGenDiagnostics(false),
+      CCLogDiagnostics(false), CCGenDiagnostics(false),
       CCPrintProcessStats(false), TargetTriple(TargetTriple), Saver(Alloc),
       CheckInputsExist(true), ProbePrecompiled(true),
       SuppressMissingInputWarning(false) {
   // Provide a sane fallback if no VFS is specified.
   if (!this->VFS)
     this->VFS = llvm::vfs::getRealFileSystem();
 
   Name = std::string(llvm::sys::path::filename(ClangExecutable));

clang/lib/Driver/ToolChains/Clang.cpp

 #include "Arch/X86.h"
 #include "CommonArgs.h"
 #include "Hexagon.h"
 #include "MSP430.h"
 #include "PS4CPU.h"
 #include "clang/Basic/CLWarnings.h"
 #include "clang/Basic/CharInfo.h"
 #include "clang/Basic/CodeGenOptions.h"
+#include "clang/Basic/HeaderInclude.h"
 #include "clang/Basic/LangOptions.h"
 #include "clang/Basic/MakeSupport.h"
 #include "clang/Basic/ObjCRuntime.h"
 #include "clang/Basic/Version.h"
 #include "clang/Config/config.h"
 #include "clang/Driver/Action.h"
 #include "clang/Driver/Distro.h"
 #include "clang/Driver/DriverDiagnostic.h"
#endif
   Args.AddAllArgs(CmdArgs, options::OPT_v);
 
   if (Args.getLastArg(options::OPT_H)) {
     CmdArgs.push_back("-H");
     CmdArgs.push_back("-sys-header-deps");
   }
   Args.AddAllArgs(CmdArgs, options::OPT_fshow_skipped_includes);
 
-  if (D.CCPrintHeaders && !D.CCGenDiagnostics) {
+  if (D.CCPrintHeadersFormat && !D.CCGenDiagnostics) {
     CmdArgs.push_back("-header-include-file");
     CmdArgs.push_back(!D.CCPrintHeadersFilename.empty()
                           ? D.CCPrintHeadersFilename.c_str()
                           : "-");
     CmdArgs.push_back("-sys-header-deps");
+    CmdArgs.push_back(Args.MakeArgString(
+        "-header-include-format=" +
+        std::string(headerIncludeFormatKindToString(D.CCPrintHeadersFormat))));
+    CmdArgs.push_back(
+        Args.MakeArgString("-header-include-filtering=" +
+                           std::string(headerIncludeFilteringKindToString(
+                               D.CCPrintHeadersFiltering))));
   }
   Args.AddLastArg(CmdArgs, options::OPT_P);
   Args.AddLastArg(CmdArgs, options::OPT_print_ivar_layout);
 
   if (D.CCLogDiagnostics && !D.CCGenDiagnostics) {
     CmdArgs.push_back("-diagnostic-log-file");
     CmdArgs.push_back(!D.CCLogDiagnosticsFilename.empty()
                           ? D.CCLogDiagnosticsFilename.c_str()

clang/lib/Frontend/CompilerInvocation.cpp

#undef DEPENDENCY_OUTPUT_OPTION_WITH_MARSHALLING
 
   // Only the -fmodule-file=<file> form.
   for (const auto *A : Args.filtered(OPT_fmodule_file)) {
     StringRef Val = A->getValue();
     if (!Val.contains('='))
       Opts.ExtraDeps.emplace_back(std::string(Val), EDK_ModuleFile);
   }
 
+  // Check for invalid combinations of header-include-format
+  // and header-include-filtering.
+  if ((Opts.HeaderIncludeFormat == HIFMT_Textual &&
+       Opts.HeaderIncludeFiltering != HIFIL_None) ||
+      (Opts.HeaderIncludeFormat == HIFMT_JSON &&
+       Opts.HeaderIncludeFiltering != HIFIL_Only_Direct_System))
+    Diags.Report(diag::err_drv_print_header_env_var_combination_cc1)
+        << Args.getLastArg(OPT_header_include_format_EQ)->getValue()
+        << Args.getLastArg(OPT_header_include_filtering_EQ)->getValue();
+
   return Diags.getNumErrors() == NumErrorsBefore;
 }
 
 static bool parseShowColorsArgs(const ArgList &Args, bool DefaultColor) {
   // Color diagnostics default to auto ("on" if terminal supports) in the driver
   // but default to off in cc1, needing an explicit OPT_fdiagnostics_color.
   // Support both clang's -f[no-]color-diagnostics and gcc's
   // -f[no-]diagnostics-colors[=never|always|auto].

clang/lib/Frontend/HeaderIncludeGen.cpp

 //===-- HeaderIncludeGen.cpp - Generate Header Includes -------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Frontend/DependencyOutputOptions.h"
 #include "clang/Frontend/Utils.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/SmallString.h"
+#include "llvm/Support/JSON.h"
 #include "llvm/Support/raw_ostream.h"
 using namespace clang;
 
 namespace {
 class HeaderIncludesCallback : public PPCallbacks {
   SourceManager &SM;
   raw_ostream *OutputFile;
   const DependencyOutputOptions &DepOpts;
public:
 
   void FileChanged(SourceLocation Loc, FileChangeReason Reason,
                    SrcMgr::CharacteristicKind FileType,
                    FileID PrevFID) override;
 
   void FileSkipped(const FileEntryRef &SkippedFile, const Token &FilenameTok,
                    SrcMgr::CharacteristicKind FileType) override;
 };
+
+/// A callback for emitting header usage information to a file in JSON. Each
+/// line in the file is a JSON object that includes the source file name and
+/// the list of headers directly or indirectly included from it. For example:
+///
+/// {"source":"/tmp/foo.c",
+///  "includes":["/usr/include/stdio.h", "/usr/include/stdlib.h"]}
+///
+/// To reduce the amount of data written to the file, we only record system
+/// headers that are directly included from a file that isn't in the system
+/// directory.
+class HeaderIncludesJSONCallback : public PPCallbacks {
+  SourceManager &SM;
+  raw_ostream *OutputFile;
+  bool OwnsOutputFile;
+  SmallVector<std::string, 16> IncludedHeaders;
+
+public:
+  HeaderIncludesJSONCallback(const Preprocessor *PP, raw_ostream *OutputFile_,
+                             bool OwnsOutputFile_)
+      : SM(PP->getSourceManager()), OutputFile(OutputFile_),
+        OwnsOutputFile(OwnsOutputFile_) {}
+
+  ~HeaderIncludesJSONCallback() override {
+    if (OwnsOutputFile)
+      delete OutputFile;
+  }
+
+  void EndOfMainFile() override;
+
+  void FileChanged(SourceLocation Loc, FileChangeReason Reason,
+                   SrcMgr::CharacteristicKind FileType,
+                   FileID PrevFID) override;
+
+  void FileSkipped(const FileEntryRef &SkippedFile, const Token &FilenameTok,
+                   SrcMgr::CharacteristicKind FileType) override;
+};
 }
 
 static void PrintHeaderInfo(raw_ostream *OutputFile, StringRef Filename,
                             bool ShowDepth, unsigned CurrentIncludeDepth,
                             bool MSStyle) {
   // Write to a temporary string to avoid unnecessary flushing on errs().
   SmallString<512> Pathname(Filename);
   if (!MSStyle)
if (EC) {
       delete OS;
     } else {
       OS->SetUnbuffered();
       OutputFile = OS;
       OwnsOutputFile = true;
     }
   }
 
+  switch (DepOpts.HeaderIncludeFormat) {
+  case HIFMT_None:
+    llvm_unreachable("unexpected header format kind");
+  case HIFMT_Textual: {
+    assert(DepOpts.HeaderIncludeFiltering == HIFIL_None &&
+           "header filtering is currently always disabled when output format is"
+           "textual");
     // Print header info for extra headers, pretending they were discovered by
     // the regular preprocessor. The primary use case is to support proper
     // generation of Make / Ninja file dependencies for implicit includes, such
     // as sanitizer ignorelists. It's only important for cl.exe compatibility,
     // the GNU way to generate rules is -M / -MM / -MD / -MMD.
     for (const auto &Header : DepOpts.ExtraDeps)
       PrintHeaderInfo(OutputFile, Header.first, ShowDepth, 2, MSStyle);
     PP.addPPCallbacks(std::make_unique<HeaderIncludesCallback>(
         &PP, ShowAllHeaders, OutputFile, DepOpts, OwnsOutputFile, ShowDepth,
         MSStyle));
+    break;
+  }
+  case HIFMT_JSON: {
+    assert(DepOpts.HeaderIncludeFiltering == HIFIL_Only_Direct_System &&
+           "only-direct-system is the only option for filtering");
+    PP.addPPCallbacks(std::make_unique<HeaderIncludesJSONCallback>(
+        &PP, OutputFile, OwnsOutputFile));
+    break;
+  }
+  }
 }
 
 void HeaderIncludesCallback::FileChanged(SourceLocation Loc,
                                          FileChangeReason Reason,
                                          SrcMgr::CharacteristicKind NewFileType,
                                          FileID PrevFID) {
   // Unless we are exiting a #include, make sure to skip ahead to the line the
   // #include directive was at.
if (!DepOpts.ShowSkippedHeaderIncludes)
     return;
 
   if (!DepOpts.IncludeSystemHeaders && isSystem(FileType))
     return;
 
   PrintHeaderInfo(OutputFile, SkippedFile.getName(), ShowDepth,
                   CurrentIncludeDepth + 1, MSStyle);
 }
+
+void HeaderIncludesJSONCallback::EndOfMainFile() {
+  const FileEntry *FE = SM.getFileEntryForID(SM.getMainFileID());
+  SmallString<256> MainFile(FE->getName());
+  SM.getFileManager().makeAbsolutePath(MainFile);
+
+  std::string Str;
+  llvm::raw_string_ostream OS(Str);
+  llvm::json::OStream JOS(OS);
+  JOS.object([&] {
+    JOS.attribute("source", MainFile.c_str());
+    JOS.attributeArray("includes", [&] {
+      llvm::StringSet<> SeenHeaders;
+      for (const std::string &H : IncludedHeaders)
+        if (SeenHeaders.insert(H).second)
+          JOS.value(H);
+    });
+  });
+  OS << "\n";
+
+  if (OutputFile->get_kind() == raw_ostream::OStreamKind::OK_FDStream) {
+    llvm::raw_fd_ostream *FDS = static_cast<llvm::raw_fd_ostream *>(OutputFile);
+    if (auto L = FDS->lock())
+      *OutputFile << Str;
+  } else
+    *OutputFile << Str;
+}
+
+/// Determine whether the header file should be recorded. The header file should
+/// be recorded only if the header file is a system header and the current file
+/// isn't a system header.
+static bool shouldRecordNewFile(SrcMgr::CharacteristicKind NewFileType,
+                                SourceLocation PrevLoc, SourceManager &SM) {
+  return SrcMgr::isSystem(NewFileType) && !SM.isInSystemHeader(PrevLoc);
+}
+
+void HeaderIncludesJSONCallback::FileChanged(
+    SourceLocation Loc, FileChangeReason Reason,
+    SrcMgr::CharacteristicKind NewFileType, FileID PrevFID) {
+  if (!shouldRecordNewFile(NewFileType, SM.getLocForStartOfFile(PrevFID), SM))
+    return;
+
+  // Unless we are exiting a #include, make sure to skip ahead to the line the
+  // #include directive was at.
+  PresumedLoc UserLoc = SM.getPresumedLoc(Loc);
+  if (UserLoc.isInvalid())
+    return;
+
+  if (Reason == PPCallbacks::EnterFile &&
+      UserLoc.getFilename() != StringRef("<command line>"))
+    IncludedHeaders.push_back(UserLoc.getFilename());
+}
+
+void HeaderIncludesJSONCallback::FileSkipped(
+    const FileEntryRef &SkippedFile, const Token &FilenameTok,
+    SrcMgr::CharacteristicKind FileType) {
+  if (!shouldRecordNewFile(FileType, FilenameTok.getLocation(), SM))
+    return;
+
+  IncludedHeaders.push_back(SkippedFile.getName().str());
+}

clang/test/Preprocessor/Inputs/print-header-json/header0.h

+#include "system3.h"
+#include "header1.h"
+#include "header2.h"

clang/test/Preprocessor/Inputs/print-header-json/system/system0.h

+#include "system1.h"
+#include "system2.h"

clang/test/Preprocessor/print-header-json.c

+// RUN: %clang_cc1 -E -header-include-format=json -header-include-filtering=only-direct-system -header-include-file %t.txt -I %S/Inputs/print-header-json -isystem %S/Inputs/print-header-json/system %s
+// RUN: cat %t.txt | FileCheck %s --check-prefix=SUPPORTED
+// RUN: not %clang_cc1 -E -header-include-format=textual -header-include-filtering=only-direct-system -header-include-file %t.txt -I %S/Inputs/print-header-json -isystem %S/Inputs/print-header-json/system %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=UNSUPPORTED0
+// RUN: not %clang_cc1 -E -header-include-format=json -header-include-filtering=none -header-include-file %t.txt -I %S/Inputs/print-header-json -isystem %S/Inputs/print-header-json/system %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=UNSUPPORTED1
+// RUN: rm %t.txt
+// RUN: env CC_PRINT_HEADERS_FORMAT=json CC_PRINT_HEADERS_FILTERING=only-direct-system CC_PRINT_HEADERS_FILE=%t.txt %clang -fsyntax-only -I %S/Inputs/print-header-json -isystem %S/Inputs/print-header-json/system %s -o /dev/null
+// RUN: env CC_PRINT_HEADERS_FORMAT=textual CC_PRINT_HEADERS_FILTERING=only-direct-system CC_PRINT_HEADERS_FILE=%t.txt not %clang -fsyntax-only -I %S/Inputs/print-header-json -isystem %S/Inputs/print-header-json/system %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=UNSUPPORTED2
+// RUN: env CC_PRINT_HEADERS_FORMAT=json CC_PRINT_HEADERS_FILTERING=none CC_PRINT_HEADERS_FILE=%t.txt not %clang -fsyntax-only -I %S/Inputs/print-header-json -isystem %S/Inputs/print-header-json/system %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=UNSUPPORTED3
+// RUN: cat %t.txt | FileCheck %s --check-prefix=SUPPORTED
+
+#include "system0.h"
+#include "header0.h"
+#include "system2.h"
+
+// SUPPORTED: {"source":"{{[^,]*}}/print-header-json.c","includes":["{{[^,]*}}system0.h","{{[^,]*}}system3.h","{{[^,]*}}system2.h"]}
+
+// UNSUPPORTED0: error: unsupported combination: -header-include-format=textual and -header-include-filtering=only-direct-system
+// UNSUPPORTED1: error: unsupported combination: -header-include-format=json and -header-include-filtering=none
+// UNSUPPORTED2: error: unsupported combination: CC_PRINT_HEADERS_FORMAT=textual and CC_PRINT_HEADERS_FILTERING=only-direct-system
+// UNSUPPORTED3: error: unsupported combination: CC_PRINT_HEADERS_FORMAT=json and CC_PRINT_HEADERS_FILTERING=none

clang/tools/driver/driver.cpp

 //===-- driver.cpp - Clang GCC-Compatible Driver --------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This is the entry point to the clang driver; it is a thin wrapper
 // for functionality in the Driver clang library.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Driver/Driver.h"
 #include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/HeaderInclude.h"
 #include "clang/Basic/Stack.h"
 #include "clang/Config/config.h"
 #include "clang/Driver/Compilation.h"
 #include "clang/Driver/DriverDiagnostic.h"
 #include "clang/Driver/Options.h"
 #include "clang/Driver/ToolChain.h"
 #include "clang/Frontend/ChainedDiagnosticConsumer.h"
 #include "clang/Frontend/CompilerInvocation.h"
static void getCLEnvVarOptions(std::string &EnvValue, llvm::StringSaver &Saver,
                                SmallVectorImpl<const char *> &Opts) {
   llvm::cl::TokenizeWindowsCommandLine(EnvValue, Saver, Opts);
   // The first instance of '#' should be replaced with '=' in each option.
   for (const char *Opt : Opts)
     if (char *NumberSignPtr = const_cast<char *>(::strchr(Opt, '#')))
       *NumberSignPtr = '=';
 }
 
-static void SetBackdoorDriverOutputsFromEnvVars(Driver &TheDriver) {
-  auto CheckEnvVar = [](const char *EnvOptSet, const char *EnvOptFile,
-                        std::string &OptFile) {
-    bool OptSet = !!::getenv(EnvOptSet);
-    if (OptSet) {
-      if (const char *Var = ::getenv(EnvOptFile))
-        OptFile = Var;
-    }
-    return OptSet;
-  };
+template <class T>
+static T checkEnvVar(const char *EnvOptSet, const char *EnvOptFile,
+  T OptVal = ::getenv(EnvOptSet);
+  if (OptVal) {
+  return OptVal;
+}
 
+static bool SetBackdoorDriverOutputsFromEnvVars(Driver &TheDriver) {
   TheDriver.CCPrintOptions =
-      CheckEnvVar("CC_PRINT_OPTIONS", "CC_PRINT_OPTIONS_FILE",
+      checkEnvVar<bool>("CC_PRINT_OPTIONS", "CC_PRINT_OPTIONS_FILE",
                         TheDriver.CCPrintOptionsFilename);
-  TheDriver.CCPrintHeaders =
-      CheckEnvVar("CC_PRINT_HEADERS", "CC_PRINT_HEADERS_FILE",
-                  TheDriver.CCPrintHeadersFilename);
+  if (checkEnvVar<bool>("CC_PRINT_HEADERS", "CC_PRINT_HEADERS_FILE",
+                        TheDriver.CCPrintHeadersFilename)) {
+    TheDriver.CCPrintHeadersFormat = HIFMT_Textual;
+    TheDriver.CCPrintHeadersFiltering = HIFIL_None;
+  } else if (const char *EnvVar = checkEnvVar<const char *>(
+                 "CC_PRINT_HEADERS_FORMAT", "CC_PRINT_HEADERS_FILE",
+                 TheDriver.CCPrintHeadersFilename)) {
+    TheDriver.CCPrintHeadersFormat = stringToHeaderIncludeFormatKind(EnvVar);
+    if (!TheDriver.CCPrintHeadersFormat) {
+      TheDriver.Diag(clang::diag::err_drv_print_header_env_var) << 0 << EnvVar;
+      return false;
+    }
+
+    const char *FilteringStr = ::getenv("CC_PRINT_HEADERS_FILTERING");
+    HeaderIncludeFilteringKind Filtering;
+    if (!stringToHeaderIncludeFiltering(FilteringStr, Filtering)) {
+      TheDriver.Diag(clang::diag::err_drv_print_header_env_var)
+          << 1 << FilteringStr;
+      return false;
+    }
+
+    if ((TheDriver.CCPrintHeadersFormat == HIFMT_Textual &&
+         Filtering != HIFIL_None) ||
+        (TheDriver.CCPrintHeadersFormat == HIFMT_JSON &&
+         Filtering != HIFIL_Only_Direct_System)) {
+      TheDriver.Diag(clang::diag::err_drv_print_header_env_var_combination)
+          << EnvVar << FilteringStr;
+      return false;
+    }
+    TheDriver.CCPrintHeadersFiltering = Filtering;
+  }
+
   TheDriver.CCLogDiagnostics =
-      CheckEnvVar("CC_LOG_DIAGNOSTICS", "CC_LOG_DIAGNOSTICS_FILE",
+      checkEnvVar<bool>("CC_LOG_DIAGNOSTICS", "CC_LOG_DIAGNOSTICS_FILE",
                         TheDriver.CCLogDiagnosticsFilename);
   TheDriver.CCPrintProcessStats =
-      CheckEnvVar("CC_PRINT_PROC_STAT", "CC_PRINT_PROC_STAT_FILE",
+      checkEnvVar<bool>("CC_PRINT_PROC_STAT", "CC_PRINT_PROC_STAT_FILE",
                         TheDriver.CCPrintStatReportFilename);
+
+  return true;
 }
 
 static void FixupDiagPrefixExeName(TextDiagnosticPrinter *DiagClient,
                                    const std::string &Path) {
   // If the clang binary happens to be named cl.exe for compatibility reasons,
   // use clang-cl.exe as the prefix to avoid confusion between clang and MSVC.
   StringRef ExeBasename(llvm::sys::path::stem(Path));
   if (ExeBasename.equals_insensitive("cl"))
int clang_main(int Argc, char **Argv) {
 
   Driver TheDriver(Path, llvm::sys::getDefaultTargetTriple(), Diags);
   SetInstallDir(Args, TheDriver, CanonicalPrefixes);
   auto TargetAndMode = ToolChain::getTargetAndModeFromProgramName(Args[0]);
   TheDriver.setTargetAndMode(TargetAndMode);
 
   insertTargetAndModeArgs(TargetAndMode, Args, SavedStrings);
 
-  SetBackdoorDriverOutputsFromEnvVars(TheDriver);
+  if (!SetBackdoorDriverOutputsFromEnvVars(TheDriver))
+    return 1;
 
   if (!UseNewCC1Process) {
     TheDriver.CC1Main = &ExecuteCC1Tool;
     // Ensure the CC1Command actually catches cc1 crashes
     llvm::CrashRecoveryContext::Enable();
   }
 
   std::unique_ptr<Compilation> C(TheDriver.BuildCompilation(Args));