[libclang] Allow skipping warnings from all included files

[nik]

Summary

Depending on the included files and the used warning flags, e.g. -
Weverything, a huge number of warnings can be reported for included
files. As processing that many diagnostics comes with a performance
impact and not all clients are interested in those diagnostics, add a
flag to skip them.

[yvvan]

Cool! That's actually quite a small change with big outcome!

include/clang/Basic/Diagnostic.h

216 ↗

(On Diff #151107)

Probably mention that it also suppresses everything with lower priority?

lib/Basic/DiagnosticIDs.cpp

484 ↗

(On Diff #151107)

{ } around return

[nik]

Addressed comments.

[yvvan]

@ilya-biryukov what do you think?

[ilya-biryukov]

Sorry for the late response, was on vacation.

Have you considered doing the same filtering in ASTUnit's StoredDiagnosticConsumer? It should not be more difficult and allows to avoid changing the clang's diagnostic interfaces. That's what we do in clangd.

I wonder if you want to handle notes and remarks in a special manner? They can be seen as part of the original diagnostic, rather than the separate diagnostic. E.g. showing a note in the main file, but not showing the diagnostic from the headers file that this note comes from, might be confusing to the users.
Maybe also add tests for diagnostics in the main file with notes/remarks in the header files and vice versa?

[yvvan]

The idea was to ignore everything including notes and remarks so that only errors from system headers are still collected.

Changing consumer might be a possible way to go as well but it requires changing (or wrapping) all consumers that we need to be affected. The advantage of this change that it also covers consumers that come from plugins that we use (tidy and clazy).

[yvvan]

But this one misses a way to set this flag for everything except libclang. We can probably change that (Nikolai is in vacation till the end of summer so it's probably my part now) and add some tests outside of Index for it (probably Frontend)

[nik]

In D48116#1144732, @ilya-biryukov wrote:

Have you considered doing the same filtering in ASTUnit's StoredDiagnosticConsumer? It should not be more difficult and allows to avoid changing the clang's diagnostic interfaces. That's what we do in clangd.

Will check.

I wonder if you want to handle notes and remarks in a special manner? They can be seen as part of the original diagnostic, rather than the separate diagnostic. E.g. showing a note in the main file, but not showing the diagnostic from the headers file that this note comes from, might be confusing to the users.

Looks like this case is handled fine.

WIthout the new option, a note is printed as expected/always (shortened output):

$ bin/c-index-test -test-load-source function ignore-warnings-from-headers.cpp -Wunused-parameter
ignore-warnings-from-headers.h:1:12: warning: unused parameter 'unusedInHeader' [-Wunused-parameter]
ignore-warnings-from-headers.cpp:1:10: note: in file included from ignore-warnings-from-headers.cpp:1:
ignore-warnings-from-headers.cpp:3:12: warning: unused parameter 'unusedInMainFile' [-Wunused-parameter]

With the new option, note is not printed and thus no confusion should arise:

$ CINDEXTEST_IGNORE_NONERRORS_FROM_INCLUDED_FILES=1 bin/c-index-test -test-load-source function ignore-warnings-from-headers.cpp -Wunused-parameter
ignore-warnings-from-headers.cpp:3:12: warning: unused parameter 'unusedInMainFile' [-Wunused-parameter]

I will add "// CHECK-NOT: note: in file included from" to the test.

Maybe also add tests for diagnostics in the main file with notes/remarks in the header files and vice versa?

"Vice versa" is covered as shown above. If the diagnostic is in main file and a note of that one refers to the header, then the note should be shown/included. This case seems also fine - I've added a test for this.

In D48116#1144838, @yvvan wrote:

But this one misses a way to set this flag for everything except libclang. We can probably change that (Nikolai is in vacation till the end of summer so it's probably my part now) and add some tests outside of Index for it (probably Frontend)

What's the use case of this?

[nik]

In D48116#1144732, @ilya-biryukov wrote:

Have you considered doing the same filtering in ASTUnit's StoredDiagnosticConsumer? It should not be more difficult and allows to avoid changing the clang's diagnostic interfaces. That's what we do in clangd.

Hmm, it's a bit strange that StoredDiagnosticConsumer should also start to filter things.

For filtering in StoredDiagnosticConsumer one needs to pass the new bool everywhere along where "bool CaptureDiagnostics" is already passed on (to end up in the StoredDiagnosticConsumer constructor) . Also, ASTUnit::CaptureDiagnostics is then not enough anymore since the new bool is also needed in getMainBufferWithPrecompiledPreamble(). One could also (2) convert "bool CaptureDiagnostics" to an enum with enumerators like CaptureNothing, CaptureAll, CaptureAllWithoutNonErrorsFromIncludes to make this a bit less invasive.

If changing clang's diagnostic interface should be avoided, I tend to go with (2). Ilya?

$ git grep "bool CaptureDiagnostics"
include/clang/Frontend/ASTUnit.h:  bool CaptureDiagnostics = false;
include/clang/Frontend/ASTUnit.h:                             ASTUnit &AST, bool CaptureDiagnostics,
include/clang/Frontend/ASTUnit.h:         IntrusiveRefCntPtr<DiagnosticsEngine> Diags, bool CaptureDiagnostics,
include/clang/Frontend/ASTUnit.h:      bool CaptureDiagnostics = false, bool AllowPCHWithCompilerErrors = false,
include/clang/Frontend/ASTUnit.h:      bool OnlyLocalDecls = false, bool CaptureDiagnostics = false,
include/clang/Frontend/ASTUnit.h:      bool OnlyLocalDecls = false, bool CaptureDiagnostics = false,
include/clang/Frontend/ASTUnit.h:      bool OnlyLocalDecls = false, bool CaptureDiagnostics = false,
lib/Frontend/ASTUnit.cpp:                             ASTUnit &AST, bool CaptureDiagnostics,
lib/Frontend/ASTUnit.cpp:    bool CaptureDiagnostics, bool AllowPCHWithCompilerErrors,
lib/Frontend/ASTUnit.cpp:                bool CaptureDiagnostics, bool UserFilesAreVolatile) {
lib/Frontend/ASTUnit.cpp:    bool OnlyLocalDecls, bool CaptureDiagnostics,
lib/Frontend/ASTUnit.cpp:    bool OnlyLocalDecls, bool CaptureDiagnostics,
lib/Frontend/ASTUnit.cpp:    bool OnlyLocalDecls, bool CaptureDiagnostics,
tools/libclang/Indexing.cpp:  bool CaptureDiagnostics = !Logger::isLoggingEnabled();

[ilya-biryukov]

In D48116#1322878, @nik wrote:

In D48116#1144732, @ilya-biryukov wrote:

Have you considered doing the same filtering in ASTUnit's StoredDiagnosticConsumer? It should not be more difficult and allows to avoid changing the clang's diagnostic interfaces. That's what we do in clangd.

Hmm, it's a bit strange that StoredDiagnosticConsumer should also start to filter things.

I

For filtering in StoredDiagnosticConsumer one needs to pass the new bool everywhere along where "bool CaptureDiagnostics" is already passed on (to end up in the StoredDiagnosticConsumer constructor) . Also, ASTUnit::CaptureDiagnostics is then not enough anymore since the new bool is also needed in getMainBufferWithPrecompiledPreamble(). One could also (2) convert "bool CaptureDiagnostics" to an enum with enumerators like CaptureNothing, CaptureAll, CaptureAllWithoutNonErrorsFromIncludes to make this a bit less invasive.
If changing clang's diagnostic interface should be avoided, I tend to go with (2). Ilya?

Yeah, LG. The changes in the ASTUnit look strictly better than changes in clang - the latter seems to already provide enough to do the filtering.
If you avoid changing the StoredDiagnosticConsumer (or writing a filtering wrapper for DiagnosticConsumer), you'll end up having some diagnostics inside headers generated after preamble was built, right?
That might be either ok or not, depending on the use-case.

[nik]

For filtering in StoredDiagnosticConsumer one needs to pass the new bool everywhere along where "bool CaptureDiagnostics" is already passed on (to end up in the StoredDiagnosticConsumer constructor) . Also, ASTUnit::CaptureDiagnostics is then not enough anymore since the new bool is also needed in getMainBufferWithPrecompiledPreamble(). One could also (2) convert "bool CaptureDiagnostics" to an enum with enumerators like CaptureNothing, CaptureAll, CaptureAllWithoutNonErrorsFromIncludes to make this a bit less invasive.
If changing clang's diagnostic interface should be avoided, I tend to go with (2). Ilya?

Yeah, LG. The changes in the ASTUnit look strictly better than changes in clang - the latter seems to already provide enough to do the filtering.
If you avoid changing the StoredDiagnosticConsumer (or writing a filtering wrapper for DiagnosticConsumer), you'll end up having some diagnostics inside headers generated after preamble was built, right?

If there is some #include after the first declaration, possibly. Why is that relevant?

[nik]

OK, filtering happens now in FilterAndStoreDiagnosticConsumer, the former
StoredDiagnosticConsumer.

[ilya-biryukov]

This LGTM wrt to layering, i.e. it's nice we don't need to change the code of diagnostics.
It's been a while since I've looked at the ASTUnit code, though, would be good if someone else took an extra look at it.

Added a few nits too.

lib/Frontend/ASTUnit.cpp

682

isWrittenInMainFile might be a better fit: it does not look at presumed locations. That would be the expected behavior in the most common case, i.e. showing errors in an IDE or a text editor.

694

Why do we need this extra check? We checked for StoredDiags and Standalone diags in the body of the statement later again.

[nik]

Ping.

[nik]

lib/Frontend/ASTUnit.cpp

682

Oh, good catch! Thanks! :)

694

Ops, looks like a left-over from a previous version, removed.

[nik]

Addressed inline comments.

[nik]

Sorry for the pointless ping, haven't seen the inline comments. They are addressed now.

I've also increased CINDEX_VERSION_MINOR so clients can detect availability of this new flag.

It's been a while since I've looked at the ASTUnit code, though, would be good if someone else took an extra look at it.

Benjamin? Argyrios? Would you be so kind? :)

[nik]

Jan?

[nik]

Ping.

[nik]

Ping.

[ilya-biryukov]

@jkorous, could you take a look or suggest someone who can review the libclang changes?
As mentioned before, I've reviewed the clang bits (which are now gone), but don't have much experience in libclang parts.

[yvvan]

libclang part is quite small here and looks ok. I would just accept it

include/clang-c/Index.h

  * The version constants for the libclang API.
  * CINDEX_VERSION_MINOR should increase when there are API additions.
  * CINDEX_VERSION_MAJOR is intended for "major" source/ABI breaking changes.
  *
  * The policy about the libclang API was always to keep it source and ABI
  * compatible, thus CINDEX_VERSION_MAJOR is expected to remain stable.
  */
 #define CINDEX_VERSION_MAJOR 0
-#define CINDEX_VERSION_MINOR 57
+#define CINDEX_VERSION_MINOR 58
 
 #define CINDEX_VERSION_ENCODE(major, minor) ( \
       ((major) * 10000)                       \
     + ((minor) *     1))
 
 #define CINDEX_VERSION CINDEX_VERSION_ENCODE( \
     CINDEX_VERSION_MAJOR,                     \
     CINDEX_VERSION_MINOR )
enum CXTranslationUnit_Flags {
   /**
    * Used to indicate that attributed types should be included in CXType.
    */
   CXTranslationUnit_IncludeAttributedTypes = 0x1000,
 
   /**
    * Used to indicate that implicit attributes should be visited.
    */
-  CXTranslationUnit_VisitImplicitAttributes = 0x2000
+  CXTranslationUnit_VisitImplicitAttributes = 0x2000,
+
+  /**
+   * Used to indicate that non-errors from included files should be ignored.
+   *
+   * If set, clang_getDiagnosticSetFromTU() will not report e.g. warnings from
+   * included files anymore. This speeds up clang_getDiagnosticSetFromTU() for
+   * the case where these warnings are not of interest, as for an IDE for
+   * example, which typically shows only the diagnostics in the main file.
+   */
+  CXTranslationUnit_IgnoreNonErrorsFromIncludedFiles = 0x4000
 };
 
 /**
  * Returns the set of flags that is suitable for parsing a translation
  * unit that is being edited.
  *
  * The set of flags returned provide options for \c clang_parseTranslationUnit()
  * to indicate that the translation unit is likely to be reparsed many times,

include/clang/Frontend/ASTUnit.h

 class Preprocessor;
 class PreprocessorOptions;
 class Sema;
 class TargetInfo;
 
 /// \brief Enumerates the available scopes for skipping function bodies.
 enum class SkipFunctionBodiesScope { None, Preamble, PreambleAndMainFile };
 
+/// \brief Enumerates the available kinds for capturing diagnostics.
+enum class CaptureDiagsKind { None, All, AllWithoutNonErrorsFromIncludes };
+
 /// Utility class for loading a ASTContext from an AST file.
 class ASTUnit {
 public:
   struct StandaloneFixIt {
     std::pair<unsigned, unsigned> RemoveRange;
     std::pair<unsigned, unsigned> InsertFromRange;
     std::string CodeToInsert;
     bool BeforePreviousInsertions;
private:
   TrivialModuleLoader ModuleLoader;
 
   // OnlyLocalDecls - when true, walking this AST should only visit declarations
   // that come from the AST itself, not from included precompiled headers.
   // FIXME: This is temporary; eventually, CIndex will always do this.
   bool OnlyLocalDecls = false;
 
   /// Whether to capture any diagnostics produced.
-  bool CaptureDiagnostics = false;
+  CaptureDiagsKind CaptureDiagnostics = CaptureDiagsKind::None;
 
   /// Track whether the main file was loaded from an AST or not.
   bool MainFileIsAST;
 
   /// What kind of translation unit this AST represents.
   TranslationUnitKind TUKind = TU_Complete;
 
   /// Whether we should time each operation.
private:
   /// completions cached.
   bool IncludeBriefCommentsInCodeCompletion : 1;
 
   /// True if non-system source files should be treated as volatile
   /// (likely to change while trying to use them).
   bool UserFilesAreVolatile : 1;
 
   static void ConfigureDiags(IntrusiveRefCntPtr<DiagnosticsEngine> Diags,
-                             ASTUnit &AST, bool CaptureDiagnostics);
+                             ASTUnit &AST, CaptureDiagsKind CaptureDiagnostics);
 
   void TranslateStoredDiagnostics(FileManager &FileMgr,
                                   SourceManager &SrcMan,
                       const SmallVectorImpl<StandaloneDiagnostic> &Diags,
                             SmallVectorImpl<StoredDiagnostic> &Out);
 
   void clearFileLevelDecls();
 
public:
 
   /// A mapping from a file name to the memory buffer that stores the
   /// remapped contents of that file.
   using RemappedFile = std::pair<std::string, llvm::MemoryBuffer *>;
 
   /// Create a ASTUnit. Gets ownership of the passed CompilerInvocation.
   static std::unique_ptr<ASTUnit>
   create(std::shared_ptr<CompilerInvocation> CI,
-         IntrusiveRefCntPtr<DiagnosticsEngine> Diags, bool CaptureDiagnostics,
-         bool UserFilesAreVolatile);
+         IntrusiveRefCntPtr<DiagnosticsEngine> Diags,
+         CaptureDiagsKind CaptureDiagnostics, bool UserFilesAreVolatile);
 
   enum WhatToLoad {
     /// Load options and the preprocessor state.
     LoadPreprocessorOnly,
 
     /// Load the AST, but do not restore Sema state.
     LoadASTOnly,
 
public:
   /// lifetime is expected to extend past that of the returned ASTUnit.
   ///
   /// \returns - The initialized ASTUnit or null if the AST failed to load.
   static std::unique_ptr<ASTUnit> LoadFromASTFile(
       const std::string &Filename, const PCHContainerReader &PCHContainerRdr,
       WhatToLoad ToLoad, IntrusiveRefCntPtr<DiagnosticsEngine> Diags,
       const FileSystemOptions &FileSystemOpts, bool UseDebugInfo = false,
       bool OnlyLocalDecls = false, ArrayRef<RemappedFile> RemappedFiles = None,
-      bool CaptureDiagnostics = false, bool AllowPCHWithCompilerErrors = false,
+      CaptureDiagsKind CaptureDiagnostics = CaptureDiagsKind::None,
+      bool AllowPCHWithCompilerErrors = false,
       bool UserFilesAreVolatile = false);
 
 private:
   /// Helper function for \c LoadFromCompilerInvocation() and
   /// \c LoadFromCommandLine(), which loads an AST from a compiler invocation.
   ///
   /// \param PrecompilePreambleAfterNParses After how many parses the preamble
   /// of this translation unit should be precompiled, to improve the performance
public:
   /// created ASTUnit was passed in \p Unit then the caller can check that.
   ///
   static ASTUnit *LoadFromCompilerInvocationAction(
       std::shared_ptr<CompilerInvocation> CI,
       std::shared_ptr<PCHContainerOperations> PCHContainerOps,
       IntrusiveRefCntPtr<DiagnosticsEngine> Diags,
       FrontendAction *Action = nullptr, ASTUnit *Unit = nullptr,
       bool Persistent = true, StringRef ResourceFilesPath = StringRef(),
-      bool OnlyLocalDecls = false, bool CaptureDiagnostics = false,
+      bool OnlyLocalDecls = false,
+      CaptureDiagsKind CaptureDiagnostics = CaptureDiagsKind::None,
       unsigned PrecompilePreambleAfterNParses = 0,
       bool CacheCodeCompletionResults = false,
       bool IncludeBriefCommentsInCodeCompletion = false,
       bool UserFilesAreVolatile = false,
       std::unique_ptr<ASTUnit> *ErrAST = nullptr);
 
   /// LoadFromCompilerInvocation - Create an ASTUnit from a source file, via a
   /// CompilerInvocation object.
   ///
   /// \param CI - The compiler invocation to use; it must have exactly one input
   /// source file. The ASTUnit takes ownership of the CompilerInvocation object.
   ///
   /// \param PCHContainerOps - The PCHContainerOperations to use for loading and
   /// creating modules.
   ///
   /// \param Diags - The diagnostics engine to use for reporting errors; its
   /// lifetime is expected to extend past that of the returned ASTUnit.
   //
   // FIXME: Move OnlyLocalDecls, UseBumpAllocator to setters on the ASTUnit, we
   // shouldn't need to specify them at construction time.
   static std::unique_ptr<ASTUnit> LoadFromCompilerInvocation(
       std::shared_ptr<CompilerInvocation> CI,
       std::shared_ptr<PCHContainerOperations> PCHContainerOps,
       IntrusiveRefCntPtr<DiagnosticsEngine> Diags, FileManager *FileMgr,
-      bool OnlyLocalDecls = false, bool CaptureDiagnostics = false,
+      bool OnlyLocalDecls = false,
+      CaptureDiagsKind CaptureDiagnostics = CaptureDiagsKind::None,
       unsigned PrecompilePreambleAfterNParses = 0,
       TranslationUnitKind TUKind = TU_Complete,
       bool CacheCodeCompletionResults = false,
       bool IncludeBriefCommentsInCodeCompletion = false,
       bool UserFilesAreVolatile = false);
 
   /// LoadFromCommandLine - Create an ASTUnit from a vector of command line
   /// arguments, which must specify exactly one source file.
public:
   /// if \p VFS is nullptr.
   ///
   // FIXME: Move OnlyLocalDecls, UseBumpAllocator to setters on the ASTUnit, we
   // shouldn't need to specify them at construction time.
   static ASTUnit *LoadFromCommandLine(
       const char **ArgBegin, const char **ArgEnd,
       std::shared_ptr<PCHContainerOperations> PCHContainerOps,
       IntrusiveRefCntPtr<DiagnosticsEngine> Diags, StringRef ResourceFilesPath,
-      bool OnlyLocalDecls = false, bool CaptureDiagnostics = false,
+      bool OnlyLocalDecls = false,
+      CaptureDiagsKind CaptureDiagnostics = CaptureDiagsKind::None,
       ArrayRef<RemappedFile> RemappedFiles = None,
       bool RemappedFilesKeepOriginalName = true,
       unsigned PrecompilePreambleAfterNParses = 0,
       TranslationUnitKind TUKind = TU_Complete,
       bool CacheCodeCompletionResults = false,
       bool IncludeBriefCommentsInCodeCompletion = false,
       bool AllowPCHWithCompilerErrors = false,
       SkipFunctionBodiesScope SkipFunctionBodies =

lib/Frontend/ASTUnit.cpp

void updated() {
     // We didn't have access to the comment options when the ASTContext was
     // constructed, so register them now.
     Context->getCommentCommandTraits().registerCommentOptions(
         LangOpt.CommentOpts);
   }
 };
 
 /// Diagnostic consumer that saves each diagnostic it is given.
-class StoredDiagnosticConsumer : public DiagnosticConsumer {
+class FilterAndStoreDiagnosticConsumer : public DiagnosticConsumer {
   SmallVectorImpl<StoredDiagnostic> *StoredDiags;
   SmallVectorImpl<ASTUnit::StandaloneDiagnostic> *StandaloneDiags;
+  bool CaptureNonErrorsFromIncludes = true;
   const LangOptions *LangOpts = nullptr;
   SourceManager *SourceMgr = nullptr;
 
 public:
-  StoredDiagnosticConsumer(
+  FilterAndStoreDiagnosticConsumer(
       SmallVectorImpl<StoredDiagnostic> *StoredDiags,
-      SmallVectorImpl<ASTUnit::StandaloneDiagnostic> *StandaloneDiags)
-      : StoredDiags(StoredDiags), StandaloneDiags(StandaloneDiags) {
+      SmallVectorImpl<ASTUnit::StandaloneDiagnostic> *StandaloneDiags,
+      bool CaptureNonErrorsFromIncludes)
+      : StoredDiags(StoredDiags), StandaloneDiags(StandaloneDiags),
+        CaptureNonErrorsFromIncludes(CaptureNonErrorsFromIncludes) {
     assert((StoredDiags || StandaloneDiags) &&
            "No output collections were passed to StoredDiagnosticConsumer.");
   }
 
   void BeginSourceFile(const LangOptions &LangOpts,
                        const Preprocessor *PP = nullptr) override {
     this->LangOpts = &LangOpts;
     if (PP)
       SourceMgr = &PP->getSourceManager();
   }
 
   void HandleDiagnostic(DiagnosticsEngine::Level Level,
                         const Diagnostic &Info) override;
 };
 
-/// RAII object that optionally captures diagnostics, if
+/// RAII object that optionally captures and filters diagnostics, if
 /// there is no diagnostic client to capture them already.
 class CaptureDroppedDiagnostics {
   DiagnosticsEngine &Diags;
-  StoredDiagnosticConsumer Client;
+  FilterAndStoreDiagnosticConsumer Client;
   DiagnosticConsumer *PreviousClient = nullptr;
   std::unique_ptr<DiagnosticConsumer> OwningPreviousClient;
 
 public:
   CaptureDroppedDiagnostics(
-      bool RequestCapture, DiagnosticsEngine &Diags,
+      CaptureDiagsKind CaptureDiagnostics, DiagnosticsEngine &Diags,
       SmallVectorImpl<StoredDiagnostic> *StoredDiags,
       SmallVectorImpl<ASTUnit::StandaloneDiagnostic> *StandaloneDiags)
-      : Diags(Diags), Client(StoredDiags, StandaloneDiags) {
-    if (RequestCapture || Diags.getClient() == nullptr) {
+      : Diags(Diags),
+        Client(StoredDiags, StandaloneDiags,
+               CaptureDiagnostics !=
+                   CaptureDiagsKind::AllWithoutNonErrorsFromIncludes) {
+    if (CaptureDiagnostics != CaptureDiagsKind::None ||
+        Diags.getClient() == nullptr) {
       OwningPreviousClient = Diags.takeClient();
       PreviousClient = Diags.getClient();
       Diags.setClient(&Client, false);
     }
   }
 
   ~CaptureDroppedDiagnostics() {
     if (Diags.getClient() == &Client)
       Diags.setClient(PreviousClient, !!OwningPreviousClient.release());
   }
 };
 
 } // namespace
 
 static ASTUnit::StandaloneDiagnostic
 makeStandaloneDiagnostic(const LangOptions &LangOpts,
                          const StoredDiagnostic &InDiag);
 
-void StoredDiagnosticConsumer::HandleDiagnostic(DiagnosticsEngine::Level Level,
-                                                const Diagnostic &Info) {
+static bool isInMainFile(const clang::Diagnostic &D) {
+  if (!D.hasSourceManager() || !D.getLocation().isValid())
+    return false;
+
+  auto &M = D.getSourceManager();
+  return M.isWrittenInMainFile(M.getExpansionLoc(D.getLocation()));
+}
+
+void FilterAndStoreDiagnosticConsumer::HandleDiagnostic(
+    DiagnosticsEngine::Level Level, const Diagnostic &Info) {
   // Default implementation (Warnings/errors count).
   DiagnosticConsumer::HandleDiagnostic(Level, Info);
 
   // Only record the diagnostic if it's part of the source manager we know
   // about. This effectively drops diagnostics from modules we're building.
   // FIXME: In the long run, ee don't want to drop source managers from modules.
   if (!Info.hasSourceManager() || &Info.getSourceManager() == SourceMgr) {
+    if (!CaptureNonErrorsFromIncludes && Level <= DiagnosticsEngine::Warning &&
+        !isInMainFile(Info)) {
+      return;
+    }
+
     StoredDiagnostic *ResultDiag = nullptr;
     if (StoredDiags) {
       StoredDiags->emplace_back(Level, Info);
       ResultDiag = &StoredDiags->back();
     }
 
     if (StandaloneDiags) {
       llvm::Optional<StoredDiagnostic> StoredDiag = None;
if (Buffer)
     return std::move(*Buffer);
   if (ErrorStr)
     *ErrorStr = Buffer.getError().message();
   return nullptr;
 }
 
 /// Configure the diagnostics object for use with ASTUnit.
 void ASTUnit::ConfigureDiags(IntrusiveRefCntPtr<DiagnosticsEngine> Diags,
-                             ASTUnit &AST, bool CaptureDiagnostics) {
+                             ASTUnit &AST,
+                             CaptureDiagsKind CaptureDiagnostics) {
   assert(Diags.get() && "no DiagnosticsEngine was provided");
-  if (CaptureDiagnostics)
-    Diags->setClient(new StoredDiagnosticConsumer(&AST.StoredDiagnostics, nullptr));
+  if (CaptureDiagnostics != CaptureDiagsKind::None)
+    Diags->setClient(new FilterAndStoreDiagnosticConsumer(
+        &AST.StoredDiagnostics, nullptr,
+        CaptureDiagnostics != CaptureDiagsKind::AllWithoutNonErrorsFromIncludes));
 }
 
 std::unique_ptr<ASTUnit> ASTUnit::LoadFromASTFile(
     const std::string &Filename, const PCHContainerReader &PCHContainerRdr,
     WhatToLoad ToLoad, IntrusiveRefCntPtr<DiagnosticsEngine> Diags,
     const FileSystemOptions &FileSystemOpts, bool UseDebugInfo,
     bool OnlyLocalDecls, ArrayRef<RemappedFile> RemappedFiles,
-    bool CaptureDiagnostics, bool AllowPCHWithCompilerErrors,
+    CaptureDiagsKind CaptureDiagnostics, bool AllowPCHWithCompilerErrors,
     bool UserFilesAreVolatile) {
   std::unique_ptr<ASTUnit> AST(new ASTUnit(true));
 
   // Recover resources if we crash before exiting this method.
   llvm::CrashRecoveryContextCleanupRegistrar<ASTUnit>
     ASTUnitCleanup(AST.get());
   llvm::CrashRecoveryContextCleanupRegistrar<DiagnosticsEngine,
     llvm::CrashRecoveryContextReleaseRefCleanup<DiagnosticsEngine>>
ASTUnit::getMainBufferWithPrecompiledPreamble(
   if (!AllowRebuild)
     return nullptr;
 
   SmallVector<StandaloneDiagnostic, 4> NewPreambleDiagsStandalone;
   SmallVector<StoredDiagnostic, 4> NewPreambleDiags;
   ASTUnitPreambleCallbacks Callbacks;
   {
     llvm::Optional<CaptureDroppedDiagnostics> Capture;
-    if (CaptureDiagnostics)
-      Capture.emplace(/*RequestCapture=*/true, *Diagnostics, &NewPreambleDiags,
+    if (CaptureDiagnostics != CaptureDiagsKind::None)
+      Capture.emplace(CaptureDiagnostics, *Diagnostics, &NewPreambleDiags,
                       &NewPreambleDiagsStandalone);
 
     // We did not previously compute a preamble, or it can't be reused anyway.
     SimpleTimer PreambleTimer(WantTiming);
     PreambleTimer.setOutput("Precompiling preamble");
 
     const bool PreviousSkipFunctionBodies =
         PreambleInvocationIn.getFrontendOpts().SkipFunctionBodies;
StringRef ASTUnit::getASTFileName() const {
   serialization::ModuleFile &
     Mod = Reader->getModuleManager().getPrimaryModule();
   return Mod.FileName;
 }
 
 std::unique_ptr<ASTUnit>
 ASTUnit::create(std::shared_ptr<CompilerInvocation> CI,
                 IntrusiveRefCntPtr<DiagnosticsEngine> Diags,
-                bool CaptureDiagnostics, bool UserFilesAreVolatile) {
+                CaptureDiagsKind CaptureDiagnostics,
+                bool UserFilesAreVolatile) {
   std::unique_ptr<ASTUnit> AST(new ASTUnit(false));
   ConfigureDiags(Diags, *AST, CaptureDiagnostics);
   IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS =
       createVFSFromCompilerInvocation(*CI, *Diags);
   AST->Diagnostics = Diags;
   AST->FileSystemOpts = CI->getFileSystemOpts();
   AST->Invocation = std::move(CI);
   AST->FileMgr = new FileManager(AST->FileSystemOpts, VFS);
   AST->UserFilesAreVolatile = UserFilesAreVolatile;
   AST->SourceMgr = new SourceManager(AST->getDiagnostics(), *AST->FileMgr,
                                      UserFilesAreVolatile);
   AST->ModuleCache = new InMemoryModuleCache;
 
   return AST;
 }
 
 ASTUnit *ASTUnit::LoadFromCompilerInvocationAction(
     std::shared_ptr<CompilerInvocation> CI,
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
     IntrusiveRefCntPtr<DiagnosticsEngine> Diags, FrontendAction *Action,
     ASTUnit *Unit, bool Persistent, StringRef ResourceFilesPath,
-    bool OnlyLocalDecls, bool CaptureDiagnostics,
+    bool OnlyLocalDecls, CaptureDiagsKind CaptureDiagnostics,
     unsigned PrecompilePreambleAfterNParses, bool CacheCodeCompletionResults,
     bool IncludeBriefCommentsInCodeCompletion, bool UserFilesAreVolatile,
     std::unique_ptr<ASTUnit> *ErrAST) {
   assert(CI && "A CompilerInvocation is required");
 
   std::unique_ptr<ASTUnit> OwnAST;
   ASTUnit *AST = Unit;
   if (!AST) {
bool ASTUnit::LoadFromCompilerInvocation(
 
   return Parse(std::move(PCHContainerOps), std::move(OverrideMainBuffer), VFS);
 }
 
 std::unique_ptr<ASTUnit> ASTUnit::LoadFromCompilerInvocation(
     std::shared_ptr<CompilerInvocation> CI,
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
     IntrusiveRefCntPtr<DiagnosticsEngine> Diags, FileManager *FileMgr,
-    bool OnlyLocalDecls, bool CaptureDiagnostics,
+    bool OnlyLocalDecls, CaptureDiagsKind CaptureDiagnostics,
     unsigned PrecompilePreambleAfterNParses, TranslationUnitKind TUKind,
     bool CacheCodeCompletionResults, bool IncludeBriefCommentsInCodeCompletion,
     bool UserFilesAreVolatile) {
   // Create the AST unit.
   std::unique_ptr<ASTUnit> AST(new ASTUnit(false));
   ConfigureDiags(Diags, *AST, CaptureDiagnostics);
   AST->Diagnostics = Diags;
   AST->OnlyLocalDecls = OnlyLocalDecls;
if (AST->LoadFromCompilerInvocation(std::move(PCHContainerOps),
     return nullptr;
   return AST;
 }
 
 ASTUnit *ASTUnit::LoadFromCommandLine(
     const char **ArgBegin, const char **ArgEnd,
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
     IntrusiveRefCntPtr<DiagnosticsEngine> Diags, StringRef ResourceFilesPath,
-    bool OnlyLocalDecls, bool CaptureDiagnostics,
+    bool OnlyLocalDecls, CaptureDiagsKind CaptureDiagnostics,
     ArrayRef<RemappedFile> RemappedFiles, bool RemappedFilesKeepOriginalName,
     unsigned PrecompilePreambleAfterNParses, TranslationUnitKind TUKind,
     bool CacheCodeCompletionResults, bool IncludeBriefCommentsInCodeCompletion,
     bool AllowPCHWithCompilerErrors, SkipFunctionBodiesScope SkipFunctionBodies,
     bool SingleFileParse, bool UserFilesAreVolatile, bool ForSerialization,
     llvm::Optional<StringRef> ModuleFormat, std::unique_ptr<ASTUnit> *ErrAST,
     IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS) {
   assert(Diags.get() && "no DiagnosticsEngine was provided");
llvm::CrashRecoveryContextCleanupRegistrar<CompilerInstance>
     CICleanup(Clang.get());
 
   auto &Inv = *CCInvocation;
   Clang->setInvocation(std::move(CCInvocation));
   OriginalSourceFile = Clang->getFrontendOpts().Inputs[0].getFile();
 
   // Set up diagnostics, capturing any diagnostics produced.
   Clang->setDiagnostics(&Diag);
-  CaptureDroppedDiagnostics Capture(true,
+  CaptureDroppedDiagnostics Capture(CaptureDiagsKind::All,
                                     Clang->getDiagnostics(),
                                     &StoredDiagnostics, nullptr);
   ProcessWarningOptions(Diag, Inv.getDiagnosticOpts());
 
   // Create the target instance.
   Clang->setTarget(TargetInfo::CreateTargetInfo(
       Clang->getDiagnostics(), Clang->getInvocation().TargetOpts));
   if (!Clang->hasTarget()) {

test/Index/ignore-warnings-from-headers.h

+void f(int unusedInHeader) {}

test/Index/ignore-warnings-from-headers.cpp

+#include "ignore-warnings-from-headers.h"
+
+void g(int unusedInMainFile) {}
+
+// RUN: env CINDEXTEST_IGNORE_NONERRORS_FROM_INCLUDED_FILES=1 c-index-test -test-load-source function %s -Wunused-parameter 2>&1 | FileCheck %s
+// CHECK-NOT: warning: unused parameter 'unusedInHeader'
+// CHECK: warning: unused parameter 'unusedInMainFile'

tools/c-index-test/c-index-test.c

static unsigned getDefaultParsingOptions() {
   if (getenv("CINDEXTEST_KEEP_GOING"))
     options |= CXTranslationUnit_KeepGoing;
   if (getenv("CINDEXTEST_LIMIT_SKIP_FUNCTION_BODIES_TO_PREAMBLE"))
     options |= CXTranslationUnit_LimitSkipFunctionBodiesToPreamble;
   if (getenv("CINDEXTEST_INCLUDE_ATTRIBUTED_TYPES"))
     options |= CXTranslationUnit_IncludeAttributedTypes;
   if (getenv("CINDEXTEST_VISIT_IMPLICIT_ATTRIBUTES"))
     options |= CXTranslationUnit_VisitImplicitAttributes;
+  if (getenv("CINDEXTEST_IGNORE_NONERRORS_FROM_INCLUDED_FILES"))
+    options |= CXTranslationUnit_IgnoreNonErrorsFromIncludedFiles;
 
   return options;
 }
 
 static void ModifyPrintingPolicyAccordingToEnv(CXPrintingPolicy Policy) {
   struct Mapping {
     const char *name;
     enum CXPrintingPolicyProperty property;

tools/c-index-test/core_main.cpp

static bool printSourceSymbolsFromModule(StringRef modulePath,
   }
 
   IntrusiveRefCntPtr<DiagnosticsEngine> Diags =
       CompilerInstance::createDiagnostics(new DiagnosticOptions());
   std::unique_ptr<ASTUnit> AU = ASTUnit::LoadFromASTFile(
       modulePath, *pchRdr, ASTUnit::LoadASTOnly, Diags,
       FileSystemOpts, /*UseDebugInfo=*/false,
       /*OnlyLocalDecls=*/true, None,
-      /*CaptureDiagnostics=*/false,
+      CaptureDiagsKind::None,
       /*AllowPCHWithCompilerErrors=*/true,
       /*UserFilesAreVolatile=*/false);
   if (!AU) {
     errs() << "failed to create TU for: " << modulePath << '\n';
     return true;
   }
 
   PrintIndexDataConsumer DataConsumer(outs());

tools/libclang/CIndex.cpp

enum CXErrorCode clang_createTranslationUnit2(CXIndex CIdx,
 
   IntrusiveRefCntPtr<DiagnosticsEngine> Diags =
       CompilerInstance::createDiagnostics(new DiagnosticOptions());
   std::unique_ptr<ASTUnit> AU = ASTUnit::LoadFromASTFile(
       ast_filename, CXXIdx->getPCHContainerOperations()->getRawReader(),
       ASTUnit::LoadEverything, Diags,
       FileSystemOpts, /*UseDebugInfo=*/false,
       CXXIdx->getOnlyLocalDecls(), None,
-      /*CaptureDiagnostics=*/true,
+      CaptureDiagsKind::All,
       /*AllowPCHWithCompilerErrors=*/true,
       /*UserFilesAreVolatile=*/true);
   *out_TU = MakeCXTranslationUnit(CXXIdx, std::move(AU));
   return *out_TU ? CXError_Success : CXError_Failure;
 }
 
 unsigned clang_defaultEditingTranslationUnitOptions() {
   return CXTranslationUnit_PrecompiledPreamble | 
clang_parseTranslationUnit_Impl(CXIndex CIdx, const char *source_filename,
 
   // Configure the diagnostics.
   IntrusiveRefCntPtr<DiagnosticsEngine>
     Diags(CompilerInstance::createDiagnostics(new DiagnosticOptions));
 
   if (options & CXTranslationUnit_KeepGoing)
     Diags->setFatalsAsError(true);
 
+  CaptureDiagsKind CaptureDiagnostics = CaptureDiagsKind::All;
+  if (options & CXTranslationUnit_IgnoreNonErrorsFromIncludedFiles)
+    CaptureDiagnostics = CaptureDiagsKind::AllWithoutNonErrorsFromIncludes;
+
   // Recover resources if we crash before exiting this function.
   llvm::CrashRecoveryContextCleanupRegistrar<DiagnosticsEngine,
     llvm::CrashRecoveryContextReleaseRefCleanup<DiagnosticsEngine> >
     DiagCleanup(Diags.get());
 
   std::unique_ptr<std::vector<ASTUnit::RemappedFile>> RemappedFiles(
       new std::vector<ASTUnit::RemappedFile>());
 
clang_parseTranslationUnit_Impl(CXIndex CIdx, const char *source_filename,
   LibclangInvocationReporter InvocationReporter(
       *CXXIdx, LibclangInvocationReporter::OperationKind::ParseOperation,
       options, llvm::makeArrayRef(*Args), /*InvocationArgs=*/None,
       unsaved_files);
   std::unique_ptr<ASTUnit> Unit(ASTUnit::LoadFromCommandLine(
       Args->data(), Args->data() + Args->size(),
       CXXIdx->getPCHContainerOperations(), Diags,
       CXXIdx->getClangResourcesPath(), CXXIdx->getOnlyLocalDecls(),
-      /*CaptureDiagnostics=*/true, *RemappedFiles.get(),
+      CaptureDiagnostics, *RemappedFiles.get(),
       /*RemappedFilesKeepOriginalName=*/true, PrecompilePreambleAfterNParses,
       TUKind, CacheCodeCompletionResults, IncludeBriefCommentsInCodeCompletion,
       /*AllowPCHWithCompilerErrors=*/true, SkipFunctionBodies, SingleFileParse,
       /*UserFilesAreVolatile=*/true, ForSerialization,
       CXXIdx->getPCHContainerOperations()->getRawReader().getFormat(),
       &ErrUnit));
 
   // Early failures in LoadFromCommandLine may return with ErrUnit unset.

tools/libclang/Indexing.cpp

static CXErrorCode clang_indexSourceFile_Impl(
   memcpy(&CB, client_index_callbacks, ClientCBSize);
 
   IndexSessionData *IdxSession = static_cast<IndexSessionData *>(cxIdxAction);
   CIndexer *CXXIdx = static_cast<CIndexer *>(IdxSession->CIdx);
 
   if (CXXIdx->isOptEnabled(CXGlobalOpt_ThreadBackgroundPriorityForIndexing))
     setThreadBackgroundPriority();
 
-  bool CaptureDiagnostics = !Logger::isLoggingEnabled();
+  CaptureDiagsKind CaptureDiagnostics = CaptureDiagsKind::All;
+  if (TU_options & CXTranslationUnit_IgnoreNonErrorsFromIncludedFiles)
+    CaptureDiagnostics = CaptureDiagsKind::AllWithoutNonErrorsFromIncludes;
+  if (Logger::isLoggingEnabled())
+    CaptureDiagnostics = CaptureDiagsKind::None;
 
   CaptureDiagnosticConsumer *CaptureDiag = nullptr;
-  if (CaptureDiagnostics)
+  if (CaptureDiagnostics != CaptureDiagsKind::None)
     CaptureDiag = new CaptureDiagnosticConsumer();
 
   // Configure the diagnostics.
   IntrusiveRefCntPtr<DiagnosticsEngine>
     Diags(CompilerInstance::createDiagnostics(new DiagnosticOptions,
                                               CaptureDiag,
                                               /*ShouldOwnClient=*/true));
 

unittests/Frontend/ASTUnitTest.cpp

std::unique_ptr<ASTUnit> createASTUnit(bool isVolatile) {
     if (!CInvok)
       return nullptr;
 
     FileManager *FileMgr =
         new FileManager(FileSystemOptions(), vfs::getRealFileSystem());
     PCHContainerOps = std::make_shared<PCHContainerOperations>();
 
     return ASTUnit::LoadFromCompilerInvocation(
-        CInvok, PCHContainerOps, Diags, FileMgr, false, false, 0, TU_Complete,
-        false, false, isVolatile);
+        CInvok, PCHContainerOps, Diags, FileMgr, false, CaptureDiagsKind::None,
+        0, TU_Complete, false, false, isVolatile);
   }
 };
 
 TEST_F(ASTUnitTest, SaveLoadPreservesLangOptionsInPrintingPolicy) {
   // Check that the printing policy is restored with the correct language
   // options when loading an ASTUnit from a file.  To this end, an ASTUnit
   // for a C++ translation unit is set up and written to a temporary file.
 

unittests/Frontend/PCHPreambleTest.cpp

std::unique_ptr<ASTUnit> ParseAST(const std::string &EntryFile) {
     PPOpts.RemappedFilesKeepOriginalName = true;
 
     IntrusiveRefCntPtr<DiagnosticsEngine>
       Diags(CompilerInstance::createDiagnostics(new DiagnosticOptions, new DiagnosticConsumer));
 
     FileManager *FileMgr = new FileManager(FSOpts, VFS);
 
     std::unique_ptr<ASTUnit> AST = ASTUnit::LoadFromCompilerInvocation(
-      CI, PCHContainerOpts, Diags, FileMgr, false, false,
+        CI, PCHContainerOpts, Diags, FileMgr, false, CaptureDiagsKind::None,
         /*PrecompilePreambleAfterNParses=*/1);
     return AST;
   }
 
   bool ReparseAST(const std::unique_ptr<ASTUnit> &AST) {
     bool reparseFailed = AST->Reparse(PCHContainerOpts, GetRemappedFiles(), VFS);
     return !reparseFailed;
   }