[PCH] Serialize skipped preprocessor ranges

[cameron314]

Summary

This fixes, for example, libclang's clang_getSkippedRanges returning zero ranges after reparsing a translation unit.

Fixes PR34971.

[erikjv]

The patch has aged a bit since upload, so does it still apply? And, can you add a testcase please?

[cameron314]

I'll rebase the patch and add a test. Thanks for looking at this!

[cameron314]

Fully rebased, with a test.

[nik]

Here are my observations:

• Your test case works fine for me even without having this change applied/build. Looks like this is already fixed in current trunk. Please confirm/test.
• Can you come up with another test case that fixes something that is not yet addressed in trunk?
• It does not fix https://bugs.llvm.org/show_bug.cgi?id=34971

Apart from that the patch applies cleanly and all tests pass (check-clang). However, some new warnings are emitted during building:

/home/nik/dev/llvm/trunk/source/tools/clang/include/clang/Lex/PreprocessingRecord.h: In constructor ‘clang::PreprocessingRecord::PreprocessingRecord(clang::SourceManager&)’:
/home/nik/dev/llvm/trunk/source/tools/clang/include/clang/Lex/PreprocessingRecord.h:339:40: warning: ‘clang::PreprocessingRecord::ExternalSource’ will be initialized after [-Wreorder]
	 ExternalPreprocessingRecordSource *ExternalSource;
										^~~~~~~~~~~~~~
/home/nik/dev/llvm/trunk/source/tools/clang/include/clang/Lex/PreprocessingRecord.h:312:10: warning:   ‘bool clang::PreprocessingRecord::SkippedRangesAllLoaded’ [-Wreorder]
	 bool SkippedRangesAllLoaded;
		  ^~~~~~~~~~~~~~~~~~~~~~
/home/nik/dev/llvm/trunk/source/tools/clang/lib/Lex/PreprocessingRecord.cpp:36:1: warning:   when initialized here [-Wreorder]

lib/Lex/PreprocessingRecord.cpp

349

Remove excess new line.

[cameron314]

• Well that's odd, because the test definitely fails for me without the patch. I'm only a few days behind the trunk.
• I'm looking at your test case now. I can reproduce it even with the patch; I'm investigating what's happening.
• I've fixed the warning, thanks! Using MSVC on my end which doesn't emit that particular warning.

[cameron314]

Alright, with my patch the c-index-test *does* correctly serialize and restore the skipped ranges; the problem is that it searches for only ranges matching the target file. When there's a preamble, it's seen as a different file than the main file, even though they're really one and the same in this case.

So my test, which uses clang_getAllSkippedRanges, passes, but c-index-test, which uses clang_getSkippedRanges, fails. I'll update my test to use both.

I'm trying to figure out how to fix this.

[cameron314]

Well, it seems like preamble PCH source location translation is fundamentally broken. The entry file has a single, positive file ID. The preamble PCH is treated as an imported module, so it has a negative file ID for the part that overlaps the preamble of the entry file. That means locations in the preamble part of the entry file can have two different file IDs depending on how they are arrived at.

I really don't know how to fix this. Any ideas?

[ilya-biryukov]

In D20124#927967, @cameron314 wrote:

Well, it seems like preamble PCH source location translation is fundamentally broken. The entry file has a single, positive file ID. The preamble PCH is treated as an imported module, so it has a negative file ID for the part that overlaps the preamble of the entry file. That means locations in the preamble part of the entry file can have two different file IDs depending on how they are arrived at.

I really don't know how to fix this. Any ideas?

Why do we store raw source locations in PPSkippedRange? Would storing SourceLocation and using ASTWriter::AddSourceLocation and ASTReader:: ReadSourceLocation do the trick?

[cameron314]

In D20124#928552, @ilya-biryukov wrote:

Why do we store raw source locations in PPSkippedRange? Would storing SourceLocation and using ASTWriter::AddSourceLocation and ASTReader:: ReadSourceLocation do the trick?

I followed the pattern used to store PPEntityOffsets; it allows the entire array to be written in one chunk. Using AddSourceLocation/ReadSourceLocation boils down to the same thing -- AddSourceLocation simply rotates by one bit, and ReadSourceLocation simply rotates back and calls ASTReader::TranslateSourceLocation (which I call from AST::ReadSkippedRange explicitly instead). So the source location translation path is exactly the same either way.

The thing is, the PCH created for the preamble is imported as a module, meaning the part of the source code that overlaps the preamble of the entry file has a different file ID from the entry file itself. Is there any way to force source locations in the preamble to map to the actual entry file instead of the module's version of the entry file?

[erikjv]

Maybe something like this works:

--- a/tools/libclang/CIndex.cpp
+++ b/tools/libclang/CIndex.cpp
@@ -8090,6 +8090,7 @@ CXSourceRangeList *clang_getSkippedRanges(CXTranslationUnit TU, CXFile file) {
   SourceManager &sm = Ctx.getSourceManager();
   FileEntry *fileEntry = static_cast<FileEntry *>(file);
   FileID wantedFileID = sm.translateFile(fileEntry);
+  bool isMainFile = wantedFileID == sm.getMainFileID();
 
   const std::vector<SourceRange> &SkippedRanges = ppRec->getSkippedRanges();
   std::vector<SourceRange> wantedRanges;
@@ -8097,6 +8098,8 @@ CXSourceRangeList *clang_getSkippedRanges(CXTranslationUnit TU, CXFile file) {
        i != ei; ++i) {
     if (sm.getFileID(i->getBegin()) == wantedFileID || sm.getFileID(i->getEnd()) == wantedFileID)
       wantedRanges.push_back(*i);
+    else if (isMainFile && (astUnit->isInPreambleFileID(i->getBegin()) || astUnit->isInPreambleFileID(i->getEnd())))
+      wantedRanges.push_back(*i);
   }
 
   skipped->count = wantedRanges.size();

[cameron314]

Brilliant, didn't know isInPreambleFileID existed. All my tests pass for me now with that change, thanks :-)
I'll update the patch on Monday.

[cameron314]

Here's the final patch that fixes clang_getSkippedRegions with regions in the preamble (as well as serializing the skipped regions in the PCH file).

[nik]

In D20124#943592, @cameron314 wrote:

Here's the final patch that fixes clang_getSkippedRegions with regions in the preamble (as well as serializing the skipped regions in the PCH file).

Works fine for me, thanks! Test from the bug report is fixed by this one.

You should probably add "Fixes PR34971." to the summary.

@erikjv @ilya-biryukov - please review as you know the details here :)

[cameron314]

Ping?
The patch is ready to go, just needs a final approval...

[erikjv]

lgtm. Ilya?

[ilya-biryukov]

LGTM, sorry for the delay.

[cameron314]

Excellent, I'll rebase and commit. Thanks everyone for your patience!

include/clang/Lex/PreprocessingRecord.h

virtual std::pair<unsigned, unsigned>
         findPreprocessedEntitiesInRange(SourceRange Range) = 0;
 
     /// \brief Optionally returns true or false if the preallocated preprocessed
     /// entity with index \p Index came from file \p FID.
     virtual Optional<bool> isPreprocessedEntityInFileID(unsigned Index,
                                                         FileID FID) {
       return None;
     }
+
+    /// \brief Read a preallocated skipped range from the external source.
+    virtual SourceRange ReadSkippedRange(unsigned Index) = 0;
   };
   
   /// \brief A record of the steps taken while preprocessing a source file,
   /// including the various preprocessing directives processed, macros 
   /// expanded, etc.
   class PreprocessingRecord : public PPCallbacks {
     SourceManager &SourceMgr;
     
class PreprocessingRecord : public PPCallbacks {
     ///
     /// The entries in this vector are loaded lazily from the external source,
     /// and are referenced by the iterator using negative indices.
     std::vector<PreprocessedEntity *> LoadedPreprocessedEntities;
 
     /// \brief The set of ranges that were skipped by the preprocessor,
     std::vector<SourceRange> SkippedRanges;
 
+    bool SkippedRangesAllLoaded = true;
+
     /// \brief Global (loaded or local) ID for a preprocessed entity.
     /// Negative values are used to indicate preprocessed entities
     /// loaded from the external source while non-negative values are used to
     /// indicate preprocessed entities introduced by the current preprocessor.
     /// Value -1 corresponds to element 0 in the loaded entities vector,
     /// value -2 corresponds to element 1 in the loaded entities vector, etc.
     /// Value 0 is an invalid value, the index to local entities is 1-based,
     /// value 1 corresponds to element 0 in the local entities vector,
class PreprocessingRecord : public PPCallbacks {
     unsigned findEndLocalPreprocessedEntity(SourceLocation Loc) const;
 
     /// \brief Allocate space for a new set of loaded preprocessed entities.
     ///
     /// \returns The index into the set of loaded preprocessed entities, which
     /// corresponds to the first newly-allocated entity.
     unsigned allocateLoadedEntities(unsigned NumEntities);
 
+    /// \brief Allocate space for a new set of loaded preprocessed skipped
+    /// ranges.
+    ///
+    /// \returns The index into the set of loaded preprocessed ranges, which
+    /// corresponds to the first newly-allocated range.
+    unsigned allocateSkippedRanges(unsigned NumRanges);
+
+    /// \brief Ensures that all external skipped ranges have been loaded.
+    void ensureSkippedRangesLoaded();
+
     /// \brief Register a new macro definition.
     void RegisterMacroDefinition(MacroInfo *Macro, MacroDefinitionRecord *Def);
 
   public:
     /// \brief Construct a new preprocessing record.
     explicit PreprocessingRecord(SourceManager &SM);
 
     /// \brief Allocate memory in the preprocessing record.
ExternalPreprocessingRecordSource *getExternalSource() const {
       return ExternalSource;
     }
 
     /// \brief Retrieve the macro definition that corresponds to the given
     /// \c MacroInfo.
     MacroDefinitionRecord *findMacroDefinition(const MacroInfo *MI);
 
     /// \brief Retrieve all ranges that got skipped while preprocessing.
-    const std::vector<SourceRange> &getSkippedRanges() const {
+    const std::vector<SourceRange> &getSkippedRanges() {
+      ensureSkippedRangesLoaded();
       return SkippedRanges;
     }
         
   private:
     friend class ASTReader;
     friend class ASTWriter;
 
     void MacroExpands(const Token &Id, const MacroDefinition &MD,

include/clang/Serialization/ASTBitCodes.h

struct PPEntityOffset {
         return SourceLocation::getFromRawEncoding(Begin);
       }
 
       SourceLocation getEnd() const {
         return SourceLocation::getFromRawEncoding(End);
       }
     };
 
+    /// \brief Source range of a skipped preprocessor region
+    struct PPSkippedRange {
+      /// \brief Raw source location of beginning of range.
+      unsigned Begin;
+      /// \brief Raw source location of end of range.
+      unsigned End;
+
+      PPSkippedRange(SourceRange R)
+        : Begin(R.getBegin().getRawEncoding()),
+          End(R.getEnd().getRawEncoding()) { }
+
+      SourceLocation getBegin() const {
+        return SourceLocation::getFromRawEncoding(Begin);
+      }
+      SourceLocation getEnd() const {
+        return SourceLocation::getFromRawEncoding(End);
+      }
+    };
+
     /// \brief Source range/offset of a preprocessed entity.
     struct DeclOffset {
       /// \brief Raw source location.
       unsigned Loc = 0;
 
       /// \brief Offset in the AST file.
       uint32_t BitOffset = 0;
 
enum ASTRecordTypes {
 
       MODULAR_CODEGEN_DECLS = 60,
 
       /// \brief Record code for \#pragma pack options.
       PACK_PRAGMA_OPTIONS = 61,
 
       /// \brief The stack of open #ifs/#ifdefs recorded in a preamble.
       PP_CONDITIONAL_STACK = 62,
+
+      /// \brief A table of skipped ranges within the preprocessing record.
+      PPD_SKIPPED_RANGES = 63
     };
 
     /// \brief Record types used within a source manager block.
     enum SourceManagerRecordTypes {
       /// \brief Describes a source location entry (SLocEntry) for a
       /// file.
       SM_SLOC_FILE_ENTRY = 1,
 

include/clang/Serialization/ASTReader.h

private:
   using GlobalPreprocessedEntityMapType =
       ContinuousRangeMap<unsigned, ModuleFile *, 4>;
 
   /// \brief Mapping from global preprocessing entity IDs to the module in
   /// which the preprocessed entity resides along with the offset that should be
   /// added to the global preprocessing entity ID to produce a local ID.
   GlobalPreprocessedEntityMapType GlobalPreprocessedEntityMap;
 
+  typedef ContinuousRangeMap<unsigned, ModuleFile *, 4>
+    GlobalSkippedRangeMapType;
+
+  /// \brief Mapping from global skipped range base IDs to the module in which
+  /// the skipped ranges reside.
+  GlobalSkippedRangeMapType GlobalSkippedRangeMap;
+
   /// \name CodeGen-relevant special data
   /// \brief Fields containing data that is relevant to CodeGen.
   //@{
 
   /// \brief The IDs of all declarations that fulfill the criteria of
   /// "interesting" decls.
   ///
   /// This contains the data loaded from all EAGERLY_DESERIALIZED_DECLS blocks
public:
   std::pair<unsigned, unsigned>
       findPreprocessedEntitiesInRange(SourceRange Range) override;
 
   /// \brief Optionally returns true or false if the preallocated preprocessed
   /// entity with index \p Index came from file \p FID.
   Optional<bool> isPreprocessedEntityInFileID(unsigned Index,
                                               FileID FID) override;
 
+  /// \brief Read a preallocated skipped range from the external source.
+  SourceRange ReadSkippedRange(unsigned Index) override;
+
   /// \brief Read the header file information for the given file entry.
   HeaderFileInfo GetHeaderFileInfo(const FileEntry *FE) override;
 
   void ReadPragmaDiagnosticMappings(DiagnosticsEngine &Diag);
 
   /// \brief Returns the number of source locations found in the chain.
   unsigned getTotalNumSLocs() const {
     return TotalNumSLocEntries;

include/clang/Serialization/Module.h

public:
   serialization::PreprocessedEntityID BasePreprocessedEntityID = 0;
 
   /// \brief Remapping table for preprocessed entity IDs in this module.
   ContinuousRangeMap<uint32_t, int, 2> PreprocessedEntityRemap;
 
   const PPEntityOffset *PreprocessedEntityOffsets = nullptr;
   unsigned NumPreprocessedEntities = 0;
 
+  /// \brief Base ID for preprocessed skipped ranges local to this module.
+  unsigned BasePreprocessedSkippedRangeID = 0;
+
+  const PPSkippedRange *PreprocessedSkippedRangeOffsets = nullptr;
+  unsigned NumPreprocessedSkippedRanges = 0;
+
   // === Header search information ===
 
   /// \brief The number of local HeaderFileInfo structures.
   unsigned LocalNumHeaderFileInfos = 0;
 
   /// \brief Actual data for the on-disk hash table of header file
   /// information.
   ///

lib/Lex/PPDirectives.cpp

while (true) {
     if (CurLexer) CurLexer->resetExtendedTokenMode();
   }
 
   // Finally, if we are out of the conditional (saw an #endif or ran off the end
   // of the file, just stop skipping and return to lexing whatever came after
   // the #if block.
   CurPPLexer->LexingRawMode = false;
 
-  if (Callbacks)
+  // The last skipped range isn't actually skipped yet if it's truncated
+  // by the end of the preamble; we'll resume parsing after the preamble.
+  if (Callbacks && (Tok.isNot(tok::eof) || !isRecordingPreamble()))
     Callbacks->SourceRangeSkipped(
         SourceRange(HashTokenLoc, CurPPLexer->getSourceLocation()),
         Tok.getLocation());
 }
 
 void Preprocessor::PTHSkipExcludedConditionalBlock() {
   while (true) {
     assert(CurPTHLexer);

lib/Lex/PreprocessingRecord.cpp

 
 unsigned PreprocessingRecord::allocateLoadedEntities(unsigned NumEntities) {
   unsigned Result = LoadedPreprocessedEntities.size();
   LoadedPreprocessedEntities.resize(LoadedPreprocessedEntities.size() 
                                     + NumEntities);
   return Result;
 }
 
+unsigned PreprocessingRecord::allocateSkippedRanges(unsigned NumRanges) {
+  unsigned Result = SkippedRanges.size();
+  SkippedRanges.resize(SkippedRanges.size() + NumRanges);
+  SkippedRangesAllLoaded = false;
+  return Result;
+}
+
+void PreprocessingRecord::ensureSkippedRangesLoaded() {
+  if (SkippedRangesAllLoaded || !ExternalSource)
+    return;
+  for (unsigned Index = 0; Index != SkippedRanges.size(); ++Index) {
+    if (SkippedRanges[Index].isInvalid())
+      SkippedRanges[Index] = ExternalSource->ReadSkippedRange(Index);
+  }
+  SkippedRangesAllLoaded = true;
+}
+
 void PreprocessingRecord::RegisterMacroDefinition(MacroInfo *Macro,
                                                   MacroDefinitionRecord *Def) {
   MacroDefinitions[Macro] = Def;
 }
 
 /// \brief Retrieve the preprocessed entity at the given ID.
 PreprocessedEntity *PreprocessingRecord::getPreprocessedEntity(PPEntityID PPID){
   if (PPID.ID < 0) {
     unsigned Index = -PPID.ID - 1;
void PreprocessingRecord::Defined(const Token &MacroNameTok,
   // This is not actually a macro expansion but record it as a macro reference.
   if (MD)
     addMacroExpansion(MacroNameTok, MD.getMacroInfo(),
                       MacroNameTok.getLocation());
 }
 
 void PreprocessingRecord::SourceRangeSkipped(SourceRange Range,
                                              SourceLocation EndifLoc) {
+  assert(Range.isValid());
   SkippedRanges.emplace_back(Range.getBegin(), EndifLoc);
 }
 
 void PreprocessingRecord::MacroExpands(const Token &Id,
                                        const MacroDefinition &MD,
                                        SourceRange Range,
                                        const MacroArgs *Args) {
   addMacroExpansion(Id, MD.getMacroInfo(), Range);
clang::InclusionDirective *ID =
                                             SourceRange(HashLoc, EndLoc));
   addPreprocessedEntity(ID);
 }
 
 size_t PreprocessingRecord::getTotalMemory() const {
   return BumpAlloc.getTotalMemory()
     + llvm::capacity_in_bytes(MacroDefinitions)
     + llvm::capacity_in_bytes(PreprocessedEntities)
-    + llvm::capacity_in_bytes(LoadedPreprocessedEntities);
+    + llvm::capacity_in_bytes(LoadedPreprocessedEntities)
+    + llvm::capacity_in_bytes(SkippedRanges);
 }

lib/Serialization/ASTReader.cpp

case PPD_ENTITIES_OFFSETS: {
         F.PreprocessedEntityRemap.insertOrReplace(
           std::make_pair(LocalBasePreprocessedEntityID,
             F.BasePreprocessedEntityID - LocalBasePreprocessedEntityID));
       }
 
       break;
     }
 
+    case PPD_SKIPPED_RANGES: {
+      F.PreprocessedSkippedRangeOffsets = (const PPSkippedRange*)Blob.data();
+      assert(Blob.size() % sizeof(PPSkippedRange) == 0);
+      F.NumPreprocessedSkippedRanges = Blob.size() / sizeof(PPSkippedRange);
+
+      if (!PP.getPreprocessingRecord())
+        PP.createPreprocessingRecord();
+      if (!PP.getPreprocessingRecord()->getExternalSource())
+        PP.getPreprocessingRecord()->SetExternalSource(*this);
+      F.BasePreprocessedSkippedRangeID = PP.getPreprocessingRecord()
+          ->allocateSkippedRanges(F.NumPreprocessedSkippedRanges);
+      
+      if (F.NumPreprocessedSkippedRanges > 0)
+        GlobalSkippedRangeMap.insert(
+            std::make_pair(F.BasePreprocessedSkippedRangeID, &F));
+      break;
+    }
+
     case DECL_UPDATE_OFFSETS:
       if (Record.size() % 2 != 0) {
         Error("invalid DECL_UPDATE_OFFSETS block in AST file");
         return Failure;
       }
       for (unsigned I = 0, N = Record.size(); I != N; I += 2) {
         GlobalDeclID ID = getGlobalDeclID(F, Record[I]);
         DeclUpdateOffsets[ID].push_back(std::make_pair(&F, Record[I + 1]));

 llvm::iterator_range<ASTReader::ModuleDeclIterator>
 ASTReader::getModuleFileLevelDecls(ModuleFile &Mod) {
   return llvm::make_range(
       ModuleDeclIterator(this, &Mod, Mod.FileSortedDecls),
       ModuleDeclIterator(this, &Mod,
                          Mod.FileSortedDecls + Mod.NumFileSortedDecls));
 }
 
+SourceRange ASTReader::ReadSkippedRange(unsigned GlobalIndex) {
+  auto I = GlobalSkippedRangeMap.find(GlobalIndex);
+  assert(I != GlobalSkippedRangeMap.end() &&
+    "Corrupted global skipped range map");
+  ModuleFile *M = I->second;
+  unsigned LocalIndex = GlobalIndex - M->BasePreprocessedSkippedRangeID;
+  assert(LocalIndex < M->NumPreprocessedSkippedRanges);
+  PPSkippedRange RawRange = M->PreprocessedSkippedRangeOffsets[LocalIndex];
+  SourceRange Range(TranslateSourceLocation(*M, RawRange.getBegin()),
+                    TranslateSourceLocation(*M, RawRange.getEnd()));
+  assert(Range.isValid());
+  return Range;
+}
+
 PreprocessedEntity *ASTReader::ReadPreprocessedEntity(unsigned Index) {
   PreprocessedEntityID PPID = Index+1;
   std::pair<ModuleFile *, unsigned> PPInfo = getModulePreprocessedEntity(Index);
   ModuleFile &M = *PPInfo.first;
   unsigned LocalIndex = PPInfo.second;
   const PPEntityOffset &PPOffs = M.PreprocessedEntityOffsets[LocalIndex];
 
   if (!PP.getPreprocessingRecord()) {

lib/Serialization/ASTWriter.cpp

#define RECORD(X) EmitRecordID(X, #X, Stream, Record)
   RECORD(METHOD_POOL);
   RECORD(PP_COUNTER_VALUE);
   RECORD(SOURCE_LOCATION_OFFSETS);
   RECORD(SOURCE_LOCATION_PRELOADS);
   RECORD(EXT_VECTOR_DECLS);
   RECORD(UNUSED_FILESCOPED_DECLS);
   RECORD(PPD_ENTITIES_OFFSETS);
   RECORD(VTABLE_USES);
+  RECORD(PPD_SKIPPED_RANGES);
   RECORD(REFERENCED_SELECTOR_POOL);
   RECORD(TU_UPDATE_LEXICAL);
   RECORD(SEMA_DECL_REFS);
   RECORD(WEAK_UNDECLARED_IDENTIFIERS);
   RECORD(PENDING_IMPLICIT_INSTANTIATIONS);
   RECORD(UPDATE_VISIBLE);
   RECORD(DECL_UPDATE_OFFSETS);
   RECORD(DECL_UPDATES);
if (NumPreprocessingRecords > 0) {
     unsigned PPEOffsetAbbrev = Stream.EmitAbbrev(std::move(Abbrev));
 
     RecordData::value_type Record[] = {PPD_ENTITIES_OFFSETS,
                                        FirstPreprocessorEntityID -
                                            NUM_PREDEF_PP_ENTITY_IDS};
     Stream.EmitRecordWithBlob(PPEOffsetAbbrev, Record,
                               bytes(PreprocessedEntityOffsets));
   }
+
+  // Write the skipped region table for the preprocessing record.
+  ArrayRef<SourceRange> SkippedRanges = PPRec.getSkippedRanges();
+  if (SkippedRanges.size() > 0) {
+    std::vector<PPSkippedRange> SerializedSkippedRanges;
+    SerializedSkippedRanges.reserve(SkippedRanges.size());
+    for (auto const& Range : SkippedRanges)
+      SerializedSkippedRanges.emplace_back(Range);
+
+    using namespace llvm;
+    auto Abbrev = std::make_shared<BitCodeAbbrev>();
+    Abbrev->Add(BitCodeAbbrevOp(PPD_SKIPPED_RANGES));
+    Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob));
+    unsigned PPESkippedRangeAbbrev = Stream.EmitAbbrev(std::move(Abbrev));
+
+    Record.clear();
+    Record.push_back(PPD_SKIPPED_RANGES);
+    Stream.EmitRecordWithBlob(PPESkippedRangeAbbrev, Record,
+                              bytes(SerializedSkippedRanges));
+  }
 }
 
 unsigned ASTWriter::getLocalOrImportedSubmoduleID(Module *Mod) {
   if (!Mod)
     return 0;
 
   llvm::DenseMap<Module *, unsigned>::iterator Known = SubmoduleIDs.find(Mod);
   if (Known != SubmoduleIDs.end())

tools/libclang/CIndex.cpp

CXSourceRangeList *clang_getSkippedRanges(CXTranslationUnit TU, CXFile file) {
   PreprocessingRecord *ppRec = astUnit->getPreprocessor().getPreprocessingRecord();
   if (!ppRec)
     return skipped;
 
   ASTContext &Ctx = astUnit->getASTContext();
   SourceManager &sm = Ctx.getSourceManager();
   FileEntry *fileEntry = static_cast<FileEntry *>(file);
   FileID wantedFileID = sm.translateFile(fileEntry);
+  bool isMainFile = wantedFileID == sm.getMainFileID();
 
   const std::vector<SourceRange> &SkippedRanges = ppRec->getSkippedRanges();
   std::vector<SourceRange> wantedRanges;
   for (std::vector<SourceRange>::const_iterator i = SkippedRanges.begin(), ei = SkippedRanges.end();
        i != ei; ++i) {
     if (sm.getFileID(i->getBegin()) == wantedFileID || sm.getFileID(i->getEnd()) == wantedFileID)
       wantedRanges.push_back(*i);
+    else if (isMainFile && (astUnit->isInPreambleFileID(i->getBegin()) || astUnit->isInPreambleFileID(i->getEnd())))
+      wantedRanges.push_back(*i);
   }
 
   skipped->count = wantedRanges.size();
   skipped->ranges = new CXSourceRange[skipped->count];
   for (unsigned i = 0, ei = skipped->count; i != ei; ++i)
     skipped->ranges[i] = cxloc::translateSourceRange(Ctx, wantedRanges[i]);
 
   return skipped;

unittests/libclang/LibclangTest.cpp

TEST_F(LibclangReparseTest, clang_parseTranslationUnit2FullArgv) {
 
   EXPECT_EQ(CXError_Success,
             clang_parseTranslationUnit2FullArgv(Index, Filename.c_str(), Argv,
                                                 sizeof(Argv) / sizeof(Argv[0]),
                                                 nullptr, 0, TUFlags, &ClangTU));
   EXPECT_EQ(0U, clang_getNumDiagnostics(ClangTU));
   DisplayDiagnostics();
 }
+
+TEST_F(LibclangReparseTest, PreprocessorSkippedRanges) {
+  std::string Header = "header.h", Main = "main.cpp";
+  WriteFile(Header,
+    "#ifdef MANGOS\n"
+    "printf(\"mmm\");\n"
+    "#endif");
+  WriteFile(Main,
+    "#include \"header.h\"\n"
+    "#ifdef GUAVA\n"
+    "#endif\n"
+    "#ifdef KIWIS\n"
+    "printf(\"mmm!!\");\n"
+    "#endif");
+
+  for (int i = 0; i != 3; ++i) {
+    unsigned flags = TUFlags | CXTranslationUnit_PrecompiledPreamble;
+    if (i == 2)
+      flags |= CXTranslationUnit_CreatePreambleOnFirstParse;
+
+    // parse once
+    ClangTU = clang_parseTranslationUnit(Index, Main.c_str(), nullptr, 0,
+                                         nullptr, 0, flags);
+    if (i != 0) {
+      // reparse
+      ASSERT_TRUE(ReparseTU(0, nullptr /* No unsaved files. */));
+    }
+
+    // Check all ranges are there
+    CXSourceRangeList *Ranges = clang_getAllSkippedRanges(ClangTU);
+    EXPECT_EQ(3U, Ranges->count);
+
+    CXSourceLocation cxl;
+    unsigned line;
+    cxl = clang_getRangeStart(Ranges->ranges[0]);
+    clang_getSpellingLocation(cxl, nullptr, &line, nullptr, nullptr);
+    EXPECT_EQ(1U, line);
+    cxl = clang_getRangeEnd(Ranges->ranges[0]);
+    clang_getSpellingLocation(cxl, nullptr, &line, nullptr, nullptr);
+    EXPECT_EQ(3U, line);
+
+    cxl = clang_getRangeStart(Ranges->ranges[1]);
+    clang_getSpellingLocation(cxl, nullptr, &line, nullptr, nullptr);
+    EXPECT_EQ(2U, line);
+    cxl = clang_getRangeEnd(Ranges->ranges[1]);
+    clang_getSpellingLocation(cxl, nullptr, &line, nullptr, nullptr);
+    EXPECT_EQ(3U, line);
+
+    cxl = clang_getRangeStart(Ranges->ranges[2]);
+    clang_getSpellingLocation(cxl, nullptr, &line, nullptr, nullptr);
+    EXPECT_EQ(4U, line);
+    cxl = clang_getRangeEnd(Ranges->ranges[2]);
+    clang_getSpellingLocation(cxl, nullptr, &line, nullptr, nullptr);
+    EXPECT_EQ(6U, line);
+
+    clang_disposeSourceRangeList(Ranges);
+
+    // Check obtaining ranges by each file works
+    CXFile cxf = clang_getFile(ClangTU, Header.c_str());
+    Ranges = clang_getSkippedRanges(ClangTU, cxf);
+    EXPECT_EQ(1U, Ranges->count);
+    cxl = clang_getRangeStart(Ranges->ranges[0]);
+    clang_getSpellingLocation(cxl, nullptr, &line, nullptr, nullptr);
+    EXPECT_EQ(1U, line);
+    clang_disposeSourceRangeList(Ranges);
+
+    cxf = clang_getFile(ClangTU, Main.c_str());
+    Ranges = clang_getSkippedRanges(ClangTU, cxf);
+    EXPECT_EQ(2U, Ranges->count);
+    cxl = clang_getRangeStart(Ranges->ranges[0]);
+    clang_getSpellingLocation(cxl, nullptr, &line, nullptr, nullptr);
+    EXPECT_EQ(2U, line);
+    cxl = clang_getRangeStart(Ranges->ranges[1]);
+    clang_getSpellingLocation(cxl, nullptr, &line, nullptr, nullptr);
+    EXPECT_EQ(4U, line);
+    clang_disposeSourceRangeList(Ranges);
+  }
+}