[Preamble] Fixed preamble breaking with BOM presence (and particularly, fluctuating BOM presence)

[cameron314]

Summary

This patch fixes preamble skipping when the preamble region includes a byte order mark (BOM). Previously, parsing would fail if preamble PCH generation was enabled and a BOM was present.

This also fixes preamble invalidation when a BOM appears or disappears. This may seem to be an obscure edge case, but it happens regularly with IDEs that pass buffer overrides that never (or always) have a BOM, yet the underlying file from the initial parse that generated a PCH might (or might not) have a BOM.

I've included a test case for these scenarios.

Note: This depends on the test infrastructure introduced in D37474.

[ilya-biryukov]

• How are various preprocessor offests (and SourceLocation offsets) are calculated? Do they account for BOM presence and ignore it?
• Are there potential problems we may run into because of the changing offsets? Could we add tests checking changing the offsets does not matter?
• Should we add checks that BOM was removed or added, but not changed? I would not expect preamble to be reusable "as is" if BOM (and therefore, input encoding) changed.

include/clang/Frontend/PrecompiledPreamble.h

102

Let's leave this class's interface immutable. It is used concurrently in clangd and having a mutable method like this would break the code.

Passing new PreambleBounds to AddImplicitPreamble and setting the offsets accordingly would do the trick, leave the interface immutable and make the fact that offsets might change more evident.

191

Let's store original PreambleBounds instead of PreambleEndsAtStartOfLine and PreambleOffset.
It would make the code easier to read.

include/clang/Lex/Lexer.h

50

Maybe pick a name that clearly states that it's a BOM size?
Or add a comment indicating that it's a BOM offset.

639

Maybe leave the old name? Doesn't SkipBytes captures the new semantics just as good?

lib/Frontend/PrecompiledPreamble.cpp

195

Could you inline usages of this function and remove it?

unittests/Frontend/PchPreambleTest.cpp

190

We're not really testing that preamble was reused.
Maybe return a flag from ASTUnit::Reparse to indicate if preamble was reused and check it here?

[cameron314]

Thanks for the response!

How are various preprocessor offests (and SourceLocation offsets) are calculated? Do they account for BOM presence and ignore it?

Everything is in byte offsets; the SourceLocation after the BOM is not the same as before the BOM. The lexer automatically skips the BOM at the beginning of the file if it sees one (Lexer::InitLexer), and everything else works normally after that. The start of the first line is after the BOM, if any, which means it doesn't affect line/column numbers.

Are there potential problems we may run into because of the changing offsets? Could we add tests checking changing the offsets does not matter?

That's a good point; I've looked into it and the PCH for the preamble is parsed using just the buffer slice that contains the preamble, excluding any BOM. That means that when we resume parsing later on a main buffer with a BOM, the SourceLocations within the preamble itself will be off. However, normally this doesn't matter since the only things in the preamble are preprocessor directives, whose positions are very rarely used. (I should note at this point that we've been using a variant of this patch in production for a few years without any problem.) So, we have two choices: Either parse the preamble with the BOM and throw out the preamble/PCH when the BOM presence changes from the main buffer, or slice the buffer when using a preamble PCH so that it never has a BOM during parsing. I'm leaning towards the second option, since it's a little cleaner and lets the preamble be reused more easily; the only downside is that an external consumer would not be able to use any absolute offsets from the AST (note that line/column offsets would be identical) in the original buffer if it has a BOM -- but in any case, absolute offsets are usually useless without the buffer itself, which if obtained from clang would always be the correct buffer.

Should we add checks that BOM was removed or added, but not changed? I would not expect preamble to be reusable "as is" if BOM (and therefore, input encoding) changed.

I'm not sure I understand this point. Clang only understands UTF-8; the BOM is either present or not, but the encoding never changes. (And the BOM itself is always the same byte sequence too.) It has no impact on the file contents.

include/clang/Frontend/PrecompiledPreamble.h

102

Fair point, I'll change this.

191

Again, good point, I'll change this.

include/clang/Lex/Lexer.h

50

I can see how this might be confusing. I'll add a comment.

639

SkipBytes moves relative to the current position, but the lexer skips the BOM implicitly on construction; I don't want to skip it twice. SetByteOffset is absolute, which makes it simple and clear to use without having to reason about implicit past state.

lib/Frontend/PrecompiledPreamble.cpp

195

I could; I think it makes sense to leave the wrapper, though, since the ASTUnit deals with the PrecompiledPreamble at its level of abstraction, and the PrecompiledPreamble deals with the lexer at its level of abstraction.

unittests/Frontend/PchPreambleTest.cpp

190

We are; if it wasn't reused, the header would have been opened again and the last assert on GetFileReadCount below would fail.

[ilya-biryukov]

In D37491#862160, @cameron314 wrote:

Are there potential problems we may run into because of the changing offsets? Could we add tests checking changing the offsets does not matter?

That's a good point; I've looked into it and the PCH for the preamble is parsed using just the buffer slice that contains the preamble, excluding any BOM. That means that when we resume parsing later on a main buffer with a BOM, the SourceLocations within the preamble itself will be off. However, normally this doesn't matter since the only things in the preamble are preprocessor directives, whose positions are very rarely used. (I should note at this point that we've been using a variant of this patch in production for a few years without any problem.) So, we have two choices: Either parse the preamble with the BOM and throw out the preamble/PCH when the BOM presence changes from the main buffer, or slice the buffer when using a preamble PCH so that it never has a BOM during parsing. I'm leaning towards the second option, since it's a little cleaner and lets the preamble be reused more easily; the only downside is that an external consumer would not be able to use any absolute offsets from the AST (note that line/column offsets would be identical) in the original buffer if it has a BOM -- but in any case, absolute offsets are usually useless without the buffer itself, which if obtained from clang would always be the correct buffer.

Maybe there's a third option option to remove the BOM from the buffer before passing it to clang?
Could you elaborate on your use-case a little more? Is there no way to consistently always pass buffers either with or without BOM?

Out of two options you mention discarding preamble on BOM changes seems like an easy option that is both correct and won't make a difference in performance since BOM rarely changes.
Looking at your use-case, it sounds like you'll only have 1 extra reparse of preamble, which is probably fine.

Should we add checks that BOM was removed or added, but not changed? I would not expect preamble to be reusable "as is" if BOM (and therefore, input encoding) changed.

I'm not sure I understand this point. Clang only understands UTF-8; the BOM is either present or not, but the encoding never changes. (And the BOM itself is always the same byte sequence too.) It has no impact on the file contents.

Sure, it's not something clang supports, it's an edge-case when clang receives "malformed" input. Does lexer only skip utf-8 BOM, but not other versions of BOM?
But you're right, it's highly unlikely anything will break in that case.

unittests/Frontend/PchPreambleTest.cpp

190

Missed that, thanks. Looks good.
Maybe add a comment explicitly noting that?

[cameron314]

Maybe there's a third option option to remove the BOM from the buffer before passing it to clang?
Could you elaborate on your use-case a little more? Is there no way to consistently always pass buffers either with or without BOM?
Out of two options you mention discarding preamble on BOM changes seems like an easy option that is both correct and won't make a difference in performance since BOM rarely changes.
Looking at your use-case, it sounds like you'll only have 1 extra reparse of preamble, which is probably fine.

In my particular use case, when the file is remapped by the IDE, there's never a BOM. But when it's not remapped, the real file may or may not have a BOM. Since the file goes back and forth between mapped and unmapped depending on whether it's saved, the BOM presence can change quite frequently, and we don't really have control over it (the BOM can change on disk too). This is a common use case for anyone integrating clang/libclang into an IDE; the rarity of UTF-8 BOMs on platforms other than Windows probably obscured this until now.

I think since we can handle the changing BOM presence in the preamble gracefully, we should. I'll draft a patch that does the slicing correctly so that the offsets are always valid.

Sure, it's not something clang supports, it's an edge-case when clang receives "malformed" input. Does lexer only skip utf-8 BOM, but not other versions of BOM?
But you're right, it's highly unlikely anything will break in that case.

Ah, I see. Yes, the lexer only skips a UTF-8 BOM, but I seem to recall seeing some code that detects BOMs in other encodings and emits an error (in the driver, possibly?).

unittests/Frontend/PchPreambleTest.cpp

190

Sure, will do.

[cameron314]

Here's an updated patch. The code required to make it work is much simpler when the BOM is simply ignored :-)

[ilya-biryukov]

Parsing errors on preamble additions and removals are definitely bad and should be fixed.
But I would argue that the right approach is to invalidate the preamble and rebuild it on BOM changes.

Current fix in ASTUnit just hides an error in the underlying APIs. For example, all other other clients of PrecompiledPreamble are still broken.

lib/Frontend/ASTUnit.cpp

1262

It seems that having only this chunk would fix your issue.
Everything else is just a non-functional refactoring, maybe let's focus on that part (and tests) in this review and send the rest as a separate change?
To keep refactoring and functional changes logically separate.

[erikjv]

Will this fix PR25023 and PR21144?

[ilya-biryukov]

In D37491#864649, @erikjv wrote:

Will this fix PR25023 and PR21144?

PR25023 should be fixed by this change. It is essentially a repro of the same bug.
Could we add a c-index-test-based test here to make sure we addressed that particular use-case?

The state of PR21144 won't be affected, as this change does not touch the code invoked during normal compilation without preambles.
If PR21144 is fixed in a way that would make SourceLocations the same regardless if BOM was present or not, we might have a better guarantee that nothing will break in case we want to reuse preamble between BOM/non-BOM versions.

[cameron314]

It seems there's other users of PrecompiledPreamble that would have to be fixed, yes. If we go with my original fix of taking into account the BOM in the preamble bounds, there's no way of reusing the PCH when the BOM appears/disappears. I still maintain this is a common use case for IDE-type clients. This type of performance bug is very hard to track down.

@erikjv: Yes, I think this will fix PR25023.
PR21144 is unrelated; clang uses UTF-8 byte offsets instead of logical-character offsets for column numbers, which makes sense to me.

lib/Frontend/ASTUnit.cpp

1262

Yes, with this form of the fix, the other changes are mostly cosmetic. I could simply revert them, it's not worth the hassle of submitting another patch.

[cameron314]

Alright, I've changed the patch so that the preamble takes into account the BOM presence and is invalidated when it changes. This automatically fixes all clients of PrecompiledPreamble, and ensures that all SourceLocations are always consistent when using a PCH generated from the preamble.

I think this should do the trick!

[ilya-biryukov]

See my comments about removing StartOffset field, but other than that looks good.

include/clang/Lex/Lexer.h

52

We could simplify it further by removing StartOffset, leaving only Size.
If you look at the code, it always uses StartOffset + Size now, which is effectively size with BOM.
What do you think?

lib/Frontend/PrecompiledPreamble.cpp

227

Maybe store BOM bytes in PreambleBytes too? Would that allow to get rid of StartOffset field (see other comment)?

[cameron314]

include/clang/Lex/Lexer.h

52

Yeah, I thought of that, but it's still nice to have the two separated.

ASTUnit.cpp, for example, checks if Size != 0 to determine if there's a preamble (without considering the offset). This isn't in the diff because it was already like that.

[ilya-biryukov]

include/clang/Lex/Lexer.h

52

Could we simply return Size = 0 from ComputePreambleBounds if we simply skipped BOM and the preamble itself is empty?
My concern is that currently it's very easy to forget adding StartOffset and simply use Size when writing code that uses PreambleBounds. If we only have Size, probability of mistakes is much lower.

[cameron314]

include/clang/Lex/Lexer.h

52

Alright, sold. There's already other places that use the size without checking the offset, it turns out.

[cameron314]

Final diff. Test passes!

include/clang/Frontend/PrecompiledPreamble.h

 class FileSystem;
 }
 
 class CompilerInstance;
 class CompilerInvocation;
 class DeclGroupRef;
 class PCHContainerOperations;
 
-/// A size of the preamble and a flag required by
-/// PreprocessorOptions::PrecompiledPreambleBytes.
-struct PreambleBounds {
-  PreambleBounds(unsigned Size, bool PreambleEndsAtStartOfLine)
-      : Size(Size), PreambleEndsAtStartOfLine(PreambleEndsAtStartOfLine) {}
-
-  /// \brief Size of the preamble in bytes.
-  unsigned Size;
-  /// \brief Whether the preamble ends at the start of a new line.
-  ///
-  /// Used to inform the lexer as to whether it's starting at the beginning of
-  /// a line after skipping the preamble.
-  bool PreambleEndsAtStartOfLine;
-};
-
 /// \brief Runs lexer to compute suggested preamble bounds.
 PreambleBounds ComputePreambleBounds(const LangOptions &LangOpts,
                                      llvm::MemoryBuffer *Buffer,
                                      unsigned MaxLines);
 
 class PreambleCallbacks;
 
 /// A class holding a PCH and all information to check whether it is valid to
bool CanReuse(const CompilerInvocation &Invocation,
                 const llvm::MemoryBuffer *MainFileBuffer, PreambleBounds Bounds,
                 vfs::FileSystem *VFS) const;
 
   /// Changes options inside \p CI to use PCH from this preamble. Also remaps
   /// main file to \p MainFileBuffer.
   void AddImplicitPreamble(CompilerInvocation &CI,
                            llvm::MemoryBuffer *MainFileBuffer) const;
 
+  /// Updates the preamble offset. This is necessary when the preamble can be
+  /// reused for another parse, but its offset has changed (e.g. when a BOM
+  /// appears or disappears between parses).
+  void UpdateOffset(unsigned NewOffset);
+
 private:
   PrecompiledPreamble(TempPCHFile PCHFile, std::vector<char> PreambleBytes,
                       bool PreambleEndsAtStartOfLine,
-                      llvm::StringMap<PreambleFileHash> FilesInPreamble);
+                      llvm::StringMap<PreambleFileHash> FilesInPreamble,
+                      unsigned PreambleOffset);
 
   /// A temp file that would be deleted on destructor call. If destructor is not
   /// called for any reason, the file will be deleted at static objects'
   /// destruction.
   /// An assertion will fire if two TempPCHFiles are created with the same name,
   /// so it's not intended to be used outside preamble-handling.
   class TempPCHFile {
   public:
private:
   /// the preamble must be thrown away.
   llvm::StringMap<PreambleFileHash> FilesInPreamble;
   /// The contents of the file that was used to precompile the preamble. Only
   /// contains first PreambleBounds::Size bytes. Used to compare if the relevant
   /// part of the file has not changed, so that preamble can be reused.
   std::vector<char> PreambleBytes;
   /// See PreambleBounds::PreambleEndsAtStartOfLine
   bool PreambleEndsAtStartOfLine;
+  /// The start offset of the bounds used to build the preamble.
+  unsigned PreambleOffset;
 };
 
 /// A set of callbacks to gather useful information while building a preamble.
 class PreambleCallbacks {
 public:
   virtual ~PreambleCallbacks() = default;
 
   /// Called after FrontendAction::Execute(), but before

include/clang/Lex/Lexer.h

enum ConflictMarkerKind {
   /// A normal or diff3 conflict marker, initiated by at least 7 "<"s,
   /// separated by at least 7 "="s or "|"s, and terminated by at least 7 ">"s.
   CMK_Normal,
   /// A Perforce-style conflict marker, initiated by 4 ">"s,
   /// separated by 4 "="s, and terminated by 4 "<"s.
   CMK_Perforce
 };
 
+/// Describes the bounds (start, size) of the preamble and a flag required by
+/// PreprocessorOptions::PrecompiledPreambleBytes.
+struct PreambleBounds {
+  PreambleBounds(unsigned Offset, unsigned Size, bool PreambleEndsAtStartOfLine)
+    : StartOffset(Offset), Size(Size),
+      PreambleEndsAtStartOfLine(PreambleEndsAtStartOfLine) {}
+
+  /// \brief Start offset of the preamble, in bytes.
+  unsigned StartOffset;
+  /// \brief Size of the preamble in bytes.
+  unsigned Size;
+  /// \brief Whether the preamble ends at the start of a new line.
+  ///
+  /// Used to inform the lexer as to whether it's starting at the beginning of
+  /// a line after skipping the preamble.
+  bool PreambleEndsAtStartOfLine;
+};
+
 /// Lexer - This provides a simple interface that turns a text buffer into a
 /// stream of tokens.  This provides no support for file reading or buffering,
 /// or buffering/seeking of tokens, only forward lexing is supported.  It relies
 /// on the specified Preprocessor object to handle preprocessor directives, etc.
 class Lexer : public PreprocessorLexer {
   void anchor() override;
 
   //===--------------------------------------------------------------------===//
public:
   /// particular file actually begins. The preamble of the main source file is
   /// a potential prefix header.
   ///
   /// \param Buffer The memory buffer containing the file's contents.
   ///
   /// \param MaxLines If non-zero, restrict the length of the preamble
   /// to fewer than this number of lines.
   ///
-  /// \returns The offset into the file where the preamble ends and the rest
-  /// of the file begins along with a boolean value indicating whether 
-  /// the preamble ends at the beginning of a new line.
-  static std::pair<unsigned, bool> ComputePreamble(StringRef Buffer,
-                                                   const LangOptions &LangOpts,
-                                                   unsigned MaxLines = 0);
+  /// \returns The offset into the file where the preamble starts, its size,
+  /// and a boolean value indicating whether the preamble ends at the
+  /// beginning of a new line.
+  static PreambleBounds ComputePreamble(StringRef Buffer,
 
   /// \brief Checks that the given token is the first token that occurs after
   /// the given location (this excludes comments and whitespace). Returns the
   /// location immediately after the specified token. If the token is not found
   /// or the location is inside a macro, the returned source location will be
   /// invalid.
   static SourceLocation findLocationAfterToken(SourceLocation loc,
                                          tok::TokenKind TKind,
private:
   /// getCharAndSizeSlowNoWarn - Same as getCharAndSizeSlow, but never emits a
   /// diagnostic.
   static char getCharAndSizeSlowNoWarn(const char *Ptr, unsigned &Size,
                                        const LangOptions &LangOpts);
 
   //===--------------------------------------------------------------------===//
   // Other lexer functions.
 
-  void SkipBytes(unsigned Bytes, bool StartOfLine);
+  void SetByteOffset(unsigned Offset, bool StartOfLine);
 
   void PropagateLineStartLeadingSpaceInfo(Token &Result);
 
   const char *LexUDSuffix(Token &Result, const char *CurPtr,
                           bool IsStringLiteral);
 
   // Helper functions to lex the remainder of a token of the specific type.
   bool LexIdentifier         (Token &Result, const char *CurPtr);

include/clang/Lex/PreprocessorOptions.h

public:
   /// build it again.
   std::shared_ptr<FailedModulesSet> FailedModules;
 
 public:
   PreprocessorOptions() : UsePredefines(true), DetailedRecord(false),
                           DisablePCHValidation(false),
                           AllowPCHWithCompilerErrors(false),
                           DumpDeserializedPCHDecls(false),
-                          PrecompiledPreambleBytes(0, true),
+                          PrecompiledPreambleBytes(0, false),
                           GeneratePreamble(false),
                           RemappedFilesKeepOriginalName(true),
                           RetainRemappedFileBuffers(false),
                           ObjCXXARCStandardLibrary(ARCXX_nolib) { }
 
   void addMacroDef(StringRef Name) { Macros.emplace_back(Name, false); }
   void addMacroUndef(StringRef Name) { Macros.emplace_back(Name, true); }
   void addRemappedFile(StringRef From, StringRef To) {
void resetNonModularOptions() {
     DumpDeserializedPCHDecls = false;
     ImplicitPCHInclude.clear();
     ImplicitPTHInclude.clear();
     TokenCache.clear();
     SingleFileParseMode = false;
     LexEditorPlaceholders = true;
     RetainRemappedFileBuffers = true;
     PrecompiledPreambleBytes.first = 0;
-    PrecompiledPreambleBytes.second = 0;
+    PrecompiledPreambleBytes.second = false;
   }
 };
 
 } // end namespace clang
 
 #endif

lib/Frontend/ASTUnit.cpp

std::unique_ptr<llvm::MemoryBuffer> MainFileBuffer =
       getBufferForFileHandlingRemapping(PreambleInvocationIn, VFS.get(),
                                         MainFilePath);
   if (!MainFileBuffer)
     return nullptr;
 
   PreambleBounds Bounds =
       ComputePreambleBounds(*PreambleInvocationIn.getLangOpts(),
                             MainFileBuffer.get(), MaxLines);
   if (!Bounds.Size)
     return nullptr;
 
   if (Preamble) {
     if (Preamble->CanReuse(PreambleInvocationIn, MainFileBuffer.get(), Bounds,
                            VFS.get())) {
       // Okay! We can re-use the precompiled preamble.
+      Preamble->UpdateOffset(Bounds.StartOffset);
 
       // Set the state of the diagnostic object to mimic its state
       // after parsing the preamble.
       getDiagnostics().Reset();
       ProcessWarningOptions(getDiagnostics(),
                             PreambleInvocationIn.getDiagnosticOpts());
       getDiagnostics().setNumWarnings(NumWarningsInPreamble);
 

lib/Frontend/FrontendActions.cpp

void PrintPreambleAction::ExecuteAction() {
 
   // We don't expect to find any #include directives in a preprocessed input.
   if (getCurrentFileKind().isPreprocessed())
     return;
 
   CompilerInstance &CI = getCompilerInstance();
   auto Buffer = CI.getFileManager().getBufferForFile(getCurrentFile());
   if (Buffer) {
-    unsigned Preamble =
-        Lexer::ComputePreamble((*Buffer)->getBuffer(), CI.getLangOpts()).first;
+    PreambleBounds Bounds =
+        Lexer::ComputePreamble((*Buffer)->getBuffer(), CI.getLangOpts());
+    unsigned Preamble = Bounds.StartOffset + Bounds.Size;
     llvm::outs().write((*Buffer)->getBufferStart(), Preamble);
   }
 }

lib/Frontend/PrecompiledPreamble.cpp

template <class T> bool moveOnNoError(llvm::ErrorOr<T> Val, T &Output) {
   if (!Val)
     return false;
   Output = std::move(*Val);
   return true;
 }
 
 } // namespace
 
 PreambleBounds clang::ComputePreambleBounds(const LangOptions &LangOpts,
                                             llvm::MemoryBuffer *Buffer,
                                             unsigned MaxLines) {
-  auto Pre = Lexer::ComputePreamble(Buffer->getBuffer(), LangOpts, MaxLines);
-  return PreambleBounds(Pre.first, Pre.second);
+  return Lexer::ComputePreamble(Buffer->getBuffer(), LangOpts, MaxLines);
 }
 
 llvm::ErrorOr<PrecompiledPreamble> PrecompiledPreamble::Build(
     const CompilerInvocation &Invocation,
     const llvm::MemoryBuffer *MainFileBuffer, PreambleBounds Bounds,
     DiagnosticsEngine &Diagnostics, IntrusiveRefCntPtr<vfs::FileSystem> VFS,
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
     PreambleCallbacks &Callbacks) {
llvm::ErrorOr<PrecompiledPreamble> PrecompiledPreamble::Build(
   // circumstances, this can fail.
   llvm::ErrorOr<PrecompiledPreamble::TempPCHFile> PreamblePCHFile =
       PrecompiledPreamble::TempPCHFile::CreateNewPreamblePCHFile();
   if (!PreamblePCHFile)
     return BuildPreambleError::CouldntCreateTempFile;
 
   // Save the preamble text for later; we'll need to compare against it for
   // subsequent reparses.
-  std::vector<char> PreambleBytes(MainFileBuffer->getBufferStart(),
-                                  MainFileBuffer->getBufferStart() +
-                                      Bounds.Size);
+  auto PreambleStart = MainFileBuffer->getBufferStart() + Bounds.StartOffset;
+  std::vector<char> PreambleBytes(PreambleStart,
+                                  PreambleStart + Bounds.Size);
   bool PreambleEndsAtStartOfLine = Bounds.PreambleEndsAtStartOfLine;
 
   // Tell the compiler invocation to generate a temporary precompiled header.
   FrontendOpts.ProgramAction = frontend::GeneratePCH;
   // FIXME: Generate the precompiled header into memory?
   FrontendOpts.OutputFile = PreamblePCHFile->getFilePath();
   PreprocessorOpts.PrecompiledPreambleBytes.first = 0;
   PreprocessorOpts.PrecompiledPreambleBytes.second = false;
llvm::ErrorOr<PrecompiledPreamble> PrecompiledPreamble::Build(
   Clang->setSourceManager(
       new SourceManager(Diagnostics, Clang->getFileManager()));
 
   auto PreambleDepCollector = std::make_shared<DependencyCollector>();
   Clang->addDependencyCollector(PreambleDepCollector);
 
   // Remap the main source file to the preamble buffer.
   StringRef MainFilePath = FrontendOpts.Inputs[0].getFile();
+  auto StartOffset = Bounds.StartOffset;
+  auto EndOffset = StartOffset + Bounds.Size;
   auto PreambleInputBuffer = llvm::MemoryBuffer::getMemBufferCopy(
-      MainFileBuffer->getBuffer().slice(0, Bounds.Size), MainFilePath);
+      MainFileBuffer->getBuffer().slice(StartOffset, EndOffset), MainFilePath);
   if (PreprocessorOpts.RetainRemappedFileBuffers) {
     // MainFileBuffer will be deleted by unique_ptr after leaving the method.
     PreprocessorOpts.addRemappedFile(MainFilePath, PreambleInputBuffer.get());
   } else {
     // In that case, remapped buffer will be deleted by CompilerInstance on
     // BeginSourceFile, so we call release() to avoid double deletion.
     PreprocessorOpts.addRemappedFile(MainFilePath,
                                      PreambleInputBuffer.release());
if (time_t ModTime = File->getModificationTime()) {
       llvm::MemoryBuffer *Buffer = SourceMgr.getMemoryBufferForFile(File);
       FilesInPreamble[File->getName()] =
           PrecompiledPreamble::PreambleFileHash::createForMemoryBuffer(Buffer);
     }
   }
 
   return PrecompiledPreamble(
       std::move(*PreamblePCHFile), std::move(PreambleBytes),
-      PreambleEndsAtStartOfLine, std::move(FilesInPreamble));
+      PreambleEndsAtStartOfLine, std::move(FilesInPreamble),
+      Bounds.StartOffset);
 }
 
 PreambleBounds PrecompiledPreamble::getBounds() const {
-  return PreambleBounds(PreambleBytes.size(), PreambleEndsAtStartOfLine);
+  return PreambleBounds(PreambleOffset, PreambleBytes.size(),
+                        PreambleEndsAtStartOfLine);
 }
 
 bool PrecompiledPreamble::CanReuse(const CompilerInvocation &Invocation,
                                    const llvm::MemoryBuffer *MainFileBuffer,
                                    PreambleBounds Bounds,
                                    vfs::FileSystem *VFS) const {
 
   assert(
-      Bounds.Size <= MainFileBuffer->getBufferSize() &&
+      Bounds.StartOffset + Bounds.Size <= MainFileBuffer->getBufferSize() &&
       "Buffer is too large. Bounds were calculated from a different buffer?");
 
   auto PreambleInvocation = std::make_shared<CompilerInvocation>(Invocation);
   PreprocessorOptions &PreprocessorOpts =
       PreambleInvocation->getPreprocessorOpts();
 
   if (!Bounds.Size)
     return false;
 
   // We've previously computed a preamble. Check whether we have the same
   // preamble now that we did before, and that there's enough space in
   // the main-file buffer within the precompiled preamble to fit the
   // new main file.
   if (PreambleBytes.size() != Bounds.Size ||
       PreambleEndsAtStartOfLine != Bounds.PreambleEndsAtStartOfLine ||
-      memcmp(PreambleBytes.data(), MainFileBuffer->getBufferStart(),
+      memcmp(PreambleBytes.data(),
+             MainFileBuffer->getBufferStart() + Bounds.StartOffset,
              Bounds.Size) != 0)
     return false;
   // The preamble has not changed. We may be able to re-use the precompiled
   // preamble.
 
   // Check that none of the files used by the preamble have changed.
   // First, make a record of those files that have been overridden via
   // remapping or unsaved_files.
bool PrecompiledPreamble::CanReuse(const CompilerInvocation &Invocation,
   return true;
 }
 
 void PrecompiledPreamble::AddImplicitPreamble(
     CompilerInvocation &CI, llvm::MemoryBuffer *MainFileBuffer) const {
   auto &PreprocessorOpts = CI.getPreprocessorOpts();
 
   // Configure ImpicitPCHInclude.
-  PreprocessorOpts.PrecompiledPreambleBytes.first = PreambleBytes.size();
+  PreprocessorOpts.PrecompiledPreambleBytes.first = PreambleOffset +
+      PreambleBytes.size();
   PreprocessorOpts.PrecompiledPreambleBytes.second = PreambleEndsAtStartOfLine;
   PreprocessorOpts.ImplicitPCHInclude = PCHFile.getFilePath();
   PreprocessorOpts.DisablePCHValidation = true;
 
   // Remap main file to point to MainFileBuffer.
   auto MainFilePath = CI.getFrontendOpts().Inputs[0].getFile();
   PreprocessorOpts.addRemappedFile(MainFilePath, MainFileBuffer);
 }
 
+void PrecompiledPreamble::UpdateOffset(unsigned NewOffset)
+{
+  PreambleOffset = NewOffset;
+}
+
 PrecompiledPreamble::PrecompiledPreamble(
     TempPCHFile PCHFile, std::vector<char> PreambleBytes,
     bool PreambleEndsAtStartOfLine,
-    llvm::StringMap<PreambleFileHash> FilesInPreamble)
+    llvm::StringMap<PreambleFileHash> FilesInPreamble,
+    unsigned PreambleOffset)
     : PCHFile(std::move(PCHFile)), FilesInPreamble(FilesInPreamble),
       PreambleBytes(std::move(PreambleBytes)),
-      PreambleEndsAtStartOfLine(PreambleEndsAtStartOfLine) {}
+      PreambleEndsAtStartOfLine(PreambleEndsAtStartOfLine),
+      PreambleOffset(PreambleOffset) {}
 
 llvm::ErrorOr<PrecompiledPreamble::TempPCHFile>
 PrecompiledPreamble::TempPCHFile::CreateNewPreamblePCHFile() {
   // FIXME: This is a hack so that we can override the preamble file during
   // crash-recovery testing, which is the only case where the preamble files
   // are not necessarily cleaned up.
   const char *TmpFile = ::getenv("CINDEXTEST_PREAMBLE_FILE");
   if (TmpFile)

lib/Lex/Lexer.cpp

namespace {
 
   enum PreambleDirectiveKind {
     PDK_Skipped,
     PDK_Unknown
   };
 
 } // end anonymous namespace
 
-std::pair<unsigned, bool> Lexer::ComputePreamble(StringRef Buffer,
-                                                 const LangOptions &LangOpts,
-                                                 unsigned MaxLines) {
+PreambleBounds Lexer::ComputePreamble(StringRef Buffer,
   // Create a lexer starting at the beginning of the file. Note that we use a
   // "fake" file source location at offset 1 so that the lexer will track our
   // position within the file.
   const unsigned StartOffset = 1;
   SourceLocation FileLoc = SourceLocation::getFromRawEncoding(StartOffset);
   Lexer TheLexer(FileLoc, LangOpts, Buffer.begin(), Buffer.begin(),
                  Buffer.end());
   TheLexer.SetCommentRetentionState(true);
PreambleBounds Lexer::ComputePreamble(StringRef Buffer,
   } while (true);
   
   SourceLocation End;
   if (ActiveCommentLoc.isValid())
     End = ActiveCommentLoc; // don't truncate a decl comment.
   else
     End = TheTok.getLocation();
 
-  return std::make_pair(End.getRawEncoding() - StartLoc.getRawEncoding(),
+  return PreambleBounds(StartLoc.getRawEncoding() - FileLoc.getRawEncoding(),
+                        End.getRawEncoding() - StartLoc.getRawEncoding(),
                         TheTok.isAtStartOfLine());
 }
 
 /// AdvanceToTokenCharacter - Given a location that specifies the start of a
 /// token, return a new location that specifies a character within the token.
 SourceLocation Lexer::AdvanceToTokenCharacter(SourceLocation TokStart,
                                               unsigned CharNo,
                                               const SourceManager &SM,
Slash:
   ++Size;
   return *Ptr;
 }
 
 //===----------------------------------------------------------------------===//
 // Helper methods for lexing.
 //===----------------------------------------------------------------------===//
 
-/// \brief Routine that indiscriminately skips bytes in the source file.
-void Lexer::SkipBytes(unsigned Bytes, bool StartOfLine) {
-  BufferPtr += Bytes;
+/// \brief Routine that indiscriminately sets the offset into the source file.
+void Lexer::SetByteOffset(unsigned Offset, bool StartOfLine) {
+  BufferPtr = BufferStart + Offset;
   if (BufferPtr > BufferEnd)
     BufferPtr = BufferEnd;
   // FIXME: What exactly does the StartOfLine bit mean?  There are two
   // possible meanings for the "start" of the line: the first token on the
   // unexpanded line, or the first token on the expanded line.
   IsAtStartOfLine = StartOfLine;
   IsAtPhysicalStartOfLine = StartOfLine;
 }

lib/Lex/Preprocessor.cpp

void Preprocessor::EnterMainSourceFile() {
   // a main file.
   if (!SourceMgr.isLoadedFileID(MainFileID)) {
     // Enter the main file source buffer.
     EnterSourceFile(MainFileID, nullptr, SourceLocation());
 
     // If we've been asked to skip bytes in the main file (e.g., as part of a
     // precompiled preamble), do so now.
     if (SkipMainFilePreamble.first > 0)
-      CurLexer->SkipBytes(SkipMainFilePreamble.first, 
+      CurLexer->SetByteOffset(SkipMainFilePreamble.first,
                               SkipMainFilePreamble.second);
-    
     // Tell the header info that the main file was entered.  If the file is later
     // #imported, it won't be re-entered.
     if (const FileEntry *FE = SourceMgr.getFileEntryForID(MainFileID))
       HeaderInfo.IncrementIncludeCount(FE);
   }
 
   // Preprocess Predefines to populate the initial preprocessor state.
   std::unique_ptr<llvm::MemoryBuffer> SB =

unittests/Frontend/PchPreambleTest.cpp

std::unique_ptr<ASTUnit> ParseAST(const std::string &EntryFile) {
 
     CI->getTargetOpts().Triple = "i386-unknown-linux-gnu";
 
     CI->getPreprocessorOpts().RemappedFileBuffers = GetRemappedFiles();
 
     PreprocessorOptions &PPOpts = CI->getPreprocessorOpts();
     PPOpts.RemappedFilesKeepOriginalName = true;
 
+    struct DebugConsumer : DiagnosticConsumer
+    {
+      void HandleDiagnostic(DiagnosticsEngine::Level DiagLevel,
+        const Diagnostic &Info) override {
+        DiagnosticConsumer::HandleDiagnostic(DiagLevel, Info);
+        SmallString<16> DiagText;
+        Info.FormatDiagnostic(DiagText);
+        llvm::errs() << DiagText << '\n';
+      }
+    };
+
     IntrusiveRefCntPtr<DiagnosticsEngine>
-      Diags(CompilerInstance::createDiagnostics(new DiagnosticOptions, new DiagnosticConsumer));
+      Diags(CompilerInstance::createDiagnostics(new DiagnosticOptions, new DebugConsumer));
 
     FileManager *FileMgr = new FileManager(FSOpts, VFS);
 
     std::unique_ptr<ASTUnit> AST = ASTUnit::LoadFromCompilerInvocation(
       CI, PCHContainerOpts, Diags, FileMgr, false, false,
       /*PrecompilePreambleAfterNParses=*/1);
     return AST;
   }
TEST_F(PchPreambleTest, ReparseWithOverriddenFileDoesNotInvalidatePreamble) {
 
   ASSERT_TRUE(ReparseAST(AST));
 
   ASSERT_NE(initialCounts[0], GetFileReadCount(MainName));
   ASSERT_EQ(initialCounts[1], GetFileReadCount(Header1));
   ASSERT_EQ(initialCounts[2], GetFileReadCount(Header2));
 }
 
+TEST_F(PchPreambleTest, ReparseWithDisappearingBom) {
+  std::string Header = "//./header.h";
+  std::string Main = "//./main.cpp";
+  AddFile(Header, "int random() { return 4; }");
+  AddFile(Main,
+    "\xef\xbb\xbf"
+    "#include \"//./header.h\"\n"
+    "int main() { return random() * 2; }");
+
+  std::unique_ptr<ASTUnit> AST(ParseAST(Main));
+  ASSERT_TRUE(AST.get());
+  ASSERT_FALSE(AST->getDiagnostics().hasErrorOccurred());
+
+  unsigned initialCounts[] = {
+    GetFileReadCount(Main),
+    GetFileReadCount(Header)
+  };
+
+  // Remove BOM but keep the rest the same
+  RemapFile(Main,
+    "#include \"//./header.h\"\n"
+    "int main() { return random() * 2; }");
+
+  ASSERT_TRUE(ReparseAST(AST));
+  ASSERT_FALSE(AST->getDiagnostics().hasErrorOccurred());
+
+  ASSERT_EQ(initialCounts[0], GetFileReadCount(Main));
+  ASSERT_EQ(initialCounts[1], GetFileReadCount(Header));
+}
+
+TEST_F(PchPreambleTest, ReparseWithAppearingBom) {
+  std::string Header = "//./header.h";
+  std::string Main = "//./main.cpp";
+  AddFile(Header, "int random() { return 4; }");
+  AddFile(Main,
+    "#include \"//./header.h\"\n"
+    "int main() { return random() * 2; }");
+
+  std::unique_ptr<ASTUnit> AST(ParseAST(Main));
+  ASSERT_TRUE(AST.get());
+  ASSERT_FALSE(AST->getDiagnostics().hasErrorOccurred());
+
+  unsigned initialCounts[] = {
+    GetFileReadCount(Main),
+    GetFileReadCount(Header)
+  };
+
+  // Suddenly, a BOM appeared
+  RemapFile(Main,
+    "\xef\xbb\xbf"
+    "#include \"//./header.h\"\n"
+    "int main() { return random() * 2; }");
+
+  ASSERT_TRUE(ReparseAST(AST));
+  ASSERT_FALSE(AST->getDiagnostics().hasErrorOccurred());
+
+  ASSERT_EQ(initialCounts[0], GetFileReadCount(Main));
+  ASSERT_EQ(initialCounts[1], GetFileReadCount(Header));
+}
+
 } // anonymous namespace