PTH-- Remove feature entirely-

[erichkeane]

Summary

When debugging a boost build with a modified
version of Clang, I discovered that the PTH implementation
stores TokenKind in 8 bits. However, we currently have 368
TokenKinds.

The result is that the value gets truncated and the wrong token
gets picked up when including PTH files. It seems that this will
go wrong every time someone uses a token that uses the 9th bit.

Upon asking on IRC, it was brought up that this was a highly
experimental features that was considered a failure. I discovered
via googling that BoostBuild (mostly Boost.Math) is the only user of this
feature, using the CC1 flag directly. I believe that this can be
transferred over to normal PCH with minimal effort:
https://github.com/boostorg/build/issues/367

Based on advice on IRC and research showing that this is a nearly
completely unused feature, this patch removes it entirely.

Note: I considered leaving the build-flags in place and making them
emit an error/warning, however since I've basically identified and
warned the only user, it seemed better to just remove them.

[erichkeane]

include/clang/Driver/Options.td

261

The default behavior is exactly to use this flag. I removed the flag since googling showed that it is never really used.

lib/Frontend/CompilerInstance.cpp

385

IdentifierInfoLookup is an abstraction around ASTReader and (previously) PTHManager. It seems that this abstraction could still be useful, so I've chosen to leave it in place.

When PTH wasn't enabled, PTHMgr was nullptr anyway, so this just uses that instead.

unittests/Tooling/DiagnosticsYamlTest.cpp

85 ↗

(On Diff #174103)

I'm unsure about what caused this change. It seems to me that these two should be equivalent, and hopefully someone will correct me if I'm wrong.

[chandlerc]

Should likely add release notes about this.

Also might be worth sending a note to cfe-dev as a heads up and give folks some time to say "wait wait".

[aaron.ballman]

In D54547#1299105, @chandlerc wrote:

Should likely add release notes about this.

Also might be worth sending a note to cfe-dev as a heads up and give folks some time to say "wait wait".

+1 to both of these points, but if doesn't cause too big of a burden for folks, I'm all in favor of it.

[riccibruno]

I don't think I can comment more broadly on this, but
there is some PTH-related code in Basic/IdentifierTable,
in IdentifierInfo::getNameStart and IdentifierInfo::getLength.
Maybe this should be removed too ?

[erichkeane]

Added to the release notes. Also, an email was sent out to cfe-dev.

@riccibruno and I are separately looking into IdentifierInfo, because it seems that there are some pretty massive optimization opportunities if we can remove PTH/indirect identifiers.

[riccibruno]

In D54547#1299883, @erichkeane wrote:

Added to the release notes. Also, an email was sent out to cfe-dev.

@riccibruno and I are separately looking into IdentifierInfo, because it seems that there are some pretty massive optimization opportunities if we can remove PTH/indirect identifiers.

As discussed on IRC:
In any case this is not blocking the removal of PTH. The code around IdentifierInfo
is very performance sensitive and it will take some time to evaluate the possible
changes, if any.

[arphaman]

Would it be better to deprecate the use of PTH right now for the current release, so that the users will be aware that we will remove PTH support in LLVM 9 once they upgrade to LLVM 8? Then we can remove it right after LLVM 8 branch is created.

[erichkeane]

In D54547#1301253, @arphaman wrote:

Would it be better to deprecate the use of PTH right now for the current release, so that the users will be aware that we will remove PTH support in LLVM 9 once they upgrade to LLVM 8? Then we can remove it right after LLVM 8 branch is created.

I considered that. However, it is:
A- Really broken
B- Only has 1 user
C- Only accessible by a cc1 option, users aren't supposed to use these anyway.

Because of this, I/we seem to have little sympathy for the users.

[voskresensky.vladimir]

I have some experience with PTH implementation, because had to fix it for Java-port of Clang (https://github.com/java-port/clank).

It was sometime ago, but making it completely workable was not hard.
As I remember the key fix was just to have PTH be EMITTED using raw-Lex mode (where real keyword-IDs are not used, so all Token Kinds nicely fit 8-bits).
It worked, because token automatically became keyword by PTHLexer::Lex:
...

   // Change the kind of this identifier to the appropriate token kind, e.g.
   // turning "for" into a keyword.
Tok.setKind(II->getTokenID());

...

We used PTH, because multiple translation units are parsed in the context of single run.
In this case preprocessing phase was upto 10x faster when token stream was deserialized from PTH (i.e. for all system headers lexed once).
Also it was helpful, because we were able to parse from concurrent threads using the same shared immutable mmaped PTH.

I thought clang-d service is using it to speed up indexing.

[erichkeane]

In D54547#1301351, @voskresensky.vladimir wrote:

I have some experience with PTH implementation, because had to fix it for Java-port of Clang (https://github.com/java-port/clank).

It was sometime ago, but making it completely workable was not hard.
As I remember the key fix was just to have PTH be EMITTED using raw-Lex mode (where real keyword-IDs are not used, so all Token Kinds nicely fit 8-bits).
It worked, because on token automatically became keyword by PTHLexer::Lex:
...

   // Change the kind of this identifier to the appropriate token kind, e.g.
   // turning "for" into a keyword.
Tok.setKind(II->getTokenID());

...

We used PTH, because multiple translation units are parsed in the context of single run.
In this case preprocessing phase was upto 10x faster when token stream was deserialized from PTH (i.e. for all system headers lexed once).
Also it was helpful, because we were able to parse from concurrent threads using the same shared immutable mmaped PTH.

I thought clang-d service is using it to speed up indexing.

Presumably, I could also just make PTH use another bit or two for the TokenID and it would work fine. However, when I mentioned this on IRC the general response was that it is a 'failed experiment'. In your use case, I wonder why you couldn't just use PCH and get even further performance improvements?

[riccibruno]

I thought clang-d service is using it to speed up indexing.

I believe that clangd is using PCH and not PTH.
(by the highly sophisticated method of grepping for pth and pch inside clangd/)

[voskresensky.vladimir]

In D54547#1301359, @erichkeane wrote:

I thought clang-d service is using it to speed up indexing.

Presumably, I could also just make PTH use another bit or two for the TokenID and it would work fine. However, when I mentioned this on IRC the general response was that it is a 'failed experiment'. In your use case, I wonder why you couldn't just use PCH and get even further performance improvements?

To be fair, I don't remember exactly. :-)
I remember we shared the same PTH for all C and C++ files and built some preprocessor-based features for incomplete code written in editor based on that.

Hmm... One more thing I had to fix was: lex in mode where we emit all preprocessor tokens there as well (like "#define MACRO A" => #, define, MACRO, A)

PHT was also used for rebuilding PCH, when something is changed. Because often single changed file can completely invalidate PCH, while PTH just contain pp-lexed tokens which doesn't carry semantic. So, PTHManager could skip providing tokens for changed file, but keep provide them for all other files.

[riccibruno]

I looked at how ASTReader create the IdentifierInfos returned by IdentifierInfo *ASTReader::get(StringRef Name),
and I ended up in ASTIdentifierLookupTrait::ReadData, which calls among other things IdentifierTable::getOwn.
The IdentifierInfo created by getOwn will always have the Entry pointer set and therefore I think it is
possible to remove the if (Entry) in IdentifierInfo::getLength and IdentifierInfo::getNameStart.

I made a patch implementing this change here D54866.

include/clang/Lex/Preprocessor.h

396

Would it make sense to remove this alias now that it
will always refer to CurLexer ?

unittests/Tooling/DiagnosticsYamlTest.cpp

85 ↗

(On Diff #174212)

It was fixed in r346884 : [Support] Teach YAMLIO about polymorphic types
so you can remove this from your patch.

[erichkeane]

include/clang/Lex/Preprocessor.h

396

There seems to be some logic around this that I'm not totally sure about dealing with IsFileLexer.

[riccibruno]

include/clang/Lex/Preprocessor.h

396

Sounds fine to me. Someone more familiar with this can
always come back later and rework this.

[jyknight]

Since you've already submitted a fix to Boost, https://github.com/boostorg/build/commit/3385fe2aa699a45e722a1013658f824b6a7c761f, I think this is fine to remove.

NOTES.txt

 //===---------------------------------------------------------------------===//
 // Random Notes
 //===---------------------------------------------------------------------===//
 
 //===---------------------------------------------------------------------===//
 
 To time GCC preprocessing speed without output, use:
    "time gcc -MM file"
 This is similar to -Eonly.
 
 //===---------------------------------------------------------------------===//
 
-Creating and using a PTH file for performance measurement (use a release build).
-
-$ clang -ccc-pch-is-pth -x objective-c-header INPUTS/Cocoa_h.m -o /tmp/tokencache
-$ clang -cc1 -token-cache /tmp/tokencache INPUTS/Cocoa_h.m
-
-//===---------------------------------------------------------------------===//
-
   C++ Template Instantiation benchmark:
      http://users.rcn.com/abrahams/instantiation_speed/index.html
 
 //===---------------------------------------------------------------------===//
 
 TODO: File Manager Speedup:
 
  We currently do a lot of stat'ing for files that don't exist, particularly

docs/InternalsManual.rst

 The Driver Library
 ==================
 
 The clang Driver and library are documented :doc:`here <DriverInternals>`.
 
 Precompiled Headers
 ===================
 
-Clang supports two implementations of precompiled headers.  The default
-implementation, precompiled headers (:doc:`PCH <PCHInternals>`) uses a
+Clang supports precompiled headers (:doc:`PCH <PCHInternals>`), which  uses a
 serialized representation of Clang's internal data structures, encoded with the
 `LLVM bitstream format <https://llvm.org/docs/BitCodeFormat.html>`_.
-Pretokenized headers (:doc:`PTH <PTHInternals>`), on the other hand, contain a
-serialized representation of the tokens encountered when preprocessing a header
-(and anything that header includes).
 
 The Frontend Library
 ====================
 
 The Frontend library contains functionality useful for building tools on top of
 the Clang libraries, for example several methods for outputting diagnostics.
 
 The Lexer and Preprocessor Library

docs/PTHInternals.rst

-==========================
-Pretokenized Headers (PTH)
-==========================
-
-This document first describes the low-level interface for using PTH and
-then briefly elaborates on its design and implementation. If you are
-interested in the end-user view, please see the :ref:`User's Manual
-<usersmanual-precompiled-headers>`.
-
-Using Pretokenized Headers with ``clang`` (Low-level Interface)
-===============================================================
-
-The Clang compiler frontend, ``clang -cc1``, supports three command line
-options for generating and using PTH files.
-
-To generate PTH files using ``clang -cc1``, use the option ``-emit-pth``:
-
-.. code-block:: console
-
-  $ clang -cc1 test.h -emit-pth -o test.h.pth
-
-This option is transparently used by ``clang`` when generating PTH
-files. Similarly, PTH files can be used as prefix headers using the
-``-include-pth`` option:
-
-.. code-block:: console
-
-  $ clang -cc1 -include-pth test.h.pth test.c -o test.s
-
-Alternatively, Clang's PTH files can be used as a raw "token-cache" (or
-"content" cache) of the source included by the original header file.
-This means that the contents of the PTH file are searched as substitutes
-for *any* source files that are used by ``clang -cc1`` to process a
-source file. This is done by specifying the ``-token-cache`` option:
-
-.. code-block:: console
-
-  $ cat test.h
-  #include <stdio.h>
-  $ clang -cc1 -emit-pth test.h -o test.h.pth
-  $ cat test.c
-  #include "test.h"
-  $ clang -cc1 test.c -o test -token-cache test.h.pth
-
-In this example the contents of ``stdio.h`` (and the files it includes)
-will be retrieved from ``test.h.pth``, as the PTH file is being used in
-this case as a raw cache of the contents of ``test.h``. This is a
-low-level interface used to both implement the high-level PTH interface
-as well as to provide alternative means to use PTH-style caching.
-
-PTH Design and Implementation
-=============================
-
-Unlike GCC's precompiled headers, which cache the full ASTs and
-preprocessor state of a header file, Clang's pretokenized header files
-mainly cache the raw lexer *tokens* that are needed to segment the
-stream of characters in a source file into keywords, identifiers, and
-operators. Consequently, PTH serves to mainly directly speed up the
-lexing and preprocessing of a source file, while parsing and
-type-checking must be completely redone every time a PTH file is used.
-
-Basic Design Tradeoffs
-----------------------
-
-In the long term there are plans to provide an alternate PCH
-implementation for Clang that also caches the work for parsing and type
-checking the contents of header files. The current implementation of PCH
-in Clang as pretokenized header files was motivated by the following
-factors:
-
-**Language independence**
-   PTH files work with any language that
-   Clang's lexer can handle, including C, Objective-C, and (in the early
-   stages) C++. This means development on language features at the
-   parsing level or above (which is basically almost all interesting
-   pieces) does not require PTH to be modified.
-
-**Simple design**
-   Relatively speaking, PTH has a simple design and
-   implementation, making it easy to test. Further, because the
-   machinery for PTH resides at the lower-levels of the Clang library
-   stack it is fairly straightforward to profile and optimize.
-
-Further, compared to GCC's PCH implementation (which is the dominate
-precompiled header file implementation that Clang can be directly
-compared against) the PTH design in Clang yields several attractive
-features:
-
-**Architecture independence**
-   In contrast to GCC's PCH files (and
-   those of several other compilers), Clang's PTH files are architecture
-   independent, requiring only a single PTH file when building a
-   program for multiple architectures.
-
-   For example, on Mac OS X one may wish to compile a "universal binary"
-   that runs on PowerPC, 32-bit Intel (i386), and 64-bit Intel
-   architectures. In contrast, GCC requires a PCH file for each
-   architecture, as the definitions of types in the AST are
-   architecture-specific. Since a Clang PTH file essentially represents
-   a lexical cache of header files, a single PTH file can be safely used
-   when compiling for multiple architectures. This can also reduce
-   compile times because only a single PTH file needs to be generated
-   during a build instead of several.
-
-**Reduced memory pressure**
-   Similar to GCC, Clang reads PTH files
-   via the use of memory mapping (i.e., ``mmap``). Clang, however,
-   memory maps PTH files as read-only, meaning that multiple invocations
-   of ``clang -cc1`` can share the same pages in memory from a
-   memory-mapped PTH file. In comparison, GCC also memory maps its PCH
-   files but also modifies those pages in memory, incurring the
-   copy-on-write costs. The read-only nature of PTH can greatly reduce
-   memory pressure for builds involving multiple cores, thus improving
-   overall scalability.
-
-**Fast generation**
-   PTH files can be generated in a small fraction
-   of the time needed to generate GCC's PCH files. Since PTH/PCH
-   generation is a serial operation that typically blocks progress
-   during a build, faster generation time leads to improved processor
-   utilization with parallel builds on multicore machines.
-
-Despite these strengths, PTH's simple design suffers some algorithmic
-handicaps compared to other PCH strategies such as those used by GCC.
-While PTH can greatly speed up the processing time of a header file, the
-amount of work required to process a header file is still roughly linear
-in the size of the header file. In contrast, the amount of work done by
-GCC to process a precompiled header is (theoretically) constant (the
-ASTs for the header are literally memory mapped into the compiler). This
-means that only the pieces of the header file that are referenced by the
-source file including the header are the only ones the compiler needs to
-process during actual compilation. While GCC's particular implementation
-of PCH mitigates some of these algorithmic strengths via the use of
-copy-on-write pages, the approach itself can fundamentally dominate at
-an algorithmic level, especially when one considers header files of
-arbitrary size.
-
-There is also a PCH implementation for Clang based on the lazy
-deserialization of ASTs. This approach theoretically has the same
-constant-time algorithmic advantages just mentioned but also retains some
-of the strengths of PTH such as reduced memory pressure (ideal for
-multi-core builds).
-
-Internal PTH Optimizations
---------------------------
-
-While the main optimization employed by PTH is to reduce lexing time of
-header files by caching pre-lexed tokens, PTH also employs several other
-optimizations to speed up the processing of header files:
-
--  ``stat`` caching: PTH files cache information obtained via calls to
-   ``stat`` that ``clang -cc1`` uses to resolve which files are included
-   by ``#include`` directives. This greatly reduces the overhead
-   involved in context-switching to the kernel to resolve included
-   files.
-
--  Fast skipping of ``#ifdef`` ... ``#endif`` chains: PTH files
-   record the basic structure of nested preprocessor blocks. When the
-   condition of the preprocessor block is false, all of its tokens are
-   immediately skipped instead of requiring them to be handled by
-   Clang's preprocessor.
-
-

docs/ReleaseNotes.rst

.. code-block:: c++
              int y : S())
           ;
       }
 
 
 Non-comprehensive list of changes in this release
 -------------------------------------------------
 
-- ...
+- The experimental feature Pretokenized Headers (PTH) was removed in its
+  entirely from Clang. The feature did not properly work with about 1/3 of the
+  possible tokens available and was unmaintained.
 
 New Compiler Flags
 ------------------
 
 - ``-fprofile-filter-files=[regexes]`` and ``-fprofile-exclude-files=[regexes]``.
 
   Clang has now options to filter or exclude some files when
   instrumenting for gcov-based profiling.

include/clang/Basic/DiagnosticFrontendKinds.td

 def err_fe_unable_to_load_plugin : Error<
     "unable to load plugin '%0': '%1'">;
 def err_fe_unable_to_create_target : Error<
     "unable to create target: '%0'">;
 def err_fe_unable_to_interface_with_target : Error<
     "unable to interface with target machine">;
 def err_fe_unable_to_open_output : Error<
     "unable to open output file '%0': '%1'">;
-def err_fe_pth_file_has_no_source_header : Error<
-    "PTH file '%0' does not designate an original source header file for -include-pth">;
 def warn_fe_macro_contains_embedded_newline : Warning<
     "macro '%0' contains embedded newline; text after the newline is ignored">;
 def warn_fe_cc_print_header_failure : Warning<
     "unable to open CC_PRINT_HEADERS file: %0 (using stderr)">;
 def warn_fe_cc_log_diagnostics_failure : Warning<
     "unable to open CC_LOG_DIAGNOSTICS file: %0 (using stderr)">;
 def warn_fe_unable_to_open_stats_file : Warning<
     "unable to open statistics output file '%0': '%1'">,

include/clang/Basic/DiagnosticLexKinds.td

 def err_bad_character_encoding : Error<
   "illegal character encoding in character literal">;
 def warn_bad_character_encoding : ExtWarn<
   "illegal character encoding in character literal">,
   InGroup<InvalidSourceEncoding>;
 def err_lexing_string : Error<"failure when lexing a string">;
 def err_placeholder_in_source : Error<"editor placeholder in source file">;
 
-
-//===----------------------------------------------------------------------===//
-// PTH Diagnostics
-//===----------------------------------------------------------------------===//
-def err_invalid_pth_file : Error<
-    "invalid or corrupt PTH file '%0'">;
-
 //===----------------------------------------------------------------------===//
 // Preprocessor Diagnostics
 //===----------------------------------------------------------------------===//
 
 let CategoryName = "User-Defined Issue" in {
 def pp_hash_warning : Warning<"%0">,
   InGroup<PoundWarning>, ShowInSystemHeader;
 def err_pp_hash_error : Error<"%0">;

 def err_pp_module_begin_without_module_end : Error<
   "no matching '#pragma clang module end' for this "
   "'#pragma clang module begin'">;
 def err_pp_module_end_without_module_begin : Error<
   "no matching '#pragma clang module begin' for this "
   "'#pragma clang module end'">;
 def note_pp_module_begin_here : Note<
   "entering module '%0' due to this pragma">;
-def err_pp_module_build_pth : Error<
-  "'#pragma clang module build' not supported in pretokenized header">;
 def err_pp_module_build_missing_end : Error<
   "no matching '#pragma clang module endbuild' for this '#pragma clang module build'">;
 
 def err_defined_macro_name : Error<"'defined' cannot be used as a macro name">;
 def err_paste_at_start : Error<
   "'##' cannot appear at start of macro expansion">;
 def err_paste_at_end : Error<"'##' cannot appear at end of macro expansion">;
 def ext_paste_comma : Extension<

include/clang/Driver/CC1Options.td

 def ast_view : Flag<["-"], "ast-view">,
   HelpText<"Build ASTs and view them with GraphViz">;
 def emit_module : Flag<["-"], "emit-module">,
   HelpText<"Generate pre-compiled module file from a module map">;
 def emit_module_interface : Flag<["-"], "emit-module-interface">,
   HelpText<"Generate pre-compiled module file from a C++ module interface">;
 def emit_header_module : Flag<["-"], "emit-header-module">,
   HelpText<"Generate pre-compiled module file from a set of header files">;
-def emit_pth : Flag<["-"], "emit-pth">,
-  HelpText<"Generate pre-tokenized header file">;
 def emit_pch : Flag<["-"], "emit-pch">,
   HelpText<"Generate pre-compiled header file">;
 def emit_llvm_bc : Flag<["-"], "emit-llvm-bc">,
   HelpText<"Build ASTs then convert to LLVM, emit .bc file">;
 def emit_llvm_only : Flag<["-"], "emit-llvm-only">,
   HelpText<"Build ASTs and convert to LLVM, discarding output">;
 def emit_codegen_only : Flag<["-"], "emit-codegen-only">,
   HelpText<"Generate machine code, but discard output">;
HelpText<"Add directory to the internal system include search path with "
            "implicit extern \"C\" semantics; these are assumed to not be "
            "user-provided and are used to model system and standard headers' "
            "paths.">;
 
 //===----------------------------------------------------------------------===//
 // Preprocessor Options
 //===----------------------------------------------------------------------===//
 
-def include_pth : Separate<["-"], "include-pth">, MetaVarName<"<file>">,
-  HelpText<"Include file before parsing">;
 def chain_include : Separate<["-"], "chain-include">, MetaVarName<"<file>">,
   HelpText<"Include and chain a header file after turning it into PCH">;
 def preamble_bytes_EQ : Joined<["-"], "preamble-bytes=">,
   HelpText<"Assume that the precompiled header is a precompiled preamble "
            "covering the first N bytes of the main file">;
-def token_cache : Separate<["-"], "token-cache">, MetaVarName<"<path>">,
-  HelpText<"Use specified token cache file">;
 def detailed_preprocessing_record : Flag<["-"], "detailed-preprocessing-record">,
   HelpText<"include a detailed record of preprocessing actions">;
 
 //===----------------------------------------------------------------------===//
 // OpenCL Options
 //===----------------------------------------------------------------------===//
 
 def cl_ext_EQ : CommaJoined<["-"], "cl-ext=">,

include/clang/Driver/Driver.h

private:
   /// Arguments originated from command line.
   std::unique_ptr<llvm::opt::InputArgList> CLOptions;
 
   /// Whether to check that input files exist when constructing compilation
   /// jobs.
   unsigned CheckInputsExist : 1;
 
 public:
-  /// Use lazy precompiled headers for PCH support.
-  unsigned CCCUsePCH : 1;
-
   /// Force clang to emit reproducer for driver invocation. This is enabled
   /// indirectly by setting FORCE_CLANG_DIAGNOSTICS_CRASH environment variable
   /// or when using the -gen-reproducer driver flag.
   unsigned GenReproducer : 1;
 
 private:
   /// Certain options suppress the 'no input files' warning.
   unsigned SuppressMissingInputWarning : 1;

include/clang/Driver/Options.td

def driver_mode : Joined<["--"], "driver-mode=">, Group<internal_driver_Group>,
   Flags<[CoreOption, DriverOption, HelpHidden]>,
   HelpText<"Set the driver mode to either 'gcc', 'g++', 'cpp', or 'cl'">;
 def rsp_quoting : Joined<["--"], "rsp-quoting=">, Group<internal_driver_Group>,
   Flags<[CoreOption, DriverOption, HelpHidden]>,
   HelpText<"Set the rsp quoting to either 'posix', or 'windows'">;
 def ccc_gcc_name : Separate<["-"], "ccc-gcc-name">, InternalDriverOpt,
   HelpText<"Name for native GCC compiler">,
   MetaVarName<"<gcc-path>">;
-def ccc_pch_is_pch : Flag<["-"], "ccc-pch-is-pch">, InternalDriverOpt,
-  HelpText<"Use lazy PCH for precompiled headers">;
-def ccc_pch_is_pth : Flag<["-"], "ccc-pch-is-pth">, InternalDriverOpt,
-  HelpText<"Use pretokenized headers for precompiled headers">;
 
 class InternalDebugOpt : Group<internal_debug_Group>,
   Flags<[DriverOption, HelpHidden, CoreOption]>;
 def ccc_install_dir : Separate<["-"], "ccc-install-dir">, InternalDebugOpt,
   HelpText<"Simulate installation in the given directory">;
 def ccc_print_phases : Flag<["-"], "ccc-print-phases">, InternalDebugOpt,
   HelpText<"Dump list of actions to perform">;
 def ccc_print_bindings : Flag<["-"], "ccc-print-bindings">, InternalDebugOpt,

include/clang/Frontend/FrontendActions.h

protected:
   void ExecuteAction() override;
 };
 
 class DumpTokensAction : public PreprocessorFrontendAction {
 protected:
   void ExecuteAction() override;
 };
 
-class GeneratePTHAction : public PreprocessorFrontendAction {
-protected:
-  void ExecuteAction() override;
-};
-
 class PreprocessOnlyAction : public PreprocessorFrontendAction {
 protected:
   void ExecuteAction() override;
 };
 
 class PrintPreprocessedAction : public PreprocessorFrontendAction {
 protected:
   void ExecuteAction() override;
 
   bool hasPCHSupport() const override { return true; }
 };
 
 }  // end namespace clang
 
 #endif

include/clang/Frontend/FrontendOptions.h

enum ActionKind {
   GenerateModuleInterface,
 
   /// Generate pre-compiled module from a set of header files.
   GenerateHeaderModule,
 
   /// Generate pre-compiled header.
   GeneratePCH,
 
-  /// Generate pre-tokenized header.
-  GeneratePTH,
-
   /// Only execute frontend initialization.
   InitOnly,
 
   /// Dump information about a module file.
   ModuleFileInfo,
 
   /// Load and verify that a PCH file is usable.
   VerifyPCH,

include/clang/Frontend/Utils.h

 /// \param ShowDepth - Whether to indent to show the nesting of the includes.
 /// \param MSStyle - Whether to print in cl.exe /showIncludes style.
 void AttachHeaderIncludeGen(Preprocessor &PP,
                             const DependencyOutputOptions &DepOpts,
                             bool ShowAllHeaders = false,
                             StringRef OutputPath = {},
                             bool ShowDepth = true, bool MSStyle = false);
 
-/// Cache tokens for use with PCH. Note that this requires a seekable stream.
-void CacheTokens(Preprocessor &PP, raw_pwrite_stream *OS);
-
 /// The ChainedIncludesSource class converts headers to chained PCHs in
 /// memory, mainly for testing.
 IntrusiveRefCntPtr<ExternalSemaSource>
 createChainedIncludesSource(CompilerInstance &CI,
                             IntrusiveRefCntPtr<ExternalSemaSource> &Reader);
 
 /// createInvocationFromCommandLine - Construct a compiler invocation object for
 /// a command line argument vector.

include/clang/Lex/PTHLexer.h

-//===- PTHLexer.h - Lexer based on Pre-tokenized input ----------*- C++ -*-===//
-//
-//                     The LLVM Compiler Infrastructure
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-//
-// This file defines the PTHLexer interface.
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_CLANG_LEX_PTHLEXER_H
-#define LLVM_CLANG_LEX_PTHLEXER_H
-
-#include "clang/Basic/SourceLocation.h"
-#include "clang/Basic/TokenKinds.h"
-#include "clang/Lex/PreprocessorLexer.h"
-#include "clang/Lex/Token.h"
-
-namespace clang {
-
-class Preprocessor;
-class PTHManager;
-
-class PTHLexer : public PreprocessorLexer {
-  SourceLocation FileStartLoc;
-
-  /// TokBuf - Buffer from PTH file containing raw token data.
-  const unsigned char* TokBuf;
-
-  /// CurPtr - Pointer into current offset of the token buffer where
-  ///  the next token will be read.
-  const unsigned char* CurPtr;
-
-  /// LastHashTokPtr - Pointer into TokBuf of the last processed '#'
-  ///  token that appears at the start of a line.
-  const unsigned char* LastHashTokPtr = nullptr;
-
-  /// PPCond - Pointer to a side table in the PTH file that provides a
-  ///  a concise summary of the preprocessor conditional block structure.
-  ///  This is used to perform quick skipping of conditional blocks.
-  const unsigned char* PPCond;
-
-  /// CurPPCondPtr - Pointer inside PPCond that refers to the next entry
-  ///  to process when doing quick skipping of preprocessor blocks.
-  const unsigned char* CurPPCondPtr;
-
-  /// ReadToken - Used by PTHLexer to read tokens TokBuf.
-  void ReadToken(Token &T);
-
-  bool LexEndOfFile(Token &Result);
-
-  /// PTHMgr - The PTHManager object that created this PTHLexer.
-  PTHManager& PTHMgr;
-
-  Token EofToken;
-
-protected:
-  friend class PTHManager;
-
-  /// Create a PTHLexer for the specified token stream.
-  PTHLexer(Preprocessor &pp, FileID FID, const unsigned char *D,
-           const unsigned char* ppcond, PTHManager &PM);
-
-public:
-  PTHLexer(const PTHLexer &) = delete;
-  PTHLexer &operator=(const PTHLexer &) = delete;
-  ~PTHLexer() override = default;
-
-  /// Lex - Return the next token.
-  bool Lex(Token &Tok);
-
-  void getEOF(Token &Tok);
-
-  /// DiscardToEndOfLine - Read the rest of the current preprocessor line as an
-  /// uninterpreted string.  This switches the lexer out of directive mode.
-  void DiscardToEndOfLine();
-
-  /// isNextPPTokenLParen - Return 1 if the next unexpanded token will return a
-  /// tok::l_paren token, 0 if it is something else and 2 if there are no more
-  /// tokens controlled by this lexer.
-  unsigned isNextPPTokenLParen() {
-    // isNextPPTokenLParen is not on the hot path, and all we care about is
-    // whether or not we are at a token with kind tok::eof or tok::l_paren.
-    // Just read the first byte from the current token pointer to determine
-    // its kind.
-    tok::TokenKind x = (tok::TokenKind)*CurPtr;
-    return x == tok::eof ? 2 : x == tok::l_paren;
-  }
-
-  /// IndirectLex - An indirect call to 'Lex' that can be invoked via
-  ///  the PreprocessorLexer interface.
-  void IndirectLex(Token &Result) override { Lex(Result); }
-
-  /// getSourceLocation - Return a source location for the token in
-  /// the current file.
-  SourceLocation getSourceLocation() override;
-
-  /// SkipBlock - Used by Preprocessor to skip the current conditional block.
-  bool SkipBlock();
-};
-
-} // namespace clang
-
-#endif // LLVM_CLANG_LEX_PTHLEXER_H

include/clang/Lex/PTHManager.h

-//===- PTHManager.h - Manager object for PTH processing ---------*- C++ -*-===//
-//
-//                     The LLVM Compiler Infrastructure
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-//
-//  This file defines the PTHManager interface.
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_CLANG_LEX_PTHMANAGER_H
-#define LLVM_CLANG_LEX_PTHMANAGER_H
-
-#include "clang/Basic/IdentifierTable.h"
-#include "clang/Basic/SourceLocation.h"
-#include "llvm/ADT/STLExtras.h"
-#include "llvm/ADT/StringRef.h"
-#include "llvm/Support/Allocator.h"
-#include "llvm/Support/OnDiskHashTable.h"
-#include <memory>
-
-namespace llvm {
-
-class MemoryBuffer;
-
-} // namespace llvm
-
-namespace clang {
-
-class DiagnosticsEngine;
-class FileSystemStatCache;
-class Preprocessor;
-class PTHLexer;
-
-class PTHManager : public IdentifierInfoLookup {
-  friend class PTHLexer;
-  friend class PTHStatCache;
-
-  class PTHFileLookupTrait;
-  class PTHStringLookupTrait;
-
-  using PTHStringIdLookup = llvm::OnDiskChainedHashTable<PTHStringLookupTrait>;
-  using PTHFileLookup = llvm::OnDiskChainedHashTable<PTHFileLookupTrait>;
-
-  /// The memory mapped PTH file.
-  std::unique_ptr<const llvm::MemoryBuffer> Buf;
-
-  /// Alloc - Allocator used for IdentifierInfo objects.
-  llvm::BumpPtrAllocator Alloc;
-
-  /// IdMap - A lazily generated cache mapping from persistent identifiers to
-  ///  IdentifierInfo*.
-  std::unique_ptr<IdentifierInfo *[], llvm::FreeDeleter> PerIDCache;
-
-  /// FileLookup - Abstract data structure used for mapping between files
-  ///  and token data in the PTH file.
-  std::unique_ptr<PTHFileLookup> FileLookup;
-
-  /// IdDataTable - Array representing the mapping from persistent IDs to the
-  ///  data offset within the PTH file containing the information to
-  ///  reconsitute an IdentifierInfo.
-  const unsigned char* const IdDataTable;
-
-  /// SortedIdTable - Abstract data structure mapping from strings to
-  ///  persistent IDs.  This is used by get().
-  std::unique_ptr<PTHStringIdLookup> StringIdLookup;
-
-  /// NumIds - The number of identifiers in the PTH file.
-  const unsigned NumIds;
-
-  /// PP - The Preprocessor object that will use this PTHManager to create
-  ///  PTHLexer objects.
-  Preprocessor* PP = nullptr;
-
-  /// SpellingBase - The base offset within the PTH memory buffer that
-  ///  contains the cached spellings for literals.
-  const unsigned char* const SpellingBase;
-
-  /// OriginalSourceFile - A null-terminated C-string that specifies the name
-  ///  if the file (if any) that was to used to generate the PTH cache.
-  const char* OriginalSourceFile;
-
-  /// This constructor is intended to only be called by the static 'Create'
-  /// method.
-  PTHManager(std::unique_ptr<const llvm::MemoryBuffer> buf,
-             std::unique_ptr<PTHFileLookup> fileLookup,
-             const unsigned char *idDataTable,
-             std::unique_ptr<IdentifierInfo *[], llvm::FreeDeleter> perIDCache,
-             std::unique_ptr<PTHStringIdLookup> stringIdLookup, unsigned numIds,
-             const unsigned char *spellingBase, const char *originalSourceFile);
-
-  /// getSpellingAtPTHOffset - Used by PTHLexer classes to get the cached
-  ///  spelling for a token.
-  unsigned getSpellingAtPTHOffset(unsigned PTHOffset, const char*& Buffer);
-
-  /// GetIdentifierInfo - Used to reconstruct IdentifierInfo objects from the
-  ///  PTH file.
-  IdentifierInfo *GetIdentifierInfo(unsigned PersistentID) {
-    // Check if the IdentifierInfo has already been resolved.
-    if (IdentifierInfo* II = PerIDCache[PersistentID])
-      return II;
-    return LazilyCreateIdentifierInfo(PersistentID);
-  }
-  IdentifierInfo* LazilyCreateIdentifierInfo(unsigned PersistentID);
-
-public:
-  // The current PTH version.
-  enum { Version = 10 };
-
-  PTHManager(const PTHManager &) = delete;
-  PTHManager &operator=(const PTHManager &) = delete;
-  ~PTHManager() override;
-
-  /// getOriginalSourceFile - Return the full path to the original header
-  ///  file name that was used to generate the PTH cache.
-  const char* getOriginalSourceFile() const {
-    return OriginalSourceFile;
-  }
-
-  /// get - Return the identifier token info for the specified named identifier.
-  ///  Unlike the version in IdentifierTable, this returns a pointer instead
-  ///  of a reference.  If the pointer is nullptr then the IdentifierInfo cannot
-  ///  be found.
-  IdentifierInfo *get(StringRef Name) override;
-
-  /// Create - This method creates PTHManager objects.  The 'file' argument
-  ///  is the name of the PTH file.  This method returns nullptr upon failure.
-  static PTHManager *Create(StringRef file, DiagnosticsEngine &Diags);
-
-  void setPreprocessor(Preprocessor *pp) { PP = pp; }
-
-  /// CreateLexer - Return a PTHLexer that "lexes" the cached tokens for the
-  ///  specified file.  This method returns nullptr if no cached tokens exist.
-  ///  It is the responsibility of the caller to 'delete' the returned object.
-  PTHLexer *CreateLexer(FileID FID);
-
-  /// createStatCache - Returns a FileSystemStatCache object for use with
-  ///  FileManager objects.  These objects use the PTH data to speed up
-  ///  calls to stat by memoizing their results from when the PTH file
-  ///  was generated.
-  std::unique_ptr<FileSystemStatCache> createStatCache();
-};
-
-} // namespace clang
-
-#endif // LLVM_CLANG_LEX_PTHMANAGER_H

include/clang/Lex/Preprocessor.h

 #include "clang/Basic/SourceLocation.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/TokenKinds.h"
 #include "clang/Lex/Lexer.h"
 #include "clang/Lex/MacroInfo.h"
 #include "clang/Lex/ModuleLoader.h"
 #include "clang/Lex/ModuleMap.h"
 #include "clang/Lex/PPCallbacks.h"
-#include "clang/Lex/PTHLexer.h"
 #include "clang/Lex/Token.h"
 #include "clang/Lex/TokenLexer.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/FoldingSet.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/PointerUnion.h"

 class HeaderSearch;
 class MacroArgs;
 class MemoryBufferCache;
 class PragmaHandler;
 class PragmaNamespace;
 class PreprocessingRecord;
 class PreprocessorLexer;
 class PreprocessorOptions;
-class PTHManager;
 class ScratchBuffer;
 class TargetInfo;
 
 /// Stores token information for comparing actual tokens with
 /// predefined values.  Only handles simple tokens and identifiers.
 class TokenValue {
   tok::TokenKind Kind;
   IdentifierInfo *II;
class Preprocessor {
   MemoryBufferCache &PCMCache;
   std::unique_ptr<ScratchBuffer> ScratchBuf;
   HeaderSearch      &HeaderInfo;
   ModuleLoader      &TheModuleLoader;
 
   /// External source of macros.
   ExternalPreprocessorSource *ExternalSource;
 
-  /// An optional PTHManager object used for getting tokens from
-  /// a token cache rather than lexing the original source file.
-  std::unique_ptr<PTHManager> PTH;
-
   /// A BumpPtrAllocator object used to quickly allocate and release
   /// objects internal to the Preprocessor.
   llvm::BumpPtrAllocator BP;
 
   /// Identifiers for builtin macros and other builtins.
   IdentifierInfo *Ident__LINE__, *Ident__FILE__;   // __LINE__, __FILE__
   IdentifierInfo *Ident__DATE__, *Ident__TIME__;   // __DATE__, __TIME__
   IdentifierInfo *Ident__INCLUDE_LEVEL__;          // __INCLUDE_LEVEL__
private:
   private:
     SmallVector<PPConditionalInfo, 4> ConditionalStack;
     State ConditionalStackState = Off;
   } PreambleConditionalStack;
 
   /// The current top of the stack that we're lexing from if
   /// not expanding a macro and we are lexing directly from source code.
   ///
-  /// Only one of CurLexer, CurPTHLexer, or CurTokenLexer will be non-null.
+  /// Only one of CurLexer, or CurTokenLexer will be non-null.
   std::unique_ptr<Lexer> CurLexer;
 
-  /// The current top of stack that we're lexing from if
-  /// not expanding from a macro and we are lexing from a PTH cache.
-  ///
-  /// Only one of CurLexer, CurPTHLexer, or CurTokenLexer will be non-null.
-  std::unique_ptr<PTHLexer> CurPTHLexer;
-
   /// The current top of the stack what we're lexing from
   /// if not expanding a macro.
   ///
-  /// This is an alias for either CurLexer or  CurPTHLexer.
+  /// This is an alias for CurLexer.
   PreprocessorLexer *CurPPLexer = nullptr;
 
   /// Used to find the current FileEntry, if CurLexer is non-null
   /// and if applicable.
   ///
   /// This allows us to implement \#include_next and find directory-specific
   /// properties.
   const DirectoryLookup *CurDirLookup = nullptr;
 
   /// The current macro we are expanding, if we are expanding a macro.
   ///
   /// One of CurLexer and CurTokenLexer must be null.
   std::unique_ptr<TokenLexer> CurTokenLexer;
 
   /// The kind of lexer we're currently working with.
   enum CurLexerKind {
     CLK_Lexer,
-    CLK_PTHLexer,
     CLK_TokenLexer,
     CLK_CachingLexer,
     CLK_LexAfterModuleImport
   } CurLexerKind = CLK_Lexer;
 
   /// If the current lexer is for a submodule that is being built, this
   /// is that submodule.
   Module *CurLexerSubmodule = nullptr;
 
   /// Keeps track of the stack of files currently
   /// \#included, and macros currently being expanded from, not counting
   /// CurLexer/CurTokenLexer.
   struct IncludeStackInfo {
     enum CurLexerKind           CurLexerKind;
     Module                     *TheSubmodule;
     std::unique_ptr<Lexer>      TheLexer;
-    std::unique_ptr<PTHLexer>   ThePTHLexer;
     PreprocessorLexer          *ThePPLexer;
     std::unique_ptr<TokenLexer> TheTokenLexer;
     const DirectoryLookup      *TheDirLookup;
 
     // The following constructors are completely useless copies of the default
     // versions, only needed to pacify MSVC.
     IncludeStackInfo(enum CurLexerKind CurLexerKind, Module *TheSubmodule,
                      std::unique_ptr<Lexer> &&TheLexer,
-                     std::unique_ptr<PTHLexer> &&ThePTHLexer,
                      PreprocessorLexer *ThePPLexer,
                      std::unique_ptr<TokenLexer> &&TheTokenLexer,
                      const DirectoryLookup *TheDirLookup)
         : CurLexerKind(std::move(CurLexerKind)),
           TheSubmodule(std::move(TheSubmodule)), TheLexer(std::move(TheLexer)),
-          ThePTHLexer(std::move(ThePTHLexer)),
           ThePPLexer(std::move(ThePPLexer)),
           TheTokenLexer(std::move(TheTokenLexer)),
           TheDirLookup(std::move(TheDirLookup)) {}
   };
   std::vector<IncludeStackInfo> IncludeMacroStack;
 
   /// Actions invoked when some preprocessor activity is
   /// encountered (e.g. a file is \#included, etc).
public:
   HeaderSearch &getHeaderSearchInfo() const { return HeaderInfo; }
 
   IdentifierTable &getIdentifierTable() { return Identifiers; }
   const IdentifierTable &getIdentifierTable() const { return Identifiers; }
   SelectorTable &getSelectorTable() { return Selectors; }
   Builtin::Context &getBuiltinInfo() { return BuiltinInfo; }
   llvm::BumpPtrAllocator &getPreprocessorAllocator() { return BP; }
 
-  void setPTHManager(PTHManager* pm);
-
-  PTHManager *getPTHManager() { return PTH.get(); }
-
   void setExternalSource(ExternalPreprocessorSource *Source) {
     ExternalSource = Source;
   }
 
   ExternalPreprocessorSource *getExternalSource() const {
     return ExternalSource;
   }
 
public:
   /// Update the current token to represent the provided
   /// identifier, in order to cache an action performed by typo correction.
   void TypoCorrectToken(const Token &Tok) {
     assert(Tok.getIdentifierInfo() && "Expected identifier token");
     if (CachedLexPos != 0 && isBacktrackEnabled())
       CachedTokens[CachedLexPos-1] = Tok;
   }
 
-  /// Recompute the current lexer kind based on the CurLexer/CurPTHLexer/
+  /// Recompute the current lexer kind based on the CurLexer/
   /// CurTokenLexer pointers.
   void recomputeCurLexerKind();
 
   /// Returns true if incremental processing is enabled
   bool isIncrementalProcessingEnabled() const { return IncrementalProcessing; }
 
   /// Enables the incremental processing
   void enableIncrementalProcessing(bool value = true) {
public:
   Module *LeaveSubmodule(bool ForPragma);
 
 private:
   friend void TokenLexer::ExpandFunctionArguments();
 
   void PushIncludeMacroStack() {
     assert(CurLexerKind != CLK_CachingLexer && "cannot push a caching lexer");
     IncludeMacroStack.emplace_back(CurLexerKind, CurLexerSubmodule,
-                                   std::move(CurLexer), std::move(CurPTHLexer),
-                                   CurPPLexer, std::move(CurTokenLexer),
-                                   CurDirLookup);
+                                   std::move(CurLexer), CurPPLexer,
+                                   std::move(CurTokenLexer), CurDirLookup);
     CurPPLexer = nullptr;
   }
 
   void PopIncludeMacroStack() {
     CurLexer = std::move(IncludeMacroStack.back().TheLexer);
-    CurPTHLexer = std::move(IncludeMacroStack.back().ThePTHLexer);
     CurPPLexer = IncludeMacroStack.back().ThePPLexer;
     CurTokenLexer = std::move(IncludeMacroStack.back().TheTokenLexer);
     CurDirLookup  = IncludeMacroStack.back().TheDirLookup;
     CurLexerSubmodule = IncludeMacroStack.back().TheSubmodule;
     CurLexerKind = IncludeMacroStack.back().CurLexerKind;
     IncludeMacroStack.pop_back();
   }
 
private:
   /// \p FoundElse is false, then \#else directives are ok, if not, then we have
   /// already seen one so a \#else directive is a duplicate.  When this returns,
   /// the caller can lex the first valid token.
   void SkipExcludedConditionalBlock(SourceLocation HashTokenLoc,
                                     SourceLocation IfTokenLoc,
                                     bool FoundNonSkipPortion, bool FoundElse,
                                     SourceLocation ElseLoc = SourceLocation());
 
-  /// A fast PTH version of SkipExcludedConditionalBlock.
-  void PTHSkipExcludedConditionalBlock();
-
   /// Information about the result for evaluating an expression for a
   /// preprocessor directive.
   struct DirectiveEvalResult {
     /// Whether the expression was evaluated as true or not.
     bool Conditional;
 
     /// True if the expression contained identifiers that were undefined.
     bool IncludedUndefinedIds;
private:
   /// Like Handle_Pragma except the pragma text is not enclosed within
   /// a string literal.
   void HandleMicrosoft__pragma(Token &Tok);
 
   /// Add a lexer to the top of the include stack and
   /// start lexing tokens from it instead of the current buffer.
   void EnterSourceFileWithLexer(Lexer *TheLexer, const DirectoryLookup *Dir);
 
-  /// Add a lexer to the top of the include stack and
-  /// start getting tokens from it using the PTH cache.
-  void EnterSourceFileWithPTH(PTHLexer *PL, const DirectoryLookup *Dir);
-
   /// Set the FileID for the preprocessor predefines.
   void setPredefinesFileID(FileID FID) {
     assert(PredefinesFileID.isInvalid() && "PredefinesFileID already set!");
     PredefinesFileID = FID;
   }
 
   /// Set the FileID for the PCH through header.
   void setPCHThroughHeaderFileID(FileID FID);
private:
 
   //===--------------------------------------------------------------------===//
   // Caching stuff.
   void CachingLex(Token &Result);
 
   bool InCachingLexMode() const {
     // If the Lexer pointers are 0 and IncludeMacroStack is empty, it means
     // that we are past EOF, not that we are in CachingLex mode.
-    return !CurPPLexer && !CurTokenLexer && !CurPTHLexer &&
-           !IncludeMacroStack.empty();
+    return !CurPPLexer && !CurTokenLexer && !IncludeMacroStack.empty();
   }
 
   void EnterCachingLexMode();
 
   void ExitCachingLexMode() {
     if (InCachingLexMode())
       RemoveTopOfLexerStack();
   }

include/clang/Lex/PreprocessorOptions.h

public:
   /// processing the rest of the file.
   bool GeneratePreamble = false;
 
   /// Whether to write comment locations into the PCH when building it.
   /// Reading the comments from the PCH can be a performance hit even if the
   /// clients don't use them.
   bool WriteCommentListToPCH = true;
 
-  /// The implicit PTH input included at the start of the translation unit, or
-  /// empty.
-  std::string ImplicitPTHInclude;
-
-  /// If given, a PTH cache file to use for speeding up header parsing.
-  std::string TokenCache;
-
   /// When enabled, preprocessor is in a mode for parsing a single file only.
   ///
   /// Disables #includes of other files and if there are unresolved identifiers
   /// in preprocessor directive conditions it causes all blocks to be parsed so
   /// that the client can get the maximum amount of information from the parser.
   bool SingleFileParseMode = false;
 
   /// When enabled, the preprocessor will construct editor placeholder tokens.
public:
   /// Reset any options that are not considered when building a
   /// module.
   void resetNonModularOptions() {
     Includes.clear();
     MacroIncludes.clear();
     ChainedIncludes.clear();
     DumpDeserializedPCHDecls = false;
     ImplicitPCHInclude.clear();
-    ImplicitPTHInclude.clear();
-    TokenCache.clear();
     SingleFileParseMode = false;
     LexEditorPlaceholders = true;
     RetainRemappedFileBuffers = true;
     PrecompiledPreambleBytes.first = 0;
     PrecompiledPreambleBytes.second = false;
   }
 };
 
 } // namespace clang
 
 #endif // LLVM_CLANG_LEX_PREPROCESSOROPTIONS_H_

lib/ARCMigrate/ARCMT.cpp

IntrusiveRefCntPtr<DiagnosticsEngine> Diags(
         new DiagnosticsEngine(DiagID, &origCI.getDiagnosticOpts(),
                               new IgnoringDiagConsumer()));
     std::string OriginalFile = ASTReader::getOriginalSourceFile(
         PPOpts.ImplicitPCHInclude, FileMgr, PCHContainerRdr, *Diags);
     if (!OriginalFile.empty())
       PPOpts.Includes.insert(PPOpts.Includes.begin(), OriginalFile);
     PPOpts.ImplicitPCHInclude.clear();
   }
-  // FIXME: Get the original header of a PTH as well.
-  CInvok->getPreprocessorOpts().ImplicitPTHInclude.clear();
   std::string define = getARCMTMacroName();
   define += '=';
   CInvok->getPreprocessorOpts().addMacroDef(define);
   CInvok->getLangOpts()->ObjCAutoRefCount = true;
   CInvok->getLangOpts()->setGC(LangOptions::NonGC);
   CInvok->getDiagnosticOpts().ErrorLimit = 0;
   CInvok->getDiagnosticOpts().PedanticErrors = 0;
 

lib/Driver/Driver.cpp

: Opts(createDriverOptTable()), Diags(Diags), VFS(std::move(VFS)),
       Mode(GCCMode), SaveTemps(SaveTempsNone), BitcodeEmbed(EmbedNone),
       LTOMode(LTOK_None), ClangExecutable(ClangExecutable),
       SysRoot(DEFAULT_SYSROOT), DriverTitle("clang LLVM compiler"),
       CCPrintOptionsFilename(nullptr), CCPrintHeadersFilename(nullptr),
       CCLogDiagnosticsFilename(nullptr), CCCPrintBindings(false),
       CCPrintOptions(false), CCPrintHeaders(false), CCLogDiagnostics(false),
       CCGenDiagnostics(false), TargetTriple(TargetTriple),
       CCCGenericGCCName(""), Saver(Alloc), CheckInputsExist(true),
-      CCCUsePCH(true), GenReproducer(false),
-      SuppressMissingInputWarning(false) {
+      GenReproducer(false), SuppressMissingInputWarning(false) {
 
   // Provide a sane fallback if no VFS is specified.
   if (!this->VFS)
     this->VFS = llvm::vfs::getRealFileSystem();
 
   Name = llvm::sys::path::filename(ClangExecutable);
   Dir = llvm::sys::path::parent_path(ClangExecutable);
   InstalledDir = Dir; // Provide a sensible default installed dir.
Compilation *Driver::BuildCompilation(ArrayRef<const char *> ArgList) {
   // FIXME: We need to figure out where this behavior should live. Most of it
   // should be outside in the client; the parts that aren't should have proper
   // options, either by introducing new ones or by overloading gcc ones like -V
   // or -b.
   CCCPrintPhases = Args.hasArg(options::OPT_ccc_print_phases);
   CCCPrintBindings = Args.hasArg(options::OPT_ccc_print_bindings);
   if (const Arg *A = Args.getLastArg(options::OPT_ccc_gcc_name))
     CCCGenericGCCName = A->getValue();
-  CCCUsePCH =
-      Args.hasFlag(options::OPT_ccc_pch_is_pch, options::OPT_ccc_pch_is_pth);
   GenReproducer = Args.hasFlag(options::OPT_gen_reproducer,
                                options::OPT_fno_crash_diagnostics,
                                !!::getenv("FORCE_CLANG_DIAGNOSTICS_CRASH"));
   // FIXME: TargetTriple is used by the target-prefixed calls to as/ld
   // and getToolChain is const.
   if (IsCLMode()) {
     // clang-cl targets MSVC-style Win32.
     llvm::Triple T(TargetTriple);

lib/Driver/ToolChains/Clang.cpp

void Clang::AddPreprocessingOptions(Compilation &C, const JobAction &JA,
 
   bool RenderedImplicitInclude = false;
   for (const Arg *A : Args.filtered(options::OPT_clang_i_Group)) {
     if (A->getOption().matches(options::OPT_include)) {
       // Handling of gcc-style gch precompiled headers.
       bool IsFirstImplicitInclude = !RenderedImplicitInclude;
       RenderedImplicitInclude = true;
 
-      // Use PCH if the user requested it.
-      bool UsePCH = D.CCCUsePCH;
-
-      bool FoundPTH = false;
       bool FoundPCH = false;
       SmallString<128> P(A->getValue());
       // We want the files to have a name like foo.h.pch. Add a dummy extension
       // so that replace_extension does the right thing.
       P += ".dummy";
-      if (UsePCH) {
-        llvm::sys::path::replace_extension(P, "pch");
-        if (llvm::sys::fs::exists(P))
-          FoundPCH = true;
-      }
 
       if (!FoundPCH) {
-        llvm::sys::path::replace_extension(P, "pth");
-        if (llvm::sys::fs::exists(P))
-          FoundPTH = true;
-      }
-
-      if (!FoundPCH && !FoundPTH) {
         llvm::sys::path::replace_extension(P, "gch");
         if (llvm::sys::fs::exists(P)) {
-          FoundPCH = UsePCH;
-          FoundPTH = !UsePCH;
+          FoundPCH = true;
         }
       }
 
-      if (FoundPCH || FoundPTH) {
+      if (FoundPCH) {
         if (IsFirstImplicitInclude) {
           A->claim();
-          if (UsePCH)
-            CmdArgs.push_back("-include-pch");
-          else
-            CmdArgs.push_back("-include-pth");
           CmdArgs.push_back(Args.MakeArgString(P));
           continue;
         } else {
           // Ignore the PCH if not first on command line and emit warning.
           D.Diag(diag::warn_drv_pch_not_first_include) << P
                                                        << A->getAsString(Args);
         }
       }
void Clang::ConstructJob(Compilation &C, const JobAction &JA,
   } else if (isa<AssembleJobAction>(JA)) {
     CmdArgs.push_back("-emit-obj");
 
     CollectArgsForIntegratedAssembler(C, Args, CmdArgs, D);
 
     // Also ignore explicit -force_cpusubtype_ALL option.
     (void)Args.hasArg(options::OPT_force__cpusubtype__ALL);
   } else if (isa<PrecompileJobAction>(JA)) {
-    // Use PCH if the user requested it.
-    bool UsePCH = D.CCCUsePCH;
-
     if (JA.getType() == types::TY_Nothing)
       CmdArgs.push_back("-fsyntax-only");
     else if (JA.getType() == types::TY_ModuleFile)
       CmdArgs.push_back(IsHeaderModulePrecompile
                             ? "-emit-header-module"
                             : "-emit-module-interface");
-    else if (UsePCH)
-      CmdArgs.push_back("-emit-pch");
     else
-      CmdArgs.push_back("-emit-pth");
+      CmdArgs.push_back("-emit-pch");
   } else if (isa<VerifyPCHJobAction>(JA)) {
     CmdArgs.push_back("-verify-pch");
   } else {
     assert((isa<CompileJobAction>(JA) || isa<BackendJobAction>(JA)) &&
            "Invalid action for clang tool.");
     if (JA.getType() == types::TY_Nothing) {
       CmdArgs.push_back("-fsyntax-only");
     } else if (JA.getType() == types::TY_LLVM_IR ||

   if (Arg *A = Args.getLastArg(options::OPT_pg))
     if (!shouldUseFramePointer(Args, Triple))
       D.Diag(diag::err_drv_argument_not_allowed_with) << "-fomit-frame-pointer"
                                                       << A->getAsString(Args);
 
   // Claim some arguments which clang supports automatically.
 
   // -fpch-preprocess is used with gcc to add a special marker in the output to
-  // include the PCH file. Clang's PTH solution is completely transparent, so we
-  // do not need to deal with it at all.
+  // include the PCH file.
   Args.ClaimAllArgs(options::OPT_fpch_preprocess);
 
   // Claim some arguments which clang doesn't support, but we don't
   // care to warn the user about.
   Args.ClaimAllArgs(options::OPT_clang_ignored_f_Group);
   Args.ClaimAllArgs(options::OPT_clang_ignored_m_Group);
 
   // Disable warnings for clang -E -emit-llvm foo.c

lib/Frontend/CMakeLists.txt

 if (CLANG_BUILT_STANDALONE)
   set(optional_deps)
 endif()
 
 add_clang_library(clangFrontend
   ASTConsumers.cpp
   ASTMerge.cpp
   ASTUnit.cpp
-  CacheTokens.cpp
   ChainedDiagnosticConsumer.cpp
   ChainedIncludesSource.cpp
   CodeGenOptions.cpp
   CompilerInstance.cpp
   CompilerInvocation.cpp
   CreateInvocationFromCommandLine.cpp
   DependencyFile.cpp
   DependencyGraph.cpp

lib/Frontend/CacheTokens.cpp

-//===--- CacheTokens.cpp - Caching of lexer tokens for PTH support --------===//
-//
-//                     The LLVM Compiler Infrastructure
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-//
-// This provides a possible implementation of PTH support for Clang that is
-// based on caching lexed tokens and identifiers.
-//
-//===----------------------------------------------------------------------===//
-
-#include "clang/Basic/Diagnostic.h"
-#include "clang/Basic/FileManager.h"
-#include "clang/Basic/FileSystemStatCache.h"
-#include "clang/Basic/IdentifierTable.h"
-#include "clang/Basic/SourceManager.h"
-#include "clang/Frontend/Utils.h"
-#include "clang/Lex/Lexer.h"
-#include "clang/Lex/PTHManager.h"
-#include "clang/Lex/Preprocessor.h"
-#include "llvm/ADT/StringMap.h"
-#include "llvm/Support/DJB.h"
-#include "llvm/Support/EndianStream.h"
-#include "llvm/Support/FileSystem.h"
-#include "llvm/Support/MemoryBuffer.h"
-#include "llvm/Support/OnDiskHashTable.h"
-#include "llvm/Support/Path.h"
-
-// FIXME: put this somewhere else?
-#ifndef S_ISDIR
-#define S_ISDIR(x) (((x)&_S_IFDIR)!=0)
-#endif
-
-using namespace clang;
-
-//===----------------------------------------------------------------------===//
-// PTH-specific stuff.
-//===----------------------------------------------------------------------===//
-
-typedef uint32_t Offset;
-
-namespace {
-class PTHEntry {
-  Offset TokenData, PPCondData;
-
-public:
-  PTHEntry() {}
-
-  PTHEntry(Offset td, Offset ppcd)
-    : TokenData(td), PPCondData(ppcd) {}
-
-  Offset getTokenOffset() const { return TokenData; }
-  Offset getPPCondTableOffset() const { return PPCondData; }
-};
-
-
-class PTHEntryKeyVariant {
-  union {
-    const FileEntry *FE;
-    // FIXME: Use "StringRef Path;" when MSVC 2013 is dropped.
-    const char *PathPtr;
-  };
-  size_t PathSize;
-  enum { IsFE = 0x1, IsDE = 0x2, IsNoExist = 0x0 } Kind;
-  FileData *Data;
-
-public:
-  PTHEntryKeyVariant(const FileEntry *fe) : FE(fe), Kind(IsFE), Data(nullptr) {}
-
-  PTHEntryKeyVariant(FileData *Data, StringRef Path)
-      : PathPtr(Path.data()), PathSize(Path.size()), Kind(IsDE),
-        Data(new FileData(*Data)) {}
-
-  explicit PTHEntryKeyVariant(StringRef Path)
-      : PathPtr(Path.data()), PathSize(Path.size()), Kind(IsNoExist),
-        Data(nullptr) {}
-
-  bool isFile() const { return Kind == IsFE; }
-
-  StringRef getString() const {
-    return Kind == IsFE ? FE->getName() : StringRef(PathPtr, PathSize);
-  }
-
-  unsigned getKind() const { return (unsigned) Kind; }
-
-  void EmitData(raw_ostream& Out) {
-    using namespace llvm::support;
-    endian::Writer LE(Out, little);
-    switch (Kind) {
-    case IsFE: {
-      // Emit stat information.
-      llvm::sys::fs::UniqueID UID = FE->getUniqueID();
-      LE.write<uint64_t>(UID.getFile());
-      LE.write<uint64_t>(UID.getDevice());
-      LE.write<uint64_t>(FE->getModificationTime());
-      LE.write<uint64_t>(FE->getSize());
-    } break;
-    case IsDE:
-      // Emit stat information.
-      LE.write<uint64_t>(Data->UniqueID.getFile());
-      LE.write<uint64_t>(Data->UniqueID.getDevice());
-      LE.write<uint64_t>(Data->ModTime);
-      LE.write<uint64_t>(Data->Size);
-      delete Data;
-      break;
-    default:
-      break;
-    }
-  }
-
-  unsigned getRepresentationLength() const {
-    return Kind == IsNoExist ? 0 : 4 * 8;
-  }
-};
-
-class FileEntryPTHEntryInfo {
-public:
-  typedef PTHEntryKeyVariant key_type;
-  typedef key_type key_type_ref;
-
-  typedef PTHEntry data_type;
-  typedef const PTHEntry& data_type_ref;
-
-  typedef unsigned hash_value_type;
-  typedef unsigned offset_type;
-
-  static hash_value_type ComputeHash(PTHEntryKeyVariant V) {
-    return llvm::djbHash(V.getString());
-  }
-
-  static std::pair<unsigned,unsigned>
-  EmitKeyDataLength(raw_ostream& Out, PTHEntryKeyVariant V,
-                    const PTHEntry& E) {
-    using namespace llvm::support;
-    endian::Writer LE(Out, little);
-
-    unsigned n = V.getString().size() + 1 + 1;
-    LE.write<uint16_t>(n);
-
-    unsigned m = V.getRepresentationLength() + (V.isFile() ? 4 + 4 : 0);
-    LE.write<uint8_t>(m);
-
-    return std::make_pair(n, m);
-  }
-
-  static void EmitKey(raw_ostream& Out, PTHEntryKeyVariant V, unsigned n){
-    using namespace llvm::support;
-    // Emit the entry kind.
-    Out << char(V.getKind());
-    // Emit the string.
-    Out.write(V.getString().data(), n - 1);
-  }
-
-  static void EmitData(raw_ostream& Out, PTHEntryKeyVariant V,
-                       const PTHEntry& E, unsigned) {
-    using namespace llvm::support;
-    endian::Writer LE(Out, little);
-
-    // For file entries emit the offsets into the PTH file for token data
-    // and the preprocessor blocks table.
-    if (V.isFile()) {
-      LE.write<uint32_t>(E.getTokenOffset());
-      LE.write<uint32_t>(E.getPPCondTableOffset());
-    }
-
-    // Emit any other data associated with the key (i.e., stat information).
-    V.EmitData(Out);
-  }
-};
-
-class OffsetOpt {
-  bool valid;
-  Offset off;
-public:
-  OffsetOpt() : valid(false) {}
-  bool hasOffset() const { return valid; }
-  Offset getOffset() const { assert(valid); return off; }
-  void setOffset(Offset o) { off = o; valid = true; }
-};
-} // end anonymous namespace
-
-typedef llvm::OnDiskChainedHashTableGenerator<FileEntryPTHEntryInfo> PTHMap;
-
-namespace {
-class PTHWriter {
-  typedef llvm::DenseMap<const IdentifierInfo*,uint32_t> IDMap;
-  typedef llvm::StringMap<OffsetOpt, llvm::BumpPtrAllocator> CachedStrsTy;
-
-  raw_pwrite_stream &Out;
-  Preprocessor& PP;
-  IDMap IM;
-  std::vector<llvm::StringMapEntry<OffsetOpt>*> StrEntries;
-  PTHMap PM;
-  CachedStrsTy CachedStrs;
-  uint32_t idcount;
-  Offset CurStrOffset;
-
-  //// Get the persistent id for the given IdentifierInfo*.
-  uint32_t ResolveID(const IdentifierInfo* II);
-
-  /// Emit a token to the PTH file.
-  void EmitToken(const Token& T);
-
-  void Emit8(uint32_t V) {
-    Out << char(V);
-  }
-
-  void Emit16(uint32_t V) {
-    using namespace llvm::support;
-    endian::write<uint16_t>(Out, V, little);
-  }
-
-  void Emit32(uint32_t V) {
-    using namespace llvm::support;
-    endian::write<uint32_t>(Out, V, little);
-  }
-
-  void EmitBuf(const char *Ptr, unsigned NumBytes) {
-    Out.write(Ptr, NumBytes);
-  }
-
-  void EmitString(StringRef V) {
-    using namespace llvm::support;
-    endian::write<uint16_t>(Out, V.size(), little);
-    EmitBuf(V.data(), V.size());
-  }
-
-  /// EmitIdentifierTable - Emits two tables to the PTH file.  The first is
-  ///  a hashtable mapping from identifier strings to persistent IDs.
-  ///  The second is a straight table mapping from persistent IDs to string data
-  ///  (the keys of the first table).
-  std::pair<Offset, Offset> EmitIdentifierTable();
-
-  /// EmitFileTable - Emit a table mapping from file name strings to PTH
-  /// token data.
-  Offset EmitFileTable() { return PM.Emit(Out); }
-
-  PTHEntry LexTokens(Lexer& L);
-  Offset EmitCachedSpellings();
-
-public:
-  PTHWriter(raw_pwrite_stream &out, Preprocessor &pp)
-      : Out(out), PP(pp), idcount(0), CurStrOffset(0) {}
-
-  PTHMap &getPM() { return PM; }
-  void GeneratePTH(StringRef MainFile);
-};
-} // end anonymous namespace
-
-uint32_t PTHWriter::ResolveID(const IdentifierInfo* II) {
-  // Null IdentifierInfo's map to the persistent ID 0.
-  if (!II)
-    return 0;
-
-  IDMap::iterator I = IM.find(II);
-  if (I != IM.end())
-    return I->second; // We've already added 1.
-
-  IM[II] = ++idcount; // Pre-increment since '0' is reserved for NULL.
-  return idcount;
-}
-
-void PTHWriter::EmitToken(const Token& T) {
-  // Emit the token kind, flags, and length.
-  Emit32(((uint32_t) T.getKind()) | ((((uint32_t) T.getFlags())) << 8)|
-         (((uint32_t) T.getLength()) << 16));
-
-  if (!T.isLiteral()) {
-    Emit32(ResolveID(T.getIdentifierInfo()));
-  } else {
-    // We cache *un-cleaned* spellings. This gives us 100% fidelity with the
-    // source code.
-    StringRef s(T.getLiteralData(), T.getLength());
-
-    // Get the string entry.
-    auto &E = *CachedStrs.insert(std::make_pair(s, OffsetOpt())).first;
-
-    // If this is a new string entry, bump the PTH offset.
-    if (!E.second.hasOffset()) {
-      E.second.setOffset(CurStrOffset);
-      StrEntries.push_back(&E);
-      CurStrOffset += s.size() + 1;
-    }
-
-    // Emit the relative offset into the PTH file for the spelling string.
-    Emit32(E.second.getOffset());
-  }
-
-  // Emit the offset into the original source file of this token so that we
-  // can reconstruct its SourceLocation.
-  Emit32(PP.getSourceManager().getFileOffset(T.getLocation()));
-}
-
-PTHEntry PTHWriter::LexTokens(Lexer& L) {
-  // Pad 0's so that we emit tokens to a 4-byte alignment.
-  // This speed up reading them back in.
-  using namespace llvm::support;
-  endian::Writer LE(Out, little);
-  uint32_t TokenOff = Out.tell();
-  for (uint64_t N = llvm::OffsetToAlignment(TokenOff, 4); N; --N, ++TokenOff)
-    LE.write<uint8_t>(0);
-
-  // Keep track of matching '#if' ... '#endif'.
-  typedef std::vector<std::pair<Offset, unsigned> > PPCondTable;
-  PPCondTable PPCond;
-  std::vector<unsigned> PPStartCond;
-  bool ParsingPreprocessorDirective = false;
-  Token Tok;
-
-  do {
-    L.LexFromRawLexer(Tok);
-  NextToken:
-
-    if ((Tok.isAtStartOfLine() || Tok.is(tok::eof)) &&
-        ParsingPreprocessorDirective) {
-      // Insert an eod token into the token cache.  It has the same
-      // position as the next token that is not on the same line as the
-      // preprocessor directive.  Observe that we continue processing
-      // 'Tok' when we exit this branch.
-      Token Tmp = Tok;
-      Tmp.setKind(tok::eod);
-      Tmp.clearFlag(Token::StartOfLine);
-      Tmp.setIdentifierInfo(nullptr);
-      EmitToken(Tmp);
-      ParsingPreprocessorDirective = false;
-    }
-
-    if (Tok.is(tok::raw_identifier)) {
-      PP.LookUpIdentifierInfo(Tok);
-      EmitToken(Tok);
-      continue;
-    }
-
-    if (Tok.is(tok::hash) && Tok.isAtStartOfLine()) {
-      // Special processing for #include.  Store the '#' token and lex
-      // the next token.
-      assert(!ParsingPreprocessorDirective);
-      Offset HashOff = (Offset) Out.tell();
-
-      // Get the next token.
-      Token NextTok;
-      L.LexFromRawLexer(NextTok);
-
-      // If we see the start of line, then we had a null directive "#".  In
-      // this case, discard both tokens.
-      if (NextTok.isAtStartOfLine())
-        goto NextToken;
-
-      // The token is the start of a directive.  Emit it.
-      EmitToken(Tok);
-      Tok = NextTok;
-
-      // Did we see 'include'/'import'/'include_next'?
-      if (Tok.isNot(tok::raw_identifier)) {
-        EmitToken(Tok);
-        continue;
-      }
-
-      IdentifierInfo* II = PP.LookUpIdentifierInfo(Tok);
-      tok::PPKeywordKind K = II->getPPKeywordID();
-
-      ParsingPreprocessorDirective = true;
-
-      switch (K) {
-      case tok::pp_not_keyword:
-        // Invalid directives "#foo" can occur in #if 0 blocks etc, just pass
-        // them through.
-      default:
-        break;
-
-      case tok::pp_include:
-      case tok::pp_import:
-      case tok::pp_include_next: {
-        // Save the 'include' token.
-        EmitToken(Tok);
-        // Lex the next token as an include string.
-        L.setParsingPreprocessorDirective(true);
-        L.LexIncludeFilename(Tok);
-        L.setParsingPreprocessorDirective(false);
-        assert(!Tok.isAtStartOfLine());
-        if (Tok.is(tok::raw_identifier))
-          PP.LookUpIdentifierInfo(Tok);
-
-        break;
-      }
-      case tok::pp_if:
-      case tok::pp_ifdef:
-      case tok::pp_ifndef: {
-        // Add an entry for '#if' and friends.  We initially set the target
-        // index to 0.  This will get backpatched when we hit #endif.
-        PPStartCond.push_back(PPCond.size());
-        PPCond.push_back(std::make_pair(HashOff, 0U));
-        break;
-      }
-      case tok::pp_endif: {
-        // Add an entry for '#endif'.  We set the target table index to itself.
-        // This will later be set to zero when emitting to the PTH file.  We
-        // use 0 for uninitialized indices because that is easier to debug.
-        unsigned index = PPCond.size();
-        // Backpatch the opening '#if' entry.
-        assert(!PPStartCond.empty());
-        assert(PPCond.size() > PPStartCond.back());
-        assert(PPCond[PPStartCond.back()].second == 0);
-        PPCond[PPStartCond.back()].second = index;
-        PPStartCond.pop_back();
-        // Add the new entry to PPCond.
-        PPCond.push_back(std::make_pair(HashOff, index));
-        EmitToken(Tok);
-
-        // Some files have gibberish on the same line as '#endif'.
-        // Discard these tokens.
-        do
-          L.LexFromRawLexer(Tok);
-        while (Tok.isNot(tok::eof) && !Tok.isAtStartOfLine());
-        // We have the next token in hand.
-        // Don't immediately lex the next one.
-        goto NextToken;
-      }
-      case tok::pp_elif:
-      case tok::pp_else: {
-        // Add an entry for #elif or #else.
-        // This serves as both a closing and opening of a conditional block.
-        // This means that its entry will get backpatched later.
-        unsigned index = PPCond.size();
-        // Backpatch the previous '#if' entry.
-        assert(!PPStartCond.empty());
-        assert(PPCond.size() > PPStartCond.back());
-        assert(PPCond[PPStartCond.back()].second == 0);
-        PPCond[PPStartCond.back()].second = index;
-        PPStartCond.pop_back();
-        // Now add '#elif' as a new block opening.
-        PPCond.push_back(std::make_pair(HashOff, 0U));
-        PPStartCond.push_back(index);
-        break;
-      }
-      }
-    }
-
-    EmitToken(Tok);
-  }
-  while (Tok.isNot(tok::eof));
-
-  assert(PPStartCond.empty() && "Error: imblanced preprocessor conditionals.");
-
-  // Next write out PPCond.
-  Offset PPCondOff = (Offset) Out.tell();
-
-  // Write out the size of PPCond so that clients can identifer empty tables.
-  Emit32(PPCond.size());
-
-  for (unsigned i = 0, e = PPCond.size(); i!=e; ++i) {
-    Emit32(PPCond[i].first - TokenOff);
-    uint32_t x = PPCond[i].second;
-    assert(x != 0 && "PPCond entry not backpatched.");
-    // Emit zero for #endifs.  This allows us to do checking when
-    // we read the PTH file back in.
-    Emit32(x == i ? 0 : x);
-  }
-
-  return PTHEntry(TokenOff, PPCondOff);
-}
-
-Offset PTHWriter::EmitCachedSpellings() {
-  // Write each cached strings to the PTH file.
-  Offset SpellingsOff = Out.tell();
-
-  for (std::vector<llvm::StringMapEntry<OffsetOpt>*>::iterator
-       I = StrEntries.begin(), E = StrEntries.end(); I!=E; ++I)
-    EmitBuf((*I)->getKeyData(), (*I)->getKeyLength()+1 /*nul included*/);
-
-  return SpellingsOff;
-}
-
-static uint32_t swap32le(uint32_t X) {
-  return llvm::support::endian::byte_swap<uint32_t, llvm::support::little>(X);
-}
-
-static void pwrite32le(raw_pwrite_stream &OS, uint32_t Val, uint64_t &Off) {
-  uint32_t LEVal = swap32le(Val);
-  OS.pwrite(reinterpret_cast<const char *>(&LEVal), 4, Off);
-  Off += 4;
-}
-
-void PTHWriter::GeneratePTH(StringRef MainFile) {
-  // Generate the prologue.
-  Out << "cfe-pth" << '\0';
-  Emit32(PTHManager::Version);
-
-  // Leave 4 words for the prologue.
-  Offset PrologueOffset = Out.tell();
-  for (unsigned i = 0; i < 4; ++i)
-    Emit32(0);
-
-  // Write the name of the MainFile.
-  if (!MainFile.empty()) {
-    EmitString(MainFile);
-  } else {
-    // String with 0 bytes.
-    Emit16(0);
-  }
-  Emit8(0);
-
-  // Iterate over all the files in SourceManager.  Create a lexer
-  // for each file and cache the tokens.
-  SourceManager &SM = PP.getSourceManager();
-  const LangOptions &LOpts = PP.getLangOpts();
-
-  for (SourceManager::fileinfo_iterator I = SM.fileinfo_begin(),
-       E = SM.fileinfo_end(); I != E; ++I) {
-    const SrcMgr::ContentCache &C = *I->second;
-    const FileEntry *FE = C.OrigEntry;
-
-    // FIXME: Handle files with non-absolute paths.
-    if (llvm::sys::path::is_relative(FE->getName()))
-      continue;
-
-    const llvm::MemoryBuffer *B = C.getBuffer(PP.getDiagnostics(), SM);
-    if (!B) continue;
-
-    FileID FID = SM.createFileID(FE, SourceLocation(), SrcMgr::C_User);
-    const llvm::MemoryBuffer *FromFile = SM.getBuffer(FID);
-    Lexer L(FID, FromFile, SM, LOpts);
-    PM.insert(FE, LexTokens(L));
-  }
-
-  // Write out the identifier table.
-  const std::pair<Offset,Offset> &IdTableOff = EmitIdentifierTable();
-
-  // Write out the cached strings table.
-  Offset SpellingOff = EmitCachedSpellings();
-
-  // Write out the file table.
-  Offset FileTableOff = EmitFileTable();
-
-  // Finally, write the prologue.
-  uint64_t Off = PrologueOffset;
-  pwrite32le(Out, IdTableOff.first, Off);
-  pwrite32le(Out, IdTableOff.second, Off);
-  pwrite32le(Out, FileTableOff, Off);
-  pwrite32le(Out, SpellingOff, Off);
-}
-
-namespace {
-/// StatListener - A simple "interpose" object used to monitor stat calls
-/// invoked by FileManager while processing the original sources used
-/// as input to PTH generation.  StatListener populates the PTHWriter's
-/// file map with stat information for directories as well as negative stats.
-/// Stat information for files are populated elsewhere.
-class StatListener : public FileSystemStatCache {
-  PTHMap &PM;
-public:
-  StatListener(PTHMap &pm) : PM(pm) {}
-  ~StatListener() override {}
-
-  LookupResult getStat(StringRef Path, FileData &Data, bool isFile,
-                       std::unique_ptr<llvm::vfs::File> *F,
-                       llvm::vfs::FileSystem &FS) override {
-    LookupResult Result = statChained(Path, Data, isFile, F, FS);
-
-    if (Result == CacheMissing) // Failed 'stat'.
-      PM.insert(PTHEntryKeyVariant(Path), PTHEntry());
-    else if (Data.IsDirectory) {
-      // Only cache directories with absolute paths.
-      if (llvm::sys::path::is_relative(Path))
-        return Result;
-
-      PM.insert(PTHEntryKeyVariant(&Data, Path), PTHEntry());
-    }
-
-    return Result;
-  }
-};
-} // end anonymous namespace
-
-void clang::CacheTokens(Preprocessor &PP, raw_pwrite_stream *OS) {
-  // Get the name of the main file.
-  const SourceManager &SrcMgr = PP.getSourceManager();
-  const FileEntry *MainFile = SrcMgr.getFileEntryForID(SrcMgr.getMainFileID());
-  SmallString<128> MainFilePath(MainFile->getName());
-
-  llvm::sys::fs::make_absolute(MainFilePath);
-
-  // Create the PTHWriter.
-  PTHWriter PW(*OS, PP);
-
-  // Install the 'stat' system call listener in the FileManager.
-  auto StatCacheOwner = llvm::make_unique<StatListener>(PW.getPM());
-  StatListener *StatCache = StatCacheOwner.get();
-  PP.getFileManager().addStatCache(std::move(StatCacheOwner),
-                                   /*AtBeginning=*/true);
-
-  // Lex through the entire file.  This will populate SourceManager with
-  // all of the header information.
-  Token Tok;
-  PP.EnterMainSourceFile();
-  do { PP.Lex(Tok); } while (Tok.isNot(tok::eof));
-
-  // Generate the PTH file.
-  PP.getFileManager().removeStatCache(StatCache);
-  PW.GeneratePTH(MainFilePath.str());
-}
-
-//===----------------------------------------------------------------------===//
-
-namespace {
-class PTHIdKey {
-public:
-  const IdentifierInfo* II;
-  uint32_t FileOffset;
-};
-
-class PTHIdentifierTableTrait {
-public:
-  typedef PTHIdKey* key_type;
-  typedef key_type  key_type_ref;
-
-  typedef uint32_t  data_type;
-  typedef data_type data_type_ref;
-
-  typedef unsigned hash_value_type;
-  typedef unsigned offset_type;
-
-  static hash_value_type ComputeHash(PTHIdKey* key) {
-    return llvm::djbHash(key->II->getName());
-  }
-
-  static std::pair<unsigned,unsigned>
-  EmitKeyDataLength(raw_ostream& Out, const PTHIdKey* key, uint32_t) {
-    using namespace llvm::support;
-    unsigned n = key->II->getLength() + 1;
-    endian::write<uint16_t>(Out, n, little);
-    return std::make_pair(n, sizeof(uint32_t));
-  }
-
-  static void EmitKey(raw_ostream& Out, PTHIdKey* key, unsigned n) {
-    // Record the location of the key data.  This is used when generating
-    // the mapping from persistent IDs to strings.
-    key->FileOffset = Out.tell();
-    Out.write(key->II->getNameStart(), n);
-  }
-
-  static void EmitData(raw_ostream& Out, PTHIdKey*, uint32_t pID,
-                       unsigned) {
-    using namespace llvm::support;
-    endian::write<uint32_t>(Out, pID, little);
-  }
-};
-} // end anonymous namespace
-
-/// EmitIdentifierTable - Emits two tables to the PTH file.  The first is
-///  a hashtable mapping from identifier strings to persistent IDs.  The second
-///  is a straight table mapping from persistent IDs to string data (the
-///  keys of the first table).
-///
-std::pair<Offset,Offset> PTHWriter::EmitIdentifierTable() {
-  // Build two maps:
-  //  (1) an inverse map from persistent IDs -> (IdentifierInfo*,Offset)
-  //  (2) a map from (IdentifierInfo*, Offset)* -> persistent IDs
-
-  // Note that we use 'calloc', so all the bytes are 0.
-  PTHIdKey *IIDMap = static_cast<PTHIdKey*>(
-      llvm::safe_calloc(idcount, sizeof(PTHIdKey)));
-
-  // Create the hashtable.
-  llvm::OnDiskChainedHashTableGenerator<PTHIdentifierTableTrait> IIOffMap;
-
-  // Generate mapping from persistent IDs -> IdentifierInfo*.
-  for (IDMap::iterator I = IM.begin(), E = IM.end(); I != E; ++I) {
-    // Decrement by 1 because we are using a vector for the lookup and
-    // 0 is reserved for NULL.
-    assert(I->second > 0);
-    assert(I->second-1 < idcount);
-    unsigned idx = I->second-1;
-
-    // Store the mapping from persistent ID to IdentifierInfo*
-    IIDMap[idx].II = I->first;
-
-    // Store the reverse mapping in a hashtable.
-    IIOffMap.insert(&IIDMap[idx], I->second);
-  }
-
-  // Write out the inverse map first.  This causes the PCIDKey entries to
-  // record PTH file offsets for the string data.  This is used to write
-  // the second table.
-  Offset StringTableOffset = IIOffMap.Emit(Out);
-
-  // Now emit the table mapping from persistent IDs to PTH file offsets.
-  Offset IDOff = Out.tell();
-  Emit32(idcount);  // Emit the number of identifiers.
-  for (unsigned i = 0 ; i < idcount; ++i)
-    Emit32(IIDMap[i].FileOffset);
-
-  // Finally, release the inverse map.
-  free(IIDMap);
-
-  return std::make_pair(IDOff, StringTableOffset);
-}

lib/Frontend/ChainedIncludesSource.cpp

IntrusiveRefCntPtr<ExternalSemaSource> clang::createChainedIncludesSource(
 
   for (unsigned i = 0, e = includes.size(); i != e; ++i) {
     bool firstInclude = (i == 0);
     std::unique_ptr<CompilerInvocation> CInvok;
     CInvok.reset(new CompilerInvocation(CI.getInvocation()));
 
     CInvok->getPreprocessorOpts().ChainedIncludes.clear();
     CInvok->getPreprocessorOpts().ImplicitPCHInclude.clear();
-    CInvok->getPreprocessorOpts().ImplicitPTHInclude.clear();
     CInvok->getPreprocessorOpts().DisablePCHValidation = true;
     CInvok->getPreprocessorOpts().Includes.clear();
     CInvok->getPreprocessorOpts().MacroIncludes.clear();
     CInvok->getPreprocessorOpts().Macros.clear();
 
     CInvok->getFrontendOpts().Inputs.clear();
     FrontendInputFile InputFile(includes[i], IK);
     CInvok->getFrontendOpts().Inputs.push_back(InputFile);

lib/Frontend/CompilerInstance.cpp

 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/LogDiagnosticPrinter.h"
 #include "clang/Frontend/SerializedDiagnosticPrinter.h"
 #include "clang/Frontend/TextDiagnosticPrinter.h"
 #include "clang/Frontend/Utils.h"
 #include "clang/Frontend/VerifyDiagnosticConsumer.h"
 #include "clang/Lex/HeaderSearch.h"
-#include "clang/Lex/PTHManager.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Sema/CodeCompleteConsumer.h"
 #include "clang/Sema/Sema.h"
 #include "clang/Serialization/ASTReader.h"
 #include "clang/Serialization/GlobalModuleIndex.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Support/BuryPointer.h"

 // Preprocessor
 
 void CompilerInstance::createPreprocessor(TranslationUnitKind TUKind) {
   const PreprocessorOptions &PPOpts = getPreprocessorOpts();
 
   // The module manager holds a reference to the old preprocessor (if any).
   ModuleManager.reset();
 
-  // Create a PTH manager if we are using some form of a token cache.
-  PTHManager *PTHMgr = nullptr;
-  if (!PPOpts.TokenCache.empty())
-    PTHMgr = PTHManager::Create(PPOpts.TokenCache, getDiagnostics());
-
   // Create the Preprocessor.
   HeaderSearch *HeaderInfo =
       new HeaderSearch(getHeaderSearchOptsPtr(), getSourceManager(),
                        getDiagnostics(), getLangOpts(), &getTarget());
   PP = std::make_shared<Preprocessor>(
       Invocation->getPreprocessorOptsPtr(), getDiagnostics(), getLangOpts(),
-      getSourceManager(), getPCMCache(), *HeaderInfo, *this, PTHMgr,
+      getSourceManager(), getPCMCache(), *HeaderInfo, *this,
+      /*IdentifierInfoLookup=*/nullptr,
       /*OwnsHeaderSearch=*/true, TUKind);
   getTarget().adjust(getLangOpts());
   PP->Initialize(getTarget(), getAuxTarget());
 
-  // Note that this is different then passing PTHMgr to Preprocessor's ctor.
-  // That argument is used as the IdentifierInfoLookup argument to
-  // IdentifierTable's ctor.
-  if (PTHMgr) {
-    PTHMgr->setPreprocessor(&*PP);
-    PP->setPTHManager(PTHMgr);
-  }
-
   if (PPOpts.DetailedRecord)
     PP->createPreprocessingRecord();
 
   // Apply remappings to the source manager.
   InitializeFileRemapping(PP->getDiagnostics(), PP->getSourceManager(),
                           PP->getFileManager(), PPOpts);
 
   // Predefine macros and configure the preprocessor.

lib/Frontend/CompilerInvocation.cpp

if (const Arg *A = Args.getLastArg(OPT_Action_Group)) {
     case OPT_emit_module:
       Opts.ProgramAction = frontend::GenerateModule; break;
     case OPT_emit_module_interface:
       Opts.ProgramAction = frontend::GenerateModuleInterface; break;
     case OPT_emit_header_module:
       Opts.ProgramAction = frontend::GenerateHeaderModule; break;
     case OPT_emit_pch:
       Opts.ProgramAction = frontend::GeneratePCH; break;
-    case OPT_emit_pth:
-      Opts.ProgramAction = frontend::GeneratePTH; break;
     case OPT_init_only:
       Opts.ProgramAction = frontend::InitOnly; break;
     case OPT_fsyntax_only:
       Opts.ProgramAction = frontend::ParseSyntaxOnly; break;
     case OPT_module_file_info:
       Opts.ProgramAction = frontend::ModuleFileInfo; break;
     case OPT_verify_pch:
       Opts.ProgramAction = frontend::VerifyPCH; break;
static bool isStrictlyPreprocessorAction(frontend::ActionKind Action) {
   case frontend::EmitLLVMOnly:
   case frontend::EmitCodeGenOnly:
   case frontend::EmitObj:
   case frontend::FixIt:
   case frontend::GenerateModule:
   case frontend::GenerateModuleInterface:
   case frontend::GenerateHeaderModule:
   case frontend::GeneratePCH:
-  case frontend::GeneratePTH:
   case frontend::ParseSyntaxOnly:
   case frontend::ModuleFileInfo:
   case frontend::VerifyPCH:
   case frontend::PluginAction:
   case frontend::RewriteObjC:
   case frontend::RewriteTest:
   case frontend::RunAnalysis:
   case frontend::TemplightDump:
static bool isStrictlyPreprocessorAction(frontend::ActionKind Action) {
   }
   llvm_unreachable("invalid frontend action");
 }
 
 static void ParsePreprocessorArgs(PreprocessorOptions &Opts, ArgList &Args,
                                   DiagnosticsEngine &Diags,
                                   frontend::ActionKind Action) {
   Opts.ImplicitPCHInclude = Args.getLastArgValue(OPT_include_pch);
-  Opts.ImplicitPTHInclude = Args.getLastArgValue(OPT_include_pth);
   Opts.PCHWithHdrStop = Args.hasArg(OPT_pch_through_hdrstop_create) ||
                         Args.hasArg(OPT_pch_through_hdrstop_use);
   Opts.PCHWithHdrStopCreate = Args.hasArg(OPT_pch_through_hdrstop_create);
   Opts.PCHThroughHeader = Args.getLastArgValue(OPT_pch_through_header_EQ);
-  if (const Arg *A = Args.getLastArg(OPT_token_cache))
-      Opts.TokenCache = A->getValue();
-  else
-    Opts.TokenCache = Opts.ImplicitPTHInclude;
   Opts.UsePredefines = !Args.hasArg(OPT_undef);
   Opts.DetailedRecord = Args.hasArg(OPT_detailed_preprocessing_record);
   Opts.DisablePCHValidation = Args.hasArg(OPT_fno_validate_pch);
   Opts.AllowPCHWithCompilerErrors = Args.hasArg(OPT_fallow_pch_with_errors);
 
   Opts.DumpDeserializedPCHDecls = Args.hasArg(OPT_dump_deserialized_pch_decls);
   for (const auto *A : Args.filtered(OPT_error_on_deserialized_pch_decl))
     Opts.DeserializedPCHDeclsToErrorOn.insert(A->getValue());

lib/Frontend/FrontendActions.cpp

void DumpTokensAction::ExecuteAction() {
   PP.EnterMainSourceFile();
   do {
     PP.Lex(Tok);
     PP.DumpToken(Tok, true);
     llvm::errs() << "\n";
   } while (Tok.isNot(tok::eof));
 }
 
-void GeneratePTHAction::ExecuteAction() {
-  CompilerInstance &CI = getCompilerInstance();
-  std::unique_ptr<raw_pwrite_stream> OS =
-      CI.createDefaultOutputFile(true, getCurrentFile());
-  if (!OS)
-    return;
-
-  CacheTokens(CI.getPreprocessor(), OS.get());
-}
-
 void PreprocessOnlyAction::ExecuteAction() {
   Preprocessor &PP = getCompilerInstance().getPreprocessor();
 
   // Ignore unknown pragmas.
   PP.IgnorePragmas();
 
   Token Tok;
   // Start parsing the specified input file.

lib/Frontend/InitPreprocessor.cpp

 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/SyncScope.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/Version.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/FrontendOptions.h"
 #include "clang/Frontend/Utils.h"
 #include "clang/Lex/HeaderSearch.h"
-#include "clang/Lex/PTHManager.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Serialization/ASTReader.h"
 #include "llvm/ADT/APFloat.h"
 using namespace clang;
 
 static bool MacroBodyEndsInBackslash(StringRef MacroBody) {
   while (!MacroBody.empty() && isWhitespace(MacroBody.back()))

 }
 
 static void AddImplicitIncludeMacros(MacroBuilder &Builder, StringRef File) {
   Builder.append(Twine("#__include_macros \"") + File + "\"");
   // Marker token to stop the __include_macros fetch loop.
   Builder.append("##"); // ##?
 }
 
-/// AddImplicitIncludePTH - Add an implicit \#include using the original file
-/// used to generate a PTH cache.
-static void AddImplicitIncludePTH(MacroBuilder &Builder, Preprocessor &PP,
-                                  StringRef ImplicitIncludePTH) {
-  PTHManager *P = PP.getPTHManager();
-  // Null check 'P' in the corner case where it couldn't be created.
-  const char *OriginalFile = P ? P->getOriginalSourceFile() : nullptr;
-
-  if (!OriginalFile) {
-    PP.getDiagnostics().Report(diag::err_fe_pth_file_has_no_source_header)
-      << ImplicitIncludePTH;
-    return;
-  }
-
-  AddImplicitInclude(Builder, OriginalFile);
-}
-
 /// Add an implicit \#include using the original file used to generate
 /// a PCH file.
 static void AddImplicitIncludePCH(MacroBuilder &Builder, Preprocessor &PP,
                                   const PCHContainerReader &PCHContainerRdr,
                                   StringRef ImplicitIncludePCH) {
   std::string OriginalFile =
       ASTReader::getOriginalSourceFile(ImplicitIncludePCH, PP.getFileManager(),
                                        PCHContainerRdr, PP.getDiagnostics());
void clang::InitializePreprocessor(
   // any -include directives.
   for (unsigned i = 0, e = InitOpts.MacroIncludes.size(); i != e; ++i)
     AddImplicitIncludeMacros(Builder, InitOpts.MacroIncludes[i]);
 
   // Process -include-pch/-include-pth directives.
   if (!InitOpts.ImplicitPCHInclude.empty())
     AddImplicitIncludePCH(Builder, PP, PCHContainerRdr,
                           InitOpts.ImplicitPCHInclude);
-  if (!InitOpts.ImplicitPTHInclude.empty())
-    AddImplicitIncludePTH(Builder, PP, InitOpts.ImplicitPTHInclude);
 
   // Process -include directives.
   for (unsigned i = 0, e = InitOpts.Includes.size(); i != e; ++i) {
     const std::string &Path = InitOpts.Includes[i];
     AddImplicitInclude(Builder, Path);
   }
 
   // Instruct the preprocessor to skip the preamble.
   PP.setSkipMainFilePreamble(InitOpts.PrecompiledPreambleBytes.first,
                              InitOpts.PrecompiledPreambleBytes.second);
 
   // Copy PredefinedBuffer into the Preprocessor.
   PP.setPredefines(Predefines.str());
 }

lib/FrontendTool/ExecuteCompilerInvocation.cpp

CreateFrontendBaseAction(CompilerInstance &CI) {
   case FixIt:                  return llvm::make_unique<FixItAction>();
   case GenerateModule:
     return llvm::make_unique<GenerateModuleFromModuleMapAction>();
   case GenerateModuleInterface:
     return llvm::make_unique<GenerateModuleInterfaceAction>();
   case GenerateHeaderModule:
     return llvm::make_unique<GenerateHeaderModuleAction>();
   case GeneratePCH:            return llvm::make_unique<GeneratePCHAction>();
-  case GeneratePTH:            return llvm::make_unique<GeneratePTHAction>();
   case InitOnly:               return llvm::make_unique<InitOnlyAction>();
   case ParseSyntaxOnly:        return llvm::make_unique<SyntaxOnlyAction>();
   case ModuleFileInfo:         return llvm::make_unique<DumpModuleInfoAction>();
   case VerifyPCH:              return llvm::make_unique<VerifyPCHAction>();
   case TemplightDump:          return llvm::make_unique<TemplightDumpAction>();
 
   case PluginAction: {
     for (FrontendPluginRegistry::iterator it =

lib/Lex/CMakeLists.txt

add_clang_library(clangLex
   ModuleMap.cpp
   PPCaching.cpp
   PPCallbacks.cpp
   PPConditionalDirectiveRecord.cpp
   PPDirectives.cpp
   PPExpressions.cpp
   PPLexerChange.cpp
   PPMacroExpansion.cpp
-  PTHLexer.cpp
   Pragma.cpp
   PreprocessingRecord.cpp
   Preprocessor.cpp
   PreprocessorLexer.cpp
   ScratchBuffer.cpp
   TokenConcatenation.cpp
   TokenLexer.cpp
 
   LINK_LIBS
   clangBasic
   )

lib/Lex/PPDirectives.cpp

 #include "clang/Lex/LiteralSupport.h"
 #include "clang/Lex/MacroInfo.h"
 #include "clang/Lex/ModuleLoader.h"
 #include "clang/Lex/ModuleMap.h"
 #include "clang/Lex/PPCallbacks.h"
 #include "clang/Lex/Pragma.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
-#include "clang/Lex/PTHLexer.h"
 #include "clang/Lex/Token.h"
 #include "clang/Lex/VariadicMacroSupport.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/ADT/StringRef.h"
void Preprocessor::SkipExcludedConditionalBlock(SourceLocation HashTokenLoc,
   assert(!CurTokenLexer && CurPPLexer && "Lexing a macro, not a file?");
 
   if (PreambleConditionalStack.reachedEOFWhileSkipping())
     PreambleConditionalStack.clearSkipInfo();
   else
     CurPPLexer->pushConditionalLevel(IfTokenLoc, /*isSkipping*/ false,
                                      FoundNonSkipPortion, FoundElse);
 
-  if (CurPTHLexer) {
-    PTHSkipExcludedConditionalBlock();
-    return;
-  }
-
   // Enter raw mode to disable identifier lookup (and thus macro expansion),
   // disabling warnings, etc.
   CurPPLexer->LexingRawMode = true;
   Token Tok;
   while (true) {
     CurLexer->Lex(Tok);
 
     if (Tok.is(tok::code_completion)) {
void Preprocessor::SkipExcludedConditionalBlock(SourceLocation HashTokenLoc,
   // The last skipped range isn't actually skipped yet if it's truncated
   // by the end of the preamble; we'll resume parsing after the preamble.
   if (Callbacks && (Tok.isNot(tok::eof) || !isRecordingPreamble()))
     Callbacks->SourceRangeSkipped(
         SourceRange(HashTokenLoc, CurPPLexer->getSourceLocation()),
         Tok.getLocation());
 }
 
-void Preprocessor::PTHSkipExcludedConditionalBlock() {
-  while (true) {
-    assert(CurPTHLexer);
-    assert(CurPTHLexer->LexingRawMode == false);
-
-    // Skip to the next '#else', '#elif', or #endif.
-    if (CurPTHLexer->SkipBlock()) {
-      // We have reached an #endif.  Both the '#' and 'endif' tokens
-      // have been consumed by the PTHLexer.  Just pop off the condition level.
-      PPConditionalInfo CondInfo;
-      bool InCond = CurPTHLexer->popConditionalLevel(CondInfo);
-      (void)InCond;  // Silence warning in no-asserts mode.
-      assert(!InCond && "Can't be skipping if not in a conditional!");
-      break;
-    }
-
-    // We have reached a '#else' or '#elif'.  Lex the next token to get
-    // the directive flavor.
-    Token Tok;
-    LexUnexpandedToken(Tok);
-
-    // We can actually look up the IdentifierInfo here since we aren't in
-    // raw mode.
-    tok::PPKeywordKind K = Tok.getIdentifierInfo()->getPPKeywordID();
-
-    if (K == tok::pp_else) {
-      // #else: Enter the else condition.  We aren't in a nested condition
-      //  since we skip those. We're always in the one matching the last
-      //  blocked we skipped.
-      PPConditionalInfo &CondInfo = CurPTHLexer->peekConditionalLevel();
-      // Note that we've seen a #else in this conditional.
-      CondInfo.FoundElse = true;
-
-      // If the #if block wasn't entered then enter the #else block now.
-      if (!CondInfo.FoundNonSkip) {
-        CondInfo.FoundNonSkip = true;
-
-        // Scan until the eod token.
-        CurPTHLexer->ParsingPreprocessorDirective = true;
-        DiscardUntilEndOfDirective();
-        CurPTHLexer->ParsingPreprocessorDirective = false;
-
-        break;
-      }
-
-      // Otherwise skip this block.
-      continue;
-    }
-
-    assert(K == tok::pp_elif);
-    PPConditionalInfo &CondInfo = CurPTHLexer->peekConditionalLevel();
-
-    // If this is a #elif with a #else before it, report the error.
-    if (CondInfo.FoundElse)
-      Diag(Tok, diag::pp_err_elif_after_else);
-
-    // If this is in a skipping block or if we're already handled this #if
-    // block, don't bother parsing the condition.  We just skip this block.
-    if (CondInfo.FoundNonSkip)
-      continue;
-
-    // Evaluate the condition of the #elif.
-    IdentifierInfo *IfNDefMacro = nullptr;
-    CurPTHLexer->ParsingPreprocessorDirective = true;
-    bool ShouldEnter = EvaluateDirectiveExpression(IfNDefMacro).Conditional;
-    CurPTHLexer->ParsingPreprocessorDirective = false;
-
-    // If this condition is true, enter it!
-    if (ShouldEnter) {
-      CondInfo.FoundNonSkip = true;
-      break;
-    }
-
-    // Otherwise, skip this block and go to the next one.
-  }
-}
-
 Module *Preprocessor::getModuleForLocation(SourceLocation Loc) {
   if (!SourceMgr.isInMainFile(Loc)) {
     // Try to determine the module of the include directive.
     // FIXME: Look into directly passing the FileEntry from LookupFile instead.
     FileID IDOfIncl = SourceMgr.getFileID(SourceMgr.getExpansionLoc(Loc));
     if (const FileEntry *EntryOfIncl = SourceMgr.getFileEntryForID(IDOfIncl)) {
       // The include comes from an included file.
       return HeaderInfo.getModuleMap()
if (Callbacks) {
     Callbacks->FileChanged(CurPPLexer->getSourceLocation(), Reason, FileKind);
   }
 }
 
 /// HandleUserDiagnosticDirective - Handle a #warning or #error directive.
 ///
 void Preprocessor::HandleUserDiagnosticDirective(Token &Tok,
                                                  bool isWarning) {
-  // PTH doesn't emit #warning or #error directives.
-  if (CurPTHLexer)
-    return CurPTHLexer->DiscardToEndOfLine();
-
   // Read the rest of the line raw.  We do this because we don't want macros
   // to be expanded and we don't require that the tokens be valid preprocessing
   // tokens.  For example, this is allowed: "#warning `   'foo".  GCC does
   // collapse multiple consecutive white space between tokens, but this isn't
   // specified by the standard.
   SmallString<128> Message;
   CurLexer->ReadToEndOfLine(&Message);
 
if (ShouldEnter && File && SuggestedModule && getLangOpts().Modules &&
     } else if (Imported.isConfigMismatch()) {
       // On a configuration mismatch, enter the header textually. We still know
       // that it's part of the corresponding module.
     } else {
       // We hit an error processing the import. Bail out.
       if (hadModuleLoaderFatalFailure()) {
         // With a fatal failure in the module loader, we abort parsing.
         Token &Result = IncludeTok;
-        if (CurLexer) {
-          Result.startToken();
-          CurLexer->FormTokenWithChars(Result, CurLexer->BufferEnd, tok::eof);
-          CurLexer->cutOffLexing();
-        } else {
-          assert(CurPTHLexer && "#include but no current lexer set!");
-          CurPTHLexer->getEOF(Result);
-        }
+        assert(CurLexer && "#include but no current lexer set!");
       }
       return;
     }
   }
 
   // The #included file will be considered to be a system header if either it is
   // in a system include directory, or if the #includer is a system include
   // header.

lib/Lex/PPLexerChange.cpp

 
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/LexDiagnostic.h"
 #include "clang/Lex/MacroInfo.h"
-#include "clang/Lex/PTHManager.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 using namespace clang;
 
 PPCallbacks::~PPCallbacks() {}
 

 bool Preprocessor::EnterSourceFile(FileID FID, const DirectoryLookup *CurDir,
                                    SourceLocation Loc) {
   assert(!CurTokenLexer && "Cannot #include a file inside a macro!");
   ++NumEnteredSourceFiles;
 
   if (MaxIncludeStackDepth < IncludeMacroStack.size())
     MaxIncludeStackDepth = IncludeMacroStack.size();
 
-  if (PTH) {
-    if (PTHLexer *PL = PTH->CreateLexer(FID)) {
-      EnterSourceFileWithPTH(PL, CurDir);
-      return false;
-    }
-  }
-
   // Get the MemoryBuffer for this FID, if it fails, we fail.
   bool Invalid = false;
   const llvm::MemoryBuffer *InputFile =
     getSourceManager().getBuffer(FID, Loc, &Invalid);
   if (Invalid) {
     SourceLocation FileStart = SourceMgr.getLocForStartOfFile(FID);
     Diag(Loc, diag::err_pp_error_opening_file)
       << std::string(SourceMgr.getBufferName(FileStart)) << "";
if (Callbacks && !CurLexer->Is_PragmaLexer) {
     SrcMgr::CharacteristicKind FileType =
        SourceMgr.getFileCharacteristic(CurLexer->getFileLoc());
 
     Callbacks->FileChanged(CurLexer->getFileLoc(),
                            PPCallbacks::EnterFile, FileType);
   }
 }
 
-/// EnterSourceFileWithPTH - Add a source file to the top of the include stack
-/// and start getting tokens from it using the PTH cache.
-void Preprocessor::EnterSourceFileWithPTH(PTHLexer *PL,
-                                          const DirectoryLookup *CurDir) {
-
-  if (CurPPLexer || CurTokenLexer)
-    PushIncludeMacroStack();
-
-  CurDirLookup = CurDir;
-  CurPTHLexer.reset(PL);
-  CurPPLexer = CurPTHLexer.get();
-  CurLexerSubmodule = nullptr;
-  if (CurLexerKind != CLK_LexAfterModuleImport)
-    CurLexerKind = CLK_PTHLexer;
-
-  // Notify the client, if desired, that we are in a new source file.
-  if (Callbacks) {
-    FileID FID = CurPPLexer->getFileID();
-    SourceLocation EnterLoc = SourceMgr.getLocForStartOfFile(FID);
-    SrcMgr::CharacteristicKind FileType =
-      SourceMgr.getFileCharacteristic(EnterLoc);
-    Callbacks->FileChanged(EnterLoc, PPCallbacks::EnterFile, FileType);
-  }
-}
-
 /// EnterMacro - Add a Macro to the top of the include stack and start lexing
 /// tokens from it instead of the current buffer.
 void Preprocessor::EnterMacro(Token &Tok, SourceLocation ILEnd,
                               MacroInfo *Macro, MacroArgs *Args) {
   std::unique_ptr<TokenLexer> TokLexer;
   if (NumCachedTokenLexers == 0) {
     TokLexer = llvm::make_unique<TokenLexer>(Tok, ILEnd, Macro, Args, *this);
   } else {

 /// the current file.  This either returns the EOF token or pops a level off
 /// the include stack and keeps going.
 bool Preprocessor::HandleEndOfFile(Token &Result, bool isEndOfMacro) {
   assert(!CurTokenLexer &&
          "Ending a file when currently in a macro!");
 
   // If we have an unclosed module region from a pragma at the end of a
   // module, complain and close it now.
-  // FIXME: This is not correct if we are building a module from PTH.
   const bool LeavingSubmodule = CurLexer && CurLexerSubmodule;
   if ((LeavingSubmodule || IncludeMacroStack.empty()) &&
       !BuildingSubmoduleStack.empty() &&
       BuildingSubmoduleStack.back().IsPragma) {
     Diag(BuildingSubmoduleStack.back().ImportLoc,
          diag::err_pp_module_begin_without_module_end);
     Module *M = LeaveSubmodule(/*ForPragma*/true);
 
bool Preprocessor::HandleEndOfFile(Token &Result, bool isEndOfMacro) {
   // If this is a #include'd file, pop it off the include stack and continue
   // lexing the #includer file.
   if (!IncludeMacroStack.empty()) {
 
     // If we lexed the code-completion file, act as if we reached EOF.
     if (isCodeCompletionEnabled() && CurPPLexer &&
         SourceMgr.getLocForStartOfFile(CurPPLexer->getFileID()) ==
             CodeCompletionFileLoc) {
-      if (CurLexer) {
-        Result.startToken();
-        CurLexer->FormTokenWithChars(Result, CurLexer->BufferEnd, tok::eof);
-        CurLexer.reset();
-      } else {
-        assert(CurPTHLexer && "Got EOF but no current lexer set!");
-        CurPTHLexer->getEOF(Result);
-        CurPTHLexer.reset();
-      }
+      assert(CurLexer && "Got EOF but no current lexer set!");
 
       CurPPLexer = nullptr;
       recomputeCurLexerKind();
       return true;
     }
 
     if (!isEndOfMacro && CurPPLexer &&
         SourceMgr.getIncludeLoc(CurPPLexer->getFileID()).isValid()) {
if (!isEndOfMacro && CurPPLexer && FoundPCHThroughHeader &&
       LeavingPCHThroughHeader = true;
     } else {
       // Client should lex another token unless we generated an EOM.
       return LeavingSubmodule;
     }
   }
 
   // If this is the end of the main file, form an EOF token.
-  if (CurLexer) {
-    const char *EndPos = getCurLexerEndPos();
-    Result.startToken();
-    CurLexer->BufferPtr = EndPos;
-    CurLexer->FormTokenWithChars(Result, EndPos, tok::eof);
+  assert(CurLexer && "Got EOF but no current lexer set!");
 
-    if (isCodeCompletionEnabled()) {
-      // Inserting the code-completion point increases the source buffer by 1,
-      // but the main FileID was created before inserting the point.
-      // Compensate by reducing the EOF location by 1, otherwise the location
-      // will point to the next FileID.
-      // FIXME: This is hacky, the code-completion point should probably be
-      // inserted before the main FileID is created.
-      if (CurLexer->getFileLoc() == CodeCompletionFileLoc)
-        Result.setLocation(Result.getLocation().getLocWithOffset(-1));
-    }
 
-    if (creatingPCHWithThroughHeader() && !LeavingPCHThroughHeader) {
-      // Reached the end of the compilation without finding the through header.
-      Diag(CurLexer->getFileLoc(), diag::err_pp_through_header_not_seen)
-          << PPOpts->PCHThroughHeader << 0;
-    }
 
-    if (!isIncrementalProcessingEnabled())
-      // We're done with lexing.
-      CurLexer.reset();
-  } else {
-    assert(CurPTHLexer && "Got EOF but no current lexer set!");
-    CurPTHLexer->getEOF(Result);
-    CurPTHLexer.reset();
-  }
 
   if (!isIncrementalProcessingEnabled())
     CurPPLexer = nullptr;
 
   if (TUKind == TU_Complete) {
     // This is the end of the top-level file. 'WarnUnusedMacroLocs' has
     // collected all macro locations that we need to warn because they are not
     // used.

lib/Lex/PPMacroExpansion.cpp

 #include "clang/Lex/DirectoryLookup.h"
 #include "clang/Lex/ExternalPreprocessorSource.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/LexDiagnostic.h"
 #include "clang/Lex/MacroArgs.h"
 #include "clang/Lex/MacroInfo.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorLexer.h"
-#include "clang/Lex/PTHLexer.h"
 #include "clang/Lex/Token.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/FoldingSet.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallString.h"

 /// isNextPPTokenLParen - Determine whether the next preprocessor token to be
 /// lexed is a '('.  If so, consume the token and return true, if not, this
 /// method should have no observable side-effect on the lexed tokens.
 bool Preprocessor::isNextPPTokenLParen() {
   // Do some quick tests for rejection cases.
   unsigned Val;
   if (CurLexer)
     Val = CurLexer->isNextPPTokenLParen();
-  else if (CurPTHLexer)
-    Val = CurPTHLexer->isNextPPTokenLParen();
   else
     Val = CurTokenLexer->isNextTokenLParen();
 
   if (Val == 2) {
     // We have run off the end.  If it's a source file we don't
     // examine enclosing ones (C99 5.1.1.2p4).  Otherwise walk up the
     // macro stack.
     if (CurPPLexer)
       return false;
     for (const IncludeStackInfo &Entry : llvm::reverse(IncludeMacroStack)) {
       if (Entry.TheLexer)
         Val = Entry.TheLexer->isNextPPTokenLParen();
-      else if (Entry.ThePTHLexer)
-        Val = Entry.ThePTHLexer->isNextPPTokenLParen();
       else
         Val = Entry.TheTokenLexer->isNextTokenLParen();
 
       if (Val != 2)
         break;
 
       // Ran off the end of a source file?
       if (Entry.ThePPLexer)

lib/Lex/PTHLexer.cpp

-//===- PTHLexer.cpp - Lex from a token stream -----------------------------===//
-//
-//                     The LLVM Compiler Infrastructure
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-//
-// This file implements the PTHLexer interface.
-//
-//===----------------------------------------------------------------------===//
-
-#include "clang/Lex/PTHLexer.h"
-#include "clang/Basic/Diagnostic.h"
-#include "clang/Basic/FileManager.h"
-#include "clang/Basic/FileSystemStatCache.h"
-#include "clang/Basic/IdentifierTable.h"
-#include "clang/Basic/SourceManager.h"
-#include "clang/Basic/TokenKinds.h"
-#include "clang/Lex/LexDiagnostic.h"
-#include "clang/Lex/PTHManager.h"
-#include "clang/Lex/Preprocessor.h"
-#include "clang/Lex/Token.h"
-#include "llvm/ADT/STLExtras.h"
-#include "llvm/ADT/StringRef.h"
-#include "llvm/Support/DJB.h"
-#include "llvm/Support/Endian.h"
-#include "llvm/Support/ErrorOr.h"
-#include "llvm/Support/FileSystem.h"
-#include "llvm/Support/MemoryBuffer.h"
-#include "llvm/Support/OnDiskHashTable.h"
-#include <cassert>
-#include <cstdint>
-#include <cstdlib>
-#include <cstring>
-#include <ctime>
-#include <memory>
-#include <utility>
-
-using namespace clang;
-
-static const unsigned StoredTokenSize = 1 + 1 + 2 + 4 + 4;
-
-//===----------------------------------------------------------------------===//
-// PTHLexer methods.
-//===----------------------------------------------------------------------===//
-
-PTHLexer::PTHLexer(Preprocessor &PP, FileID FID, const unsigned char *D,
-                   const unsigned char *ppcond, PTHManager &PM)
-    : PreprocessorLexer(&PP, FID), TokBuf(D), CurPtr(D), PPCond(ppcond),
-      CurPPCondPtr(ppcond), PTHMgr(PM) {
-  FileStartLoc = PP.getSourceManager().getLocForStartOfFile(FID);
-}
-
-bool PTHLexer::Lex(Token& Tok) {
-  //===--------------------------------------==//
-  // Read the raw token data.
-  //===--------------------------------------==//
-  using namespace llvm::support;
-
-  // Shadow CurPtr into an automatic variable.
-  const unsigned char *CurPtrShadow = CurPtr;
-
-  // Read in the data for the token.
-  unsigned Word0 = endian::readNext<uint32_t, little, aligned>(CurPtrShadow);
-  uint32_t IdentifierID =
-      endian::readNext<uint32_t, little, aligned>(CurPtrShadow);
-  uint32_t FileOffset =
-      endian::readNext<uint32_t, little, aligned>(CurPtrShadow);
-
-  tok::TokenKind TKind = (tok::TokenKind) (Word0 & 0xFF);
-  Token::TokenFlags TFlags = (Token::TokenFlags) ((Word0 >> 8) & 0xFF);
-  uint32_t Len = Word0 >> 16;
-
-  CurPtr = CurPtrShadow;
-
-  //===--------------------------------------==//
-  // Construct the token itself.
-  //===--------------------------------------==//
-
-  Tok.startToken();
-  Tok.setKind(TKind);
-  Tok.setFlag(TFlags);
-  assert(!LexingRawMode);
-  Tok.setLocation(FileStartLoc.getLocWithOffset(FileOffset));
-  Tok.setLength(Len);
-
-  // Handle identifiers.
-  if (Tok.isLiteral()) {
-    Tok.setLiteralData((const char*) (PTHMgr.SpellingBase + IdentifierID));
-  }
-  else if (IdentifierID) {
-    MIOpt.ReadToken();
-    IdentifierInfo *II = PTHMgr.GetIdentifierInfo(IdentifierID-1);
-
-    Tok.setIdentifierInfo(II);
-
-    // Change the kind of this identifier to the appropriate token kind, e.g.
-    // turning "for" into a keyword.
-    Tok.setKind(II->getTokenID());
-
-    if (II->isHandleIdentifierCase())
-      return PP->HandleIdentifier(Tok);
-
-    return true;
-  }
-
-  //===--------------------------------------==//
-  // Process the token.
-  //===--------------------------------------==//
-  if (TKind == tok::eof) {
-    // Save the end-of-file token.
-    EofToken = Tok;
-
-    assert(!ParsingPreprocessorDirective);
-    assert(!LexingRawMode);
-
-    return LexEndOfFile(Tok);
-  }
-
-  if (TKind == tok::hash && Tok.isAtStartOfLine()) {
-    LastHashTokPtr = CurPtr - StoredTokenSize;
-    assert(!LexingRawMode);
-    PP->HandleDirective(Tok);
-
-    return false;
-  }
-
-  if (TKind == tok::eod) {
-    assert(ParsingPreprocessorDirective);
-    ParsingPreprocessorDirective = false;
-    return true;
-  }
-
-  MIOpt.ReadToken();
-  return true;
-}
-
-bool PTHLexer::LexEndOfFile(Token &Result) {
-  // If we hit the end of the file while parsing a preprocessor directive,
-  // end the preprocessor directive first.  The next token returned will
-  // then be the end of file.
-  if (ParsingPreprocessorDirective) {
-    ParsingPreprocessorDirective = false; // Done parsing the "line".
-    return true;  // Have a token.
-  }
-
-  assert(!LexingRawMode);
-
-  // If we are in a #if directive, emit an error.
-  while (!ConditionalStack.empty()) {
-    if (PP->getCodeCompletionFileLoc() != FileStartLoc)
-      PP->Diag(ConditionalStack.back().IfLoc,
-               diag::err_pp_unterminated_conditional);
-    ConditionalStack.pop_back();
-  }
-
-  // Finally, let the preprocessor handle this.
-  return PP->HandleEndOfFile(Result);
-}
-
-// FIXME: We can just grab the last token instead of storing a copy
-// into EofToken.
-void PTHLexer::getEOF(Token& Tok) {
-  assert(EofToken.is(tok::eof));
-  Tok = EofToken;
-}
-
-void PTHLexer::DiscardToEndOfLine() {
-  assert(ParsingPreprocessorDirective && ParsingFilename == false &&
-         "Must be in a preprocessing directive!");
-
-  // We assume that if the preprocessor wishes to discard to the end of
-  // the line that it also means to end the current preprocessor directive.
-  ParsingPreprocessorDirective = false;
-
-  // Skip tokens by only peeking at their token kind and the flags.
-  // We don't need to actually reconstruct full tokens from the token buffer.
-  // This saves some copies and it also reduces IdentifierInfo* lookup.
-  const unsigned char* p = CurPtr;
-  while (true) {
-    // Read the token kind.  Are we at the end of the file?
-    tok::TokenKind x = (tok::TokenKind) (uint8_t) *p;
-    if (x == tok::eof) break;
-
-    // Read the token flags.  Are we at the start of the next line?
-    Token::TokenFlags y = (Token::TokenFlags) (uint8_t) p[1];
-    if (y & Token::StartOfLine) break;
-
-    // Skip to the next token.
-    p += StoredTokenSize;
-  }
-
-  CurPtr = p;
-}
-
-/// SkipBlock - Used by Preprocessor to skip the current conditional block.
-bool PTHLexer::SkipBlock() {
-  using namespace llvm::support;
-
-  assert(CurPPCondPtr && "No cached PP conditional information.");
-  assert(LastHashTokPtr && "No known '#' token.");
-
-  const unsigned char *HashEntryI = nullptr;
-  uint32_t TableIdx;
-
-  do {
-    // Read the token offset from the side-table.
-    uint32_t Offset = endian::readNext<uint32_t, little, aligned>(CurPPCondPtr);
-
-    // Read the target table index from the side-table.
-    TableIdx = endian::readNext<uint32_t, little, aligned>(CurPPCondPtr);
-
-    // Compute the actual memory address of the '#' token data for this entry.
-    HashEntryI = TokBuf + Offset;
-
-    // Optimization: "Sibling jumping".  #if...#else...#endif blocks can
-    //  contain nested blocks.  In the side-table we can jump over these
-    //  nested blocks instead of doing a linear search if the next "sibling"
-    //  entry is not at a location greater than LastHashTokPtr.
-    if (HashEntryI < LastHashTokPtr && TableIdx) {
-      // In the side-table we are still at an entry for a '#' token that
-      // is earlier than the last one we saw.  Check if the location we would
-      // stride gets us closer.
-      const unsigned char* NextPPCondPtr =
-        PPCond + TableIdx*(sizeof(uint32_t)*2);
-      assert(NextPPCondPtr >= CurPPCondPtr);
-      // Read where we should jump to.
-      const unsigned char *HashEntryJ =
-          TokBuf + endian::readNext<uint32_t, little, aligned>(NextPPCondPtr);
-
-      if (HashEntryJ <= LastHashTokPtr) {
-        // Jump directly to the next entry in the side table.
-        HashEntryI = HashEntryJ;
-        TableIdx = endian::readNext<uint32_t, little, aligned>(NextPPCondPtr);
-        CurPPCondPtr = NextPPCondPtr;
-      }
-    }
-  }
-  while (HashEntryI < LastHashTokPtr);
-  assert(HashEntryI == LastHashTokPtr && "No PP-cond entry found for '#'");
-  assert(TableIdx && "No jumping from #endifs.");
-
-  // Update our side-table iterator.
-  const unsigned char* NextPPCondPtr = PPCond + TableIdx*(sizeof(uint32_t)*2);
-  assert(NextPPCondPtr >= CurPPCondPtr);
-  CurPPCondPtr = NextPPCondPtr;
-
-  // Read where we should jump to.
-  HashEntryI =
-      TokBuf + endian::readNext<uint32_t, little, aligned>(NextPPCondPtr);
-  uint32_t NextIdx = endian::readNext<uint32_t, little, aligned>(NextPPCondPtr);
-
-  // By construction NextIdx will be zero if this is a #endif.  This is useful
-  // to know to obviate lexing another token.
-  bool isEndif = NextIdx == 0;
-
-  // This case can occur when we see something like this:
-  //
-  //  #if ...
-  //   /* a comment or nothing */
-  //  #elif
-  //
-  // If we are skipping the first #if block it will be the case that CurPtr
-  // already points 'elif'.  Just return.
-
-  if (CurPtr > HashEntryI) {
-    assert(CurPtr == HashEntryI + StoredTokenSize);
-    // Did we reach a #endif?  If so, go ahead and consume that token as well.
-    if (isEndif)
-      CurPtr += StoredTokenSize * 2;
-    else
-      LastHashTokPtr = HashEntryI;
-
-    return isEndif;
-  }
-
-  // Otherwise, we need to advance.  Update CurPtr to point to the '#' token.
-  CurPtr = HashEntryI;
-
-  // Update the location of the last observed '#'.  This is useful if we
-  // are skipping multiple blocks.
-  LastHashTokPtr = CurPtr;
-
-  // Skip the '#' token.
-  assert(((tok::TokenKind)*CurPtr) == tok::hash);
-  CurPtr += StoredTokenSize;
-
-  // Did we reach a #endif?  If so, go ahead and consume that token as well.
-  if (isEndif) {
-    CurPtr += StoredTokenSize * 2;
-  }
-
-  return isEndif;
-}
-
-SourceLocation PTHLexer::getSourceLocation() {
-  // getSourceLocation is not on the hot path.  It is used to get the location
-  // of the next token when transitioning back to this lexer when done
-  // handling a #included file.  Just read the necessary data from the token
-  // data buffer to construct the SourceLocation object.
-  // NOTE: This is a virtual function; hence it is defined out-of-line.
-  using namespace llvm::support;
-
-  const unsigned char *OffsetPtr = CurPtr + (StoredTokenSize - 4);
-  uint32_t Offset = endian::readNext<uint32_t, little, aligned>(OffsetPtr);
-  return FileStartLoc.getLocWithOffset(Offset);
-}
-
-//===----------------------------------------------------------------------===//
-// PTH file lookup: map from strings to file data.
-//===----------------------------------------------------------------------===//
-
-/// PTHFileLookup - This internal data structure is used by the PTHManager
-///  to map from FileEntry objects managed by FileManager to offsets within
-///  the PTH file.
-namespace {
-
-class PTHFileData {
-  const uint32_t TokenOff;
-  const uint32_t PPCondOff;
-
-public:
-  PTHFileData(uint32_t tokenOff, uint32_t ppCondOff)
-      : TokenOff(tokenOff), PPCondOff(ppCondOff) {}
-
-  uint32_t getTokenOffset() const { return TokenOff; }
-  uint32_t getPPCondOffset() const { return PPCondOff; }
-};
-
-class PTHFileLookupCommonTrait {
-public:
-  using internal_key_type = std::pair<unsigned char, StringRef>;
-  using hash_value_type = unsigned;
-  using offset_type = unsigned;
-
-  static hash_value_type ComputeHash(internal_key_type x) {
-    return llvm::djbHash(x.second);
-  }
-
-  static std::pair<unsigned, unsigned>
-  ReadKeyDataLength(const unsigned char*& d) {
-    using namespace llvm::support;
-
-    unsigned keyLen =
-        (unsigned)endian::readNext<uint16_t, little, unaligned>(d);
-    unsigned dataLen = (unsigned) *(d++);
-    return std::make_pair(keyLen, dataLen);
-  }
-
-  static internal_key_type ReadKey(const unsigned char* d, unsigned) {
-    unsigned char k = *(d++); // Read the entry kind.
-    return std::make_pair(k, (const char*) d);
-  }
-};
-
-} // namespace
-
-class PTHManager::PTHFileLookupTrait : public PTHFileLookupCommonTrait {
-public:
-  using external_key_type = const FileEntry *;
-  using data_type = PTHFileData;
-
-  static internal_key_type GetInternalKey(const FileEntry* FE) {
-    return std::make_pair((unsigned char) 0x1, FE->getName());
-  }
-
-  static bool EqualKey(internal_key_type a, internal_key_type b) {
-    return a.first == b.first && a.second == b.second;
-  }
-
-  static PTHFileData ReadData(const internal_key_type& k,
-                              const unsigned char* d, unsigned) {
-    using namespace llvm::support;
-
-    assert(k.first == 0x1 && "Only file lookups can match!");
-    uint32_t x = endian::readNext<uint32_t, little, unaligned>(d);
-    uint32_t y = endian::readNext<uint32_t, little, unaligned>(d);
-    return PTHFileData(x, y);
-  }
-};
-
-class PTHManager::PTHStringLookupTrait {
-public:
-  using data_type = uint32_t;
-  using external_key_type = const std::pair<const char *, unsigned>;
-  using internal_key_type = external_key_type;
-  using hash_value_type = uint32_t;
-  using offset_type = unsigned;
-
-  static bool EqualKey(const internal_key_type& a,
-                       const internal_key_type& b) {
-    return (a.second == b.second) ? memcmp(a.first, b.first, a.second) == 0
-                                  : false;
-  }
-
-  static hash_value_type ComputeHash(const internal_key_type& a) {
-    return llvm::djbHash(StringRef(a.first, a.second));
-  }
-
-  // This hopefully will just get inlined and removed by the optimizer.
-  static const internal_key_type&
-  GetInternalKey(const external_key_type& x) { return x; }
-
-  static std::pair<unsigned, unsigned>
-  ReadKeyDataLength(const unsigned char*& d) {
-    using namespace llvm::support;
-
-    return std::make_pair(
-        (unsigned)endian::readNext<uint16_t, little, unaligned>(d),
-        sizeof(uint32_t));
-  }
-
-  static std::pair<const char*, unsigned>
-  ReadKey(const unsigned char* d, unsigned n) {
-      assert(n >= 2 && d[n-1] == '\0');
-      return std::make_pair((const char*) d, n-1);
-    }
-
-  static uint32_t ReadData(const internal_key_type& k, const unsigned char* d,
-                           unsigned) {
-    using namespace llvm::support;
-
-    return endian::readNext<uint32_t, little, unaligned>(d);
-  }
-};
-
-//===----------------------------------------------------------------------===//
-// PTHManager methods.
-//===----------------------------------------------------------------------===//
-
-PTHManager::PTHManager(
-    std::unique_ptr<const llvm::MemoryBuffer> buf,
-    std::unique_ptr<PTHFileLookup> fileLookup, const unsigned char *idDataTable,
-    std::unique_ptr<IdentifierInfo *[], llvm::FreeDeleter> perIDCache,
-    std::unique_ptr<PTHStringIdLookup> stringIdLookup, unsigned numIds,
-    const unsigned char *spellingBase, const char *originalSourceFile)
-    : Buf(std::move(buf)), PerIDCache(std::move(perIDCache)),
-      FileLookup(std::move(fileLookup)), IdDataTable(idDataTable),
-      StringIdLookup(std::move(stringIdLookup)), NumIds(numIds),
-      SpellingBase(spellingBase), OriginalSourceFile(originalSourceFile) {}
-
-PTHManager::~PTHManager() = default;
-
-static void InvalidPTH(DiagnosticsEngine &Diags, const char *Msg) {
-  Diags.Report(Diags.getCustomDiagID(DiagnosticsEngine::Error, "%0")) << Msg;
-}
-
-PTHManager *PTHManager::Create(StringRef file, DiagnosticsEngine &Diags) {
-  // Memory map the PTH file.
-  llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> FileOrErr =
-      llvm::MemoryBuffer::getFile(file);
-
-  if (!FileOrErr) {
-    // FIXME: Add ec.message() to this diag.
-    Diags.Report(diag::err_invalid_pth_file) << file;
-    return nullptr;
-  }
-  std::unique_ptr<llvm::MemoryBuffer> File = std::move(FileOrErr.get());
-
-  using namespace llvm::support;
-
-  // Get the buffer ranges and check if there are at least three 32-bit
-  // words at the end of the file.
-  const unsigned char *BufBeg = (const unsigned char*)File->getBufferStart();
-  const unsigned char *BufEnd = (const unsigned char*)File->getBufferEnd();
-
-  // Check the prologue of the file.
-  if ((BufEnd - BufBeg) < (signed)(sizeof("cfe-pth") + 4 + 4) ||
-      memcmp(BufBeg, "cfe-pth", sizeof("cfe-pth")) != 0) {
-    Diags.Report(diag::err_invalid_pth_file) << file;
-    return nullptr;
-  }
-
-  // Read the PTH version.
-  const unsigned char *p = BufBeg + (sizeof("cfe-pth"));
-  unsigned Version = endian::readNext<uint32_t, little, aligned>(p);
-
-  if (Version < PTHManager::Version) {
-    InvalidPTH(Diags,
-        Version < PTHManager::Version
-        ? "PTH file uses an older PTH format that is no longer supported"
-        : "PTH file uses a newer PTH format that cannot be read");
-    return nullptr;
-  }
-
-  // Compute the address of the index table at the end of the PTH file.
-  const unsigned char *PrologueOffset = p;
-
-  if (PrologueOffset >= BufEnd) {
-    Diags.Report(diag::err_invalid_pth_file) << file;
-    return nullptr;
-  }
-
-  // Construct the file lookup table.  This will be used for mapping from
-  // FileEntry*'s to cached tokens.
-  const unsigned char* FileTableOffset = PrologueOffset + sizeof(uint32_t)*2;
-  const unsigned char *FileTable =
-      BufBeg + endian::readNext<uint32_t, little, aligned>(FileTableOffset);
-
-  if (!(FileTable > BufBeg && FileTable < BufEnd)) {
-    Diags.Report(diag::err_invalid_pth_file) << file;
-    return nullptr; // FIXME: Proper error diagnostic?
-  }
-
-  std::unique_ptr<PTHFileLookup> FL(PTHFileLookup::Create(FileTable, BufBeg));
-
-  // Warn if the PTH file is empty.  We still want to create a PTHManager
-  // as the PTH could be used with -include-pth.
-  if (FL->isEmpty())
-    InvalidPTH(Diags, "PTH file contains no cached source data");
-
-  // Get the location of the table mapping from persistent ids to the
-  // data needed to reconstruct identifiers.
-  const unsigned char* IDTableOffset = PrologueOffset + sizeof(uint32_t)*0;
-  const unsigned char *IData =
-      BufBeg + endian::readNext<uint32_t, little, aligned>(IDTableOffset);
-
-  if (!(IData >= BufBeg && IData < BufEnd)) {
-    Diags.Report(diag::err_invalid_pth_file) << file;
-    return nullptr;
-  }
-
-  // Get the location of the hashtable mapping between strings and
-  // persistent IDs.
-  const unsigned char* StringIdTableOffset = PrologueOffset + sizeof(uint32_t)*1;
-  const unsigned char *StringIdTable =
-      BufBeg + endian::readNext<uint32_t, little, aligned>(StringIdTableOffset);
-  if (!(StringIdTable >= BufBeg && StringIdTable < BufEnd)) {
-    Diags.Report(diag::err_invalid_pth_file) << file;
-    return nullptr;
-  }
-
-  std::unique_ptr<PTHStringIdLookup> SL(
-      PTHStringIdLookup::Create(StringIdTable, BufBeg));
-
-  // Get the location of the spelling cache.
-  const unsigned char* spellingBaseOffset = PrologueOffset + sizeof(uint32_t)*3;
-  const unsigned char *spellingBase =
-      BufBeg + endian::readNext<uint32_t, little, aligned>(spellingBaseOffset);
-  if (!(spellingBase >= BufBeg && spellingBase < BufEnd)) {
-    Diags.Report(diag::err_invalid_pth_file) << file;
-    return nullptr;
-  }
-
-  // Get the number of IdentifierInfos and pre-allocate the identifier cache.
-  uint32_t NumIds = endian::readNext<uint32_t, little, aligned>(IData);
-
-  // Pre-allocate the persistent ID -> IdentifierInfo* cache.  We use calloc()
-  // so that we in the best case only zero out memory once when the OS returns
-  // us new pages.
-  std::unique_ptr<IdentifierInfo *[], llvm::FreeDeleter> PerIDCache;
-
-  if (NumIds) {
-    PerIDCache.reset((IdentifierInfo **)calloc(NumIds, sizeof(PerIDCache[0])));
-    if (!PerIDCache) {
-      InvalidPTH(Diags, "Could not allocate memory for processing PTH file");
-      return nullptr;
-    }
-  }
-
-  // Compute the address of the original source file.
-  const unsigned char* originalSourceBase = PrologueOffset + sizeof(uint32_t)*4;
-  unsigned len =
-      endian::readNext<uint16_t, little, unaligned>(originalSourceBase);
-  if (!len) originalSourceBase = nullptr;
-
-  // Create the new PTHManager.
-  return new PTHManager(std::move(File), std::move(FL), IData,
-                        std::move(PerIDCache), std::move(SL), NumIds,
-                        spellingBase, (const char *)originalSourceBase);
-}
-
-IdentifierInfo* PTHManager::LazilyCreateIdentifierInfo(unsigned PersistentID) {
-  using namespace llvm::support;
-
-  // Look in the PTH file for the string data for the IdentifierInfo object.
-  const unsigned char* TableEntry = IdDataTable + sizeof(uint32_t)*PersistentID;
-  const unsigned char *IDData =
-      (const unsigned char *)Buf->getBufferStart() +
-      endian::readNext<uint32_t, little, aligned>(TableEntry);
-  assert(IDData < (const unsigned char*)Buf->getBufferEnd());
-
-  // Allocate the object.
-  std::pair<IdentifierInfo,const unsigned char*> *Mem =
-      Alloc.Allocate<std::pair<IdentifierInfo, const unsigned char *>>();
-
-  Mem->second = IDData;
-  assert(IDData[0] != '\0');
-  IdentifierInfo *II = new ((void*) Mem) IdentifierInfo();
-
-  // Store the new IdentifierInfo in the cache.
-  PerIDCache[PersistentID] = II;
-  assert(II->getNameStart() && II->getNameStart()[0] != '\0');
-  return II;
-}
-
-IdentifierInfo* PTHManager::get(StringRef Name) {
-  // Double check our assumption that the last character isn't '\0'.
-  assert(Name.empty() || Name.back() != '\0');
-  PTHStringIdLookup::iterator I =
-      StringIdLookup->find(std::make_pair(Name.data(), Name.size()));
-  if (I == StringIdLookup->end()) // No identifier found?
-    return nullptr;
-
-  // Match found.  Return the identifier!
-  assert(*I > 0);
-  return GetIdentifierInfo(*I-1);
-}
-
-PTHLexer *PTHManager::CreateLexer(FileID FID) {
-  const FileEntry *FE = PP->getSourceManager().getFileEntryForID(FID);
-  if (!FE)
-    return nullptr;
-
-  using namespace llvm::support;
-
-  // Lookup the FileEntry object in our file lookup data structure.  It will
-  // return a variant that indicates whether or not there is an offset within
-  // the PTH file that contains cached tokens.
-  PTHFileLookup::iterator I = FileLookup->find(FE);
-
-  if (I == FileLookup->end()) // No tokens available?
-    return nullptr;
-
-  const PTHFileData& FileData = *I;
-
-  const unsigned char *BufStart = (const unsigned char *)Buf->getBufferStart();
-  // Compute the offset of the token data within the buffer.
-  const unsigned char* data = BufStart + FileData.getTokenOffset();
-
-  // Get the location of pp-conditional table.
-  const unsigned char* ppcond = BufStart + FileData.getPPCondOffset();
-  uint32_t Len = endian::readNext<uint32_t, little, aligned>(ppcond);
-  if (Len == 0) ppcond = nullptr;
-
-  assert(PP && "No preprocessor set yet!");
-  return new PTHLexer(*PP, FID, data, ppcond, *this);
-}
-
-//===----------------------------------------------------------------------===//
-// 'stat' caching.
-//===----------------------------------------------------------------------===//
-
-namespace {
-
-class PTHStatData {
-public:
-  uint64_t Size;
-  time_t ModTime;
-  llvm::sys::fs::UniqueID UniqueID;
-  const bool HasData = false;
-  bool IsDirectory;
-
-  PTHStatData() = default;
-  PTHStatData(uint64_t Size, time_t ModTime, llvm::sys::fs::UniqueID UniqueID,
-              bool IsDirectory)
-      : Size(Size), ModTime(ModTime), UniqueID(UniqueID), HasData(true),
-        IsDirectory(IsDirectory) {}
-};
-
-class PTHStatLookupTrait : public PTHFileLookupCommonTrait {
-public:
-  using external_key_type = StringRef; // const char*
-  using data_type = PTHStatData;
-
-  static internal_key_type GetInternalKey(StringRef path) {
-    // The key 'kind' doesn't matter here because it is ignored in EqualKey.
-    return std::make_pair((unsigned char) 0x0, path);
-  }
-
-  static bool EqualKey(internal_key_type a, internal_key_type b) {
-    // When doing 'stat' lookups we don't care about the kind of 'a' and 'b',
-    // just the paths.
-    return a.second == b.second;
-  }
-
-  static data_type ReadData(const internal_key_type& k, const unsigned char* d,
-                            unsigned) {
-    if (k.first /* File or Directory */) {
-      bool IsDirectory = true;
-      if (k.first == 0x1 /* File */) {
-        IsDirectory = false;
-        d += 4 * 2; // Skip the first 2 words.
-      }
-
-      using namespace llvm::support;
-
-      uint64_t File = endian::readNext<uint64_t, little, unaligned>(d);
-      uint64_t Device = endian::readNext<uint64_t, little, unaligned>(d);
-      llvm::sys::fs::UniqueID UniqueID(Device, File);
-      time_t ModTime = endian::readNext<uint64_t, little, unaligned>(d);
-      uint64_t Size = endian::readNext<uint64_t, little, unaligned>(d);
-      return data_type(Size, ModTime, UniqueID, IsDirectory);
-    }
-
-    // Negative stat.  Don't read anything.
-    return data_type();
-  }
-};
-
-} // namespace
-
-namespace clang {
-
-class PTHStatCache : public FileSystemStatCache {
-  using CacheTy = llvm::OnDiskChainedHashTable<PTHStatLookupTrait>;
-
-  CacheTy Cache;
-
-public:
-  PTHStatCache(PTHManager::PTHFileLookup &FL)
-      : Cache(FL.getNumBuckets(), FL.getNumEntries(), FL.getBuckets(),
-              FL.getBase()) {}
-
-  LookupResult getStat(StringRef Path, FileData &Data, bool isFile,
-                       std::unique_ptr<llvm::vfs::File> *F,
-                       llvm::vfs::FileSystem &FS) override {
-    // Do the lookup for the file's data in the PTH file.
-    CacheTy::iterator I = Cache.find(Path);
-
-    // If we don't get a hit in the PTH file just forward to 'stat'.
-    if (I == Cache.end())
-      return statChained(Path, Data, isFile, F, FS);
-
-    const PTHStatData &D = *I;
-
-    if (!D.HasData)
-      return CacheMissing;
-
-    Data.Name = Path;
-    Data.Size = D.Size;
-    Data.ModTime = D.ModTime;
-    Data.UniqueID = D.UniqueID;
-    Data.IsDirectory = D.IsDirectory;
-    Data.IsNamedPipe = false;
-    Data.InPCH = true;
-
-    return CacheExists;
-  }
-};
-
-} // namespace clang
-
-std::unique_ptr<FileSystemStatCache> PTHManager::createStatCache() {
-  return llvm::make_unique<PTHStatCache>(*FileLookup);
-}

lib/Lex/Pragma.cpp

 #include "clang/Lex/LexDiagnostic.h"
 #include "clang/Lex/Lexer.h"
 #include "clang/Lex/LiteralSupport.h"
 #include "clang/Lex/MacroInfo.h"
 #include "clang/Lex/ModuleLoader.h"
 #include "clang/Lex/PPCallbacks.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorLexer.h"
-#include "clang/Lex/PTHLexer.h"
 #include "clang/Lex/Token.h"
 #include "clang/Lex/TokenLexer.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringSwitch.h"
void Preprocessor::HandlePragmaOnce(Token &OnceTok) {
 
   // Get the current file lexer we're looking at.  Ignore _Pragma 'files' etc.
   // Mark the file as a once-only file now.
   HeaderInfo.MarkFileIncludeOnce(getCurrentFileLexer()->getFileEntry());
 }
 
 void Preprocessor::HandlePragmaMark() {
   assert(CurPPLexer && "No current lexer?");
-  if (CurLexer)
-    CurLexer->ReadToEndOfLine();
-  else
-    CurPTHLexer->DiscardToEndOfLine();
 }
 
 /// HandlePragmaPoison - Handle \#pragma GCC poison.  PoisonTok is the 'poison'.
 void Preprocessor::HandlePragmaPoison() {
   Token Tok;
 
   while (true) {
     // Read the next token to poison.  While doing this, pretend that we are
void Preprocessor::HandlePragmaModuleBuild(Token &Tok) {
   IdentifierInfo *ModuleName = ModuleNameLoc.first;
 
   LexUnexpandedToken(Tok);
   if (Tok.isNot(tok::eod)) {
     Diag(Tok, diag::ext_pp_extra_tokens_at_eol) << "pragma";
     DiscardUntilEndOfDirective();
   }
 
-  if (CurPTHLexer) {
-    // FIXME: Support this somehow?
-    Diag(Loc, diag::err_pp_module_build_pth);
-    return;
-  }
-
   CurLexer->LexingRawMode = true;
 
   auto TryConsumeIdentifier = [&](StringRef Ident) -> bool {
     if (Tok.getKind() != tok::raw_identifier ||
         Tok.getRawIdentifier() != Ident)
       return false;
     CurLexer->Lex(Tok);
     return true;

lib/Lex/Preprocessor.cpp

 #include "clang/Lex/ExternalPreprocessorSource.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/LexDiagnostic.h"
 #include "clang/Lex/Lexer.h"
 #include "clang/Lex/LiteralSupport.h"
 #include "clang/Lex/MacroArgs.h"
 #include "clang/Lex/MacroInfo.h"
 #include "clang/Lex/ModuleLoader.h"
-#include "clang/Lex/PTHLexer.h"
-#include "clang/Lex/PTHManager.h"
 #include "clang/Lex/Pragma.h"
 #include "clang/Lex/PreprocessingRecord.h"
 #include "clang/Lex/PreprocessorLexer.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Lex/ScratchBuffer.h"
 #include "clang/Lex/Token.h"
 #include "clang/Lex/TokenLexer.h"
 #include "llvm/ADT/APInt.h"

 }
 
 void Preprocessor::FinalizeForModelFile() {
   NumEnteredSourceFiles = 1;
 
   PragmaHandlers = std::move(PragmaHandlersBackup);
 }
 
-void Preprocessor::setPTHManager(PTHManager* pm) {
-  PTH.reset(pm);
-  FileMgr.addStatCache(PTH->createStatCache());
-}
-
 void Preprocessor::DumpToken(const Token &Tok, bool DumpFlags) const {
   llvm::errs() << tok::getTokenName(Tok.getKind()) << " '"
                << getSpelling(Tok) << "'";
 
   if (!DumpFlags) return;
 
   llvm::errs() << "\t";
   if (Tok.isAtStartOfLine())
for (Preprocessor::macro_iterator I = macro_begin(), E = macro_end();
     }
   }
   return BestSpelling;
 }
 
 void Preprocessor::recomputeCurLexerKind() {
   if (CurLexer)
     CurLexerKind = CLK_Lexer;
-  else if (CurPTHLexer)
-    CurLexerKind = CLK_PTHLexer;
   else if (CurTokenLexer)
     CurLexerKind = CLK_TokenLexer;
   else
     CurLexerKind = CLK_CachingLexer;
 }
 
 bool Preprocessor::SetCodeCompletionPoint(const FileEntry *File,
                                           unsigned CompleteLine,

 void Preprocessor::Lex(Token &Result) {
   // We loop here until a lex function returns a token; this avoids recursion.
   bool ReturnedToken;
   do {
     switch (CurLexerKind) {
     case CLK_Lexer:
       ReturnedToken = CurLexer->Lex(Result);
       break;
-    case CLK_PTHLexer:
-      ReturnedToken = CurPTHLexer->Lex(Result);
-      break;
     case CLK_TokenLexer:
       ReturnedToken = CurTokenLexer->Lex(Result);
       break;
     case CLK_CachingLexer:
       CachingLex(Result);
       ReturnedToken = true;
       break;
     case CLK_LexAfterModuleImport:

lib/Serialization/ASTReader.cpp

bool ASTReader::ParsePreprocessorOptions(const RecordData &Record,
   // Macro Includes
   for (unsigned N = Record[Idx++]; N; --N) {
     PPOpts.MacroIncludes.push_back(ReadString(Record, Idx));
   }
 
   PPOpts.UsePredefines = Record[Idx++];
   PPOpts.DetailedRecord = Record[Idx++];
   PPOpts.ImplicitPCHInclude = ReadString(Record, Idx);
-  PPOpts.ImplicitPTHInclude = ReadString(Record, Idx);
   PPOpts.ObjCXXARCStandardLibrary =
     static_cast<ObjCXXARCStandardLibraryKind>(Record[Idx++]);
   SuggestedPredefines.clear();
   return Listener.ReadPreprocessorOptions(PPOpts, Complain,
                                           SuggestedPredefines);
 }
 
 std::pair<ModuleFile *, unsigned>

lib/Serialization/ASTWriter.cpp

#include "clang/Basic/Sanitizers.def"
   Record.push_back(PPOpts.MacroIncludes.size());
   for (unsigned I = 0, N = PPOpts.MacroIncludes.size(); I != N; ++I)
     AddString(PPOpts.MacroIncludes[I], Record);
 
   Record.push_back(PPOpts.UsePredefines);
   // Detailed record is important since it is used for the module cache hash.
   Record.push_back(PPOpts.DetailedRecord);
   AddString(PPOpts.ImplicitPCHInclude, Record);
-  AddString(PPOpts.ImplicitPTHInclude, Record);
   Record.push_back(static_cast<unsigned>(PPOpts.ObjCXXARCStandardLibrary));
   Stream.EmitRecord(PREPROCESSOR_OPTIONS, Record);
 
   // Leave the options block.
   Stream.ExitBlock();
 
   // Original file name and file ID
   SourceManager &SM = Context.getSourceManager();

test/Driver/pth.c

-// Test transparent PTH support.
-
-// RUN: %clang -no-canonical-prefixes -ccc-pch-is-pth -x c-header %s -o %t.h.pth -### 2> %t.log
-// RUN: FileCheck -check-prefix CHECK1 -input-file %t.log %s
-
-// CHECK1: "{{.*[/\\]}}clang{{.*}}" "-cc1" {{.*}} "-o" "{{.*}}.h.pth" "-x" "c-header" "{{.*}}pth.c"
-
-// RUN: touch %t.h.pth
-// RUN: %clang -no-canonical-prefixes -ccc-pch-is-pth -E -include %t.h %s -### 2> %t.log
-// RUN: FileCheck -check-prefix CHECK2 -input-file %t.log %s
-
-// CHECK2: "{{.*[/\\]}}clang{{.*}}" "-cc1" {{.*}}"-include-pth" "{{.*}}.h.pth" {{.*}}"-x" "c" "{{.*}}pth.c"

test/PCH/emit-pth.c

-// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-pth -o %t1 %s
-// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-pth -o - %s > %t2
-// RUN: cmp %t1 %t2
-// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-pth -o - %s | \
-// RUN: FileCheck %s
-
-// CHECK: cfe-pth

test/PCH/pth.c

-// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-pth -o %t %S/pth.h
-// RUN: not %clang_cc1 -triple i386-unknown-unknown -include-pth %t -fsyntax-only %s 2>&1 | FileCheck %s
-
-#error This is the only diagnostic
-
-// CHECK: This is the only diagnostic
-// CHECK: 1 error generated.

test/Preprocessor/include-pth.c

-// RUN: %clang_cc1 -emit-pth %s -o %t
-// RUN: %clang_cc1 -include-pth %t %s -E | grep 'file_to_include' | count 2
-#include "file_to_include.h"