Index: NOTES.txt =================================================================== --- NOTES.txt +++ NOTES.txt @@ -10,13 +10,6 @@ //===---------------------------------------------------------------------===// -Creating and using a PTH file for performance measurement (use a release build). - -$ clang -ccc-pch-is-pth -x objective-c-header INPUTS/Cocoa_h.m -o /tmp/tokencache -$ clang -cc1 -token-cache /tmp/tokencache INPUTS/Cocoa_h.m - -//===---------------------------------------------------------------------===// - C++ Template Instantiation benchmark: http://users.rcn.com/abrahams/instantiation_speed/index.html Index: docs/InternalsManual.rst =================================================================== --- docs/InternalsManual.rst +++ docs/InternalsManual.rst @@ -559,13 +559,9 @@ Precompiled Headers =================== -Clang supports two implementations of precompiled headers. The default -implementation, precompiled headers (:doc:`PCH `) uses a +Clang supports precompiled headers (:doc:`PCH `), which uses a serialized representation of Clang's internal data structures, encoded with the `LLVM bitstream format `_. -Pretokenized headers (:doc:`PTH `), on the other hand, contain a -serialized representation of the tokens encountered when preprocessing a header -(and anything that header includes). The Frontend Library ==================== Index: docs/PTHInternals.rst =================================================================== --- docs/PTHInternals.rst +++ docs/PTHInternals.rst @@ -1,163 +0,0 @@ -========================== -Pretokenized Headers (PTH) -========================== - -This document first describes the low-level interface for using PTH and -then briefly elaborates on its design and implementation. If you are -interested in the end-user view, please see the :ref:`User's Manual -`. - -Using Pretokenized Headers with ``clang`` (Low-level Interface) -=============================================================== - -The Clang compiler frontend, ``clang -cc1``, supports three command line -options for generating and using PTH files. - -To generate PTH files using ``clang -cc1``, use the option ``-emit-pth``: - -.. code-block:: console - - $ clang -cc1 test.h -emit-pth -o test.h.pth - -This option is transparently used by ``clang`` when generating PTH -files. Similarly, PTH files can be used as prefix headers using the -``-include-pth`` option: - -.. code-block:: console - - $ clang -cc1 -include-pth test.h.pth test.c -o test.s - -Alternatively, Clang's PTH files can be used as a raw "token-cache" (or -"content" cache) of the source included by the original header file. -This means that the contents of the PTH file are searched as substitutes -for *any* source files that are used by ``clang -cc1`` to process a -source file. This is done by specifying the ``-token-cache`` option: - -.. code-block:: console - - $ cat test.h - #include - $ clang -cc1 -emit-pth test.h -o test.h.pth - $ cat test.c - #include "test.h" - $ clang -cc1 test.c -o test -token-cache test.h.pth - -In this example the contents of ``stdio.h`` (and the files it includes) -will be retrieved from ``test.h.pth``, as the PTH file is being used in -this case as a raw cache of the contents of ``test.h``. This is a -low-level interface used to both implement the high-level PTH interface -as well as to provide alternative means to use PTH-style caching. - -PTH Design and Implementation -============================= - -Unlike GCC's precompiled headers, which cache the full ASTs and -preprocessor state of a header file, Clang's pretokenized header files -mainly cache the raw lexer *tokens* that are needed to segment the -stream of characters in a source file into keywords, identifiers, and -operators. Consequently, PTH serves to mainly directly speed up the -lexing and preprocessing of a source file, while parsing and -type-checking must be completely redone every time a PTH file is used. - -Basic Design Tradeoffs ----------------------- - -In the long term there are plans to provide an alternate PCH -implementation for Clang that also caches the work for parsing and type -checking the contents of header files. The current implementation of PCH -in Clang as pretokenized header files was motivated by the following -factors: - -**Language independence** - PTH files work with any language that - Clang's lexer can handle, including C, Objective-C, and (in the early - stages) C++. This means development on language features at the - parsing level or above (which is basically almost all interesting - pieces) does not require PTH to be modified. - -**Simple design** - Relatively speaking, PTH has a simple design and - implementation, making it easy to test. Further, because the - machinery for PTH resides at the lower-levels of the Clang library - stack it is fairly straightforward to profile and optimize. - -Further, compared to GCC's PCH implementation (which is the dominate -precompiled header file implementation that Clang can be directly -compared against) the PTH design in Clang yields several attractive -features: - -**Architecture independence** - In contrast to GCC's PCH files (and - those of several other compilers), Clang's PTH files are architecture - independent, requiring only a single PTH file when building a - program for multiple architectures. - - For example, on Mac OS X one may wish to compile a "universal binary" - that runs on PowerPC, 32-bit Intel (i386), and 64-bit Intel - architectures. In contrast, GCC requires a PCH file for each - architecture, as the definitions of types in the AST are - architecture-specific. Since a Clang PTH file essentially represents - a lexical cache of header files, a single PTH file can be safely used - when compiling for multiple architectures. This can also reduce - compile times because only a single PTH file needs to be generated - during a build instead of several. - -**Reduced memory pressure** - Similar to GCC, Clang reads PTH files - via the use of memory mapping (i.e., ``mmap``). Clang, however, - memory maps PTH files as read-only, meaning that multiple invocations - of ``clang -cc1`` can share the same pages in memory from a - memory-mapped PTH file. In comparison, GCC also memory maps its PCH - files but also modifies those pages in memory, incurring the - copy-on-write costs. The read-only nature of PTH can greatly reduce - memory pressure for builds involving multiple cores, thus improving - overall scalability. - -**Fast generation** - PTH files can be generated in a small fraction - of the time needed to generate GCC's PCH files. Since PTH/PCH - generation is a serial operation that typically blocks progress - during a build, faster generation time leads to improved processor - utilization with parallel builds on multicore machines. - -Despite these strengths, PTH's simple design suffers some algorithmic -handicaps compared to other PCH strategies such as those used by GCC. -While PTH can greatly speed up the processing time of a header file, the -amount of work required to process a header file is still roughly linear -in the size of the header file. In contrast, the amount of work done by -GCC to process a precompiled header is (theoretically) constant (the -ASTs for the header are literally memory mapped into the compiler). This -means that only the pieces of the header file that are referenced by the -source file including the header are the only ones the compiler needs to -process during actual compilation. While GCC's particular implementation -of PCH mitigates some of these algorithmic strengths via the use of -copy-on-write pages, the approach itself can fundamentally dominate at -an algorithmic level, especially when one considers header files of -arbitrary size. - -There is also a PCH implementation for Clang based on the lazy -deserialization of ASTs. This approach theoretically has the same -constant-time algorithmic advantages just mentioned but also retains some -of the strengths of PTH such as reduced memory pressure (ideal for -multi-core builds). - -Internal PTH Optimizations --------------------------- - -While the main optimization employed by PTH is to reduce lexing time of -header files by caching pre-lexed tokens, PTH also employs several other -optimizations to speed up the processing of header files: - -- ``stat`` caching: PTH files cache information obtained via calls to - ``stat`` that ``clang -cc1`` uses to resolve which files are included - by ``#include`` directives. This greatly reduces the overhead - involved in context-switching to the kernel to resolve included - files. - -- Fast skipping of ``#ifdef`` ... ``#endif`` chains: PTH files - record the basic structure of nested preprocessor blocks. When the - condition of the preprocessor block is false, all of its tokens are - immediately skipped instead of requiring them to be handled by - Clang's preprocessor. - - Index: docs/ReleaseNotes.rst =================================================================== --- docs/ReleaseNotes.rst +++ docs/ReleaseNotes.rst @@ -116,7 +116,9 @@ Non-comprehensive list of changes in this release ------------------------------------------------- -- ... +- The experimental feature Pretokenized Headers (PTH) was removed in its + entirely from Clang. The feature did not properly work with about 1/3 of the + possible tokens available and was unmaintained. New Compiler Flags ------------------ Index: include/clang/Basic/DiagnosticFrontendKinds.td =================================================================== --- include/clang/Basic/DiagnosticFrontendKinds.td +++ include/clang/Basic/DiagnosticFrontendKinds.td @@ -99,8 +99,6 @@ "unable to interface with target machine">; def err_fe_unable_to_open_output : Error< "unable to open output file '%0': '%1'">; -def err_fe_pth_file_has_no_source_header : Error< - "PTH file '%0' does not designate an original source header file for -include-pth">; def warn_fe_macro_contains_embedded_newline : Warning< "macro '%0' contains embedded newline; text after the newline is ignored">; def warn_fe_cc_print_header_failure : Warning< Index: include/clang/Basic/DiagnosticLexKinds.td =================================================================== --- include/clang/Basic/DiagnosticLexKinds.td +++ include/clang/Basic/DiagnosticLexKinds.td @@ -258,13 +258,6 @@ def err_lexing_string : Error<"failure when lexing a string">; def err_placeholder_in_source : Error<"editor placeholder in source file">; - -//===----------------------------------------------------------------------===// -// PTH Diagnostics -//===----------------------------------------------------------------------===// -def err_invalid_pth_file : Error< - "invalid or corrupt PTH file '%0'">; - //===----------------------------------------------------------------------===// // Preprocessor Diagnostics //===----------------------------------------------------------------------===// @@ -570,8 +563,6 @@ "'#pragma clang module end'">; def note_pp_module_begin_here : Note< "entering module '%0' due to this pragma">; -def err_pp_module_build_pth : Error< - "'#pragma clang module build' not supported in pretokenized header">; def err_pp_module_build_missing_end : Error< "no matching '#pragma clang module endbuild' for this '#pragma clang module build'">; Index: include/clang/Driver/CC1Options.td =================================================================== --- include/clang/Driver/CC1Options.td +++ include/clang/Driver/CC1Options.td @@ -562,8 +562,6 @@ HelpText<"Generate pre-compiled module file from a C++ module interface">; def emit_header_module : Flag<["-"], "emit-header-module">, HelpText<"Generate pre-compiled module file from a set of header files">; -def emit_pth : Flag<["-"], "emit-pth">, - HelpText<"Generate pre-tokenized header file">; def emit_pch : Flag<["-"], "emit-pch">, HelpText<"Generate pre-compiled header file">; def emit_llvm_bc : Flag<["-"], "emit-llvm-bc">, @@ -794,15 +792,11 @@ // Preprocessor Options //===----------------------------------------------------------------------===// -def include_pth : Separate<["-"], "include-pth">, MetaVarName<"">, - HelpText<"Include file before parsing">; def chain_include : Separate<["-"], "chain-include">, MetaVarName<"">, HelpText<"Include and chain a header file after turning it into PCH">; def preamble_bytes_EQ : Joined<["-"], "preamble-bytes=">, HelpText<"Assume that the precompiled header is a precompiled preamble " "covering the first N bytes of the main file">; -def token_cache : Separate<["-"], "token-cache">, MetaVarName<"">, - HelpText<"Use specified token cache file">; def detailed_preprocessing_record : Flag<["-"], "detailed-preprocessing-record">, HelpText<"include a detailed record of preprocessing actions">; Index: include/clang/Driver/Driver.h =================================================================== --- include/clang/Driver/Driver.h +++ include/clang/Driver/Driver.h @@ -227,9 +227,6 @@ unsigned CheckInputsExist : 1; public: - /// Use lazy precompiled headers for PCH support. - unsigned CCCUsePCH : 1; - /// Force clang to emit reproducer for driver invocation. This is enabled /// indirectly by setting FORCE_CLANG_DIAGNOSTICS_CRASH environment variable /// or when using the -gen-reproducer driver flag. Index: include/clang/Driver/Options.td =================================================================== --- include/clang/Driver/Options.td +++ include/clang/Driver/Options.td @@ -258,10 +258,6 @@ def ccc_gcc_name : Separate<["-"], "ccc-gcc-name">, InternalDriverOpt, HelpText<"Name for native GCC compiler">, MetaVarName<"">; -def ccc_pch_is_pch : Flag<["-"], "ccc-pch-is-pch">, InternalDriverOpt, - HelpText<"Use lazy PCH for precompiled headers">; -def ccc_pch_is_pth : Flag<["-"], "ccc-pch-is-pth">, InternalDriverOpt, - HelpText<"Use pretokenized headers for precompiled headers">; class InternalDebugOpt : Group, Flags<[DriverOption, HelpHidden, CoreOption]>; Index: include/clang/Frontend/FrontendActions.h =================================================================== --- include/clang/Frontend/FrontendActions.h +++ include/clang/Frontend/FrontendActions.h @@ -261,11 +261,6 @@ void ExecuteAction() override; }; -class GeneratePTHAction : public PreprocessorFrontendAction { -protected: - void ExecuteAction() override; -}; - class PreprocessOnlyAction : public PreprocessorFrontendAction { protected: void ExecuteAction() override; Index: include/clang/Frontend/FrontendOptions.h =================================================================== --- include/clang/Frontend/FrontendOptions.h +++ include/clang/Frontend/FrontendOptions.h @@ -88,9 +88,6 @@ /// Generate pre-compiled header. GeneratePCH, - /// Generate pre-tokenized header. - GeneratePTH, - /// Only execute frontend initialization. InitOnly, Index: include/clang/Frontend/Utils.h =================================================================== --- include/clang/Frontend/Utils.h +++ include/clang/Frontend/Utils.h @@ -183,9 +183,6 @@ StringRef OutputPath = {}, bool ShowDepth = true, bool MSStyle = false); -/// Cache tokens for use with PCH. Note that this requires a seekable stream. -void CacheTokens(Preprocessor &PP, raw_pwrite_stream *OS); - /// The ChainedIncludesSource class converts headers to chained PCHs in /// memory, mainly for testing. IntrusiveRefCntPtr Index: include/clang/Lex/PTHLexer.h =================================================================== --- include/clang/Lex/PTHLexer.h +++ include/clang/Lex/PTHLexer.h @@ -1,107 +0,0 @@ -//===- PTHLexer.h - Lexer based on Pre-tokenized input ----------*- C++ -*-===// -// -// The LLVM Compiler Infrastructure -// -// This file is distributed under the University of Illinois Open Source -// License. See LICENSE.TXT for details. -// -//===----------------------------------------------------------------------===// -// -// This file defines the PTHLexer interface. -// -//===----------------------------------------------------------------------===// - -#ifndef LLVM_CLANG_LEX_PTHLEXER_H -#define LLVM_CLANG_LEX_PTHLEXER_H - -#include "clang/Basic/SourceLocation.h" -#include "clang/Basic/TokenKinds.h" -#include "clang/Lex/PreprocessorLexer.h" -#include "clang/Lex/Token.h" - -namespace clang { - -class Preprocessor; -class PTHManager; - -class PTHLexer : public PreprocessorLexer { - SourceLocation FileStartLoc; - - /// TokBuf - Buffer from PTH file containing raw token data. - const unsigned char* TokBuf; - - /// CurPtr - Pointer into current offset of the token buffer where - /// the next token will be read. - const unsigned char* CurPtr; - - /// LastHashTokPtr - Pointer into TokBuf of the last processed '#' - /// token that appears at the start of a line. - const unsigned char* LastHashTokPtr = nullptr; - - /// PPCond - Pointer to a side table in the PTH file that provides a - /// a concise summary of the preprocessor conditional block structure. - /// This is used to perform quick skipping of conditional blocks. - const unsigned char* PPCond; - - /// CurPPCondPtr - Pointer inside PPCond that refers to the next entry - /// to process when doing quick skipping of preprocessor blocks. - const unsigned char* CurPPCondPtr; - - /// ReadToken - Used by PTHLexer to read tokens TokBuf. - void ReadToken(Token &T); - - bool LexEndOfFile(Token &Result); - - /// PTHMgr - The PTHManager object that created this PTHLexer. - PTHManager& PTHMgr; - - Token EofToken; - -protected: - friend class PTHManager; - - /// Create a PTHLexer for the specified token stream. - PTHLexer(Preprocessor &pp, FileID FID, const unsigned char *D, - const unsigned char* ppcond, PTHManager &PM); - -public: - PTHLexer(const PTHLexer &) = delete; - PTHLexer &operator=(const PTHLexer &) = delete; - ~PTHLexer() override = default; - - /// Lex - Return the next token. - bool Lex(Token &Tok); - - void getEOF(Token &Tok); - - /// DiscardToEndOfLine - Read the rest of the current preprocessor line as an - /// uninterpreted string. This switches the lexer out of directive mode. - void DiscardToEndOfLine(); - - /// isNextPPTokenLParen - Return 1 if the next unexpanded token will return a - /// tok::l_paren token, 0 if it is something else and 2 if there are no more - /// tokens controlled by this lexer. - unsigned isNextPPTokenLParen() { - // isNextPPTokenLParen is not on the hot path, and all we care about is - // whether or not we are at a token with kind tok::eof or tok::l_paren. - // Just read the first byte from the current token pointer to determine - // its kind. - tok::TokenKind x = (tok::TokenKind)*CurPtr; - return x == tok::eof ? 2 : x == tok::l_paren; - } - - /// IndirectLex - An indirect call to 'Lex' that can be invoked via - /// the PreprocessorLexer interface. - void IndirectLex(Token &Result) override { Lex(Result); } - - /// getSourceLocation - Return a source location for the token in - /// the current file. - SourceLocation getSourceLocation() override; - - /// SkipBlock - Used by Preprocessor to skip the current conditional block. - bool SkipBlock(); -}; - -} // namespace clang - -#endif // LLVM_CLANG_LEX_PTHLEXER_H Index: include/clang/Lex/PTHManager.h =================================================================== --- include/clang/Lex/PTHManager.h +++ include/clang/Lex/PTHManager.h @@ -1,149 +0,0 @@ -//===- PTHManager.h - Manager object for PTH processing ---------*- C++ -*-===// -// -// The LLVM Compiler Infrastructure -// -// This file is distributed under the University of Illinois Open Source -// License. See LICENSE.TXT for details. -// -//===----------------------------------------------------------------------===// -// -// This file defines the PTHManager interface. -// -//===----------------------------------------------------------------------===// - -#ifndef LLVM_CLANG_LEX_PTHMANAGER_H -#define LLVM_CLANG_LEX_PTHMANAGER_H - -#include "clang/Basic/IdentifierTable.h" -#include "clang/Basic/SourceLocation.h" -#include "llvm/ADT/STLExtras.h" -#include "llvm/ADT/StringRef.h" -#include "llvm/Support/Allocator.h" -#include "llvm/Support/OnDiskHashTable.h" -#include - -namespace llvm { - -class MemoryBuffer; - -} // namespace llvm - -namespace clang { - -class DiagnosticsEngine; -class FileSystemStatCache; -class Preprocessor; -class PTHLexer; - -class PTHManager : public IdentifierInfoLookup { - friend class PTHLexer; - friend class PTHStatCache; - - class PTHFileLookupTrait; - class PTHStringLookupTrait; - - using PTHStringIdLookup = llvm::OnDiskChainedHashTable; - using PTHFileLookup = llvm::OnDiskChainedHashTable; - - /// The memory mapped PTH file. - std::unique_ptr Buf; - - /// Alloc - Allocator used for IdentifierInfo objects. - llvm::BumpPtrAllocator Alloc; - - /// IdMap - A lazily generated cache mapping from persistent identifiers to - /// IdentifierInfo*. - std::unique_ptr PerIDCache; - - /// FileLookup - Abstract data structure used for mapping between files - /// and token data in the PTH file. - std::unique_ptr FileLookup; - - /// IdDataTable - Array representing the mapping from persistent IDs to the - /// data offset within the PTH file containing the information to - /// reconsitute an IdentifierInfo. - const unsigned char* const IdDataTable; - - /// SortedIdTable - Abstract data structure mapping from strings to - /// persistent IDs. This is used by get(). - std::unique_ptr StringIdLookup; - - /// NumIds - The number of identifiers in the PTH file. - const unsigned NumIds; - - /// PP - The Preprocessor object that will use this PTHManager to create - /// PTHLexer objects. - Preprocessor* PP = nullptr; - - /// SpellingBase - The base offset within the PTH memory buffer that - /// contains the cached spellings for literals. - const unsigned char* const SpellingBase; - - /// OriginalSourceFile - A null-terminated C-string that specifies the name - /// if the file (if any) that was to used to generate the PTH cache. - const char* OriginalSourceFile; - - /// This constructor is intended to only be called by the static 'Create' - /// method. - PTHManager(std::unique_ptr buf, - std::unique_ptr fileLookup, - const unsigned char *idDataTable, - std::unique_ptr perIDCache, - std::unique_ptr stringIdLookup, unsigned numIds, - const unsigned char *spellingBase, const char *originalSourceFile); - - /// getSpellingAtPTHOffset - Used by PTHLexer classes to get the cached - /// spelling for a token. - unsigned getSpellingAtPTHOffset(unsigned PTHOffset, const char*& Buffer); - - /// GetIdentifierInfo - Used to reconstruct IdentifierInfo objects from the - /// PTH file. - IdentifierInfo *GetIdentifierInfo(unsigned PersistentID) { - // Check if the IdentifierInfo has already been resolved. - if (IdentifierInfo* II = PerIDCache[PersistentID]) - return II; - return LazilyCreateIdentifierInfo(PersistentID); - } - IdentifierInfo* LazilyCreateIdentifierInfo(unsigned PersistentID); - -public: - // The current PTH version. - enum { Version = 10 }; - - PTHManager(const PTHManager &) = delete; - PTHManager &operator=(const PTHManager &) = delete; - ~PTHManager() override; - - /// getOriginalSourceFile - Return the full path to the original header - /// file name that was used to generate the PTH cache. - const char* getOriginalSourceFile() const { - return OriginalSourceFile; - } - - /// get - Return the identifier token info for the specified named identifier. - /// Unlike the version in IdentifierTable, this returns a pointer instead - /// of a reference. If the pointer is nullptr then the IdentifierInfo cannot - /// be found. - IdentifierInfo *get(StringRef Name) override; - - /// Create - This method creates PTHManager objects. The 'file' argument - /// is the name of the PTH file. This method returns nullptr upon failure. - static PTHManager *Create(StringRef file, DiagnosticsEngine &Diags); - - void setPreprocessor(Preprocessor *pp) { PP = pp; } - - /// CreateLexer - Return a PTHLexer that "lexes" the cached tokens for the - /// specified file. This method returns nullptr if no cached tokens exist. - /// It is the responsibility of the caller to 'delete' the returned object. - PTHLexer *CreateLexer(FileID FID); - - /// createStatCache - Returns a FileSystemStatCache object for use with - /// FileManager objects. These objects use the PTH data to speed up - /// calls to stat by memoizing their results from when the PTH file - /// was generated. - std::unique_ptr createStatCache(); -}; - -} // namespace clang - -#endif // LLVM_CLANG_LEX_PTHMANAGER_H Index: include/clang/Lex/Preprocessor.h =================================================================== --- include/clang/Lex/Preprocessor.h +++ include/clang/Lex/Preprocessor.h @@ -29,7 +29,6 @@ #include "clang/Lex/ModuleLoader.h" #include "clang/Lex/ModuleMap.h" #include "clang/Lex/PPCallbacks.h" -#include "clang/Lex/PTHLexer.h" #include "clang/Lex/Token.h" #include "clang/Lex/TokenLexer.h" #include "llvm/ADT/ArrayRef.h" @@ -79,7 +78,6 @@ class PreprocessingRecord; class PreprocessorLexer; class PreprocessorOptions; -class PTHManager; class ScratchBuffer; class TargetInfo; @@ -143,10 +141,6 @@ /// External source of macros. ExternalPreprocessorSource *ExternalSource; - /// An optional PTHManager object used for getting tokens from - /// a token cache rather than lexing the original source file. - std::unique_ptr PTH; - /// A BumpPtrAllocator object used to quickly allocate and release /// objects internal to the Preprocessor. llvm::BumpPtrAllocator BP; @@ -392,19 +386,13 @@ /// The current top of the stack that we're lexing from if /// not expanding a macro and we are lexing directly from source code. /// - /// Only one of CurLexer, CurPTHLexer, or CurTokenLexer will be non-null. + /// Only one of CurLexer, or CurTokenLexer will be non-null. std::unique_ptr CurLexer; - /// The current top of stack that we're lexing from if - /// not expanding from a macro and we are lexing from a PTH cache. - /// - /// Only one of CurLexer, CurPTHLexer, or CurTokenLexer will be non-null. - std::unique_ptr CurPTHLexer; - /// The current top of the stack what we're lexing from /// if not expanding a macro. /// - /// This is an alias for either CurLexer or CurPTHLexer. + /// This is an alias for CurLexer. PreprocessorLexer *CurPPLexer = nullptr; /// Used to find the current FileEntry, if CurLexer is non-null @@ -422,7 +410,6 @@ /// The kind of lexer we're currently working with. enum CurLexerKind { CLK_Lexer, - CLK_PTHLexer, CLK_TokenLexer, CLK_CachingLexer, CLK_LexAfterModuleImport @@ -439,7 +426,6 @@ enum CurLexerKind CurLexerKind; Module *TheSubmodule; std::unique_ptr TheLexer; - std::unique_ptr ThePTHLexer; PreprocessorLexer *ThePPLexer; std::unique_ptr TheTokenLexer; const DirectoryLookup *TheDirLookup; @@ -448,13 +434,11 @@ // versions, only needed to pacify MSVC. IncludeStackInfo(enum CurLexerKind CurLexerKind, Module *TheSubmodule, std::unique_ptr &&TheLexer, - std::unique_ptr &&ThePTHLexer, PreprocessorLexer *ThePPLexer, std::unique_ptr &&TheTokenLexer, const DirectoryLookup *TheDirLookup) : CurLexerKind(std::move(CurLexerKind)), TheSubmodule(std::move(TheSubmodule)), TheLexer(std::move(TheLexer)), - ThePTHLexer(std::move(ThePTHLexer)), ThePPLexer(std::move(ThePPLexer)), TheTokenLexer(std::move(TheTokenLexer)), TheDirLookup(std::move(TheDirLookup)) {} @@ -838,10 +822,6 @@ Builtin::Context &getBuiltinInfo() { return BuiltinInfo; } llvm::BumpPtrAllocator &getPreprocessorAllocator() { return BP; } - void setPTHManager(PTHManager* pm); - - PTHManager *getPTHManager() { return PTH.get(); } - void setExternalSource(ExternalPreprocessorSource *Source) { ExternalSource = Source; } @@ -1456,7 +1436,7 @@ CachedTokens[CachedLexPos-1] = Tok; } - /// Recompute the current lexer kind based on the CurLexer/CurPTHLexer/ + /// Recompute the current lexer kind based on the CurLexer/ /// CurTokenLexer pointers. void recomputeCurLexerKind(); @@ -1915,15 +1895,13 @@ void PushIncludeMacroStack() { assert(CurLexerKind != CLK_CachingLexer && "cannot push a caching lexer"); IncludeMacroStack.emplace_back(CurLexerKind, CurLexerSubmodule, - std::move(CurLexer), std::move(CurPTHLexer), - CurPPLexer, std::move(CurTokenLexer), - CurDirLookup); + std::move(CurLexer), CurPPLexer, + std::move(CurTokenLexer), CurDirLookup); CurPPLexer = nullptr; } void PopIncludeMacroStack() { CurLexer = std::move(IncludeMacroStack.back().TheLexer); - CurPTHLexer = std::move(IncludeMacroStack.back().ThePTHLexer); CurPPLexer = IncludeMacroStack.back().ThePPLexer; CurTokenLexer = std::move(IncludeMacroStack.back().TheTokenLexer); CurDirLookup = IncludeMacroStack.back().TheDirLookup; @@ -1992,9 +1970,6 @@ bool FoundNonSkipPortion, bool FoundElse, SourceLocation ElseLoc = SourceLocation()); - /// A fast PTH version of SkipExcludedConditionalBlock. - void PTHSkipExcludedConditionalBlock(); - /// Information about the result for evaluating an expression for a /// preprocessor directive. struct DirectiveEvalResult { @@ -2060,10 +2035,6 @@ /// start lexing tokens from it instead of the current buffer. void EnterSourceFileWithLexer(Lexer *TheLexer, const DirectoryLookup *Dir); - /// Add a lexer to the top of the include stack and - /// start getting tokens from it using the PTH cache. - void EnterSourceFileWithPTH(PTHLexer *PL, const DirectoryLookup *Dir); - /// Set the FileID for the preprocessor predefines. void setPredefinesFileID(FileID FID) { assert(PredefinesFileID.isInvalid() && "PredefinesFileID already set!"); @@ -2094,8 +2065,7 @@ bool InCachingLexMode() const { // If the Lexer pointers are 0 and IncludeMacroStack is empty, it means // that we are past EOF, not that we are in CachingLex mode. - return !CurPPLexer && !CurTokenLexer && !CurPTHLexer && - !IncludeMacroStack.empty(); + return !CurPPLexer && !CurTokenLexer && !IncludeMacroStack.empty(); } void EnterCachingLexMode(); Index: include/clang/Lex/PreprocessorOptions.h =================================================================== --- include/clang/Lex/PreprocessorOptions.h +++ include/clang/Lex/PreprocessorOptions.h @@ -110,13 +110,6 @@ /// clients don't use them. bool WriteCommentListToPCH = true; - /// The implicit PTH input included at the start of the translation unit, or - /// empty. - std::string ImplicitPTHInclude; - - /// If given, a PTH cache file to use for speeding up header parsing. - std::string TokenCache; - /// When enabled, preprocessor is in a mode for parsing a single file only. /// /// Disables #includes of other files and if there are unresolved identifiers @@ -204,8 +197,6 @@ ChainedIncludes.clear(); DumpDeserializedPCHDecls = false; ImplicitPCHInclude.clear(); - ImplicitPTHInclude.clear(); - TokenCache.clear(); SingleFileParseMode = false; LexEditorPlaceholders = true; RetainRemappedFileBuffers = true; Index: lib/ARCMigrate/ARCMT.cpp =================================================================== --- lib/ARCMigrate/ARCMT.cpp +++ lib/ARCMigrate/ARCMT.cpp @@ -190,8 +190,6 @@ PPOpts.Includes.insert(PPOpts.Includes.begin(), OriginalFile); PPOpts.ImplicitPCHInclude.clear(); } - // FIXME: Get the original header of a PTH as well. - CInvok->getPreprocessorOpts().ImplicitPTHInclude.clear(); std::string define = getARCMTMacroName(); define += '='; CInvok->getPreprocessorOpts().addMacroDef(define); Index: lib/Driver/Driver.cpp =================================================================== --- lib/Driver/Driver.cpp +++ lib/Driver/Driver.cpp @@ -101,8 +101,7 @@ CCPrintOptions(false), CCPrintHeaders(false), CCLogDiagnostics(false), CCGenDiagnostics(false), TargetTriple(TargetTriple), CCCGenericGCCName(""), Saver(Alloc), CheckInputsExist(true), - CCCUsePCH(true), GenReproducer(false), - SuppressMissingInputWarning(false) { + GenReproducer(false), SuppressMissingInputWarning(false) { // Provide a sane fallback if no VFS is specified. if (!this->VFS) @@ -1007,8 +1006,6 @@ CCCPrintBindings = Args.hasArg(options::OPT_ccc_print_bindings); if (const Arg *A = Args.getLastArg(options::OPT_ccc_gcc_name)) CCCGenericGCCName = A->getValue(); - CCCUsePCH = - Args.hasFlag(options::OPT_ccc_pch_is_pch, options::OPT_ccc_pch_is_pth); GenReproducer = Args.hasFlag(options::OPT_gen_reproducer, options::OPT_fno_crash_diagnostics, !!::getenv("FORCE_CLANG_DIAGNOSTICS_CRASH")); Index: lib/Driver/ToolChains/Clang.cpp =================================================================== --- lib/Driver/ToolChains/Clang.cpp +++ lib/Driver/ToolChains/Clang.cpp @@ -1152,42 +1152,26 @@ bool IsFirstImplicitInclude = !RenderedImplicitInclude; RenderedImplicitInclude = true; - // Use PCH if the user requested it. - bool UsePCH = D.CCCUsePCH; - - bool FoundPTH = false; bool FoundPCH = false; SmallString<128> P(A->getValue()); // We want the files to have a name like foo.h.pch. Add a dummy extension // so that replace_extension does the right thing. P += ".dummy"; - if (UsePCH) { - llvm::sys::path::replace_extension(P, "pch"); - if (llvm::sys::fs::exists(P)) - FoundPCH = true; - } + llvm::sys::path::replace_extension(P, "pch"); + if (llvm::sys::fs::exists(P)) + FoundPCH = true; if (!FoundPCH) { - llvm::sys::path::replace_extension(P, "pth"); - if (llvm::sys::fs::exists(P)) - FoundPTH = true; - } - - if (!FoundPCH && !FoundPTH) { llvm::sys::path::replace_extension(P, "gch"); if (llvm::sys::fs::exists(P)) { - FoundPCH = UsePCH; - FoundPTH = !UsePCH; + FoundPCH = true; } } - if (FoundPCH || FoundPTH) { + if (FoundPCH) { if (IsFirstImplicitInclude) { A->claim(); - if (UsePCH) - CmdArgs.push_back("-include-pch"); - else - CmdArgs.push_back("-include-pth"); + CmdArgs.push_back("-include-pch"); CmdArgs.push_back(Args.MakeArgString(P)); continue; } else { @@ -3467,19 +3451,14 @@ // Also ignore explicit -force_cpusubtype_ALL option. (void)Args.hasArg(options::OPT_force__cpusubtype__ALL); } else if (isa(JA)) { - // Use PCH if the user requested it. - bool UsePCH = D.CCCUsePCH; - if (JA.getType() == types::TY_Nothing) CmdArgs.push_back("-fsyntax-only"); else if (JA.getType() == types::TY_ModuleFile) CmdArgs.push_back(IsHeaderModulePrecompile ? "-emit-header-module" : "-emit-module-interface"); - else if (UsePCH) - CmdArgs.push_back("-emit-pch"); else - CmdArgs.push_back("-emit-pth"); + CmdArgs.push_back("-emit-pch"); } else if (isa(JA)) { CmdArgs.push_back("-verify-pch"); } else { @@ -5247,8 +5226,7 @@ // Claim some arguments which clang supports automatically. // -fpch-preprocess is used with gcc to add a special marker in the output to - // include the PCH file. Clang's PTH solution is completely transparent, so we - // do not need to deal with it at all. + // include the PCH file. Args.ClaimAllArgs(options::OPT_fpch_preprocess); // Claim some arguments which clang doesn't support, but we don't Index: lib/Frontend/CMakeLists.txt =================================================================== --- lib/Frontend/CMakeLists.txt +++ lib/Frontend/CMakeLists.txt @@ -16,7 +16,6 @@ ASTConsumers.cpp ASTMerge.cpp ASTUnit.cpp - CacheTokens.cpp ChainedDiagnosticConsumer.cpp ChainedIncludesSource.cpp CodeGenOptions.cpp Index: lib/Frontend/CacheTokens.cpp =================================================================== --- lib/Frontend/CacheTokens.cpp +++ lib/Frontend/CacheTokens.cpp @@ -1,700 +0,0 @@ -//===--- CacheTokens.cpp - Caching of lexer tokens for PTH support --------===// -// -// The LLVM Compiler Infrastructure -// -// This file is distributed under the University of Illinois Open Source -// License. See LICENSE.TXT for details. -// -//===----------------------------------------------------------------------===// -// -// This provides a possible implementation of PTH support for Clang that is -// based on caching lexed tokens and identifiers. -// -//===----------------------------------------------------------------------===// - -#include "clang/Basic/Diagnostic.h" -#include "clang/Basic/FileManager.h" -#include "clang/Basic/FileSystemStatCache.h" -#include "clang/Basic/IdentifierTable.h" -#include "clang/Basic/SourceManager.h" -#include "clang/Frontend/Utils.h" -#include "clang/Lex/Lexer.h" -#include "clang/Lex/PTHManager.h" -#include "clang/Lex/Preprocessor.h" -#include "llvm/ADT/StringMap.h" -#include "llvm/Support/DJB.h" -#include "llvm/Support/EndianStream.h" -#include "llvm/Support/FileSystem.h" -#include "llvm/Support/MemoryBuffer.h" -#include "llvm/Support/OnDiskHashTable.h" -#include "llvm/Support/Path.h" - -// FIXME: put this somewhere else? -#ifndef S_ISDIR -#define S_ISDIR(x) (((x)&_S_IFDIR)!=0) -#endif - -using namespace clang; - -//===----------------------------------------------------------------------===// -// PTH-specific stuff. -//===----------------------------------------------------------------------===// - -typedef uint32_t Offset; - -namespace { -class PTHEntry { - Offset TokenData, PPCondData; - -public: - PTHEntry() {} - - PTHEntry(Offset td, Offset ppcd) - : TokenData(td), PPCondData(ppcd) {} - - Offset getTokenOffset() const { return TokenData; } - Offset getPPCondTableOffset() const { return PPCondData; } -}; - - -class PTHEntryKeyVariant { - union { - const FileEntry *FE; - // FIXME: Use "StringRef Path;" when MSVC 2013 is dropped. - const char *PathPtr; - }; - size_t PathSize; - enum { IsFE = 0x1, IsDE = 0x2, IsNoExist = 0x0 } Kind; - FileData *Data; - -public: - PTHEntryKeyVariant(const FileEntry *fe) : FE(fe), Kind(IsFE), Data(nullptr) {} - - PTHEntryKeyVariant(FileData *Data, StringRef Path) - : PathPtr(Path.data()), PathSize(Path.size()), Kind(IsDE), - Data(new FileData(*Data)) {} - - explicit PTHEntryKeyVariant(StringRef Path) - : PathPtr(Path.data()), PathSize(Path.size()), Kind(IsNoExist), - Data(nullptr) {} - - bool isFile() const { return Kind == IsFE; } - - StringRef getString() const { - return Kind == IsFE ? FE->getName() : StringRef(PathPtr, PathSize); - } - - unsigned getKind() const { return (unsigned) Kind; } - - void EmitData(raw_ostream& Out) { - using namespace llvm::support; - endian::Writer LE(Out, little); - switch (Kind) { - case IsFE: { - // Emit stat information. - llvm::sys::fs::UniqueID UID = FE->getUniqueID(); - LE.write(UID.getFile()); - LE.write(UID.getDevice()); - LE.write(FE->getModificationTime()); - LE.write(FE->getSize()); - } break; - case IsDE: - // Emit stat information. - LE.write(Data->UniqueID.getFile()); - LE.write(Data->UniqueID.getDevice()); - LE.write(Data->ModTime); - LE.write(Data->Size); - delete Data; - break; - default: - break; - } - } - - unsigned getRepresentationLength() const { - return Kind == IsNoExist ? 0 : 4 * 8; - } -}; - -class FileEntryPTHEntryInfo { -public: - typedef PTHEntryKeyVariant key_type; - typedef key_type key_type_ref; - - typedef PTHEntry data_type; - typedef const PTHEntry& data_type_ref; - - typedef unsigned hash_value_type; - typedef unsigned offset_type; - - static hash_value_type ComputeHash(PTHEntryKeyVariant V) { - return llvm::djbHash(V.getString()); - } - - static std::pair - EmitKeyDataLength(raw_ostream& Out, PTHEntryKeyVariant V, - const PTHEntry& E) { - using namespace llvm::support; - endian::Writer LE(Out, little); - - unsigned n = V.getString().size() + 1 + 1; - LE.write(n); - - unsigned m = V.getRepresentationLength() + (V.isFile() ? 4 + 4 : 0); - LE.write(m); - - return std::make_pair(n, m); - } - - static void EmitKey(raw_ostream& Out, PTHEntryKeyVariant V, unsigned n){ - using namespace llvm::support; - // Emit the entry kind. - Out << char(V.getKind()); - // Emit the string. - Out.write(V.getString().data(), n - 1); - } - - static void EmitData(raw_ostream& Out, PTHEntryKeyVariant V, - const PTHEntry& E, unsigned) { - using namespace llvm::support; - endian::Writer LE(Out, little); - - // For file entries emit the offsets into the PTH file for token data - // and the preprocessor blocks table. - if (V.isFile()) { - LE.write(E.getTokenOffset()); - LE.write(E.getPPCondTableOffset()); - } - - // Emit any other data associated with the key (i.e., stat information). - V.EmitData(Out); - } -}; - -class OffsetOpt { - bool valid; - Offset off; -public: - OffsetOpt() : valid(false) {} - bool hasOffset() const { return valid; } - Offset getOffset() const { assert(valid); return off; } - void setOffset(Offset o) { off = o; valid = true; } -}; -} // end anonymous namespace - -typedef llvm::OnDiskChainedHashTableGenerator PTHMap; - -namespace { -class PTHWriter { - typedef llvm::DenseMap IDMap; - typedef llvm::StringMap CachedStrsTy; - - raw_pwrite_stream &Out; - Preprocessor& PP; - IDMap IM; - std::vector*> StrEntries; - PTHMap PM; - CachedStrsTy CachedStrs; - uint32_t idcount; - Offset CurStrOffset; - - //// Get the persistent id for the given IdentifierInfo*. - uint32_t ResolveID(const IdentifierInfo* II); - - /// Emit a token to the PTH file. - void EmitToken(const Token& T); - - void Emit8(uint32_t V) { - Out << char(V); - } - - void Emit16(uint32_t V) { - using namespace llvm::support; - endian::write(Out, V, little); - } - - void Emit32(uint32_t V) { - using namespace llvm::support; - endian::write(Out, V, little); - } - - void EmitBuf(const char *Ptr, unsigned NumBytes) { - Out.write(Ptr, NumBytes); - } - - void EmitString(StringRef V) { - using namespace llvm::support; - endian::write(Out, V.size(), little); - EmitBuf(V.data(), V.size()); - } - - /// EmitIdentifierTable - Emits two tables to the PTH file. The first is - /// a hashtable mapping from identifier strings to persistent IDs. - /// The second is a straight table mapping from persistent IDs to string data - /// (the keys of the first table). - std::pair EmitIdentifierTable(); - - /// EmitFileTable - Emit a table mapping from file name strings to PTH - /// token data. - Offset EmitFileTable() { return PM.Emit(Out); } - - PTHEntry LexTokens(Lexer& L); - Offset EmitCachedSpellings(); - -public: - PTHWriter(raw_pwrite_stream &out, Preprocessor &pp) - : Out(out), PP(pp), idcount(0), CurStrOffset(0) {} - - PTHMap &getPM() { return PM; } - void GeneratePTH(StringRef MainFile); -}; -} // end anonymous namespace - -uint32_t PTHWriter::ResolveID(const IdentifierInfo* II) { - // Null IdentifierInfo's map to the persistent ID 0. - if (!II) - return 0; - - IDMap::iterator I = IM.find(II); - if (I != IM.end()) - return I->second; // We've already added 1. - - IM[II] = ++idcount; // Pre-increment since '0' is reserved for NULL. - return idcount; -} - -void PTHWriter::EmitToken(const Token& T) { - // Emit the token kind, flags, and length. - Emit32(((uint32_t) T.getKind()) | ((((uint32_t) T.getFlags())) << 8)| - (((uint32_t) T.getLength()) << 16)); - - if (!T.isLiteral()) { - Emit32(ResolveID(T.getIdentifierInfo())); - } else { - // We cache *un-cleaned* spellings. This gives us 100% fidelity with the - // source code. - StringRef s(T.getLiteralData(), T.getLength()); - - // Get the string entry. - auto &E = *CachedStrs.insert(std::make_pair(s, OffsetOpt())).first; - - // If this is a new string entry, bump the PTH offset. - if (!E.second.hasOffset()) { - E.second.setOffset(CurStrOffset); - StrEntries.push_back(&E); - CurStrOffset += s.size() + 1; - } - - // Emit the relative offset into the PTH file for the spelling string. - Emit32(E.second.getOffset()); - } - - // Emit the offset into the original source file of this token so that we - // can reconstruct its SourceLocation. - Emit32(PP.getSourceManager().getFileOffset(T.getLocation())); -} - -PTHEntry PTHWriter::LexTokens(Lexer& L) { - // Pad 0's so that we emit tokens to a 4-byte alignment. - // This speed up reading them back in. - using namespace llvm::support; - endian::Writer LE(Out, little); - uint32_t TokenOff = Out.tell(); - for (uint64_t N = llvm::OffsetToAlignment(TokenOff, 4); N; --N, ++TokenOff) - LE.write(0); - - // Keep track of matching '#if' ... '#endif'. - typedef std::vector > PPCondTable; - PPCondTable PPCond; - std::vector PPStartCond; - bool ParsingPreprocessorDirective = false; - Token Tok; - - do { - L.LexFromRawLexer(Tok); - NextToken: - - if ((Tok.isAtStartOfLine() || Tok.is(tok::eof)) && - ParsingPreprocessorDirective) { - // Insert an eod token into the token cache. It has the same - // position as the next token that is not on the same line as the - // preprocessor directive. Observe that we continue processing - // 'Tok' when we exit this branch. - Token Tmp = Tok; - Tmp.setKind(tok::eod); - Tmp.clearFlag(Token::StartOfLine); - Tmp.setIdentifierInfo(nullptr); - EmitToken(Tmp); - ParsingPreprocessorDirective = false; - } - - if (Tok.is(tok::raw_identifier)) { - PP.LookUpIdentifierInfo(Tok); - EmitToken(Tok); - continue; - } - - if (Tok.is(tok::hash) && Tok.isAtStartOfLine()) { - // Special processing for #include. Store the '#' token and lex - // the next token. - assert(!ParsingPreprocessorDirective); - Offset HashOff = (Offset) Out.tell(); - - // Get the next token. - Token NextTok; - L.LexFromRawLexer(NextTok); - - // If we see the start of line, then we had a null directive "#". In - // this case, discard both tokens. - if (NextTok.isAtStartOfLine()) - goto NextToken; - - // The token is the start of a directive. Emit it. - EmitToken(Tok); - Tok = NextTok; - - // Did we see 'include'/'import'/'include_next'? - if (Tok.isNot(tok::raw_identifier)) { - EmitToken(Tok); - continue; - } - - IdentifierInfo* II = PP.LookUpIdentifierInfo(Tok); - tok::PPKeywordKind K = II->getPPKeywordID(); - - ParsingPreprocessorDirective = true; - - switch (K) { - case tok::pp_not_keyword: - // Invalid directives "#foo" can occur in #if 0 blocks etc, just pass - // them through. - default: - break; - - case tok::pp_include: - case tok::pp_import: - case tok::pp_include_next: { - // Save the 'include' token. - EmitToken(Tok); - // Lex the next token as an include string. - L.setParsingPreprocessorDirective(true); - L.LexIncludeFilename(Tok); - L.setParsingPreprocessorDirective(false); - assert(!Tok.isAtStartOfLine()); - if (Tok.is(tok::raw_identifier)) - PP.LookUpIdentifierInfo(Tok); - - break; - } - case tok::pp_if: - case tok::pp_ifdef: - case tok::pp_ifndef: { - // Add an entry for '#if' and friends. We initially set the target - // index to 0. This will get backpatched when we hit #endif. - PPStartCond.push_back(PPCond.size()); - PPCond.push_back(std::make_pair(HashOff, 0U)); - break; - } - case tok::pp_endif: { - // Add an entry for '#endif'. We set the target table index to itself. - // This will later be set to zero when emitting to the PTH file. We - // use 0 for uninitialized indices because that is easier to debug. - unsigned index = PPCond.size(); - // Backpatch the opening '#if' entry. - assert(!PPStartCond.empty()); - assert(PPCond.size() > PPStartCond.back()); - assert(PPCond[PPStartCond.back()].second == 0); - PPCond[PPStartCond.back()].second = index; - PPStartCond.pop_back(); - // Add the new entry to PPCond. - PPCond.push_back(std::make_pair(HashOff, index)); - EmitToken(Tok); - - // Some files have gibberish on the same line as '#endif'. - // Discard these tokens. - do - L.LexFromRawLexer(Tok); - while (Tok.isNot(tok::eof) && !Tok.isAtStartOfLine()); - // We have the next token in hand. - // Don't immediately lex the next one. - goto NextToken; - } - case tok::pp_elif: - case tok::pp_else: { - // Add an entry for #elif or #else. - // This serves as both a closing and opening of a conditional block. - // This means that its entry will get backpatched later. - unsigned index = PPCond.size(); - // Backpatch the previous '#if' entry. - assert(!PPStartCond.empty()); - assert(PPCond.size() > PPStartCond.back()); - assert(PPCond[PPStartCond.back()].second == 0); - PPCond[PPStartCond.back()].second = index; - PPStartCond.pop_back(); - // Now add '#elif' as a new block opening. - PPCond.push_back(std::make_pair(HashOff, 0U)); - PPStartCond.push_back(index); - break; - } - } - } - - EmitToken(Tok); - } - while (Tok.isNot(tok::eof)); - - assert(PPStartCond.empty() && "Error: imblanced preprocessor conditionals."); - - // Next write out PPCond. - Offset PPCondOff = (Offset) Out.tell(); - - // Write out the size of PPCond so that clients can identifer empty tables. - Emit32(PPCond.size()); - - for (unsigned i = 0, e = PPCond.size(); i!=e; ++i) { - Emit32(PPCond[i].first - TokenOff); - uint32_t x = PPCond[i].second; - assert(x != 0 && "PPCond entry not backpatched."); - // Emit zero for #endifs. This allows us to do checking when - // we read the PTH file back in. - Emit32(x == i ? 0 : x); - } - - return PTHEntry(TokenOff, PPCondOff); -} - -Offset PTHWriter::EmitCachedSpellings() { - // Write each cached strings to the PTH file. - Offset SpellingsOff = Out.tell(); - - for (std::vector*>::iterator - I = StrEntries.begin(), E = StrEntries.end(); I!=E; ++I) - EmitBuf((*I)->getKeyData(), (*I)->getKeyLength()+1 /*nul included*/); - - return SpellingsOff; -} - -static uint32_t swap32le(uint32_t X) { - return llvm::support::endian::byte_swap(X); -} - -static void pwrite32le(raw_pwrite_stream &OS, uint32_t Val, uint64_t &Off) { - uint32_t LEVal = swap32le(Val); - OS.pwrite(reinterpret_cast(&LEVal), 4, Off); - Off += 4; -} - -void PTHWriter::GeneratePTH(StringRef MainFile) { - // Generate the prologue. - Out << "cfe-pth" << '\0'; - Emit32(PTHManager::Version); - - // Leave 4 words for the prologue. - Offset PrologueOffset = Out.tell(); - for (unsigned i = 0; i < 4; ++i) - Emit32(0); - - // Write the name of the MainFile. - if (!MainFile.empty()) { - EmitString(MainFile); - } else { - // String with 0 bytes. - Emit16(0); - } - Emit8(0); - - // Iterate over all the files in SourceManager. Create a lexer - // for each file and cache the tokens. - SourceManager &SM = PP.getSourceManager(); - const LangOptions &LOpts = PP.getLangOpts(); - - for (SourceManager::fileinfo_iterator I = SM.fileinfo_begin(), - E = SM.fileinfo_end(); I != E; ++I) { - const SrcMgr::ContentCache &C = *I->second; - const FileEntry *FE = C.OrigEntry; - - // FIXME: Handle files with non-absolute paths. - if (llvm::sys::path::is_relative(FE->getName())) - continue; - - const llvm::MemoryBuffer *B = C.getBuffer(PP.getDiagnostics(), SM); - if (!B) continue; - - FileID FID = SM.createFileID(FE, SourceLocation(), SrcMgr::C_User); - const llvm::MemoryBuffer *FromFile = SM.getBuffer(FID); - Lexer L(FID, FromFile, SM, LOpts); - PM.insert(FE, LexTokens(L)); - } - - // Write out the identifier table. - const std::pair &IdTableOff = EmitIdentifierTable(); - - // Write out the cached strings table. - Offset SpellingOff = EmitCachedSpellings(); - - // Write out the file table. - Offset FileTableOff = EmitFileTable(); - - // Finally, write the prologue. - uint64_t Off = PrologueOffset; - pwrite32le(Out, IdTableOff.first, Off); - pwrite32le(Out, IdTableOff.second, Off); - pwrite32le(Out, FileTableOff, Off); - pwrite32le(Out, SpellingOff, Off); -} - -namespace { -/// StatListener - A simple "interpose" object used to monitor stat calls -/// invoked by FileManager while processing the original sources used -/// as input to PTH generation. StatListener populates the PTHWriter's -/// file map with stat information for directories as well as negative stats. -/// Stat information for files are populated elsewhere. -class StatListener : public FileSystemStatCache { - PTHMap &PM; -public: - StatListener(PTHMap &pm) : PM(pm) {} - ~StatListener() override {} - - LookupResult getStat(StringRef Path, FileData &Data, bool isFile, - std::unique_ptr *F, - llvm::vfs::FileSystem &FS) override { - LookupResult Result = statChained(Path, Data, isFile, F, FS); - - if (Result == CacheMissing) // Failed 'stat'. - PM.insert(PTHEntryKeyVariant(Path), PTHEntry()); - else if (Data.IsDirectory) { - // Only cache directories with absolute paths. - if (llvm::sys::path::is_relative(Path)) - return Result; - - PM.insert(PTHEntryKeyVariant(&Data, Path), PTHEntry()); - } - - return Result; - } -}; -} // end anonymous namespace - -void clang::CacheTokens(Preprocessor &PP, raw_pwrite_stream *OS) { - // Get the name of the main file. - const SourceManager &SrcMgr = PP.getSourceManager(); - const FileEntry *MainFile = SrcMgr.getFileEntryForID(SrcMgr.getMainFileID()); - SmallString<128> MainFilePath(MainFile->getName()); - - llvm::sys::fs::make_absolute(MainFilePath); - - // Create the PTHWriter. - PTHWriter PW(*OS, PP); - - // Install the 'stat' system call listener in the FileManager. - auto StatCacheOwner = llvm::make_unique(PW.getPM()); - StatListener *StatCache = StatCacheOwner.get(); - PP.getFileManager().addStatCache(std::move(StatCacheOwner), - /*AtBeginning=*/true); - - // Lex through the entire file. This will populate SourceManager with - // all of the header information. - Token Tok; - PP.EnterMainSourceFile(); - do { PP.Lex(Tok); } while (Tok.isNot(tok::eof)); - - // Generate the PTH file. - PP.getFileManager().removeStatCache(StatCache); - PW.GeneratePTH(MainFilePath.str()); -} - -//===----------------------------------------------------------------------===// - -namespace { -class PTHIdKey { -public: - const IdentifierInfo* II; - uint32_t FileOffset; -}; - -class PTHIdentifierTableTrait { -public: - typedef PTHIdKey* key_type; - typedef key_type key_type_ref; - - typedef uint32_t data_type; - typedef data_type data_type_ref; - - typedef unsigned hash_value_type; - typedef unsigned offset_type; - - static hash_value_type ComputeHash(PTHIdKey* key) { - return llvm::djbHash(key->II->getName()); - } - - static std::pair - EmitKeyDataLength(raw_ostream& Out, const PTHIdKey* key, uint32_t) { - using namespace llvm::support; - unsigned n = key->II->getLength() + 1; - endian::write(Out, n, little); - return std::make_pair(n, sizeof(uint32_t)); - } - - static void EmitKey(raw_ostream& Out, PTHIdKey* key, unsigned n) { - // Record the location of the key data. This is used when generating - // the mapping from persistent IDs to strings. - key->FileOffset = Out.tell(); - Out.write(key->II->getNameStart(), n); - } - - static void EmitData(raw_ostream& Out, PTHIdKey*, uint32_t pID, - unsigned) { - using namespace llvm::support; - endian::write(Out, pID, little); - } -}; -} // end anonymous namespace - -/// EmitIdentifierTable - Emits two tables to the PTH file. The first is -/// a hashtable mapping from identifier strings to persistent IDs. The second -/// is a straight table mapping from persistent IDs to string data (the -/// keys of the first table). -/// -std::pair PTHWriter::EmitIdentifierTable() { - // Build two maps: - // (1) an inverse map from persistent IDs -> (IdentifierInfo*,Offset) - // (2) a map from (IdentifierInfo*, Offset)* -> persistent IDs - - // Note that we use 'calloc', so all the bytes are 0. - PTHIdKey *IIDMap = static_cast( - llvm::safe_calloc(idcount, sizeof(PTHIdKey))); - - // Create the hashtable. - llvm::OnDiskChainedHashTableGenerator IIOffMap; - - // Generate mapping from persistent IDs -> IdentifierInfo*. - for (IDMap::iterator I = IM.begin(), E = IM.end(); I != E; ++I) { - // Decrement by 1 because we are using a vector for the lookup and - // 0 is reserved for NULL. - assert(I->second > 0); - assert(I->second-1 < idcount); - unsigned idx = I->second-1; - - // Store the mapping from persistent ID to IdentifierInfo* - IIDMap[idx].II = I->first; - - // Store the reverse mapping in a hashtable. - IIOffMap.insert(&IIDMap[idx], I->second); - } - - // Write out the inverse map first. This causes the PCIDKey entries to - // record PTH file offsets for the string data. This is used to write - // the second table. - Offset StringTableOffset = IIOffMap.Emit(Out); - - // Now emit the table mapping from persistent IDs to PTH file offsets. - Offset IDOff = Out.tell(); - Emit32(idcount); // Emit the number of identifiers. - for (unsigned i = 0 ; i < idcount; ++i) - Emit32(IIDMap[i].FileOffset); - - // Finally, release the inverse map. - free(IIDMap); - - return std::make_pair(IDOff, StringTableOffset); -} Index: lib/Frontend/ChainedIncludesSource.cpp =================================================================== --- lib/Frontend/ChainedIncludesSource.cpp +++ lib/Frontend/ChainedIncludesSource.cpp @@ -129,7 +129,6 @@ CInvok->getPreprocessorOpts().ChainedIncludes.clear(); CInvok->getPreprocessorOpts().ImplicitPCHInclude.clear(); - CInvok->getPreprocessorOpts().ImplicitPTHInclude.clear(); CInvok->getPreprocessorOpts().DisablePCHValidation = true; CInvok->getPreprocessorOpts().Includes.clear(); CInvok->getPreprocessorOpts().MacroIncludes.clear(); Index: lib/Frontend/CompilerInstance.cpp =================================================================== --- lib/Frontend/CompilerInstance.cpp +++ lib/Frontend/CompilerInstance.cpp @@ -30,7 +30,6 @@ #include "clang/Frontend/Utils.h" #include "clang/Frontend/VerifyDiagnosticConsumer.h" #include "clang/Lex/HeaderSearch.h" -#include "clang/Lex/PTHManager.h" #include "clang/Lex/Preprocessor.h" #include "clang/Lex/PreprocessorOptions.h" #include "clang/Sema/CodeCompleteConsumer.h" @@ -376,30 +375,18 @@ // The module manager holds a reference to the old preprocessor (if any). ModuleManager.reset(); - // Create a PTH manager if we are using some form of a token cache. - PTHManager *PTHMgr = nullptr; - if (!PPOpts.TokenCache.empty()) - PTHMgr = PTHManager::Create(PPOpts.TokenCache, getDiagnostics()); - // Create the Preprocessor. HeaderSearch *HeaderInfo = new HeaderSearch(getHeaderSearchOptsPtr(), getSourceManager(), getDiagnostics(), getLangOpts(), &getTarget()); PP = std::make_shared( Invocation->getPreprocessorOptsPtr(), getDiagnostics(), getLangOpts(), - getSourceManager(), getPCMCache(), *HeaderInfo, *this, PTHMgr, + getSourceManager(), getPCMCache(), *HeaderInfo, *this, + /*IdentifierInfoLookup=*/nullptr, /*OwnsHeaderSearch=*/true, TUKind); getTarget().adjust(getLangOpts()); PP->Initialize(getTarget(), getAuxTarget()); - // Note that this is different then passing PTHMgr to Preprocessor's ctor. - // That argument is used as the IdentifierInfoLookup argument to - // IdentifierTable's ctor. - if (PTHMgr) { - PTHMgr->setPreprocessor(&*PP); - PP->setPTHManager(PTHMgr); - } - if (PPOpts.DetailedRecord) PP->createPreprocessingRecord(); Index: lib/Frontend/CompilerInvocation.cpp =================================================================== --- lib/Frontend/CompilerInvocation.cpp +++ lib/Frontend/CompilerInvocation.cpp @@ -1619,8 +1619,6 @@ Opts.ProgramAction = frontend::GenerateHeaderModule; break; case OPT_emit_pch: Opts.ProgramAction = frontend::GeneratePCH; break; - case OPT_emit_pth: - Opts.ProgramAction = frontend::GeneratePTH; break; case OPT_init_only: Opts.ProgramAction = frontend::InitOnly; break; case OPT_fsyntax_only: @@ -3020,7 +3018,6 @@ case frontend::GenerateModuleInterface: case frontend::GenerateHeaderModule: case frontend::GeneratePCH: - case frontend::GeneratePTH: case frontend::ParseSyntaxOnly: case frontend::ModuleFileInfo: case frontend::VerifyPCH: @@ -3049,15 +3046,10 @@ DiagnosticsEngine &Diags, frontend::ActionKind Action) { Opts.ImplicitPCHInclude = Args.getLastArgValue(OPT_include_pch); - Opts.ImplicitPTHInclude = Args.getLastArgValue(OPT_include_pth); Opts.PCHWithHdrStop = Args.hasArg(OPT_pch_through_hdrstop_create) || Args.hasArg(OPT_pch_through_hdrstop_use); Opts.PCHWithHdrStopCreate = Args.hasArg(OPT_pch_through_hdrstop_create); Opts.PCHThroughHeader = Args.getLastArgValue(OPT_pch_through_header_EQ); - if (const Arg *A = Args.getLastArg(OPT_token_cache)) - Opts.TokenCache = A->getValue(); - else - Opts.TokenCache = Opts.ImplicitPTHInclude; Opts.UsePredefines = !Args.hasArg(OPT_undef); Opts.DetailedRecord = Args.hasArg(OPT_detailed_preprocessing_record); Opts.DisablePCHValidation = Args.hasArg(OPT_fno_validate_pch); Index: lib/Frontend/FrontendActions.cpp =================================================================== --- lib/Frontend/FrontendActions.cpp +++ lib/Frontend/FrontendActions.cpp @@ -750,16 +750,6 @@ } while (Tok.isNot(tok::eof)); } -void GeneratePTHAction::ExecuteAction() { - CompilerInstance &CI = getCompilerInstance(); - std::unique_ptr OS = - CI.createDefaultOutputFile(true, getCurrentFile()); - if (!OS) - return; - - CacheTokens(CI.getPreprocessor(), OS.get()); -} - void PreprocessOnlyAction::ExecuteAction() { Preprocessor &PP = getCompilerInstance().getPreprocessor(); Index: lib/Frontend/InitPreprocessor.cpp =================================================================== --- lib/Frontend/InitPreprocessor.cpp +++ lib/Frontend/InitPreprocessor.cpp @@ -21,7 +21,6 @@ #include "clang/Frontend/FrontendOptions.h" #include "clang/Frontend/Utils.h" #include "clang/Lex/HeaderSearch.h" -#include "clang/Lex/PTHManager.h" #include "clang/Lex/Preprocessor.h" #include "clang/Lex/PreprocessorOptions.h" #include "clang/Serialization/ASTReader.h" @@ -76,23 +75,6 @@ Builder.append("##"); // ##? } -/// AddImplicitIncludePTH - Add an implicit \#include using the original file -/// used to generate a PTH cache. -static void AddImplicitIncludePTH(MacroBuilder &Builder, Preprocessor &PP, - StringRef ImplicitIncludePTH) { - PTHManager *P = PP.getPTHManager(); - // Null check 'P' in the corner case where it couldn't be created. - const char *OriginalFile = P ? P->getOriginalSourceFile() : nullptr; - - if (!OriginalFile) { - PP.getDiagnostics().Report(diag::err_fe_pth_file_has_no_source_header) - << ImplicitIncludePTH; - return; - } - - AddImplicitInclude(Builder, OriginalFile); -} - /// Add an implicit \#include using the original file used to generate /// a PCH file. static void AddImplicitIncludePCH(MacroBuilder &Builder, Preprocessor &PP, @@ -1177,8 +1159,6 @@ if (!InitOpts.ImplicitPCHInclude.empty()) AddImplicitIncludePCH(Builder, PP, PCHContainerRdr, InitOpts.ImplicitPCHInclude); - if (!InitOpts.ImplicitPTHInclude.empty()) - AddImplicitIncludePTH(Builder, PP, InitOpts.ImplicitPTHInclude); // Process -include directives. for (unsigned i = 0, e = InitOpts.Includes.size(); i != e; ++i) { Index: lib/FrontendTool/ExecuteCompilerInvocation.cpp =================================================================== --- lib/FrontendTool/ExecuteCompilerInvocation.cpp +++ lib/FrontendTool/ExecuteCompilerInvocation.cpp @@ -65,7 +65,6 @@ case GenerateHeaderModule: return llvm::make_unique(); case GeneratePCH: return llvm::make_unique(); - case GeneratePTH: return llvm::make_unique(); case InitOnly: return llvm::make_unique(); case ParseSyntaxOnly: return llvm::make_unique(); case ModuleFileInfo: return llvm::make_unique(); Index: lib/Lex/CMakeLists.txt =================================================================== --- lib/Lex/CMakeLists.txt +++ lib/Lex/CMakeLists.txt @@ -17,7 +17,6 @@ PPExpressions.cpp PPLexerChange.cpp PPMacroExpansion.cpp - PTHLexer.cpp Pragma.cpp PreprocessingRecord.cpp Preprocessor.cpp Index: lib/Lex/PPDirectives.cpp =================================================================== --- lib/Lex/PPDirectives.cpp +++ lib/Lex/PPDirectives.cpp @@ -31,7 +31,6 @@ #include "clang/Lex/Pragma.h" #include "clang/Lex/Preprocessor.h" #include "clang/Lex/PreprocessorOptions.h" -#include "clang/Lex/PTHLexer.h" #include "clang/Lex/Token.h" #include "clang/Lex/VariadicMacroSupport.h" #include "llvm/ADT/ArrayRef.h" @@ -383,11 +382,6 @@ CurPPLexer->pushConditionalLevel(IfTokenLoc, /*isSkipping*/ false, FoundNonSkipPortion, FoundElse); - if (CurPTHLexer) { - PTHSkipExcludedConditionalBlock(); - return; - } - // Enter raw mode to disable identifier lookup (and thus macro expansion), // disabling warnings, etc. CurPPLexer->LexingRawMode = true; @@ -585,83 +579,6 @@ Tok.getLocation()); } -void Preprocessor::PTHSkipExcludedConditionalBlock() { - while (true) { - assert(CurPTHLexer); - assert(CurPTHLexer->LexingRawMode == false); - - // Skip to the next '#else', '#elif', or #endif. - if (CurPTHLexer->SkipBlock()) { - // We have reached an #endif. Both the '#' and 'endif' tokens - // have been consumed by the PTHLexer. Just pop off the condition level. - PPConditionalInfo CondInfo; - bool InCond = CurPTHLexer->popConditionalLevel(CondInfo); - (void)InCond; // Silence warning in no-asserts mode. - assert(!InCond && "Can't be skipping if not in a conditional!"); - break; - } - - // We have reached a '#else' or '#elif'. Lex the next token to get - // the directive flavor. - Token Tok; - LexUnexpandedToken(Tok); - - // We can actually look up the IdentifierInfo here since we aren't in - // raw mode. - tok::PPKeywordKind K = Tok.getIdentifierInfo()->getPPKeywordID(); - - if (K == tok::pp_else) { - // #else: Enter the else condition. We aren't in a nested condition - // since we skip those. We're always in the one matching the last - // blocked we skipped. - PPConditionalInfo &CondInfo = CurPTHLexer->peekConditionalLevel(); - // Note that we've seen a #else in this conditional. - CondInfo.FoundElse = true; - - // If the #if block wasn't entered then enter the #else block now. - if (!CondInfo.FoundNonSkip) { - CondInfo.FoundNonSkip = true; - - // Scan until the eod token. - CurPTHLexer->ParsingPreprocessorDirective = true; - DiscardUntilEndOfDirective(); - CurPTHLexer->ParsingPreprocessorDirective = false; - - break; - } - - // Otherwise skip this block. - continue; - } - - assert(K == tok::pp_elif); - PPConditionalInfo &CondInfo = CurPTHLexer->peekConditionalLevel(); - - // If this is a #elif with a #else before it, report the error. - if (CondInfo.FoundElse) - Diag(Tok, diag::pp_err_elif_after_else); - - // If this is in a skipping block or if we're already handled this #if - // block, don't bother parsing the condition. We just skip this block. - if (CondInfo.FoundNonSkip) - continue; - - // Evaluate the condition of the #elif. - IdentifierInfo *IfNDefMacro = nullptr; - CurPTHLexer->ParsingPreprocessorDirective = true; - bool ShouldEnter = EvaluateDirectiveExpression(IfNDefMacro).Conditional; - CurPTHLexer->ParsingPreprocessorDirective = false; - - // If this condition is true, enter it! - if (ShouldEnter) { - CondInfo.FoundNonSkip = true; - break; - } - - // Otherwise, skip this block and go to the next one. - } -} - Module *Preprocessor::getModuleForLocation(SourceLocation Loc) { if (!SourceMgr.isInMainFile(Loc)) { // Try to determine the module of the include directive. @@ -1387,10 +1304,6 @@ /// void Preprocessor::HandleUserDiagnosticDirective(Token &Tok, bool isWarning) { - // PTH doesn't emit #warning or #error directives. - if (CurPTHLexer) - return CurPTHLexer->DiscardToEndOfLine(); - // Read the rest of the line raw. We do this because we don't want macros // to be expanded and we don't require that the tokens be valid preprocessing // tokens. For example, this is allowed: "#warning ` 'foo". GCC does @@ -2007,14 +1920,10 @@ if (hadModuleLoaderFatalFailure()) { // With a fatal failure in the module loader, we abort parsing. Token &Result = IncludeTok; - if (CurLexer) { - Result.startToken(); - CurLexer->FormTokenWithChars(Result, CurLexer->BufferEnd, tok::eof); - CurLexer->cutOffLexing(); - } else { - assert(CurPTHLexer && "#include but no current lexer set!"); - CurPTHLexer->getEOF(Result); - } + assert(CurLexer && "#include but no current lexer set!"); + Result.startToken(); + CurLexer->FormTokenWithChars(Result, CurLexer->BufferEnd, tok::eof); + CurLexer->cutOffLexing(); } return; } Index: lib/Lex/PPLexerChange.cpp =================================================================== --- lib/Lex/PPLexerChange.cpp +++ lib/Lex/PPLexerChange.cpp @@ -19,7 +19,6 @@ #include "clang/Lex/HeaderSearch.h" #include "clang/Lex/LexDiagnostic.h" #include "clang/Lex/MacroInfo.h" -#include "clang/Lex/PTHManager.h" #include "llvm/ADT/StringSwitch.h" #include "llvm/Support/FileSystem.h" #include "llvm/Support/MemoryBuffer.h" @@ -76,13 +75,6 @@ if (MaxIncludeStackDepth < IncludeMacroStack.size()) MaxIncludeStackDepth = IncludeMacroStack.size(); - if (PTH) { - if (PTHLexer *PL = PTH->CreateLexer(FID)) { - EnterSourceFileWithPTH(PL, CurDir); - return false; - } - } - // Get the MemoryBuffer for this FID, if it fails, we fail. bool Invalid = false; const llvm::MemoryBuffer *InputFile = @@ -131,31 +123,6 @@ } } -/// EnterSourceFileWithPTH - Add a source file to the top of the include stack -/// and start getting tokens from it using the PTH cache. -void Preprocessor::EnterSourceFileWithPTH(PTHLexer *PL, - const DirectoryLookup *CurDir) { - - if (CurPPLexer || CurTokenLexer) - PushIncludeMacroStack(); - - CurDirLookup = CurDir; - CurPTHLexer.reset(PL); - CurPPLexer = CurPTHLexer.get(); - CurLexerSubmodule = nullptr; - if (CurLexerKind != CLK_LexAfterModuleImport) - CurLexerKind = CLK_PTHLexer; - - // Notify the client, if desired, that we are in a new source file. - if (Callbacks) { - FileID FID = CurPPLexer->getFileID(); - SourceLocation EnterLoc = SourceMgr.getLocForStartOfFile(FID); - SrcMgr::CharacteristicKind FileType = - SourceMgr.getFileCharacteristic(EnterLoc); - Callbacks->FileChanged(EnterLoc, PPCallbacks::EnterFile, FileType); - } -} - /// EnterMacro - Add a Macro to the top of the include stack and start lexing /// tokens from it instead of the current buffer. void Preprocessor::EnterMacro(Token &Tok, SourceLocation ILEnd, @@ -340,7 +307,6 @@ // If we have an unclosed module region from a pragma at the end of a // module, complain and close it now. - // FIXME: This is not correct if we are building a module from PTH. const bool LeavingSubmodule = CurLexer && CurLexerSubmodule; if ((LeavingSubmodule || IncludeMacroStack.empty()) && !BuildingSubmoduleStack.empty() && @@ -437,15 +403,10 @@ if (isCodeCompletionEnabled() && CurPPLexer && SourceMgr.getLocForStartOfFile(CurPPLexer->getFileID()) == CodeCompletionFileLoc) { - if (CurLexer) { - Result.startToken(); - CurLexer->FormTokenWithChars(Result, CurLexer->BufferEnd, tok::eof); - CurLexer.reset(); - } else { - assert(CurPTHLexer && "Got EOF but no current lexer set!"); - CurPTHLexer->getEOF(Result); - CurPTHLexer.reset(); - } + assert(CurLexer && "Got EOF but no current lexer set!"); + Result.startToken(); + CurLexer->FormTokenWithChars(Result, CurLexer->BufferEnd, tok::eof); + CurLexer.reset(); CurPPLexer = nullptr; recomputeCurLexerKind(); @@ -523,39 +484,34 @@ } // If this is the end of the main file, form an EOF token. - if (CurLexer) { - const char *EndPos = getCurLexerEndPos(); - Result.startToken(); - CurLexer->BufferPtr = EndPos; - CurLexer->FormTokenWithChars(Result, EndPos, tok::eof); - - if (isCodeCompletionEnabled()) { - // Inserting the code-completion point increases the source buffer by 1, - // but the main FileID was created before inserting the point. - // Compensate by reducing the EOF location by 1, otherwise the location - // will point to the next FileID. - // FIXME: This is hacky, the code-completion point should probably be - // inserted before the main FileID is created. - if (CurLexer->getFileLoc() == CodeCompletionFileLoc) - Result.setLocation(Result.getLocation().getLocWithOffset(-1)); - } - - if (creatingPCHWithThroughHeader() && !LeavingPCHThroughHeader) { - // Reached the end of the compilation without finding the through header. - Diag(CurLexer->getFileLoc(), diag::err_pp_through_header_not_seen) - << PPOpts->PCHThroughHeader << 0; - } - - if (!isIncrementalProcessingEnabled()) - // We're done with lexing. - CurLexer.reset(); - } else { - assert(CurPTHLexer && "Got EOF but no current lexer set!"); - CurPTHLexer->getEOF(Result); - CurPTHLexer.reset(); + assert(CurLexer && "Got EOF but no current lexer set!"); + const char *EndPos = getCurLexerEndPos(); + Result.startToken(); + CurLexer->BufferPtr = EndPos; + CurLexer->FormTokenWithChars(Result, EndPos, tok::eof); + + if (isCodeCompletionEnabled()) { + // Inserting the code-completion point increases the source buffer by 1, + // but the main FileID was created before inserting the point. + // Compensate by reducing the EOF location by 1, otherwise the location + // will point to the next FileID. + // FIXME: This is hacky, the code-completion point should probably be + // inserted before the main FileID is created. + if (CurLexer->getFileLoc() == CodeCompletionFileLoc) + Result.setLocation(Result.getLocation().getLocWithOffset(-1)); + } + + if (creatingPCHWithThroughHeader() && !LeavingPCHThroughHeader) { + // Reached the end of the compilation without finding the through header. + Diag(CurLexer->getFileLoc(), diag::err_pp_through_header_not_seen) + << PPOpts->PCHThroughHeader << 0; } if (!isIncrementalProcessingEnabled()) + // We're done with lexing. + CurLexer.reset(); + + if (!isIncrementalProcessingEnabled()) CurPPLexer = nullptr; if (TUKind == TU_Complete) { Index: lib/Lex/PPMacroExpansion.cpp =================================================================== --- lib/Lex/PPMacroExpansion.cpp +++ lib/Lex/PPMacroExpansion.cpp @@ -29,7 +29,6 @@ #include "clang/Lex/MacroInfo.h" #include "clang/Lex/Preprocessor.h" #include "clang/Lex/PreprocessorLexer.h" -#include "clang/Lex/PTHLexer.h" #include "clang/Lex/Token.h" #include "llvm/ADT/ArrayRef.h" #include "llvm/ADT/DenseMap.h" @@ -429,8 +428,6 @@ unsigned Val; if (CurLexer) Val = CurLexer->isNextPPTokenLParen(); - else if (CurPTHLexer) - Val = CurPTHLexer->isNextPPTokenLParen(); else Val = CurTokenLexer->isNextTokenLParen(); @@ -443,8 +440,6 @@ for (const IncludeStackInfo &Entry : llvm::reverse(IncludeMacroStack)) { if (Entry.TheLexer) Val = Entry.TheLexer->isNextPPTokenLParen(); - else if (Entry.ThePTHLexer) - Val = Entry.ThePTHLexer->isNextPPTokenLParen(); else Val = Entry.TheTokenLexer->isNextTokenLParen(); Index: lib/Lex/PTHLexer.cpp =================================================================== --- lib/Lex/PTHLexer.cpp +++ lib/Lex/PTHLexer.cpp @@ -1,748 +0,0 @@ -//===- PTHLexer.cpp - Lex from a token stream -----------------------------===// -// -// The LLVM Compiler Infrastructure -// -// This file is distributed under the University of Illinois Open Source -// License. See LICENSE.TXT for details. -// -//===----------------------------------------------------------------------===// -// -// This file implements the PTHLexer interface. -// -//===----------------------------------------------------------------------===// - -#include "clang/Lex/PTHLexer.h" -#include "clang/Basic/Diagnostic.h" -#include "clang/Basic/FileManager.h" -#include "clang/Basic/FileSystemStatCache.h" -#include "clang/Basic/IdentifierTable.h" -#include "clang/Basic/SourceManager.h" -#include "clang/Basic/TokenKinds.h" -#include "clang/Lex/LexDiagnostic.h" -#include "clang/Lex/PTHManager.h" -#include "clang/Lex/Preprocessor.h" -#include "clang/Lex/Token.h" -#include "llvm/ADT/STLExtras.h" -#include "llvm/ADT/StringRef.h" -#include "llvm/Support/DJB.h" -#include "llvm/Support/Endian.h" -#include "llvm/Support/ErrorOr.h" -#include "llvm/Support/FileSystem.h" -#include "llvm/Support/MemoryBuffer.h" -#include "llvm/Support/OnDiskHashTable.h" -#include -#include -#include -#include -#include -#include -#include - -using namespace clang; - -static const unsigned StoredTokenSize = 1 + 1 + 2 + 4 + 4; - -//===----------------------------------------------------------------------===// -// PTHLexer methods. -//===----------------------------------------------------------------------===// - -PTHLexer::PTHLexer(Preprocessor &PP, FileID FID, const unsigned char *D, - const unsigned char *ppcond, PTHManager &PM) - : PreprocessorLexer(&PP, FID), TokBuf(D), CurPtr(D), PPCond(ppcond), - CurPPCondPtr(ppcond), PTHMgr(PM) { - FileStartLoc = PP.getSourceManager().getLocForStartOfFile(FID); -} - -bool PTHLexer::Lex(Token& Tok) { - //===--------------------------------------==// - // Read the raw token data. - //===--------------------------------------==// - using namespace llvm::support; - - // Shadow CurPtr into an automatic variable. - const unsigned char *CurPtrShadow = CurPtr; - - // Read in the data for the token. - unsigned Word0 = endian::readNext(CurPtrShadow); - uint32_t IdentifierID = - endian::readNext(CurPtrShadow); - uint32_t FileOffset = - endian::readNext(CurPtrShadow); - - tok::TokenKind TKind = (tok::TokenKind) (Word0 & 0xFF); - Token::TokenFlags TFlags = (Token::TokenFlags) ((Word0 >> 8) & 0xFF); - uint32_t Len = Word0 >> 16; - - CurPtr = CurPtrShadow; - - //===--------------------------------------==// - // Construct the token itself. - //===--------------------------------------==// - - Tok.startToken(); - Tok.setKind(TKind); - Tok.setFlag(TFlags); - assert(!LexingRawMode); - Tok.setLocation(FileStartLoc.getLocWithOffset(FileOffset)); - Tok.setLength(Len); - - // Handle identifiers. - if (Tok.isLiteral()) { - Tok.setLiteralData((const char*) (PTHMgr.SpellingBase + IdentifierID)); - } - else if (IdentifierID) { - MIOpt.ReadToken(); - IdentifierInfo *II = PTHMgr.GetIdentifierInfo(IdentifierID-1); - - Tok.setIdentifierInfo(II); - - // Change the kind of this identifier to the appropriate token kind, e.g. - // turning "for" into a keyword. - Tok.setKind(II->getTokenID()); - - if (II->isHandleIdentifierCase()) - return PP->HandleIdentifier(Tok); - - return true; - } - - //===--------------------------------------==// - // Process the token. - //===--------------------------------------==// - if (TKind == tok::eof) { - // Save the end-of-file token. - EofToken = Tok; - - assert(!ParsingPreprocessorDirective); - assert(!LexingRawMode); - - return LexEndOfFile(Tok); - } - - if (TKind == tok::hash && Tok.isAtStartOfLine()) { - LastHashTokPtr = CurPtr - StoredTokenSize; - assert(!LexingRawMode); - PP->HandleDirective(Tok); - - return false; - } - - if (TKind == tok::eod) { - assert(ParsingPreprocessorDirective); - ParsingPreprocessorDirective = false; - return true; - } - - MIOpt.ReadToken(); - return true; -} - -bool PTHLexer::LexEndOfFile(Token &Result) { - // If we hit the end of the file while parsing a preprocessor directive, - // end the preprocessor directive first. The next token returned will - // then be the end of file. - if (ParsingPreprocessorDirective) { - ParsingPreprocessorDirective = false; // Done parsing the "line". - return true; // Have a token. - } - - assert(!LexingRawMode); - - // If we are in a #if directive, emit an error. - while (!ConditionalStack.empty()) { - if (PP->getCodeCompletionFileLoc() != FileStartLoc) - PP->Diag(ConditionalStack.back().IfLoc, - diag::err_pp_unterminated_conditional); - ConditionalStack.pop_back(); - } - - // Finally, let the preprocessor handle this. - return PP->HandleEndOfFile(Result); -} - -// FIXME: We can just grab the last token instead of storing a copy -// into EofToken. -void PTHLexer::getEOF(Token& Tok) { - assert(EofToken.is(tok::eof)); - Tok = EofToken; -} - -void PTHLexer::DiscardToEndOfLine() { - assert(ParsingPreprocessorDirective && ParsingFilename == false && - "Must be in a preprocessing directive!"); - - // We assume that if the preprocessor wishes to discard to the end of - // the line that it also means to end the current preprocessor directive. - ParsingPreprocessorDirective = false; - - // Skip tokens by only peeking at their token kind and the flags. - // We don't need to actually reconstruct full tokens from the token buffer. - // This saves some copies and it also reduces IdentifierInfo* lookup. - const unsigned char* p = CurPtr; - while (true) { - // Read the token kind. Are we at the end of the file? - tok::TokenKind x = (tok::TokenKind) (uint8_t) *p; - if (x == tok::eof) break; - - // Read the token flags. Are we at the start of the next line? - Token::TokenFlags y = (Token::TokenFlags) (uint8_t) p[1]; - if (y & Token::StartOfLine) break; - - // Skip to the next token. - p += StoredTokenSize; - } - - CurPtr = p; -} - -/// SkipBlock - Used by Preprocessor to skip the current conditional block. -bool PTHLexer::SkipBlock() { - using namespace llvm::support; - - assert(CurPPCondPtr && "No cached PP conditional information."); - assert(LastHashTokPtr && "No known '#' token."); - - const unsigned char *HashEntryI = nullptr; - uint32_t TableIdx; - - do { - // Read the token offset from the side-table. - uint32_t Offset = endian::readNext(CurPPCondPtr); - - // Read the target table index from the side-table. - TableIdx = endian::readNext(CurPPCondPtr); - - // Compute the actual memory address of the '#' token data for this entry. - HashEntryI = TokBuf + Offset; - - // Optimization: "Sibling jumping". #if...#else...#endif blocks can - // contain nested blocks. In the side-table we can jump over these - // nested blocks instead of doing a linear search if the next "sibling" - // entry is not at a location greater than LastHashTokPtr. - if (HashEntryI < LastHashTokPtr && TableIdx) { - // In the side-table we are still at an entry for a '#' token that - // is earlier than the last one we saw. Check if the location we would - // stride gets us closer. - const unsigned char* NextPPCondPtr = - PPCond + TableIdx*(sizeof(uint32_t)*2); - assert(NextPPCondPtr >= CurPPCondPtr); - // Read where we should jump to. - const unsigned char *HashEntryJ = - TokBuf + endian::readNext(NextPPCondPtr); - - if (HashEntryJ <= LastHashTokPtr) { - // Jump directly to the next entry in the side table. - HashEntryI = HashEntryJ; - TableIdx = endian::readNext(NextPPCondPtr); - CurPPCondPtr = NextPPCondPtr; - } - } - } - while (HashEntryI < LastHashTokPtr); - assert(HashEntryI == LastHashTokPtr && "No PP-cond entry found for '#'"); - assert(TableIdx && "No jumping from #endifs."); - - // Update our side-table iterator. - const unsigned char* NextPPCondPtr = PPCond + TableIdx*(sizeof(uint32_t)*2); - assert(NextPPCondPtr >= CurPPCondPtr); - CurPPCondPtr = NextPPCondPtr; - - // Read where we should jump to. - HashEntryI = - TokBuf + endian::readNext(NextPPCondPtr); - uint32_t NextIdx = endian::readNext(NextPPCondPtr); - - // By construction NextIdx will be zero if this is a #endif. This is useful - // to know to obviate lexing another token. - bool isEndif = NextIdx == 0; - - // This case can occur when we see something like this: - // - // #if ... - // /* a comment or nothing */ - // #elif - // - // If we are skipping the first #if block it will be the case that CurPtr - // already points 'elif'. Just return. - - if (CurPtr > HashEntryI) { - assert(CurPtr == HashEntryI + StoredTokenSize); - // Did we reach a #endif? If so, go ahead and consume that token as well. - if (isEndif) - CurPtr += StoredTokenSize * 2; - else - LastHashTokPtr = HashEntryI; - - return isEndif; - } - - // Otherwise, we need to advance. Update CurPtr to point to the '#' token. - CurPtr = HashEntryI; - - // Update the location of the last observed '#'. This is useful if we - // are skipping multiple blocks. - LastHashTokPtr = CurPtr; - - // Skip the '#' token. - assert(((tok::TokenKind)*CurPtr) == tok::hash); - CurPtr += StoredTokenSize; - - // Did we reach a #endif? If so, go ahead and consume that token as well. - if (isEndif) { - CurPtr += StoredTokenSize * 2; - } - - return isEndif; -} - -SourceLocation PTHLexer::getSourceLocation() { - // getSourceLocation is not on the hot path. It is used to get the location - // of the next token when transitioning back to this lexer when done - // handling a #included file. Just read the necessary data from the token - // data buffer to construct the SourceLocation object. - // NOTE: This is a virtual function; hence it is defined out-of-line. - using namespace llvm::support; - - const unsigned char *OffsetPtr = CurPtr + (StoredTokenSize - 4); - uint32_t Offset = endian::readNext(OffsetPtr); - return FileStartLoc.getLocWithOffset(Offset); -} - -//===----------------------------------------------------------------------===// -// PTH file lookup: map from strings to file data. -//===----------------------------------------------------------------------===// - -/// PTHFileLookup - This internal data structure is used by the PTHManager -/// to map from FileEntry objects managed by FileManager to offsets within -/// the PTH file. -namespace { - -class PTHFileData { - const uint32_t TokenOff; - const uint32_t PPCondOff; - -public: - PTHFileData(uint32_t tokenOff, uint32_t ppCondOff) - : TokenOff(tokenOff), PPCondOff(ppCondOff) {} - - uint32_t getTokenOffset() const { return TokenOff; } - uint32_t getPPCondOffset() const { return PPCondOff; } -}; - -class PTHFileLookupCommonTrait { -public: - using internal_key_type = std::pair; - using hash_value_type = unsigned; - using offset_type = unsigned; - - static hash_value_type ComputeHash(internal_key_type x) { - return llvm::djbHash(x.second); - } - - static std::pair - ReadKeyDataLength(const unsigned char*& d) { - using namespace llvm::support; - - unsigned keyLen = - (unsigned)endian::readNext(d); - unsigned dataLen = (unsigned) *(d++); - return std::make_pair(keyLen, dataLen); - } - - static internal_key_type ReadKey(const unsigned char* d, unsigned) { - unsigned char k = *(d++); // Read the entry kind. - return std::make_pair(k, (const char*) d); - } -}; - -} // namespace - -class PTHManager::PTHFileLookupTrait : public PTHFileLookupCommonTrait { -public: - using external_key_type = const FileEntry *; - using data_type = PTHFileData; - - static internal_key_type GetInternalKey(const FileEntry* FE) { - return std::make_pair((unsigned char) 0x1, FE->getName()); - } - - static bool EqualKey(internal_key_type a, internal_key_type b) { - return a.first == b.first && a.second == b.second; - } - - static PTHFileData ReadData(const internal_key_type& k, - const unsigned char* d, unsigned) { - using namespace llvm::support; - - assert(k.first == 0x1 && "Only file lookups can match!"); - uint32_t x = endian::readNext(d); - uint32_t y = endian::readNext(d); - return PTHFileData(x, y); - } -}; - -class PTHManager::PTHStringLookupTrait { -public: - using data_type = uint32_t; - using external_key_type = const std::pair; - using internal_key_type = external_key_type; - using hash_value_type = uint32_t; - using offset_type = unsigned; - - static bool EqualKey(const internal_key_type& a, - const internal_key_type& b) { - return (a.second == b.second) ? memcmp(a.first, b.first, a.second) == 0 - : false; - } - - static hash_value_type ComputeHash(const internal_key_type& a) { - return llvm::djbHash(StringRef(a.first, a.second)); - } - - // This hopefully will just get inlined and removed by the optimizer. - static const internal_key_type& - GetInternalKey(const external_key_type& x) { return x; } - - static std::pair - ReadKeyDataLength(const unsigned char*& d) { - using namespace llvm::support; - - return std::make_pair( - (unsigned)endian::readNext(d), - sizeof(uint32_t)); - } - - static std::pair - ReadKey(const unsigned char* d, unsigned n) { - assert(n >= 2 && d[n-1] == '\0'); - return std::make_pair((const char*) d, n-1); - } - - static uint32_t ReadData(const internal_key_type& k, const unsigned char* d, - unsigned) { - using namespace llvm::support; - - return endian::readNext(d); - } -}; - -//===----------------------------------------------------------------------===// -// PTHManager methods. -//===----------------------------------------------------------------------===// - -PTHManager::PTHManager( - std::unique_ptr buf, - std::unique_ptr fileLookup, const unsigned char *idDataTable, - std::unique_ptr perIDCache, - std::unique_ptr stringIdLookup, unsigned numIds, - const unsigned char *spellingBase, const char *originalSourceFile) - : Buf(std::move(buf)), PerIDCache(std::move(perIDCache)), - FileLookup(std::move(fileLookup)), IdDataTable(idDataTable), - StringIdLookup(std::move(stringIdLookup)), NumIds(numIds), - SpellingBase(spellingBase), OriginalSourceFile(originalSourceFile) {} - -PTHManager::~PTHManager() = default; - -static void InvalidPTH(DiagnosticsEngine &Diags, const char *Msg) { - Diags.Report(Diags.getCustomDiagID(DiagnosticsEngine::Error, "%0")) << Msg; -} - -PTHManager *PTHManager::Create(StringRef file, DiagnosticsEngine &Diags) { - // Memory map the PTH file. - llvm::ErrorOr> FileOrErr = - llvm::MemoryBuffer::getFile(file); - - if (!FileOrErr) { - // FIXME: Add ec.message() to this diag. - Diags.Report(diag::err_invalid_pth_file) << file; - return nullptr; - } - std::unique_ptr File = std::move(FileOrErr.get()); - - using namespace llvm::support; - - // Get the buffer ranges and check if there are at least three 32-bit - // words at the end of the file. - const unsigned char *BufBeg = (const unsigned char*)File->getBufferStart(); - const unsigned char *BufEnd = (const unsigned char*)File->getBufferEnd(); - - // Check the prologue of the file. - if ((BufEnd - BufBeg) < (signed)(sizeof("cfe-pth") + 4 + 4) || - memcmp(BufBeg, "cfe-pth", sizeof("cfe-pth")) != 0) { - Diags.Report(diag::err_invalid_pth_file) << file; - return nullptr; - } - - // Read the PTH version. - const unsigned char *p = BufBeg + (sizeof("cfe-pth")); - unsigned Version = endian::readNext(p); - - if (Version < PTHManager::Version) { - InvalidPTH(Diags, - Version < PTHManager::Version - ? "PTH file uses an older PTH format that is no longer supported" - : "PTH file uses a newer PTH format that cannot be read"); - return nullptr; - } - - // Compute the address of the index table at the end of the PTH file. - const unsigned char *PrologueOffset = p; - - if (PrologueOffset >= BufEnd) { - Diags.Report(diag::err_invalid_pth_file) << file; - return nullptr; - } - - // Construct the file lookup table. This will be used for mapping from - // FileEntry*'s to cached tokens. - const unsigned char* FileTableOffset = PrologueOffset + sizeof(uint32_t)*2; - const unsigned char *FileTable = - BufBeg + endian::readNext(FileTableOffset); - - if (!(FileTable > BufBeg && FileTable < BufEnd)) { - Diags.Report(diag::err_invalid_pth_file) << file; - return nullptr; // FIXME: Proper error diagnostic? - } - - std::unique_ptr FL(PTHFileLookup::Create(FileTable, BufBeg)); - - // Warn if the PTH file is empty. We still want to create a PTHManager - // as the PTH could be used with -include-pth. - if (FL->isEmpty()) - InvalidPTH(Diags, "PTH file contains no cached source data"); - - // Get the location of the table mapping from persistent ids to the - // data needed to reconstruct identifiers. - const unsigned char* IDTableOffset = PrologueOffset + sizeof(uint32_t)*0; - const unsigned char *IData = - BufBeg + endian::readNext(IDTableOffset); - - if (!(IData >= BufBeg && IData < BufEnd)) { - Diags.Report(diag::err_invalid_pth_file) << file; - return nullptr; - } - - // Get the location of the hashtable mapping between strings and - // persistent IDs. - const unsigned char* StringIdTableOffset = PrologueOffset + sizeof(uint32_t)*1; - const unsigned char *StringIdTable = - BufBeg + endian::readNext(StringIdTableOffset); - if (!(StringIdTable >= BufBeg && StringIdTable < BufEnd)) { - Diags.Report(diag::err_invalid_pth_file) << file; - return nullptr; - } - - std::unique_ptr SL( - PTHStringIdLookup::Create(StringIdTable, BufBeg)); - - // Get the location of the spelling cache. - const unsigned char* spellingBaseOffset = PrologueOffset + sizeof(uint32_t)*3; - const unsigned char *spellingBase = - BufBeg + endian::readNext(spellingBaseOffset); - if (!(spellingBase >= BufBeg && spellingBase < BufEnd)) { - Diags.Report(diag::err_invalid_pth_file) << file; - return nullptr; - } - - // Get the number of IdentifierInfos and pre-allocate the identifier cache. - uint32_t NumIds = endian::readNext(IData); - - // Pre-allocate the persistent ID -> IdentifierInfo* cache. We use calloc() - // so that we in the best case only zero out memory once when the OS returns - // us new pages. - std::unique_ptr PerIDCache; - - if (NumIds) { - PerIDCache.reset((IdentifierInfo **)calloc(NumIds, sizeof(PerIDCache[0]))); - if (!PerIDCache) { - InvalidPTH(Diags, "Could not allocate memory for processing PTH file"); - return nullptr; - } - } - - // Compute the address of the original source file. - const unsigned char* originalSourceBase = PrologueOffset + sizeof(uint32_t)*4; - unsigned len = - endian::readNext(originalSourceBase); - if (!len) originalSourceBase = nullptr; - - // Create the new PTHManager. - return new PTHManager(std::move(File), std::move(FL), IData, - std::move(PerIDCache), std::move(SL), NumIds, - spellingBase, (const char *)originalSourceBase); -} - -IdentifierInfo* PTHManager::LazilyCreateIdentifierInfo(unsigned PersistentID) { - using namespace llvm::support; - - // Look in the PTH file for the string data for the IdentifierInfo object. - const unsigned char* TableEntry = IdDataTable + sizeof(uint32_t)*PersistentID; - const unsigned char *IDData = - (const unsigned char *)Buf->getBufferStart() + - endian::readNext(TableEntry); - assert(IDData < (const unsigned char*)Buf->getBufferEnd()); - - // Allocate the object. - std::pair *Mem = - Alloc.Allocate>(); - - Mem->second = IDData; - assert(IDData[0] != '\0'); - IdentifierInfo *II = new ((void*) Mem) IdentifierInfo(); - - // Store the new IdentifierInfo in the cache. - PerIDCache[PersistentID] = II; - assert(II->getNameStart() && II->getNameStart()[0] != '\0'); - return II; -} - -IdentifierInfo* PTHManager::get(StringRef Name) { - // Double check our assumption that the last character isn't '\0'. - assert(Name.empty() || Name.back() != '\0'); - PTHStringIdLookup::iterator I = - StringIdLookup->find(std::make_pair(Name.data(), Name.size())); - if (I == StringIdLookup->end()) // No identifier found? - return nullptr; - - // Match found. Return the identifier! - assert(*I > 0); - return GetIdentifierInfo(*I-1); -} - -PTHLexer *PTHManager::CreateLexer(FileID FID) { - const FileEntry *FE = PP->getSourceManager().getFileEntryForID(FID); - if (!FE) - return nullptr; - - using namespace llvm::support; - - // Lookup the FileEntry object in our file lookup data structure. It will - // return a variant that indicates whether or not there is an offset within - // the PTH file that contains cached tokens. - PTHFileLookup::iterator I = FileLookup->find(FE); - - if (I == FileLookup->end()) // No tokens available? - return nullptr; - - const PTHFileData& FileData = *I; - - const unsigned char *BufStart = (const unsigned char *)Buf->getBufferStart(); - // Compute the offset of the token data within the buffer. - const unsigned char* data = BufStart + FileData.getTokenOffset(); - - // Get the location of pp-conditional table. - const unsigned char* ppcond = BufStart + FileData.getPPCondOffset(); - uint32_t Len = endian::readNext(ppcond); - if (Len == 0) ppcond = nullptr; - - assert(PP && "No preprocessor set yet!"); - return new PTHLexer(*PP, FID, data, ppcond, *this); -} - -//===----------------------------------------------------------------------===// -// 'stat' caching. -//===----------------------------------------------------------------------===// - -namespace { - -class PTHStatData { -public: - uint64_t Size; - time_t ModTime; - llvm::sys::fs::UniqueID UniqueID; - const bool HasData = false; - bool IsDirectory; - - PTHStatData() = default; - PTHStatData(uint64_t Size, time_t ModTime, llvm::sys::fs::UniqueID UniqueID, - bool IsDirectory) - : Size(Size), ModTime(ModTime), UniqueID(UniqueID), HasData(true), - IsDirectory(IsDirectory) {} -}; - -class PTHStatLookupTrait : public PTHFileLookupCommonTrait { -public: - using external_key_type = StringRef; // const char* - using data_type = PTHStatData; - - static internal_key_type GetInternalKey(StringRef path) { - // The key 'kind' doesn't matter here because it is ignored in EqualKey. - return std::make_pair((unsigned char) 0x0, path); - } - - static bool EqualKey(internal_key_type a, internal_key_type b) { - // When doing 'stat' lookups we don't care about the kind of 'a' and 'b', - // just the paths. - return a.second == b.second; - } - - static data_type ReadData(const internal_key_type& k, const unsigned char* d, - unsigned) { - if (k.first /* File or Directory */) { - bool IsDirectory = true; - if (k.first == 0x1 /* File */) { - IsDirectory = false; - d += 4 * 2; // Skip the first 2 words. - } - - using namespace llvm::support; - - uint64_t File = endian::readNext(d); - uint64_t Device = endian::readNext(d); - llvm::sys::fs::UniqueID UniqueID(Device, File); - time_t ModTime = endian::readNext(d); - uint64_t Size = endian::readNext(d); - return data_type(Size, ModTime, UniqueID, IsDirectory); - } - - // Negative stat. Don't read anything. - return data_type(); - } -}; - -} // namespace - -namespace clang { - -class PTHStatCache : public FileSystemStatCache { - using CacheTy = llvm::OnDiskChainedHashTable; - - CacheTy Cache; - -public: - PTHStatCache(PTHManager::PTHFileLookup &FL) - : Cache(FL.getNumBuckets(), FL.getNumEntries(), FL.getBuckets(), - FL.getBase()) {} - - LookupResult getStat(StringRef Path, FileData &Data, bool isFile, - std::unique_ptr *F, - llvm::vfs::FileSystem &FS) override { - // Do the lookup for the file's data in the PTH file. - CacheTy::iterator I = Cache.find(Path); - - // If we don't get a hit in the PTH file just forward to 'stat'. - if (I == Cache.end()) - return statChained(Path, Data, isFile, F, FS); - - const PTHStatData &D = *I; - - if (!D.HasData) - return CacheMissing; - - Data.Name = Path; - Data.Size = D.Size; - Data.ModTime = D.ModTime; - Data.UniqueID = D.UniqueID; - Data.IsDirectory = D.IsDirectory; - Data.IsNamedPipe = false; - Data.InPCH = true; - - return CacheExists; - } -}; - -} // namespace clang - -std::unique_ptr PTHManager::createStatCache() { - return llvm::make_unique(*FileLookup); -} Index: lib/Lex/Pragma.cpp =================================================================== --- lib/Lex/Pragma.cpp +++ lib/Lex/Pragma.cpp @@ -31,7 +31,6 @@ #include "clang/Lex/PPCallbacks.h" #include "clang/Lex/Preprocessor.h" #include "clang/Lex/PreprocessorLexer.h" -#include "clang/Lex/PTHLexer.h" #include "clang/Lex/Token.h" #include "clang/Lex/TokenLexer.h" #include "llvm/ADT/ArrayRef.h" @@ -404,10 +403,7 @@ void Preprocessor::HandlePragmaMark() { assert(CurPPLexer && "No current lexer?"); - if (CurLexer) - CurLexer->ReadToEndOfLine(); - else - CurPTHLexer->DiscardToEndOfLine(); + CurLexer->ReadToEndOfLine(); } /// HandlePragmaPoison - Handle \#pragma GCC poison. PoisonTok is the 'poison'. @@ -810,12 +806,6 @@ DiscardUntilEndOfDirective(); } - if (CurPTHLexer) { - // FIXME: Support this somehow? - Diag(Loc, diag::err_pp_module_build_pth); - return; - } - CurLexer->LexingRawMode = true; auto TryConsumeIdentifier = [&](StringRef Ident) -> bool { Index: lib/Lex/Preprocessor.cpp =================================================================== --- lib/Lex/Preprocessor.cpp +++ lib/Lex/Preprocessor.cpp @@ -44,8 +44,6 @@ #include "clang/Lex/MacroArgs.h" #include "clang/Lex/MacroInfo.h" #include "clang/Lex/ModuleLoader.h" -#include "clang/Lex/PTHLexer.h" -#include "clang/Lex/PTHManager.h" #include "clang/Lex/Pragma.h" #include "clang/Lex/PreprocessingRecord.h" #include "clang/Lex/PreprocessorLexer.h" @@ -224,11 +222,6 @@ PragmaHandlers = std::move(PragmaHandlersBackup); } -void Preprocessor::setPTHManager(PTHManager* pm) { - PTH.reset(pm); - FileMgr.addStatCache(PTH->createStatCache()); -} - void Preprocessor::DumpToken(const Token &Tok, bool DumpFlags) const { llvm::errs() << tok::getTokenName(Tok.getKind()) << " '" << getSpelling(Tok) << "'"; @@ -379,8 +372,6 @@ void Preprocessor::recomputeCurLexerKind() { if (CurLexer) CurLexerKind = CLK_Lexer; - else if (CurPTHLexer) - CurLexerKind = CLK_PTHLexer; else if (CurTokenLexer) CurLexerKind = CLK_TokenLexer; else @@ -877,9 +868,6 @@ case CLK_Lexer: ReturnedToken = CurLexer->Lex(Result); break; - case CLK_PTHLexer: - ReturnedToken = CurPTHLexer->Lex(Result); - break; case CLK_TokenLexer: ReturnedToken = CurTokenLexer->Lex(Result); break; Index: lib/Serialization/ASTReader.cpp =================================================================== --- lib/Serialization/ASTReader.cpp +++ lib/Serialization/ASTReader.cpp @@ -5388,7 +5388,6 @@ PPOpts.UsePredefines = Record[Idx++]; PPOpts.DetailedRecord = Record[Idx++]; PPOpts.ImplicitPCHInclude = ReadString(Record, Idx); - PPOpts.ImplicitPTHInclude = ReadString(Record, Idx); PPOpts.ObjCXXARCStandardLibrary = static_cast(Record[Idx++]); SuggestedPredefines.clear(); Index: lib/Serialization/ASTWriter.cpp =================================================================== --- lib/Serialization/ASTWriter.cpp +++ lib/Serialization/ASTWriter.cpp @@ -1695,7 +1695,6 @@ // Detailed record is important since it is used for the module cache hash. Record.push_back(PPOpts.DetailedRecord); AddString(PPOpts.ImplicitPCHInclude, Record); - AddString(PPOpts.ImplicitPTHInclude, Record); Record.push_back(static_cast(PPOpts.ObjCXXARCStandardLibrary)); Stream.EmitRecord(PREPROCESSOR_OPTIONS, Record); Index: test/Driver/pth.c =================================================================== --- test/Driver/pth.c +++ test/Driver/pth.c @@ -1,12 +0,0 @@ -// Test transparent PTH support. - -// RUN: %clang -no-canonical-prefixes -ccc-pch-is-pth -x c-header %s -o %t.h.pth -### 2> %t.log -// RUN: FileCheck -check-prefix CHECK1 -input-file %t.log %s - -// CHECK1: "{{.*[/\\]}}clang{{.*}}" "-cc1" {{.*}} "-o" "{{.*}}.h.pth" "-x" "c-header" "{{.*}}pth.c" - -// RUN: touch %t.h.pth -// RUN: %clang -no-canonical-prefixes -ccc-pch-is-pth -E -include %t.h %s -### 2> %t.log -// RUN: FileCheck -check-prefix CHECK2 -input-file %t.log %s - -// CHECK2: "{{.*[/\\]}}clang{{.*}}" "-cc1" {{.*}}"-include-pth" "{{.*}}.h.pth" {{.*}}"-x" "c" "{{.*}}pth.c" Index: test/PCH/emit-pth.c =================================================================== --- test/PCH/emit-pth.c +++ test/PCH/emit-pth.c @@ -1,7 +0,0 @@ -// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-pth -o %t1 %s -// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-pth -o - %s > %t2 -// RUN: cmp %t1 %t2 -// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-pth -o - %s | \ -// RUN: FileCheck %s - -// CHECK: cfe-pth Index: test/PCH/pth.c =================================================================== --- test/PCH/pth.c +++ test/PCH/pth.c @@ -1,7 +0,0 @@ -// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-pth -o %t %S/pth.h -// RUN: not %clang_cc1 -triple i386-unknown-unknown -include-pth %t -fsyntax-only %s 2>&1 | FileCheck %s - -#error This is the only diagnostic - -// CHECK: This is the only diagnostic -// CHECK: 1 error generated. Index: test/Preprocessor/include-pth.c =================================================================== --- test/Preprocessor/include-pth.c +++ test/Preprocessor/include-pth.c @@ -1,3 +0,0 @@ -// RUN: %clang_cc1 -emit-pth %s -o %t -// RUN: %clang_cc1 -include-pth %t %s -E | grep 'file_to_include' | count 2 -#include "file_to_include.h"