[ASTWriter] Provide capability to output a PCM/PCH file that does not write out information about its output path

[akyrtzi]

Summary

This is useful to enable sharing of the same PCH file even when it's intended for a different output path.

The only information this option disables writing is for ORIGINAL_PCH_DIR record which is treated as optional and (when present) used as fallback for resolving input file paths relative to it.

[benlangmuir]

Is the functionality provided by ORIGINAL_PCH_DIR still useful for making it possible to move a PCH and its referenced headers? It's not completely clear to me when this feature is used in practice. It would be nice to remove it or change the default behaviour if possible, rather than require a new option, but I'm open to this approach if we think we can't change the default.

[akyrtzi]

@v.g.vassilev is the functionality of "write ORIGINAL_PCH_DIR and resolve headers relative to it if PCH file and headers moved together" used by Cling?

[akyrtzi]

Also pinging @rmaz who made a related change, is this used by Facebook?

[v.g.vassilev]

In D130710#3685470, @akyrtzi wrote:

@v.g.vassilev is the functionality of "write ORIGINAL_PCH_DIR and resolve headers relative to it if PCH file and headers moved together" used by Cling?

We currently use -fmodules-embed-all-files which zips all header files in a blob in the PCH/PCM that gives us flexibility to not require the headers to be present at a specific location. We have mostly moved to PCMs and the PCH is of less interest to us. Does that patch help making the implicitly built module files easier to relocate?

[akyrtzi]

In D130710#3685509, @v.g.vassilev wrote:

In D130710#3685470, @akyrtzi wrote:

@v.g.vassilev is the functionality of "write ORIGINAL_PCH_DIR and resolve headers relative to it if PCH file and headers moved together" used by Cling?

We currently use -fmodules-embed-all-files which zips all header files in a blob in the PCH/PCM that gives us flexibility to not require the headers to be present at a specific location. We have mostly moved to PCMs and the PCH is of less interest to us. Does that patch help making the implicitly built module files easier to relocate?

Not sure whether ORIGINAL_PCH_DIR relates to implicitly built module files or not, I'm wondering whether ORIGINAL_PCH_DIR is still useful and, if I understand correctly, it's not useful for Cling since it embeds the header file contents, right?

[akyrtzi]

Add FIXME comment to consider either removing ORIGINAL_PCH_DIR or changing the default.

[akyrtzi]

In D130710#3685436, @benlangmuir wrote:

Is the functionality provided by ORIGINAL_PCH_DIR still useful for making it possible to move a PCH and its referenced headers? It's not completely clear to me when this feature is used in practice. It would be nice to remove it or change the default behaviour if possible, rather than require a new option, but I'm open to this approach if we think we can't change the default.

Given that there was a recent change related to ORIGINAL_PCH_DIR, I'm reluctant to change the default at this point, I added a FIXME for following up to see if we can remove it or change the default later on.

[rmaz]

In D130710#3688338, @akyrtzi wrote:

In D130710#3685436, @benlangmuir wrote:

Is the functionality provided by ORIGINAL_PCH_DIR still useful for making it possible to move a PCH and its referenced headers? It's not completely clear to me when this feature is used in practice. It would be nice to remove it or change the default behaviour if possible, rather than require a new option, but I'm open to this approach if we think we can't change the default.

Given that there was a recent change related to ORIGINAL_PCH_DIR, I'm reluctant to change the default at this point, I added a FIXME for following up to see if we can remove it or change the default later on.

We don't use this functionality, no. My change was made for the same reason: to be able to relocate and cache pcm files when used in conjunction with -fmodule-file-home-is-cwd, removing the field entirely would be preferable from our perspective.

clang/include/clang/Driver/Options.td

def pic_is_pie : Flag<["-"], "pic-is-pie">,
   MarshallingInfoFlag<LangOpts<"PIE">>;
 def fhalf_no_semantic_interposition : Flag<["-"], "fhalf-no-semantic-interposition">,
   HelpText<"Like -fno-semantic-interposition but don't use local aliases">,
   MarshallingInfoFlag<LangOpts<"HalfNoSemanticInterposition">>;
 def fno_validate_pch : Flag<["-"], "fno-validate-pch">,
   HelpText<"Disable validation of precompiled headers">,
   MarshallingInfoFlag<PreprocessorOpts<"DisablePCHOrModuleValidation">, "DisableValidationForModuleKind::None">,
   Normalizer<"makeFlagToValueNormalizer(DisableValidationForModuleKind::All)">;
+def fpcm_output_path_independent : Flag<["-"], "fpcm-output-path-independent">,
+  HelpText<"Output a PCM/PCH file that does not write out information about its output path">,
+  MarshallingInfoFlag<FrontendOpts<"OutputPathIndependentPCM">>;
 def fallow_pcm_with_errors : Flag<["-"], "fallow-pcm-with-compiler-errors">,
   HelpText<"Accept a PCM file that was created with compiler errors">,
   MarshallingInfoFlag<FrontendOpts<"AllowPCMWithCompilerErrors">>;
 def fallow_pch_with_errors : Flag<["-"], "fallow-pch-with-compiler-errors">,
   HelpText<"Accept a PCH file that was created with compiler errors">,
   MarshallingInfoFlag<PreprocessorOpts<"AllowPCHWithCompilerErrors">>,
   ImpliedByAnyOf<[fallow_pcm_with_errors.KeyPath]>;
 def fallow_pch_with_different_modules_cache_path :

clang/include/clang/Frontend/FrontendOptions.h

public:
   unsigned IncludeTimestamps : 1;
 
   /// Should a temporary file be used during compilation.
   unsigned UseTemporary : 1;
 
   /// When using -emit-module, treat the modulemap as a system module.
   unsigned IsSystemModule : 1;
 
+  /// Output a PCM/PCH file that does not write out information about its output
+  /// path.
+  ///
+  /// FIXME: This only controls whether \p ORIGINAL_PCH_DIR record is written
+  /// out or not. Consider either removing that record entirely if it's no
+  /// longer relevant or switching the default to not write it unless an option
+  /// is set to true.
+  unsigned OutputPathIndependentPCM : 1;
+
   /// Output (and read) PCM files regardless of compiler errors.
   unsigned AllowPCMWithCompilerErrors : 1;
 
   CodeCompleteOptions CodeCompleteOpts;
 
   /// Specifies the output format of the AST.
   ASTDumpOutputFormat ASTDumpFormat = ADOF_Default;
 
FrontendOptions()
         ShowStats(false), TimeTrace(false), ShowVersion(false),
         FixWhatYouCan(false), FixOnlyWarnings(false), FixAndRecompile(false),
         FixToTemporaries(false), ARCMTMigrateEmitARCErrors(false),
         SkipFunctionBodies(false), UseGlobalModuleIndex(true),
         GenerateGlobalModuleIndex(true), ASTDumpDecls(false),
         ASTDumpLookups(false), BuildingImplicitModule(false),
         BuildingImplicitModuleUsesLock(true), ModulesEmbedAllFiles(false),
         IncludeTimestamps(true), UseTemporary(true),
-        AllowPCMWithCompilerErrors(false), TimeTraceGranularity(500) {}
+        OutputPathIndependentPCM(false), AllowPCMWithCompilerErrors(false),
+        TimeTraceGranularity(500) {}
 
   /// getInputKindForExtension - Return the appropriate input kind for a file
   /// extension. For example, "c" would return Language::C.
   ///
   /// \return The input kind for the extension, or Language::Unknown if the
   /// extension is not recognized.
   static InputKind getInputKindForExtension(StringRef Extension);
 };
 
 } // namespace clang
 
 #endif // LLVM_CLANG_FRONTEND_FRONTENDOPTIONS_H

clang/include/clang/Serialization/ASTWriter.h

public:
   /// are relative to the given system root. If we're writing a module, its
   /// build directory will be used in preference to this if both are available.
   ///
   /// \return the module signature, which eventually will be a hash of
   /// the module but currently is merely a random 32-bit number.
   ASTFileSignature WriteAST(Sema &SemaRef, StringRef OutputFile,
                             Module *WritingModule, StringRef isysroot,
                             bool hasErrors = false,
-                            bool ShouldCacheASTInMemory = false);
+                            bool ShouldCacheASTInMemory = false,
+                            bool OutputPathIndependent = false);
 
   /// Emit a token.
   void AddToken(const Token &Tok, RecordDataImpl &Record);
 
   /// Emit a AlignPackInfo.
   void AddAlignPackInfo(const Sema::AlignPackInfo &Info,
                         RecordDataImpl &Record);
 
class PCHGenerator : public SemaConsumer {
   std::string OutputFile;
   std::string isysroot;
   Sema *SemaPtr;
   std::shared_ptr<PCHBuffer> Buffer;
   llvm::BitstreamWriter Stream;
   ASTWriter Writer;
   bool AllowASTWithErrors;
   bool ShouldCacheASTInMemory;
+  bool OutputPathIndependent;
 
 protected:
   ASTWriter &getWriter() { return Writer; }
   const ASTWriter &getWriter() const { return Writer; }
   SmallVectorImpl<char> &getPCH() const { return Buffer->Data; }
 
 public:
   PCHGenerator(const Preprocessor &PP, InMemoryModuleCache &ModuleCache,
                StringRef OutputFile, StringRef isysroot,
                std::shared_ptr<PCHBuffer> Buffer,
                ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
                bool AllowASTWithErrors = false, bool IncludeTimestamps = true,
-               bool ShouldCacheASTInMemory = false);
+               bool ShouldCacheASTInMemory = false,
+               bool OutputPathIndependent = false);
   ~PCHGenerator() override;
 
   void InitializeSema(Sema &S) override { SemaPtr = &S; }
   void HandleTranslationUnit(ASTContext &Ctx) override;
   ASTMutationListener *GetASTMutationListener() override;
   ASTDeserializationListener *GetASTDeserializationListener() override;
   bool hasEmittedPCH() const { return Buffer->IsComplete; }
 };
 
 } // namespace clang
 
 #endif // LLVM_CLANG_SERIALIZATION_ASTWRITER_H

clang/lib/Frontend/FrontendActions.cpp

GeneratePCHAction::CreateASTConsumer(CompilerInstance &CI, StringRef InFile) {
 
   const auto &FrontendOpts = CI.getFrontendOpts();
   auto Buffer = std::make_shared<PCHBuffer>();
   std::vector<std::unique_ptr<ASTConsumer>> Consumers;
   Consumers.push_back(std::make_unique<PCHGenerator>(
       CI.getPreprocessor(), CI.getModuleCache(), OutputFile, Sysroot, Buffer,
       FrontendOpts.ModuleFileExtensions,
       CI.getPreprocessorOpts().AllowPCHWithCompilerErrors,
-      FrontendOpts.IncludeTimestamps, +CI.getLangOpts().CacheGeneratedPCH));
+      FrontendOpts.IncludeTimestamps, +CI.getLangOpts().CacheGeneratedPCH,
+      FrontendOpts.OutputPathIndependentPCM));
   Consumers.push_back(CI.getPCHContainerWriter().CreatePCHContainerGenerator(
       CI, std::string(InFile), OutputFile, std::move(OS), Buffer));
 
   return std::make_unique<MultiplexConsumer>(std::move(Consumers));
 }
 
 bool GeneratePCHAction::ComputeASTConsumerArguments(CompilerInstance &CI,
                                                     std::string &Sysroot) {
GenerateModuleAction::CreateASTConsumer(CompilerInstance &CI,
   Consumers.push_back(std::make_unique<PCHGenerator>(
       CI.getPreprocessor(), CI.getModuleCache(), OutputFile, Sysroot, Buffer,
       CI.getFrontendOpts().ModuleFileExtensions,
       /*AllowASTWithErrors=*/
       +CI.getFrontendOpts().AllowPCMWithCompilerErrors,
       /*IncludeTimestamps=*/
       +CI.getFrontendOpts().BuildingImplicitModule,
       /*ShouldCacheASTInMemory=*/
-      +CI.getFrontendOpts().BuildingImplicitModule));
+      +CI.getFrontendOpts().BuildingImplicitModule,
+      /*OutputPathIndependent=*/
+      +CI.getFrontendOpts().OutputPathIndependentPCM));
   Consumers.push_back(CI.getPCHContainerWriter().CreatePCHContainerGenerator(
       CI, std::string(InFile), OutputFile, std::move(OS), Buffer));
   return std::make_unique<MultiplexConsumer>(std::move(Consumers));
 }
 
 bool GenerateModuleAction::shouldEraseOutputFiles() {
   return !getCompilerInstance().getFrontendOpts().AllowPCMWithCompilerErrors &&
          ASTFrontendAction::shouldEraseOutputFiles();

clang/lib/Serialization/ASTWriter.cpp

 
 time_t ASTWriter::getTimestampForOutput(const FileEntry *E) const {
   return IncludeTimestamps ? E->getModificationTime() : 0;
 }
 
 ASTFileSignature ASTWriter::WriteAST(Sema &SemaRef, StringRef OutputFile,
                                      Module *WritingModule, StringRef isysroot,
                                      bool hasErrors,
-                                     bool ShouldCacheASTInMemory) {
+                                     bool ShouldCacheASTInMemory,
+                                     bool OutputPathIndependent) {
   WritingAST = true;
 
   ASTHasCompilerErrors = hasErrors;
 
   // Emit the file header.
   Stream.Emit((unsigned)'C', 8);
   Stream.Emit((unsigned)'P', 8);
   Stream.Emit((unsigned)'C', 8);
   Stream.Emit((unsigned)'H', 8);
 
   WriteBlockInfoBlock();
 
   Context = &SemaRef.Context;
   PP = &SemaRef.PP;
   this->WritingModule = WritingModule;
-  ASTFileSignature Signature =
-      WriteASTCore(SemaRef, isysroot, OutputFile, WritingModule);
+  ASTFileSignature Signature = WriteASTCore(
+      SemaRef, isysroot, OutputPathIndependent ? StringRef() : OutputFile,
+      WritingModule);
   Context = nullptr;
   PP = nullptr;
   this->WritingModule = nullptr;
   this->BaseDirectory.clear();
 
   WritingAST = false;
   if (ShouldCacheASTInMemory) {
     // Construct MemoryBuffer and update buffer manager.

clang/lib/Serialization/GeneratePCH.cpp

 
 using namespace clang;
 
 PCHGenerator::PCHGenerator(
     const Preprocessor &PP, InMemoryModuleCache &ModuleCache,
     StringRef OutputFile, StringRef isysroot, std::shared_ptr<PCHBuffer> Buffer,
     ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
     bool AllowASTWithErrors, bool IncludeTimestamps,
-    bool ShouldCacheASTInMemory)
+    bool ShouldCacheASTInMemory, bool OutputPathIndependent)
     : PP(PP), OutputFile(OutputFile), isysroot(isysroot.str()),
       SemaPtr(nullptr), Buffer(std::move(Buffer)), Stream(this->Buffer->Data),
       Writer(Stream, this->Buffer->Data, ModuleCache, Extensions,
              IncludeTimestamps),
       AllowASTWithErrors(AllowASTWithErrors),
-      ShouldCacheASTInMemory(ShouldCacheASTInMemory) {
+      ShouldCacheASTInMemory(ShouldCacheASTInMemory),
+      OutputPathIndependent(OutputPathIndependent) {
   this->Buffer->IsComplete = false;
 }
 
 PCHGenerator::~PCHGenerator() {
 }
 
 void PCHGenerator::HandleTranslationUnit(ASTContext &Ctx) {
   // Don't create a PCH if there were fatal failures during module loading.
void PCHGenerator::HandleTranslationUnit(ASTContext &Ctx) {
 
   // Emit the PCH file to the Buffer.
   assert(SemaPtr && "No Sema?");
   Buffer->Signature =
       Writer.WriteAST(*SemaPtr, OutputFile, Module, isysroot,
                       // For serialization we are lenient if the errors were
                       // only warn-as-error kind.
                       PP.getDiagnostics().hasUncompilableErrorOccurred(),
-                      ShouldCacheASTInMemory);
+                      ShouldCacheASTInMemory, OutputPathIndependent);
 
   Buffer->IsComplete = true;
 }
 
 ASTMutationListener *PCHGenerator::GetASTMutationListener() {
   return &Writer;
 }
 
 ASTDeserializationListener *PCHGenerator::GetASTDeserializationListener() {
   return &Writer;
 }

clang/test/PCH/pch-output-path-independent.c

+// RUN: rm -rf %t && mkdir -p %t/a %t/b
+
+// RUN: %clang_cc1 -triple x86_64-apple-macos11 -emit-pch %s -o %t/a/t1.pch -fpcm-output-path-independent
+// RUN: %clang_cc1 -triple x86_64-apple-macos11 -emit-pch %s -o %t/b/t2.pch -fpcm-output-path-independent
+
+// RUN: diff %t/a/t1.pch %t/b/t2.pch