Make AST reading work better with LLVM_APPEND_VC_REV=NO

[thakis]

Summary

With LLVM_APPEND_VC_REV=NO, Modules/merge-lifetime-extended-temporary.cpp
would fail if it ran before a0f50d731639350c7a7 (which changed
the serialization format) and then after, for these reasons:

1. With LLVM_APPEND_VC_REV=NO, the module hash before and after the change was the same.

2. Modules/merge-lifetime-extended-temporary.cpp is the only test we have that uses -fmodule-cache-path=%t that a) actually writes to the cache path b) doesn't do rm -rf %t at the top of the test

So the old run would write a module file, and then the new run would
try to load it, but the serialized format changed.

Do several things to fix this:

1. Include clang::serialization::VERSION_MAJOR/VERSION_MINOR in the module hash, so that when the AST format changes (...and we remember to bump these), we use a different module cache dir.
2. Bump VERSION_MAJOR, since a0f50d731639350c7a7 changed the on-disk format in a way that a gch file written before that change can't be read after that change.
3. Add rm -rf %t to all tests that pass -fmodule-cache-path=%t. This is unneccessary from a correctness PoV after 1 and 2, but makes it so that we don't amass many cache dirs over time. (Arguably, it also makes it so that the test suite doesn't catch when we change the serialization format but don't bump clang::serialization::VERSION_MAJOR/VERSION_MINOR; oh well.)

[thakis]

clang/lib/Frontend/CompilerInvocation.cpp

3637

Arguably, we should omit the full repo version from the hash: There's no reason to use a new cache dir just because someone fixed a typo in the (say) mlir docs. We'd also be better about remembering to bump clang::serialization::VERSION_MAJOR/MINOR in that case.

[thakis]

rnk, could you maybe take a look?

[rnk]

lgtm

clang/include/clang/Serialization/ASTBitCodes.h

namespace serialization {
     /// Whenever the AST file format changes in a way that makes it
     /// incompatible with previous versions (such that a reader
     /// designed for the previous version could not support reading
     /// the new version), this number should be increased.
     ///
     /// Version 4 of AST files also requires that the version control branch and
     /// revision match exactly, since there is no backward compatibility of
     /// AST files at this time.
-    const unsigned VERSION_MAJOR = 8;
+    const unsigned VERSION_MAJOR = 9;
 
     /// AST file minor version number supported by this version of
     /// Clang.
     ///
     /// Whenever the AST format changes in a way that is still
     /// compatible with previous versions (such that a reader designed
     /// for the previous version could still support reading the new
     /// version by ignoring new kinds of subblocks), this number

clang/lib/Frontend/CompilerInvocation.cpp

 #include "clang/Frontend/FrontendOptions.h"
 #include "clang/Frontend/FrontendPluginRegistry.h"
 #include "clang/Frontend/MigratorOptions.h"
 #include "clang/Frontend/PreprocessorOutputOptions.h"
 #include "clang/Frontend/Utils.h"
 #include "clang/Lex/HeaderSearchOptions.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Sema/CodeCompleteOptions.h"
+#include "clang/Serialization/ASTBitCodes.h"
 #include "clang/Serialization/ModuleFileExtension.h"
 #include "clang/StaticAnalyzer/Core/AnalyzerOptions.h"
 #include "llvm/ADT/APInt.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/CachedHashString.h"
 #include "llvm/ADT/Hashing.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
std::string CompilerInvocation::getModuleHash() const {
   using llvm::hash_code;
   using llvm::hash_value;
   using llvm::hash_combine;
   using llvm::hash_combine_range;
 
   // Start the signature with the compiler version.
   // FIXME: We'd rather use something more cryptographically sound than
   // CityHash, but this will do for now.
   hash_code code = hash_value(getClangFullRepositoryVersion());
 
+  // Also include the serialization version, in case LLVM_APPEND_VC_REV is off
+  // and getClangFullRepositoryVersion() doesn't include git revision.
+  code = hash_combine(code, serialization::VERSION_MAJOR,
+                      serialization::VERSION_MINOR);
+
   // Extend the signature with the language options
 #define LANGOPT(Name, Bits, Default, Description) \
    code = hash_combine(code, LangOpts->Name);
 #define ENUM_LANGOPT(Name, Type, Bits, Default, Description) \
   code = hash_combine(code, static_cast<unsigned>(LangOpts->get##Name()));
 #define BENIGN_LANGOPT(Name, Bits, Default, Description)
 #define BENIGN_ENUM_LANGOPT(Name, Type, Bits, Default, Description)
 #include "clang/Basic/LangOptions.def"

clang/test/Modules/diagnostics.modulemap

+// RUN: rm -rf %t
 // RUN: not %clang_cc1 -fmodules -fmodules-cache-path=%t -fmodule-map-file=%S/Inputs/diagnostics-aux.modulemap -fmodule-map-file=%s -fsyntax-only -x c++ /dev/null 2>&1 | FileCheck %s --implicit-check-not error:
 
 // CHECK: In file included from {{.*}}diagnostics-aux.modulemap:3:
 // CHECK: diagnostics-aux-2.modulemap:2:3: error: expected
 
 // PR22299: Ensure we can produce diagnostics for duplicate modules from -fmodule-map-file=.
 //
 // CHECK: diagnostics.modulemap:[[@LINE+2]]:8: error: redefinition of module 'foo'

clang/test/Modules/exception-spec.cpp

+// RUN: rm -rf %t
 // RUN: %clang_cc1 -x c++ -std=c++17 -fmodules -fmodules-local-submodule-visibility -fmodules-cache-path=%t %s -verify
 
 // expected-no-diagnostics
 
 #pragma clang module build PR38627
 module PR38627 {}
 #pragma clang module contents
 #pragma clang module begin PR38627

clang/test/Modules/merge-lifetime-extended-temporary.cpp

+// RUN: rm -rf %t
 // RUN: %clang_cc1 -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -x c++ -I%S/Inputs/merge-lifetime-extended-temporary -verify -std=c++11 %s -DORDER=1
 // RUN: %clang_cc1 -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -x c++ -I%S/Inputs/merge-lifetime-extended-temporary -verify -std=c++11 %s -DORDER=2
 
 // expected-no-diagnostics
 #if ORDER == 1
 #include "c.h"
 #include "b.h"
 #else
 #include "b.h"
 #include "c.h"
 #endif
 
 static_assert(PtrTemp1 == &LETemp, "");
 static_assert(PtrTemp1 == PtrTemp2, "");

clang/test/Modules/objc-method-redecl.m

+// RUN: rm -rf %t
 // RUN: %clang_cc1 -fmodules -fmodules-cache-path=%t -x objective-c-header -emit-pch %S/Inputs/objc-method-redecl.h -o %t.pch -Wno-objc-root-class
 // RUN: %clang_cc1 -fmodules -fmodules-cache-path=%t -x objective-c -include-pch %t.pch %s -verify -Wno-objc-root-class
 // expected-no-diagnostics
 
 @implementation T
 - (void)test {
 }
 @end

clang/test/Modules/using-decl-inheritance.cpp

+// RUN: rm -rf %t
 // RUN: %clang_cc1 -x c++ -fmodules -fmodules-local-submodule-visibility -fmodules-cache-path=%t %s -verify
 // RUN: %clang_cc1 -x c++ -fmodules -fmodules-cache-path=%t %s -verify
 
 // expected-no-diagnostics
 
 #pragma clang module build A
   module A { }
 #pragma clang module contents