[dsymutil] Timestmap verification for __swift_ast

[JDevlieghere]

Summary

This patch adds timestamp verification for swiftmodule files.

• A new flag is provided to allows us to continue testing of the code for embedding the__swift_ast. (git doesn't maintain timestamps)
• Adds a new test for fat (arm) binaries.

[fjricci]

Looks good to me

[JDevlieghere]

• Format changes

[aprantl]

test/tools/dsymutil/ARM/swift-ast.test

1 ↗

(On Diff #118246)

if the test is disabling the new functionality (-no-swiftmodule-timestamp) what is it testing?

[JDevlieghere]

test/tools/dsymutil/ARM/swift-ast.test

1 ↗

(On Diff #118246)

I used this to verify my implementation locally, and figured I might as well upstream it as it covers two scenarios (arm & universal mach-o) not covered by the existing test. Let me know if you would leave this out or have it moved into in a separate commit.

[aprantl]

test/tools/dsymutil/ARM/swift-ast.test

1 ↗

(On Diff #118246)

the extra test is fine, I was just wondering if the test shouldn't be testing for the "Timestamp mismatch" warning.

[JDevlieghere]

• Update tests to check for warnings

llvm/trunk/test/tools/dsymutil/ARM/swift-ast.test

+RUN: llvm-dsymutil -oso-prepend-path %p/.. %p/../Inputs/swift-ast.macho.fat -o %T/swift-ast.dSYM -verbose -no-swiftmodule-timestamp | FileCheck %s --check-prefix=DSYMUTIL
+RUN: llvm-readobj -sections -section-data %T/swift-ast.dSYM/Contents/Resources/DWARF/swift-ast.macho.fat | FileCheck %s --check-prefix=READOBJ
+
+The tested object file has been created by the dummy Swift code:
+let x = 1
+
+Compiled with:
+  swiftc /tmp/test.swift -Onone -target armv7-apple-ios8.1 -c -o swift-ast-armv7.o
+  ld swift-ast-armv7.o -add_ast_path swift-ast.swiftmodule -arch armv7 -ios_version_min 8.1  -syslibroot /path/to/iPhoneOS.sdk -L/path/to/usr/lib/swift/iphoneos -lobjc -lSystem -o swift-ast.macho.armv7
+
+  swiftc /tmp/test.swift -Onone -target arm64-apple-ios8.1 -c -o swift-ast-arm64.o
+  ld swift-ast-arm64.o -add_ast_path swift-ast.swiftmodule -arch arm64 -ios_version_min 8.1  -syslibroot /path/to/iPhoneOS.sdk -L/path/to/usr/lib/swift/iphoneos -lobjc -lSystem -o swift-ast.macho.arm64
+
+  lipo -create swift-ast.macho.armv7 swift-ast.macho.arm64 -o swift-ast.macho.fat
+
+DSYMUTIL: filename:{{.*}}swift-ast.swiftmodule
+DSYMUTIL: DEBUG MAP OBJECT:{{.*}}swift-ast.swiftmodule
+
+READOBJ: Name:{{.*}}__swift_ast
+READOBJ: |SWIFTMODULE DATA|
+READOBJ-NEXT: |.|
+READOBJ: |SWIFTMODULE DATA|
+READOBJ-NEXT: |.|
+
+RUN: llvm-dsymutil -oso-prepend-path %p/.. %p/../Inputs/swift-ast.macho.fat -no-output -verbose 2>&1 | FileCheck %s --check-prefix=TIMESTAMP
+TIMESTAMP: Warning: Timestamp mismatch

llvm/trunk/test/tools/dsymutil/X86/swift-ast-x86_64.test

-RUN: llvm-dsymutil -oso-prepend-path %p/.. %p/../Inputs/swift-ast.macho.x86_64 -o %T/swift-ast.dSYM -verbose | FileCheck %s --check-prefix=DSYMUTIL
+RUN: llvm-dsymutil -oso-prepend-path %p/.. %p/../Inputs/swift-ast.macho.x86_64 -o %T/swift-ast.dSYM -verbose -no-swiftmodule-timestamp | FileCheck %s --check-prefix=DSYMUTIL
 RUN: llvm-readobj -sections -section-data %T/swift-ast.dSYM/Contents/Resources/DWARF/swift-ast.macho.x86_64 | FileCheck %s --check-prefix=READOBJ
 
 The tested object file has been created by the dummy Swift code:
 let x = 1
 
-Compiled with: swiftc /tmp/test.swift -Onone -target x86_64-apple-macosx10.9 -c
-Linked with: ld swift-ast.o -add_ast_path Inputs/swift-ast.swiftmodule -arch x86_64 -lSystem -macosx_version_min 10.9.0
+Compiled with:
+  swiftc /tmp/test.swift -Onone -target x86_64-apple-macosx10.9 -c
+  ld swift-ast.o -add_ast_path Inputs/swift-ast.swiftmodule -arch x86_64 -lSystem -macosx_version_min 10.9.0
 
 DSYMUTIL: filename:{{.*}}swift-ast.swiftmodule
 DSYMUTIL: DEBUG MAP OBJECT:{{.*}}swift-ast.swiftmodule
 
 READOBJ: Name:{{.*}}__swift_ast
 READOBJ: |SWIFTMODULE DATA|
+READOBJ-NEXT: |.|
+
+RUN: llvm-dsymutil -oso-prepend-path %p/.. %p/../Inputs/swift-ast.macho.x86_64 -no-output -verbose 2>&1 | FileCheck %s --check-prefix=TIMESTAMP
+TIMESTAMP: Warning: Timestamp mismatch

llvm/trunk/tools/dsymutil/BinaryHolder.h

class BinaryHolder {
   /// corresponding archive to memory and return the MemoryBufferRefs
   /// corresponding to the described member. Multiple buffers are
   /// returned when there are multiple architectures available for the
   /// requested file.
   ErrorOr<std::vector<MemoryBufferRef>>
   MapArchiveAndGetMemberBuffers(StringRef Filename,
                                 sys::TimePoint<std::chrono::seconds> Timestamp);
 
-  void changeBackingMemoryBuffer(std::unique_ptr<MemoryBuffer> &&MemBuf);
-  ErrorOr<const object::ObjectFile &> getObjfileForArch(const Triple &T);
-
-public:
-  BinaryHolder(bool Verbose) : Verbose(Verbose) {}
-
   /// Return the MemoryBufferRef that holds the memory mapping for the
   /// given \p Filename. This function will try to parse archive
   /// member specifications of the form /path/to/archive.a(member.o).
   ///
   /// The returned MemoryBufferRefs points to a buffer owned by this
   /// object. The buffer is valid until the next call to
   /// GetMemoryBufferForFile() on this object.
   /// Multiple buffers are returned when there are multiple
   /// architectures available for the requested file.
   ErrorOr<std::vector<MemoryBufferRef>>
   GetMemoryBuffersForFile(StringRef Filename,
                           sys::TimePoint<std::chrono::seconds> Timestamp);
 
+  void changeBackingMemoryBuffer(std::unique_ptr<MemoryBuffer> &&MemBuf);
+  ErrorOr<const object::ObjectFile &> getObjfileForArch(const Triple &T);
+
+public:
+  BinaryHolder(bool Verbose) : Verbose(Verbose) {}
+
   /// Get the ObjectFiles designated by the \p Filename. This
   /// might be an archive member specification of the form
   /// /path/to/archive.a(member.o).
   ///
   /// Calling this function invalidates the previous mapping owned by
   /// the BinaryHolder. Multiple buffers are returned when there are
   /// multiple architectures available for the requested file.
   ErrorOr<std::vector<const object::ObjectFile *>>

llvm/trunk/tools/dsymutil/DwarfLinker.cpp

public:
 
   /// Emit the abbreviation table \p Abbrevs to the debug_abbrev section.
   void emitAbbrevs(const std::vector<std::unique_ptr<DIEAbbrev>> &Abbrevs,
                    unsigned DwarfVersion);
 
   /// Emit the string table described by \p Pool.
   void emitStrings(const NonRelocatableStringpool &Pool);
 
-  /// Emit the swift_ast section stored in \p Buffers.
-  void emitSwiftAST(const std::vector<MemoryBufferRef> &Buffers);
+  /// Emit the swift_ast section stored in \p Buffer.
+  void emitSwiftAST(StringRef Buffer);
 
   /// Emit debug_ranges for \p FuncRange by translating the
   /// original \p Entries.
   void emitRangesEntries(
       int64_t UnitPcOffset, uint64_t OrigLowPc,
       const FunctionIntervals::const_iterator &FuncRange,
       const std::vector<DWARFDebugRangeList::RangeListEntry> &Entries,
       unsigned AddressSize);
void DwarfStreamer::emitStrings(const NonRelocatableStringpool &Pool) {
   Asm->OutStreamer->SwitchSection(MOFI->getDwarfStrSection());
   for (auto *Entry = Pool.getFirstEntry(); Entry;
        Entry = Pool.getNextEntry(Entry))
     Asm->OutStreamer->EmitBytes(
         StringRef(Entry->getKey().data(), Entry->getKey().size() + 1));
 }
 
 /// Emit the swift_ast section stored in \p Buffers.
-void DwarfStreamer::emitSwiftAST(const std::vector<MemoryBufferRef> &Buffers) {
+void DwarfStreamer::emitSwiftAST(StringRef Buffer) {
   MCSection *SwiftASTSection = MOFI->getDwarfSwiftASTSection();
   SwiftASTSection->setAlignment(1 << 5);
   MS->SwitchSection(SwiftASTSection);
-  for (auto Buf : Buffers)
-    MS->EmitBytes(Buf.getBuffer());
+  MS->EmitBytes(Buffer);
 }
 
 /// Emit the debug_range section contents for \p FuncRange by
 /// translating the original \p Entries. The debug_range section
 /// format is totally trivial, consisting just of pairs of address
 /// sized addresses describing the ranges.
 void DwarfStreamer::emitRangesEntries(
     int64_t UnitPcOffset, uint64_t OrigLowPc,
for (const auto &Obj : Map.objects()) {
     CurrentDebugObject = Obj.get();
 
     if (Options.Verbose)
       outs() << "DEBUG MAP OBJECT: " << Obj->getObjectFilename() << "\n";
 
     // N_AST objects (swiftmodule files) should get dumped directly into the
     // appropriate DWARF section.
     if (Obj->getType() == MachO::N_AST) {
-      auto ErrOrMemBufferRefs = BinHolder.GetMemoryBuffersForFile(
-          Obj->getObjectFilename(), Obj->getTimestamp());
-      if (ErrOrMemBufferRefs.getError())
+      StringRef File = Obj->getObjectFilename();
+      auto ErrorOrMem = MemoryBuffer::getFile(File);
+      if (!ErrorOrMem) {
+        errs() << "Warning: Could not open " << File << "\n";
+        continue;
+      }
+      sys::fs::file_status Stat;
+      if (auto errc = sys::fs::status(File, Stat)) {
+        errs() << "Warning: " << errc.message() << "\n";
         continue;
+      }
+      if (!Options.NoTimestamp && Stat.getLastModificationTime() !=
+                                      sys::TimePoint<>(Obj->getTimestamp())) {
+        errs() << "Warning: Timestamp mismatch for " << File << ": "
+               << Stat.getLastModificationTime() << " and "
+               << sys::TimePoint<>(Obj->getTimestamp()) << "\n";
+        continue;
+      }
+
+      // Copy the module into the .swift_ast section.
       if (!Options.NoOutput)
-        Streamer->emitSwiftAST(ErrOrMemBufferRefs.get());
+        Streamer->emitSwiftAST((*ErrorOrMem)->getBuffer());
       continue;
     }
 
     auto ErrOrObj = loadObject(BinHolder, *Obj, Map);
     if (!ErrOrObj)
       continue;
 
     // Look for relocations that correspond to debug map entries.

llvm/trunk/tools/dsymutil/dsymutil.h

 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/ErrorOr.h"
 #include <memory>
 
 namespace llvm {
 namespace dsymutil {
 
 struct LinkOptions {
-  bool Verbose;  ///< Verbosity
-  bool NoOutput; ///< Skip emitting output
-  bool NoODR;    ///< Do not unique types according to ODR
+  bool Verbose;            ///< Verbosity
+  bool NoOutput;           ///< Skip emitting output
+  bool NoODR;              ///< Do not unique types according to ODR
+  bool NoTimestamp;        ///< Do not check swiftmodule timestamp
   std::string PrependPath; ///< -oso-prepend-path
 
-  LinkOptions() : Verbose(false), NoOutput(false) {}
+  LinkOptions() : Verbose(false), NoOutput(false), NoTimestamp(false) {}
 };
 
 /// \brief Extract the DebugMaps from the given file.
 /// The file has to be a MachO object file. Multiple debug maps can be
 /// returned when the file is universal (aka fat) binary.
 llvm::ErrorOr<std::vector<std::unique_ptr<DebugMap>>>
 parseDebugMap(StringRef InputFile, ArrayRef<std::string> Archs,
               StringRef PrependPath, bool Verbose, bool InputIsYAML);

llvm/trunk/tools/dsymutil/dsymutil.cpp

 
 static opt<bool> Verbose("verbose", desc("Verbosity level"), init(false),
                          cat(DsymCategory));
 
 static opt<bool>
     NoOutput("no-output",
              desc("Do the link in memory, but do not emit the result file."),
              init(false), cat(DsymCategory));
-
+static opt<bool>
+    NoTimestamp("no-swiftmodule-timestamp",
+                desc("Don't check timestamp for swiftmodule files."),
+                init(false), cat(DsymCategory));
 static list<std::string> ArchFlags(
     "arch",
     desc("Link DWARF debug information only for specified CPU architecture\n"
          "types. This option can be specified multiple times, once for each\n"
          "desired architecture.  All cpu architectures will be linked by\n"
          "default."),
     ZeroOrMore, cat(DsymCategory));
 
int main(int argc, char **argv) {
   if (Version) {
     llvm::cl::PrintVersionMessage();
     return 0;
   }
 
   Options.Verbose = Verbose;
   Options.NoOutput = NoOutput;
   Options.NoODR = NoODR;
+  Options.NoTimestamp = NoTimestamp;
   Options.PrependPath = OsoPrependPath;
 
   llvm::InitializeAllTargetInfos();
   llvm::InitializeAllTargetMCs();
   llvm::InitializeAllTargets();
   llvm::InitializeAllAsmPrinters();
 
   if (!FlatOut && OutputFileOpt == "-") {