[Coverage] Load code coverage data from archives

[vsk]

Summary

Support loading code coverage data from regular archives, thin archives,
and from MachO universal binaries which contain archives.

Testing: check-llvm, check-profile (with {A,UB}San enabled)

rdar://51538999

[Dor1s]

Left some questions / suggestions, but don't see any problems, LGTM!

llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp

595

nit: I guess it's stupid, but maybe sizeof(uint32_t) isntead of 4? And the same in the following conditions.

597

I think we can make it a bit more concise (2 branches instead of 4) by passing Endian as a template parameter. May need a separate check though to make sure it's either support::endianness::little or support::endianness::big.

742

Why do we use emplace_back here and on line 794, but push_back in the other places?

757

Should we log anything here?

[vsk]

• Address review feedback.

llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp

595

If it were possible to write 'sizeof(intptr32_t)', I think that would be clearer. I'm not sure 'sizeof(uint32_t)' is clearer than '4', though.

597

As 'Endian' is not a constant-expression, I don't think it's possible to pass it in as a template parameter. I think it could be passed as a regular parameter, but that would eliminate two branches here while adding two branches to 'readCoverageMappingData'.

742

Fixed.

757

I don't think so. It's expected that at least one of the slices in a universal binary will not match the desired architecture. I'll add a comment to that effect.

[Dor1s]

Thanks for the replies, Vedant. Still LGTM!

llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp

597

Ah, good point, my bad!

[davidxl]

is thin archive supported?

[davidxl]

llvm-cov's user manual probably also needs some update.

[vsk]

• Add support for thin archives (and a test).
• Update the llvm-cov CommandGuide doc.

[davidxl]

lgtm

llvm/include/llvm/ProfileData/Coverage/CoverageMappingReader.h

private:
   std::vector<CounterMappingRegion> MappingRegions;
 
   BinaryCoverageReader() = default;
 
 public:
   BinaryCoverageReader(const BinaryCoverageReader &) = delete;
   BinaryCoverageReader &operator=(const BinaryCoverageReader &) = delete;
 
+  static Expected<std::vector<std::unique_ptr<BinaryCoverageReader>>>
+  create(MemoryBufferRef ObjectBuffer, StringRef Arch);
+
   static Expected<std::unique_ptr<BinaryCoverageReader>>
-  create(std::unique_ptr<MemoryBuffer> &ObjectBuffer,
-         StringRef Arch);
+  createCoverageReaderFromBuffer(StringRef Coverage,
+                                 InstrProfSymtab &&ProfileNames,
+                                 uint8_t BytesInAddress,
+                                 support::endianness Endian);
 
   Error readNextRecord(CoverageMappingRecord &Record) override;
 };
 
 } // end namespace coverage
 } // end namespace llvm
 
 #endif // LLVM_PROFILEDATA_COVERAGE_COVERAGEMAPPINGREADER_H

llvm/lib/ProfileData/Coverage/CoverageMapping.cpp

CoverageMapping::load(ArrayRef<StringRef> ObjectFilenames,
 
   SmallVector<std::unique_ptr<CoverageMappingReader>, 4> Readers;
   SmallVector<std::unique_ptr<MemoryBuffer>, 4> Buffers;
   for (const auto &File : llvm::enumerate(ObjectFilenames)) {
     auto CovMappingBufOrErr = MemoryBuffer::getFileOrSTDIN(File.value());
     if (std::error_code EC = CovMappingBufOrErr.getError())
       return errorCodeToError(EC);
     StringRef Arch = Arches.empty() ? StringRef() : Arches[File.index()];
-    auto CoverageReaderOrErr =
-        BinaryCoverageReader::create(CovMappingBufOrErr.get(), Arch);
-    if (Error E = CoverageReaderOrErr.takeError())
+    MemoryBufferRef CovMappingBufRef =
+        CovMappingBufOrErr.get()->getMemBufferRef();
+    auto CoverageReadersOrErr =
+        BinaryCoverageReader::create(CovMappingBufRef, Arch);
+    if (Error E = CoverageReadersOrErr.takeError())
       return std::move(E);
-    Readers.push_back(std::move(CoverageReaderOrErr.get()));
+    for (auto &Reader : CoverageReadersOrErr.get())
+      Readers.push_back(std::move(Reader));
     Buffers.push_back(std::move(CovMappingBufOrErr.get()));
   }
   return load(Readers, *ProfileReader);
 }
 
 namespace {
 
 /// Distributes functions into instantiation sets.

llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp

if (auto E = NextHeaderOrErr.takeError())
       return E;
     Buf = NextHeaderOrErr.get();
   }
   return Error::success();
 }
 
 static const char *TestingFormatMagic = "llvmcovmtestdata";
 
-static Error loadTestingFormat(StringRef Data, InstrProfSymtab &ProfileNames,
-                               StringRef &CoverageMapping,
-                               uint8_t &BytesInAddress,
-                               support::endianness &Endian) {
-  BytesInAddress = 8;
-  Endian = support::endianness::little;
+Expected<std::unique_ptr<BinaryCoverageReader>>
+BinaryCoverageReader::createCoverageReaderFromBuffer(
+    StringRef Coverage, InstrProfSymtab &&ProfileNames, uint8_t BytesInAddress,
+    support::endianness Endian) {
+  std::unique_ptr<BinaryCoverageReader> Reader(new BinaryCoverageReader());
+  Reader->ProfileNames = std::move(ProfileNames);
+  if (BytesInAddress == 4 && Endian == support::endianness::little) {
+    if (Error E =
+            readCoverageMappingData<uint32_t, support::endianness::little>(
+                Reader->ProfileNames, Coverage, Reader->MappingRecords,
+                Reader->Filenames))
+      return std::move(E);
+  } else if (BytesInAddress == 4 && Endian == support::endianness::big) {
+    if (Error E = readCoverageMappingData<uint32_t, support::endianness::big>(
+            Reader->ProfileNames, Coverage, Reader->MappingRecords,
+            Reader->Filenames))
+      return std::move(E);
+  } else if (BytesInAddress == 8 && Endian == support::endianness::little) {
+    if (Error E =
+            readCoverageMappingData<uint64_t, support::endianness::little>(
+                Reader->ProfileNames, Coverage, Reader->MappingRecords,
+                Reader->Filenames))
+      return std::move(E);
+  } else if (BytesInAddress == 8 && Endian == support::endianness::big) {
+    if (Error E = readCoverageMappingData<uint64_t, support::endianness::big>(
+            Reader->ProfileNames, Coverage, Reader->MappingRecords,
+            Reader->Filenames))
+      return std::move(E);
+  } else
+    return make_error<CoverageMapError>(coveragemap_error::malformed);
+  return Reader;
+}
+
+static Expected<std::unique_ptr<BinaryCoverageReader>>
+loadTestingFormat(StringRef Data) {
+  uint8_t BytesInAddress = 8;
+  support::endianness Endian = support::endianness::little;
 
   Data = Data.substr(StringRef(TestingFormatMagic).size());
   if (Data.empty())
     return make_error<CoverageMapError>(coveragemap_error::truncated);
   unsigned N = 0;
   uint64_t ProfileNamesSize = decodeULEB128(Data.bytes_begin(), &N);
   if (N > Data.size())
     return make_error<CoverageMapError>(coveragemap_error::malformed);
   Data = Data.substr(N);
   if (Data.empty())
     return make_error<CoverageMapError>(coveragemap_error::truncated);
   N = 0;
   uint64_t Address = decodeULEB128(Data.bytes_begin(), &N);
   if (N > Data.size())
     return make_error<CoverageMapError>(coveragemap_error::malformed);
   Data = Data.substr(N);
   if (Data.size() < ProfileNamesSize)
     return make_error<CoverageMapError>(coveragemap_error::malformed);
+  InstrProfSymtab ProfileNames;
   if (Error E = ProfileNames.create(Data.substr(0, ProfileNamesSize), Address))
-    return E;
-  CoverageMapping = Data.substr(ProfileNamesSize);
+    return std::move(E);
+  StringRef CoverageMapping = Data.substr(ProfileNamesSize);
   // Skip the padding bytes because coverage map data has an alignment of 8.
   if (CoverageMapping.empty())
     return make_error<CoverageMapError>(coveragemap_error::truncated);
   size_t Pad = alignmentAdjustment(CoverageMapping.data(), 8);
   if (CoverageMapping.size() < Pad)
     return make_error<CoverageMapError>(coveragemap_error::malformed);
   CoverageMapping = CoverageMapping.substr(Pad);
-  return Error::success();
+  return BinaryCoverageReader::createCoverageReaderFromBuffer(
+      CoverageMapping, std::move(ProfileNames), BytesInAddress, Endian);
 }
 
 static Expected<SectionRef> lookupSection(ObjectFile &OF, StringRef Name) {
   // On COFF, the object file section name may end in "$M". This tells the
   // linker to sort these sections between "$A" and "$Z". The linker removes the
   // dollar and everything after it in the final binary. Do the same to match.
   bool IsCOFF = isa<COFFObjectFile>(OF);
   auto stripSuffix = [IsCOFF](StringRef N) {
     return IsCOFF ? N.split('$').first : N;
   };
   Name = stripSuffix(Name);
 
   StringRef FoundName;
   for (const auto &Section : OF.sections()) {
     if (auto EC = Section.getName(FoundName))
       return errorCodeToError(EC);
     if (stripSuffix(FoundName) == Name)
       return Section;
   }
   return make_error<CoverageMapError>(coveragemap_error::no_data_found);
 }
 
-static Error loadBinaryFormat(MemoryBufferRef ObjectBuffer,
-                              InstrProfSymtab &ProfileNames,
-                              StringRef &CoverageMapping,
-                              uint8_t &BytesInAddress,
-                              support::endianness &Endian, StringRef Arch) {
-  auto BinOrErr = createBinary(ObjectBuffer);
-  if (!BinOrErr)
-    return BinOrErr.takeError();
-  auto Bin = std::move(BinOrErr.get());
+static Expected<std::unique_ptr<BinaryCoverageReader>>
+loadBinaryFormat(std::unique_ptr<Binary> Bin, StringRef Arch) {
   std::unique_ptr<ObjectFile> OF;
   if (auto *Universal = dyn_cast<MachOUniversalBinary>(Bin.get())) {
     // If we have a universal binary, try to look up the object for the
     // appropriate architecture.
     auto ObjectFileOrErr = Universal->getObjectForArch(Arch);
     if (!ObjectFileOrErr)
       return ObjectFileOrErr.takeError();
     OF = std::move(ObjectFileOrErr.get());
   } else if (isa<ObjectFile>(Bin.get())) {
     // For any other object file, upcast and take ownership.
     OF.reset(cast<ObjectFile>(Bin.release()));
     // If we've asked for a particular arch, make sure they match.
     if (!Arch.empty() && OF->getArch() != Triple(Arch).getArch())
       return errorCodeToError(object_error::arch_not_found);
   } else
     // We can only handle object files.
     return make_error<CoverageMapError>(coveragemap_error::malformed);
 
   // The coverage uses native pointer sizes for the object it's written in.
-  BytesInAddress = OF->getBytesInAddress();
-  Endian = OF->isLittleEndian() ? support::endianness::little
+  uint8_t BytesInAddress = OF->getBytesInAddress();
+  support::endianness Endian = OF->isLittleEndian()
+                                   ? support::endianness::little
                                    : support::endianness::big;
 
   // Look for the sections that we are interested in.
   auto ObjFormat = OF->getTripleObjectFormat();
   auto NamesSection =
       lookupSection(*OF, getInstrProfSectionName(IPSK_name, ObjFormat,
                                                  /*AddSegmentInfo=*/false));
   if (auto E = NamesSection.takeError())
-    return E;
+    return std::move(E);
   auto CoverageSection =
       lookupSection(*OF, getInstrProfSectionName(IPSK_covmap, ObjFormat,
                                                  /*AddSegmentInfo=*/false));
   if (auto E = CoverageSection.takeError())
-    return E;
+    return std::move(E);
 
   // Get the contents of the given sections.
-  if (Expected<StringRef> E = CoverageSection->getContents())
-    CoverageMapping = *E;
-  else
-    return E.takeError();
+  auto CoverageMappingOrErr = CoverageSection->getContents();
+  if (!CoverageMappingOrErr)
+    return CoverageMappingOrErr.takeError();
 
+  InstrProfSymtab ProfileNames;
   if (Error E = ProfileNames.create(*NamesSection))
-    return E;
+    return std::move(E);
 
-  return Error::success();
+  return BinaryCoverageReader::createCoverageReaderFromBuffer(
+      CoverageMappingOrErr.get(), std::move(ProfileNames), BytesInAddress,
+      Endian);
 }
 
-Expected<std::unique_ptr<BinaryCoverageReader>>
-BinaryCoverageReader::create(std::unique_ptr<MemoryBuffer> &ObjectBuffer,
+Expected<std::vector<std::unique_ptr<BinaryCoverageReader>>>
+BinaryCoverageReader::create(MemoryBufferRef ObjectBuffer,
                              StringRef Arch) {
-  std::unique_ptr<BinaryCoverageReader> Reader(new BinaryCoverageReader());
+  std::vector<std::unique_ptr<BinaryCoverageReader>> Readers;
 
-  StringRef Coverage;
-  uint8_t BytesInAddress;
-  support::endianness Endian;
-  Error E = Error::success();
-  consumeError(std::move(E));
-  if (ObjectBuffer->getBuffer().startswith(TestingFormatMagic))
+  if (ObjectBuffer.getBuffer().startswith(TestingFormatMagic)) {
     // This is a special format used for testing.
-    E = loadTestingFormat(ObjectBuffer->getBuffer(), Reader->ProfileNames,
-                          Coverage, BytesInAddress, Endian);
-  else
-    E = loadBinaryFormat(ObjectBuffer->getMemBufferRef(), Reader->ProfileNames,
-                         Coverage, BytesInAddress, Endian, Arch);
-  if (E)
-    return std::move(E);
+    auto ReaderOrErr = loadTestingFormat(ObjectBuffer.getBuffer());
+    if (!ReaderOrErr)
+      return ReaderOrErr.takeError();
+    Readers.push_back(std::move(ReaderOrErr.get()));
+    return Readers;
+  }
 
-  if (BytesInAddress == 4 && Endian == support::endianness::little)
-    E = readCoverageMappingData<uint32_t, support::endianness::little>(
-        Reader->ProfileNames, Coverage, Reader->MappingRecords,
-        Reader->Filenames);
-  else if (BytesInAddress == 4 && Endian == support::endianness::big)
-    E = readCoverageMappingData<uint32_t, support::endianness::big>(
-        Reader->ProfileNames, Coverage, Reader->MappingRecords,
-        Reader->Filenames);
-  else if (BytesInAddress == 8 && Endian == support::endianness::little)
-    E = readCoverageMappingData<uint64_t, support::endianness::little>(
-        Reader->ProfileNames, Coverage, Reader->MappingRecords,
-        Reader->Filenames);
-  else if (BytesInAddress == 8 && Endian == support::endianness::big)
-    E = readCoverageMappingData<uint64_t, support::endianness::big>(
-        Reader->ProfileNames, Coverage, Reader->MappingRecords,
-        Reader->Filenames);
-  else
-    return make_error<CoverageMapError>(coveragemap_error::malformed);
-  if (E)
-    return std::move(E);
-  return std::move(Reader);
+  auto BinOrErr = createBinary(ObjectBuffer);
+  if (!BinOrErr)
+    return BinOrErr.takeError();
+  std::unique_ptr<Binary> Bin = std::move(BinOrErr.get());
+
+  // MachO universal binaries which contain archives need to be treated as
+  // archives, not as regular binaries.
+  if (auto *Universal = dyn_cast<MachOUniversalBinary>(Bin.get())) {
+    for (auto &ObjForArch : Universal->objects()) {
+      // Skip slices within the universal binary which target the wrong arch.
+      std::string ObjArch = ObjForArch.getArchFlagName();
+      if (Arch != ObjArch)
+        continue;
+
+      auto ArchiveOrErr = ObjForArch.getAsArchive();
+      if (!ArchiveOrErr) {
+        // If this is not an archive, try treating it as a regular object.
+        consumeError(ArchiveOrErr.takeError());
+        break;
+      }
+
+      return BinaryCoverageReader::create(
+          ArchiveOrErr.get()->getMemoryBufferRef(), Arch);
+    }
+  }
+
+  // Load coverage out of archive members.
+  if (auto *Ar = dyn_cast<Archive>(Bin.get())) {
+    Error Err = Error::success();
+    for (auto &Child : Ar->children(Err)) {
+      Expected<MemoryBufferRef> ChildBufOrErr = Child.getMemoryBufferRef();
+      if (!ChildBufOrErr)
+        return ChildBufOrErr.takeError();
+
+      auto ChildReadersOrErr =
+          BinaryCoverageReader::create(ChildBufOrErr.get(), Arch);
+      if (!ChildReadersOrErr)
+        return ChildReadersOrErr.takeError();
+      for (auto &Reader : ChildReadersOrErr.get())
+        Readers.push_back(std::move(Reader));
+    }
+    if (Err)
+      return std::move(Err);
+    return Readers;
+  }
+
+  auto ReaderOrErr = loadBinaryFormat(std::move(Bin), Arch);
+  if (!ReaderOrErr)
+    return ReaderOrErr.takeError();
+  Readers.push_back(std::move(ReaderOrErr.get()));
+  return Readers;
 }
 
 Error BinaryCoverageReader::readNextRecord(CoverageMappingRecord &Record) {
   if (CurrentRecord >= MappingRecords.size())
     return make_error<CoverageMapError>(coveragemap_error::eof);
 
   FunctionsFilenames.clear();
   Expressions.clear();

llvm/test/tools/llvm-cov/Inputs/universal_bin_wrapping_archives/obj1.c

+void f1() {}

llvm/test/tools/llvm-cov/Inputs/universal_bin_wrapping_archives/obj2.c

+void f2() {}

llvm/test/tools/llvm-cov/Inputs/universal_bin_wrapping_archives/universal_bin_wrapping_archives.proftext

+f1
+0x0
+1
+100
+f2
+0x0
+1
+100

llvm/test/tools/llvm-cov/universal_bin_wrapping_archives.test

+The coverage reader should be able to handle archives, and archives embedded within
+MachO universal binaries.
+
+---
+Steps to re-generate these files on macOS:
+
+clang -fprofile-instr-generate -fcoverage-mapping -c obj1.c -o obj1_32.o -arch i386
+clang -fprofile-instr-generate -fcoverage-mapping -c obj2.c -o obj2_32.o -arch i386
+clang -fprofile-instr-generate -fcoverage-mapping -c obj1.c -o obj1_64.o -arch x86_64
+clang -fprofile-instr-generate -fcoverage-mapping -c obj2.c -o obj2_64.o -arch x86_64
+ar -q archive_32 obj1_32.o obj2_32.o
+ar -q archive_64 obj1_64.o obj2_64.o
+lipo -output universal_bin_wrapping_archives -create archive_32 archive_64
+---
+
+RUN: llvm-profdata merge %S/Inputs/universal_bin_wrapping_archives/universal_bin_wrapping_archives.proftext -o %t.profdata
+
+RUN: llvm-cov show %S/Inputs/universal_bin_wrapping_archives/universal_bin_wrapping_archives \
+RUN:   -instr-profile %t.profdata -path-equivalence=/tmp,%S/Inputs/universal_bin_wrapping_archives %s -arch i386 \
+RUN:   | FileCheck %s --check-prefix=SHOW_ARCHIVE
+
+RUN: llvm-cov show %S/Inputs/universal_bin_wrapping_archives/universal_bin_wrapping_archives \
+RUN:   -instr-profile %t.profdata -path-equivalence=/tmp,%S/Inputs/universal_bin_wrapping_archives %s -arch x86_64 \
+RUN:   | FileCheck %s --check-prefix=SHOW_ARCHIVE
+
+SHOW_ARCHIVE: {{.*}}obj1.c:
+SHOW_ARCHIVE-NEXT: 1|    100|void f1() {}
+SHOW_ARCHIVE: {{.*}}obj2.c:
+SHOW_ARCHIVE-NEXT: 1|    100|void f2() {}
+
+RUN: llvm-cov report %S/Inputs/universal_bin_wrapping_archives/universal_bin_wrapping_archives \
+RUN:   -instr-profile %t.profdata -path-equivalence=/tmp,%S/Inputs/universal_bin_wrapping_archives %s -arch i386 \
+RUN:   | FileCheck %s --check-prefix=REPORT_ARCHIVE
+
+RUN: llvm-cov report %S/Inputs/universal_bin_wrapping_archives/universal_bin_wrapping_archives \
+RUN:   -instr-profile %t.profdata -path-equivalence=/tmp,%S/Inputs/universal_bin_wrapping_archives %s -arch x86_64 \
+RUN:   | FileCheck %s --check-prefix=REPORT_ARCHIVE
+
+REPORT_ARCHIVE: obj1.c 1 0 100.00% 1 0 100.00% 1 0 100.00%
+REPORT_ARCHIVE: obj2.c 1 0 100.00% 1 0 100.00% 1 0 100.00%
+REPORT_ARCHIVE: TOTAL 2 0 100.00% 2 0 100.00% 2 0 100.00%