[llvm-ar][libObject] Fix relative paths when nesting thin archives.

[rupprecht]

Summary

When adding one thin archive to another, we currently chop off the relative path to the flattened members. For instance, when adding foo/child.a (which contains x.txt) to parent.a, when flattening it we should add it as foo/x.txt (which exists) instead of x.txt (which does not exist).

As a note, this also undoes the IsNew parameter of handling relative paths in r288280. The unit test there still passes.

This was reported as part of testing the kernel build with llvm-ar: https://patchwork.kernel.org/patch/10767545/ (see the second point).

[ruiu]

llvm/lib/Object/ArchiveWriter.cpp

238 ↗

(On Diff #185624)

So, M.MemberName is relative to ArcName if it is thin, and it is used as-is if not thin. It doesn't look like we really have to distinguish two here. Could you call computeRelativePath early in llvm-ar and set its result to MemberName? Then I think you can remove special handling of thin archive here.

llvm/test/tools/llvm-ar/flatten-thin-archive-directories.test

15 ↗

(On Diff #185624)

Are you going to fix this bug as well?

[rupprecht]

llvm/lib/Object/ArchiveWriter.cpp

238 ↗

(On Diff #185624)

Hmm, good point, that would be a lot simpler. I'll try it.

llvm/test/tools/llvm-ar/flatten-thin-archive-directories.test

15 ↗

(On Diff #185624)

I'll do it separately -- see D57845

[rupprecht]

• Move computeRelativePath special case from libObject to llvm-ar
• Remove obsolete test fixme

[rupprecht]

llvm/lib/Object/ArchiveWriter.cpp

238 ↗

(On Diff #185624)

That seems to have worked, thanks for the suggestion!

[ruiu]

LGTM

llvm/lib/Object/ArchiveWriter.cpp

238 ↗

(On Diff #185624)

This function is now too trivial to be a function. I'd remove this function by replacing a call site with OS << M.MemberName << "/\n";.

llvm/tools/llvm-ar/llvm-ar.cpp

618 ↗

(On Diff #185644)

Remove the redundant assignment.

[rupprecht]

• Fix related test to always delete the archive first
• Inline addToStringTable
• Remove redundant assignment

[hans]

I reverted this in r353507 as it broke the Chromium build; see https://crbug.com/930058

[rupprecht]

In D57842#1390249, @hans wrote:

I reverted this in r353507 as it broke the Chromium build; see https://crbug.com/930058

Sorry for the breakage! Thanks for reverting this.

Do you have repro instructions? I'd like to reland this soon.

[rupprecht]

Add regression test for chromium linking issue, and update LibDriver call site to correctly compute relative paths.

[rupprecht]

Reopened + updated patch with LibDriver fixes. I believe this should fix the chromium build, but haven't verified it. The new test case (llvm/test/tools/llvm-lib/thin-relative.test) should capture the issue, and was failing with the old patch but is now fixed.

[hans]

In D57842#1391462, @rupprecht wrote:

Reopened + updated patch with LibDriver fixes. I believe this should fix the chromium build, but haven't verified it. The new test case (llvm/test/tools/llvm-lib/thin-relative.test) should capture the issue, and was failing with the old patch but is now fixed.

Thanks! I'm not the right person to review this, but I can keep an eye on our buildbots when it lands.

[ruiu]

LGTM

llvm/trunk/include/llvm/Object/ArchiveWriter.h

 namespace llvm {
 
 struct NewArchiveMember {
   std::unique_ptr<MemoryBuffer> Buf;
   StringRef MemberName;
   sys::TimePoint<std::chrono::seconds> ModTime;
   unsigned UID = 0, GID = 0, Perms = 0644;
 
-  bool IsNew = false;
   NewArchiveMember() = default;
   NewArchiveMember(MemoryBufferRef BufRef);
 
   static Expected<NewArchiveMember>
   getOldMember(const object::Archive::Child &OldMember, bool Deterministic);
 
   static Expected<NewArchiveMember> getFile(StringRef FileName,
                                             bool Deterministic);

llvm/trunk/lib/Object/ArchiveWriter.cpp

 Expected<NewArchiveMember>
 NewArchiveMember::getOldMember(const object::Archive::Child &OldMember,
                                bool Deterministic) {
   Expected<llvm::MemoryBufferRef> BufOrErr = OldMember.getMemoryBufferRef();
   if (!BufOrErr)
     return BufOrErr.takeError();
 
   NewArchiveMember M;
-  assert(M.IsNew == false);
   M.Buf = MemoryBuffer::getMemBuffer(*BufOrErr, false);
   M.MemberName = M.Buf->getBufferIdentifier();
   if (!Deterministic) {
     auto ModTimeOrErr = OldMember.getLastModified();
     if (!ModTimeOrErr)
       return ModTimeOrErr.takeError();
     M.ModTime = ModTimeOrErr.get();
     Expected<unsigned> UIDOrErr = OldMember.getUID();
ErrorOr<std::unique_ptr<MemoryBuffer>> MemberBufferOrErr =
       MemoryBuffer::getOpenFile(FD, FileName, Status.getSize(), false);
   if (!MemberBufferOrErr)
     return errorCodeToError(MemberBufferOrErr.getError());
 
   if (close(FD) != 0)
     return errorCodeToError(std::error_code(errno, std::generic_category()));
 
   NewArchiveMember M;
-  M.IsNew = true;
   M.Buf = std::move(*MemberBufferOrErr);
   M.MemberName = M.Buf->getBufferIdentifier();
   if (!Deterministic) {
     M.ModTime = std::chrono::time_point_cast<std::chrono::seconds>(
         Status.getLastModificationTime());
     M.UID = Status.getUser();
     M.GID = Status.getGroup();
     M.Perms = Status.permissions();
printBSDMemberHeader(raw_ostream &Out, uint64_t Pos, StringRef Name,
   while (Pad--)
     Out.write(uint8_t(0));
 }
 
 static bool useStringTable(bool Thin, StringRef Name) {
   return Thin || Name.size() >= 16 || Name.contains('/');
 }
 
-// Compute the relative path from From to To.
-static std::string computeRelativePath(StringRef From, StringRef To) {
-  if (sys::path::is_absolute(From) || sys::path::is_absolute(To))
-    return To;
-
-  StringRef DirFrom = sys::path::parent_path(From);
-  auto FromI = sys::path::begin(DirFrom);
-  auto ToI = sys::path::begin(To);
-  while (*FromI == *ToI) {
-    ++FromI;
-    ++ToI;
-  }
-
-  SmallString<128> Relative;
-  for (auto FromE = sys::path::end(DirFrom); FromI != FromE; ++FromI)
-    sys::path::append(Relative, "..");
-
-  for (auto ToE = sys::path::end(To); ToI != ToE; ++ToI)
-    sys::path::append(Relative, *ToI);
-
-#ifdef _WIN32
-  // Replace backslashes with slashes so that the path is portable between *nix
-  // and Windows.
-  std::replace(Relative.begin(), Relative.end(), '\\', '/');
-#endif
-
-  return Relative.str();
-}
-
 static bool is64BitKind(object::Archive::Kind Kind) {
   switch (Kind) {
   case object::Archive::K_GNU:
   case object::Archive::K_BSD:
   case object::Archive::K_DARWIN:
   case object::Archive::K_COFF:
     return false;
   case object::Archive::K_DARWIN64:
   case object::Archive::K_GNU64:
     return true;
   }
   llvm_unreachable("not supported for writting");
 }
 
-static void addToStringTable(raw_ostream &Out, StringRef ArcName,
-                             const NewArchiveMember &M, bool Thin) {
-  StringRef ID = M.Buf->getBufferIdentifier();
-  if (Thin) {
-    if (M.IsNew)
-      Out << computeRelativePath(ArcName, ID);
-    else
-      Out << ID;
-  } else
-    Out << M.MemberName;
-  Out << "/\n";
-}
-
-static void printMemberHeader(raw_ostream &Out, uint64_t Pos,
-                              raw_ostream &StringTable,
-                              StringMap<uint64_t> &MemberNames,
-                              object::Archive::Kind Kind, bool Thin,
-                              StringRef ArcName, const NewArchiveMember &M,
-                              sys::TimePoint<std::chrono::seconds> ModTime,
-                              unsigned Size) {
-
+static void
+printMemberHeader(raw_ostream &Out, uint64_t Pos, raw_ostream &StringTable,
+                  StringMap<uint64_t> &MemberNames, object::Archive::Kind Kind,
+                  bool Thin, const NewArchiveMember &M,
+                  sys::TimePoint<std::chrono::seconds> ModTime, unsigned Size) {
   if (isBSDLike(Kind))
     return printBSDMemberHeader(Out, Pos, M.MemberName, ModTime, M.UID, M.GID,
                                 M.Perms, Size);
   if (!useStringTable(Thin, M.MemberName))
     return printGNUSmallMemberHeader(Out, M.MemberName, ModTime, M.UID, M.GID,
                                      M.Perms, Size);
   Out << '/';
   uint64_t NamePos;
   if (Thin) {
     NamePos = StringTable.tell();
-    addToStringTable(StringTable, ArcName, M, Thin);
+    StringTable << M.MemberName << "/\n";
   } else {
     auto Insertion = MemberNames.insert({M.MemberName, uint64_t(0)});
     if (Insertion.second) {
       Insertion.first->second = StringTable.tell();
-      addToStringTable(StringTable, ArcName, M, Thin);
+      StringTable << M.MemberName << "/\n";
     }
     NamePos = Insertion.first->second;
   }
   printWithSpacePadding(Out, NamePos, 15);
   printRestOfMemberHeader(Out, ModTime, M.UID, M.GID, M.Perms, Size);
 }
 
 namespace {
if (auto EC = S.printName(SymNames))
       return errorCodeToError(EC);
     SymNames << '\0';
   }
   return Ret;
 }
 
 static Expected<std::vector<MemberData>>
 computeMemberData(raw_ostream &StringTable, raw_ostream &SymNames,
-                  object::Archive::Kind Kind, bool Thin, StringRef ArcName,
-                  bool Deterministic, ArrayRef<NewArchiveMember> NewMembers) {
+                  object::Archive::Kind Kind, bool Thin, bool Deterministic,
+                  ArrayRef<NewArchiveMember> NewMembers) {
   static char PaddingData[8] = {'\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n'};
 
   // This ignores the symbol table, but we only need the value mod 8 and the
   // symbol table is aligned to be a multiple of 8 bytes
   uint64_t Pos = 0;
 
   std::vector<MemberData> Ret;
   bool HasObject = false;
for (const NewArchiveMember &M : NewMembers) {
     StringRef Padding = StringRef(PaddingData, MemberPadding + TailPadding);
 
     sys::TimePoint<std::chrono::seconds> ModTime;
     if (UniqueTimestamps)
       // Increment timestamp for each file of a given name.
       ModTime = sys::toTimePoint(FilenameCount[M.MemberName]++);
     else
       ModTime = M.ModTime;
-    printMemberHeader(Out, Pos, StringTable, MemberNames, Kind, Thin, ArcName,
-                      M, ModTime, Buf.getBufferSize() + MemberPadding);
+    printMemberHeader(Out, Pos, StringTable, MemberNames, Kind, Thin, M,
+                      ModTime, Buf.getBufferSize() + MemberPadding);
     Out.flush();
 
     Expected<std::vector<unsigned>> Symbols =
         getSymbols(Buf, SymNames, HasObject);
     if (auto E = Symbols.takeError())
       return std::move(E);
 
     Pos += Header.size() + Data.size() + Padding.size();
Error llvm::writeArchive(StringRef ArcName,
   assert((!Thin || !isBSDLike(Kind)) && "Only the gnu format has a thin mode");
 
   SmallString<0> SymNamesBuf;
   raw_svector_ostream SymNames(SymNamesBuf);
   SmallString<0> StringTableBuf;
   raw_svector_ostream StringTable(StringTableBuf);
 
   Expected<std::vector<MemberData>> DataOrErr = computeMemberData(
-      StringTable, SymNames, Kind, Thin, ArcName, Deterministic, NewMembers);
+      StringTable, SymNames, Kind, Thin, Deterministic, NewMembers);
   if (Error E = DataOrErr.takeError())
     return E;
   std::vector<MemberData> &Data = *DataOrErr;
 
   if (!StringTableBuf.empty())
     Data.insert(Data.begin(), computeStringTable(StringTableBuf));
 
   // We would like to detect if we need to switch to a 64-bit symbol table.

llvm/trunk/test/tools/llvm-ar/flatten-thin-archive-directories.test

+# This test creates a thin archive in a directory and adds it to a thin archive
+# in the parent directory. The relative path should be included when flattening
+# the archive.
+
+RUN: mkdir -p %t/foo
+RUN: touch %t/foo/a.txt
+RUN: rm -f %t/archive.a %t/foo/archive.a
+
+# These tests must be run in the same directory as %t/archive.a. cd %t is
+# included on each line to make debugging this test case easier.
+RUN: cd %t && llvm-ar rcST foo/archive.a foo/a.txt
+RUN: cd %t && llvm-ar rcST archive.a foo/archive.a
+RUN: cd %t && llvm-ar t archive.a | FileCheck %s --match-full-lines
+
+CHECK: foo/a.txt

llvm/trunk/test/tools/llvm-ar/flatten-thin-archive.test

 # This test creates a thin archive that contains a thin archive, a regular
 # archive, and a file.
 #
 # The inner thin archive should be flattened, but the regular archive should
 # not. The order of members in the archive should match the input order, with
 # flattened members appearing together.
 
 RUN: touch %t-a.txt %t-b.txt %t-c.txt %t-d.txt %t-e.txt
-RUN: rm -f %t-a-plus-b.a %t.a
+RUN: rm -f %t-a-plus-b.a %t-d-plus-e.a %t.a
 RUN: llvm-ar rcsT %t-a-plus-b.a %t-a.txt %t-b.txt
 RUN: llvm-ar rcs %t-d-plus-e.a %t-d.txt %t-e.txt
 RUN: llvm-ar rcsT %t.a %t-a-plus-b.a %t-c.txt %t-d-plus-e.a
 RUN: llvm-ar t %t.a | FileCheck %s
 
 CHECK:      a.txt
 CHECK-NEXT: b.txt
 CHECK-NEXT: c.txt
 CHECK-NEXT: -d-plus-e.a

llvm/trunk/tools/llvm-ar/llvm-ar.cpp

 // This variable holds the name of the archive file as given on the
 // command line.
 static std::string ArchiveName;
 
 // This variable holds the list of member files to proecess, as given
 // on the command line.
 static std::vector<StringRef> Members;
 
+// Static buffer to hold StringRefs.
+static BumpPtrAllocator Alloc;
+
 // Extract the member filename from the command line for the [relpos] argument
 // associated with a, b, and i modifiers
 static void getRelPos() {
   if (PositionalArgs.empty())
     fail("Expected [relpos] for a, b, or i modifier");
   RelPos = PositionalArgs[0];
   PositionalArgs.erase(PositionalArgs.begin());
 }
static void performReadOperation(ArchiveOperation Operation,
 
   if (Members.empty())
     return;
   for (StringRef Name : Members)
     WithColor::error(errs(), ToolName) << "'" << Name << "' was not found\n";
   exit(1);
 }
 
+// Compute the relative path from From to To.
+static std::string computeRelativePath(StringRef From, StringRef To) {
+  if (sys::path::is_absolute(From) || sys::path::is_absolute(To))
+    return To;
+
+  StringRef DirFrom = sys::path::parent_path(From);
+  auto FromI = sys::path::begin(DirFrom);
+  auto ToI = sys::path::begin(To);
+  while (*FromI == *ToI) {
+    ++FromI;
+    ++ToI;
+  }
+
+  SmallString<128> Relative;
+  for (auto FromE = sys::path::end(DirFrom); FromI != FromE; ++FromI)
+    sys::path::append(Relative, "..");
+
+  for (auto ToE = sys::path::end(To); ToI != ToE; ++ToI)
+    sys::path::append(Relative, *ToI);
+
+#ifdef _WIN32
+  // Replace backslashes with slashes so that the path is portable between *nix
+  // and Windows.
+  std::replace(Relative.begin(), Relative.end(), '\\', '/');
+#endif
+
+  return Relative.str();
+}
+
 static void addChildMember(std::vector<NewArchiveMember> &Members,
                            const object::Archive::Child &M,
                            bool FlattenArchive = false) {
   if (Thin && !M.getParent()->isThin())
     fail("Cannot convert a regular archive to a thin one");
   Expected<NewArchiveMember> NMOrErr =
       NewArchiveMember::getOldMember(M, Deterministic);
   failIfError(NMOrErr.takeError());
+  // If the child member we're trying to add is thin, use the path relative to
+  // the archive it's in, so the file resolves correctly.
+  if (Thin && FlattenArchive) {
+    StringSaver Saver(Alloc);
+    Expected<std::string> FileNameOrErr = M.getFullName();
+    failIfError(FileNameOrErr.takeError());
+    NMOrErr->MemberName =
+        Saver.save(computeRelativePath(ArchiveName, *FileNameOrErr));
+  }
   if (FlattenArchive &&
       identify_magic(NMOrErr->Buf->getBuffer()) == file_magic::archive) {
     Expected<std::string> FileNameOrErr = M.getFullName();
     failIfError(FileNameOrErr.takeError());
     object::Archive &Lib = readLibrary(*FileNameOrErr);
     // When creating thin archives, only flatten if the member is also thin.
     if (!Thin || Lib.isThin()) {
       Error Err = Error::success();
       // Only Thin archives are recursively flattened.
       for (auto &Child : Lib.children(Err))
         addChildMember(Members, Child, /*FlattenArchive=*/Thin);
       failIfError(std::move(Err));
       return;
     }
   }
   Members.push_back(std::move(*NMOrErr));
 }
 
 static void addMember(std::vector<NewArchiveMember> &Members,
                       StringRef FileName, bool FlattenArchive = false) {
   Expected<NewArchiveMember> NMOrErr =
       NewArchiveMember::getFile(FileName, Deterministic);
   failIfError(NMOrErr.takeError(), FileName);
+  StringSaver Saver(Alloc);
+  // For regular archives, use the basename of the object path for the member
+  // name. For thin archives, use the full relative paths so the file resolves
+  // correctly.
+  NMOrErr->MemberName =
+      Thin ? Saver.save(computeRelativePath(ArchiveName, FileName))
+           : sys::path::filename(NMOrErr->MemberName);
   if (FlattenArchive &&
       identify_magic(NMOrErr->Buf->getBuffer()) == file_magic::archive) {
     object::Archive &Lib = readLibrary(FileName);
     // When creating thin archives, only flatten if the member is also thin.
     if (!Thin || Lib.isThin()) {
       Error Err = Error::success();
       // Only Thin archives are recursively flattened.
       for (auto &Child : Lib.children(Err))
         addChildMember(Members, Child, /*FlattenArchive=*/Thin);
       failIfError(std::move(Err));
       return;
     }
   }
-  // Use the basename of the object path for the member name.
-  NMOrErr->MemberName = sys::path::filename(NMOrErr->MemberName);
   Members.push_back(std::move(*NMOrErr));
 }
 
 enum InsertAction {
   IA_AddOldMember,
   IA_AddNewMember,
   IA_Delete,
   IA_MoveOldMember,
for (auto &Child : OldArchive->children(Err)) {
           InsertPos = Pos + 1;
       }
 
       std::vector<StringRef>::iterator MemberI = Members.end();
       InsertAction Action =
           computeInsertAction(Operation, Child, Name, MemberI);
       switch (Action) {
       case IA_AddOldMember:
-        addChildMember(Ret, Child);
+        addChildMember(Ret, Child, /*FlattenArchive=*/Thin);
         break;
       case IA_AddNewMember:
         addMember(Ret, *MemberI);
         break;
       case IA_Delete:
         break;
       case IA_MoveOldMember:
-        addChildMember(Moved, Child);
+        addChildMember(Moved, Child, /*FlattenArchive=*/Thin);
         break;
       case IA_MoveNewMember:
         addMember(Moved, *MemberI);
         break;
       }
       if (MemberI != Members.end())
         Members.erase(MemberI);
     }
auto Command = StringSwitch<MRICommand>(CommandStr.lower())
                        .Default(MRICommand::Invalid);
 
     switch (Command) {
     case MRICommand::AddLib: {
       object::Archive &Lib = readLibrary(Rest);
       {
         Error Err = Error::success();
         for (auto &Member : Lib.children(Err))
-          addChildMember(NewMembers, Member);
+          addChildMember(NewMembers, Member, /*FlattenArchive=*/Thin);
         failIfError(std::move(Err));
       }
       break;
     }
     case MRICommand::AddMod:
       addMember(NewMembers, Rest);
       break;
     case MRICommand::Create:
if (arg == "-version" || arg == "--version") {
     cl::PrintVersionMessage();
     return true;
   }
   return false;
 }
 
 static int ar_main(int argc, char **argv) {
   SmallVector<const char *, 0> Argv(argv, argv + argc);
-  BumpPtrAllocator Alloc;
   StringSaver Saver(Alloc);
   cl::ExpandResponseFiles(Saver, cl::TokenizeGNUCommandLine, Argv);
   for (size_t i = 1; i < Argv.size(); ++i) {
     StringRef Arg = Argv[i];
     const char *match;
     auto MatchFlagWithArg = [&](const char *expected) {
       size_t len = strlen(expected);
       if (Arg == expected) {