Add a class to create a tar archive file.

[ruiu]

Summary

In LLD, we create cpio archive files for --reproduce command.
cpio was not a bad choice because it is very easy to create, but
it was sometimes hard to use because people are not familiar with
cpio command.

I noticed that creating a tar archive isn't as hard as I thought.
So I implemented it in this patch.

Create a tar archive instead of a cpio for --reproduce.

[silvas]

This is a good idea. Tar is a much more common format that users will be more familiar with. I'm not an expert on Tar files, so I can't review that part of the patch, but hopefully the bots will show if any tar implementations don't like the files we produce.

Sorry, I know this is a bit unrelated to this patch per se, but one thing that has always bothered me is that --reproduce foo doesn't produce a file called foo; this is very non-unixy. I think it sort of made sense for cpio since that is a less known format, but I think that .tar is common enough that we can expect people to write --reproduce foo.tar. That makes any examples self-documenting (users will read --reproduce repro.tar and understand that it creates a tar file; rather than --reproduce repro and not know what file format it is). What do you think?

[ruiu]

• Split into two patches for LLVM and LLD.

[ruiu]

I think not adding an extension is a good idea. I made that change in D28103.

llvm/trunk/include/llvm/Support/TarWriter.h

+//===-- llvm/Support/TarWriter.h - Tar archive file creator -----*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_TAR_WRITER_H
+#define LLVM_SUPPORT_TAR_WRITER_H
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/raw_ostream.h"
+
+namespace llvm {
+class TarWriter {
+public:
+  static Expected<std::unique_ptr<TarWriter>> create(StringRef OutputPath,
+                                                     StringRef BaseDir);
+
+  void append(StringRef Path, StringRef Data);
+
+private:
+  TarWriter(int FD, StringRef BaseDir);
+  raw_fd_ostream OS;
+  std::string BaseDir;
+};
+}
+
+#endif

llvm/trunk/lib/Support/CMakeLists.txt

add_llvm_library(LLVMSupport
   SpecialCaseList.cpp
   Statistic.cpp
   StringExtras.cpp
   StringMap.cpp
   StringPool.cpp
   StringSaver.cpp
   StringRef.cpp
   SystemUtils.cpp
+  TarWriter.cpp
   TargetParser.cpp
   ThreadPool.cpp
   Timer.cpp
   ToolOutputFile.cpp
   TrigramIndex.cpp
   Triple.cpp
   Twine.cpp
   Unicode.cpp

llvm/trunk/lib/Support/TarWriter.cpp

+//===-- TarWriter.cpp - Tar archive file creator --------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// TarWriter class provides a feature to create a tar archive file.
+//
+// I put emphasis on simplicity over comprehensiveness when
+// implementing this class because we don't need a full-fledged
+// archive file generator in LLVM at the moment.
+//
+// The filename field in the Unix V7 tar header is 100 bytes, which is
+// apparently too small. Various extensions were proposed and
+// implemented to fix the issue. The writer implemented in this file
+// uses PAX extension headers.
+//
+// Note that we emit PAX headers even if filenames fit in the V7
+// header for the sake of simplicity. So, generated files are N
+// kilobyte larger than the ideal where N is the number of files in
+// archives. In practice, I think you don't need to worry about that.
+//
+// The PAX header is standardized in IEEE Std 1003.1-2001.
+//
+// The struct definition of UstarHeader is copied from
+// https://www.freebsd.org/cgi/man.cgi?query=tar&sektion=5
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/TarWriter.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/MathExtras.h"
+
+using namespace llvm;
+
+// Each file in an archive must be aligned to this block size.
+static const int BlockSize = 512;
+
+struct UstarHeader {
+  char Name[100];
+  char Mode[8];
+  char Uid[8];
+  char Gid[8];
+  char Size[12];
+  char Mtime[12];
+  char Checksum[8];
+  char TypeFlag;
+  char Linkname[100];
+  char Magic[6];
+  char Version[2];
+  char Uname[32];
+  char Gname[32];
+  char DevMajor[8];
+  char DevMinor[8];
+  char Prefix[155];
+  char Pad[12];
+};
+static_assert(sizeof(UstarHeader) == BlockSize, "invalid Ustar header");
+
+// A PAX attribute is in the form of "<length> <key>=<value>\n"
+// where <length> is the length of the entire string including
+// the length field itself. An example string is this.
+//
+//   25 ctime=1084839148.1212\n
+//
+// This function create such string.
+static std::string formatPax(StringRef Key, const Twine &Val) {
+  int Len = Key.size() + Val.str().size() + 3; // +3 for " ", "=" and "\n"
+
+  // We need to compute total size twice because appending
+  // a length field could change total size by one.
+  int Total = Len + Twine(Len).str().size();
+  Total = Len + Twine(Total).str().size();
+  return (Twine(Total) + " " + Key + "=" + Val + "\n").str();
+}
+
+// Headers in tar files must be aligned to 512 byte boundaries.
+// This function writes null bytes so that the file is a multiple
+// of 512 bytes.
+static void pad(raw_fd_ostream &OS) {
+  uint64_t Pos = OS.tell();
+  OS.seek(alignTo(Pos, BlockSize));
+}
+
+// Computes a checksum for a tar header.
+static void computeChecksum(UstarHeader &Hdr) {
+  // Before computing a checksum, checksum field must be
+  // filled with space characters.
+  memset(Hdr.Checksum, ' ', sizeof(Hdr.Checksum));
+
+  // Compute a checksum and set it to the checksum field.
+  unsigned Chksum = 0;
+  for (size_t I = 0; I < sizeof(Hdr); ++I)
+    Chksum += reinterpret_cast<uint8_t *>(&Hdr)[I];
+  sprintf(Hdr.Checksum, "%06o", Chksum);
+}
+
+// Create a tar header and write it to a given output stream.
+static void writePaxHeader(raw_fd_ostream &OS, const Twine &Path) {
+  // A PAX header consists of a 512-byte header followed
+  // by key-value strings. First, create key-value strings.
+  std::string PaxAttr = formatPax("path", Path);
+
+  // Create a 512-byte header.
+  UstarHeader Hdr = {};
+  sprintf(Hdr.Size, "%011lo", PaxAttr.size());
+  Hdr.TypeFlag = 'x';            // PAX magic
+  memcpy(Hdr.Magic, "ustar", 6); // Ustar magic
+  computeChecksum(Hdr);
+
+  // Write them down.
+  OS << StringRef(reinterpret_cast<char *>(&Hdr), sizeof(Hdr));
+  OS << PaxAttr;
+  pad(OS);
+}
+
+// The PAX header is an extended format, so a PAX header needs
+// to be followed by a "real" header.
+static void writeUstarHeader(raw_fd_ostream &OS, size_t Size) {
+  UstarHeader Hdr = {};
+  strcpy(Hdr.Mode, "0000664");
+  sprintf(Hdr.Size, "%011lo", Size);
+  memcpy(Hdr.Magic, "ustar", 6);
+
+  computeChecksum(Hdr);
+  OS << StringRef(reinterpret_cast<char *>(&Hdr), sizeof(Hdr));
+}
+
+// We want to use '/' as a path separator even on Windows.
+// This function canonicalizes a given path.
+static std::string canonicalize(std::string S) {
+#ifdef LLVM_ON_WIN32
+  std::replace(S.begin(), S.end(), '\\', '/');
+#endif
+  return S;
+}
+
+// Creates a TarWriter instance and returns it.
+Expected<std::unique_ptr<TarWriter>> TarWriter::create(StringRef OutputPath,
+                                                       StringRef BaseDir) {
+  int FD;
+  if (std::error_code EC = openFileForWrite(OutputPath, FD, sys::fs::F_None))
+    return make_error<StringError>("cannot open " + OutputPath, EC);
+  return std::unique_ptr<TarWriter>(new TarWriter(FD, BaseDir));
+}
+
+TarWriter::TarWriter(int FD, StringRef BaseDir)
+    : OS(FD, /*shouldClose=*/true, /*unbuffered=*/false), BaseDir(BaseDir) {}
+
+// Append a given file to an archive.
+void TarWriter::append(StringRef Path, StringRef Data) {
+  // Write Path and Data.
+  writePaxHeader(OS, BaseDir + "/" + canonicalize(Path));
+  writeUstarHeader(OS, Data.size());
+  OS << Data;
+  pad(OS);
+
+  // POSIX requires tar archives end with two null blocks.
+  // Here, we write the terminator and then seek back, so that
+  // the file being output is terminated correctly at any moment.
+  uint64_t Pos = OS.tell();
+  OS << std::string(BlockSize * 2, '\0');
+  OS.seek(Pos);
+  OS.flush();
+}