[tools]Introduce llvm-lipo

[alexander-shaposhnikov]

Summary

This diff starts the implementation of llvm-lipo which is supposed to be a drop-in replacement for the well-known tool lipo.
Lipo manipulates fat binaries, in particular, it can create a "fat" binary, extract regular object files from a fat binary,
report some useful information about the input.

In this diff to get things started we implement the option -verify_arch.
Next we are planning to contribute the implementation of

"-archs": (along the lines of the current diff).

"-thin": (takes a fat binary and writes the "slice" for a particular architecture into a separate file "as is". libObject's functionality is enough for it.

"-create": (takes multiple regular mach-o object files for different architectures and generates a "fat" binary, we are planning to use the standard libObject functionality for reading/inspecting/validating the input files,
populate MachO::fat_header and MachO::fat_arch structures and write them and the slices "as is".

Test plan:
make check-all

[mtrent]

I hate to be "That Guy" but this really needs a docs/CommandGuide file.

I'm super-excited to see this work. This is shall we say "a really bad time of year" to get a thoughtful review from me ( for some reason :p ) but I'll move this to the top of my pile, circumstances permitting.

[smeenai]

Looks good, though I'm not super familiar with the libObject APIs (yet) :)

test/tools/llvm-lipo/verify-arch-macho-binary.test

3 ↗

(On Diff #199542)

lipo only supports this form of specifying flags (single dash, no equals). Do we want to support the others?

22 ↗

(On Diff #199542)

Can we strip this down as much as we can? E.g. https://reviews.llvm.org/diffusion/L/browse/llvm/trunk/test/tools/llvm-objdump/AArch64/Inputs/arm64e.macho.yaml

test/tools/llvm-lipo/verify-arch-universal-binary.test

10 ↗

(On Diff #199542)

Same comment about minimizing this as much as possible.

tools/llvm-lipo/llvm-lipo.cpp

33 ↗

(On Diff #199542)

Shouldn't this be OneOrMore?

74 ↗

(On Diff #199542)

Would it be better to use a SmallVectorImpl as the parameter type?

81 ↗

(On Diff #199542)

Super nit: I think this would read more naturally if the left and right hand sides of the == were switched.

82 ↗

(On Diff #199542)

Do we need the explicit Twine constructor here? I thought there was an operator+ for const char * and StringRef.

95 ↗

(On Diff #199542)

Triple(Arch).getArch() is used a bunch. Should we save it to a variable?

[alexander-shaposhnikov]

will update this diff tomorrow

[mtrent]

A second thought on the topic of lipo compatibility. libObject is probably reasonable if your primary interest is MH_OBJECT files. There exist a number of final-linked-images that libObject's fat parser will not currently understand; these are formats that cannot be produced by llvm tools today. I wonder if choosing a different name for the tool might give you some flexibility to "do things differently" than lipo. That said, I don't have a better name in mind. :)

test/tools/llvm-lipo/verify-arch-macho-binary.test

3 ↗

(On Diff #199542)

I don't see a problem with supporting "--" options here, from the point of view of lipo compatibility. I say, do it.

There are other ways where llvm-lipo is going to be an 'awkward fit' as a drop-in replacement for lipo. For example, lipo is a little unusual in that when -verify_arch is specified, all the remaining options are interpreted as architectures. So llvm-lipo interprets "-verify_arch x86_64 /bin/ls" as "verify /bin/ls has an arch for x86_64 in it" but lipo itself says [paraphrasing] "/bin/ls is not an architecture." Either you will type "input_file -verify_arch x86_64 i386" or you will type "-verify_arch x86_64 -verify_arch i386 input_file" (where input_file could appear anywhere), and whichever usage you specify will be specific either to "llvm-lipo" or the historical lipo.

Pragmatically I think you're going to want to get as close as you can, to avoid confusion. But recognize there are going to be differences if you want to use llvm's argument parsing.

tools/llvm-lipo/llvm-lipo.cpp

33 ↗

(On Diff #199542)

No. -verify_arch is one of several mutually-exclusive lipo commands. You cannot specify both -verify_arch and -info for example. So ZeroOrMore is correct, and the llvm-lipo command driver is going to have to enforce this exclusivity.

95 ↗

(On Diff #199542)

Do you mean "O->getArch()"? I agree, if so.

96 ↗

(On Diff #199542)

the lipo -verify_arch command does not print an error if the command is able to run successfully, as it's primarily for using with shell scripts and Makefiles. Is a verbose message really necessary here? If less, please implement a -non_verbose option.

[mtrent]

tools/llvm-lipo/llvm-lipo.cpp

96 ↗

(On Diff #199542)

Sigh. s/If less/If yes/;

[alexander-shaposhnikov]

Address comments, have not added docs in this commit yet, can do as a follow-up or will update this diff a bit later.

[alexander-shaposhnikov]

regarding -verify_arch X Y Z - yeah, I can try to make it work with LLVM's command line options parser, not sure if this interface is ideal / better than -verify_arch X -verify_arch Y -verify_arch Z, but yeah, on the other hand, this would be an incompatibility / would require customers of the old tool to update their scripts/code. The downside of the old interface - the positional argument should go before -verify_arch. To be honest at the moment I don't see how to express the old behavior of -verify_arch using libCommandLine, any suggestions would be helpful, alternatively we can try to live with this incompatibility.

[mtrent]

In D61927#1503871, @alexshap wrote:

regarding -verify_arch X Y Z - yeah, I can try to make it work with LLVM's command line options parser, not sure if this interface is ideal / better than -verify_arch X -verify_arch Y -verify_arch Z, but yeah, on the other hand, this would be an incompatibility / would require customers of the old tool to update their scripts/code. The downside of the old interface - the positional argument should go before -verify_arch. To be honest at the moment I don't see how to express the old behavior of -verify_arch using libCommandLine, any suggestions would be helpful, alternatively we can try to live with this incompatibility.

Well, let's have a look at Apple's command syntax. I re-wrote the man page for the latest release, so it's a little clearer to see what's going on. I'll summarize here.

These commands/options take no arguments and are no trouble at all for llvm's command-line parser: -archs, -detailed_info, -info, -hideARM64

These commands/options take a single argument, and only one of these options is allowed. Again, easy mode: -thin, -output.

These commands take a single argument, and any number of them can appear (as long as this is the only such command that appears). Easy mode, you have to handle the command exclusivity yourself in your command driver anyway, so that's not really any extra work: -extract, -extract_family, -remove,

These commands / options take a fixed number of arguments, where n > 1: -create, -replace, -arch, -segalign. I'm under the impression this is something llvm does not support of the box, because we see this problem wtih "llvm-nm -s". But it's possible that the "AdditionalVals" parameter on class Option is meant to deal with this situation. But, if so, no one has circled back to llvm-nm. Some of these commands are pretty critical.

And this command is the worst of the bunch, taking any number of arguments. It's also the one you implemented first :): -verify_arch.

Looking ahead to your task of making a drop-in replacement for Apple's lipo, you're going to have to tackle -verify_arch, -create, -replace, -arch, and -segalign. That's a non-trivial surface area of the lipo tool. My advice is to see if you can model the complete set of lipo command-line options you want to reverse-engineer -- don't worry about implementing the feature, just parse the arguments. If -verify_arch is the only one that doesn't fit the mold, I think "just live with it" is the right call. But if you can't get -create to work with an arbitrary number of -arch or -arch_blank options, or if you can't get -replace working, you're going to have trouble making a viable "drop-in-replacement."

HTH!

[alexander-shaposhnikov]

@mtrent, good call, will have a look and update this diff

[alexander-shaposhnikov]

This comment was removed by alexander-shaposhnikov.

[alexander-shaposhnikov]

Use a bigger hammer to parse command line options

[mtrent]

That's a pretty big hammer. Let's give it a try!

llvm/trunk/test/CMakeLists.txt

set(LLVM_TEST_DEPENDS
           llvm-dwp
           llvm-elfabi
           llvm-exegesis
           llvm-extract
           llvm-isel-fuzzer
           llvm-jitlink
           llvm-lib
           llvm-link
+          llvm-lipo
           llvm-lto2
           llvm-mc
           llvm-mca
           llvm-modextract
           llvm-mt
           llvm-nm
           llvm-objcopy
           llvm-objdump

llvm/trunk/test/tools/llvm-lipo/help-message.test

+# RUN: llvm-lipo -h | FileCheck --check-prefix=LIPO-USAGE %s
+# RUN: llvm-lipo --help | FileCheck --check-prefix=LIPO-USAGE %s
+
+# RUN: llvm-lipo -version | FileCheck --check-prefix=LIPO-VERSION %s
+# RUN: llvm-lipo --version | FileCheck --check-prefix=LIPO-VERSION %s
+
+# RUN: not llvm-lipo 2>&1 | FileCheck --check-prefix=LIPO-USAGE %s
+# RUN: not llvm-lipo -abcabc 2>&1 | FileCheck --check-prefix=LIPO-UNKNOWN-ARG %s
+# RUN: not llvm-lipo --abcabc 2>&1 | FileCheck --check-prefix=LIPO-UNKNOWN-ARG %s
+
+# LIPO-USAGE:    USAGE: llvm-lipo
+# LIPO-UNKNOWN-ARG:    unknown argument '{{-+}}abcabc'
+# LIPO-VERSION: {{ version }}

llvm/trunk/test/tools/llvm-lipo/verify-arch-macho-binary.test

+# RUN: yaml2obj %s > %t
+
+# RUN: llvm-lipo %t -verify_arch i386
+# RUN: llvm-lipo %t --verify_arch i386
+
+# RUN: not llvm-lipo %t -verify_arch aarch64
+# RUN: not llvm-lipo %t -verify_arch aarch64 i386 
+
+# INVALID_ARCH: Invalid architecture: aarch101
+# RUN: not llvm-lipo %t -verify_arch aarch101 2>&1 | FileCheck --check-prefix=INVALID_ARCH %s
+
+# INVALID_OBJ: The file was not recognized as a valid object file
+# RUN: touch %t.empty
+# RUN: not llvm-lipo %t.empty -verify_arch aarch101 2>&1 | FileCheck --check-prefix=INVALID_OBJ %s
+
+# NO_INPUT_OBJ: at least one input file should be specified
+# RUN: not llvm-lipo -verify_arch i386 2>&1 | FileCheck --check-prefix=NO_INPUT_OBJ %s
+
+# MULTIPLE_INPUT_OBJ: verify_arch expects a single input file
+# RUN: not llvm-lipo %t %t -verify_arch i386 2>&1 | FileCheck --check-prefix=MULTIPLE_INPUT_OBJ %s
+
+--- !mach-o
+FileHeader:      
+  magic:           0xFEEDFACE
+  cputype:         0x00000007
+  cpusubtype:      0x00000003
+  filetype:        0x00000001
+  ncmds:           0
+  sizeofcmds:      0
+  flags:           0x00002000
+...

llvm/trunk/test/tools/llvm-lipo/verify-arch-universal-binary.test

+# RUN: yaml2obj %s > %t
+
+# RUN: llvm-lipo %t -verify_arch i386
+# RUN: llvm-lipo %t -verify_arch i386 x86_64
+
+# RUN: not llvm-lipo %t -verify_arch aarch64
+# RUN: not llvm-lipo %t -verify_arch aarch64 i386 
+
+--- !fat-mach-o
+FatHeader:       
+  magic:           0xCAFEBABE
+  nfat_arch:       2
+FatArchs:        
+  - cputype:         0x00000007
+    cpusubtype:      0x00000003
+    offset:          0x0000000000001000
+    size:            28
+    align:           12
+  - cputype:         0x01000007
+    cpusubtype:      0x00000003
+    offset:          0x0000000000002000
+    size:            32
+    align:           12
+Slices:          
+  - !mach-o
+    FileHeader:      
+      magic:           0xFEEDFACE
+      cputype:         0x00000007
+      cpusubtype:      0x00000003
+      filetype:        0x00000001
+      ncmds:           0
+      sizeofcmds:      0
+      flags:           0x00002000
+  - !mach-o
+    FileHeader:      
+      magic:           0xFEEDFACF
+      cputype:         0x01000007
+      cpusubtype:      0x00000003
+      filetype:        0x00000001
+      ncmds:           0
+      sizeofcmds:      0
+      flags:           0x00002000
+      reserved:        0x00000000
+...

llvm/trunk/tools/llvm-lipo/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  ${LLVM_TARGETS_TO_BUILD}
+  Object
+  Option
+  Support
+)
+
+set(LLVM_TARGET_DEFINITIONS LipoOpts.td)
+tablegen(LLVM LipoOpts.inc -gen-opt-parser-defs)
+add_public_tablegen_target(LipoOptsTableGen)
+
+add_llvm_tool(llvm-lipo
+  llvm-lipo.cpp
+  DEPENDS
+  LipoOptsTableGen
+)

llvm/trunk/tools/llvm-lipo/LLVMBuild.txt

+;===- ./tools/llvm-lipo/LLVMBuild.txt --------------------------*- Conf -*--===;
+;
+; Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+; See https://llvm.org/LICENSE.txt for license information.
+; SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+;
+;===------------------------------------------------------------------------===;
+;
+; This is an LLVMBuild description file for the components in this subdirectory.
+;
+; For more information on the LLVMBuild system, please see:
+;
+;   http://llvm.org/docs/LLVMBuild.html
+;
+;===------------------------------------------------------------------------===;
+[component_0]
+type = Tool
+name = llvm-lipo
+parent = Tools
+required_libraries = Object Option Support

llvm/trunk/tools/llvm-lipo/LipoOpts.td

+include "llvm/Option/OptParser.td"
+
+def help : Flag<["-", "--"], "help">;
+def h : Flag<["-"], "h">, Alias<help>;
+
+def version : Flag<["-", "--"], "version">,
+              HelpText<"Print the version and exit.">;
+
+def verify_arch : Option<["-", "--"], "verify_arch", KIND_REMAINING_ARGS>,
+                  HelpText<"Verify that the specified arch_types are present in the input file">;

llvm/trunk/tools/llvm-lipo/llvm-lipo.cpp

+//===-- llvm-lipo.cpp - a tool for manipulating universal binaries --------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// A utility for creating / splitting / inspecting universal binaries.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/Triple.h"
+#include "llvm/Object/Binary.h"
+#include "llvm/Object/MachO.h"
+#include "llvm/Object/MachOUniversal.h"
+#include "llvm/Object/ObjectFile.h"
+#include "llvm/Option/Arg.h"
+#include "llvm/Option/ArgList.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/WithColor.h"
+
+using namespace llvm;
+using namespace llvm::object;
+
+static const StringRef ToolName = "llvm-lipo";
+
+LLVM_ATTRIBUTE_NORETURN static void reportError(Twine Message) {
+  WithColor::error(errs(), ToolName) << Message << "\n";
+  errs().flush();
+  exit(EXIT_FAILURE);
+}
+
+LLVM_ATTRIBUTE_NORETURN static void reportError(StringRef File, Error E) {
+  assert(E);
+  std::string Buf;
+  raw_string_ostream OS(Buf);
+  logAllUnhandledErrors(std::move(E), OS);
+  OS.flush();
+  WithColor::error(errs(), ToolName) << "'" << File << "': " << Buf;
+  exit(EXIT_FAILURE);
+}
+
+namespace {
+enum LipoID {
+  LIPO_INVALID = 0, // This is not an option ID.
+#define OPTION(PREFIX, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM,  \
+               HELPTEXT, METAVAR, VALUES)                                      \
+  LIPO_##ID,
+#include "LipoOpts.inc"
+#undef OPTION
+};
+
+#define PREFIX(NAME, VALUE) const char *const LIPO_##NAME[] = VALUE;
+#include "LipoOpts.inc"
+#undef PREFIX
+
+static const opt::OptTable::Info LipoInfoTable[] = {
+#define OPTION(PREFIX, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM,  \
+               HELPTEXT, METAVAR, VALUES)                                      \
+  {LIPO_##PREFIX, NAME,      HELPTEXT,                                         \
+   METAVAR,       LIPO_##ID, opt::Option::KIND##Class,                         \
+   PARAM,         FLAGS,     LIPO_##GROUP,                                     \
+   LIPO_##ALIAS,  ALIASARGS, VALUES},
+#include "LipoOpts.inc"
+#undef OPTION
+};
+
+class LipoOptTable : public opt::OptTable {
+public:
+  LipoOptTable() : OptTable(LipoInfoTable) {}
+};
+
+struct Config {
+  SmallVector<std::string, 1> InputFiles;
+  SmallVector<std::string, 1> VerifyArchList;
+};
+
+} // end namespace
+
+static Config parseLipoOptions(ArrayRef<const char *> ArgsArr) {
+  Config C;
+  LipoOptTable T;
+  unsigned MissingArgumentIndex, MissingArgumentCount;
+  llvm::opt::InputArgList InputArgs =
+      T.ParseArgs(ArgsArr, MissingArgumentIndex, MissingArgumentCount);
+
+  if (InputArgs.size() == 0) {
+    // PrintHelp does not accept Twine.
+    T.PrintHelp(errs(), "llvm-lipo input[s] option[s]", "llvm-lipo");
+    exit(EXIT_FAILURE);
+  }
+
+  if (InputArgs.hasArg(LIPO_help)) {
+    // PrintHelp does not accept Twine.
+    T.PrintHelp(outs(), "llvm-lipo input[s] option[s]", "llvm-lipo");
+    exit(EXIT_SUCCESS);
+  }
+
+  if (InputArgs.hasArg(LIPO_version)) {
+    outs() << ToolName + "\n";
+    cl::PrintVersionMessage();
+    exit(EXIT_SUCCESS);
+  }
+
+  for (auto Arg : InputArgs.filtered(LIPO_UNKNOWN))
+    reportError("unknown argument '" + Arg->getAsString(InputArgs) + "'");
+
+  for (auto Arg : InputArgs.filtered(LIPO_INPUT))
+    C.InputFiles.push_back(Arg->getValue());
+  if (C.InputFiles.empty())
+    reportError("at least one input file should be specified");
+
+  if (InputArgs.hasArg(LIPO_verify_arch)) {
+    for (auto A : InputArgs.getAllArgValues(LIPO_verify_arch))
+      C.VerifyArchList.push_back(A);
+    if (C.VerifyArchList.empty())
+      reportError(
+          "verify_arch requires at least one architecture to be specified");
+    if (C.InputFiles.size() > 1)
+      reportError("verify_arch expects a single input file");
+  }
+  return C;
+}
+
+static SmallVector<OwningBinary<Binary>, 1>
+readInputBinaries(ArrayRef<std::string> InputFiles) {
+  SmallVector<OwningBinary<Binary>, 1> InputBinaries;
+  for (StringRef InputFile : InputFiles) {
+    Expected<OwningBinary<llvm::object::Binary>> BinaryOrErr =
+        createBinary(InputFile);
+    if (!BinaryOrErr)
+      reportError(InputFile, BinaryOrErr.takeError());
+    if (!isa<MachOObjectFile>(BinaryOrErr->getBinary()) &&
+        !isa<MachOUniversalBinary>(BinaryOrErr->getBinary()))
+      reportError("File " + InputFile + " has unsupported binary format");
+    InputBinaries.push_back(std::move(*BinaryOrErr));
+  }
+  return InputBinaries;
+}
+
+LLVM_ATTRIBUTE_NORETURN
+static void verifyArch(ArrayRef<OwningBinary<Binary>> InputBinaries,
+                       ArrayRef<std::string> VerifyArchList) {
+  assert(!InputBinaries.empty() &&
+         "The list of input binaries should be non-empty");
+  assert(!VerifyArchList.empty() &&
+         "The list of architectures should be non-empty");
+  assert(InputBinaries.size() == 1 && "Incorrect number of input binaries");
+
+  for (StringRef Arch : VerifyArchList)
+    if (Triple(Arch).getArch() == Triple::ArchType::UnknownArch)
+      reportError("Invalid architecture: " + Arch);
+
+  if (auto UO =
+          dyn_cast<MachOUniversalBinary>(InputBinaries.front().getBinary())) {
+    for (StringRef Arch : VerifyArchList) {
+      Expected<std::unique_ptr<MachOObjectFile>> Obj =
+          UO->getObjectForArch(Arch);
+      if (!Obj)
+        exit(EXIT_FAILURE);
+    }
+  } else if (auto O =
+                 dyn_cast<MachOObjectFile>(InputBinaries.front().getBinary())) {
+    const Triple::ArchType ObjectArch = O->getArch();
+    for (StringRef Arch : VerifyArchList)
+      if (ObjectArch != Triple(Arch).getArch())
+        exit(EXIT_FAILURE);
+  } else {
+    llvm_unreachable("Unexpected binary format");
+  }
+  exit(EXIT_SUCCESS);
+}
+
+int main(int argc, char **argv) {
+  InitLLVM X(argc, argv);
+  Config C = parseLipoOptions(makeArrayRef(argv + 1, argc));
+  SmallVector<OwningBinary<Binary>, 1> InputBinaries =
+      readInputBinaries(C.InputFiles);
+  if (!C.VerifyArchList.empty())
+    verifyArch(InputBinaries, C.VerifyArchList);
+  return EXIT_SUCCESS;
+}