[tools]Introduce llvm-lipo

[alexander-shaposhnikov]

Summary

This diff starts the implementation of llvm-lipo which is supposed to be a drop-in replacement for the well-known tool lipo.
Lipo manipulates fat binaries, in particular, it can create a "fat" binary, extract regular object files from a fat binary,
report some useful information about the input.

In this diff to get things started we implement the option -verify_arch.
Next we are planning to contribute the implementation of

"-archs": (along the lines of the current diff).

"-thin": (takes a fat binary and writes the "slice" for a particular architecture into a separate file "as is". libObject's functionality is enough for it.

"-create": (takes multiple regular mach-o object files for different architectures and generates a "fat" binary, we are planning to use the standard libObject functionality for reading/inspecting/validating the input files,
populate MachO::fat_header and MachO::fat_arch structures and write them and the slices "as is".

Test plan:
make check-all

[mtrent]

I hate to be "That Guy" but this really needs a docs/CommandGuide file.

I'm super-excited to see this work. This is shall we say "a really bad time of year" to get a thoughtful review from me ( for some reason :p ) but I'll move this to the top of my pile, circumstances permitting.

[smeenai]

Looks good, though I'm not super familiar with the libObject APIs (yet) :)

test/tools/llvm-lipo/verify-arch-macho-binary.test

3

lipo only supports this form of specifying flags (single dash, no equals). Do we want to support the others?

22

Can we strip this down as much as we can? E.g. https://reviews.llvm.org/diffusion/L/browse/llvm/trunk/test/tools/llvm-objdump/AArch64/Inputs/arm64e.macho.yaml

test/tools/llvm-lipo/verify-arch-universal-binary.test

10

Same comment about minimizing this as much as possible.

tools/llvm-lipo/llvm-lipo.cpp

33

Shouldn't this be OneOrMore?

74

Would it be better to use a SmallVectorImpl as the parameter type?

81

Super nit: I think this would read more naturally if the left and right hand sides of the == were switched.

82

Do we need the explicit Twine constructor here? I thought there was an operator+ for const char * and StringRef.

95

Triple(Arch).getArch() is used a bunch. Should we save it to a variable?

[alexander-shaposhnikov]

will update this diff tomorrow

[mtrent]

A second thought on the topic of lipo compatibility. libObject is probably reasonable if your primary interest is MH_OBJECT files. There exist a number of final-linked-images that libObject's fat parser will not currently understand; these are formats that cannot be produced by llvm tools today. I wonder if choosing a different name for the tool might give you some flexibility to "do things differently" than lipo. That said, I don't have a better name in mind. :)

test/tools/llvm-lipo/verify-arch-macho-binary.test

3

I don't see a problem with supporting "--" options here, from the point of view of lipo compatibility. I say, do it.

There are other ways where llvm-lipo is going to be an 'awkward fit' as a drop-in replacement for lipo. For example, lipo is a little unusual in that when -verify_arch is specified, all the remaining options are interpreted as architectures. So llvm-lipo interprets "-verify_arch x86_64 /bin/ls" as "verify /bin/ls has an arch for x86_64 in it" but lipo itself says [paraphrasing] "/bin/ls is not an architecture." Either you will type "input_file -verify_arch x86_64 i386" or you will type "-verify_arch x86_64 -verify_arch i386 input_file" (where input_file could appear anywhere), and whichever usage you specify will be specific either to "llvm-lipo" or the historical lipo.

Pragmatically I think you're going to want to get as close as you can, to avoid confusion. But recognize there are going to be differences if you want to use llvm's argument parsing.

tools/llvm-lipo/llvm-lipo.cpp

33

No. -verify_arch is one of several mutually-exclusive lipo commands. You cannot specify both -verify_arch and -info for example. So ZeroOrMore is correct, and the llvm-lipo command driver is going to have to enforce this exclusivity.

95

Do you mean "O->getArch()"? I agree, if so.

96

the lipo -verify_arch command does not print an error if the command is able to run successfully, as it's primarily for using with shell scripts and Makefiles. Is a verbose message really necessary here? If less, please implement a -non_verbose option.

[mtrent]

tools/llvm-lipo/llvm-lipo.cpp

96

Sigh. s/If less/If yes/;

[alexander-shaposhnikov]

Address comments, have not added docs in this commit yet, can do as a follow-up or will update this diff a bit later.

[alexander-shaposhnikov]

regarding -verify_arch X Y Z - yeah, I can try to make it work with LLVM's command line options parser, not sure if this interface is ideal / better than -verify_arch X -verify_arch Y -verify_arch Z, but yeah, on the other hand, this would be an incompatibility / would require customers of the old tool to update their scripts/code. The downside of the old interface - the positional argument should go before -verify_arch. To be honest at the moment I don't see how to express the old behavior of -verify_arch using libCommandLine, any suggestions would be helpful, alternatively we can try to live with this incompatibility.

[mtrent]

In D61927#1503871, @alexshap wrote:

regarding -verify_arch X Y Z - yeah, I can try to make it work with LLVM's command line options parser, not sure if this interface is ideal / better than -verify_arch X -verify_arch Y -verify_arch Z, but yeah, on the other hand, this would be an incompatibility / would require customers of the old tool to update their scripts/code. The downside of the old interface - the positional argument should go before -verify_arch. To be honest at the moment I don't see how to express the old behavior of -verify_arch using libCommandLine, any suggestions would be helpful, alternatively we can try to live with this incompatibility.

Well, let's have a look at Apple's command syntax. I re-wrote the man page for the latest release, so it's a little clearer to see what's going on. I'll summarize here.

These commands/options take no arguments and are no trouble at all for llvm's command-line parser: -archs, -detailed_info, -info, -hideARM64

These commands/options take a single argument, and only one of these options is allowed. Again, easy mode: -thin, -output.

These commands take a single argument, and any number of them can appear (as long as this is the only such command that appears). Easy mode, you have to handle the command exclusivity yourself in your command driver anyway, so that's not really any extra work: -extract, -extract_family, -remove,

These commands / options take a fixed number of arguments, where n > 1: -create, -replace, -arch, -segalign. I'm under the impression this is something llvm does not support of the box, because we see this problem wtih "llvm-nm -s". But it's possible that the "AdditionalVals" parameter on class Option is meant to deal with this situation. But, if so, no one has circled back to llvm-nm. Some of these commands are pretty critical.

And this command is the worst of the bunch, taking any number of arguments. It's also the one you implemented first :): -verify_arch.

Looking ahead to your task of making a drop-in replacement for Apple's lipo, you're going to have to tackle -verify_arch, -create, -replace, -arch, and -segalign. That's a non-trivial surface area of the lipo tool. My advice is to see if you can model the complete set of lipo command-line options you want to reverse-engineer -- don't worry about implementing the feature, just parse the arguments. If -verify_arch is the only one that doesn't fit the mold, I think "just live with it" is the right call. But if you can't get -create to work with an arbitrary number of -arch or -arch_blank options, or if you can't get -replace working, you're going to have trouble making a viable "drop-in-replacement."

HTH!

[alexander-shaposhnikov]

@mtrent, good call, will have a look and update this diff

[alexander-shaposhnikov]

This comment was removed by alexander-shaposhnikov.

[alexander-shaposhnikov]

Use a bigger hammer to parse command line options

[mtrent]

That's a pretty big hammer. Let's give it a try!

test/tools/llvm-lipo/verify-arch-macho-binary.test

+# RUN: yaml2obj %s > %t
+
+# RUN: llvm-lipo %t -verify_arch i386
+# RUN: llvm-lipo %t --verify_arch i386
+# RUN: llvm-lipo %t --verify_arch=i386
+
+# NO_AARCH64: does not contain aarch64
+# RUN: not llvm-lipo %t -verify_arch aarch64 2>&1 | FileCheck --check-prefix=NO_AARCH64 %s
+# RUN: not llvm-lipo %t -verify_arch aarch64 -verify_arch i386 2>&1 | FileCheck --check-prefix=NO_AARCH64 %s
+
+# INVALID_ARCH: Invalid architecture: aarch101
+# RUN: not llvm-lipo %t -verify_arch aarch101 2>&1 | FileCheck --check-prefix=INVALID_ARCH %s
+
+# INVALID_OBJ: The file was not recognized as a valid object file
+# RUN: touch %t.empty
+# RUN: not llvm-lipo %t.empty -verify_arch aarch101 2>&1 | FileCheck --check-prefix=INVALID_OBJ %s
+
+# MULTIPLE_INPUT_OBJ: verify_arch expects a single input file
+# RUN: not llvm-lipo %t %t -verify_arch i386 2>&1 | FileCheck --check-prefix=MULTIPLE_INPUT_OBJ %s
+
+--- !mach-o
+FileHeader:      
+  magic:           0xFEEDFACE
+  cputype:         0x00000007
+  cpusubtype:      0x00000003
+  filetype:        0x00000001
+  ncmds:           3
+  sizeofcmds:      296
+  flags:           0x00002000
+LoadCommands:    
+  - cmd:             LC_SEGMENT
+    cmdsize:         192
+    segname:         ''
+    vmaddr:          0
+    vmsize:          84
+    fileoff:         324
+    filesize:        84
+    maxprot:         7
+    initprot:        7
+    nsects:          2
+    flags:           0
+    Sections:        
+      - sectname:        __text
+        segname:         __TEXT
+        addr:            0x0000000000000000
+        size:            31
+        offset:          0x00000144
+        align:           4
+        reloff:          0x00000000
+        nreloc:          0
+        flags:           0x80000400
+        reserved1:       0x00000000
+        reserved2:       0x00000000
+        reserved3:       0x00000000
+      - sectname:        __eh_frame
+        segname:         __TEXT
+        addr:            0x0000000000000020
+        size:            52
+        offset:          0x00000164
+        align:           2
+        reloff:          0x00000000
+        nreloc:          0
+        flags:           0x6800000B
+        reserved1:       0x00000000
+        reserved2:       0x00000000
+        reserved3:       0x00000000
+  - cmd:             LC_SYMTAB
+    cmdsize:         24
+    symoff:          408
+    nsyms:           1
+    stroff:          420
+    strsize:         8
+  - cmd:             LC_DYSYMTAB
+    cmdsize:         80
+    ilocalsym:       0
+    nlocalsym:       0
+    iextdefsym:      0
+    nextdefsym:      1
+    iundefsym:       1
+    nundefsym:       0
+    tocoff:          0
+    ntoc:            0
+    modtaboff:       0
+    nmodtab:         0
+    extrefsymoff:    0
+    nextrefsyms:     0
+    indirectsymoff:  0
+    nindirectsyms:   0
+    extreloff:       0
+    nextrel:         0
+    locreloff:       0
+    nlocrel:         0
+LinkEditData:    
+  NameList:        
+    - n_strx:          1
+      n_type:          0x0F
+      n_sect:          1
+      n_desc:          0
+      n_value:         0
+  StringTable:     
+    - ''
+    - _main
+    - ''
+...

test/tools/llvm-lipo/verify-arch-universal-binary.test

+# RUN: yaml2obj %s > %t
+
+# RUN: llvm-lipo %t -verify_arch i386
+# RUN: llvm-lipo %t -verify_arch i386 -verify_arch x86_64
+
+# NO_AARCH64: does not contain aarch64
+# RUN: not llvm-lipo %t -verify_arch aarch64 2>&1 | FileCheck --check-prefix=NO_AARCH64 %s
+# RUN: not llvm-lipo %t -verify_arch aarch64 -verify_arch i386 2>&1 | FileCheck --check-prefix=NO_AARCH64 %s
+
+--- !fat-mach-o
+FatHeader:       
+  magic:           0xCAFEBABE
+  nfat_arch:       2
+FatArchs:        
+  - cputype:         0x00000007
+    cpusubtype:      0x00000003
+    offset:          0x0000000000001000
+    size:            428
+    align:           12
+  - cputype:         0x01000007
+    cpusubtype:      0x00000003
+    offset:          0x0000000000002000
+    size:            488
+    align:           12
+Slices:          
+  - !mach-o
+    FileHeader:      
+      magic:           0xFEEDFACE
+      cputype:         0x00000007
+      cpusubtype:      0x00000003
+      filetype:        0x00000001
+      ncmds:           3
+      sizeofcmds:      296
+      flags:           0x00002000
+    LoadCommands:    
+      - cmd:             LC_SEGMENT
+        cmdsize:         192
+        segname:         ''
+        vmaddr:          0
+        vmsize:          84
+        fileoff:         324
+        filesize:        84
+        maxprot:         7
+        initprot:        7
+        nsects:          2
+        flags:           0
+        Sections:        
+          - sectname:        __text
+            segname:         __TEXT
+            addr:            0x0000000000000000
+            size:            31
+            offset:          0x00000144
+            align:           4
+            reloff:          0x00000000
+            nreloc:          0
+            flags:           0x80000400
+            reserved1:       0x00000000
+            reserved2:       0x00000000
+            reserved3:       0x00000000
+          - sectname:        __eh_frame
+            segname:         __TEXT
+            addr:            0x0000000000000020
+            size:            52
+            offset:          0x00000164
+            align:           2
+            reloff:          0x00000000
+            nreloc:          0
+            flags:           0x6800000B
+            reserved1:       0x00000000
+            reserved2:       0x00000000
+            reserved3:       0x00000000
+      - cmd:             LC_SYMTAB
+        cmdsize:         24
+        symoff:          408
+        nsyms:           1
+        stroff:          420
+        strsize:         8
+      - cmd:             LC_DYSYMTAB
+        cmdsize:         80
+        ilocalsym:       0
+        nlocalsym:       0
+        iextdefsym:      0
+        nextdefsym:      1
+        iundefsym:       1
+        nundefsym:       0
+        tocoff:          0
+        ntoc:            0
+        modtaboff:       0
+        nmodtab:         0
+        extrefsymoff:    0
+        nextrefsyms:     0
+        indirectsymoff:  0
+        nindirectsyms:   0
+        extreloff:       0
+        nextrel:         0
+        locreloff:       0
+        nlocrel:         0
+    LinkEditData:    
+      NameList:        
+        - n_strx:          1
+          n_type:          0x0F
+          n_sect:          1
+          n_desc:          0
+          n_value:         0
+      StringTable:     
+        - ''
+        - _main
+        - ''
+  - !mach-o
+    FileHeader:      
+      magic:           0xFEEDFACF
+      cputype:         0x01000007
+      cpusubtype:      0x00000003
+      filetype:        0x00000001
+      ncmds:           3
+      sizeofcmds:      336
+      flags:           0x00002000
+      reserved:        0x00000000
+    LoadCommands:    
+      - cmd:             LC_SEGMENT_64
+        cmdsize:         232
+        segname:         ''
+        vmaddr:          0
+        vmsize:          96
+        fileoff:         368
+        filesize:        96
+        maxprot:         7
+        initprot:        7
+        nsects:          2
+        flags:           0
+        Sections:        
+          - sectname:        __text
+            segname:         __TEXT
+            addr:            0x0000000000000000
+            size:            26
+            offset:          0x00000170
+            align:           4
+            reloff:          0x00000000
+            nreloc:          0
+            flags:           0x80000400
+            reserved1:       0x00000000
+            reserved2:       0x00000000
+            reserved3:       0x00000000
+          - sectname:        __eh_frame
+            segname:         __TEXT
+            addr:            0x0000000000000020
+            size:            64
+            offset:          0x00000190
+            align:           3
+            reloff:          0x00000000
+            nreloc:          0
+            flags:           0x6800000B
+            reserved1:       0x00000000
+            reserved2:       0x00000000
+            reserved3:       0x00000000
+      - cmd:             LC_SYMTAB
+        cmdsize:         24
+        symoff:          464
+        nsyms:           1
+        stroff:          480
+        strsize:         8
+      - cmd:             LC_DYSYMTAB
+        cmdsize:         80
+        ilocalsym:       0
+        nlocalsym:       0
+        iextdefsym:      0
+        nextdefsym:      1
+        iundefsym:       1
+        nundefsym:       0
+        tocoff:          0
+        ntoc:            0
+        modtaboff:       0
+        nmodtab:         0
+        extrefsymoff:    0
+        nextrefsyms:     0
+        indirectsymoff:  0
+        nindirectsyms:   0
+        extreloff:       0
+        nextrel:         0
+        locreloff:       0
+        nlocrel:         0
+    LinkEditData:    
+      NameList:        
+        - n_strx:          1
+          n_type:          0x0F
+          n_sect:          1
+          n_desc:          0
+          n_value:         0
+      StringTable:     
+        - ''
+        - _main
+        - ''
+...

tools/llvm-lipo/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  ${LLVM_TARGETS_TO_BUILD}
+  Object
+  Support
+)
+
+add_llvm_tool(llvm-lipo
+  llvm-lipo.cpp
+)

tools/llvm-lipo/LLVMBuild.txt

+;===- ./tools/llvm-lipo/LLVMBuild.txt --------------------------*- Conf -*--===;
+;
+; Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+; See https://llvm.org/LICENSE.txt for license information.
+; SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+;
+;===------------------------------------------------------------------------===;
+;
+; This is an LLVMBuild description file for the components in this subdirectory.
+;
+; For more information on the LLVMBuild system, please see:
+;
+;   http://llvm.org/docs/LLVMBuild.html
+;
+;===------------------------------------------------------------------------===;
+[component_0]
+type = Tool
+name = llvm-lipo
+parent = Tools
+required_libraries = Object Option Support

tools/llvm-lipo/llvm-lipo.cpp

+//===-- llvm-lipo.cpp - a tool for manipulating universal binaries --------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// A utility for creating / splitting / inspecting universal binaries.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/Triple.h"
+#include "llvm/Object/Binary.h"
+#include "llvm/Object/MachO.h"
+#include "llvm/Object/MachOUniversal.h"
+#include "llvm/Object/ObjectFile.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/WithColor.h"
+
+using namespace llvm;
+using namespace llvm::object;
+
+cl::OptionCategory LipoCategory("Specific Options");
+
+static cl::list<std::string> InputFiles(cl::Positional, cl::OneOrMore,
+                                        cl::desc("<input files>"),
+                                        cl::cat(LipoCategory));
+
+static cl::list<std::string> VerifyArch(
+    "verify_arch", cl::ZeroOrMore,
+    cl::desc(
+        "Verify the specified architectures are present in the input files"),
+    cl::value_desc("architecture"), cl::cat(LipoCategory));
+
+static const StringRef ToolName = "llvm-lipo";
+
+LLVM_ATTRIBUTE_NORETURN static void reportError(Twine Message) {
+  WithColor::error(errs(), ToolName) << Message << "\n";
+  errs().flush();
+  exit(EXIT_FAILURE);
+}
+
+LLVM_ATTRIBUTE_NORETURN static void reportError(StringRef File, Error E) {
+  assert(E);
+  std::string Buf;
+  raw_string_ostream OS(Buf);
+  logAllUnhandledErrors(std::move(E), OS);
+  OS.flush();
+  WithColor::error(errs(), ToolName) << "'" << File << "': " << Buf;
+  exit(EXIT_FAILURE);
+}
+
+static SmallVector<OwningBinary<Binary>, 1> readInputBinaries() {
+  SmallVector<OwningBinary<Binary>, 1> InputBinaries;
+  for (StringRef InputFile : InputFiles) {
+    Expected<OwningBinary<llvm::object::Binary>> BinaryOrErr =
+        createBinary(InputFile);
+    if (!BinaryOrErr)
+      reportError(InputFile, BinaryOrErr.takeError());
+    if (!isa<MachOObjectFile>(BinaryOrErr->getBinary()) &&
+        !isa<MachOUniversalBinary>(BinaryOrErr->getBinary()))
+      reportError(Twine("File ") + InputFile +
+                  " has unsupported binary format");
+    InputBinaries.push_back(std::move(*BinaryOrErr));
+  }
+  return InputBinaries;
+}
+
+LLVM_ATTRIBUTE_NORETURN
+static void
+verifyArch(const SmallVector<OwningBinary<Binary>, 1> &InputBinaries) {
+  assert(!InputBinaries.empty() &&
+         "The list of input binaries should be non-empty");
+  if (InputBinaries.size() > 1)
+    reportError("verify_arch expects a single input file");
+
+  for (StringRef Arch : VerifyArch)
+    if (Triple::ArchType::UnknownArch == Triple(Arch).getArch())
+      reportError(Twine("Invalid architecture: ") + Arch);
+
+  if (auto UO =
+          dyn_cast<MachOUniversalBinary>(InputBinaries.front().getBinary())) {
+    for (StringRef Arch : VerifyArch) {
+      Expected<std::unique_ptr<MachOObjectFile>> Obj =
+          UO->getObjectForArch(Arch);
+      if (!Obj)
+        reportError(UO->getFileName(), Obj.takeError());
+    }
+  } else if (auto O =
+                 dyn_cast<MachOObjectFile>(InputBinaries.front().getBinary())) {
+    for (StringRef Arch : VerifyArch)
+      if (O->getArch() != Triple(Arch).getArch())
+        reportError(Twine(O->getFileName()) + " does not contain " + Arch);
+  } else {
+    llvm_unreachable("Unexpected binary format");
+  }
+  exit(EXIT_SUCCESS);
+}
+
+int main(int argc, char **argv) {
+  InitLLVM X(argc, argv);
+  cl::ParseCommandLineOptions(
+      argc, argv,
+      "A tool for creating/splitting/inspecting universal binaries\n");
+
+  SmallVector<OwningBinary<Binary>, 1> InputBinaries = readInputBinaries();
+  if (!VerifyArch.empty())
+    verifyArch(InputBinaries);
+
+  return EXIT_SUCCESS;
+}