LibDriver, llvm-lib: introduce.

[pcc]

Summary

llvm-lib is intended to be a lib.exe compatible utility that also
understands bitcode. The implementation lives in a library so that
lld can use it to implement /lib.

[• rafael]

The lib format is really similar to the .a format.

I guess the only question I have about this direction is how would it
compare with having lld fork+exec llvm-lib.exe and implementing
llvm-lib.exe in the same binary as llvm-ar.exe (similar to how we do
for llvm-ranlib.exe).

[pcc]

• Make llvm-lib an alias of llvm-ar, add Makefiles

[pcc]

• Add response file support

[• rafael]

Why do you need LibDriver? Why not just put these files in tools/llvm-ar?

[pcc]

I'll want to use the lib driver from lld, per our earlier discussion here: http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20150601/280581.html

[• rafael]

This is fine from the llvm side.

Rui, is this a convenient interface for lld to use?

lib/LibDriver/LibDriver.cpp

72

We really should just move the string saver that rui implemented to llvm and make it the one true string saver.
At least add a fixme about that.

[echristo]

+namespace {
+class StrDupSaver : public cl::StringSaver {

+ std::vector<char *> Dups;

We really should just move the string saver that rui implemented to llvm
and make it the one true string saver.
At least add a fixme about that.

Given we do it all over clang as well it'd be nice, yes.

-eric

[pcc]

Rui, is this a convenient interface for lld to use?

D10347 is the lld side of this.

[• rafael]

LGTM

[ruiu]

2015/06/09 午後2:31 "Rafael Ávila de Espíndola" <[e-mail]>:

This is fine from the llvm side.

Rui, is this a convenient interface for lld to use?

Comment at: lib/LibDriver/LibDriver.cpp:71
@@ +70,3 @@
+namespace {
+class StrDupSaver : public cl::StringSaver {

+ std::vector<char *> Dups;

We really should just move the string saver that rui implemented to llvm

and make it the one true string saver.

At least add a fixme about that.

Let me do that. I'll send a patch later.

http://reviews.llvm.org/D10297

EMAIL PREFERENCES

http://reviews.llvm.org/settings/panel/emailpreferences/

[• rafael]

• We really should just move the string saver that rui implemented to llvm

and make it the one true string saver.

At least add a fixme about that.

I am giving that a try :-)

Cheers,
Rafael

include/llvm/LibDriver/LibDriver.h

+//===- llvm/LibDriver/LibDriver.h - lib.exe-compatible driver ---*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Defines an interface to a lib.exe-compatible driver that also understands
+// bitcode files. Used by llvm-lib and lld-link2 /lib.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBDRIVER_LIBDRIVER_H
+#define LLVM_LIBDRIVER_LIBDRIVER_H
+
+namespace llvm {
+
+int libDriverMain(int argc, const char **argv);
+
+}
+
+#endif

lib/CMakeLists.txt

 add_subdirectory(DebugInfo)
 add_subdirectory(ExecutionEngine)
 add_subdirectory(Target)
 add_subdirectory(AsmParser)
 add_subdirectory(LineEditor)
 add_subdirectory(ProfileData)
 add_subdirectory(Fuzzer)
 add_subdirectory(Passes)
+add_subdirectory(LibDriver)

lib/LLVMBuild.txt

 ; For more information on the LLVMBuild system, please see:
 ;
 ;   http://llvm.org/docs/LLVMBuild.html
 ;
 ;===------------------------------------------------------------------------===;
 
 [common]
 subdirectories = Analysis AsmParser Bitcode CodeGen DebugInfo ExecutionEngine
- LineEditor Linker IR IRReader LTO MC Object Option Passes ProfileData Support
- TableGen Target Transforms
+ LibDriver LineEditor Linker IR IRReader LTO MC Object Option Passes ProfileData
+ Support TableGen Target Transforms
 
 [component_0]
 type = Group
 name = Libraries
 parent = $ROOT

lib/LibDriver/CMakeLists.txt

+set(LLVM_TARGET_DEFINITIONS Options.td)
+tablegen(LLVM Options.inc -gen-opt-parser-defs)
+add_public_tablegen_target(LibOptionsTableGen)
+
+add_llvm_library(LLVMLibDriver
+  LibDriver.cpp
+  )
+add_dependencies(LLVMLibDriver LibOptionsTableGen)

lib/LibDriver/LLVMBuild.txt

-;===- ./lib/LLVMBuild.txt --------------------------------------*- Conf -*--===;
+;===- ./lib/LibDriver/LLVMBuild.txt ----------------------------*- Conf -*--===;
 ;
 ;                     The LLVM Compiler Infrastructure
 ;
 ; This file is distributed under the University of Illinois Open Source
 ; License. See LICENSE.TXT for details.
 ;
 ;===------------------------------------------------------------------------===;
 ;
 ; This is an LLVMBuild description file for the components in this subdirectory.
 ;
 ; For more information on the LLVMBuild system, please see:
 ;
 ;   http://llvm.org/docs/LLVMBuild.html
 ;
 ;===------------------------------------------------------------------------===;
 
-[common]
-subdirectories = Analysis AsmParser Bitcode CodeGen DebugInfo ExecutionEngine
- LineEditor Linker IR IRReader LTO MC Object Option Passes ProfileData Support
- TableGen Target Transforms
-
 [component_0]
-type = Group
-name = Libraries
-parent = $ROOT
+type = Library
+name = LibDriver
+parent = Libraries
+required_libraries = Object Option Support

lib/LibDriver/LibDriver.cpp

+//===- LibDriver.cpp - lib.exe-compatible driver --------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Defines an interface to a lib.exe-compatible driver that also understands
+// bitcode files. Used by llvm-lib and lld-link2 /lib.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/LibDriver/LibDriver.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Object/ArchiveWriter.h"
+#include "llvm/Option/Arg.h"
+#include "llvm/Option/ArgList.h"
+#include "llvm/Option/Option.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace llvm;
+
+namespace {
+
+enum {
+  OPT_INVALID = 0,
+#define OPTION(_1, _2, ID, _4, _5, _6, _7, _8, _9, _10, _11) OPT_##ID,
+#include "Options.inc"
+#undef OPTION
+};
+
+#define PREFIX(NAME, VALUE) const char *const NAME[] = VALUE;
+#include "Options.inc"
+#undef PREFIX
+
+static const llvm::opt::OptTable::Info infoTable[] = {
+#define OPTION(X1, X2, ID, KIND, GROUP, ALIAS, X6, X7, X8, X9, X10)    \
+  {                                                                    \
+    X1, X2, X9, X10, OPT_##ID, llvm::opt::Option::KIND##Class, X8, X7, \
+    OPT_##GROUP, OPT_##ALIAS, X6                                       \
+  },
+#include "Options.inc"
+#undef OPTION
+};
+
+class LibOptTable : public llvm::opt::OptTable {
+public:
+  LibOptTable() : OptTable(infoTable, llvm::array_lengthof(infoTable), true) {}
+};
+
+}
+
+static std::string getOutputPath(llvm::opt::InputArgList *Args) {
+  if (auto *Arg = Args->getLastArg(OPT_out))
+    return Arg->getValue();
+  for (auto *Arg : Args->filtered(OPT_INPUT)) {
+    if (!StringRef(Arg->getValue()).endswith_lower(".obj"))
+      continue;
+    SmallString<128> Val = StringRef(Arg->getValue());
+    llvm::sys::path::replace_extension(Val, ".lib");
+    return Val.str();
+  }
+  llvm_unreachable("internal error");
+}
+
+int llvm::libDriverMain(int Argc, const char **Argv) {
+  LibOptTable Table;
+  unsigned MissingIndex;
+  unsigned MissingCount;
+  std::unique_ptr<llvm::opt::InputArgList> Args(
+      Table.ParseArgs(&Argv[1], &Argv[Argc], MissingIndex, MissingCount));
+  if (MissingCount) {
+    llvm::errs() << "missing arg value for \""
+                 << Args->getArgString(MissingIndex)
+                 << "\", expected " << MissingCount
+                 << (MissingCount == 1 ? " argument.\n" : " arguments.\n");
+    return 1;
+  }
+  for (auto *Arg : Args->filtered(OPT_UNKNOWN))
+    llvm::errs() << "ignoring unknown argument: " << Arg->getSpelling() << "\n";
+
+  if (Args->filtered_begin(OPT_INPUT) == Args->filtered_end()) {
+    llvm::errs() << "no input files.\n";
+    return 1;
+  }
+
+  std::vector<llvm::NewArchiveIterator> Members;
+  for (auto *Arg : Args->filtered(OPT_INPUT))
+    Members.emplace_back(Arg->getValue(),
+                         llvm::sys::path::filename(Arg->getValue()));
+
+  std::pair<StringRef, std::error_code> Result = llvm::writeArchive(
+      getOutputPath(Args.get()), Members, /*WriteSymtab=*/true);
+  if (Result.second) {
+    if (Result.first.empty())
+      Result.first = Argv[0];
+    llvm::errs() << Result.first << ": " << Result.second.message() << "\n";
+    return 1;
+  }
+
+  return 0;
+}

lib/LibDriver/Makefile

+##===- lib/LibDriver/Makefile ------------------------------*- Makefile -*-===##
+#
+#                     The LLVM Compiler Infrastructure
+#
+# This file is distributed under the University of Illinois Open Source
+# License. See LICENSE.TXT for details.
+#
+##===----------------------------------------------------------------------===##
+
+LEVEL = ../..
+LIBRARYNAME = LLVMLibDriver
+BUILD_ARCHIVE := 1
+BUILT_SOURCES = Options.inc
+TABLEGEN_INC_FILES_COMMON = 1
+
+include $(LEVEL)/Makefile.common
+
+$(ObjDir)/Options.inc.tmp : Options.td $(LLVM_TBLGEN) $(ObjDir)/.dir
+  $(Echo) "Building lib Driver Option tables with tblgen"
+  $(Verb) $(LLVMTableGen) -gen-opt-parser-defs -o $(call SYSPATH, $@) $<

lib/LibDriver/Options.td

+include "llvm/Option/OptParser.td"
+
+// lib.exe accepts options starting with either a dash or a slash.
+
+// Flag that takes no arguments.
+class F<string name> : Flag<["/", "-", "-?"], name>;
+
+// Flag that takes one argument after ":".
+class P<string name, string help> :
+      Joined<["/", "-", "-?"], name#":">, HelpText<help>;
+
+def out    : P<"out", "Path to file to write output">;
+
+//==============================================================================
+// The flags below do nothing. They are defined only for lib.exe compatibility.
+//==============================================================================
+def nologo : F<"nologo">;

lib/Makefile

 ##===- lib/Makefile ----------------------------------------*- Makefile -*-===##
 #
 #                     The LLVM Compiler Infrastructure
 #
 # This file is distributed under the University of Illinois Open Source
 # License. See LICENSE.TXT for details.
 #
 ##===----------------------------------------------------------------------===##
 LEVEL = ..
 
 include $(LEVEL)/Makefile.config
 
 PARALLEL_DIRS := IR AsmParser Bitcode Analysis Transforms CodeGen Target      \
                  ExecutionEngine Linker LTO MC Object Option DebugInfo        \
-                 IRReader LineEditor ProfileData Passes
+                 IRReader LineEditor ProfileData Passes LibDriver
 
 include $(LEVEL)/Makefile.common

test/CMakeLists.txt

set(LLVM_TEST_DEPENDS
           llvm-c-test
           llvm-cov
           llvm-cxxdump
           llvm-diff
           llvm-dis
           llvm-dsymutil
           llvm-dwarfdump
           llvm-extract
+          llvm-lib
           llvm-link
           llvm-lto
           llvm-mc
           llvm-mcmarkup
           llvm-nm
           llvm-objdump
           llvm-profdata
           llvm-ranlib

test/Object/nm-archive.test

 COFF-o: {{.*}}/archive-test.a-coff-i386:trivial-object-test.coff-i386: 00000000 T _main
 COFF-o: {{.*}}/archive-test.a-coff-i386:trivial-object-test.coff-i386:          U _puts
 
 RUN: llvm-as %p/Inputs/trivial.ll -o=%t1
 RUN: rm -f %t2
 RUN: llvm-ar rcs %t2 %t1
 RUN: llvm-nm %t2 | FileCheck %s -check-prefix BITCODE
 
+RUN: rm -f %t2
+RUN: llvm-lib /out:%t2 %t1
+RUN: llvm-nm %t2 | FileCheck %s -check-prefix BITCODE
+
 BITCODE:          U SomeOtherFunction
 BITCODE-NEXT:          T main
 BITCODE-NEXT:          U puts
 BITCODE-NEXT:          D var
 
 
 Test we don't error with an archive with no symtab.
 RUN: llvm-nm %p/Inputs/archive-test.a-gnu-no-symtab

test/lit.cfg

for pattern in [r"\bbugpoint\b(?!-)",
                 r"\bllvm-cov\b",
                 r"\bllvm-cxxdump\b",
                 r"\bllvm-diff\b",
                 r"\bllvm-dis\b",
                 r"\bllvm-dsymutil\b",
                 r"\bllvm-dwarfdump\b",
                 r"\bllvm-extract\b",
                 r"\bllvm-go\b",
+                r"\bllvm-lib\b",
                 r"\bllvm-link\b",
                 r"\bllvm-lto\b",
                 r"\bllvm-mc\b",
                 r"\bllvm-mcmarkup\b",
                 r"\bllvm-nm\b",
                 r"\bllvm-objdump\b",
                 r"\bllvm-profdata\b",
                 r"\bllvm-ranlib\b",

tools/llvm-ar/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   ${LLVM_TARGETS_TO_BUILD}
   Core
+  LibDriver
   Object
   Support
   )
 
 add_llvm_tool(llvm-ar
   llvm-ar.cpp
   )
 

 
 add_custom_command(OUTPUT ${llvm_ranlib}
                    COMMAND ${CMAKE_COMMAND} -E ${LLVM_LINK_OR_COPY} "${llvm_ar_binary}" "${llvm_ranlib}"
                    DEPENDS llvm-ar)
 
 add_custom_target(llvm-ranlib ALL DEPENDS ${llvm_ranlib})
 set_target_properties(llvm-ranlib PROPERTIES FOLDER Tools)
 
+set(llvm_lib "${LLVM_RUNTIME_OUTPUT_INTDIR}/llvm-lib${CMAKE_EXECUTABLE_SUFFIX}")
+
+add_custom_command(OUTPUT ${llvm_lib}
+                   COMMAND ${CMAKE_COMMAND} -E ${LLVM_LINK_OR_COPY} "${llvm_ar_binary}" "${llvm_lib}"
+                   DEPENDS llvm-ar)
+
+add_custom_target(llvm-lib ALL DEPENDS ${llvm_lib})
+set_target_properties(llvm-lib PROPERTIES FOLDER Tools)
+
 install(SCRIPT install_symlink.cmake -DCMAKE_INSTALL_PREFIX=\"${CMAKE_INSTALL_PREFIX}\")

tools/llvm-ar/Makefile

 ##===- tools/llvm-ar/Makefile ------------------------------*- Makefile -*-===##
 # 
 #                     The LLVM Compiler Infrastructure
 #
 # This file is distributed under the University of Illinois Open Source
 # License. See LICENSE.TXT for details.
 # 
 ##===----------------------------------------------------------------------===##
 
 LEVEL := ../..
 TOOLNAME := llvm-ar
 TOOLALIAS = llvm-ranlib
-LINK_COMPONENTS := all-targets bitreader support object
+LINK_COMPONENTS := all-targets bitreader libdriver support object
 
 # This tool has no plugins, optimize startup time.
 TOOL_NO_EXPORTS := 1
 
 include $(LEVEL)/Makefile.common
+
+all-local::
+  $(Verb) $(AliasTool) $(notdir $(ToolBuildPath)) $(ToolDir)/llvm-lib$(EXEEXT)

tools/llvm-ar/install_symlink.cmake

 
 set(bindir "${DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/")
 
 message("Creating llvm-ranlib")
 
 execute_process(
   COMMAND "${CMAKE_COMMAND}" -E ${LINK_OR_COPY} "llvm-ar${EXECUTABLE_SUFFIX}" "llvm-ranlib${EXECUTABLE_SUFFIX}"
   WORKING_DIRECTORY "${bindir}")
+
+message("Creating llvm-lib")
+
+execute_process(
+  COMMAND "${CMAKE_COMMAND}" -E ${LINK_OR_COPY} "llvm-ar${EXECUTABLE_SUFFIX}" "llvm-lib${EXECUTABLE_SUFFIX}"
+  WORKING_DIRECTORY "${bindir}")

tools/llvm-ar/llvm-ar.cpp

 // Builds up (relatively) standard unix archive files (.a) containing LLVM
 // bitcode or other files.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/Module.h"
+#include "llvm/LibDriver/LibDriver.h"
 #include "llvm/Object/Archive.h"
 #include "llvm/Object/ArchiveWriter.h"
 #include "llvm/Object/ObjectFile.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/LineIterator.h"

 
 int main(int argc, char **argv) {
   ToolName = argv[0];
   // Print a stack trace if we signal out.
   sys::PrintStackTraceOnErrorSignal();
   PrettyStackTraceProgram X(argc, argv);
   llvm_shutdown_obj Y;  // Call llvm_shutdown() on exit.
 
+  llvm::InitializeAllTargetInfos();
+  llvm::InitializeAllTargetMCs();
+  llvm::InitializeAllAsmParsers();
+
+  StringRef Stem = sys::path::stem(ToolName);
+  if (Stem.find("ranlib") == StringRef::npos &&
+      Stem.find("lib") != StringRef::npos)
+    return libDriverMain(argc, const_cast<const char **>(argv));
+
   // Have the command line options parsed and handle things
   // like --help and --version.
   cl::ParseCommandLineOptions(argc, argv,
     "LLVM Archiver (llvm-ar)\n\n"
     "  This program archives bitcode files into single libraries\n"
   );
 
-  llvm::InitializeAllTargetInfos();
-  llvm::InitializeAllTargetMCs();
-  llvm::InitializeAllAsmParsers();
-
-  StringRef Stem = sys::path::stem(ToolName);
   if (Stem.find("ar") != StringRef::npos)
     return ar_main();
   if (Stem.find("ranlib") != StringRef::npos)
     return ranlib_main();
-  fail("Not ranlib or ar!");
+  fail("Not ranlib, ar or lib!");
 }