Add llvm-modextract tool.

[pcc]

Summary

This program is for testing features that rely on multi-module bitcode files.
It takes a multi-module bitcode file, extracts one of the modules and writes
it to the output file.

[mehdi_amini]

Could we have a cheap way to extract modules without parsing the IR?

(I assume we need D26179 to get in before we can have a test with multiple modules)

[pcc]

Could we have a cheap way to extract modules without parsing the IR?

Most likely, but it would probably require a change to the bitcode format because we represent the VST offset as an offset from the start of the file rather than the start of the module.

(I assume we need D26179 to get in before we can have a test with multiple modules)

Right.

[mehdi_amini]

LGTM.

In D26778#605102, @pcc wrote:

Could we have a cheap way to extract modules without parsing the IR?

Most likely, but it would probably require a change to the bitcode format because we represent the VST offset as an offset from the start of the file rather than the start of the module.

Right I think any relative offset should begin with the module block! I'm fine keeping this as future work.

[silvas]

This is just a drive-by comment, but you might be able to get away with less boilerplate by just teaching llvm-extract to do this. I don't feels strongly about this suggestion so do whatever you find best.

[mehdi_amini]

I agree with Sean! Reusing llvm-extract seems appropriate to me here.

[mehdi_amini]

Having a llvm-bcutils tool is a good idea, but that a larger refactoring. We should think about it, but not block committing this patch either.

[mehdi_amini]

In D26778#605103, @mehdi_amini wrote:

LGTM.

In D26778#605102, @pcc wrote:

Could we have a cheap way to extract modules without parsing the IR?

Most likely, but it would probably require a change to the bitcode format because we represent the VST offset as an offset from the start of the file rather than the start of the module.

Right I think any relative offset should begin with the module block! I'm fine keeping this as future work.

Actually this is a problem and I think we should address it now.

The issue is that if we go with the current solution, we'll have to support "forever" the fact that a bitcode file with multiple module can have offset from the beginning of the file.
It seems to me that we have to first update the bitcode to make sure each offset start from the beginning of the module.

[mehdi_amini]

llvm/tools/llvm-modextract/llvm-modextract.cpp

54

For symmetry with llvm-cat, can we have the same -b option:

std::error_code EC;
std::unique_ptr<tool_output_file> Out(
    new tool_output_file(OutputFilename, EC, sys::fs::F_None));
ExitOnErr(errorCodeToError(EC));
 
if (BinaryExtract) {
   // Write header
   ...
   //Dump Ms[ModuleIndex].getBuffer() in Out->os()
   ...
   return 0;
}

std::unique_ptr<Module> M = ExitOnErr(Ms[ModuleIndex].parseModule(Context));
WriteBitcodeToFile(M.get(), Out->os());

[pcc]

• add -b flag

[mehdi_amini]

Great! Thanks. LGTM.

llvm/include/llvm/Bitcode/BitcodeReader.h

class BitcodeModule {
     friend Expected<std::vector<BitcodeModule>>
     getBitcodeModuleList(MemoryBufferRef Buffer);
 
     Expected<std::unique_ptr<Module>>
     getModuleImpl(LLVMContext &Context, bool MaterializeAll,
                   bool ShouldLazyLoadMetadata);
 
   public:
-    ArrayRef<uint8_t> getBuffer() const { return Buffer; }
+    StringRef getBuffer() const {
+      return StringRef((const char *)Buffer.begin(), Buffer.size());
+    }
 
     /// Read the bitcode module and prepare for lazy deserialization of function
     /// bodies. If ShouldLazyLoadMetadata is true, lazily load metadata as well.
     Expected<std::unique_ptr<Module>>
     getLazyModule(LLVMContext &Context, bool ShouldLazyLoadMetadata);
 
     /// Read the entire bitcode module and return it.
     Expected<std::unique_ptr<Module>> parseModule(LLVMContext &Context);

llvm/test/CMakeLists.txt

set(LLVM_TEST_DEPENDS
           llvm-dwarfdump
           llvm-dwp
           llvm-extract
           llvm-lib
           llvm-link
           llvm-lto2
           llvm-mc
           llvm-mcmarkup
+          llvm-modextract
           llvm-nm
           llvm-objdump
           llvm-opt-report
           llvm-pdbdump
           llvm-profdata
           llvm-ranlib
           llvm-readobj
           llvm-rtdyld

llvm/test/lit.cfg

for pattern in [r"\bbugpoint\b(?!-)",
                 r"\bllvm-dwarfdump\b",
                 r"\bllvm-extract\b",
                 r"\bllvm-lib\b",
                 r"\bllvm-link\b",
                 r"\bllvm-lto\b",
                 r"\bllvm-lto2\b",
                 r"\bllvm-mc\b",
                 r"\bllvm-mcmarkup\b",
+                r"\bllvm-modextract\b",
                 r"\bllvm-nm\b",
                 r"\bllvm-objdump\b",
                 r"\bllvm-pdbdump\b",
                 r"\bllvm-profdata\b",
                 r"\bllvm-ranlib\b",
                 r"\bllvm-readobj\b",
                 r"\bllvm-rtdyld\b",
                 r"\bllvm-size\b",

llvm/test/tools/llvm-modextract/single.ll

+; RUN: llvm-as -o %t %s
+; RUN: llvm-modextract -n 0 -o - %t | llvm-dis | FileCheck %s
+; RUN: not llvm-modextract -n 1 -o - %t 2>&1 | FileCheck --check-prefix=ERROR %s
+; RUN: llvm-modextract -b -n 0 -o - %t | llvm-dis | FileCheck %s
+; RUN: not llvm-modextract -b -n 1 -o - %t 2>&1 | FileCheck --check-prefix=ERROR %s
+
+; CHECK: define void @f()
+; ERROR: llvm-modextract: error: module index out of range; bitcode file contains 1 module(s)
+
+define void @f() {
+  ret void
+}

llvm/tools/LLVMBuild.txt

  llvm-dwarfdump
  llvm-dwp
  llvm-extract
  llvm-jitlistener
  llvm-link
  llvm-lto
  llvm-mc
  llvm-mcmarkup
+ llvm-modextract
  llvm-nm
  llvm-objdump
  llvm-pdbdump
  llvm-profdata
  llvm-rtdyld
  llvm-size
  llvm-split
  opt
  verify-uselistorder
 
 [component_0]
 type = Group
 name = Tools
 parent = $ROOT

llvm/tools/llvm-modextract/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  IRReader
+  BitWriter
+  Core
+  Support
+  )
+
+add_llvm_tool(llvm-modextract
+  llvm-modextract.cpp
+  )

llvm/tools/llvm-modextract/LLVMBuild.txt

-;===- ./tools/LLVMBuild.txt ------------------------------------*- Conf -*--===;
+;===- ./tools/llvm-modextract/LLVMBuild.txt --------------------*- Conf -*--===;
 ;
 ;                     The LLVM Compiler Infrastructure
 ;
 ; This file is distributed under the University of Illinois Open Source
 ; License. See LICENSE.TXT for details.
 ;
 ;===------------------------------------------------------------------------===;
 ;
 ; This is an LLVMBuild description file for the components in this subdirectory.
 ;
 ; For more information on the LLVMBuild system, please see:
 ;
 ;   http://llvm.org/docs/LLVMBuild.html
 ;
 ;===------------------------------------------------------------------------===;
 
-[common]
-subdirectories =
- bugpoint
- dsymutil
- llc
- lli
- llvm-ar
- llvm-as
- llvm-bcanalyzer
- llvm-cat
- llvm-cov
- llvm-diff
- llvm-dis
- llvm-dwarfdump
- llvm-dwp
- llvm-extract
- llvm-jitlistener
- llvm-link
- llvm-lto
- llvm-mc
- llvm-mcmarkup
- llvm-nm
- llvm-objdump
- llvm-pdbdump
- llvm-profdata
- llvm-rtdyld
- llvm-size
- llvm-split
- opt
- verify-uselistorder
-
 [component_0]
-type = Group
-name = Tools
-parent = $ROOT
+type = Tool
+name = llvm-modextract
+parent = Tools
+required_libraries = BitReader BitWriter

llvm/tools/llvm-modextract/llvm-modextract.cpp

+//===-- llvm-modextract.cpp - LLVM module extractor utility ---------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This program is for testing features that rely on multi-module bitcode files.
+// It takes a multi-module bitcode file, extracts one of the modules and writes
+// it to the output file.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Bitcode/BitcodeReader.h"
+#include "llvm/Bitcode/BitcodeWriter.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/ToolOutputFile.h"
+
+using namespace llvm;
+
+static cl::opt<bool>
+    BinaryExtract("b", cl::desc("Whether to perform binary extraction"));
+
+static cl::opt<std::string> OutputFilename("o", cl::Required,
+                                           cl::desc("Output filename"),
+                                           cl::value_desc("filename"));
+
+static cl::opt<std::string>
+    InputFilename(cl::Positional, cl::desc("<input bitcode>"), cl::init("-"));
+
+static cl::opt<unsigned> ModuleIndex("n", cl::Required,
+                                     cl::desc("Index of module to extract"),
+                                     cl::value_desc("index"));
+
+int main(int argc, char **argv) {
+  cl::ParseCommandLineOptions(argc, argv, "Module extractor");
+
+  ExitOnError ExitOnErr("llvm-modextract: error: ");
+
+  std::unique_ptr<MemoryBuffer> MB =
+      ExitOnErr(errorOrToExpected(MemoryBuffer::getFileOrSTDIN(InputFilename)));
+  std::vector<BitcodeModule> Ms = ExitOnErr(getBitcodeModuleList(*MB));
+
+  LLVMContext Context;
+  if (ModuleIndex >= Ms.size()) {
+    errs() << "llvm-modextract: error: module index out of range; bitcode file "
+              "contains "
+           << Ms.size() << " module(s)\n";
+    return 1;
+  }
+
+  std::error_code EC;
+  std::unique_ptr<tool_output_file> Out(
+      new tool_output_file(OutputFilename, EC, sys::fs::F_None));
+  ExitOnErr(errorCodeToError(EC));
+
+  if (BinaryExtract) {
+    SmallVector<char, 0> Header;
+    BitcodeWriter Writer(Header);
+    Out->os() << Header << Ms[ModuleIndex].getBuffer();
+    return 0;
+  }
+
+  std::unique_ptr<Module> M = ExitOnErr(Ms[ModuleIndex].parseModule(Context));
+  WriteBitcodeToFile(M.get(), Out->os());
+
+  return 0;
+}