llvm-mc-fuzzer: A fuzzing tool for the MC layer.

[dsanders]

Summary

Only the disassembler is supported in this patch but it has already found a few
issues in the Mips disassembler (mostly invalid instructions being successfully
disassembled).

[dsanders]

The recent threads on libFuzzer inspired me to try it out and llvm-mc-fuzzer is
the result. It seems to be useful and has already found a few issues with the
Mips disassembler, so I thought I'd post the patch to see if there's interest
in bringing this upstream.

[dsanders]

I forgot to mention: I also have some python scripts to convert our disassembler test inputs to the raw binary format used by libFuzzer and back. I haven't included those in this patch since they are just quick hacks. They'll need implementing properly to be suitable for upstreaming.

[kcc]

Nice!

Could you please also add a line about your findings
to http://llvm.org/docs/LibFuzzer.html#trophies
(or tell me what to add there) ?

Once committed, I'll add it to the fuzzer bot:
lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fuzzer

tools/llvm-mc-fuzzer/llvm-mc-fuzzer.cpp

23

please use a C++ constant

63

why not vector?

82

Since the command line is unusual (compared to other uses of libFuzzer)
please add a comment with usage example(s)

104

Do you really need to copy these here?
Why not just pass I.c_str()?

110

Do you need it to be this complex?
For assembling you can write a separate target binary

[dsanders]

tools/llvm-mc-fuzzer/llvm-mc-fuzzer.cpp

63

I didn't give it much thought. I'll switch it to a vector.

104

Unfortunately, yes but there is an alternate solution. The problem is that c_str() returns a const char * but fuzzer::FuzzerDriver() expects an array of char *. It's unsafe to just drop the const-ness so I make a non-const copy.

If fuzzer::FuzzerDriver's second was const char ** then I could avoid the copy.

110

My thinking was that it would be nice if llvm-mc-fuzzer had a similar command line to llvm-mc. If it's preferred to use a separate binary then I don't mind doing that.

[dsanders]

Replaced macro with C++ constant.
Added trophy to libFuzzer documentation.
Changed DataCopy to a std::vector.
Added usage examples.

[kcc]

docs/LibFuzzer.rst

452–459

Links maybe?

tools/llvm-mc-fuzzer/llvm-mc-fuzzer.cpp

99

typeo: contrain

105

I've just added more interface variants:
Can you try this one?

int FuzzerDriver(const std::vector<std::string> &Args, UserCallback Callback);

111

Ok, makes sense

[silvas]

Nice! I know Russell had been looking at using fuzz-testing to test
round-tripping through assembly, which seems like a perfect fit for a
libFuzzer-based tool. Russell, is this something that you are still working
on? Maybe llvm-mc-fuzzer will grow that functionality some day.

• Sean Silva

[russell.gallop]

I found a number of bugs comparing direct object emission and via assembly with the check_cfc tool. Sean I and spoke about using fuzzing in that area. I expect it would find issues but I haven't done any work on it so no objections if someone wanted to add this to llvm-mc-fuzzer.

[dsanders]

Added links to libFuzzer.rst and better explained it.
Fixed contrain -> constrain.
Switched to the new FuzzerDriver() interface.

[dsanders]

In D12723#243586, @silvas wrote:

Nice! I know Russell had been looking at using fuzz-testing to test
round-tripping through assembly, which seems like a perfect fit for a
libFuzzer-based tool. Russell, is this something that you are still working
on? Maybe llvm-mc-fuzzer will grow that functionality some day.

• Sean Silva

I'm keen to get that functionality too. Mips's move instructions will be a bit troublesome here since many distinct opcodes disassemble to 'move $1, $2' but that string only assembles to a single opcode.

One feature that would be helpful from the Fuzzer is the ability for the callback to be able to classify inputs into various bins. For example, "this input is invalid", "this input disassembled but failed to complete the round trip", "this input completed a round trip but the encodings don't match", etc. At the moment, we need to determine this when converting inputs into test cases which seems redundant when the callback already knew what happened.

tools/llvm-mc-fuzzer/llvm-mc-fuzzer.cpp

105

That worked nicely. Thanks

[kcc]

One feature that would be helpful from the Fuzzer is the ability for the callback to be able to classify inputs into various bins. For example, "this input is invalid", "this input disassembled but failed to complete the round trip", "this input completed a round trip but the encodings don't match", etc. At the moment, we need to determine this when converting inputs into test cases which seems redundant when the callback already knew what happened.

Yes, I've seen similar requests already.
goFuzz does it this way:

The function must return 1 if the input is interesting in some way (for example, it was parsed successfully, that is, it is lexically correct, go-fuzz will give more priority to such inputs); -1 if the input must not be added to corpus even if gives new coverage; and 0 otherwise; other values are reserved for future use.

So, I'll probably add some similar functionality (probably not in the nearest two weeks though).

[kcc]

LGTM, thanks for doing this!

tools/llvm-mc-fuzzer/llvm-mc-fuzzer.cpp

127

I would just call fuzzer::FuzzerDriver and not create the "TestOneInput" temporary.
But feel free to ignore this comment.

docs/LibFuzzer.rst

* LLVM:
   * Clang: https://llvm.org/bugs/show_bug.cgi?id=23057
 
   * Clang-format: https://llvm.org/bugs/show_bug.cgi?id=23052
 
   * libc++: https://llvm.org/bugs/show_bug.cgi?id=24411
 
   * llvm-as: https://llvm.org/bugs/show_bug.cgi?id=24639
 
-
+  * Disassembler:
+    * Mips: Discovered a number of untested instructions for the Mips target
+      (see valid-mips*.s in http://reviews.llvm.org/rL247405,
+      http://reviews.llvm.org/rL247414, http://reviews.llvm.org/rL247416,
+      http://reviews.llvm.org/rL247417, http://reviews.llvm.org/rL247420,
+      and http://reviews.llvm.org/rL247422) as well some instructions that
+      successfully disassembled on ISA's where they were not valid (see
+      invalid-xfail.s files in the same commits).
 
 .. _pcre2: http://www.pcre.org/
 
 .. _AFL: http://lcamtuf.coredump.cx/afl/
 
 .. _SanitizerCoverage: http://clang.llvm.org/docs/SanitizerCoverage.html
 .. _SanitizerCoverageTraceDataFlow: http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-data-flow
 .. _DataFlowSanitizer: http://clang.llvm.org/docs/DataFlowSanitizer.html
 
 .. _Heartbleed: http://en.wikipedia.org/wiki/Heartbleed
 
 .. _FuzzerInterface.h: https://github.com/llvm-mirror/llvm/blob/master/lib/Fuzzer/FuzzerInterface.h

tools/llvm-mc-fuzzer/CMakeLists.txt

+if( LLVM_USE_SANITIZE_COVERAGE )
+  include_directories(BEFORE
+    ${CMAKE_CURRENT_SOURCE_DIR}/../../lib/Fuzzer)
+
+  set(LLVM_LINK_COMPONENTS
+      AllTargetsDescs
+      AllTargetsDisassemblers
+      AllTargetsInfos
+      MC
+      MCDisassembler
+      Support
+      )
+  add_llvm_tool(llvm-mc-fuzzer 
+                llvm-mc-fuzzer.cpp)
+  target_link_libraries(llvm-mc-fuzzer
+                        LLVMFuzzerNoMain
+                        )
+endif()

tools/llvm-mc-fuzzer/llvm-mc-fuzzer.cpp

+//===--- llvm-mc-fuzzer.cpp - Fuzzer for the MC layer ---------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm-c/Disassembler.h"
+#include "llvm-c/Target.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/MC/SubtargetFeature.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/raw_ostream.h"
+#include "FuzzerInterface.h"
+
+using namespace llvm;
+
+const unsigned AssemblyTextBufSize = 80;
+
+enum ActionType {
+  AC_Assemble,
+  AC_Disassemble
+};
+
+static cl::opt<ActionType>
+Action(cl::desc("Action to perform:"),
+       cl::init(AC_Assemble),
+       cl::values(clEnumValN(AC_Assemble, "assemble",
+                             "Assemble a .s file (default)"),
+                  clEnumValN(AC_Disassemble, "disassemble",
+                             "Disassemble strings of hex bytes"),
+                  clEnumValEnd));
+
+static cl::opt<std::string>
+    TripleName("triple", cl::desc("Target triple to assemble for, "
+                                  "see -version for available targets"));
+
+static cl::opt<std::string>
+    MCPU("mcpu",
+         cl::desc("Target a specific cpu type (-mcpu=help for details)"),
+         cl::value_desc("cpu-name"), cl::init(""));
+
+static cl::list<std::string>
+    MAttrs("mattr", cl::CommaSeparated,
+           cl::desc("Target specific attributes (-mattr=help for details)"),
+           cl::value_desc("a1,+a2,-a3,..."));
+// The feature string derived from -mattr's values.
+std::string FeaturesStr;
+
+static cl::list<std::string>
+    FuzzerArgv("fuzzer-args", cl::Positional,
+               cl::desc("Options to pass to the fuzzer"), cl::ZeroOrMore,
+               cl::PositionalEatsArgs);
+
+void DisassembleOneInput(const uint8_t *Data, size_t Size) {
+  char AssemblyText[AssemblyTextBufSize];
+
+  std::vector<uint8_t> DataCopy(Data, Data + Size);
+
+  LLVMDisasmContextRef Ctx = LLVMCreateDisasmCPUFeatures(
+      TripleName.c_str(), MCPU.c_str(), FeaturesStr.c_str(), nullptr, 0,
+      nullptr, nullptr);
+  assert(Ctx);
+  uint8_t *p = DataCopy.data();
+  unsigned Consumed;
+  do {
+    Consumed = LLVMDisasmInstruction(Ctx, p, Size, 0, AssemblyText,
+                                     AssemblyTextBufSize);
+    Size -= Consumed;
+    p += Consumed;
+  } while (Consumed != 0);
+  LLVMDisasmDispose(Ctx);
+}
+
+int main(int argc, char **argv) {
+  // The command line is unusual compared to other fuzzers due to the need to
+  // specify the target. Options like -triple, -mcpu, and -mattr work like
+  // their counterparts in llvm-mc, while -fuzzer-args collects options for the
+  // fuzzer itself.
+  //
+  // Examples:
+  //
+  // Fuzz the big-endian MIPS32R6 disassembler using 100,000 inputs of up to
+  // 4-bytes each and use the contents of ./corpus as the test corpus:
+  //   llvm-mc-fuzzer -triple mips-linux-gnu -mcpu=mips32r6 -disassemble \
+  //       -fuzzer-args -max_len=4 -runs=100000 ./corpus
+  //
+  // Infinitely fuzz the little-endian MIPS64R2 disassembler with the MSA
+  // feature enabled using up to 64-byte inputs:
+  //   llvm-mc-fuzzer -triple mipsel-linux-gnu -mcpu=mips64r2 -mattr=msa \
+  //       -disassemble -fuzzer-args ./corpus
+  //
+  // If your aim is to find instructions that are not tested, then it is
+  // advisable to constrain the maximum input size to a single instruction
+  // using -max_len as in the first example. This results in a test corpus of
+  // individual instructions that test unique paths. Without this constraint,
+  // there will be considerable redundancy in the corpus.
+
+  LLVMInitializeAllTargetInfos();
+  LLVMInitializeAllTargetMCs();
+  LLVMInitializeAllDisassemblers();
+
+  cl::ParseCommandLineOptions(argc, argv);
+
+  // Package up features to be passed to target/subtarget
+  // We have to pass it via a global since the callback doesn't
+  // permit any user data.
+  if (MAttrs.size()) {
+    SubtargetFeatures Features;
+    for (unsigned i = 0; i != MAttrs.size(); ++i)
+      Features.AddFeature(MAttrs[i]);
+    FeaturesStr = Features.getString();
+  }
+
+  // Insert the program name into the FuzzerArgv.
+  FuzzerArgv.insert(FuzzerArgv.begin(), argv[0]);
+
+  void (*TestOneInput)(const uint8_t *, size_t) = nullptr;
+
+  if (Action == AC_Assemble)
+    errs() << "error: -assemble is not implemented\n";
+  else if (Action == AC_Disassemble)
+    TestOneInput = DisassembleOneInput;
+  else
+    llvm_unreachable("Unknown action");
+
+  return fuzzer::FuzzerDriver(FuzzerArgv, TestOneInput);
+}