Adding proto-fuzzer support for more extensions with new driver flow

[jocewei]

Summary

• Uses Protobuf to represent RV32 General (G) Instruction Set with extensions I,M,A,F,D,C, with no constraints on either register or immediate operand values (that is, with no constraints on register class or immediate range). Also includes pseudoinstructions.
• Updated fuzzer driver flow:

(1) to invoke fuzzed llvm-mc assembler, GNU assembler, and GNU disassembler.
(2) To use the status of calling each tool to categorize test failures.

• Created single fuzzer executable that supports rv32imafdc.
• Kept the example fuzzer with sample grammar and corresponding Python script.

Related: https://reviews.llvm.org/D49521

[asb]

Hi Jocelyn. What should I be referring to as the base of this patch? e.g. D49521 doesn't contain proto_to_asm_rv32i.cpp?

I might be looking at the wrong version of the protobuf API or overlooking other issues here, but might it be possible to use FindValueByNumber to get hold of an EnumDescriptor and hence a string name for protobuf enums, getting rid of some of the boilerplate in proto_to_asm? https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor#EnumDescriptor.FindValueByNumber.details

[jocewei]

In D50164#1185810, @asb wrote:

Hi Jocelyn. What should I be referring to as the base of this patch? e.g. D49521 doesn't contain proto_to_asm_rv32i.cpp?

This patch adds two new files: proto_to_asm_rv32i.cpp and rv32i_asm.proto. These files add to proto_to_asm.cpp and asm_proto.proto from the previous patch (https://reviews.llvm.org/D49521), which we chose to leave untouched so as to preserve the simpler example grammar that only contains the add and sub instructions.

I might be looking at the wrong version of the protobuf API or overlooking other issues here, but might it be possible to use FindValueByNumber to get hold of an EnumDescriptor and hence a string name for protobuf enums, getting rid of some of the boilerplate in proto_to_asm? https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor#EnumDescriptor.FindValueByNumber.details

In proto_to_asm, to handle the protobuf enums, we can use int values 0-31 instead of Register::X0 through Register::X31, or int values 0-9 instead of RTypeOpcode_Op_ADD, etc. But this means that if the order of items in the protobuf enum cannot change without us also modifying the corresponding proto_to_asm code. Based on what I read, I'm not sure if using EnumDescriptor / EnumValueDescriptor would be advantageous.

[apazos]

tools/llvm-mc-assemble-proto-fuzzer/proto-files/rv32i_asm.proto

113

When you have the Load instructions ready, upload to this same patch so it has RV32I complete in this patch, except for the CSR instrs.

[asb]

In D50164#1186326, @jocewei wrote:

In D50164#1185810, @asb wrote:

Hi Jocelyn. What should I be referring to as the base of this patch? e.g. D49521 doesn't contain proto_to_asm_rv32i.cpp?

This patch adds two new files: proto_to_asm_rv32i.cpp and rv32i_asm.proto. These files add to proto_to_asm.cpp and asm_proto.proto from the previous patch (https://reviews.llvm.org/D49521), which we chose to leave untouched so as to preserve the simpler example grammar that only contains the add and sub instructions.

But the patch as upload modifies a few lines of proto_to_asm_rv32i.cpp rather than adding it from scratch, which is why I wondered if there's an intermediate patch or a problem with the diff generation.

I might be looking at the wrong version of the protobuf API or overlooking other issues here, but might it be possible to use FindValueByNumber to get hold of an EnumDescriptor and hence a string name for protobuf enums, getting rid of some of the boilerplate in proto_to_asm? https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor#EnumDescriptor.FindValueByNumber.details

In proto_to_asm, to handle the protobuf enums, we can use int values 0-31 instead of Register::X0 through Register::X31, or int values 0-9 instead of RTypeOpcode_Op_ADD, etc. But this means that if the order of items in the protobuf enum cannot change without us also modifying the corresponding proto_to_asm code. Based on what I read, I'm not sure if using EnumDescriptor / EnumValueDescriptor would be advantageous.

I was just thinking of getting rid of the manual enum -> string conversion. Looks like you'd need to convert the protobuf-generated string to lower-case though.

[jocewei]

In D50164#1187184, @asb wrote:

In D50164#1186326, @jocewei wrote:

In D50164#1185810, @asb wrote:

Hi Jocelyn. What should I be referring to as the base of this patch? e.g. D49521 doesn't contain proto_to_asm_rv32i.cpp?

This patch adds two new files: proto_to_asm_rv32i.cpp and rv32i_asm.proto. These files add to proto_to_asm.cpp and asm_proto.proto from the previous patch (https://reviews.llvm.org/D49521), which we chose to leave untouched so as to preserve the simpler example grammar that only contains the add and sub instructions.

But the patch as upload modifies a few lines of proto_to_asm_rv32i.cpp rather than adding it from scratch, which is why I wondered if there's an intermediate patch or a problem with the diff generation.

You're right, there was a missing patch, now added: https://reviews.llvm.org/D50262

I might be looking at the wrong version of the protobuf API or overlooking other issues here, but might it be possible to use FindValueByNumber to get hold of an EnumDescriptor and hence a string name for protobuf enums, getting rid of some of the boilerplate in proto_to_asm? https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor#EnumDescriptor.FindValueByNumber.details

In proto_to_asm, to handle the protobuf enums, we can use int values 0-31 instead of Register::X0 through Register::X31, or int values 0-9 instead of RTypeOpcode_Op_ADD, etc. But this means that if the order of items in the protobuf enum cannot change without us also modifying the corresponding proto_to_asm code. Based on what I read, I'm not sure if using EnumDescriptor / EnumValueDescriptor would be advantageous.

I was just thinking of getting rid of the manual enum -> string conversion. Looks like you'd need to convert the protobuf-generated string to lower-case though.

Okay, that makes sense. I'll try that, thanks!

[jocewei]

Added Load instructions to fuzzer's Protobuf representation of the RV32I ISA (rv32i_asm.proto and proto_to_asm_rv32i.cpp)

[apazos]

Jocelyn https://reviews.llvm.org/D50262 is unfinished work and should not be pushed, it is confusing the review.
You should update this patch with any missing files and with the latest version you have. Abandon https://reviews.llvm.org/D50262.

[apazos]

I think Alex's suggestion is doable. We should try to simplify the printing of the instructions.

const EnumValueDescriptor * D = FindValueByNumber(X.name());
if (D) {

OS << D.name(); 
return OS;

}
assert(..);

[jocewei]

Rebased patch and added fuzzer support for more extensions (m,c)

[jocewei]

Cleaned up proto_to_asm, simplified grammar to have fewer messages and be more easily extensible

[jocewei]

Added missing instructions to rv32i.proto and rv32c.proto

[jocewei]

Updated with pseudoinstructions added to the Protobuf

[jocewei]

Abandoning revision because tool has been moved to llvm/tools from clang/tools, as of this patch: https://reviews.llvm.org/D51144

tools/llvm-mc-assemble-proto-fuzzer/README.txt

 
 -------------------------------------------------------------------------------
 Running:
 -------------------------------------------------------------------------------
 Run the Python script. Specify any args you would like to run the
 fuzzer with, including the corpus directory and the full path to objdump.
 For example:
 
-$ python mcfuzz.py --corpus corpus --max-len 32 --runs 10 --triple riscv32
-  --out outputdir --objdump /full/path/to/riscv32-unknown-linux-gnu-objdump
-  --mattr +c,+m
+$ python mcfuzz.py --corpus corpus --max-len 32 --runs 10 --triple riscv32 \
+  --out outputdir --objdump /full/path/to/riscv32-unknown-linux-gnu-objdump \
+  --mattr +c,+m --march rv32imc \
+  --assemble /full/path/to/riscv32-unknown-linux-gnu-as
 
 Run the script with the --verbose flag if you would like to see the output
 of the fuzzer as it runs.
 
 Use the --riscv-no-aliases flag if you want the generated .s files (from
 proto-to-asm) to be printed in their original form (ie to print
   `sub x3,x0,x3` to the file instead of
   `neg x3,x4`). Note that using this flag (--riscv-no-aliases) may cause

 script concludes.
 
 The last part of the script runs a diff of all the .s and .objdump files, and
 prints out a summary of results (how many diffs passed and how many failed).
 
 -------------------------------------------------------------------------------
 Notes:
 -------------------------------------------------------------------------------
-The fuzzer currently supports all the instructions in the RV32I ISA, except
-for the following instructions: FENCE, FENCE.I, ECALL, EBREAK, CSRRW, CSRRS,
-CSRRC, CSRRWI, CSRRSI, and CSRRCI. Currently, running the Python script will
-result in some failures as instructions are given aliases by the golden
-disassembler, and therefore the diff fails.
+Currently, running the Python script will result in some failures as
+instructions are given aliases by the golden disassembler, and therefore
+the diff fails.

tools/llvm-mc-assemble-proto-fuzzer/mcfuzz.py

 #!/usr/bin/env python
 
 import argparse
 import difflib
 import os
 import shlex
 import subprocess
 import sys
 import tempfile
 
+
+def main():
     parser = argparse.ArgumentParser()
+    parse_arguments(parser)
+    args = parser.parse_args()
+
+    # Step 1: Invoke fuzzer to generate a corpus.
+    call_fuzzer(args)
+
+    # If user did not specify output dir, skip steps 2-8; exit.
+    if args.out is None:
+        print("No output directory specified; exiting after populating "
+              "corpus directory.")
+        sys.exit(0)
+
+    # List of corpus files that both LLVM AS and GNU AS fail to assemble.
+    list_of_llvm_as_and_gnu_as_fails = []
+    # List of corpus files that only LLVM AS fails to assemble.
+    list_of_llvm_as_fails = []
+    # List of corpus files that only GNU AS fails to assemble.
+    list_of_gnu_as_fails = []
+    # List of corpus files that both LLVM AS and GNU AS can assemble.
+    list_of_llvm_as_and_gnu_as_passes = []
+    # Keep track of which diffs pass / fail.
+    passes = 0
+    fails = 0
+    list_of_diffs = []
+
+    # Iterate through the corpus body.
 
+    # For each file, re-run the fuzzer to generate an object file with the fuzz
+    # target. Also run the golden assembler (gnu as) to generate a reference
+    # object file. If both assemblers behave the same way, proceed to generate
+    # a .s file (using proto-to-asm tool), disassemble the object file
+    # generated by the fuzz target (using objdump) and then compare the
+    # resulting .objdump with your .s file.
+    for filename in os.listdir(args.corpus):
+        # Step 2: Run fuzzer with filetype=obj; check for error in fuzz target.
+        target_error_occurred = call_fuzzer_on_corpus_file(args, filename)
+
+        # Step 3: Run proto-to-asm on corpus file to generate .s file.
+        call_proto_to_asm(args, filename)
+
+        # Step 4: Run golden assembler (gnu) and check for error.
+        golden_error_occurred = call_golden_assembler(args, filename)
+
+        # Step 5: Compare behavior of fuzz target and golden assembler.
+        ####################################################################
+        #-- Initial, tentative interpretation of results, based on the   --#
+        #-- status (pass/fail) of tools (LLVM AS, GNU AS, GNU OBJDUMP)   --#
+        #------------------------------------------------------------------#
+        # LLVM AS | GNU AS | GNU OBJDUMP |    Conclusion                   #
+        #------------------------------------------------------------------#
+        #    0    |   0    |     x       | invalid/unimplemented instr     #
+        #    0    |   1    |     x       | LLVM MC bug/unimplemented instr #
+        #    1    |   0    |     x       | LLVM MC bug                     #
+        #    1    |   1    |     0       | LLVM MC bug and GCC bug         #
+        #    1    |   1    |     1       | success                         #
+        ####################################################################
+        if target_error_occurred and golden_error_occurred:
+            print("Both assemblers failed to assemble file: " + filename)
+            list_of_llvm_as_and_gnu_as_fails.append(filename)
+            continue
+        elif target_error_occurred:
+            print("Only the target AS failed to assemble file: " + filename)
+            list_of_llvm_as_fails.append(filename)
+            continue
+        elif golden_error_occurred:
+            print("Only the golden AS failed to assemble file: " + filename)
+            list_of_gnu_as_fails.append(filename)
+            continue
+        else:
+            print("Both assemblers assembled file: " + filename)
+            list_of_llvm_as_and_gnu_as_passes.append(filename)
+
+        # Step 6: Call objdump on each .o file (generated by fuzz target) in
+        # the output directory, to generate corresponding .objdump files.
+        objdump_file = call_objdump(args, filename)
+
+        # Step 7: Process files generated by objdump so that the files only
+        # contain instructions.
+        process_objdump_file(objdump_file, args, filename)
+
+        # Step 8: Diff the .s file generated by proto-to-asm with .objdump file
+        # generated by objdump.
+        passes, fails, list_of_diffs = print_file_status(args, filename,
+                                                         passes, fails,
+                                                         list_of_diffs)
+
+    print_result(passes, fails, list_of_llvm_as_and_gnu_as_fails,
+                 list_of_llvm_as_fails, list_of_gnu_as_fails,
+                 list_of_llvm_as_and_gnu_as_passes, list_of_diffs)
+
+    sys.exit(0 if fails == 0 else 1)
+
+
+def parse_arguments(parser):
     # Flags for the directory names, corpus and outputdir.
-parser.add_argument("--corpus", type=str, help="corpus directory name", \
+    parser.add_argument("--corpus", type=str, help="corpus directory name",
                         required=True)
-parser.add_argument("--out", type=str, \
-      help="output directory name for obj files", default=None)
+    parser.add_argument("--out", type=str,
+                        help="output directory name for obj files",
+                        default=None)
 
     # Flags for llvm-mc-assemble-proto-fuzzer: triple, mattr, filetype.
-parser.add_argument("--triple", type=str, help="specify the triple", \
+    parser.add_argument("--triple", type=str, help="specify the triple",
                         default="riscv32")
-parser.add_argument("--mattr", type=str, help="specify mattr", \
+    parser.add_argument("--mattr", type=str, help="specify mattr",
                         default="")
-parser.add_argument("--filetype", type=str, help="asm or obj", \
+    parser.add_argument("--march", type=str, help="specify march",
+                        default="rv32i")
+    parser.add_argument("--filetype", type=str, help="asm or obj",
                         default="obj")
 
     # These args are passed in after the -fuzzer-args flag.
     parser.add_argument("--runs", type=int, help="number of runs", default=100)
-parser.add_argument("--max-len", type=int, \
+    parser.add_argument("--max-len", type=int,
                         help="limit instruction size for fuzzing", default=40)
 
     # Flag specifies the name of objdump executable.
-parser.add_argument("--objdump", type=str, help="specify the path to" \
-      " objdump", default="/prj/llvm-arm/home/common/build_tools/" \
-      "riscv32-gnu-7.2.0/bin/riscv32-unknown-linux-gnu-objdump")
+    parser.add_argument("--objdump", type=str, help="specify the path to"
+                        " objdump", default="/prj/llvm-arm/home/common/"
+                        "build_tools/riscv32-gnu-7.2.0/bin/"
+                        "riscv32-unknown-linux-gnu-objdump")
+
+    # Flag specifies the name of golden assembler executable.
+    parser.add_argument("--assemble", type=str, help="specify the path to"
+                        " golden assembler", default="/prj/llvm-arm/home/"
+                        "common/build_tools/riscv32-gnu-7.2.0/bin/"
+                        "riscv32-unknown-linux-gnu-as")
 
     # Flag specifies that we should print out everything
     parser.add_argument("--verbose", dest="verbose", action="store_true")
     parser.set_defaults(verbose=False)
 
     parser.add_argument("--riscv-no-aliases", dest="NoAliases",
                         action="store_true")
     parser.set_defaults(NoAliases=False)
 
-args = parser.parse_args()
 
-# Step 1: Invoke fuzzer to generate a corpus.
-cmd = 'llvm-mc-assemble-proto-fuzzer-rv32i {corpus} -triple={triple} ' \
-      '-mattr={mattr}' + ' -filetype={filetype} -fuzzer-args -runs={runs} ' \
-      '-max_len={max_len}'
-cmd = cmd.format(corpus=args.corpus, triple=args.triple, mattr=args.mattr, \
-      filetype=args.filetype, runs=args.runs, max_len=args.max_len)
+# This function calls the llvm-mc-assemble-proto-fuzzer with a given extension
+def call_fuzzer(args):
+    cmd = 'llvm-mc-assemble-proto-fuzzer-{march} {corpus} \
+          -triple={triple} -mattr={mattr} -filetype={filetype} -fuzzer-args ' \
+          + '-runs={runs} -max_len={max_len}'
+    cmd = cmd.format(march=args.march, corpus=args.corpus, triple=args.triple,
+                     mattr=args.mattr, filetype=args.filetype,
+                     runs=args.runs, max_len=args.max_len)
     fuzz_command = shlex.split(cmd)
 
-fuzz_proc = subprocess.Popen(fuzz_command, stdout=subprocess.PIPE, \
+    fuzz_proc = subprocess.Popen(fuzz_command, stdout=subprocess.PIPE,
                                  stderr=subprocess.STDOUT)
     fuzz_out, fuzz_err = fuzz_proc.communicate()
     if fuzz_proc.returncode != 0:
-    raise ValueError('failed to run fuzz {}: {}'.format(fuzz_command, fuzz_err))
+        raise ValueError('failed to run fuzz {}: {}'.format(fuzz_command,
+                         fuzz_err))
     if args.verbose:
         print(fuzz_out)
 
-# If user specified an output directory, proceed to step 2; otherwise, exit.
-if args.out is None:
-    print("No output directory specified; exiting after populating "
-          "corpus directory.")
-    sys.exit(0)
 
-# Keep track of which diffs pass / fail.
-passes = 0
-fails = 0
-list_of_failures = []
-# Iterate through the corpus body.
-# For each file, generate a .s and a .o file. Then, use objdump to
-# generate a .objdump file, which we will compare to the .s file.
-# Diff the .s with the corresponding .objdump file.
-for filename in os.listdir(args.corpus):
+def call_fuzzer_on_corpus_file(args, filename):
+    target_error_occurred = False
+    cmd = 'llvm-mc-assemble-proto-fuzzer-{march} {corpus}/{file} ' \
+          + '-triple={triple} -mattr={mattr} -out={out} ' \
+          + '-filetype=obj -fuzzer-args -runs=1'
+    cmd = cmd.format(march=args.march, corpus=args.corpus, file=filename,
+                     triple=args.triple, mattr=args.mattr, out=args.out)
+    target_as_command = shlex.split(cmd)
+    target_as_proc = subprocess.Popen(target_as_command,
+                                      stdout=subprocess.PIPE,
+                                      stderr=subprocess.STDOUT)
+    target_as_out, target_as_err = target_as_proc.communicate()
+    if target_as_proc.returncode != 0:
+        raise ValueError('failed to run fuzzer {}: {}'.format(
+                         target_as_command, target_as_err))
+    if target_as_out:
+        if "error" in target_as_out:
+            print("Fuzz Target failed to assemble this input: " + filename)
+            target_error_occurred = True
+    if args.verbose:
+        print(target_as_out)
+    return target_error_occurred
 
-    filename_prefix = args.out + "/" + filename
 
-    # Step 2: Run proto-to-asm on corpus file to generate .s file.
-    proto_to_asm_command = ["llvm-mc-assemble-proto-to-asm-rv32i", \
-          args.corpus + "/" + filename, "-riscv-no-aliases=" +
-          str(args.NoAliases)]
+def call_proto_to_asm(args, filename):
+    filename_prefix = args.out + "/" + filename
+    cmd = 'llvm-mc-assemble-proto-to-asm-{march} {corpus}/{file}' \
+          + ' -riscv-no-aliases={NoAliases}'
+    cmd = cmd.format(march=args.march, corpus=args.corpus, file=filename,
+                     NoAliases=args.NoAliases)
+    proto_to_asm_command = shlex.split(cmd)
     asm_file = open(filename_prefix + ".s", "w+r")
     asm_proc = subprocess.Popen(proto_to_asm_command, stdout=asm_file,
                                 stderr=subprocess.STDOUT)
     asm_out, asm_err = asm_proc.communicate()
     if asm_proc.returncode != 0:
         raise ValueError('failed to run {}: {}'.format(proto_to_asm_command,
                          asm_err))
+    elif asm_out:
+        print("Asm_out: " + asm_out)
+    asm_file.close()
 
-    # Step 3: Generate .o files in the outputdir.
-    cmd = 'llvm-mc-assemble-proto-fuzzer-rv32i {corpus}/{file} ' \
-          + '-triple={triple} -mattr={mattr} -out={out} ' \
-          + '-filetype=obj -fuzzer-args -runs=1'
-    cmd = cmd.format(corpus=args.corpus, file=filename, triple=args.triple, \
-          mattr=args.mattr, out=args.out)
-    obj_files_command = shlex.split(cmd)
-    obj_proc = subprocess.Popen(obj_files_command, stdout=subprocess.PIPE,
+
+def call_golden_assembler(args, filename):
+    golden_error_occurred = False
+    cmd = '{assemble} {dirname}/{filename}.s -march={march} ' \
+          + '-o {dirname}/{filename}.out'
+    cmd = cmd.format(assemble=args.assemble, march=args.march,
+                     dirname=args.out, filename=filename)
+    golden_as_command = shlex.split(cmd)
+    golden_as_proc = subprocess.Popen(golden_as_command,
+                                      stdout=subprocess.PIPE,
                                       stderr=subprocess.STDOUT)
-    obj_out, obj_err = obj_proc.communicate()
-    if obj_proc.returncode != 0:
-        raise ValueError('failed to run fuzzer {}: {}'.format( \
-              obj_files_command, obj_err))
-    if args.verbose:
-        print(obj_out)
+    golden_as_out, golden_as_err = golden_as_proc.communicate()
+    if golden_as_proc.returncode != 0:
+        if (golden_as_out):
+            print("golden_as_out: " + golden_as_out)
+        golden_error_occurred = True
+    elif args.verbose:
+        print(golden_as_out)
+    return golden_error_occurred
 
-    # Step 4: Call objdump on each .o file in the output directory,
-    # to generate corresponding .objdump files.
+
+def call_objdump(args, filename):
+    filename_prefix = args.out + "/" + filename
     objdump_file = open(filename_prefix + ".objdump", "w+r")
     cmd = '{objdump} -dr -M numeric {filename}.o'
     cmd = cmd.format(objdump=args.objdump, filename=filename_prefix)
     objdump_command = shlex.split(cmd)
     objdump_proc = subprocess.Popen(objdump_command,
-          stdout=objdump_file, stderr=subprocess.STDOUT)
+                                    stdout=objdump_file,
+                                    stderr=subprocess.STDOUT)
     objdump_out, objdump_err = objdump_proc.communicate()
     if objdump_proc.returncode != 0:
-        raise ValueError('failed to run objdump {}: ' \
+        raise ValueError('failed to run objdump {}: '
                          + '{}'.format(objdump_command, objdump_err))
+    return objdump_file
 
-    # Step 5: Process asm files so that files only contain asm instructions.
-    asm_file.seek(0)
-    lines = asm_file.readlines()
-    asm_file.close()
-    write_asm_file = open(filename_prefix + ".parsed_s", "w")
-    # Remove the first line of .s file, which contains the filename. For example:
-    # // corpus/dff0318decde43ce5065a4209412aa2c68d01318
-    for line in lines:
-        if "\t" in line:
-            write_asm_file.write(line)
-    write_asm_file.close()
 
-    # Step 6: Process files generated by objdump so that the files only contain
-    # instructions.
+def process_objdump_file(objdump_file, args, filename):
+    filename_prefix = args.out + "/" + filename
     objdump_file.seek(0)
     temp_objdump_file = tempfile.TemporaryFile("w+r")
-    # Only the lines containing asm instructions have tabs in them; remove all
-    # other lines (for example, header lines and whitespace).
+    # Only the lines containing asm instructions have tabs in them; remove
+    # all other lines (for example, header lines and whitespace).
     for line in objdump_file:
         if "\t" in line:
             temp_objdump_file.write(line)
     objdump_file.close()
     with open(filename_prefix + ".parsed_objdump", "w") as write_objdump_file:
         temp_objdump_file.seek(0)
         # Each line of the objdump output looks something like this:
         # 0:   00318033                add     x0,x3,x3
         # We remove the first two columns, leaving only the asm instruction.
         for line in temp_objdump_file:
             parts = line.split()[2:]
             for part in parts:
                 write_objdump_file.write("\t")
                 write_objdump_file.write(part)
             write_objdump_file.write("\n")
         temp_objdump_file.close()
 
-    # Step 7: Diff the .s file generated by proto-to-asm and the .objdump file
-    # generated by objdump.
+
+def print_file_status(args, filename, passes, fails, list_of_diffs):
+    filename_prefix = args.out + "/" + filename
     prefix = "Checking " + filename + "..."
-    file1 = open(filename_prefix + ".parsed_s", "r")
+    file1 = open(filename_prefix + ".s", "r")
     file2 = open(filename_prefix + ".parsed_objdump", "r")
     diff = difflib.ndiff(file1.readlines(), file2.readlines())
     delta = ''.join(x[0:] for x in diff if x.startswith(('- ', '+ ')))
     if delta:
         print(prefix + "FAILURE!")
         fails = fails + 1
-        list_of_failures.append(filename)
-        list_of_failures.append(delta)
+        list_of_diffs.append(filename)
+        list_of_diffs.append(delta)
     else:
         print(prefix + "SUCCESS!")
         passes = passes + 1
-
-print("Succeeded: " + str(passes) + "\t\tFailed: " + str(fails) \
-      + "\t\t[Total: " + str(passes + fails) + "]")
+    file1.close()
+    file2.close()
+    return passes, fails, list_of_diffs
+
+
+def print_result(passes, fails, list1, list2, list3, list4, list5):
+    print("Both llvm-mc-assembler and golden (gnu) assembler fail:")
+    count1 = 0
+    str = ""
+    for item in list1:
+        count1 = count1 + 1
+        str = str + item + "\t"
+        if (count1 % 3 == 0):
+            str = str + "\n"
+    print(str)
+
+    str = ""
+    count2 = 0
+    print("Target assembler llvm-mc fails:")
+    for item in list2:
+        count2 = count2 + 1
+        str = str + item + "\t"
+        if (count2 % 3 == 0):
+            str = str + "\n"
+    print(str)
+
+    str = ""
+    count3 = 0
+    print("Golden (gnu) assembler fails:")
+    for item in list3:
+        count3 = count3 + 1
+        str = str + item + "\t"
+        if (count3 % 3 == 0):
+            str = str + "\n"
+    print(str)
+
+    str = ""
+    count4 = 0
+    print("Both assemblers pass:")
+    for item in list4:
+        count4 = count4 + 1
+        str = str + item + "\t"
+        if (count4 % 3 == 0):
+            str = str + "\n"
+    print(str)
 
     if (fails != 0):
         print("The following files failed...")
-    for item in list_of_failures:
+        for item in list5:
             print(item)
+    print("Both llvm as and gnu assembler fail: {}".format(count1))
+    print("Only llvm as fails: {}".format(count2))
+    print("Only gnu as fails: {}".format(count3))
+    print("Both llvm as and gnu assembler pass: {}".format(count4))
+    print("[Succeeded: {}/{}\tFailed: {}/{}]".format(passes,
+          passes + fails, fails, passes + fails))
 
-sys.exit(0 if fails == 0 else 1)
+if __name__ == "__main__":
+    main()

tools/llvm-mc-assemble-proto-fuzzer/proto-files/rv32i_asm.proto

 //===-- rv32i_asm.proto - Protobuf description of ASM ---------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
-/// This file describes a subset of ASM (RV32I) as a protobuf.  It is used to
-///  more easily find interesting inputs for fuzzing the llvm mc layer.
+/// This file describes a subset of ASM (RV32I) as a protobuf, with no
+///  constraints on register or immediate operands.  It is used to more
+///  easily find interesting inputs for fuzzing the llvm mc layer.
+///  The fuzzer currently supports all the instructions in the RV32I ISA,
+///  except for the following instructions: FENCE, FENCE.I, ECALL, EBREAK,
+///  CSRRW, CSRRS, CSRRC, CSRRWI, CSRRSI, and CSRRCI.
 ///
 //===----------------------------------------------------------------------===//
 
 syntax = "proto2";
 
-message ITypeShiftImmediate {
-  required uint32 imm = 1;
+message Immediate {
+  oneof imm_oneof {
+    sint32 s_imm = 1;
+    uint32 u_imm = 2;
   }
-
-message ITypeImmediate {
-  required sint32 imm = 1;
-}
-
-message STypeImmediate {
-  required sint32 imm = 1;
-}
-
-message BTypeImmediate {
-  required sint32 imm = 1;
-}
-
-message UTypeImmediate {
-  required uint32 imm = 1;
-}
-
-message JTypeImmediate {
-  required sint32 imm = 1;
 }
 
 message Register {
   enum RegName {
     X0 = 0;   X1 = 1;   X2 = 2;   X3 = 3;   X4 = 4;
     X5 = 5;   X6 = 6;   X7 = 7;   X8 = 8;   X9 = 9;
     X10 = 10; X11 = 11; X12 = 12; X13 = 13; X14 = 14;
     X15 = 15; X16 = 16; X17 = 17; X18 = 18; X19 = 19;
enum Op {
     SLTIU = 5; JALR = 6;
   };
   required Op op = 1;
 }
 
 message ITypeOperands {
   required Register operand1 = 1;
   required Register operand2 = 2;
-  required ITypeImmediate operand3 = 3;
+  required Immediate operand3 = 3;
 }
 
 message ITypeStatement {
   required ITypeOpcode opcode = 1;
   required ITypeOperands operands = 2;
 }
 
 message ITypeShiftOpcode {
   enum Op {
     SRAI = 0; SRLI = 1; SLLI = 2;
   };
   required Op op = 1;
 }
 
 message ITypeShiftOperands {
   required Register operand1 = 1;
   required Register operand2 = 2;
-  required ITypeShiftImmediate operand3 = 3;
+  required Immediate operand3 = 3;
 }
 
 message ITypeShiftStatement {
   required ITypeShiftOpcode opcode = 1;
   required ITypeShiftOperands operands = 2;
 }
 
 message STypeOpcode {
   enum Op {
     SW = 0; SH = 1; SB = 2;
   };
   required Op op = 1;
 }
 
 message STypeOperands {
   required Register operand1 = 1;
-  required STypeImmediate operand2 = 2;
+  required Immediate operand2 = 2;
   required Register operand3 = 3;
 }
 
 message STypeStatement {
   required STypeOpcode opcode = 1;
   required STypeOperands operands = 2;
 }
 
 message BTypeOpcode {
   enum Op {
     BEQ = 0; BNE = 1; BGE = 2; BLT = 3; BGEU = 4;
     BLTU = 5;
   };
   required Op op = 1;
 }
 
 message BTypeOperands {
   required Register operand1 = 1;
   required Register operand2 = 2;
-  required BTypeImmediate operand3 = 3;
+  required Immediate operand3 = 3;
 }
 
 message BTypeStatement {
   required BTypeOpcode opcode = 1;
   required BTypeOperands operands = 2;
 }
 
 message UTypeOpcode {
   enum Op {
     LUI = 0; AUIPC = 1;
   };
   required Op op = 1;
 }
 
 message UTypeOperands {
   required Register operand1 = 1;
-  required UTypeImmediate operand2 = 2;
+  required Immediate operand2 = 2;
 }
 
 message UTypeStatement {
   required UTypeOpcode opcode = 1;
   required UTypeOperands operands = 2;
 }
 
 message JTypeOpcode {
   enum Op {
     JAL = 0;
   };
   required Op op = 1;
 }
 
 message JTypeOperands {
   required Register operand1 = 1;
-  required JTypeImmediate operand2 = 2;
+  required Immediate operand2 = 2;
 }
 
 message JTypeStatement {
   required JTypeOpcode opcode = 1;
   required JTypeOperands operands = 2;
 }
 
 message AsmStatement {

tools/llvm-mc-assemble-proto-fuzzer/proto-to-asm/proto_to_asm_main.cpp

for (int i = 1; i < argc; i++) {
     else
       NewArgv.push_back(argv[i]);
   }
   cl::ParseCommandLineOptions((int) ModifiedArgv.size(), &ModifiedArgv[0]);
   for (int i = 1; i < (int) NewArgv.size(); i++) {
     std::fstream in(NewArgv[i]);
     std::string str((std::istreambuf_iterator<char>(in)),
                     std::istreambuf_iterator<char>());
-    std::cout << "// " << NewArgv[i] << std::endl;
     std::cout << mc_proto_fuzzer::ProtoToASM(
         reinterpret_cast<const uint8_t *>(str.data()), str.size(),
         (bool) NoAliases);
   }
   return 0;
 }

tools/llvm-mc-assemble-proto-fuzzer/proto-to-asm/proto_to_asm_rv32i.cpp

 //==-- proto_to_asm_rv32i.cpp - Protobuf-ASM conversion --------------------==//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // Implements functions for converting between protobufs for the assembly
 // language grammar for RV32I instruction set and assembly language
-// instructions. 
+// instructions.
 //
 //===----------------------------------------------------------------------===//
 #include "proto_to_asm.h"
 #include "rv32i_asm.pb.h"
 
 #include <ostream>
 #include <sstream>
 
 namespace mc_proto_fuzzer {
 
 static bool NoAliases = false;
 
-std::ostream &operator<<(std::ostream &OS, const ITypeShiftImmediate &X) {
-  // Limit immediate range to [0,31].
-  return OS << X.imm() % 32;
-}
-std::ostream &operator<<(std::ostream &OS, const ITypeImmediate &X) {
-  // Limit immediate range to [-2048, 2047].
-  return OS << X.imm() % 2048;
-}
-std::ostream &operator<<(std::ostream &OS, const STypeImmediate &X) {
-  // Limit immediate range to [-2048, 2047].
-  return OS << X.imm() % 2048;
-}
-std::ostream &operator<<(std::ostream &OS, const BTypeImmediate &X) {
-  // Limit immediate range to [-2048, 2046], in multiples of 2.
-  return OS << ((X.imm() % 2048) / 2) * 2;
-}
-std::ostream &operator<<(std::ostream &OS, const UTypeImmediate &X) {
-  // Limit immediate range to [0, 1048575].
-  return OS << X.imm() % 1048576;
-}
-std::ostream &operator<<(std::ostream &OS, const JTypeImmediate &X) {
-  // Limit immediate range to [-1048576, 1048574], in multiples of 2.
-  return OS << (((X.imm() % 1048576) / 2) * 2);
+std::ostream &operator<<(std::ostream &OS, const Immediate &X) {
+  if (X.has_s_imm())
+    return OS << X.s_imm();
+  else
+    return OS << X.u_imm();
 }
 std::ostream &operator<<(std::ostream &OS, const Register &X) {
   switch (X.name()) {
     case Register::X0:  OS << "x0"; break;
     case Register::X1:  OS << "x1"; break;
     case Register::X2:  OS << "x2"; break;
     case Register::X3:  OS << "x3"; break;
     case Register::X4:  OS << "x4"; break;
switch (X.name()) {
     case Register::X24: OS << "x24"; break;
     case Register::X25: OS << "x25"; break;
     case Register::X26: OS << "x26"; break;
     case Register::X27: OS << "x27"; break;
     case Register::X28: OS << "x28"; break;
     case Register::X29: OS << "x29"; break;
     case Register::X30: OS << "x30"; break;
     case Register::X31: OS << "x31"; break;
+    default: assert(!"The default case was reached.");
   }
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const RTypeOpcode &X) {
   switch (X.op()) {
     case RTypeOpcode_Op_ADD:  OS << "add"; break;
     case RTypeOpcode_Op_SUB:  OS << "sub"; break;
     case RTypeOpcode_Op_SLL:  OS << "sll"; break;
     case RTypeOpcode_Op_SLT:  OS << "slt"; break;
     case RTypeOpcode_Op_SLTU: OS << "sltu"; break;
     case RTypeOpcode_Op_XOR:  OS << "xor"; break;
     case RTypeOpcode_Op_SRL:  OS << "srl"; break;
     case RTypeOpcode_Op_SRA:  OS << "sra"; break;
     case RTypeOpcode_Op_OR:   OS << "or"; break;
     case RTypeOpcode_Op_AND:  OS << "and"; break;
+    default: assert(!"The default case was reached.");
   }
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const RTypeOperands &X) {
   OS << X.operand1();
   OS << "," << X.operand2();
   OS << "," << X.operand3();
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const RTypeStatement &X) {
   OS << "\t" << X.opcode() << "\t";
   OS << X.operands() << "\n";
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const ITypeShiftOpcode &X) {
   switch (X.op()) {
     case ITypeShiftOpcode_Op_SRAI: OS << "srai"; break;
     case ITypeShiftOpcode_Op_SRLI: OS << "srli"; break;
     case ITypeShiftOpcode_Op_SLLI: OS << "slli"; break;
+    default: assert(!"The default case was reached.");
   }
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const ITypeShiftOperands &X) {
   OS << X.operand1();
   OS << "," << X.operand2();
   OS << "," << X.operand3();
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const ITypeShiftStatement &X) {
   OS << "\t" << X.opcode() << "\t";
   OS << X.operands() << "\n";
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const ITypeOpcode &X) {
   switch (X.op()) {
     case ITypeOpcode_Op_ADDI:  OS << "addi"; break;
     case ITypeOpcode_Op_SLTI:  OS << "slti"; break;
     case ITypeOpcode_Op_ANDI:  OS << "andi"; break;
     case ITypeOpcode_Op_ORI:   OS << "ori"; break;
     case ITypeOpcode_Op_XORI:  OS << "xori"; break;
     case ITypeOpcode_Op_SLTIU: OS << "sltiu"; break;
     case ITypeOpcode_Op_JALR:  OS << "jalr"; break;
+    default: assert(!"The default case was reached.");
   }
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const ITypeOperands &X) {
   OS << X.operand1();
   OS << "," << X.operand2();
   OS << "," << X.operand3();
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const ITypeStatement &X) {
   OS << "\t" << X.opcode() << "\t";
   OS << X.operands() << "\n";
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const STypeOpcode &X) {
   switch (X.op()) {
     case STypeOpcode_Op_SW: OS << "sw"; break;
     case STypeOpcode_Op_SH: OS << "sh"; break;
     case STypeOpcode_Op_SB: OS << "sb"; break;
+    default: assert(!"The default case was reached.");
   }
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const STypeOperands &X) {
   OS << X.operand1();
   OS << "," << X.operand2();
   OS << "(" << X.operand3() << ")";
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const STypeStatement &X) {
   OS << "\t" << X.opcode() << "\t";
   OS << X.operands() << "\n";
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const BTypeOpcode &X) {
   switch (X.op()) {
     case BTypeOpcode_Op_BEQ:  OS << "beq"; break;
     case BTypeOpcode_Op_BNE:  OS << "bne"; break;
     case BTypeOpcode_Op_BGE:  OS << "bge"; break;
     case BTypeOpcode_Op_BLT:  OS << "blt"; break;
     case BTypeOpcode_Op_BGEU: OS << "bgeu"; break;
     case BTypeOpcode_Op_BLTU: OS << "bltu"; break;
+    default: assert(!"The default case was reached.");
   }
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const BTypeOperands &X) {
   OS << X.operand1();
   OS << "," << X.operand2();
   OS << "," << X.operand3();
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const BTypeStatement &X) {
   OS << "\t" << X.opcode() << "\t";
   OS << X.operands() << "\n";
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const UTypeOpcode &X) {
   switch (X.op()) {
     case UTypeOpcode_Op_LUI:   OS << "lui"; break;
     case UTypeOpcode_Op_AUIPC: OS << "auipc"; break;
+    default: assert(!"The default case was reached.");
   }
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const UTypeOperands &X) {
   OS << X.operand1();
   OS << "," << X.operand2();
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const UTypeStatement &X) {
   OS << "\t" << X.opcode() << "\t";
   OS << X.operands() << "\n";
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const JTypeOpcode &X) {
   switch (X.op()) {
     case JTypeOpcode_Op_JAL: OS << "jal"; break;
+    default: assert(!"The default case was reached.");
   }
   return OS;
 }
 std::ostream &operator<<(std::ostream &OS, const JTypeOperands &X) {
   OS << X.operand1();
   OS << "," << X.operand2();
   return OS;
 }