[MLIR][SPIRV] Extend automation script to generate coverage report.

[ergawy]

Summary

This patch adds a new cli argument to the automation script to generate
a report of the current SPIRV spec instruction coverage. It dumps to the
standard output a YAML string with the coverage information.

[ergawy]

This is a trivial extension to the automation script to make it slightly easier to know the current state of affairs. Please feel free to reject it if you find it adds too much code without as much value :)

[antiagainst]

This is pretty cool! Thanks @ergawy! Just two comments, could you wrap to 80 characters and add a report_coverage.sh wrapper to feed in some directories like other .sh wrappers?

[ergawy]

Create wrapper script and wrap lines at 80 chars.

[antiagainst]

It seems you accidentally deleted the first commit. You can choose the base SHA to diff against when using arc diff <base-sha-to-diff-against>. the patch is against a non-existing commit; I see errors when trying to apply it to the top of the tree. Could you rebase against the top of the tree so I can land it? Thanks.

[ergawy]

Rebase on top of master.

[ergawy]

arc diff against master.

[ergawy]

In D82006#2106555, @antiagainst wrote:

It seems you accidentally deleted the first commit. You can choose the base SHA to diff against when using arc diff <base-sha-to-diff-against>. the patch is against a non-existing commit; I see errors when trying to apply it to the top of the tree. Could you rebase against the top of the tree so I can land it? Thanks.

Thank you for the explanation, I was wondering what the problem is since I am not really familiar with the Differential/Arcanist workflow. Hopefully, the problem is solved now.

[ergawy]

Squash commits.

[ergawy]

Sorry for the mess. It seems that CI fails only for Linux for some reason. Will try to find my around. Any tips are much appreciated.

[ergawy]

Rebase on top of master again.

[ergawy]

@antiagainst Sorry for the noise, but I don't think there anything I can do regarding the failures happening on CI here. Can you please have a look and tell if I am mistaken?

[antiagainst]

In D82006#2109200, @ergawy wrote:

@antiagainst Sorry for the noise, but I don't think there anything I can do regarding the failures happening on CI here. Can you please have a look and tell if I am mistaken?

No worries! The script is just for offline use; it's not ever integrated into build/test process. I don't think it can cause buildbot failures.

mlir/utils/spirv/gen_spirv_dialect.py

 #
 # The 'operand_kinds' dict of spirv.core.grammar.json contains all supported
 # SPIR-V enum classes.
 
 import itertools
 import re
 import requests
 import textwrap
+import yaml
 
 SPIRV_HTML_SPEC_URL = 'https://www.khronos.org/registry/spir-v/specs/unified1/SPIRV.html'
 SPIRV_JSON_SPEC_URL = 'https://raw.githubusercontent.com/KhronosGroup/SPIRV-Headers/master/include/spirv/unified1/spirv.core.grammar.json'
 
 AUTOGEN_OP_DEF_SEPARATOR = '\n// -----\n\n'
 AUTOGEN_ENUM_SECTION_MARKER = 'enum section. Generated from SPIR-V spec; DO NOT MODIFY!'
 AUTOGEN_OPCODE_SECTION_MARKER = (
     'opcode section. Generated from SPIR-V spec; DO NOT MODIFY!')
def get_spirv_grammar_from_json_spec():
   spec = response.content
 
   import json
   spirv = json.loads(spec)
 
   return spirv['operand_kinds'], spirv['instructions']
 
 
-def split_list_into_sublists(items, offset):
+def split_list_into_sublists(items):
   """Split the list of items into multiple sublists.
 
   This is to make sure the string composed from each sublist won't exceed
   80 characters.
 
   Arguments:
     - items: a list of strings
-    - offset: the offset in calculating each sublist's length
   """
   chuncks = []
   chunk = []
   chunk_len = 0
 
   for item in items:
     chunk_len += len(item) + 2
     if chunk_len > 80:
def gen_operand_kind_enum_attr(operand_kind, capability_mapping):
   case_defs = '\n'.join(case_defs)
 
   # Generate the list of enum case names
   fmt_str = 'SPV_{acronym}_{symbol}';
   case_names = [fmt_str.format(acronym=kind_acronym,symbol=case[0])
                 for case in kind_cases]
 
   # Split them into sublists and concatenate into multiple lines
-  case_names = split_list_into_sublists(case_names, 6)
+  case_names = split_list_into_sublists(case_names)
   case_names = ['{:6}'.format('') + ', '.join(sublist)
                 for sublist in case_names]
   case_names = ',\n'.join(case_names)
 
   # Generate the enum attribute definition
   enum_attr = '''def SPV_{name}Attr :
     SPV_{category}EnumAttr<"{name}", "valid SPIR-V {name}", [
 {cases}
opcode_defs = [
           offset=(max_len + 1 - len(inst['opname']))) for inst in instructions
   ]
   opcode_str = '\n'.join(opcode_defs)
 
   decl_fmt_str = 'SPV_OC_{name}'
   opcode_list = [
       decl_fmt_str.format(name=inst['opname']) for inst in instructions
   ]
-  opcode_list = split_list_into_sublists(opcode_list, 6)
+  opcode_list = split_list_into_sublists(opcode_list)
   opcode_list = [
       '{:6}'.format('') + ', '.join(sublist) for sublist in opcode_list
   ]
   opcode_list = ',\n'.join(opcode_list)
   enum_attr = 'def SPV_OpcodeAttr :\n'\
               '    SPV_I32EnumAttr<"{name}", "valid SPIR-V instructions", [\n'\
               '{lst}\n'\
               '    ]>;'.format(name='Opcode', lst=opcode_list)
   return opcode_str + '\n\n' + enum_attr
 
+def map_cap_to_opnames(instructions):
+  """Maps capabilities to instructions enabled by those capabilities
+
+  Arguments:
+    - instructions: a list containing a subset of SPIR-V instructions' grammar
+  Returns:
+    - A map with keys representing capabilities and values of lists of
+    instructions enabled by the corresponding key
+  """
+  cap_to_inst = {}
+
+  for inst in instructions:
+    caps = inst['capabilities'] if 'capabilities' in inst else ['0_core_0']
+    for cap in caps:
+      if cap not in cap_to_inst:
+        cap_to_inst[cap] = []
+      cap_to_inst[cap].append(inst['opname'])
+
+  return cap_to_inst
+
+def gen_instr_coverage_report(path, instructions):
+  """Dumps to standard output a YAML report of current instruction coverage
+
+  Arguments:
+    - path: the path to SPIRBase.td
+    - instructions: a list containing all SPIR-V instructions' grammar
+  """
+  with open(path, 'r') as f:
+    content = f.read()
+
+  content = content.split(AUTOGEN_OPCODE_SECTION_MARKER)
+
+  existing_opcodes = [k[11:] for k in re.findall('def SPV_OC_\w+', content[1])]
+  existing_instructions = list(
+          filter(lambda inst: (inst['opname'] in existing_opcodes),
+              instructions))
+
+  instructions_opnames = [inst['opname'] for inst in instructions]
+
+  remaining_opcodes = list(set(instructions_opnames) - set(existing_opcodes))
+  remaining_instructions = list(
+          filter(lambda inst: (inst['opname'] in remaining_opcodes),
+              instructions))
+
+  rem_cap_to_instr = map_cap_to_opnames(remaining_instructions)
+  ex_cap_to_instr = map_cap_to_opnames(existing_instructions)
+
+  rem_cap_to_cov = {}
+
+  # Calculate coverage for each capability
+  for cap in rem_cap_to_instr:
+    if cap not in ex_cap_to_instr:
+      rem_cap_to_cov[cap] = 0.0
+    else:
+      rem_cap_to_cov[cap] = \
+              (len(ex_cap_to_instr[cap]) / (len(ex_cap_to_instr[cap]) \
+              + len(rem_cap_to_instr[cap])))
+
+  report = {}
+
+  # Merge the 3 maps into one report
+  for cap in rem_cap_to_instr:
+    report[cap] = {}
+    report[cap]['Supported Instructions'] = \
+            ex_cap_to_instr[cap] if cap in ex_cap_to_instr else []
+    report[cap]['Unsupported Instructions']  = rem_cap_to_instr[cap]
+    report[cap]['Coverage'] = '{}%'.format(int(rem_cap_to_cov[cap] * 100))
+
+  print(yaml.dump(report))
 
 def update_td_opcodes(path, instructions, filter_list):
   """Updates SPIRBase.td with new generated opcode cases.
 
   Arguments:
     - path: the path to SPIRBase.td
     - instructions: a list containing all SPIR-V instructions' grammar
     - filter_list: a list containing new opnames to add
if len(split) == 2:
     return rest[:index], rest[index + len(end):]
   return '', split[0]
 
 
 def extract_td_op_info(op_def):
   """Extracts potentially manually specified sections in op's definition.
 
   Arguments: - A string containing the op's TableGen definition
-    - doc: the instruction's SPIR-V HTML doc
 
   Returns:
     - A dict containing potential manually specified sections
   """
   # Get opname
   opname = [o[8:-2] for o in re.findall('def SPV_\w+Op', op_def)]
   assert len(opname) == 1, 'more than one ops in the same section!'
   opname = opname[0]
cli_parser.add_argument(
       help='SPIR-V instruction to be added to ops file')
   cli_parser.add_argument(
       '--inst-category',
       dest='inst_category',
       type=str,
       default='Op',
       help='SPIR-V instruction category used for choosing '\
            'the TableGen base class to define this op')
+  cli_parser.add_argument('--gen-inst-coverage', dest='gen_inst_coverage', action='store_true')
+  cli_parser.set_defaults(gen_inst_coverage=False)
 
   args = cli_parser.parse_args()
 
   operand_kinds, instructions = get_spirv_grammar_from_json_spec()
 
   # Define new enum attr
   if args.new_enum is not None:
     assert args.base_td_path is not None
if args.new_inst is not None:
     assert args.op_td_path is not None
     docs = get_spirv_doc_from_html_spec()
     capability_mapping = get_capability_mapping(operand_kinds)
     update_td_op_definitions(args.op_td_path, instructions, docs, args.new_inst,
                              args.inst_category, capability_mapping)
     print('Done. Note that this script just generates a template; ', end='')
     print('please read the spec and update traits, arguments, and ', end='')
     print('results accordingly.')
+
+  if args.gen_inst_coverage:
+    gen_instr_coverage_report(args.base_td_path, instructions)

mlir/utils/spirv/report_coverage.sh

+#!/bin/bash
+
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# Script for reporting current level of SPIR-V spec instruction coverage in spv
+# Dialect. It dumps to standard output a YAML string of current coverage.
+#
+# Run as:
+# ./report_coverage.sh
+
+set -e
+
+current_file="$(readlink -f "$0")"
+current_dir="$(dirname "$current_file")"
+
+python3 ${current_dir}/gen_spirv_dialect.py \
+  --base-td-path ${current_dir}/../../include/mlir/Dialect/SPIRV/SPIRVBase.td \
+  --gen-inst-coverage