Add an option to print out annotation remark count.

[zjaffal]

Summary

This adds a annotation-count option to llvm-remarkutil.

llvm-remarkutil annotation-count -remark=REMARK

This will print out the remark count for a pass that uses annotation remarks.

[thegameg]

llvm/tools/llvm-remarkutil/RemarkUtil.cpp

88

I think --remark is too vague. It is very specific to the type of the annotation remark, so maybe --annotation-type?

303

Why assert? The type can be anything right? If it doesn't match --remark it should just print nothing, no?

[paquette]

llvm/test/tools/llvm-remarkutil/Inputs/annotation-count.yaml

28

probably need a newline here to make all YAML parsers work (I've had problems with MIR tests missing newlines at the end)

llvm/tools/llvm-remarkutil/RemarkUtil.cpp

88

+1

290

comment is incorrect

306

I think the message on this assert is incorrect

[paquette]

llvm/tools/llvm-remarkutil/RemarkUtil.cpp

306

And yeah these probably shouldn't be asserts. IIUC this should only happen if you provide a file without annotation remarks. In which case, either print nothing, or provide an error which says "file did not contain any annotation remarks" or something.

[jroelofs]

llvm/test/tools/llvm-remarkutil/no-annotation-count.test

6

Missing colon (to make it a FileCheck directive)

llvm/tools/llvm-remarkutil/RemarkUtil.cpp

57

s/consisten/consistent/

[paquette]

llvm/tools/llvm-remarkutil/RemarkUtil.cpp

43

Make it explicit here that this is only from annotation remarks?

57

typos

[thegameg]

llvm/tools/llvm-remarkutil/RemarkUtil.cpp

306

I think the latter is not a bad idea to keep as an assert or some sort of hard error: it's technically a malformed annotation remark to not have a count field (is that true, @zjaffal?). But yes, the message is wrong

[zjaffal]

Thanks all for your comments I will update the patch accordingly

llvm/tools/llvm-remarkutil/RemarkUtil.cpp

306

All annotation remarks should have a count so I think this assert should still be there.

[paquette]

llvm/docs/CommandGuide/llvm-remarkutil.rst

23

Can you also add some documentation with a summary, like the other options have below?

[zjaffal]

Address comments.

[zjaffal]

add new lines at the end yaml and rst files

[jroelofs]

llvm/test/tools/llvm-remarkutil/no-annotation-count.test

6

still missing

[zjaffal]

llvm/test/tools/llvm-remarkutil/no-annotation-count.test

6

I will update it when committing

[thegameg]

LGTM

[aidengrossman]

It seems like when this commit relanded there was a dangling reference left in the documentation (llvm-remarkutils.rst, with annotation-count_subcommand not pointing to anything. This was breaking my sphinx builds. I've committed b4ba5c7984d29841ac92ef54a6bac44fb1eeab9b to fix the dangling reference and fix the builds. Let me know if anything needs to be adjusted. Thanks.

llvm/docs/CommandGuide/llvm-remarkutil.rst

 `remark <https://llvm.org/docs/Remarks.html>`_ formats.
 
 Subcommands
 -----------
 
   * :ref:`bitstream2yaml_subcommand` - Reserialize bitstream remarks to YAML.
   * :ref:`yaml2bitstream_subcommand` - Reserialize YAML remarks to bitstream.
   * :ref:`instruction-count_subcommand` - Output function instruction counts.
-
+  * :ref:`annotation-count_subcommand` - Output remark type count from annotation remarks.
 .. _bitstream2yaml_subcommand:
 
 bitstream2yaml
 ~~~~~~~~~~~~~~
 
 .. program:: llvm-remarkutil bitstream2yaml
 
 USAGE: :program:`llvm-remarkutil` bitstream2yaml <input file> -o <output file>

 
 Instruction count remarks require asm-printer remarks.
 
 CSV format is as follows:
 
 ::
   Function,InstructionCount
   foo,123
+
+annotation-count
+~~~~~~~~~~~~~~~~~
+
+.. program:: llvm-remarkutil annotation-count
+
+USAGE: :program:`llvm-remarkutil` annotation-count <input file> --parser=<bitstream|yaml> --annotation-type=<type> -o <output file>
+
+Summary
+^^^^^^^
+
+Outputs a count for annotation-type `<type>` remark for every function. The count expresses
+the number of remark checks inserted at the function.
+
+Annotation count remarks require AnnotationRemarksPass remarks.
+
+CSV format is as follows:
+
+::
+  Function,Count
+  foo,123

llvm/test/tools/llvm-remarkutil/Inputs/annotation-count.yaml

+--- !Analysis
+Pass:            annotation-remarks
+Name:            AnnotationSummary
+Function:        func1 
+Args:
+  - String:          'Annotated '
+  - count:           '1'
+  - String:          ' instructions with '
+  - type:            remark
+--- !Analysis
+Pass:            annotation-remarks
+Name:            AnnotationSummary
+Function:        func2 
+Args:
+  - String:          'Annotated '
+  - count:           '2'
+  - String:          ' instructions with '
+  - type:            remark
+--- !Analysis
+Pass:            annotation-remarks
+Name:            AnnotationSummary
+Function:        func3
+Args:
+  - String:          'Annotated '
+  - count:           '3'
+  - String:          ' instructions with '
+  - type:            remark

llvm/test/tools/llvm-remarkutil/annotation-count.test

+RUN: llvm-remarkutil annotation-count --parser=yaml --annotation-type=remark %p/Inputs/annotation-count.yaml | FileCheck %s
+RUN: llvm-remarkutil yaml2bitstream %p/Inputs/annotation-count.yaml | llvm-remarkutil annotation-count --parser=bitstream --annotation-type=remark | FileCheck %s
+
+; CHECK-LABEL: Function,Count
+; CHECK: func1,1
+; CHECK: func2,2
+; CHECK: func3,3

llvm/test/tools/llvm-remarkutil/broken-bitstream-remark.test

 RUN: not llvm-remarkutil bitstream2yaml %p/Inputs/broken-remark -o - 2>&1 | FileCheck %s
 RUN: not llvm-remarkutil instruction-count --parser=bitstream %p/Inputs/broken-remark -o - 2>&1 | FileCheck %s
+RUN: not llvm-remarkutil annotation-count --parser=bitstream --annotation-type=remark %p/Inputs/broken-remark -o - 2>&1 | FileCheck %s
 CHECK: error: Unknown magic number: expecting RMRK, got --- .

llvm/test/tools/llvm-remarkutil/broken-yaml-remark.test

 RUN: not llvm-remarkutil yaml2bitstream %p/Inputs/broken-remark -o - 2>&1 | FileCheck %s
 RUN: not llvm-remarkutil instruction-count --parser=yaml %p/Inputs/broken-remark -o - 2>&1 | FileCheck %s
+RUN: not llvm-remarkutil annotation-count --parser=yaml --annotation-type=remark %p/Inputs/broken-remark -o - 2>&1 | FileCheck %s
 CHECK: error: Type, Pass, Name or Function missing

llvm/test/tools/llvm-remarkutil/empty-file.test

 RUN: not llvm-remarkutil yaml2bitstream %p/Inputs/empty-file -o - 2>&1 | FileCheck %s --check-prefix=YAMLPARSER
 RUN: not llvm-remarkutil instruction-count --parser=yaml %p/Inputs/empty-file -o - 2>&1 | FileCheck %s --check-prefix=YAMLPARSER
+RUN: not llvm-remarkutil annotation-count --parser=yaml --annotation-type=remark %p/Inputs/empty-file -o - 2>&1 | FileCheck %s --check-prefix=YAMLPARSER
 RUN: llvm-remarkutil bitstream2yaml %p/Inputs/empty-file -o - 2>&1 | FileCheck %s --allow-empty --check-prefix=BITSTREAM2YAML
 RUN: llvm-remarkutil instruction-count --parser=bitstream %p/Inputs/empty-file -o - 2>&1 | FileCheck %s --allow-empty --check-prefix=SIZEBITSTREAM
+RUN: llvm-remarkutil annotation-count --parser=bitstream --annotation-type=remark %p/Inputs/empty-file -o - 2>&1 | FileCheck %s --allow-empty --check-prefix=ANNOTATIONBITSTREAM
 
 ; YAMLPARSER: error: document root is not of mapping type.
 
 ; An empty bitstream file is valid.
 ; BITSTREAM2YAML-NOT: error
 
 ; SIZEBITSTREAM-LABEL: Function,InstructionCount
-; SIZEBITSTREAM-EMPTY
+; SIZEBITSTREAM-EMPTY:
+
+; ANNOTATIONBITSTREAM-LABEL: Function,Count
+; ANNOTATIONBITSTREAM-EMPTY: 

llvm/test/tools/llvm-remarkutil/no-annotation-count.test

+RUN: llvm-remarkutil annotation-count --parser=yaml --annotation-type=remark %p/Inputs/made-up-fake-remarks.yaml | FileCheck %s
+RUN: llvm-remarkutil yaml2bitstream %p/Inputs/made-up-fake-remarks.yaml | llvm-remarkutil annotation-count --parser=bitstream --annotation-type=remark | FileCheck %s
+
+; CHECK-LABEL: Function,Count
+; CHECK-EMPTY:

llvm/tools/llvm-remarkutil/RemarkUtil.cpp

static cl::SubCommand
     YAML2Bitstream("yaml2bitstream",
                    "Convert YAML remarks to bitstream remarks");
 static cl::SubCommand
     Bitstream2YAML("bitstream2yaml",
                    "Convert bitstream remarks to YAML remarks");
 static cl::SubCommand InstructionCount(
     "instruction-count",
     "Function instruction count information (requires asm-printer remarks)");
+static cl::SubCommand
+    AnnotationCount("annotation-count",
+                    "Collect count information from annotation remarks (uses "
+                    "AnnotationRemarksPass)");
 } // namespace subopts
 
 // Keep input + output help + names consistent across the various modes via a
 // hideous macro.
 #define INPUT_OUTPUT_COMMAND_LINE_OPTIONS(SUBOPT)                              \
   static cl::opt<std::string> InputFileName(                                   \
       cl::Positional, cl::cat(RemarkUtilCategory), cl::init("-"),              \
       cl::desc("<input file>"), cl::sub(SUBOPT));                              \
   static cl::opt<std::string> OutputFileName(                                  \
       "o", cl::init("-"), cl::cat(RemarkUtilCategory), cl::desc("Output"),     \
       cl::value_desc("filename"), cl::sub(SUBOPT));
+
+// Keep Input format and names consistent accross the modes via a macro.
+#define INPUT_FORMAT_COMMAND_LINE_OPTIONS(SUBOPT)                              \
+  static cl::opt<Format> InputFormat(                                          \
+      "parser", cl::desc("Input remark format to parse"),                      \
+      cl::values(clEnumValN(Format::YAML, "yaml", "YAML"),                     \
+                 clEnumValN(Format::Bitstream, "bitstream", "Bitstream")),     \
+      cl::sub(SUBOPT));
 namespace yaml2bitstream {
 /// Remark format to parse.
 static constexpr Format InputFormat = Format::YAML;
 /// Remark format to output.
 static constexpr Format OutputFormat = Format::Bitstream;
 INPUT_OUTPUT_COMMAND_LINE_OPTIONS(subopts::YAML2Bitstream)
 } // namespace yaml2bitstream
 
 namespace bitstream2yaml {
 /// Remark format to parse.
 static constexpr Format InputFormat = Format::Bitstream;
 /// Remark format to output.
 static constexpr Format OutputFormat = Format::YAML;
 INPUT_OUTPUT_COMMAND_LINE_OPTIONS(subopts::Bitstream2YAML)
 } // namespace bitstream2yaml
 
 namespace instructioncount {
-static cl::opt<Format> InputFormat(
-    "parser", cl::desc("Input remark format to parse"),
-    cl::values(clEnumValN(Format::YAML, "yaml", "YAML"),
-               clEnumValN(Format::Bitstream, "bitstream", "Bitstream")),
-    cl::sub(subopts::InstructionCount));
+INPUT_FORMAT_COMMAND_LINE_OPTIONS(subopts::InstructionCount)
 INPUT_OUTPUT_COMMAND_LINE_OPTIONS(subopts::InstructionCount)
 } // namespace instructioncount
 
+namespace annotationcount {
+INPUT_FORMAT_COMMAND_LINE_OPTIONS(subopts::AnnotationCount)
+static cl::opt<std::string> AnnotationTypeToCollect(
+    "annotation-type", cl::desc("annotation-type remark to collect count for"),
+    cl::sub(subopts::AnnotationCount));
+INPUT_OUTPUT_COMMAND_LINE_OPTIONS(subopts::AnnotationCount)
+} // namespace annotationcount
+
 /// \returns A MemoryBuffer for the input file on success, and an Error
 /// otherwise.
 static Expected<std::unique_ptr<MemoryBuffer>>
 getInputMemoryBuffer(StringRef InputFileName) {
   auto MaybeBuf = MemoryBuffer::getFileOrSTDIN(InputFileName);
   if (auto ErrorCode = MaybeBuf.getError())
     return createStringError(ErrorCode,
                              Twine("Cannot open file '" + InputFileName +
static Error tryInstructionCount() {
   if (!E.isA<EndOfFileError>())
     return E;
   consumeError(std::move(E));
   OF->keep();
   return Error::success();
 }
 } // namespace instructioncount
 
+namespace annotationcount {
+static Error tryAnnotationCount() {
+  // Create the output buffer.
+  auto MaybeOF = getOutputFileWithFlags(OutputFileName,
+                                        /*Flags = */ sys::fs::OF_TextWithCRLF);
+  if (!MaybeOF)
+    return MaybeOF.takeError();
+  auto OF = std::move(*MaybeOF);
+  // Create a parser for the user-specified input format.
+  auto MaybeBuf = getInputMemoryBuffer(InputFileName);
+  if (!MaybeBuf)
+    return MaybeBuf.takeError();
+  auto MaybeParser = createRemarkParser(InputFormat, (*MaybeBuf)->getBuffer());
+  if (!MaybeParser)
+    return MaybeParser.takeError();
+  // Emit CSV header.
+  OF->os() << "Function,Count\n";
+  // Parse all remarks. When we see the specified remark collect the count
+  // information.
+  auto &Parser = **MaybeParser;
+  auto MaybeRemark = Parser.next();
+  for (; MaybeRemark; MaybeRemark = Parser.next()) {
+    auto &Remark = **MaybeRemark;
+    if (Remark.RemarkName != "AnnotationSummary")
+      continue;
+    auto *RemarkNameArg = find_if(Remark.Args, [](const Argument &Arg) {
+      return Arg.Key == "type" && Arg.Val == AnnotationTypeToCollect;
+    });
+    if (RemarkNameArg == Remark.Args.end())
+      continue;
+    auto *CountArg = find_if(
+        Remark.Args, [](const Argument &Arg) { return Arg.Key == "count"; });
+    assert(CountArg != Remark.Args.end() &&
+           "Expected annotation-type remark to have a count key?");
+    OF->os() << Remark.FunctionName << "," << CountArg->Val << "\n";
+  }
+  auto E = MaybeRemark.takeError();
+  if (!E.isA<EndOfFileError>())
+    return E;
+  consumeError(std::move(E));
+  OF->keep();
+  return Error::success();
+}
+
+} // namespace annotationcount
 /// Handle user-specified suboptions (e.g. yaml2bitstream, bitstream2yaml).
 /// \returns An Error if the specified suboption fails or if no suboption was
 /// specified. Otherwise, Error::success().
 static Error handleSuboptions() {
   if (subopts::Bitstream2YAML)
     return bitstream2yaml::tryBitstream2YAML();
   if (subopts::YAML2Bitstream)
     return yaml2bitstream::tryYAML2Bitstream();
   if (subopts::InstructionCount)
     return instructioncount::tryInstructionCount();
+  if (subopts::AnnotationCount)
+    return annotationcount::tryAnnotationCount();
+
   return make_error<StringError>(
       "Please specify a subcommand. (See -help for options)",
       inconvertibleErrorCode());
 }
 
 int main(int argc, const char **argv) {
   InitLLVM X(argc, argv);
   cl::HideUnrelatedOptions(RemarkUtilCategory);
   cl::ParseCommandLineOptions(argc, argv, "Remark file utilities\n");
   ExitOnErr.setBanner(std::string(argv[0]) + ": error: ");
   ExitOnErr(handleSuboptions());
 }