[llvm-remarkutil] Add an option to display DebugLoc when collecting counts for remarks.

[thegameg]

Why not use the YAML output directly in this case?

[zjaffal]

In D148374#4269616, @thegameg wrote:

Why not use the YAML output directly in this case?

If you want to generate the count for functions there is the possibility that the function name is not unique so you will end up with different counts for the same function. While the DebugLoc is not unique but it is better than just using the function name as an identifier.

[paquette]

llvm/test/tools/llvm-remarkutil/annotation-count-with-dbg-loc.test

8

add newline

llvm/tools/llvm-remarkutil/RemarkUtil.cpp

265

We can reduce duplication in the output string here:

if (UseDebugLoc)
  OF->os() << "Source,"
OF->os() << "Function,InstructionCount\n";

287

Similar to the case above, we can just emit the debug location here, and then eliminate the else.

// Emit debug location if the user wants it.
if (UseDebugLoc) {
  std::string Loc = ...;
  OF->os() << Loc << ",";
}
// Always emit function name and instruction count.
OF->os() << Remark.FunctionName << ...;

316

Same here

325

I think we can factor this out into a "shouldSkipRemark" and share it between the instruction count + annotation remark modes.

342

Same as other feedback for string output

[zjaffal]

refactor code to reduce duplication

[paquette]

LGTM

llvm/docs/CommandGuide/llvm-remarkutil.rst

 
 .. _instruction-count_subcommand:
 
 instruction-count
 ~~~~~~~~~~~~~~~~~
 
 .. program:: llvm-remarkutil instruction-count
 
-USAGE: :program:`llvm-remarkutil` instruction-count <input file> --parser=<bitstream|yaml> -o <output file>
+USAGE: :program:`llvm-remarkutil` instruction-count <input file> --parser=<bitstream|yaml> [--use-debug-loc] -o <output file>
 
 Summary
 ^^^^^^^
 
 Outputs instruction count remarks for every function. Instruction count remarks
 encode the number of instructions in a function at assembly printing time.
 
 Instruction count remarks require asm-printer remarks.
 
 CSV format is as follows:
 
 ::
   Function,InstructionCount
   foo,123
 
+if `--use-debug-loc` is passed then the CSV will include the source path, line number and column. 
+
+::
+  Source,Function,InstructionCount
+  path:line:column,foo,3
+
 .. _annotation-count_subcommand:
 
 annotation-count
 ~~~~~~~~~~~~~~~~~
 
 .. program:: llvm-remarkutil annotation-count
 
-USAGE: :program:`llvm-remarkutil` annotation-count <input file> --parser=<bitstream|yaml> --annotation-type=<type> -o <output file>
+USAGE: :program:`llvm-remarkutil` annotation-count <input file> --parser=<bitstream|yaml> --annotation-type=<type>  [--use-debug-loc] -o <output file>
 
 Summary
 ^^^^^^^
 
 Outputs a count for annotation-type `<type>` remark for every function. The count expresses
 the number of remark checks inserted at the function.
 
 Annotation count remarks require AnnotationRemarksPass remarks.
 
 CSV format is as follows:
 
 ::
   Function,Count
   foo,123
+
+if `--use-debug-loc` is passed then the CSV will include the source path, line number and column. 
+
+::
+  Source,Function,Count
+  path:line:column,foo,3

llvm/test/tools/llvm-remarkutil/Inputs/annotation-count-with-dbg-loc.yaml

+--- !Analysis
+Pass:            annotation-remarks
+Name:            AnnotationSummary
+DebugLoc:        { File: path/to/anno.c, Line: 1, Column: 2 }
+Function:        func1
+Args:
+  - String:          'Annotated '
+  - count:           '1'
+  - String:          ' instructions with '
+  - type:            remark
+--- !Analysis
+Pass:            annotation-remarks
+Name:            AnnotationSummary
+DebugLoc:        { File: path/to/anno2.c, Line: 1, Column: 2 }
+Function:        func2 
+Args:
+  - String:          'Annotated '
+  - count:           '2'
+  - String:          ' instructions with '
+  - type:            remark
+--- !Analysis
+Pass:            annotation-remarks
+Name:            AnnotationSummary
+DebugLoc:        { File: path/to/anno3.c, Line: 1, Column: 2 }
+Function:        func3
+Args:
+  - String:          'Annotated '
+  - count:           '3'
+  - String:          ' instructions with '
+  - type:            remark

llvm/test/tools/llvm-remarkutil/Inputs/instruction-count-with-dbg-loc.yaml

+--- !Analysis
+Pass:            asm-printer
+Name:            InstructionCount
+DebugLoc:        { File: path/to/inst.c, Line: 1, Column: 2 }
+Function:        func1
+Args:
+  - NumInstructions: '1'
+  - String:          ' instructions in function'
+...
+--- !Analysis
+Pass:            asm-printer
+Name:            InstructionCount
+DebugLoc:        { File: path/to/inst2.c, Line: 1, Column: 2 }
+Function:        func2
+Args:
+  - NumInstructions: '2'
+  - String:          ' instructions in function'
+...
+--- !Analysis
+Pass:            asm-printer
+Name:            InstructionCount
+DebugLoc:        { File: path/to/inst3.c, Line: 1, Column: 2 }
+Function:        func3
+Args:
+  - NumInstructions: '3'
+  - String:          ' instructions in function'
+...

llvm/test/tools/llvm-remarkutil/annotation-count-with-dbg-loc.test

+RUN: llvm-remarkutil annotation-count --use-debug-loc --parser=yaml --annotation-type=remark %p/Inputs/annotation-count-with-dbg-loc.yaml | FileCheck %s
+RUN: llvm-remarkutil yaml2bitstream %p/Inputs/annotation-count-with-dbg-loc.yaml | llvm-remarkutil annotation-count --use-debug-loc --parser=bitstream --annotation-type=remark | FileCheck %s
+
+; CHECK-LABEL: Source,Function,Count
+; CHECK: path/to/anno.c:1:2,func1,1
+; CHECK: path/to/anno2.c:1:2,func2,2
+; CHECK: path/to/anno3.c:1:2,func3,3

llvm/test/tools/llvm-remarkutil/instruction-count-with-dbg-loc.test

+RUN: llvm-remarkutil instruction-count --use-debug-loc --parser=yaml %p/Inputs/instruction-count-with-dbg-loc.yaml | FileCheck %s
+RUN: llvm-remarkutil yaml2bitstream %p/Inputs/instruction-count-with-dbg-loc.yaml | llvm-remarkutil instruction-count --use-debug-loc --parser=bitstream | FileCheck %s
+
+; CHECK-LABEL: Source,Function,InstructionCount
+; CHECK: path/to/inst.c:1:2,func1,1
+; CHECK: path/to/inst2.c:1:2,func2,2
+; CHECK: path/to/inst3.c:1:2,func3,3

llvm/test/tools/llvm-remarkutil/no-debug-log.test

+RUN: llvm-remarkutil instruction-count --use-debug-loc --parser=yaml %p/Inputs/instruction-count.yaml | FileCheck --check-prefix=INSTRUCTIONCOUNT %s
+RUN: llvm-remarkutil yaml2bitstream %p/Inputs/instruction-count.yaml | llvm-remarkutil instruction-count --use-debug-loc --parser=bitstream | FileCheck %s --check-prefix=INSTRUCTIONCOUNT
+RUN: llvm-remarkutil annotation-count --use-debug-loc --parser=yaml --annotation-type=remark %p/Inputs/annotation-count.yaml | FileCheck --check-prefix=ANNOTATIONCOUNT %s
+RUN: llvm-remarkutil yaml2bitstream %p/Inputs/instruction-count.yaml | llvm-remarkutil annotation-count --use-debug-loc --parser=bitstream --annotation-type=remark | FileCheck %s --check-prefix=ANNOTATIONCOUNT
+
+; ANNOTATIONCOUNT-LABEL: Source,Function,Count
+; INSTRUCTIONCOUNT-LABEL: Source,Function,InstructionCount

llvm/tools/llvm-remarkutil/RemarkUtil.cpp

 
 // Keep Input format and names consistent accross the modes via a macro.
 #define INPUT_FORMAT_COMMAND_LINE_OPTIONS(SUBOPT)                              \
   static cl::opt<Format> InputFormat(                                          \
       "parser", cl::desc("Input remark format to parse"),                      \
       cl::values(clEnumValN(Format::YAML, "yaml", "YAML"),                     \
                  clEnumValN(Format::Bitstream, "bitstream", "Bitstream")),     \
       cl::sub(SUBOPT));
+
+#define DEBUG_LOC_INFO_COMMAND_LINE_OPTIONS(SUBOPT)                            \
+  static cl::opt<bool> UseDebugLoc(                                            \
+      "use-debug-loc",                                                         \
+      cl::desc(                                                                \
+          "Add debug loc information when generating tables for "              \
+          "functions. The loc is represented as (path:line number:column "     \
+          "number)"),                                                          \
+      cl::init(false), cl::sub(SUBOPT));
 namespace yaml2bitstream {
 /// Remark format to parse.
 static constexpr Format InputFormat = Format::YAML;
 /// Remark format to output.
 static constexpr Format OutputFormat = Format::Bitstream;
 INPUT_OUTPUT_COMMAND_LINE_OPTIONS(subopts::YAML2Bitstream)
 } // namespace yaml2bitstream
 
 namespace bitstream2yaml {
 /// Remark format to parse.
 static constexpr Format InputFormat = Format::Bitstream;
 /// Remark format to output.
 static constexpr Format OutputFormat = Format::YAML;
 INPUT_OUTPUT_COMMAND_LINE_OPTIONS(subopts::Bitstream2YAML)
 } // namespace bitstream2yaml
 
 namespace instructioncount {
 INPUT_FORMAT_COMMAND_LINE_OPTIONS(subopts::InstructionCount)
 INPUT_OUTPUT_COMMAND_LINE_OPTIONS(subopts::InstructionCount)
+DEBUG_LOC_INFO_COMMAND_LINE_OPTIONS(subopts::InstructionCount)
 } // namespace instructioncount
 
 namespace annotationcount {
 INPUT_FORMAT_COMMAND_LINE_OPTIONS(subopts::AnnotationCount)
 static cl::opt<std::string> AnnotationTypeToCollect(
     "annotation-type", cl::desc("annotation-type remark to collect count for"),
     cl::sub(subopts::AnnotationCount));
 INPUT_OUTPUT_COMMAND_LINE_OPTIONS(subopts::AnnotationCount)
+DEBUG_LOC_INFO_COMMAND_LINE_OPTIONS(subopts::AnnotationCount)
 } // namespace annotationcount
 
 /// \returns A MemoryBuffer for the input file on success, and an Error
 /// otherwise.
 static Expected<std::unique_ptr<MemoryBuffer>>
 getInputMemoryBuffer(StringRef InputFileName) {
   auto MaybeBuf = MemoryBuffer::getFileOrSTDIN(InputFileName);
   if (auto ErrorCode = MaybeBuf.getError())

 getOutputFileForRemarks(StringRef OutputFileName, Format OutputFormat) {
   assert((OutputFormat == Format::YAML || OutputFormat == Format::Bitstream) &&
          "Expected one of YAML or Bitstream!");
   return getOutputFileWithFlags(OutputFileName, OutputFormat == Format::YAML
                                                     ? sys::fs::OF_TextWithCRLF
                                                     : sys::fs::OF_None);
 }
 
+static bool shouldSkipRemark(bool UseDebugLoc, Remark &Remark) {
+  return UseDebugLoc && !Remark.Loc.has_value();
+}
+
 namespace yaml2bitstream {
 /// Parses all remarks in the input YAML file.
 /// \p [out] ParsedRemarks - Filled with remarks parsed from the input file.
 /// \p [out] StrTab - A string table populated for later remark serialization.
 /// \returns Error::success() if all remarks were successfully parsed, and an
 /// Error otherwise.
 static Error
 tryParseRemarksFromYAMLFile(std::vector<std::unique_ptr<Remark>> &ParsedRemarks,
static Error tryInstructionCount() {
   // Create a parser for the user-specified input format.
   auto MaybeBuf = getInputMemoryBuffer(InputFileName);
   if (!MaybeBuf)
     return MaybeBuf.takeError();
   auto MaybeParser = createRemarkParser(InputFormat, (*MaybeBuf)->getBuffer());
   if (!MaybeParser)
     return MaybeParser.takeError();
   // Emit CSV header.
+  if (UseDebugLoc)
+    OF->os() << "Source,";
   OF->os() << "Function,InstructionCount\n";
   // Parse all remarks. Whenever we see an instruction count remark, output
   // the file name and the number of instructions.
   auto &Parser = **MaybeParser;
   auto MaybeRemark = Parser.next();
   for (; MaybeRemark; MaybeRemark = Parser.next()) {
     auto &Remark = **MaybeRemark;
     if (Remark.RemarkName != "InstructionCount")
       continue;
+    if (shouldSkipRemark(UseDebugLoc, Remark))
+      continue;
     auto *InstrCountArg = find_if(Remark.Args, [](const Argument &Arg) {
       return Arg.Key == "NumInstructions";
     });
     assert(InstrCountArg != Remark.Args.end() &&
            "Expected instruction count remarks to have a NumInstructions key?");
+    if (UseDebugLoc) {
+      std::string Loc = Remark.Loc->SourceFilePath.str() + ":" +
+                        std::to_string(Remark.Loc->SourceLine) + +":" +
+                        std::to_string(Remark.Loc->SourceColumn);
+      OF->os() << Loc << ",";
+    }
     OF->os() << Remark.FunctionName << "," << InstrCountArg->Val << "\n";
   }
   auto E = MaybeRemark.takeError();
   if (!E.isA<EndOfFileError>())
     return E;
   consumeError(std::move(E));
   OF->keep();
   return Error::success();
 }
static Error tryAnnotationCount() {
   // Create a parser for the user-specified input format.
   auto MaybeBuf = getInputMemoryBuffer(InputFileName);
   if (!MaybeBuf)
     return MaybeBuf.takeError();
   auto MaybeParser = createRemarkParser(InputFormat, (*MaybeBuf)->getBuffer());
   if (!MaybeParser)
     return MaybeParser.takeError();
   // Emit CSV header.
+  if (UseDebugLoc)
+    OF->os() << "Source,";
   OF->os() << "Function,Count\n";
   // Parse all remarks. When we see the specified remark collect the count
   // information.
   auto &Parser = **MaybeParser;
   auto MaybeRemark = Parser.next();
   for (; MaybeRemark; MaybeRemark = Parser.next()) {
     auto &Remark = **MaybeRemark;
     if (Remark.RemarkName != "AnnotationSummary")
       continue;
+    if (shouldSkipRemark(UseDebugLoc, Remark))
+      continue;
     auto *RemarkNameArg = find_if(Remark.Args, [](const Argument &Arg) {
       return Arg.Key == "type" && Arg.Val == AnnotationTypeToCollect;
     });
     if (RemarkNameArg == Remark.Args.end())
       continue;
     auto *CountArg = find_if(
         Remark.Args, [](const Argument &Arg) { return Arg.Key == "count"; });
     assert(CountArg != Remark.Args.end() &&
            "Expected annotation-type remark to have a count key?");
+    if (UseDebugLoc) {
+      std::string Loc = Remark.Loc->SourceFilePath.str() + ":" +
+                        std::to_string(Remark.Loc->SourceLine) + +":" +
+                        std::to_string(Remark.Loc->SourceColumn);
+      OF->os() << Loc << ",";
+    }
     OF->os() << Remark.FunctionName << "," << CountArg->Val << "\n";
   }
   auto E = MaybeRemark.takeError();
   if (!E.isA<EndOfFileError>())
     return E;
   consumeError(std::move(E));
   OF->keep();
   return Error::success();
 }