[FileCheck] Annotate input dump (2/7)

[jdenny]

Summary

This patch implements input annotations for diagnostics that suggest
fuzzy matches for directives for which no matches were found. Instead
of using the usual ^~~, which is used by later patches for good
matches, these annotations use ? so that fuzzy matches are visually
distinct. No tildes are included as these diagnostics (independently
of this patch) currently identify only the start of the match.

For example:

$ FileCheck -dump-input=help
The following description was requested by -dump-input=help to
explain the input annotations printed by -dump-input=always and
-dump-input=fail:

  - L:     labels line number L of the input file
  - T:L    labels the only match result for a pattern of type T from line L of
           the check file
  - T:L'N  labels the Nth match result for a pattern of type T from line L of
           the check file
  - X~~    marks search range when no match is found
  - ?      marks fuzzy match when no match is found
  - colors error, fuzzy match

If you are not seeing color above or in input dumps, try: -color

$ FileCheck -v -dump-input=always check1 < input1 |& sed -n '/^<<<</,$p'
<<<<<<
          1: ; abc def
          2: ; ghI jkl
next:3'0     X~~~~~~~~ error: no match found
next:3'1       ?       possible intended match
>>>>>>

$ cat check1
CHECK: abc
CHECK-SAME: def
CHECK-NEXT: ghi
CHECK-SAME: jkl

$ cat input1
; abc def
; ghI jkl

This patch introduces the concept of multiple "match results" per
directive. In the above example, the first match result for the
CHECK-NEXT directive is the failed match, for which the annotation
shows the search range. The second match result is the fuzzy match.
Later patches will introduce other cases of multiple match results per
directive.

When colors are enabled, ? is colored magenta. That is, it doesn't
indicate the actual error, which a red X~~ marker indicates, but its
color suggests it's closely related.

[jdenny]

Rebased onto updated patches earlier in series.

[jdenny]

Update for changes earlier in patch series.

[jdenny]

Propagate changes started earlier in patch series.

[jdenny]

Continue changes started earlier in patch series.

[probinson]

One small comment fix needed, otherwise LGTM.

llvm/include/llvm/Support/FileCheck.h

169 ↗

(On Diff #177381)

Comments should be proper sentences.

[jdenny]

Adjust comment, as suggested by probinson.

llvm/trunk/include/llvm/Support/FileCheck.h

public:
   bool ParsePattern(StringRef PatternStr, StringRef Prefix, SourceMgr &SM,
                     unsigned LineNumber, const FileCheckRequest &Req);
   size_t Match(StringRef Buffer, size_t &MatchLen,
                StringMap<StringRef> &VariableTable) const;
   void PrintVariableUses(const SourceMgr &SM, StringRef Buffer,
                          const StringMap<StringRef> &VariableTable,
                          SMRange MatchRange = None) const;
   void PrintFuzzyMatch(const SourceMgr &SM, StringRef Buffer,
-                       const StringMap<StringRef> &VariableTable) const;
+                       const StringMap<StringRef> &VariableTable,
+                       std::vector<FileCheckDiag> *Diags) const;
 
   bool hasVariable() const {
     return !(VariableUses.empty() && VariableDefs.empty());
   }
 
   Check::FileCheckType getCheckTy() const { return CheckTy; }
 
   int getCount() const { return CheckTy.getCount(); }

 //===----------------------------------------------------------------------===//
 
 struct FileCheckDiag {
   /// What is the FileCheck directive for this diagnostic?
   Check::FileCheckType CheckTy;
   /// Where is the FileCheck directive for this diagnostic?
   unsigned CheckLine, CheckCol;
   /// What kind of match result does this diagnostic describe?
+  ///
+  /// There might be more than one of these for the same directive.  For
+  /// example, there might be a fuzzy match after a fail.
   enum MatchType {
     // TODO: More members will appear with later patches in this series.
     /// Indicates no match for an expected pattern.
     MatchNoneButExpected,
+    /// Indicates a possible intended match because there's no perfect match.
+    MatchFuzzy,
     MatchTypeCount,
   } MatchTy;
-  /// The search range.
+  /// The match range if MatchTy is not MatchNoneButExpected, or the search
+  /// range otherwise.
   unsigned InputStartLine, InputStartCol, InputEndLine, InputEndCol;
   FileCheckDiag(const SourceMgr &SM, const Check::FileCheckType &CheckTy,
                 SMLoc CheckLoc, MatchType MatchTy, SMRange InputRange);
 };
 
 //===----------------------------------------------------------------------===//
 // Check Strings.
 //===----------------------------------------------------------------------===//

llvm/trunk/lib/Support/FileCheck.cpp

static SMRange ProcessMatchResult(FileCheckDiag::MatchType MatchTy,
   // more match types.
   if (Diags && MatchTy != FileCheckDiag::MatchTypeCount)
     Diags->emplace_back(SM, CheckTy, Loc, MatchTy, Range);
   return Range;
 }
 
 void FileCheckPattern::PrintFuzzyMatch(
     const SourceMgr &SM, StringRef Buffer,
-    const StringMap<StringRef> &VariableTable) const {
+    const StringMap<StringRef> &VariableTable,
+    std::vector<FileCheckDiag> *Diags) const {
   // Attempt to find the closest/best fuzzy match.  Usually an error happens
   // because some string in the output didn't exactly match. In these cases, we
   // would like to show the user a best guess at what "should have" matched, to
   // save them having to actually check the input manually.
   size_t NumLinesForward = 0;
   size_t Best = StringRef::npos;
   double BestQuality = 0;
 
if (Quality < BestQuality || Best == StringRef::npos) {
       BestQuality = Quality;
     }
   }
 
   // Print the "possible intended match here" line if we found something
   // reasonable and not equal to what we showed in the "scanning from here"
   // line.
   if (Best && Best != StringRef::npos && BestQuality < 50) {
-    SM.PrintMessage(SMLoc::getFromPointer(Buffer.data() + Best),
-                    SourceMgr::DK_Note, "possible intended match here");
+    SMRange MatchRange =
+        ProcessMatchResult(FileCheckDiag::MatchFuzzy, SM, getLoc(),
+                           getCheckTy(), Buffer, Best, 0, Diags);
+    SM.PrintMessage(MatchRange.Start, SourceMgr::DK_Note,
+                    "possible intended match here");
 
     // FIXME: If we wanted to be really friendly we would show why the match
     // failed, as it can be hard to spot simple one character differences.
   }
 }
 
 /// Finds the closing sequence of a regex variable usage or definition.
 ///
SMRange SearchRange = ProcessMatchResult(
       ExpectedMatch ? FileCheckDiag::MatchNoneButExpected
                     : FileCheckDiag::MatchTypeCount,
       SM, Loc, Pat.getCheckTy(), Buffer, 0, Buffer.size(), Diags);
   SM.PrintMessage(SearchRange.Start, SourceMgr::DK_Note, "scanning from here");
 
   // Allow the pattern to print additional information if desired.
   Pat.PrintVariableUses(SM, Buffer, VariableTable);
   if (ExpectedMatch)
-    Pat.PrintFuzzyMatch(SM, Buffer, VariableTable);
+    Pat.PrintFuzzyMatch(SM, Buffer, VariableTable, Diags);
 }
 
 static void PrintNoMatch(bool ExpectedMatch, const SourceMgr &SM,
                          const FileCheckString &CheckStr, int MatchedCount,
                          StringRef Buffer, StringMap<StringRef> &VariableTable,
                          bool VerboseVerbose,
                          std::vector<FileCheckDiag> *Diags) {
   PrintNoMatch(ExpectedMatch, SM, CheckStr.Prefix, CheckStr.Loc, CheckStr.Pat,

llvm/trunk/test/FileCheck/dump-input-annotations.txt

 ;--------------------------------------------------
-; Use -strict-whitespace to check marker alignment here.
+; Use -strict-whitespace to check marker and note alignment here.
 ; (Also check multiline marker where start/end columns vary across lines.)
 ;
 ; In the remaining checks, don't use -strict-whitespace and thus check just the
 ; presence, order, and lengths of markers.  That way, if we ever change padding
 ; within line labels, we don't have to adjust so many tests.
 ;--------------------------------------------------
 
 ; RUN: echo 'hello world' > %t.in
 ; RUN: echo 'goodbye' >> %t.in
 ; RUN: echo 'world' >> %t.in
+; RUN: echo 'unicorn' >> %t.in
 
 ; RUN: echo 'CHECK: hello' > %t.chk
 ; RUN: echo 'CHECK: universe' >> %t.chk
 
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -strict-whitespace -match-full-lines -check-prefix=ALIGN %s
 
 ; ALIGN:Full input was:
 ; ALIGN-NEXT:<<<<<<
-; ALIGN-NEXT:         1: hello world
-; ALIGN-NEXT:check:2           X~~~~
-; ALIGN-NEXT:         2: goodbye
-; ALIGN-NEXT:check:2     ~~~~~~~
-; ALIGN-NEXT:         3: world
-; ALIGN-NEXT:check:2     ~~~~~ error: no match found
+; ALIGN-NEXT:           1: hello world
+; ALIGN-NEXT:check:2'0           X~~~~
+; ALIGN-NEXT:           2: goodbye
+; ALIGN-NEXT:check:2'0     ~~~~~~~
+; ALIGN-NEXT:           3: world
+; ALIGN-NEXT:check:2'0     ~~~~~
+; ALIGN-NEXT:           4: unicorn
+; ALIGN-NEXT:check:2'0     ~~~~~~~ error: no match found
+; ALIGN-NEXT:check:2'1     ?       possible intended match
 ; ALIGN-NEXT:>>>>>>
 ; ALIGN-NOT:{{.}}
 
 ;--------------------------------------------------
-; CHECK (also: multi-line search range)
+; CHECK (also: multi-line search range, fuzzy match)
 ;--------------------------------------------------
 
 ; Good match and no match.
 
 ; RUN: echo 'hello'   > %t.in
 ; RUN: echo 'again'   >> %t.in
 ; RUN: echo 'whirled' >> %t.in
 
 ; RUN: echo 'CHECK: hello' > %t.chk
 ; RUN: echo 'CHECK: world' >> %t.chk
 
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefix=CHK
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=CHK,CHK-V
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -vv 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=CHK,CHK-V
 
 ; CHK:        <<<<<<
-; CHK-NEXT:            1: hello
-; CHK-NEXT:            2: again
-; CHK-NEXT:   check:2     X~~~~
-; CHK-NEXT:            3: whirled
-; CHK-NEXT:   check:2     ~~~~~~~ error: no match found
+; CHK-NEXT:              1: hello
+; CHK-NEXT:              2: again
+; CHK-NEXT:   check:2'0     X~~~~
+; CHK-NEXT:              3: whirled
+; CHK-NEXT:   check:2'0     ~~~~~~~ error: no match found
+; CHK-NEXT:   check:2'1     ?       possible intended match
 ; CHK-NEXT:   >>>>>>
 ; CHK-NOT:    {{.}}
 
 ;--------------------------------------------------
 ; CHECK-COUNT-<num>
 ;--------------------------------------------------
 
 ; Good match and no match.

 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=LAB
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=LAB,LAB-V
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -vv 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=LAB,LAB-V,LAB-VV
 
 ; LAB:         <<<<<<
-; LAB-NEXT:             1: lab0
-; LAB-NEXT:             2: foo
-; LAB-NEXT:    label:3     X~~
-; LAB-NEXT:             3: lab1
-; LAB-NEXT:    label:3     ~~~~
-; LAB-NEXT:             4: bar
-; LAB-NEXT:    label:3     ~~~ error: no match found
+; LAB-NEXT:               1: lab0
+; LAB-NEXT:               2: foo
+; LAB-NEXT:    label:3'0     X~~
+; LAB-NEXT:               3: lab1
+; LAB-NEXT:    label:3'0     ~~~~
+; LAB-NEXT:    label:3'1     ?    possible intended match
+; LAB-NEXT:               4: bar
+; LAB-NEXT:    label:3'0     ~~~ error: no match found
 ; LAB-NEXT:    >>>>>>
 ; LAB-NOT:     {{.}}
 
 

llvm/trunk/utils/FileCheck/FileCheck.cpp

struct MarkerStyle {
   MarkerStyle(char Lead, raw_ostream::Colors Color, const std::string &Note)
       : Lead(Lead), Color(Color), Note(Note) {}
 };
 
 static MarkerStyle GetMarker(FileCheckDiag::MatchType MatchTy) {
   switch (MatchTy) {
   case FileCheckDiag::MatchNoneButExpected:
     return MarkerStyle('X', raw_ostream::RED, "error: no match found");
+  case FileCheckDiag::MatchFuzzy:
+    return MarkerStyle('?', raw_ostream::MAGENTA, "possible intended match");
   case FileCheckDiag::MatchTypeCount:
     llvm_unreachable_internal("unexpected match type");
   }
   llvm_unreachable_internal("unexpected match type");
 }
 
 static void DumpInputAnnotationHelp(raw_ostream &OS) {
   OS << "The following description was requested by -dump-input=help to\n"
      << "explain the input annotations printed by -dump-input=always and\n"
      << "-dump-input=fail:\n\n";
 
   // Labels for input lines.
   OS << "  - ";
   WithColor(OS, raw_ostream::SAVEDCOLOR, true) << "L:";
   OS << "     labels line number L of the input file\n";
 
   // Labels for annotation lines.
   OS << "  - ";
   WithColor(OS, raw_ostream::SAVEDCOLOR, true) << "T:L";
-  OS << "    labels the match result for a pattern of type T from "
+  OS << "    labels the only match result for a pattern of type T from "
      << "line L of\n"
      << "           the check file\n";
+  OS << "  - ";
+  WithColor(OS, raw_ostream::SAVEDCOLOR, true) << "T:L'N";
+  OS << "  labels the Nth match result for a pattern of type T from line "
+     << "L of\n"
+     << "           the check file\n";
 
   // Markers on annotation lines.
   OS << "  - ";
   WithColor(OS, raw_ostream::SAVEDCOLOR, true) << "X~~";
-  OS << "    marks search range when no match is found\n";
+  OS << "    marks search range when no match is found\n"
+     << "  - ";
+  WithColor(OS, raw_ostream::SAVEDCOLOR, true) << "?";
+  OS << "      marks fuzzy match when no match is found\n";
 
   // Colors.
   OS << "  - colors ";
   WithColor(OS, raw_ostream::RED, true) << "error";
+  OS << ", ";
+  WithColor(OS, raw_ostream::MAGENTA, true) << "fuzzy match";
   OS << "\n\n"
      << "If you are not seeing color above or in input dumps, try: -color\n";
 }
 
 /// An annotation for a single input line.
 struct InputAnnotation {
   /// The check file line (one-origin indexing) where the directive that
   /// produced this annotation is located.
   unsigned CheckLine;
+  /// The index of the match result for this check.
+  unsigned CheckDiagIndex;
   /// The label for this annotation.
   std::string Label;
   /// What input line (one-origin indexing) this annotation marks.  This might
   /// be different from the starting line of the original diagnostic if this is
   /// a non-initial fragment of a diagnostic that has been broken across
   /// multiple lines.
   unsigned InputLine;
   /// The column range (one-origin indexing, open end) in which to to mark the
case Check::CheckNone:
     llvm_unreachable("invalid FileCheckType");
   }
   llvm_unreachable("unknown FileCheckType");
 }
 
 static void BuildInputAnnotations(const std::vector<FileCheckDiag> &Diags,
                                   std::vector<InputAnnotation> &Annotations,
                                   unsigned &LabelWidth) {
+  // How many diagnostics has the current check seen so far?
+  unsigned CheckDiagCount = 0;
   // What's the widest label?
   LabelWidth = 0;
   for (auto DiagItr = Diags.begin(), DiagEnd = Diags.end(); DiagItr != DiagEnd;
        ++DiagItr) {
     InputAnnotation A;
 
     // Build label, which uniquely identifies this check result.
     A.CheckLine = DiagItr->CheckLine;
     llvm::raw_string_ostream Label(A.Label);
     Label << GetCheckTypeAbbreviation(DiagItr->CheckTy) << ":"
           << DiagItr->CheckLine;
+    A.CheckDiagIndex = UINT_MAX;
+    auto DiagNext = std::next(DiagItr);
+    if (DiagNext != DiagEnd && DiagItr->CheckTy == DiagNext->CheckTy &&
+        DiagItr->CheckLine == DiagNext->CheckLine)
+      A.CheckDiagIndex = CheckDiagCount++;
+    else if (CheckDiagCount) {
+      A.CheckDiagIndex = CheckDiagCount;
+      CheckDiagCount = 0;
+    }
+    if (A.CheckDiagIndex != UINT_MAX)
+      Label << "'" << A.CheckDiagIndex;
+    else
+      A.CheckDiagIndex = 0;
     Label.flush();
     LabelWidth = std::max((std::string::size_type)LabelWidth, A.Label.size());
 
     MarkerStyle Marker = GetMarker(DiagItr->MatchTy);
     A.Marker = Marker;
 
     // Compute the mark location, and break annotation into multiple
     // annotations if it spans multiple lines.
if (DiagItr->InputStartLine == DiagItr->InputEndLine) {
         // If a range ends before the first column on a line, then it has no
         // characters on that line, so there's nothing to render.
         if (DiagItr->InputEndCol == 1 && L == E) {
           Annotations.back().Marker.Note = Marker.Note;
           break;
         }
         InputAnnotation B;
         B.CheckLine = A.CheckLine;
+        B.CheckDiagIndex = A.CheckDiagIndex;
         B.Label = A.Label;
         B.InputLine = L;
         B.Marker = Marker;
         B.Marker.Lead = '~';
         B.InputStartCol = 1;
         if (L != E) {
           B.InputEndCol = UINT_MAX;
           B.Marker.Note = "";
static void DumpAnnotatedInput(
   //
   // First, sort in the order of input lines to make it easier to find relevant
   // annotations while iterating input lines in the implementation below.
   // FileCheck diagnostics are not always reported and recorded in the order of
   // input lines due to, for example, CHECK-DAG and CHECK-NOT.
   //
   // Second, for annotations for the same input line, sort in the order of the
   // FileCheck directive's line in the check file (where there's at most one
-  // directive per line).  The rationale of this choice is that, for any input
-  // line, this sort establishes a total order of annotations that, with
-  // respect to match results, is consistent across multiple lines, thus
-  // making match results easier to track from one line to the next when they
-  // span multiple lines.
+  // directive per line) and then by the index of the match result for that
+  // directive.  The rationale of this choice is that, for any input line, this
+  // sort establishes a total order of annotations that, with respect to match
+  // results, is consistent across multiple lines, thus making match results
+  // easier to track from one line to the next when they span multiple lines.
   std::sort(Annotations.begin(), Annotations.end(),
             [](const InputAnnotation &A, const InputAnnotation &B) {
               if (A.InputLine != B.InputLine)
                 return A.InputLine < B.InputLine;
+              if (A.CheckLine != B.CheckLine)
                 return A.CheckLine < B.CheckLine;
+              assert(A.CheckDiagIndex != B.CheckDiagIndex &&
+                     "expected diagnostic indices to be unique within a "
+                     " check line");
+              return A.CheckDiagIndex < B.CheckDiagIndex;
             });
 
   // Compute the width of the label column.
   const unsigned char *InputFilePtr = InputFileText.bytes_begin(),
                       *InputFileEnd = InputFileText.bytes_end();
   unsigned LineCount = InputFileText.count('\n');
   if (InputFileEnd[-1] != '\n')
     ++LineCount;