[analyzer] Fix HTML report deduplication.

[NoQ]

Summary

Folks, I deleted some code.

Some of it was... Perl code.

Not sure if anybody noticed but scan-build didn't deduplicate reports correctly. Its algorithm was extremely primitive (basically it hashed each html file and deleted files with duplicate hash). This obviously didn't work with cross-file reports (i.e., bug path starts in the main file but ends in a header) which we intended to deduplicate by the end-of-path location, because the main file name is included in the report and therefore affects md5. D42269 doesn't help either. Long story short, it didn't honor the modern advancements in IssueHash at all.

I could make scan-build read the issue hash but I think my solution is even more elegant so hear me out. I put the issue hash into the report filename instead, replacing the random section. When clang tries to emit a duplicate report, it'll simply fail because the file with that name is already present. Moreover, as per tradition, I reduce the issue hash to the first 6 characters, so bug report filenames look exactly like before (report-123abc.html), except now they're automatically stable and deterministic! Such truncation is, of course, entirely cosmetic and absolutely unnecessary but I think it's pretty cool.

The flag -analyzer-config stable-report-filename=true now becomes redundant because reports are stable anyway (in fact, they weren't actually stable before the patch even with the flag, because they depended on the race between clang invocations to emit the reports; I changed it to include a snippet of the issue hash as well instead of the race-y index). That said, I'm conflicted about removing the flag entirely because it also produces more verbose/readable filenames that people seem to enjoy. I think we should enable it by default instead, as soon as we make sure it doesn't produce extremely long filenames.

scan-build --generate-index-only no longer does deduplication, as seen on the updated test "rebuild_index". If deduplication is desired, it can typically be achieved with a simple

find -name '*.html' -exec mv "{}" . \;

[steakhal]

AFAIK from CodeChecker's side we are fine with this change. I think we are not using the stable-report-filename analyzer config either.
Aside from these, the less Perl the better, I guess.

clang/lib/StaticAnalyzer/Core/HTMLDiagnostics.cpp

314

I don't frequently see stringstreams in the codebase.
Why don't you use the llvm alternative?

std::string s;
llvm::raw_string_ostream os(s);

[vsavchenko]

This is incredible! Thanks for addressing it! I've encountered this many times.

clang/lib/StaticAnalyzer/Core/HTMLDiagnostics.cpp

314

nit: whatever stream you choose, can you please start it with a capital letter? 💅

323–324

Can we make a mirror for this option and mark the other one as deprecated?

[ASDenysPetrov]

Nice work! Unfortunately I'm not able to run tests on my Windows env, but I've run you tests files manually. It works for me.

P.S. BTW, is there any workarounds to make current tests supported on Windows? I know there is REQUIRES instruction (https://llvm.org/docs/TestingGuide.html#constraining-test-execution) but I didn't find any sufficient description of it.

clang/lib/StaticAnalyzer/Core/HTMLDiagnostics.cpp

323–324

Nice idea. I'm in favor of a mirorring.

337

Do you think 6 trimmed characters are enough to avoid collisions?

[NoQ]

Address review comments!

[NoQ]

In D105167#2949521, @ASDenysPetrov wrote:

Nice work! Unfortunately I'm not able to run tests on my Windows env, but I've run you tests files manually. It works for me.

P.S. BTW, is there any workarounds to make current tests supported on Windows? I know there is REQUIRES instruction (https://llvm.org/docs/TestingGuide.html#constraining-test-execution) but I didn't find any sufficient description of it.

Yes, you can always remove the REQUIRES: directive. The problem with generally enabling these tests on Windows is that scan-build is written in Perl and the Perl interpreter isn't shipped with Windows by default. Nothing else in LLVM is written in Perl so requiring a Perl installation to run LLVM tests just for this single use case is counter-productive.

clang/lib/StaticAnalyzer/Core/HTMLDiagnostics.cpp

323–324

The new option is -analyzer-config verbose-report-filename=true and the old option is preserved and acts exactly the same; the help text says that it's deprecated.

337

It's basically same as before. Previously these 6 digits were completely random, now they're chunks of MD5 hashes of the real thing which is probably as random.

That's 16.7 million variants. The probability of collision on 6 digits with 100 warnings is around 0.0001%, with 1000 warnings it goes up to 3% (Birthday Problem). Even though 1000 warnings are a real possibility on a huge project (eg., LLVM or WebKit), this is way above the point where you care about collisions.

I think there's nothing that prevents us from bumping it (at least in the non-verbose filename mode; in the verbose filename mode it may cause slightly more breakages in the case where it's close to hitting the file system's filename length limit). Bumping from 6 digits to 8 digits would reduce the collision probability on 1000 warnings to 0.0001%. I'll think about it a bit more :)

clang/include/clang/Analysis/PathDiagnostic.h

struct PathDiagnosticConsumerOptions {
   /// without re-compiling the program under analysis.
   bool ShouldDisplayMacroExpansions = false;
 
   /// Whether to include LLVM statistics of the process in the diagnostic.
   /// Useful for profiling the tool on large real-world codebases.
   bool ShouldSerializeStats = false;
 
   /// If the consumer intends to produce multiple output files, should it
-  /// use randomly generated file names for these files (with the tiny risk of
-  /// having random collisions) or deterministic human-readable file names
-  /// (with a larger risk of deterministic collisions or invalid characters
-  /// in the file name). We should not really give this choice to the users
-  /// because deterministic mode is always superior when done right, but
-  /// for some consumers this mode is experimental and needs to be
-  /// off by default.
-  bool ShouldWriteStableReportFilename = false;
+  /// use a pseudo-random file name name or a human-readable file name.
+  bool ShouldWriteVerboseReportFilename = false;
 
   /// Whether the consumer should treat consumed diagnostics as hard errors.
   /// Useful for breaking your build when issues are found.
   bool ShouldDisplayWarningsAsErrors = false;
 
   /// Whether the consumer should attempt to rewrite the source file
   /// with fix-it hints attached to the diagnostics it consumes.
   bool ShouldApplyFixIts = false;

clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.h

#undef ANALYZER_OPTION_DEPENDS_ON_USER_MODE
   ///
   /// \sa CXXMemberInliningMode
   bool mayInlineCXXMemberFunction(CXXInlineableMemberKind K) const;
 
   ento::PathDiagnosticConsumerOptions getDiagOpts() const {
     return {FullCompilerInvocation,
             ShouldDisplayMacroExpansions,
             ShouldSerializeStats,
-            ShouldWriteStableReportFilename,
+            // The stable report filename option is deprecated because
+            // file names are now always stable. Now the old option acts as
+            // an alias to the new verbose filename option because this
+            // closely mimics the behavior under the old option.
+            ShouldWriteStableReportFilename || ShouldWriteVerboseReportFilename,
             AnalyzerWerror,
             ShouldApplyFixIts,
             ShouldDisplayCheckerNameForText};
   }
 };
 
 using AnalyzerOptionsRef = IntrusiveRefCntPtr<AnalyzerOptions>;
 

clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def

 
 ANALYZER_OPTION(bool, ShouldReportIssuesInMainSourceFile,
                 "report-in-main-source-file",
                 "Whether or not the diagnostic report should be always "
                 "reported in the main source file and not the headers.",
                 false)
 
 ANALYZER_OPTION(bool, ShouldWriteStableReportFilename, "stable-report-filename",
-                "Whether or not the report filename should be random or not.",
+                "Deprecated: report filenames are now always stable. "
+                "See also 'verbose-report-filename'.",
+                false)
+
+ANALYZER_OPTION(bool, ShouldWriteVerboseReportFilename, "verbose-report-filename",
+                "Whether or not the report filename should contain extra "
+                "information about the issue.",
                 false)
 
 ANALYZER_OPTION(
     bool, ShouldSerializeStats, "serialize-stats",
     "Whether the analyzer should serialize statistics to plist output. "
     "Statistics would be serialized in JSON format inside the main dictionary "
     "under the statistics key. Available only if compiled in assert mode or "
     "with LLVM statistics explicitly enabled.",

clang/lib/StaticAnalyzer/Core/HTMLDiagnostics.cpp

 
 void HTMLDiagnostics::FlushDiagnosticsImpl(
   std::vector<const PathDiagnostic *> &Diags,
   FilesMade *filesMade) {
   for (const auto Diag : Diags)
     ReportDiag(*Diag, filesMade);
 }
 
+static llvm::SmallString<32> getIssueHash(const PathDiagnostic &D,
+                                          const Preprocessor &PP) {
+  SourceManager &SMgr = PP.getSourceManager();
+  PathDiagnosticLocation UPDLoc = D.getUniqueingLoc();
+  FullSourceLoc L(SMgr.getExpansionLoc(UPDLoc.isValid()
+                                           ? UPDLoc.asLocation()
+                                           : D.getLocation().asLocation()),
+                  SMgr);
+  return getIssueHash(L, D.getCheckerName(), D.getBugType(),
+                      D.getDeclWithIssue(), PP.getLangOpts());
+}
+
 void HTMLDiagnostics::ReportDiag(const PathDiagnostic& D,
                                  FilesMade *filesMade) {
   // Create the HTML directory if it is missing.
   if (!createdDir) {
     createdDir = true;
     if (std::error_code ec = llvm::sys::fs::create_directories(Directory)) {
       llvm::errs() << "warning: could not create directory '"
                    << Directory << "': " << ec.message() << '\n';
void HTMLDiagnostics::ReportDiag(const PathDiagnostic& D,
 
   // The path as already been prechecked that the path is non-empty.
   assert(!path.empty());
   const SourceManager &SMgr = path.front()->getLocation().getManager();
 
   // Create a new rewriter to generate HTML.
   Rewriter R(const_cast<SourceManager&>(SMgr), PP.getLangOpts());
 
-  // The file for the first path element is considered the main report file, it
-  // will usually be equivalent to SMgr.getMainFileID(); however, it might be a
-  // header when -analyzer-opt-analyze-headers is used.
-  FileID ReportFile = path.front()->getLocation().asLocation().getExpansionLoc().getFileID();
-
   // Get the function/method name
   SmallString<128> declName("unknown");
   int offsetDecl = 0;
   if (const Decl *DeclWithIssue = D.getDeclWithIssue()) {
       if (const auto *ND = dyn_cast<NamedDecl>(DeclWithIssue))
           declName = ND->getDeclName().getAsString();
 
       if (const Stmt *Body = DeclWithIssue->getBody()) {
void HTMLDiagnostics::ReportDiag(const PathDiagnostic& D,
   std::string report = GenerateHTML(D, R, SMgr, path, declName.c_str());
   if (report.empty()) {
     llvm::errs() << "warning: no diagnostics generated for main file.\n";
     return;
   }
 
   // Create a path for the target HTML file.
   int FD;
-  SmallString<128> Model, ResultPath;
 
-  if (!DiagOpts.ShouldWriteStableReportFilename) {
-      llvm::sys::path::append(Model, Directory, "report-%%%%%%.html");
-      if (std::error_code EC =
-          llvm::sys::fs::make_absolute(Model)) {
-          llvm::errs() << "warning: could not make '" << Model
-                       << "' absolute: " << EC.message() << '\n';
-        return;
-      }
-      if (std::error_code EC = llvm::sys::fs::createUniqueFile(
-              Model, FD, ResultPath, llvm::sys::fs::OF_Text)) {
-        llvm::errs() << "warning: could not create file in '" << Directory
-                     << "': " << EC.message() << '\n';
-        return;
-      }
-  } else {
-      int i = 1;
-      std::error_code EC;
-      do {
-          // Find a filename which is not already used
-          const FileEntry* Entry = SMgr.getFileEntryForID(ReportFile);
-          std::stringstream filename;
-          Model = "";
-          filename << "report-"
-                   << llvm::sys::path::filename(Entry->getName()).str()
-                   << "-" << declName.c_str()
-                   << "-" << offsetDecl
-                   << "-" << i << ".html";
-          llvm::sys::path::append(Model, Directory,
-                                  filename.str());
-          EC = llvm::sys::fs::openFileForReadWrite(
-              Model, FD, llvm::sys::fs::CD_CreateNew, llvm::sys::fs::OF_None);
-          if (EC && EC != llvm::errc::file_exists) {
-              llvm::errs() << "warning: could not create file '" << Model
-                           << "': " << EC.message() << '\n';
-              return;
-          }
-          i++;
-      } while (EC);
-  }
+  SmallString<128> FileNameStr;
+  llvm::raw_svector_ostream FileName(FileNameStr);
+  FileName << "report-";
+
+  // Historically, neither the stable report filename nor the unstable report
+  // filename were actually stable. That said, the stable report filename
+  // was more stable because it was mostly composed of information
+  // about the bug report instead of being completely random.
+  // Now both stable and unstable report filenames are in fact stable
+  // but the stable report filename is still more verbose.
+  if (DiagOpts.ShouldWriteVerboseReportFilename) {
+    // FIXME: This code relies on knowing what constitutes the issue hash.
+    // Otherwise deduplication won't work correctly.
+    FileID ReportFile =
+        path.back()->getLocation().asLocation().getExpansionLoc().getFileID();
+
+    const FileEntry *Entry = SMgr.getFileEntryForID(ReportFile);
+
+    FileName << llvm::sys::path::filename(Entry->getName()).str() << "-"
+             << declName.c_str() << "-" << offsetDecl << "-";
+  }
+
+  FileName << StringRef(getIssueHash(D, PP)).substr(0, 6).str() << ".html";
+
+  SmallString<128> ResultPath;
+  llvm::sys::path::append(ResultPath, Directory, FileName.str());
+  if (std::error_code EC = llvm::sys::fs::make_absolute(ResultPath)) {
+    llvm::errs() << "warning: could not make '" << ResultPath
+
+  if (std::error_code EC = llvm::sys::fs::openFileForReadWrite(
+          ResultPath, FD, llvm::sys::fs::CD_CreateNew,
+          llvm::sys::fs::OF_None)) {
+    // Existence of the file corresponds to the situation where a different
+    // Clang instance has emitted a bug report with the same issue hash.
+    // This is an entirely normal situation that does not deserve a warning,
+    // as apart from hash collisions this can happen because the reports
+    // are in fact similar enough to be considered duplicates of each other.
+    if (EC != llvm::errc::file_exists) {
 
   llvm::raw_fd_ostream os(FD, true);
 
   if (filesMade)
     filesMade->addDiagnostic(D, getName(),
                              llvm::sys::path::filename(ResultPath));
 
   // Emit the HTML to disk.
// Embed meta-data tags.
     if (!BugType.empty())
       os << "\n<!-- BUGTYPE " << BugType << " -->\n";
 
     PathDiagnosticLocation UPDLoc = D.getUniqueingLoc();
     FullSourceLoc L(SMgr.getExpansionLoc(UPDLoc.isValid()
                                              ? UPDLoc.asLocation()
                                              : D.getLocation().asLocation()),
                     SMgr);
-    const Decl *DeclWithIssue = D.getDeclWithIssue();
 
     StringRef BugCategory = D.getCategory();
     if (!BugCategory.empty())
       os << "\n<!-- BUGCATEGORY " << BugCategory << " -->\n";
 
     os << "\n<!-- BUGFILE " << DirName << Entry->getName() << " -->\n";
 
     os << "\n<!-- FILENAME " << llvm::sys::path::filename(Entry->getName()) << " -->\n";
 
     os  << "\n<!-- FUNCTIONNAME " <<  declName << " -->\n";
 
-    os << "\n<!-- ISSUEHASHCONTENTOFLINEINCONTEXT "
-       << getIssueHash(L, D.getCheckerName(), D.getBugType(), DeclWithIssue,
-                       PP.getLangOpts())
+    os << "\n<!-- ISSUEHASHCONTENTOFLINEINCONTEXT " << getIssueHash(D, PP)
        << " -->\n";
 
     os << "\n<!-- BUGLINE "
        << LineNumber
        << " -->\n";
 
     os << "\n<!-- BUGCOLUMN "
       << ColumnNumber

clang/test/Analysis/analyzer-config.c

 // CHECK-NEXT: suppress-c++-stdlib = true
 // CHECK-NEXT: suppress-inlined-defensive-checks = true
 // CHECK-NEXT: suppress-null-return-paths = true
 // CHECK-NEXT: track-conditions = true
 // CHECK-NEXT: track-conditions-debug = false
 // CHECK-NEXT: unix.DynamicMemoryModeling:AddNoOwnershipChangeNotes = false
 // CHECK-NEXT: unix.DynamicMemoryModeling:Optimistic = false
 // CHECK-NEXT: unroll-loops = false
+// CHECK-NEXT: verbose-report-filename = false
 // CHECK-NEXT: widen-loops = false

clang/test/Analysis/scan-build/Inputs/deduplication/1.c

+#include "header.h"
+
+void bar() {
+  foo();
+}

clang/test/Analysis/scan-build/Inputs/deduplication/2.c

+#include "header.h"
+
+void bar() {
+  foo();
+}

clang/test/Analysis/scan-build/Inputs/deduplication/header.h

+int foo() {
+  int x = 0;
+  return 1 / x;
+}

clang/test/Analysis/scan-build/deduplication.test

+// FIXME: Actually, "perl".
+REQUIRES: shell
+
+RUN: rm -rf %t.output_dir && mkdir %t.output_dir
+RUN: %scan-build -o %t.output_dir \
+RUN:             %clang -S %S/Inputs/deduplication/1.c \
+RUN:                       %S/Inputs/deduplication/2.c \
+RUN:     | FileCheck %s -check-prefix CHECK-STDOUT
+
+RUN: ls %t.output_dir/*/ | FileCheck %s -check-prefix CHECK-FILENAMES
+
+RUN: rm -rf %t.output_dir && mkdir %t.output_dir
+RUN: %scan-build -o %t.output_dir \
+RUN:             -analyzer-config stable-report-filename=true \
+RUN:             %clang -S %S/Inputs/deduplication/1.c \
+RUN:                       %S/Inputs/deduplication/2.c \
+RUN:     | FileCheck %s -check-prefix CHECK-STDOUT
+
+RUN: ls %t.output_dir/*/ | FileCheck %s -check-prefix CHECK-FILENAMES
+
+RUN: rm -rf %t.output_dir && mkdir %t.output_dir
+RUN: %scan-build -o %t.output_dir \
+RUN:             -analyzer-config verbose-report-filename=true \
+RUN:             %clang -S %S/Inputs/deduplication/1.c \
+RUN:                       %S/Inputs/deduplication/2.c \
+RUN:     | FileCheck %s -check-prefix CHECK-STDOUT
+
+RUN: ls %t.output_dir/*/ | FileCheck %s -check-prefix CHECK-FILENAMES
+
+// Only one report file should be generated.
+
+CHECK-STDOUT: scan-build: Using '{{.*}}' for static analysis
+CHECK-STDOUT: scan-build: 1 bug found.
+CHECK-STDOUT: scan-build: Run 'scan-view {{.*}}' to examine bug reports.
+
+
+CHECK-FILENAMES: index.html
+CHECK-FILENAMES-NEXT: report-{{.*}}.html
+CHECK-FILENAMES-NEXT: scanview.css
+CHECK-FILENAMES-NEXT: sorttable.js

clang/test/Analysis/scan-build/rebuild_index/rebuild_index.test

 // FIXME: Actually, "perl".
 REQUIRES: shell
 
 RUN: rm -rf %t.output_dir && mkdir %t.output_dir
 RUN: cp %S/report-1.html %t.output_dir
 RUN: cp %S/report-2.html %t.output_dir
-RUN: cp %S/report-3.html %t.output_dir
 RUN: mkdir %t.output_dir/subdirectory
-RUN: cp %S/subdirectory/report-4.html %t.output_dir/subdirectory
+RUN: cp %S/subdirectory/report-3.html %t.output_dir/subdirectory
 
 RUN: %scan-build --generate-index-only %t.output_dir
 
 RUN: ls %t.output_dir | FileCheck -check-prefix CHECK-FILES %s
 
 CHECK-FILES:      index.html
 CHECK-FILES-NEXT: report-1.html
 CHECK-FILES-NEXT: report-2.html
-
-// report-3.html is a duplicate of report-1.html so it's not present.
-CHECK-FILES-NOT:  report-3.html
 CHECK-FILES-NEXT: scanview.css
 CHECK-FILES-NEXT: sorttable.js
 CHECK-FILES-NEXT: subdirectory
 
 RUN: ls %t.output_dir/subdirectory | FileCheck -check-prefix CHECK-SUB %s
 
-CHECK-SUB: report-4.html
+CHECK-SUB: report-3.html
 
 RUN: cat %t.output_dir/index.html | FileCheck -check-prefix CHECK-INDEX %s
 
 CHECK-INDEX:      cat1
 CHECK-INDEX-NEXT: bug1
 CHECK-INDEX-NEXT: cat2
 CHECK-INDEX-NEXT: bug2
-CHECK-INDEX-NEXT: cat4
-CHECK-INDEX-NEXT: bug4
+CHECK-INDEX-NEXT: cat3
+CHECK-INDEX-NEXT: bug3
 
 CHECK-INDEX:     report-1.html#EndPath
 CHECK-INDEX:     report-2.html#EndPath
-CHECK-INDEX-NOT: report-3.html#EndPath
-CHECK-INDEX:     subdirectory/report-4.html#EndPath
+CHECK-INDEX:     subdirectory/report-3.html#EndPath

clang/test/Analysis/scan-build/rebuild_index/report-3.html

-<!-- BUGTYPE bug1 -->
-<!-- BUGFILE file1 -->
-<!-- BUGPATHLENGTH 1 -->
-<!-- BUGLINE 1 -->
-<!-- BUGCATEGORY cat1 -->
-<!-- BUGDESC desc1 -->
-<!-- FUNCTIONNAME func1 -->
-<!-- BUGMETAEND -->

clang/test/Analysis/scan-build/rebuild_index/subdirectory/report-3.html

+<!-- BUGTYPE bug3 -->
+<!-- BUGFILE file3 -->
+<!-- BUGPATHLENGTH 3 -->
+<!-- BUGLINE 3 -->
+<!-- BUGCATEGORY cat3 -->
+<!-- BUGDESC desc3 -->
+<!-- FUNCTIONNAME func3 -->
+<!-- BUGMETAEND -->

clang/test/Analysis/scan-build/rebuild_index/subdirectory/report-4.html

-<!-- BUGTYPE bug4 -->
-<!-- BUGFILE file4 -->
-<!-- BUGPATHLENGTH 4 -->
-<!-- BUGLINE 4 -->
-<!-- BUGCATEGORY cat4 -->
-<!-- BUGDESC desc4 -->
-<!-- FUNCTIONNAME func4 -->
-<!-- BUGMETAEND -->

clang/tools/scan-build/bin/scan-build

 #!/usr/bin/env perl
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 ##===----------------------------------------------------------------------===##
 #
 # A script designed to wrap a build so that all calls to gcc are intercepted
 # and piped to the static analyzer.
 #
 ##===----------------------------------------------------------------------===##
 
 use strict;
 use warnings;
 use FindBin qw($RealBin);
-use Digest::MD5;
 use File::Basename;
 use File::Find;
 use File::Copy qw(copy);
 use File::Path qw( rmtree mkpath );
 use Term::ANSIColor;
 use Term::ANSIColor qw(:constants);
 use Cwd qw/ getcwd abs_path /;
 use Sys::Hostname;
sub SetHtmlEnv {
   if ($Options{Verbose}) {
     Diag("Emitting reports for this run to '$Dir'.\n");
   }
 
   $ENV{'CCC_ANALYZER_HTML'} = $Dir;
 }
 
 ##----------------------------------------------------------------------------##
-# ComputeDigest - Compute a digest of the specified file.
-##----------------------------------------------------------------------------##
-
-sub ComputeDigest {
-  my $FName = shift;
-  DieDiag("Cannot read $FName to compute Digest.\n") if (! -r $FName);
-
-  # Use Digest::MD5.  We don't have to be cryptographically secure.  We're
-  # just looking for duplicate files that come from a non-malicious source.
-  # We use Digest::MD5 because it is a standard Perl module that should
-  # come bundled on most systems.
-  open(FILE, $FName) or DieDiag("Cannot open $FName when computing Digest.\n");
-  binmode FILE;
-  my $Result = Digest::MD5->new->addfile(*FILE)->hexdigest;
-  close(FILE);
-
-  # Return the digest.
-  return $Result;
-}
-
-##----------------------------------------------------------------------------##
 #  UpdatePrefix - Compute the common prefix of files.
 ##----------------------------------------------------------------------------##
 
 my $Prefix;
 
 sub UpdatePrefix {
   my $x = shift;
   my $y = basename($x);

 
 ##----------------------------------------------------------------------------##
 # ScanFile - Scan a report file for various identifying attributes.
 ##----------------------------------------------------------------------------##
 
 # Sometimes a source file is scanned more than once, and thus produces
 # multiple error reports.  We use a cache to solve this problem.
 
-my %AlreadyScanned;
-
 sub ScanFile {
 
   my $Index = shift;
   my $Dir = shift;
   my $FName = shift;
   my $Stats = shift;
 
-  # Compute a digest for the report file.  Determine if we have already
-  # scanned a file that looks just like it.
-
-  my $digest = ComputeDigest("$Dir/$FName");
-
-  if (defined $AlreadyScanned{$digest}) {
-    # Redundant file.  Remove it.
-    unlink("$Dir/$FName");
-    return;
-  }
-
-  $AlreadyScanned{$digest} = 1;
-
   # At this point the report file is not world readable.  Make it happen.
   chmod(0644, "$Dir/$FName");
 
   # Scan the report file for tags.
   open(IN, "$Dir/$FName") or DieDiag("Cannot open '$Dir/$FName'\n");
 
   my $BugType        = "";
   my $BugFile        = "";