[CompilationDatabase] remove duplicate elements

[rizsotto.mailinglist]

Summary

Hi Guys,

I would refer to an earlier conversation on the cfe-dev list...

On Wed, Nov 30, 2016 at 10:34 PM Laszlo Nagy <[e-mail]> wrote:

• compilation flags like -MD, -MT x, -MF x, etc... might cause "duplicate" items in the compilation database. (the only difference would be the presence of these flags) i did modify the command line and filter out these flags. but if i'm following your logic, tooling libs shall notice that there is no semantic difference between the two compilation and run only once. (which is not the case now) would you be happy with that?

If there's no semantic difference, I'd argue that there should only be one entry in the compilation db. On the other hand, I'd not object to patches making the tooling infra deduplicate :)

So, I'm preparing a patch to get rid of duplicate compilation database entries. The current version is just a scratch, did not implemented the filtering itself, but just studied how this can be done. Looking forward for you comments, ideas!

[rizsotto.mailinglist]

Implemented preprocessor flag filtering.

[joerg]

Personally, I think this should have been handled by whatever tool created the JSON database file in first place.

[rizsotto.mailinglist]

All right, then everything stays the same. :)

lib/Tooling/JSONCompilationDatabase.cpp

 #include "clang/Tooling/CompilationDatabase.h"
 #include "clang/Tooling/CompilationDatabasePluginRegistry.h"
 #include "clang/Tooling/Tooling.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/StringSaver.h"
+#include <set>
 #include <system_error>
 
 namespace clang {
 namespace tooling {
 
 namespace {
 
 /// \brief A parser for escaped strings of command line arguments.
nodeToCommandLine(JSONCommandLineSyntax Syntax,
   }
   std::vector<std::string> Arguments;
   for (auto *Node : Nodes) {
     Arguments.push_back(Node->getValue(Storage));
   }
   return Arguments;
 }
 
+static std::vector<std::string>
+deduplicateCommandLine(const std::vector<std::string> &Nodes) {
+  struct FlagFilter {
+    private:
+      int State;
+      const std::map<std::string, int> Flags = {
+        {"-MD", 0}, {"-MMD", 0}, {"-MG", 0}, {"-MP", 0}, {"-MF", 1}, {"-MT", 1}, {"-MQ", 1}
+      };
+    public:
+      FlagFilter() : State(0) { }
+
+      bool operator()(const std::string &Current) {
+        if (State != 0) {
+          --State;
+          return false;
+        }
+        const auto Check = Flags.find(Current);
+        if (Check != Flags.end()) {
+          State = Check->second;
+          return false;
+        }
+        return true;
+      }
+  };
+
+  std::vector<std::string> Result;
+  std::copy_if(Nodes.begin(), Nodes.end(), std::back_inserter(Result), FlagFilter());
+  return Result;
+}
+
+struct CompareCompileCommand {
+  bool operator()(const CompileCommand &Lhs, const CompileCommand &Rhs) const {
+    return (Lhs.Directory == Rhs.Directory) &&
+           (Lhs.Filename == Rhs.Filename) &&
+           (Lhs.CommandLine == Rhs.CommandLine) &&
+           (Lhs.Output == Rhs.Output);
+  }
+};
+
 void JSONCompilationDatabase::getCommands(
     ArrayRef<CompileCommandRef> CommandsRef,
     std::vector<CompileCommand> &Commands) const {
+  // To filter out duplicate elements put the result first into a set.
+  std::set<CompileCommand, CompareCompileCommand> Results;
   for (int I = 0, E = CommandsRef.size(); I != E; ++I) {
     SmallString<8> DirectoryStorage;
     SmallString<32> FilenameStorage;
     SmallString<32> OutputStorage;
     auto Output = std::get<3>(CommandsRef[I]);
-    Commands.emplace_back(
+    Results.emplace(
         std::get<0>(CommandsRef[I])->getValue(DirectoryStorage),
         std::get<1>(CommandsRef[I])->getValue(FilenameStorage),
-        nodeToCommandLine(Syntax, std::get<2>(CommandsRef[I])),
+        deduplicateCommandLine(
+          nodeToCommandLine(Syntax, std::get<2>(CommandsRef[I]))),
         Output ? Output->getValue(OutputStorage) : "");
   }
+  // Copy the unique elements into the real return value.
+  std::copy(Results.begin(), Results.end(), std::back_inserter(Commands));
 }
 
 bool JSONCompilationDatabase::parse(std::string &ErrorMessage) {
   llvm::yaml::document_iterator I = YAMLStream.begin();
   if (I == YAMLStream.end()) {
     ErrorMessage = "Error while parsing YAML.";
     return false;
   }