[mlir][pdll] Add extra-dirs for LSP includes.

[jpienaar]

Summary

Enable specifying additional include directories to search. This is
consistent with what one can do with clangd (although there it is more
general compilation options) and Python LSP. We would in general expect
these to be provided by compilation database equivalent.

[rriddle]

Nice, thanks.

mlir/lib/Tools/mlir-pdll-lsp-server/MlirPdllLspServerMain.cpp

58

Unrelated, but this should probably say PDLL. Oops

mlir/lib/Tools/mlir-pdll-lsp-server/PDLLServer.h

79

nit: Can you move this into the Impl? It helps keep all of the class state contained in one place.

[NathanielMcVicar]

This is causing failures on the Windows builbot: https://lab.llvm.org/buildbot/#/builders/13/builds/19626/steps/6/logs/stdio
I believe you simply need #include <string> to make MSVC happy.

C:\PROGRA~2\MICROS~3\2019\COMMUN~1\VC\Tools\MSVC\1429~1.301\bin\Hostx64\x64\cl.exe  /nologo /TP -DBUILD_EXAMPLES -DGTEST_HAS_RTTI=0 -DMLIR_CUDA_CONVERSIONS_ENABLED=1 -DMLIR_ROCM_CONVERSIONS_ENABLED=1 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:\buildbot\mlir-x64-windows-ninja\build\tools\mlir\lib\Tools\mlir-pdll-lsp-server -IC:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\lib\Tools\mlir-pdll-lsp-server -IC:\buildbot\mlir-x64-windows-ninja\build\include -IC:\buildbot\mlir-x64-windows-ninja\llvm-project\llvm\include -IC:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\include -IC:\buildbot\mlir-x64-windows-ninja\build\tools\mlir\include /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Zc:strictStrings /Oi /Zc:rvalueCast /bigobj /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4324 -w14062 -we4238 /Gw /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++14 /showIncludes /Fotools\mlir\lib\Tools\mlir-pdll-lsp-server\CMakeFiles\MLIRPdllLspServerLib.dir\PDLLServer.cpp.obj /Fdtools\mlir\lib\Tools\mlir-pdll-lsp-server\CMakeFiles\MLIRPdllLspServerLib.dir\MLIRPdllLspServerLib.pdb /FS -c C:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\lib\Tools\mlir-pdll-lsp-server\PDLLServer.cpp
C:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\lib\Tools\mlir-pdll-lsp-server\PDLLServer.h(32): error C2039: 'string': is not a member of 'std'
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include\vector(24): note: see declaration of 'std'
C:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\lib\Tools\mlir-pdll-lsp-server\PDLLServer.h(32): error C2065: 'string': undeclared identifier
C:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\lib\Tools\mlir-pdll-lsp-server\PDLLServer.h(32): error C2923: 'std::vector': 'string' is not a valid template type argument for parameter '_Ty'
C:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\lib\Tools\mlir-pdll-lsp-server\PDLLServer.h(32): note: see declaration of 'string'

[jpienaar]

In D123474#3448889, @NathanielMcVicar wrote:

This is causing failures on the Windows builbot: https://lab.llvm.org/buildbot/#/builders/13/builds/19626/steps/6/logs/stdio
I believe you simply need #include <string> to make MSVC happy.

C:\PROGRA~2\MICROS~3\2019\COMMUN~1\VC\Tools\MSVC\1429~1.301\bin\Hostx64\x64\cl.exe  /nologo /TP -DBUILD_EXAMPLES -DGTEST_HAS_RTTI=0 -DMLIR_CUDA_CONVERSIONS_ENABLED=1 -DMLIR_ROCM_CONVERSIONS_ENABLED=1 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:\buildbot\mlir-x64-windows-ninja\build\tools\mlir\lib\Tools\mlir-pdll-lsp-server -IC:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\lib\Tools\mlir-pdll-lsp-server -IC:\buildbot\mlir-x64-windows-ninja\build\include -IC:\buildbot\mlir-x64-windows-ninja\llvm-project\llvm\include -IC:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\include -IC:\buildbot\mlir-x64-windows-ninja\build\tools\mlir\include /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Zc:strictStrings /Oi /Zc:rvalueCast /bigobj /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4324 -w14062 -we4238 /Gw /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++14 /showIncludes /Fotools\mlir\lib\Tools\mlir-pdll-lsp-server\CMakeFiles\MLIRPdllLspServerLib.dir\PDLLServer.cpp.obj /Fdtools\mlir\lib\Tools\mlir-pdll-lsp-server\CMakeFiles\MLIRPdllLspServerLib.dir\MLIRPdllLspServerLib.pdb /FS -c C:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\lib\Tools\mlir-pdll-lsp-server\PDLLServer.cpp
C:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\lib\Tools\mlir-pdll-lsp-server\PDLLServer.h(32): error C2039: 'string': is not a member of 'std'
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include\vector(24): note: see declaration of 'std'
C:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\lib\Tools\mlir-pdll-lsp-server\PDLLServer.h(32): error C2065: 'string': undeclared identifier
C:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\lib\Tools\mlir-pdll-lsp-server\PDLLServer.h(32): error C2923: 'std::vector': 'string' is not a valid template type argument for parameter '_Ty'
C:\buildbot\mlir-x64-windows-ninja\llvm-project\mlir\lib\Tools\mlir-pdll-lsp-server\PDLLServer.h(32): note: see declaration of 'string'

Thanks for flagging, will address. (funnily I didn't get email from Windows buildbot yet)

mlir/lib/Tools/mlir-pdll-lsp-server/MlirPdllLspServerMain.cpp

llvm::cl::opt<Logger::Level> logLevel{
           clEnumValN(Logger::Level::Debug, "verbose", "Low level details")),
       llvm::cl::init(Logger::Level::Info),
   };
   llvm::cl::opt<bool> prettyPrint{
       "pretty",
       llvm::cl::desc("Pretty-print JSON output"),
       llvm::cl::init(false),
   };
+  llvm::cl::list<std::string> extraIncludeDirs(
+      "I", llvm::cl::desc("Extra directory of include files"),
+      llvm::cl::value_desc("directory"), llvm::cl::Prefix);
+
   llvm::cl::ParseCommandLineOptions(argc, argv, "MLIR LSP Language Server");
 
   if (litTest) {
     inputStyle = JSONStreamStyle::Delimited;
     logLevel = Logger::Level::Debug;
     prettyPrint = true;
   }
 
   // Configure the logger.
   Logger::setLogLevel(logLevel);
 
   // Configure the transport used for communication.
   llvm::sys::ChangeStdinToBinary();
   JSONTransport transport(stdin, llvm::outs(), inputStyle, prettyPrint);
 
   // Configure the servers and start the main language server.
-  PDLLServer server;
+  PDLLServer::Options options(extraIncludeDirs);
+  PDLLServer server(options);
   return runPdllLSPServer(server, transport);
 }

mlir/lib/Tools/mlir-pdll-lsp-server/PDLLServer.h

 struct SignatureHelp;
 class URIForFile;
 
 /// This class implements all of the PDLL related functionality necessary for a
 /// language server. This class allows for keeping the PDLL specific logic
 /// separate from the logic that involves LSP server/client communication.
 class PDLLServer {
 public:
-  PDLLServer();
+  struct Options {
+    Options(const std::vector<std::string> &extraDirs) : extraDirs(extraDirs){};
+
+    /// Additional list of include directories to search.
+    const std::vector<std::string> &extraDirs;
+  };
+
+  PDLLServer(const Options &options);
   ~PDLLServer();
 
   /// Add or update the document, with the provided `version`, at the given URI.
   /// Any diagnostics emitted for this document should be added to
   /// `diagnostics`.
   void addOrUpdateDocument(const URIForFile &uri, StringRef contents,
                            int64_t version,
                            std::vector<Diagnostic> &diagnostics);
public:
   CompletionList getCodeCompletion(const URIForFile &uri,
                                    const Position &completePos);
 
   /// Get the signature help for the position within the given file.
   SignatureHelp getSignatureHelp(const URIForFile &uri,
                                  const Position &helpPos);
 
 private:
-  struct Impl;
+  /// PDLL LSP options.
+  const Options &options;
 
+  struct Impl;
   std::unique_ptr<Impl> impl;
 };
 
 } // namespace lsp
 } // namespace mlir
 
 #endif // LIB_MLIR_TOOLS_MLIRPDLLSPSERVER_SERVER_H_

mlir/lib/Tools/mlir-pdll-lsp-server/PDLLServer.cpp

 // PDLDocument
 //===----------------------------------------------------------------------===//
 
 namespace {
 /// This class represents all of the information pertaining to a specific PDL
 /// document.
 struct PDLDocument {
   PDLDocument(const lsp::URIForFile &uri, StringRef contents,
+              const std::vector<std::string> &extraDirs,
               std::vector<lsp::Diagnostic> &diagnostics);
   PDLDocument(const PDLDocument &) = delete;
   PDLDocument &operator=(const PDLDocument &) = delete;
 
   //===--------------------------------------------------------------------===//
   // Definitions and References
   //===--------------------------------------------------------------------===//
 
struct PDLDocument {
   FailureOr<ast::Module *> astModule;
 
   /// The index of the parsed module.
   PDLIndex index;
 };
 } // namespace
 
 PDLDocument::PDLDocument(const lsp::URIForFile &uri, StringRef contents,
+                         const std::vector<std::string> &extraDirs,
                          std::vector<lsp::Diagnostic> &diagnostics)
     : astContext(odsContext) {
   auto memBuffer = llvm::MemoryBuffer::getMemBufferCopy(contents, uri.file());
   if (!memBuffer) {
     lsp::Logger::error("Failed to create memory buffer for file", uri.file());
     return;
   }
 
   // TODO: Properly provide include directories from the client.
   llvm::SmallString<32> uriDirectory(uri.file());
   llvm::sys::path::remove_filename(uriDirectory);
   includeDirs.push_back(uriDirectory.str().str());
+  includeDirs.insert(includeDirs.end(), extraDirs.begin(), extraDirs.end());
 
   sourceMgr.setIncludeDirs(includeDirs);
   sourceMgr.AddNewSourceBuffer(std::move(memBuffer), SMLoc());
 
   astContext.getDiagEngine().setHandlerFn([&](const ast::Diagnostic &diag) {
     if (auto lspDiag = getLspDiagnoticFromDiag(sourceMgr, diag, uri))
       diagnostics.push_back(std::move(*lspDiag));
   });

 // PDLTextFileChunk
 //===----------------------------------------------------------------------===//
 
 namespace {
 /// This class represents a single chunk of an PDL text file.
 struct PDLTextFileChunk {
   PDLTextFileChunk(uint64_t lineOffset, const lsp::URIForFile &uri,
                    StringRef contents,
+                   const std::vector<std::string> &extraDirs,
                    std::vector<lsp::Diagnostic> &diagnostics)
-      : lineOffset(lineOffset), document(uri, contents, diagnostics) {}
+      : lineOffset(lineOffset),
+        document(uri, contents, extraDirs, diagnostics) {}
 
   /// Adjust the line number of the given range to anchor at the beginning of
   /// the file, instead of the beginning of this chunk.
   void adjustLocForChunkOffset(lsp::Range &range) {
     adjustLocForChunkOffset(range.start);
     adjustLocForChunkOffset(range.end);
   }
   /// Adjust the line number of the given position to anchor at the beginning of

 // PDLTextFile
 //===----------------------------------------------------------------------===//
 
 namespace {
 /// This class represents a text file containing one or more PDL documents.
 class PDLTextFile {
 public:
   PDLTextFile(const lsp::URIForFile &uri, StringRef fileContents,
-              int64_t version, std::vector<lsp::Diagnostic> &diagnostics);
+              int64_t version, const std::vector<std::string> &extraDirs,
+              std::vector<lsp::Diagnostic> &diagnostics);
 
   /// Return the current version of this text file.
   int64_t getVersion() const { return version; }
 
   //===--------------------------------------------------------------------===//
   // LSP Queries
   //===--------------------------------------------------------------------===//
 
private:
   /// The chunks of this file. The order of these chunks is the order in which
   /// they appear in the text file.
   std::vector<std::unique_ptr<PDLTextFileChunk>> chunks;
 };
 } // namespace
 
 PDLTextFile::PDLTextFile(const lsp::URIForFile &uri, StringRef fileContents,
                          int64_t version,
+                         const std::vector<std::string> &extraDirs,
                          std::vector<lsp::Diagnostic> &diagnostics)
     : contents(fileContents.str()), version(version), totalNumLines(0) {
   // Split the file into separate PDL documents.
   // TODO: Find a way to share the split file marker with other tools. We don't
   // want to use `splitAndProcessBuffer` here, but we do want to make sure this
   // marker doesn't go out of sync.
   SmallVector<StringRef, 8> subContents;
   StringRef(contents).split(subContents, "// -----");
   chunks.emplace_back(std::make_unique<PDLTextFileChunk>(
-      /*lineOffset=*/0, uri, subContents.front(), diagnostics));
+      /*lineOffset=*/0, uri, subContents.front(), extraDirs, diagnostics));
 
   uint64_t lineOffset = subContents.front().count('\n');
   for (StringRef docContents : llvm::drop_begin(subContents)) {
     unsigned currentNumDiags = diagnostics.size();
-    auto chunk = std::make_unique<PDLTextFileChunk>(lineOffset, uri,
-                                                    docContents, diagnostics);
+    auto chunk = std::make_unique<PDLTextFileChunk>(
+        lineOffset, uri, docContents, extraDirs, diagnostics);
     lineOffset += docContents.count('\n');
 
     // Adjust locations used in diagnostics to account for the offset from the
     // beginning of the file.
     for (lsp::Diagnostic &diag :
          llvm::drop_begin(diagnostics, currentNumDiags)) {
       chunk->adjustLocForChunkOffset(diag.range);
 
struct lsp::PDLLServer::Impl {
   /// The files held by the server, mapped by their URI file name.
   llvm::StringMap<std::unique_ptr<PDLTextFile>> files;
 };
 
 //===----------------------------------------------------------------------===//
 // PDLLServer
 //===----------------------------------------------------------------------===//
 
-lsp::PDLLServer::PDLLServer() : impl(std::make_unique<Impl>()) {}
+lsp::PDLLServer::PDLLServer(const Options &options)
+    : options(options), impl(std::make_unique<Impl>()) {}
 lsp::PDLLServer::~PDLLServer() = default;
 
 void lsp::PDLLServer::addOrUpdateDocument(
     const URIForFile &uri, StringRef contents, int64_t version,
     std::vector<Diagnostic> &diagnostics) {
-  impl->files[uri.file()] =
-      std::make_unique<PDLTextFile>(uri, contents, version, diagnostics);
+  impl->files[uri.file()] = std::make_unique<PDLTextFile>(
+      uri, contents, version, options.extraDirs, diagnostics);
 }
 
 Optional<int64_t> lsp::PDLLServer::removeDocument(const URIForFile &uri) {
   auto it = impl->files.find(uri.file());
   if (it == impl->files.end())
     return llvm::None;
 
   int64_t version = it->second->getVersion();