[clangd] Fix a selection tree crash for unmatched-bracket code.

[hokein]

Summary

Fixes https://github.com/clangd/clangd/issues/999

[sammccall]

Wow, nice find!

I'd argue the bug is probably rather in TokenBuffer::expandedTokens(SourceRange) that produced this span. The idea that it might *rarely* include the EOF token seems like a magnet for bugs like these.

(Of course it's a special case of a more general "recovery location points at an unrelated token" problem)...

Even if we want to fix it in TokenBuffer, doing it right before the release cut might be risky. So:

• fix it here
• or fix it in TokenBuffer
• or fix it here and then change after the branch cut

I'm happy with any of these options, up to you.

clang-tools-extra/clangd/Selection.cpp

test(llvm::ArrayRef<syntax::Token> ExpandedTokens) const {
     // This is a significant performance improvement when a lot of nodes
     // surround the selection, including when generated by macros.
     if (MaybeSelectedExpanded.empty() ||
         &ExpandedTokens.front() > &MaybeSelectedExpanded.back() ||
         &ExpandedTokens.back() < &MaybeSelectedExpanded.front()) {
       return SelectionTree::Unselected;
     }
 
+    // The eof token is used as a sentinel.
+    // In general, source range from an AST node should not claim the eof token,
+    // but it could occur for unmatched-bracket cases.
+    // FIXME: fix it in TokenBuffer, expandedTokens(SourceRange) should not
+    // return the eof token.
+    if (ExpandedTokens.back().kind() == tok::eof)
+      ExpandedTokens = ExpandedTokens.drop_back();
+
     SelectionTree::Selection Result = NoTokens;
     while (!ExpandedTokens.empty()) {
       // Take consecutive tokens from the same context together for efficiency.
       SourceLocation Start = ExpandedTokens.front().location();
       FileID FID = SM.getFileID(Start);
       // Comparing SourceLocations against bounds is cheaper than getFileID().
       SourceLocation Limit = SM.getComposedLoc(FID, SM.getFileIDSize(FID));
       auto Batch = ExpandedTokens.take_while([&](const syntax::Token &T) {

clang-tools-extra/clangd/unittests/SelectionTests.cpp

Case Cases[] = {
        "TypedefTypeLoc"},
 
       // lambda captured var-decl
       {R"cpp(
         void test(int bar) {
           auto l = [^[[foo = bar]]] { };
         })cpp",
        "VarDecl"},
+      {R"cpp(
+        /*error-ok*/
+        void func() [[{^]])cpp",
+       "CompoundStmt"},
   };
 
   for (const Case &C : Cases) {
     trace::TestTracer Tracer;
     Annotations Test(C.Code);
 
     TestTU TU;
     TU.Code = std::string(Test.code());