[clangd] Update TUStatus test to handle async PreambleThread

[kadircet]

Summary

Currently it doesn't matter because we run PreambleThread in sync mode.
Once we start running it in async, test order will become non-deterministic.

[sammccall]

clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp

866

This seems unfortunate because it doesn't test the production configuration.

How many possible sequences are there? Are there fewer if we blockuntilidle between adddoc + locatesymbolat?

Can we use testing::AnyOf to fudge around the nondeterminism?

894

FWIW, I found the comments useful, and don't understand e.g. why we have two identical lines anymore.

897

why do we never go idle at the end?

[kadircet]

• Assert on the execution order instead with simplifications.

[kadircet]

clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp

866

I didn't want to do that as it was rather asserting on action ordering of preamble and ast worker threads rather than checking for whether we emit TUStatuses, but I suppose this test was always trying to assert both.

Introducing some more simplications by turning of diagnostics and also making use of the assumption that ASTWorker will block for first preamble build.

897

because astworker doesn't emit queue related notifications(running action, queued, idle) in RunSync mode.

[sammccall]

Thanks! Now having read it, I'm slightly nervous we're not catching all the interleavings.
An alternate approach if this is too messy would be just to capture the two streams of enums into separate vectors, drop duplicates, and assert the sequence of each.
Anyway, up to you what to do here, I'm happy if we're testing at least something in the async configuration.

clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp

866

This makes sense, at the risk of being a massive pain do you think we should have a separate test that (only) creates diagnostics?

912

haha, including the (idle,runningaction) case twice is fun :-) Maybe not necessary though

918

Hmm... are you sure the preamble can't still be running at this point? What stops it from pausing forever before exiting?

[kadircet]

• Record actions for each thread separately.

[kadircet]

going with separately recording each thread. that way we can also test building of diagnostics.

clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp

866

changed the test to record each thread separately while deduplicating, so we can also build the diags now.

918

it has to go idle before stopping, since we block after issuing the update.

clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp

 //===-- TUSchedulerTests.cpp ------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "Annotations.h"
 #include "Cancellation.h"
+#include "ClangdServer.h"
 #include "Context.h"
 #include "Diagnostics.h"
 #include "Matchers.h"
 #include "ParsedAST.h"
 #include "Path.h"
 #include "Preamble.h"
 #include "TUScheduler.h"
 #include "TestFS.h"
TEST_F(TUSchedulerTests, Run) {
   });
   TaskRun.wait();
 }
 
 TEST_F(TUSchedulerTests, TUStatus) {
   class CaptureTUStatus : public ClangdServer::Callbacks {
   public:
     void onFileUpdated(PathRef File, const TUStatus &Status) override {
+      auto ASTAction = Status.ASTActivity.K;
+      auto PreambleAction = Status.PreambleActivity;
       std::lock_guard<std::mutex> Lock(Mutex);
-      AllStatus.push_back(Status);
+      // Only push the action if it has changed. Since TUStatus can be published
+      // from either Preamble or AST thread and when one changes the other stays
+      // the same.
+      // Note that this can result in missing some updates when something other
+      // than action kind changes, e.g. when AST is built/reused the action kind
+      // stays as Building.
+      if (ASTActions.empty() || ASTActions.back() != ASTAction)
+        ASTActions.push_back(ASTAction);
+      if (PreambleActions.empty() || PreambleActions.back() != PreambleAction)
+        PreambleActions.push_back(PreambleAction);
     }
 
-    std::vector<TUStatus> allStatus() {
+    std::vector<PreambleAction> preambleStatuses() {
       std::lock_guard<std::mutex> Lock(Mutex);
-      return AllStatus;
+      return PreambleActions;
+    }
+
+    std::vector<ASTAction::Kind> astStatuses() {
+      std::lock_guard<std::mutex> Lock(Mutex);
+      return ASTActions;
     }
 
   private:
     std::mutex Mutex;
-    std::vector<TUStatus> AllStatus;
+    std::vector<ASTAction::Kind> ASTActions;
+    std::vector<PreambleAction> PreambleActions;
   } CaptureTUStatus;
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   ClangdServer Server(CDB, FS, ClangdServer::optsForTest(), &CaptureTUStatus);
   Annotations Code("int m^ain () {}");
 
   // We schedule the following tasks in the queue:
   //   [Update] [GoToDefinition]
   Server.addDocument(testPath("foo.cpp"), Code.code(), "1",
-                     WantDiagnostics::Yes);
+                     WantDiagnostics::Auto);
+  ASSERT_TRUE(Server.blockUntilIdleForTest());
   Server.locateSymbolAt(testPath("foo.cpp"), Code.point(),
                         [](Expected<std::vector<LocatedSymbol>> Result) {
                           ASSERT_TRUE((bool)Result);
                         });
-
   ASSERT_TRUE(Server.blockUntilIdleForTest());
 
-  EXPECT_THAT(CaptureTUStatus.allStatus(),
+  EXPECT_THAT(CaptureTUStatus.preambleStatuses(),
+              ElementsAre(
+                  // PreambleThread starts idle, as the update is first handled
+                  // by ASTWorker.
+                  PreambleAction::Idle,
+                  // Then it starts building first preamble and releases that to
+                  // ASTWorker.
+                  PreambleAction::Building,
+                  // Then goes idle and stays that way as we don't receive any
+                  // more update requests.
+                  PreambleAction::Idle));
+  EXPECT_THAT(CaptureTUStatus.astStatuses(),
               ElementsAre(
-                  // Everything starts with ASTWorker starting to execute an
-                  // update
-                  TUState(PreambleAction::Idle, ASTAction::RunningAction),
-                  // We build the preamble
-                  TUState(PreambleAction::Building, ASTAction::RunningAction),
-                  // We built the preamble, and issued ast built on ASTWorker
-                  // thread. Preambleworker goes idle afterwards.
-                  TUState(PreambleAction::Idle, ASTAction::RunningAction),
-                  // Start task for building the ast, as a result of building
-                  // preamble, on astworker thread.
-                  TUState(PreambleAction::Idle, ASTAction::RunningAction),
-                  // AST build starts.
-                  TUState(PreambleAction::Idle, ASTAction::Building),
-                  // AST built finished successfully
-                  TUState(PreambleAction::Idle, ASTAction::Building),
-                  // Running go to def
-                  TUState(PreambleAction::Idle, ASTAction::RunningAction),
-                  // ASTWorker goes idle.
-                  TUState(PreambleAction::Idle, ASTAction::Idle)));
+                  // Starts handling the update action and blocks until the
+                  // first preamble is built.
+                  ASTAction::RunningAction,
+                  // Afterwqards it builds an AST for that preamble to publish
+                  // diagnostics.
+                  ASTAction::Building,
+                  // Then goes idle.
+                  ASTAction::Idle,
+                  // Afterwards we start executing go-to-def.
+                  ASTAction::RunningAction,
+                  // Then go idle.
+                  ASTAction::Idle));
 }
 
 TEST_F(TUSchedulerTests, CommandLineErrors) {
   // We should see errors from command-line parsing inside the main file.
   CDB.ExtraClangFlags = {"-fsome-unknown-flag"};
 
   // (!) 'Ready' must live longer than TUScheduler.
   Notification Ready;
 
   TUScheduler S(CDB, optsForTest(), captureDiags());
   std::vector<Diag> Diagnostics;
   updateWithDiags(S, testPath("foo.cpp"), "void test() {}",
                   WantDiagnostics::Yes, [&](std::vector<Diag> D) {
                     Diagnostics = std::move(D);
                     Ready.notify();
                   });
   Ready.wait();
 
   EXPECT_THAT(
       Diagnostics,
       ElementsAre(AllOf(
           Field(&Diag::ID, Eq(diag::err_drv_unknown_argument)),
           Field(&Diag::Name, Eq("drv_unknown_argument")),