[CodeComplete] Fix completion in the middle of idents in macro calls

[ilya-biryukov]

Summary

This patch removes IdentifierInfo from completion token after remembering
the identifier in the preprocessor.

Prior to this patch, completion token had the IdentifierInfo set to null when
completing at the start of identifier and to the II for completion prefix
when in the middle of identifier.

This patch unifies how code completion token is handled when it is insterted
before the identifier and in the middle of the identifier.

The actual IdentifierInfo can still be obtained from the Preprocessor.

[bkramer]

I think we can go ahead with this. I expect code completion in the middle of a token not to be used much though, so there might be more broken cases.

lib/Lex/Preprocessor.cpp

case CLK_CachingLexer:
       break;
     case CLK_LexAfterModuleImport:
       LexAfterModuleImport(Result);
       ReturnedToken = true;
       break;
     }
   } while (!ReturnedToken);
 
-  if (Result.is(tok::code_completion))
+  if (Result.is(tok::code_completion) && Result.getIdentifierInfo()) {
+    // Remember the identifier before code completion token.
     setCodeCompletionIdentifierInfo(Result.getIdentifierInfo());
+    // Set IdenfitierInfo to null to avoid confusing code that handles both
+    // identifiers and completion tokens.
+    Result.setIdentifierInfo(nullptr);
+  }
 
   LastTokenWasAt = Result.is(tok::at);
 }
 
 /// \brief Lex a token following the 'import' contextual keyword.
 ///
 void Preprocessor::LexAfterModuleImport(Token &Result) {
   // Figure out what kind of lexer we actually have.

test/CodeCompletion/inside-macros.cpp

+#define ID(X) X
+
+void test(bool input_var) {
+  ID(input_var) = true;
+  // Check that input_var shows up when completing at the start, in the middle
+  // and at the end of the identifier.
+  //
+  // RUN: %clang_cc1 -fsyntax-only -code-completion-at=%s:4:6 %s -o - | FileCheck %s
+  // RUN: %clang_cc1 -fsyntax-only -code-completion-at=%s:4:8 %s -o - | FileCheck %s
+  // RUN: %clang_cc1 -fsyntax-only -code-completion-at=%s:4:15 %s -o - | FileCheck %s
+
+  // CHECK: input_var
+}