[libclang] Optionally add code completion results for arrow instead of dot

[yvvan]

Summary

Follow up for https://reviews.llvm.org/D41537 - libclang part is extracted into this review

[yvvan]

The base revision has changed - some minor changes were also required here.

[yvvan]

Output range of token being replaced instead of it's text

As a side-effect introduce a call to get a Token starting from specified location.

[yvvan]

Update according to c++ part changes

[nik]

Generates build errors here:

In file included from /d2/llvm/qtc/source/tools/clang/tools/c-index-test/c-index-test.c:4:0:
/d2/llvm/qtc/source/tools/clang/include/clang-c/Index.h:1332:62: warning: comma at end of enumerator list [-Wpedantic]
/d2/llvm/qtc/source/tools/clang/tools/c-index-test/c-index-test.c: In function 'tokens_spelling_at_range':
/d2/llvm/qtc/source/tools/clang/tools/c-index-test/c-index-test.c:2263:3: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
/d2/llvm/qtc/source/tools/clang/tools/c-index-test/c-index-test.c:2263:3: note: use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code
/d2/llvm/qtc/source/tools/clang/tools/c-index-test/c-index-test.c: In function 'print_completion_result':
/d2/llvm/qtc/source/tools/clang/tools/c-index-test/c-index-test.c:2365:3: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
At top level:
/d2/llvm/qtc/source/tools/clang/tools/c-index-test/c-index-test.c:2257:13: warning: 'tokens_spelling_at_range' defined but not used [-Wunused-function]

include/clang-c/Index.h

4759

s/tokens/token

5261

s/string/index

5269

• Please use proper documentation style, e.g. make use of "\brief", "\param", "\returns". Currently one has to somewhat guess that the returned string is related to the replacement_range parameter.

• Is this a behavior change now? What happens to old code not aware of this function?

5271

Please avoid double negation.

5273

s/editors/clients/

5274

Do you mean replacement_range instead of RemoveRange?

5289

The vec_ptr-> example should go to its own line.

5339

Currently there is not more, so I would avoid the "etc.".

test/Index/complete-arrow-dot.cpp

21

That's pretty verbose, consider to use wildcards/regexes instead just checking for "requires fix-it:".

tools/c-index-test/c-index-test.c

2479

Keep terminology of the API - s/CORRECTIONS/FIXITS

[yvvan]

Address review comments

include/clang-c/Index.h

5269

'\brief' is not used anymore.

Is this a behavior change now?

No, without CXCodeComplete_IncludeFixIts set you won't event have extra completions. And existing ones won't get any fix-its (because why would they?)

5274

Yes, it's repalcement_range here

5289

I've fixed it the same way it's done in CodeCompleteConsumer.h

5339

It might change without Index.h being updated.

[nik]

include/clang-c/Index.h

4753

"Tokenize the source code" sounds as if more than one token will be handled.

Suggestion: Reduce to "Get the raw lexical token starting with the given location".

5261

Indicate that calling this makes only sense if CXCodeComplete_IncludeFixIts was set.

5269

Still \param and \returns are useful since there the range is an output argument.

5271

Clarify that you refer to clang_codeCompleteAt here.

Suggestion: By default, clang_codeCompleteAt()....

5273

Add empty line before "For the client..." for readability.

5274

...of the cursor after applying the fixits

5282

new line

5286

new line

5288

"one of the ...", same below.

5341

Suggestion: CXCodeComplete_IncludeCompletionsWithFixIts

[nik]

• Sometimes you refer to "fixits", sometimes "fix-its" and some times "FixIts". Unify to what is already there.
• The term "completion items" is new so far. Use "completions" for consistency.

[yvvan]

Address review comments

[nik]

include/clang-c/Index.h

5302

Please document the parameters. It's more important here than for any other function that was introduced.

5338

s/completion items/completions/g

[yvvan]

Address comments about Index.h new functions parameters

[yvvan]

Diff with unix line endings

include/clang-c/Index.h

  * The version constants for the libclang API.
  * CINDEX_VERSION_MINOR should increase when there are API additions.
  * CINDEX_VERSION_MAJOR is intended for "major" source/ABI breaking changes.
  *
  * The policy about the libclang API was always to keep it source and ABI
  * compatible, thus CINDEX_VERSION_MAJOR is expected to remain stable.
  */
 #define CINDEX_VERSION_MAJOR 0
-#define CINDEX_VERSION_MINOR 48
+#define CINDEX_VERSION_MINOR 49
 
 #define CINDEX_VERSION_ENCODE(major, minor) ( \
       ((major) * 10000)                       \
     + ((minor) *     1))
 
 #define CINDEX_VERSION CINDEX_VERSION_ENCODE( \
     CINDEX_VERSION_MAJOR,                     \
     CINDEX_VERSION_MINOR )
enum CXTranslationUnit_Flags {
   CXTranslationUnit_KeepGoing = 0x200,
 
   /**
    * Sets the preprocessor in a mode for parsing a single file only.
    */
   CXTranslationUnit_SingleFileParse = 0x400,
 
   /**
-   * \brief Used in combination with CXTranslationUnit_SkipFunctionBodies to
+   * Used in combination with CXTranslationUnit_SkipFunctionBodies to
    * constrain the skipping of function bodies to the preamble.
    *
    * The function bodies of the main file are not skipped.
    */
   CXTranslationUnit_LimitSkipFunctionBodiesToPreamble = 0x800
 };
 
 /**

  * Describes a single preprocessing token.
  */
 typedef struct {
   unsigned int_data[4];
   void *ptr_data;
 } CXToken;
 
 /**
+ * Get the raw lexical token starting with the given location.
+ *
+ * \param TU the translation unit whose text is being tokenized.
+ *
+ * \param Location the source location with which the token starts.
+ *
+ * \returns The token starting with the given location or NULL if no such token
+ * exist. The returned pointer must be freed with clang_disposeTokens before the
+ * translation unit is destroyed.
+ */
+CINDEX_LINKAGE CXToken *clang_getToken(CXTranslationUnit TU,
+                                       CXSourceLocation Location);
+
+/**
  * Determine the kind of the given token.
  */
 CINDEX_LINKAGE CXTokenKind clang_getTokenKind(CXToken);
 
 /**
  * Determine the spelling of the given token.
  *
  * The spelling of a token is the textual representation of that token, e.g.,
typedef struct {
   /**
    * The number of code-completion results stored in the
    * \c Results array.
    */
   unsigned NumResults;
 } CXCodeCompleteResults;
 
 /**
+ * Retrieve the number of fix-its for the given completion index.
+ *
+ * Calling this makes sense only if CXCodeComplete_IncludeCompletionsWithFixIts
+ * option was set.
+ */
+CINDEX_LINKAGE unsigned
+clang_getCompletionNumFixIts(CXCodeCompleteResults *results,
+                             unsigned completion_index);
+
+/**
+ * Fix-its that *must* be applied before inserting the text for the
+ * corresponding completion.
+ *
+ * By default, clang_codeCompleteAt() only returns completions with empty
+ * fix-its. Extra completions with non-empty fix-its should be explicitly
+ * requested by setting CXCodeComplete_IncludeCompletionsWithFixIts.
+ *
+ * For the clients to be able to compute position of the cursor after applying
+ * fix-its, the following conditions are guaranteed to hold for
+ * replacement_range of the stored fix-its:
+ *  - Ranges in the fix-its are guaranteed to never contain the completion
+ *  point (or identifier under completion point, if any) inside them, except
+ *  at the start or at the end of the range.
+ *  - If a fix-it range starts or ends with completion point (or starts or
+ *  ends after the identifier under completion point), it will contain at
+ *  least one character. It allows to unambiguously recompute completion
+ *  point after applying the fix-it.
+ *
+ * The intuition is that provided fix-its change code around the identifier we
+ * complete, but are not allowed to touch the identifier itself or the
+ * completion point. One example of completions with corrections are the ones
+ * replacing '.' with '->' and vice versa:
+ *
+ * std::unique_ptr<std::vector<int>> vec_ptr;
+ * In 'vec_ptr.^', one of the completions is 'push_back', it requires
+ * replacing '.' with '->'.
+ * In 'vec_ptr->^', one of the completions is 'release', it requires
+ * replacing '->' with '.'.
+ */
+CINDEX_LINKAGE CXString clang_getCompletionFixIt(
+    CXCodeCompleteResults *results, unsigned completion_index,
+    unsigned fixit_index, CXSourceRange *replacement_range);
+
+/**
  * Flags that can be passed to \c clang_codeCompleteAt() to
  * modify its behavior.
  *
  * The enumerators in this enumeration can be bitwise-OR'd together to
  * provide multiple options to \c clang_codeCompleteAt().
  */
 enum CXCodeComplete_Flags {
   /**
enum CXCodeComplete_Flags {
    */
   CXCodeComplete_IncludeBriefComments = 0x04,
 
   /**
    * Whether to speed up completion by omitting top- or namespace-level entities
    * defined in the preamble. There's no guarantee any particular entity is
    * omitted. This may be useful if the headers are indexed externally.
    */
-  CXCodeComplete_SkipPreamble = 0x08
+  CXCodeComplete_SkipPreamble = 0x08,
+
+  /**
+   * Whether to include completion items with small
+   * fix-its, e.g. change '.' to '->' on member access, etc.
+   */
+  CXCodeComplete_IncludeCompletionsWithFixIts = 0x10
 };
 
 /**
  * Bits that represent the context under which completion is occurring.
  *
  * The enumerators in this enumeration may be bitwise-OR'd together if multiple
  * contexts are occurring simultaneously.
  */

include/clang/Sema/CodeCompleteConsumer.h

public:
   ResultKind Kind;
 
   /// The cursor kind that describes this result.
   CXCursorKind CursorKind;
 
   /// The availability of this result.
   CXAvailabilityKind Availability = CXAvailability_Available;
 
-  /// FixIts that *must* be applied before inserting the text for the
-  /// corresponding completion item.
+  /// Fix-its that *must* be applied before inserting the text for the
+  /// corresponding completion.
   ///
-  /// Completion items with non-empty fixits will not be returned by default,
-  /// they should be explicitly requested by setting
-  /// CompletionOptions::IncludeFixIts. For the editors to be able to
-  /// compute position of the cursor for the completion item itself, the
-  /// following conditions are guaranteed to hold for RemoveRange of the stored
-  /// fixits:
-  ///  - Ranges in the fixits are guaranteed to never contain the completion
+  /// By default, CodeCompletionBuilder only returns completions with empty
+  /// fix-its. Extra completions with non-empty fix-its should be explicitly
+  /// requested by setting CompletionOptions::IncludeFixIts.
+  ///
+  /// For the clients to be able to compute position of the cursor after
+  /// applying fix-its, the following conditions are guaranteed to hold for
+  /// RemoveRange of the stored fix-its:
+  ///  - Ranges in the fix-its are guaranteed to never contain the completion
   ///  point (or identifier under completion point, if any) inside them, except
   ///  at the start or at the end of the range.
-  ///  - If a fixit range starts or ends with completion point (or starts or
+  ///  - If a fix-it range starts or ends with completion point (or starts or
   ///  ends after the identifier under completion point), it will contain at
   ///  least one character. It allows to unambiguously recompute completion
-  ///  point after applying the fixit.
-  /// The intuition is that provided fixits change code around the identifier we
-  /// complete, but are not allowed to touch the identifier itself or the
-  /// completion point. One example of completion items with corrections are the
-  /// ones replacing '.' with '->' and vice versa:
+  ///  point after applying the fix-it.
+  ///
+  /// The intuition is that provided fix-its change code around the identifier
+  /// we complete, but are not allowed to touch the identifier itself or the
+  /// completion point. One example of completions with corrections are the ones
+  /// replacing '.' with '->' and vice versa:
+  ///
   /// std::unique_ptr<std::vector<int>> vec_ptr;
-  /// In 'vec_ptr.^', one of completion items is 'push_back', it requires
+  /// In 'vec_ptr.^', one of the completions is 'push_back', it requires
   /// replacing '.' with '->'.
-  /// In 'vec_ptr->^', one of completion items is 'release', it requires
+  /// In 'vec_ptr->^', one of the completions is 'release', it requires
   /// replacing '->' with '.'.
   std::vector<FixItHint> FixIts;
 
   /// Whether this result is hidden by another name.
   bool Hidden : 1;
 
   /// Whether this result was found via lookup into a base class.
   bool QualifierIsInformative : 1;

test/Index/complete-arrow-dot.cpp

+struct X {
+  void doSomething();
+
+  int field;
+};
+
+void X::doSomething() {
+  // RUN: c-index-test -code-completion-at=%s:10:8 %s | FileCheck %s
+  // RUN: env CINDEXTEST_COMPLETION_INCLUDE_FIXITS=1 c-index-test -code-completion-at=%s:10:8 %s | FileCheck -check-prefix=CHECK-WITH-CORRECTION %s
+  this.;
+}
+
+void doSomething() {
+  // RUN: c-index-test -code-completion-at=%s:17:6 %s | FileCheck -check-prefix=CHECK-ARROW-TO-DOT %s
+  // RUN: env CINDEXTEST_COMPLETION_INCLUDE_FIXITS=1 c-index-test -code-completion-at=%s:17:6 %s | FileCheck -check-prefix=CHECK-ARROW-TO-DOT-WITH-CORRECTION %s
+  X x;
+  x->
+}
+
+// CHECK-NOT: CXXMethod:{ResultType void}{TypedText doSomething}{LeftParen (}{RightParen )} (34) (requires fix-it:{{.*}}
+// CHECK-NOT: FieldDecl:{ResultType int}{TypedText field} (35) (requires fix-it:{{.*}}
+// CHECK-NOT: CXXMethod:{ResultType X &}{TypedText operator=}{LeftParen (}{Placeholder const X &}{RightParen )} (79) (requires fix-it:{{.*}}
+// CHECK-NOT: CXXMethod:{ResultType X &}{TypedText operator=}{LeftParen (}{Placeholder X &&}{RightParen )} (79) (requires fix-it:{{.*}}
+// CHECK-NOT: StructDecl:{TypedText X}{Text ::} (75) (requires fix-it:{{.*}}
+// CHECK-NOT: CXXDestructor:{ResultType void}{TypedText ~X}{LeftParen (}{RightParen )} (79) (requires fix-it:{{.*}}
+// CHECK: Completion contexts:
+// CHECK-NEXT: Dot member access
+
+// CHECK-WITH-CORRECTION: CXXMethod:{ResultType void}{TypedText doSomething}{LeftParen (}{RightParen )} (34) (requires fix-it:{{.*}}
+// CHECK-WITH-CORRECTION-NEXT: FieldDecl:{ResultType int}{TypedText field} (35) (requires fix-it:{{.*}}
+// CHECK-WITH-CORRECTION-NEXT: CXXMethod:{ResultType X &}{TypedText operator=}{LeftParen (}{Placeholder const X &}{RightParen )} (79) (requires fix-it:{{.*}}
+// CHECK-WITH-CORRECTION-NEXT: CXXMethod:{ResultType X &}{TypedText operator=}{LeftParen (}{Placeholder X &&}{RightParen )} (79) (requires fix-it:{{.*}}
+// CHECK-WITH-CORRECTION-NEXT: StructDecl:{TypedText X}{Text ::} (75) (requires fix-it:{{.*}}
+// CHECK-WITH-CORRECTION-NEXT: CXXDestructor:{ResultType void}{TypedText ~X}{LeftParen (}{RightParen )} (79) (requires fix-it:{{.*}}
+// CHECK-WITH-CORRECTION-NEXT: Completion contexts:
+// CHECK-WITH-CORRECTION-NEXT: Dot member access
+
+// CHECK-ARROW-TO-DOT-NOT: CXXMethod:{ResultType void}{TypedText doSomething}{LeftParen (}{RightParen )} (34) (requires fix-it:{{.*}}
+// CHECK-ARROW-TO-DOT-NOT: FieldDecl:{ResultType int}{TypedText field} (35) (requires fix-it:{{.*}}
+// CHECK-ARROW-TO-DOT-NOT: CXXMethod:{ResultType X &}{TypedText operator=}{LeftParen (}{Placeholder const X &}{RightParen )} (79) (requires fix-it:{{.*}}
+// CHECK-ARROW-TO-DOT-NOT: CXXMethod:{ResultType X &}{TypedText operator=}{LeftParen (}{Placeholder X &&}{RightParen )} (79) (requires fix-it:{{.*}}
+// CHECK-ARROW-TO-DOT-NOT: StructDecl:{TypedText X}{Text ::} (75) (requires fix-it:{{.*}}
+// CHECK-ARROW-TO-DOT-NOT: CXXDestructor:{ResultType void}{TypedText ~X}{LeftParen (}{RightParen )} (79) (requires fix-it:{{.*}}
+// CHECK-ARROW-TO-DOT: Completion contexts:
+// CHECK-ARROW-TO-DOT-NEXT: Unknown
+
+// CHECK-ARROW-TO-DOT-WITH-CORRECTION: CXXMethod:{ResultType void}{TypedText doSomething}{LeftParen (}{RightParen )} (34) (requires fix-it:{{.*}}
+// CHECK-ARROW-TO-DOT-WITH-CORRECTION-NEXT: FieldDecl:{ResultType int}{TypedText field} (35) (requires fix-it:{{.*}}
+// CHECK-ARROW-TO-DOT-WITH-CORRECTION-NEXT: CXXMethod:{ResultType X &}{TypedText operator=}{LeftParen (}{Placeholder const X &}{RightParen )} (79) (requires fix-it:{{.*}}
+// CHECK-ARROW-TO-DOT-WITH-CORRECTION-NEXT: CXXMethod:{ResultType X &}{TypedText operator=}{LeftParen (}{Placeholder X &&}{RightParen )} (79) (requires fix-it:{{.*}}
+// CHECK-ARROW-TO-DOT-WITH-CORRECTION-NEXT: StructDecl:{TypedText X}{Text ::} (75) (requires fix-it:{{.*}}
+// CHECK-ARROW-TO-DOT-WITH-CORRECTION-NEXT: CXXDestructor:{ResultType void}{TypedText ~X}{LeftParen (}{RightParen )} (79) (requires fix-it:{{.*}}
+// CHECK-ARROW-TO-DOT-WITH-CORRECTION-NEXT: Completion contexts:
+// CHECK-ARROW-TO-DOT-WITH-CORRECTION-NEXT: Arrow member access

tools/c-index-test/c-index-test.c

for (I = 0; I != N; ++I) {
     fprintf(file, "{%s %s}",
             clang_getCompletionChunkKindSpelling(Kind),
             cstr ? cstr : "");
     clang_disposeString(text);
   }
 
 }
 
-static void print_completion_result(CXCompletionResult *completion_result,
+static void print_line_column(CXSourceLocation location, FILE *file) {
+    unsigned line, column;
+    clang_getExpansionLocation(location, NULL, &line, &column, NULL);
+    fprintf(file, "%d:%d", line, column);
+}
+
+static void print_token_range(CXTranslationUnit translation_unit,
+                              CXSourceLocation start, FILE *file) {
+  CXToken *token = clang_getToken(translation_unit, start);
+
+  fprintf(file, "{");
+  if (token != NULL) {
+    CXSourceRange token_range = clang_getTokenExtent(translation_unit, *token);
+    print_line_column(clang_getRangeStart(token_range), file);
+    fprintf(file, "-");
+    print_line_column(clang_getRangeEnd(token_range), file);
+    clang_disposeTokens(translation_unit, token, 1);
+  }
+
+  fprintf(file, "}");
+}
+
+static void print_completion_result(CXTranslationUnit translation_unit,
+                                    CXCodeCompleteResults *completion_results,
+                                    unsigned index,
                                     FILE *file) {
+  CXCompletionResult *completion_result = completion_results->Results + index;
   CXString ks = clang_getCursorKindSpelling(completion_result->CursorKind);
   unsigned annotationCount;
   enum CXCursorKind ParentKind;
   CXString ParentName;
   CXString BriefComment;
   CXString Annotation;
   const char *BriefCommentCString;
   
static void print_completion_result(CXTranslationUnit translation_unit,
 
   BriefComment = clang_getCompletionBriefComment(
                                         completion_result->CompletionString);
   BriefCommentCString = clang_getCString(BriefComment);
   if (BriefCommentCString && *BriefCommentCString != '\0') {
     fprintf(file, "(brief comment: %s)", BriefCommentCString);
   }
   clang_disposeString(BriefComment);
-  
+  unsigned i;
+  for (i = 0; i < clang_getCompletionNumFixIts(completion_results, index);
+       ++i) {
+    CXSourceRange correction_range;
+    CXString FixIt = clang_getCompletionFixIt(completion_results, index, i,
+                                              &correction_range);
+    fprintf(file, " (requires fix-it: ");
+    print_token_range(translation_unit, clang_getRangeStart(correction_range),
+                      file);
+    fprintf(file, " to \"%s\")", clang_getCString(FixIt));
+    clang_disposeString(FixIt);
+  }
+
   fprintf(file, "\n");
 }
 
 void print_completion_contexts(unsigned long long contexts, FILE *file) {
   fprintf(file, "Completion contexts:\n");
   if (contexts == CXCompletionContext_Unknown) {
     fprintf(file, "Unknown\n");
   }
int perform_code_completion(int argc, const char **argv, int timing_only) {
   const char *InvocationPath;
 
   if (getenv("CINDEXTEST_CODE_COMPLETE_PATTERNS"))
     completionOptions |= CXCodeComplete_IncludeCodePatterns;
   if (getenv("CINDEXTEST_COMPLETION_BRIEF_COMMENTS"))
     completionOptions |= CXCodeComplete_IncludeBriefComments;
   if (getenv("CINDEXTEST_COMPLETION_SKIP_PREAMBLE"))
     completionOptions |= CXCodeComplete_SkipPreamble;
+  if (getenv("CINDEXTEST_COMPLETION_INCLUDE_FIXITS"))
+    completionOptions |= CXCodeComplete_IncludeCompletionsWithFixIts;
   
   if (timing_only)
     input += strlen("-code-completion-timing=");
   else
     input += strlen("-code-completion-at=");
 
   if ((errorCode = parse_file_line_column(input, &filename, &line, &column,
                                           0, 0)))
if (results) {
     enum CXCursorKind containerKind;
     CXString objCSelector;
     const char *selectorString;
     if (!timing_only) {      
       /* Sort the code-completion results based on the typed text. */
       clang_sortCodeCompletionResults(results->Results, results->NumResults);
 
       for (i = 0; i != n; ++i)
-        print_completion_result(results->Results + i, stdout);
+        print_completion_result(TU, results, i, stdout);
     }
     n = clang_codeCompleteGetNumDiagnostics(results);
     for (i = 0; i != n; ++i) {
       CXDiagnostic diag = clang_codeCompleteGetDiagnostic(results, i);
       PrintDiagnostic(diag);
       clang_disposeDiagnostic(diag);
     }
     
while (indent > 0) {
     --indent;
   }
 }
 
 static void printLocation(CXSourceLocation L) {
   CXFile File;
   CXString FileName;
   unsigned line, column, offset;
-  
   clang_getExpansionLocation(L, &File, &line, &column, &offset);
   FileName = clang_getFileName(File);
-  
   fprintf(stderr, "%s:%d:%d", clang_getCString(FileName), line, column);
   clang_disposeString(FileName);
 }
 
 static void printRanges(CXDiagnostic D, unsigned indent) {
   unsigned i, n = clang_getDiagnosticNumRanges(D);
   
   for (i = 0; i < n; ++i) {

tools/libclang/CIndex.cpp

if (Tok.isLiteral()) {
       CXTok.int_data[0] = CXToken_Punctuation;
       CXTok.ptr_data = nullptr;
     }
     CXTokens.push_back(CXTok);
     previousWasAt = Tok.is(tok::at);
   } while (Lex.getBufferLocation() < EffectiveBufferEnd);
 }
 
+CXToken *clang_getToken(CXTranslationUnit TU, CXSourceLocation Location) {
+  LOG_FUNC_SECTION {
+    *Log << TU << ' ' << Location;
+  }
+
+  if (isNotUsableTU(TU)) {
+    LOG_BAD_TU(TU);
+    return NULL;
+  }
+
+  ASTUnit *CXXUnit = cxtu::getASTUnit(TU);
+  if (!CXXUnit)
+    return NULL;
+
+  SourceLocation Begin = cxloc::translateSourceLocation(Location);
+  if (Begin.isInvalid())
+    return NULL;
+  SourceManager &SM = CXXUnit->getSourceManager();
+  std::pair<FileID, unsigned> DecomposedEnd = SM.getDecomposedLoc(Begin);
+  DecomposedEnd.second += Lexer::MeasureTokenLength(Begin, SM, CXXUnit->getLangOpts());
+
+  SourceLocation End = SM.getComposedLoc(DecomposedEnd.first, DecomposedEnd.second);
+
+  SmallVector<CXToken, 32> CXTokens;
+  getTokens(CXXUnit, SourceRange(Begin, End), CXTokens);
+
+  if (CXTokens.empty())
+    return NULL;
+
+  CXTokens.resize(1);
+  CXToken *Token = static_cast<CXToken *>(llvm::safe_malloc(sizeof(CXToken)));
+
+  memmove(Token, CXTokens.data(), sizeof(CXToken));
+  return Token;
+}
+
 void clang_tokenize(CXTranslationUnit TU, CXSourceRange Range,
                     CXToken **Tokens, unsigned *NumTokens) {
   LOG_FUNC_SECTION {
     *Log << TU << ' ' << Range;
   }
 
   if (Tokens)
     *Tokens = nullptr;

tools/libclang/CIndexCodeCompletion.cpp

 // code completion.
 //
 //===----------------------------------------------------------------------===//
 
 #include "CIndexer.h"
 #include "CIndexDiagnostic.h"
 #include "CLog.h"
 #include "CXCursor.h"
+#include "CXSourceLocation.h"
 #include "CXString.h"
 #include "CXTranslationUnit.h"
 #include "clang/AST/Decl.h"
 #include "clang/AST/DeclObjC.h"
 #include "clang/AST/Type.h"
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Frontend/ASTUnit.h"
struct AllocatedCXCodeCompleteResults : public CXCodeCompleteResults {
 
   /// a boolean value indicating whether there is complete information
   /// about the container
   unsigned ContainerIsIncomplete;
   
   /// A string containing the Objective-C selector entered thus far for a
   /// message send.
   std::string Selector;
+
+  /// Vector of fix-its for each completion result that *must* be applied
+  /// before that result for the corresponding completion item.
+  std::vector<std::vector<FixItHint>> FixItsVector;
 };
 
 } // end anonymous namespace
 
+unsigned clang_getCompletionNumFixIts(CXCodeCompleteResults *results,
+                                      unsigned completion_index) {
+  AllocatedCXCodeCompleteResults *allocated_results = (AllocatedCXCodeCompleteResults *)results;
+
+  if (!allocated_results || allocated_results->FixItsVector.size() <= completion_index)
+    return 0;
+
+  return static_cast<unsigned>(allocated_results->FixItsVector[completion_index].size());
+}
+
+CXString clang_getCompletionFixIt(CXCodeCompleteResults *results,
+                                  unsigned completion_index,
+                                  unsigned fixit_index,
+                                  CXSourceRange *replacement_range) {
+  AllocatedCXCodeCompleteResults *allocated_results = (AllocatedCXCodeCompleteResults *)results;
+
+  if (!allocated_results || allocated_results->FixItsVector.size() <= completion_index) {
+    if (replacement_range)
+      *replacement_range = clang_getNullRange();
+    return cxstring::createNull();
+  }
+
+  ArrayRef<FixItHint> FixIts = allocated_results->FixItsVector[completion_index];
+  if (FixIts.size() <= fixit_index) {
+    if (replacement_range)
+      *replacement_range = clang_getNullRange();
+    return cxstring::createNull();
+  }
+
+  const FixItHint &FixIt = FixIts[fixit_index];
+  if (replacement_range) {
+    *replacement_range = cxloc::translateSourceRange(
+        *allocated_results->SourceMgr, allocated_results->LangOpts,
+        FixIt.RemoveRange);
+  }
+
+  return cxstring::createRef(FixIt.CodeToInsert.c_str());
+}
+
 /// Tracks the number of code-completion result objects that are 
 /// currently active.
 ///
 /// Used for debugging purposes only.
 static std::atomic<unsigned> CodeCompletionResultObjects;
 
 AllocatedCXCodeCompleteResults::AllocatedCXCodeCompleteResults(
     IntrusiveRefCntPtr<FileManager> FileMgr)
CaptureCompletionResults(const CodeCompleteOptions &Opts,
         TU(TranslationUnit) { }
     ~CaptureCompletionResults() override { Finish(); }
 
     void ProcessCodeCompleteResults(Sema &S, 
                                     CodeCompletionContext Context,
                                     CodeCompletionResult *Results,
                                     unsigned NumResults) override {
       StoredResults.reserve(StoredResults.size() + NumResults);
+      if (includeFixIts())
+        AllocatedResults.FixItsVector.reserve(NumResults);
       for (unsigned I = 0; I != NumResults; ++I) {
-        CodeCompletionString *StoredCompletion        
+        CodeCompletionString *StoredCompletion
           = Results[I].CreateCodeCompletionString(S, Context, getAllocator(),
                                                   getCodeCompletionTUInfo(),
                                                   includeBriefComments());
         
         CXCompletionResult R;
         R.CursorKind = Results[I].CursorKind;
         R.CompletionString = StoredCompletion;
         StoredResults.push_back(R);
+        if (includeFixIts())
+          AllocatedResults.FixItsVector.emplace_back(std::move(Results[I].FixIts));
       }
-      
       enum CodeCompletionContext::Kind contextKind = Context.getKind();
       
       AllocatedResults.ContextKind = contextKind;
       AllocatedResults.Contexts = getContextsForContextKind(contextKind, S);
       
       AllocatedResults.Selector = "";
       ArrayRef<IdentifierInfo *> SelIdents = Context.getSelIdents();
       for (ArrayRef<IdentifierInfo *>::iterator I = SelIdents.begin(),

 
 static CXCodeCompleteResults *
 clang_codeCompleteAt_Impl(CXTranslationUnit TU, const char *complete_filename,
                           unsigned complete_line, unsigned complete_column,
                           ArrayRef<CXUnsavedFile> unsaved_files,
                           unsigned options) {
   bool IncludeBriefComments = options & CXCodeComplete_IncludeBriefComments;
   bool SkipPreamble = options & CXCodeComplete_SkipPreamble;
+  bool IncludeFixIts = options & CXCodeComplete_IncludeCompletionsWithFixIts;
 
 #ifdef UDP_CODE_COMPLETION_LOGGER
 #ifdef UDP_CODE_COMPLETION_LOGGER_PORT
   const llvm::TimeRecord &StartTime =  llvm::TimeRecord::getCurrentTime();
 #endif
 #endif
-
   bool EnableLogging = getenv("LIBCLANG_CODE_COMPLETION_LOGGING") != nullptr;
 
   if (cxtu::isNotUsableTU(TU)) {
     LOG_BAD_TU(TU);
     return nullptr;
   }
 
   ASTUnit *AST = cxtu::getASTUnit(TU);
AllocatedCXCodeCompleteResults *Results = new AllocatedCXCodeCompleteResults(
       &AST->getFileManager());
   Results->Results = nullptr;
   Results->NumResults = 0;
   
   // Create a code-completion consumer to capture the results.
   CodeCompleteOptions Opts;
   Opts.IncludeBriefComments = IncludeBriefComments;
   Opts.LoadExternal = !SkipPreamble;
+  Opts.IncludeFixIts = IncludeFixIts;
   CaptureCompletionResults Capture(Opts, *Results, &TU);
 
   // Perform completion.
   std::vector<const char *> CArgs;
   for (const auto &Arg : TU->Arguments)
     CArgs.push_back(Arg.c_str());
   std::string CompletionInvocation =
       llvm::formatv("-code-completion-at={0}:{1}:{2}", complete_filename,

 namespace {
   struct OrderCompletionResults {
     bool operator()(const CXCompletionResult &XR, 
                     const CXCompletionResult &YR) const {
       CodeCompletionString *X
         = (CodeCompletionString *)XR.CompletionString;
       CodeCompletionString *Y
         = (CodeCompletionString *)YR.CompletionString;
-      
       SmallString<256> XBuffer;
       StringRef XText = GetTypedName(X, XBuffer);
       SmallString<256> YBuffer;
       StringRef YText = GetTypedName(Y, YBuffer);
       
       if (XText.empty() || YText.empty())
         return !XText.empty();
             

tools/libclang/libclang.exports

 clang_getChildDiagnostics
 clang_getClangVersion
 clang_getCompletionAnnotation
 clang_getCompletionAvailability
 clang_getCompletionBriefComment
 clang_getCompletionChunkCompletionString
 clang_getCompletionChunkKind
 clang_getCompletionChunkText
+clang_getCompletionNumFixIts
+clang_getCompletionFixIt
 clang_getCompletionNumAnnotations
 clang_getCompletionParent
 clang_getCompletionPriority
 clang_getCursor
 clang_getCursorAvailability
 clang_getCursorCompletionString
 clang_getCursorDefinition
 clang_getCursorDisplayName

 clang_getRemappings
 clang_getRemappingsFromFileList
 clang_getResultType
 clang_getSkippedRanges
 clang_getSpecializedCursorTemplate
 clang_getSpellingLocation
 clang_getTUResourceUsageName
 clang_getTemplateCursorKind
+clang_getToken
 clang_getTokenExtent
 clang_getTokenKind
 clang_getTokenLocation
 clang_getTokenSpelling
 clang_getTranslationUnitCursor
 clang_getTranslationUnitSpelling
 clang_getTranslationUnitTargetInfo
 clang_getTypeDeclaration