libclang: Avoid returning an extra token in clang_tokenize

[emilio]

Summary

This is a bug that has hit me, and apparently more people (see [1]). There's a
bug open on this since 2011[2] with a patch attached, but got no attention, so I
rebased the patch and added a test for it.

[1]: https://github.com/servo/rust-bindgen/pull/219#discussion_r86691290
[2]: https://llvm.org/bugs/show_bug.cgi?id=9069

[emilio]

Guy, I added you as a reviewer looking at the blame log, feel free to redirect since it's my first patch to LLVM. Thanks!

[akyrtzi]

The fix is correct, the comment for CXSourceRange says that it "Identifies a half-open character range in the source code".
But you don't need the other changes to test this. Here's an example:

$ cat /tmp/t.c
int gv = 0;
$ c-index-test -test-annotate-tokens=/tmp/t.c:1:10:1:11 /tmp/t.c
Literal: "0" [1:10 - 1:11] IntegerLiteral=
Punctuation: ";" [1:11 - 1:12]

All you have to do is a similar invocation with a character range for a token, CHECK that the token shows up and CHECK-NOT that the following token does not show up.
And add this to an existing test file that tests -test-annotate-tokens no need to add an additional test file.

[emilio]

I thought the extra test would be helpful since I'm not aware of any test that used clang_getCursorExtent + clang_tokenize, but I guess I can revert that, no problem. Thanks for the review! :)

[emilio]

Updated per comments, let me know if I can do anything else, thanks!

[akyrtzi]

Committed in r286421, thank you!

[emilio]

Thanks for commiting the patch @akyrtzi (and for fixing up the test path, I kind of suspected there had to be a better place for it but didn't look too close apparently :P).

Thanks again!

[skalinichev]

This patch breaks python binding tests (which I fixes just now).
Please, next time launch python tests too

[emilio]

Sorry @skalinichev, didn't know about those. Thanks for fixing.

test/Index/targeted-annotation.c

 // TOP: Keyword: "int" [11:8 - 11:11] VarDecl=TopVar:11:12
 // TOP: Identifier: "TopVar" [11:12 - 11:18] VarDecl=TopVar:11:12
 // TOP: Punctuation: ";" [11:18 - 11:19]
+
+// RUN: env CINDEXTEST_FAILONERROR=1 c-index-test -test-annotate-tokens=%S/targeted-top.h:11:12:11:18 %s -include %t.h \
+// RUN:  | FileCheck %s -check-prefix=CHECK
+
+// CHECK: Identifier: "TopVar" [11:12 - 11:18] VarDecl=TopVar:11:12
+// CHECK-NOT: Punctuation: ";" [11:18 - 11:19]

tools/libclang/CIndex.cpp

     }
     CXTokens.push_back(CXTok);
     previousWasAt = Tok.is(tok::at);
-  } while (Lex.getBufferLocation() <= EffectiveBufferEnd);
+  } while (Lex.getBufferLocation() < EffectiveBufferEnd);
 }
 
 void clang_tokenize(CXTranslationUnit TU, CXSourceRange Range,