Thanks for reviewing Kristina & Aaron, much appreciated!

• Change #cmakedefine01 to #cmakedefine for consistency.
• Add comments to getLastAccessedTime()/ getLastModificationTime() APIs to make sure people are aware that time resolution can differ among file systems.
• Improve commit summary.

I think that should be documented on the public API part with a mention that the resolution is expected to differ depending on the file system and OS. WDYT?

That is a good idea, the way I consider file_status is that it provides a cross-platform way to get at the file properties. It provides times in nanoseconds but leaves the precision as implementation detail of the underlying file system, which makes sense, people should not expect anything more than that.
I'll add some comments.

In D54826#1307402, @aaron.ballman wrote:

Is it going to be an issue that the Windows side of things has a more wild idea of file timestamp resolution? NTFS has a theoretical max precision of 100ns intervals, though according to MSDN, the access time on NTFS has a resolution of 1 hour, which is better than FAT file systems, where the resolution is 1 day. It seems odd to rely on ns resolution for access time, as that seems like it's going to be highly platform and filesystem dependent.

I can't speak for the Windows side of things but what you are pointing out is a general question about applications taking into account that file properties may differ across platforms. This is orthogonal to file_status itself reporting file properties accurately.
If current file_status has 1 hour resolution on NTFS for access time, this patch is not going to make it any better or worse, nor affect behavior of applications that are querying files from NTFS.

I think this is the most important bit of information here. It should have been in the description of the differential.
If the current precision was 1ms (or even 1us), then yes, going all the way down to 1ns may not be important/worthwhile.
But 1 sec precision is clearly laughably not great, and should be improved.

Ah good point, I didn't consider that people are not broadly aware that current file_status has only 'up to second' precision.

On second look, if you are concerned with adding additional fields in basic_file_status, I can replace:

time_t fs_st_atime = 0;
time_t fs_st_mtime = 0;

with uint64_t's which are adequate to include nanosecond precision, while these time_t's are 64bits as well but waste space in the structure.

if there's no consumer for such API (pending review) there is no point in adding it.

To make sure it's clear, this is not a new API. getLastModificationTime() already reports a a timepoint in nanoseconds, but it is always in second intervals. This patch allows it to include the nanoseconds that are part of the underlying file status.

The general changes to the support libraries are usually facilitated because of missing functionality that is required somewhere else and it being the most logical place to add it. If so please attach related revisions to this one (through "Edit Related Revisions") so it's more clear why this was added

There have been enhancements to file_status that added more info without the requirement you stated (attach related revisions to this one), like:
https://reviews.llvm.org/D31110
https://reviews.llvm.org/D18456

Moved the toTimePoint conversion function to llvm/Support/Chrono.h so it can be broadly available.

LGTM!

Good enough, thanks for looking into this!

CXIndexDataConsumer.cpp uses ASTNode.OrigD, that code is exercised when you run c-index-test -index-file <...>. I'd recommend to at least check if that command would produce a different output for the test case that you detected originally.

Is it possible to add a regression test case ? I assume this is fixing some issue, we should make sure we don't regress again.

We could leave disableSourceFileDiagnostics off until someone finds a use case for it.

Thanks for the explanation. Please do add documentation comments for the new method so people using ASTUnit in their own code have an idea when and why they would need to call this. Something like "if you intend to emit additional diagnostics after the ASTUnit is created [...]".
Also consider making the naming more clear to match the intended purpose, like enableEmittingAdditionalDiagnostics() or something similar.
Otherwise LGTM.

It's not clear to me what kind of issue you are addressing, for example is the unit test hitting an assertion hit without your changes ? Or is there something else ?
Also it would be good to add some documentation comments to clarify what's the use case and when ASTUnit::beginSourceFile() should be getting called.

That would be good. How would one go about asking Clang to generate this extra information? Would a Clang Plugin be suitable for this?

That's an interesting idea that we could explore, but I don't have much experience with that mechanism to comment on.

Hey Marc,

Ah, sorry I mislead you. To test this try using c-index-test -index-file /path/to/file, see other examples in test/Index, e.g. test/Index/index-file.cpp

Could you add a test case for this change ? See examples in test/Index/Core.
Also for the test case code: template <template <typename> class Actor = actor>, is the actor reference in this code reported ?
See the test examples on how to print out and test how the source symbols are indexed.

LGTM!

Nice one! One minor change I'd suggest is to change SymbolProperty to enum class SymbolProperty : SymbolPropertySet, so that if it needs to increase we only change the type in one place.

@malaperle, to clarify we are not suggesting that you write your own parser, the suggestion is to use clang in 'fast-scan' mode to get the structure of the declarations of a single file, see CXTranslationUnit_SingleFileParse (along with enabling skipping of bodies). We have found clang is super fast when you only try to get the structure of a file like this. We can make convenient APIs to provide the syntactic structure of declarations based on their location.

To be honest, I want this functionality to get in as much as you do, and I'm more than happy to prioritize the code review for it :) But the current patch size makes the reviewing really hard (e.g. I would never have caught the BLOCK issues hadn't I tried running the original patch myself). I'm not sure if it's really a common practice to check in a big chunk of code without careful code review and leave potential improvements as followups. I'm sure @klimek would have thoughts about this.

Hey Eric,

LGTM.

CC'ed @arphaman.

FYI, I had added -target in this test in r296263 but it was still failing for some reason in some bots, that's why I then switched to triple via r296265.
I'm not sure if it's going work now or not for all bots, but it does, that is definitely better.

I'd prefer to avoid including whitespace-only changes (there are a couple of lines in the diff with only whitespace change), otherwise LGTM!

FullSourceLoc could be useful, it wraps the SourceManager that the SourceLocation come from.

Comparing SourceLocations from different translation units is not meaningful and my concern is that treating source locations like this can very easily lead to errors where by mistake the code is resolving a SourceLocation with the wrong translation unit and not the translation unit that it came from.

Update doc-comment for Preprocessor::EvaluateDirectiveExpression

Provide doc-comments for struct DirectiveEvalResult.

In D34263#783770, @voskresensky.vladimir wrote:

What I find problematic in this patch is PPOpts->SingleFileParseMode checks.
Let's suppose we implement what I mentioned above => how is it going to co-exist nicely? I think code will be less understandable with both: check of flag and call to PPCallbacks.
What I propose is to move the logic from the PPOpts single flag into PPCallbacks. And from check of flag to query of PPCallbacks.
Of course when you create PPCallbacks you can consult global SingleFileParseMode to create default implementation to answer "any symbol is defined", so you get your use-case easily handled, but it also gives others more flexibility.

There is a bit of misunderstanding on what this patch is doing, the patch is not about answering "any symbol is defined". See my cfe-dev email (http://lists.llvm.org/pipermail/cfe-dev/2017-June/054254.html) that provides some more context.
The patch is doing these 2 things:

• Keep track of whether a preprocessor directive condition used a symbol that was not resolved at all
• If above is true then make the preprocessor parse all directive blocks, not arbitrary choose one of them.

In D34263#783694, @klimek wrote:

how many patches for single file mode are coming down the road, though? I'm somewhat concerned about the overall complexity it'll add to clang.

There is no other patch for using this preprocessor option. Any related improvements down the road will be about general improvements for error recovery, for example things like this:

• Introduce an UnresolvedTypename type instead of changing unresolved types to int.
• For @interace A : B, don't completely drop B from the super-class list of A if it is unresolved.

These kind of improvements are not conditional.

Improving doc comment.

Enhanced doc-comment for the preprocessor option, and fixed indentation on a couple of calls.

Hey Vladimir, what you are proposing is orthogonal to this patch. You are proposing for "the client to provide the value for an undefined identifier", and the patch is about the client not knowing what the value should be so it fallbacks to parsing all tokens to get the max amount of info. Note that both of the techniques can be combined well, if the client provides the value, the preprocessor will take it into account, otherwise if it is stays unresolved it will fallback to lexing all tokens.
But what you are proposing is not a replacement for what the patch is doing.

I retract my comment, I see that getMainExecutable on OSX calls realpath as well, so it's good to use realpath in this code to make sure they are in-sync with how the compiler will determine the path.

Getting the real path is notoriously slow (at least it's horrible in OSX, not sure about linux). Since this is about dropping the '/../' part, could we do some simple canonicalization of removing the dots ? Not sure if there is an existing function that does that.

Also used the new Decl:: function in a place in libclang/CIndex.cpp and committed in r303484

I recommend to add a Decl::getExternalSourceSymbolAttr() to reuse this and avoid code duplication.

Committed in pieces with r298392, r300948, r300949.

Committed in r298170.

Committed in r297972.

Committed in r296282 with a couple of changes:

• Made it so that parameter definitions have a 'child' relation with their function/method (instead of 'containedBy'). This allows looking up parameters of a function by looking up their children.
• Given that parameters show up in parent-child hierarchies, I think it is a bit better to make them to a separate 'SymbolKind' instead of a sub-kind. This preserves the property that 'variable' symbol kind does not ever enter a parent-child hierarchy.

LGTM!

Committed in r295416, thanks!

Committed via r289995, thanks!

When I try just the test case (without the other changes), it doesn't fail when using r288438; so I'm not sure whether the test is actually testing the regression.

LGTM, thanks! Committed in r288438.

LGTM, thanks! Committed in r287024.

Committed in r286421, thank you!

The fix is correct, the comment for CXSourceRange says that it "Identifies a half-open character range in the source code".
But you don't need the other changes to test this. Here's an example:

LGTM.

Another recommendation for follow-up. When invoking completion on the right-hand side of the assignment it should provide a block literal completion with high priority. For example, when completing like this:

What do you think of the following possible priority heuristic

What if the user just wants to invoke the block, this is as common or more, like:

I could disable crash recovery by calling clang_toggleCrashRecovery(false) after clang_createIndex has been called but this doesn't work because the right JVM handler isn't reinstalled

LGTM!

Thanks for making the changes; I'd recommend to go ahead and commit and we can iterate post-commit if necessary.

In D17820#366638, @milianw wrote:

But, we currently always request code completion at a word start boundary so nothing would change for us. That said, I see how this patch could be seen as a breaking behavior change, and thus should probably only get enabled by an explicit option - if at all.

We should not bake-in filtering into the clang CodeComplete API. What kind of filtering to use should be on a higher level, different clients may want different filtering, e.g. prefix filtering, fuzzy filtering, etc.

Could you check if you can decouple VerifyDiagnosticConsumer from depending on a particular DiagnosticsEngine ?

I'm not sure removing all the Importer.FromDiag() calls is the right thing. You are changing

If I apply just the test changes without the rest of the changes, I don't see any failures, so I'm not sure these are effective tests.

Doesn't this mean that _all_ of the Importer.FromDiag() calls will be ignored by VerifyDiagnosticConsumer ? Why specifically change only this two and what are we going to do with the others ?
This seems more like needing a fix higher up in the stack, not at the ASTImporter level.

LGTM, committed in r261274.

Tests need to be updated, for example:

LGTM.