Are you sure the difference is not random noise? It's not clear to me under which conditions you would encounter the same region in NewParsedRegions within the same TU. I assume this would be the case if the same file is parsed multiple times, but I'd expect this to be uncommon within the same TU.
I'd recommend to add some counters of when NewParsedRegions.count(region) is true and when ParsedRegions.count(region) is true, and print them at the end.

In D65846#1619752, @bruno wrote:

clang -fmodules -fmodules-cache-path=... is supposed to create the directory for the cache path, including the parent directories, AFAIK. If this doesn't happen it is a behavior change (and undesirable IMO).

Is c-index-test invoking clang or do we just have a similar interface? Perhaps it's not doing it right (haven't seen this problem happening directly while invoking clang). Should -fallow-pch-with-compiler-errors be considered somehow here?

In D65846#1619645, @jfb wrote:

My current guess is that this part of the test:

c-index-test -write-pch %t.h.pch %s -fmodules -fmodules-cache-path=%t.mcp -Xclang -triple -Xclang x86_64-apple-darwin

Is expected to generate the unknown type name error, but when than happens it ignores -fmodules-cache-path=%t.mcp and doesn't create the directory. The next line expects the directory to exist, which is why it can't create the lock file (because the directory it's trying to create it in doesn't exist).

Sorry, it's not clear to me what is the issue.

In D64384#1596521, @jkorous wrote:

For now I went with IndexWhileBuilding name but I'm open to discussion.

Thanks for reviewing Kristina & Aaron, much appreciated!

• Change #cmakedefine01 to #cmakedefine for consistency.
• Add comments to getLastAccessedTime()/ getLastModificationTime() APIs to make sure people are aware that time resolution can differ among file systems.
• Improve commit summary.

I think that should be documented on the public API part with a mention that the resolution is expected to differ depending on the file system and OS. WDYT?

That is a good idea, the way I consider file_status is that it provides a cross-platform way to get at the file properties. It provides times in nanoseconds but leaves the precision as implementation detail of the underlying file system, which makes sense, people should not expect anything more than that.
I'll add some comments.

In D54826#1307402, @aaron.ballman wrote:

Is it going to be an issue that the Windows side of things has a more wild idea of file timestamp resolution? NTFS has a theoretical max precision of 100ns intervals, though according to MSDN, the access time on NTFS has a resolution of 1 hour, which is better than FAT file systems, where the resolution is 1 day. It seems odd to rely on ns resolution for access time, as that seems like it's going to be highly platform and filesystem dependent.

I can't speak for the Windows side of things but what you are pointing out is a general question about applications taking into account that file properties may differ across platforms. This is orthogonal to file_status itself reporting file properties accurately.
If current file_status has 1 hour resolution on NTFS for access time, this patch is not going to make it any better or worse, nor affect behavior of applications that are querying files from NTFS.

I think this is the most important bit of information here. It should have been in the description of the differential.
If the current precision was 1ms (or even 1us), then yes, going all the way down to 1ns may not be important/worthwhile.
But 1 sec precision is clearly laughably not great, and should be improved.

Ah good point, I didn't consider that people are not broadly aware that current file_status has only 'up to second' precision.

On second look, if you are concerned with adding additional fields in basic_file_status, I can replace:

time_t fs_st_atime = 0;
time_t fs_st_mtime = 0;

with uint64_t's which are adequate to include nanosecond precision, while these time_t's are 64bits as well but waste space in the structure.

if there's no consumer for such API (pending review) there is no point in adding it.

To make sure it's clear, this is not a new API. getLastModificationTime() already reports a a timepoint in nanoseconds, but it is always in second intervals. This patch allows it to include the nanoseconds that are part of the underlying file status.

The general changes to the support libraries are usually facilitated because of missing functionality that is required somewhere else and it being the most logical place to add it. If so please attach related revisions to this one (through "Edit Related Revisions") so it's more clear why this was added

There have been enhancements to file_status that added more info without the requirement you stated (attach related revisions to this one), like:
https://reviews.llvm.org/D31110
https://reviews.llvm.org/D18456

Moved the toTimePoint conversion function to llvm/Support/Chrono.h so it can be broadly available.

LGTM!

Good enough, thanks for looking into this!

CXIndexDataConsumer.cpp uses ASTNode.OrigD, that code is exercised when you run c-index-test -index-file <...>. I'd recommend to at least check if that command would produce a different output for the test case that you detected originally.

Is it possible to add a regression test case ? I assume this is fixing some issue, we should make sure we don't regress again.

We could leave disableSourceFileDiagnostics off until someone finds a use case for it.

Thanks for the explanation. Please do add documentation comments for the new method so people using ASTUnit in their own code have an idea when and why they would need to call this. Something like "if you intend to emit additional diagnostics after the ASTUnit is created [...]".
Also consider making the naming more clear to match the intended purpose, like enableEmittingAdditionalDiagnostics() or something similar.
Otherwise LGTM.

It's not clear to me what kind of issue you are addressing, for example is the unit test hitting an assertion hit without your changes ? Or is there something else ?
Also it would be good to add some documentation comments to clarify what's the use case and when ASTUnit::beginSourceFile() should be getting called.

That would be good. How would one go about asking Clang to generate this extra information? Would a Clang Plugin be suitable for this?

That's an interesting idea that we could explore, but I don't have much experience with that mechanism to comment on.

Hey Marc,

Ah, sorry I mislead you. To test this try using c-index-test -index-file /path/to/file, see other examples in test/Index, e.g. test/Index/index-file.cpp

Could you add a test case for this change ? See examples in test/Index/Core.
Also for the test case code: template <template <typename> class Actor = actor>, is the actor reference in this code reported ?
See the test examples on how to print out and test how the source symbols are indexed.

LGTM!

Nice one! One minor change I'd suggest is to change SymbolProperty to enum class SymbolProperty : SymbolPropertySet, so that if it needs to increase we only change the type in one place.

@malaperle, to clarify we are not suggesting that you write your own parser, the suggestion is to use clang in 'fast-scan' mode to get the structure of the declarations of a single file, see CXTranslationUnit_SingleFileParse (along with enabling skipping of bodies). We have found clang is super fast when you only try to get the structure of a file like this. We can make convenient APIs to provide the syntactic structure of declarations based on their location.

To be honest, I want this functionality to get in as much as you do, and I'm more than happy to prioritize the code review for it :) But the current patch size makes the reviewing really hard (e.g. I would never have caught the BLOCK issues hadn't I tried running the original patch myself). I'm not sure if it's really a common practice to check in a big chunk of code without careful code review and leave potential improvements as followups. I'm sure @klimek would have thoughts about this.

Hey Eric,

LGTM.

CC'ed @arphaman.

FYI, I had added -target in this test in r296263 but it was still failing for some reason in some bots, that's why I then switched to triple via r296265.
I'm not sure if it's going work now or not for all bots, but it does, that is definitely better.

I'd prefer to avoid including whitespace-only changes (there are a couple of lines in the diff with only whitespace change), otherwise LGTM!

FullSourceLoc could be useful, it wraps the SourceManager that the SourceLocation come from.