[VFS] vfs::directory_iterator yields path and file type instead of full Status
ClosedPublic

Authored by sammccall on Sep 11 2018, 3:34 AM.

Details

Summary

Most callers I can find are using only getName(). Type is used by the
recursive iterator.

Now we don't have to call stat() on every listed file (on most platforms).
Exceptions are e.g. Solaris where readdir() doesn't include type information.
On those platforms we'll still stat() - see D51918.

The result is significantly faster (stat() can be slow).
My motivation: this may allow us to improve clang IO on large TUs with long
include search paths. Caching readdir() results may allow us to skip many stat()
and open() operations on nonexistent files.

Diff Detail

Repository
rC Clang
sammccall created this revision.Sep 11 2018, 3:34 AM
sammccall added inline comments.Sep 11 2018, 3:41 AM
include/clang/Basic/VirtualFileSystem.h
135

Backwards-compatibility notes:

  • Almost all users of directory_iterator require no source changes (with this change)
  • Implementations of VFS require changes if they support directory iteration and do not merely wrap other VFSes. Anecdotally, most do not require changes.

So this weird API seems worth having to make out-of-tree breakages less painful.
Happy to update the internal uses though if that seems worthwhile.

bkramer added inline comments.Sep 12 2018, 10:57 AM
include/clang/Basic/VirtualFileSystem.h
135

Can we mirror llvm::sys::fs::directory_entry's interface? I want the APIs to be as close as possible. Upgrading users is not a big deal.

sammccall added inline comments.Sep 12 2018, 12:01 PM
include/clang/Basic/VirtualFileSystem.h
135

How much of the interface are you talking about? :-)

Making these private and calling the accessors file() and type() is easy of course, and consistency is nice.

Supporting status() with similar semantics+performance is both complicated and... not a good idea, I think. See the other patch where I added a comment like "this interface is wrong" and didn't fix it :-)

The other random functions on fs::directory_entry seem like random implementation cruft to me.

sammccall updated this revision to Diff 165250.Sep 13 2018, 4:32 AM

Path/getName() -> path(), Type -> type() for consistency with fs::directory_entry

sammccall added inline comments.Sep 13 2018, 4:34 AM
include/clang/Basic/VirtualFileSystem.h
135

I've done the path() and type() rename, so now we're consistent with fs::directory_entry.
And inconsistent with clang::vfs::Status, and with clang::DirectoryEntry (from FileManager).
Can't win em all!

Added inline comments regarding code style and internal/external naming.

include/clang/Basic/VirtualFileSystem.h
135

The consistency is mostly related to naming of class members or function args or locals, I think the external APIs should use whatever is most suitable, in this case path() and type() and status() would make sense style wise, I think. Just my two cents in, not insisting on any style comments this time :)

bkramer added inline comments.Sep 13 2018, 5:42 AM
include/clang/Basic/VirtualFileSystem.h
135

Not exposing random fields and having the same names as the other directory_entry is what I wanted :)

status() is deep in YAGNI territory.

(Just checking - IIUC neither of you are asking for changes, and this is waiting on review of the latest snapshot?)

Fine with me, I wasn't really suggesting a revision just making a remark more or less. Sorry if it came off that way.

This revision is now accepted and ready to land.Sep 14 2018, 1:36 AM
This revision was automatically updated to reflect the committed changes.