This is an archive of the discontinued LLVM Phabricator instance.

[clang-tidy] add concurrency-async-no-new-threads
AbandonedPublic

Authored by segoon on Jan 13 2021, 11:11 AM.

Details

Summary

Checks for functions and types that create new system threads.

The check by default searches for types/functions from the following categories:

  • C++ std
  • Boost.Thread, part of Boost.Compute
  • C11 threads
  • POSIX threads (pthreads)
  • Linux syscalls

Custom functions and types can be added via options.

Diff Detail

Event Timeline

segoon created this revision.Jan 13 2021, 11:11 AM
segoon requested review of this revision.Jan 13 2021, 11:11 AM
Eugene.Zelenko added inline comments.
clang-tools-extra/docs/clang-tidy/checks/concurrency-async-no-new-threads.rst
7

Please make first statement same as in Release Notes.

26

Please add information about default value. Same below.

Eugene.Zelenko set the repository for this revision to rG LLVM Github Monorepo.
Eugene.Zelenko added a project: Restricted Project.
Eugene.Zelenko added a subscriber: cfe-commits.
Herald added a project: Restricted Project. · View Herald TranscriptJan 14 2021, 7:20 AM
segoon updated this revision to Diff 316664.Jan 14 2021, 8:02 AM
segoon edited the summary of this revision. (Show Details)
  • fix the first document line
  • add default values

Eugene.Zelenko, thanks for the review! fixed

Eugene.Zelenko, njames93, any comments on the patch?

Eugene.Zelenko, njames93, any comments on the patch?

Sorry, you need to wait for approval from one of maintainers mentioned as reviewers. I check only documentation and for minor code issues.

segoon updated this revision to Diff 322696.Feb 10 2021, 8:25 AM
  • explicitly state in docs that the check must be used only for async code

alexfh, aaron.ballman, hi! Any comments on the patch?

aaron.ballman added inline comments.Feb 10 2021, 11:41 AM
clang-tools-extra/clang-tidy/concurrency/AsyncNoNewThreadsCheck.cpp
23

The trailing comment markers don't really add much.

34

If we're going to add these, we should probably also add ones for Win32 and Mac OS as well, like CreateThread, CreateRemoteThread, _beginthread, _beginthreadex, etc.

clang-tools-extra/docs/clang-tidy/checks/concurrency-async-no-new-threads.rst
9–11

FWIW, this suggests to me that what you really want is a way for APIs to opt into this behavior. There's no reason why you wouldn't have a complex system that has both threads and coroutines in it, but it does stand to reason that you may want to say "this function, and everything called within this function, should not create any system threads" in some situations.

The note below helps call out the expectations from the check, but it requires the developer to restructure the way they write code pretty drastically in order to make the checking behavior more reasonable, which does not seem ideal.

segoon added inline comments.Feb 11 2021, 5:45 AM
clang-tools-extra/clang-tidy/concurrency/AsyncNoNewThreadsCheck.cpp
23

it's a hack for clang-format, otherwise it contatenates the lines, creating unmaintainable mess of strings. "One line - one name" is much more suitable.

34

I don't mind, but I'm not an expert in WinAPI or Windows programming. So, this part should be by someone with expertise of Windows in separate patches.

clang-tools-extra/docs/clang-tidy/checks/concurrency-async-no-new-threads.rst
9–11

I think it is a complex problem, so it should be separated into smaller tasks.

Step one - checks with hardcoded functions/types with user-guided enabling on a per-file basis. A semi-automated check.

Step two - try to solve other parts of the puzzle. Maybe try to add [clang:coroutine_safe] tag and teach clang static analyzer to deduce coroutine safety property and use it for enabling/disabling the cheks. Maybe reuse other (not yet implemented) heuristics from static analyzer (or other tools) to identify coroutine functions and check only these functions. I'm not an expert in static analyzer, so other LLVM developers might find a clever heuristics when to enable/disable these checks or maybe how to deduce blacklisted/whitelisted functions/types lists (e.g. for concurrency-async-{fs,blocking}).

Indeed, the current approach has its own limitations. But it may be a first step in the right direction.

aaron.ballman added inline comments.Feb 11 2021, 6:59 AM
clang-tools-extra/clang-tidy/concurrency/AsyncNoNewThreadsCheck.cpp
23

Ah, I didn't know that'd change the behavior of clang-format, that's neat!

FWIW, we usually don't do a whole lot of markup to avoid clang-format issues (such as the clang-format: on/off markers). Instead, we usually just ignore the LINT warnings in code review and check the code in as-is. This helps reduce clutter in the code base. In this case, the comments aren't adding a ton of clutter so maybe they're fine. But they definitely look odd as a reader of the code, which is a bit distracting.

34

I don't think it needs to be done in separate patches. I can give you the full list of things I care about.

CreateThread (https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createthread)
CreateRemoteThread (https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createremotethread)
CreateRemoteThreadEx (https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createremotethreadex)
_beginthread
_beginthreadex (https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/beginthread-beginthreadex?view=msvc-160)
clang-tools-extra/docs/clang-tidy/checks/concurrency-async-no-new-threads.rst
9–11

My fear with the current approach is that I don't think projects usually split their code in the way this check requires, and so the check will not be very useful in practice because it will be too chatty. Have you tried running this check over some large projects that use coroutines to see what the diagnostic results look like?

curdeius added inline comments.
clang-tools-extra/docs/clang-tidy/checks/concurrency-async-no-new-threads.rst
15

Typo: Insead -> Instead

segoon added inline comments.Feb 11 2021, 7:32 AM
clang-tools-extra/clang-tidy/concurrency/AsyncNoNewThreadsCheck.cpp
23

IMO having an ability to run clang-format for sources is very handy. It makes me sad if running clang-format against the source forces me to set some markers or reverting a part of clang-format changes in the source parts I have never touched. An ability to simply rerun checks/linters is very powerful.

34

Oh, great, thank you! Added.

clang-tools-extra/docs/clang-tidy/checks/concurrency-async-no-new-threads.rst
9–11

I haven't conducted a survey, but I can say for Yandex.Taxi where the patch has originated. The codebase is ~350KLOC. The main parts are:

  1. a coroutine framework that uses blocking functions / threads creation /etc. inside - concurrency-async-* disabled
  2. coroutine libraries consisting of coroutine-only code - concurrency-async-* enabled
  3. services with coroutine-only code - concurrency-async-* enabled

All 3 parts are clearly separated from each other from the filesystem prospective. Actually I think there are plenty of other invariants that must be met for coroutine code, so separate coroutine/non-coroutine namespaces (and consequently, filesystem separation) is a must. E.g. there are filesystem::RewriteFileContents() and filesystem::blocking::RewriteFileContents() - trivial to realize whether you may call it or not. I admit that other projects may have quite different approach to namespaces/modules, though.

segoon updated this revision to Diff 323006.Feb 11 2021, 7:33 AM
  • add WinAPI support
segoon updated this revision to Diff 323008.Feb 11 2021, 7:34 AM

-fix typo

aaron.ballman added inline comments.Feb 11 2021, 9:38 AM
clang-tools-extra/clang-tidy/concurrency/AsyncNoNewThreadsCheck.cpp
23

I empathize, but clang-format is a moving target that hasn't existed as long as the rest of the project, so the project has a mismatch of formats. This is why the usual rule of thumb is: clang-format a full file and accept what it produces (on new files, like this one) or clang-format your diff (on existing files). If you don't like what clang-format produces in either situation, we typically format it by hand and then don't worry about it until the next time someone touches that code. Of course, I notice that we don't mention clang-format anywhere in the style guide at all. :-(

I won't insist on a change here, but I also would not be surprised if the useless comment markers were removed by someone during a "cleanup" either.

41

Missing the :: in front of all of these identifiers.

clang-tools-extra/docs/clang-tidy/checks/concurrency-async-no-new-threads.rst
9–11

I admit that other projects may have quite different approach to namespaces/modules, though.

That's why I'm hoping you can take some measurements of other projects -- it wouldn't be good to add a check that's highly specific to one project's coding style (unless it's a check for a particular style guide that's published somewhere), and I worry a bit that's what is happening here.

It seems that the related file structure (IOW, "separated coroutine code and non-coroutine code" rule) is indeed specific to Yandex.Taxi. I asked multiple developers in different companies and no one could confirm they align with the rule. The checks in the current form seem to have lower value outside of Yandex.Taxi than I thought. So, I'll close PRs with the checks in the current form (unless someone tell me what part of the check(s) could be used in another way). Thanks for all who reviewed the patches and made suggestions :)

FWIW, we'll still use the checks in Yandex.Taxi. A clang static analyzer check or some other graph analysis / manual attributes marking approach would fit better for the LLVM upstream.