This is an archive of the discontinued LLVM Phabricator instance.

[Analyzer] Only add iterator note tags to the operations of the affected iterators
Needs ReviewPublic

Authored by baloghadamsoftware on Mar 5 2020, 5:09 AM.

Details

Reviewers
NoQ
Szelethus
Summary

If an error happens which is related to some iterators the Iterator Modeling checker adds note tags to all the iterator operations along the bug path. This may be disturbing if there are other iterators beside the ones which are affected by the bug. This patch restricts the note tags to only the affected iterators and adjust the debug checkers to be able to test this change.

Diff Detail

Event Timeline

Updated according to the comments to D75514.

Its very interesting to see how the idea of interestingness propagation through NoteTags works in practice -- I feel like many other checkers will need to adopt something like this, and its great that we have the first use case presented here. The patch from afar looks good, I only left some nits to ease the readability for the uninitiated.

clang/lib/StaticAnalyzer/Checkers/DebugIteratorModeling.cpp
90–115

Now that we have multiple SVals around, can we rename V? Also, I would appreciate some comments. As I understand it, ExprInspectionChecker now marks the arguments as interesting, so if we write this:

clang_analyzer_express(clang_analyzer_iterator_position(i2));

clang_analyzer_iterator_position(i2) will be interesting, and this function propagates this interestingness to i2, correct?

103

We won't need this anymore :)

clang/lib/StaticAnalyzer/Checkers/ExprInspectionChecker.cpp
432 ↗(On Diff #249320)

Why the need for the Optional stuff here?

clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
125–126

The usage of this function is very non-obvious, plenty of comments would be appreciated here as well.

Szelethus added inline comments.Mar 11 2020, 10:48 AM
clang/test/Analysis/iterator-modelling.cpp
169 ↗(On Diff #249320)

Interestingness won't be propagated here because clang_analyzer_iterator_container(i2) == &v is interesting, not clang_analyzer_iterator_container(i2), correct?

Updated according to the comments.

baloghadamsoftware marked 6 inline comments as done.Mar 13 2020, 4:57 AM
baloghadamsoftware added inline comments.
clang/lib/StaticAnalyzer/Checkers/DebugIteratorModeling.cpp
90–115

Yes.

clang/test/Analysis/iterator-modelling.cpp
169 ↗(On Diff #249320)

Currently only clang_analyzer_express() marks its argument as interesting. This could be extended in the future, however the argument of clang_analyzer_eval() is usually the result of the comparison, not the symbolic comparison itself so the sides of the comparison are not reachable at that point.

NoQ added inline comments.Mar 15 2020, 8:13 PM
clang/lib/StaticAnalyzer/Checkers/DebugIteratorModeling.cpp
107–108

What region would it be in the common scenario? Will it be the argument region or the region from which the value was copied into the argument region? Can we say explicitly "let's just take the argument region" because it's clearly the right thing to do?

clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
732–733

I'm opposed to this code for the same reason that i'm opposed to it in the debug checker. Parent region is an undocumented implementation detail of RegionStore. It is supposed to be immaterial to the user. You should not rely on its exact value.

@baloghadamsoftware Can we eliminate all such code from iterator checkers and instead identify all iterators by regions in which they're stored? Does my improved C++ support help with this anyhow whenever it kicks in?

clang/test/Analysis/iterator-modelling.cpp
69 ↗(On Diff #250180)

Let's also add a test for a simple copy, i.e. auto j = i;. Does it say Iterator 'i' copied? What about moved?

clang/test/Analysis/iterator-range.cpp
42

This line ultimately needs a note as well, i.e. Iterator points to end of container 'V' or something like that.

baloghadamsoftware marked 2 inline comments as done.Mar 17 2020, 11:13 AM
baloghadamsoftware added inline comments.
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
732–733

How to find the region where it is stored? I am open to find better solutions, but it was the only one I could find so far. If we ignore LazyCompoundVal then everything falls apart, we can remove all the iterator-related checkers.

NoQ added inline comments.Mar 23 2020, 7:27 PM
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
732–733

It's the region from which you loaded it. If you obtained it as Call.getArgSVal() then it's the parameter region for the call. If you obtained it as Call.getReturnValue() then it's the target region can be obtained by looking at the construction context for the call.

Szelethus added inline comments.Mar 24 2020, 9:01 AM
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
732–733

LazyCompoundVal and friends seem to be a constantly emerging headache for almost everyone. For how long I've spent in the analyzer, and have religiously studied conversations and your workbook about it, I still feel anxious whenever it comes up.

It would be great to have this documented in the code sometime.

clang/test/Analysis/iterator-modelling.cpp
169 ↗(On Diff #249320)

Fair enough.

baloghadamsoftware marked an inline comment as done.Mar 26 2020, 7:42 AM
baloghadamsoftware added inline comments.
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
732–733

Do you mean CallEvent::getParameterLocation() for arguments? What is the construction context for the call? How can it be obtained?

baloghadamsoftware marked an inline comment as done.Mar 26 2020, 8:20 AM
baloghadamsoftware added inline comments.
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
732–733

I do not know exactly how many place LazyCompoundVal appears, but one place for sure is in MaterializeTemporaryExpr::getSubExpr(). What to use there instead?

baloghadamsoftware marked an inline comment as done.Mar 26 2020, 8:43 AM
baloghadamsoftware added inline comments.
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
732–733

I also get it in the Val parameter of checkBind().

baloghadamsoftware marked an inline comment as done.Mar 27 2020, 6:16 AM
baloghadamsoftware added inline comments.
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
732–733

Now I spent a whole day in vain. You probably mean ExprEngine::getObjectUnderConstruction(), (which takes ConstructionContextItem as its argument) but it turned out that there are no objects under construction in checkPostCall(). (Stack dump says constructing_objects as null.) It seems that the only working solution is the current one. I am not opposed to find better working solutions, but we cannot spend months to completely rewrite parts of the analyzer for such a simple patch. And note tags are definitely needed for iterator checkers.

DenisDvlp removed a subscriber: DenisDvlp.
NoQ added inline comments.Apr 7 2020, 10:20 PM
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
732–733

"A whole day"? "One simple patch"? Give me a break.

We've been discussing this problem since your very first implementation of the iterator checker dozens of patches ago, and i spent six months of my full time work trying to make this part of the analyzer operate in an obvious, principled manner, even made a dev meeting talk about it in order to explain how it works. And all you do is keep insisting that your solution is "working" even though literally nobody understands how, even you.

Out of all the contributors who bring patches to me every day, only @Szelethus is actively addressing the technical debt. This is not "one simple patch". This has to stop at some point, and i expect you, being a fairly senior contributor at this point, to put at least a slight effort into good engineering practices, apply the necessary amount of critical thinking, take basic responsibility for your code.

I don't mind if you address this issue immediately after this patch if you urgently need this patch landed. But i wouldn't like this to go on forever.

dump says constructing_objects as null

You shouldn't be spending the whole day before noticing it. Whenever something isn't working, the first thing you do should be dump the ExplodedGraph and look at it. You would have noticed it right away.

Was the object never there or was it removed too early? If there are no objects under construction tracked but you need them tracked, make them tracked for as long as you need. That's the whole point of the objects-under-construction map.

baloghadamsoftware marked an inline comment as done.Apr 7 2020, 11:25 PM
baloghadamsoftware added inline comments.
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
732–733

Sorry, @NoQ, I wrote this comment before starting the WIP patch. I agree that we should have clean solutions and I do not like hacking at all. Also at the University I do not accept solutions that just work by chance (i.e. pointers randomly pointing to memory where it luckily does not crash).

However, unfortunately we are a profit-oriented company where our small team has tons of internal customers. Their requests should be top-priority. I already got lots of comments from my teammates and bosses that I spend too much time on a single topic (iterator checkers). If I cannot increase my performance the company may decide that these checkers remain in downstream. I am trying very hard to avoid that. The point of open-source projects is that many are contributing and get other's contributions for free. So, theoretically it should be cheaper than downstream development. However, in this project there are very few contributors. So if I have to spend 20x the time for open-sourcing every single patch than just developing a working solution downstream then open-source development turns out to be much more expensive. There is a risk that our company will not take that. This is the reason I try to balance between fully proper solutions and less proper, but fully tested and working solutions. You must understand that I am in a situation where I must do this for not losing the possibility to open-source my work.

@Szelethus is in somewhat better situation as a student. The company is not so strict with students. The money we spend on students is more of a "research" investment but employees are paid for serving the customers. Unfortunately, I am not allowed to spend years for something that is not directly visible for them. So I am willing to fix this issue properly but I need your help because I must be as fast as possible.

Anyway, constructing_objects is only null because we do not have LocationContext in the dumping function.

NoQ added inline comments.Apr 8 2020, 6:30 AM
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
732–733

I wrote this comment before starting the WIP patch

Dang! I almost knew something was wrong :( Sorry, please accept my apologies :(

baloghadamsoftware marked an inline comment as done.Apr 8 2020, 6:45 AM
baloghadamsoftware added inline comments.
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp
732–733

No problem, I phrased a bit too harsh. Please accept my apologies! I really want to solve this problem now properly, so please follow my comments at that patch. Every day I solve one problem and face two more so the number of problems grows exponentially, just like the virus.

Rebased, no tricks with LazyCompoundVals are used anymore.