If an error happens which is related to some iterators the Iterator Modeling checker adds note tags to all the iterator operations along the bug path. This may be disturbing if there are other iterators beside the ones which are affected by the bug. This patch restricts the note tags to only the affected iterators and adjust the debug checkers to be able to test this change.
Diff Detail
Event Timeline
Its very interesting to see how the idea of interestingness propagation through NoteTags works in practice -- I feel like many other checkers will need to adopt something like this, and its great that we have the first use case presented here. The patch from afar looks good, I only left some nits to ease the readability for the uninitiated.
clang/lib/StaticAnalyzer/Checkers/DebugIteratorModeling.cpp | ||
---|---|---|
90–115 | Now that we have multiple SVals around, can we rename V? Also, I would appreciate some comments. As I understand it, ExprInspectionChecker now marks the arguments as interesting, so if we write this: clang_analyzer_express(clang_analyzer_iterator_position(i2)); clang_analyzer_iterator_position(i2) will be interesting, and this function propagates this interestingness to i2, correct? | |
103 | We won't need this anymore :) | |
clang/lib/StaticAnalyzer/Checkers/ExprInspectionChecker.cpp | ||
432 ↗ | (On Diff #249320) | Why the need for the Optional stuff here? |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
125–126 | The usage of this function is very non-obvious, plenty of comments would be appreciated here as well. |
clang/test/Analysis/iterator-modelling.cpp | ||
---|---|---|
169 ↗ | (On Diff #249320) | Interestingness won't be propagated here because clang_analyzer_iterator_container(i2) == &v is interesting, not clang_analyzer_iterator_container(i2), correct? |
clang/lib/StaticAnalyzer/Checkers/DebugIteratorModeling.cpp | ||
---|---|---|
90–115 | Yes. | |
clang/test/Analysis/iterator-modelling.cpp | ||
169 ↗ | (On Diff #249320) | Currently only clang_analyzer_express() marks its argument as interesting. This could be extended in the future, however the argument of clang_analyzer_eval() is usually the result of the comparison, not the symbolic comparison itself so the sides of the comparison are not reachable at that point. |
clang/lib/StaticAnalyzer/Checkers/DebugIteratorModeling.cpp | ||
---|---|---|
107–108 | What region would it be in the common scenario? Will it be the argument region or the region from which the value was copied into the argument region? Can we say explicitly "let's just take the argument region" because it's clearly the right thing to do? | |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
732–733 | I'm opposed to this code for the same reason that i'm opposed to it in the debug checker. Parent region is an undocumented implementation detail of RegionStore. It is supposed to be immaterial to the user. You should not rely on its exact value. @baloghadamsoftware Can we eliminate all such code from iterator checkers and instead identify all iterators by regions in which they're stored? Does my improved C++ support help with this anyhow whenever it kicks in? | |
clang/test/Analysis/iterator-modelling.cpp | ||
69 ↗ | (On Diff #250180) | Let's also add a test for a simple copy, i.e. auto j = i;. Does it say Iterator 'i' copied? What about moved? |
clang/test/Analysis/iterator-range.cpp | ||
42 | This line ultimately needs a note as well, i.e. Iterator points to end of container 'V' or something like that. |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
---|---|---|
732–733 | How to find the region where it is stored? I am open to find better solutions, but it was the only one I could find so far. If we ignore LazyCompoundVal then everything falls apart, we can remove all the iterator-related checkers. |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
---|---|---|
732–733 | It's the region from which you loaded it. If you obtained it as Call.getArgSVal() then it's the parameter region for the call. If you obtained it as Call.getReturnValue() then it's the target region can be obtained by looking at the construction context for the call. |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
---|---|---|
732–733 | LazyCompoundVal and friends seem to be a constantly emerging headache for almost everyone. For how long I've spent in the analyzer, and have religiously studied conversations and your workbook about it, I still feel anxious whenever it comes up. It would be great to have this documented in the code sometime. | |
clang/test/Analysis/iterator-modelling.cpp | ||
169 ↗ | (On Diff #249320) | Fair enough. |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
---|---|---|
732–733 | Do you mean CallEvent::getParameterLocation() for arguments? What is the construction context for the call? How can it be obtained? |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
---|---|---|
732–733 | I do not know exactly how many place LazyCompoundVal appears, but one place for sure is in MaterializeTemporaryExpr::getSubExpr(). What to use there instead? |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
---|---|---|
732–733 | I also get it in the Val parameter of checkBind(). |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
---|---|---|
732–733 | Now I spent a whole day in vain. You probably mean ExprEngine::getObjectUnderConstruction(), (which takes ConstructionContextItem as its argument) but it turned out that there are no objects under construction in checkPostCall(). (Stack dump says constructing_objects as null.) It seems that the only working solution is the current one. I am not opposed to find better working solutions, but we cannot spend months to completely rewrite parts of the analyzer for such a simple patch. And note tags are definitely needed for iterator checkers. |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
---|---|---|
732–733 | "A whole day"? "One simple patch"? Give me a break. We've been discussing this problem since your very first implementation of the iterator checker dozens of patches ago, and i spent six months of my full time work trying to make this part of the analyzer operate in an obvious, principled manner, even made a dev meeting talk about it in order to explain how it works. And all you do is keep insisting that your solution is "working" even though literally nobody understands how, even you. Out of all the contributors who bring patches to me every day, only @Szelethus is actively addressing the technical debt. This is not "one simple patch". This has to stop at some point, and i expect you, being a fairly senior contributor at this point, to put at least a slight effort into good engineering practices, apply the necessary amount of critical thinking, take basic responsibility for your code. I don't mind if you address this issue immediately after this patch if you urgently need this patch landed. But i wouldn't like this to go on forever.
You shouldn't be spending the whole day before noticing it. Whenever something isn't working, the first thing you do should be dump the ExplodedGraph and look at it. You would have noticed it right away. Was the object never there or was it removed too early? If there are no objects under construction tracked but you need them tracked, make them tracked for as long as you need. That's the whole point of the objects-under-construction map. |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
---|---|---|
732–733 | Sorry, @NoQ, I wrote this comment before starting the WIP patch. I agree that we should have clean solutions and I do not like hacking at all. Also at the University I do not accept solutions that just work by chance (i.e. pointers randomly pointing to memory where it luckily does not crash). However, unfortunately we are a profit-oriented company where our small team has tons of internal customers. Their requests should be top-priority. I already got lots of comments from my teammates and bosses that I spend too much time on a single topic (iterator checkers). If I cannot increase my performance the company may decide that these checkers remain in downstream. I am trying very hard to avoid that. The point of open-source projects is that many are contributing and get other's contributions for free. So, theoretically it should be cheaper than downstream development. However, in this project there are very few contributors. So if I have to spend 20x the time for open-sourcing every single patch than just developing a working solution downstream then open-source development turns out to be much more expensive. There is a risk that our company will not take that. This is the reason I try to balance between fully proper solutions and less proper, but fully tested and working solutions. You must understand that I am in a situation where I must do this for not losing the possibility to open-source my work. @Szelethus is in somewhat better situation as a student. The company is not so strict with students. The money we spend on students is more of a "research" investment but employees are paid for serving the customers. Unfortunately, I am not allowed to spend years for something that is not directly visible for them. So I am willing to fix this issue properly but I need your help because I must be as fast as possible. Anyway, constructing_objects is only null because we do not have LocationContext in the dumping function. |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
---|---|---|
732–733 |
Dang! I almost knew something was wrong :( Sorry, please accept my apologies :( |
clang/lib/StaticAnalyzer/Checkers/IteratorModeling.cpp | ||
---|---|---|
732–733 | No problem, I phrased a bit too harsh. Please accept my apologies! I really want to solve this problem now properly, so please follow my comments at that patch. Every day I solve one problem and face two more so the number of problems grows exponentially, just like the virus. |
We won't need this anymore :)