[analyzer] Bump default performance thresholds?

[NoQ]

Summary

Because we now have faster CPUs and more RAM and stuff, should we now skew the balance to finding more bugs?

We could probably make a few rounds of such changes, observing any delayed feedback from users who use default settings and aren't watching phabricator, and rolling back in case we degrade dramatically on specific smaller projects.

As the first step, i've recently tested the following changes to default -analyzer-options:

• max-nodes: 150000 -> 225000 (+50%) - the limit on the size of the exploded graph.
• max-inlinable-size: 50 -> 100 (+100%) - the limit on the number of CFG blocks in inlined functions.

Totally, this gives 10% performance degradation and finds 5% more bugs on a large-ish codebase. max-inlinable-size change skews the analyzer to find more IPA-based bugs than before (+/-5% added/lost), and also overally slightly improves the number of bugs found; max-nodes increase brings back some of these positives.

Generally, it would also be good to make the analyzer work in a more obvious manner in terms of why does or doesn't it cover certain paths, inline certain functions, etc.- currently this is a mess of unobvious heuristics, and if we could make it less obvious by lifting some of these heuristics, it may be an additional benefit of this work as well.

[a.sidorin]

Hi Artem,

Could you tell what code bases did you use to collect your statistics? I'll try to check the patch on our code bases. I think we should be careful about default settings. Maybe we should introduce another UMK_* for deeper analysis instead?

[zaks.anna]

Maybe we should introduce another UMK_* for deeper analysis instead?

The difference here is not substantial enough to warrant a new level. The general motivation for bumping these numbers is that we've set the timeouts many years ago and the hardware improved quite a bit since then.

[zaks.anna]

Once Artem gives more details about the codebase he tested on, I think it would be fine to commit this. We can revert/address concerns later if @a.sidorin or anyone else raises concerns based on further testing on their codebases. @a.sidorin, would this work for you?

[NoQ]

This was an mixture of internal apple projects (user apps, drivers, deamons, whatever) with a relatively balanced selection of languages and levels of analyzer adoption. They amounted to ~16 hours of analysis CPU time (i.e. 4 hours on a quad-core machine per run). I've also ran it on LLVM separately, and had similar observations. I'm totally welcoming the feedback from other codebases!

In D34277#782605, @zaks.anna wrote:

Maybe we should introduce another UMK_* for deeper analysis instead?

The difference here is not substantial enough to warrant a new level. The general motivation for bumping these numbers is that we've set the timeouts many years ago and the hardware improved quite a bit since then.

Yeah, the point was mostly about default settings, for people who don't bother to tweak them, and adding more options essentially defeats the purpose.

[xazax.hun]

While I have no objections, I am wondering whether this is the good way to spend the performance budget. In particular, there are patches to generate more symbolic expressions, that could also degrade the performance (but fix some fixmes along the way).

[a.sidorin]

Ok, I hope this will work. Anyway, we can always revert this patch in case of any problems.

[NoQ]

Gabor makes such a good point. Maybe we should commit the zombie symbols patch as well (:

[zaks.anna]

In particular, there are patches to generate more symbolic expressions, that could also degrade the performance (but fix some fixmes along the way).

The patch you are talking about might be promising, but needs much more investigation and tuning for performance vs issues found. I do not think we should block this patch until the investigation is done.

cfe/trunk/lib/StaticAnalyzer/Core/AnalyzerOptions.cpp

if (!MaxInlinableSize.hasValue()) {
     UserModeKind HighLevelMode = getUserMode();
     switch (HighLevelMode) {
       default:
         llvm_unreachable("Invalid mode.");
       case UMK_Shallow:
         DefaultValue = 4;
         break;
       case UMK_Deep:
-        DefaultValue = 50;
+        DefaultValue = 100;
         break;
     }
 
     MaxInlinableSize = getOptionAsInteger("max-inlinable-size", DefaultValue);
   }
   return MaxInlinableSize.getValue();
 }
 
if (!MaxNodesPerTopLevelFunction.hasValue()) {
     UserModeKind HighLevelMode = getUserMode();
     switch (HighLevelMode) {
       default:
         llvm_unreachable("Invalid mode.");
       case UMK_Shallow:
         DefaultValue = 75000;
         break;
       case UMK_Deep:
-        DefaultValue = 150000;
+        DefaultValue = 225000;
         break;
     }
     MaxNodesPerTopLevelFunction = getOptionAsInteger("max-nodes", DefaultValue);
   }
   return MaxNodesPerTopLevelFunction.getValue();
 }
 
 bool AnalyzerOptions::shouldSynthesizeBodies() {

cfe/trunk/test/Analysis/analyzer-config.c

 // CHECK-NEXT: cfg-conditional-static-initializers = true
 // CHECK-NEXT: cfg-temporary-dtors = false
 // CHECK-NEXT: faux-bodies = true
 // CHECK-NEXT: graph-trim-interval = 1000
 // CHECK-NEXT: inline-lambdas = true
 // CHECK-NEXT: ipa = dynamic-bifurcate
 // CHECK-NEXT: ipa-always-inline-size = 3
 // CHECK-NEXT: leak-diagnostics-reference-allocation = false
-// CHECK-NEXT: max-inlinable-size = 50
-// CHECK-NEXT: max-nodes = 150000
+// CHECK-NEXT: max-inlinable-size = 100
+// CHECK-NEXT: max-nodes = 225000
 // CHECK-NEXT: max-times-inline-large = 32
 // CHECK-NEXT: min-cfg-size-treat-functions-as-large = 14
 // CHECK-NEXT: mode = deep
 // CHECK-NEXT: region-store-small-struct-limit = 2
 // CHECK-NEXT: widen-loops = false
 // CHECK-NEXT: [stats]
 // CHECK-NEXT: num-entries = 15
 

cfe/trunk/test/Analysis/analyzer-config.cpp

 // CHECK-NEXT: cfg-conditional-static-initializers = true
 // CHECK-NEXT: cfg-temporary-dtors = false
 // CHECK-NEXT: faux-bodies = true
 // CHECK-NEXT: graph-trim-interval = 1000
 // CHECK-NEXT: inline-lambdas = true
 // CHECK-NEXT: ipa = dynamic-bifurcate
 // CHECK-NEXT: ipa-always-inline-size = 3
 // CHECK-NEXT: leak-diagnostics-reference-allocation = false
-// CHECK-NEXT: max-inlinable-size = 50
-// CHECK-NEXT: max-nodes = 150000
+// CHECK-NEXT: max-inlinable-size = 100
+// CHECK-NEXT: max-nodes = 225000
 // CHECK-NEXT: max-times-inline-large = 32
 // CHECK-NEXT: min-cfg-size-treat-functions-as-large = 14
 // CHECK-NEXT: mode = deep
 // CHECK-NEXT: region-store-small-struct-limit = 2
 // CHECK-NEXT: widen-loops = false
 // CHECK-NEXT: [stats]
 // CHECK-NEXT: num-entries = 20