[Static Analyzer] New checker hook: checkInitialState

[xazax.hun]

Summary

Adds a new analyzer hook:

ProgramStateRef checkInitialState(const EntryPointInfo& EPInfo) /* non-const */;

This allows checkers to act on entry points, set up their initial state (by returning a new state) or prevent the analyzer from continuing from this entry point (by returning nullptr). It also serves to balance the existing checkEndFunction() and checkEndAnalysis() hooks.

EntryPointInfo is currently a very simple class containing a const Decl* of the declaration being used as an entry point and a ProgramStateRef of the initial state. It can later be extended, if we want to add more information to it.

Original discussion: http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20151123/143961.html and http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20151130/144002.html
Original-original discussion (very old, ~2 years ago, when the idea first came up): http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20131216/095565.html

Artem Dergachev has commented that a similar hook allowing to add multiple transitions using a CheckerContext might be more favourable:

At a glance, I wonder if it's worth it to provide a CheckerContext
inside this callback and then handle transitions properly (which would
allow the checker to split the program state at the very beginning of
the function). I cannot instantly think of a use-case (hmm, maybe
somebody would like to eagerly discriminate between a NULL and non-NULL
pointer argument of the function), but at the same time I don't see any
obvious problems with adding it, especially because it'd be hard to
change the API when the use-case appears.

That wasn't a use case I had in mind, but it might be a good idea. That would serve a more general function, and I'm thinking that its interface would look something like:

void checkEntryPoint(const Decl *D, CheckerContext &Context) const;

Other thoughts?

[NoQ]

Yeah, that's what i had in mind. Additionally, Decl can be obtained as Context.getStackFrame().getDecl() (and in fact the getStackFrame() thing itself is of interest as well), so there's no need to pass it as an extra argument. On the other hand, CallEvent might be of interest to have easier access to function arguments (not sure how much sense there is in a CallEvent object corresponding to the top frame, even though values inside it are pretty obvious).

[xazax.hun]

In the meantime CheckBeginFunction has been implemented separately. I think you should "abandon" this revision so it is shown as closed.

include/clang/StaticAnalyzer/Core/Checker.h

 #include "clang/Analysis/ProgramPoint.h"
 #include "clang/StaticAnalyzer/Core/CheckerManager.h"
 #include "clang/StaticAnalyzer/Core/PathSensitive/SVals.h"
 #include "llvm/Support/Casting.h"
 
 namespace clang {
 namespace ento {
   class BugReporter;
+  class EntryPointInfo;
 
 namespace check {
 
 template <typename DECL>
 class ASTDecl {
   template <typename CHECKER>
   static void _checkDecl(void *checker, const Decl *D, AnalysisManager& mgr,
                          BugReporter &BR) {

 public:
   template <typename CHECKER>
   static void _register(CHECKER *checker, CheckerManager &mgr) {
     mgr._registerListenerForEvent<EVENT>(
                  CheckerManager::CheckEventFunc(checker, _checkEvent<CHECKER>));
   }
 };
 
+class InitialState {
+  template <typename CHECKER>
+  static ProgramStateRef _checkInitialState(void *checker, const EntryPointInfo &info) {
+    return ((CHECKER *)checker)->checkInitialState(info);
+  }
+
+public:
+  template <typename CHECKER>
+  static void _register(CHECKER *checker, CheckerManager &mgr) {
+    mgr._registerForInitialState(CheckerManager::CheckInitialStateFunc(checker, _checkInitialState<CHECKER>));
+  }
+};
+
 } // end check namespace
 
 namespace eval {
 
 class Assume {
   template <typename CHECKER>
   static ProgramStateRef _evalAssume(void *checker,
                                          ProgramStateRef state,

include/clang/StaticAnalyzer/Core/CheckerManager.h

namespace ento {
   class ExplodedNode;
   class ExplodedNodeSet;
   class ExplodedGraph;
   class ProgramState;
   class NodeBuilder;
   struct NodeBuilderContext;
   class MemRegion;
   class SymbolReaper;
+  class EntryPointInfo;
 
 template <typename T> class CheckerFn;
 
 template <typename RET, typename... Ps>
 class CheckerFn<RET(Ps...)> {
   typedef RET (*Func)(void *, Ps...);
   Func Fn;
 public:
//===----------------------------------------------------------------------===//
   /// method CheckerBase::printState if it has custom data to print.
   /// \param Out The output stream
   /// \param State The state being printed
   /// \param NL The preferred representation of a newline.
   /// \param Sep The preferred separator between different kinds of data.
   void runCheckersForPrintState(raw_ostream &Out, ProgramStateRef State,
                                 const char *NL, const char *Sep);
 
+  /// \brief Run checkers for manipulating the initial program state at the
+  /// start of the analysis.
+  ///
+  /// \param D AST node of entry point
+  /// \param State Initial program state
+  /// \returns Checkers can modify the state by returning a new one.
+  ProgramStateRef runCheckersForInitialState(const Decl *D, ProgramStateRef State);
+
 //===----------------------------------------------------------------------===//
 // Internal registration functions for AST traversing.
 //===----------------------------------------------------------------------===//
 
   // Functions used by the registration mechanism, checkers should not touch
   // these directly.
 
   typedef CheckerFn<void (const Decl *, AnalysisManager&, BugReporter &)>
//===----------------------------------------------------------------------===//
   
   typedef CheckerFn<bool (const CallExpr *, CheckerContext &)>
       EvalCallFunc;
 
   typedef CheckerFn<void (const TranslationUnitDecl *,
                           AnalysisManager&, BugReporter &)>
       CheckEndOfTranslationUnit;
 
+  typedef CheckerFn<ProgramStateRef (const EntryPointInfo &)>
+      CheckInitialStateFunc;
+
   typedef bool (*HandlesStmtFunc)(const Stmt *D);
   void _registerForPreStmt(CheckStmtFunc checkfn,
                            HandlesStmtFunc isForStmtFn);
   void _registerForPostStmt(CheckStmtFunc checkfn,
                             HandlesStmtFunc isForStmtFn);
 
   void _registerForPreObjCMessage(CheckObjCMessageFunc checkfn);
   void _registerForPostObjCMessage(CheckObjCMessageFunc checkfn);
//===----------------------------------------------------------------------===//
   void _registerForConstPointerEscape(CheckPointerEscapeFunc checkfn);
 
   void _registerForEvalAssume(EvalAssumeFunc checkfn);
 
   void _registerForEvalCall(EvalCallFunc checkfn);
 
   void _registerForEndOfTranslationUnit(CheckEndOfTranslationUnit checkfn);
 
+  void _registerForInitialState(CheckInitialStateFunc checkfn);
+
 //===----------------------------------------------------------------------===//
 // Internal registration functions for events.
 //===----------------------------------------------------------------------===//
 
   typedef void *EventTag;
   typedef CheckerFn<void (const void *event)> CheckEventFunc;
 
   template <typename EVENT>
private:
   std::vector<CheckPointerEscapeFunc> PointerEscapeCheckers;
 
   std::vector<EvalAssumeFunc> EvalAssumeCheckers;
 
   std::vector<EvalCallFunc> EvalCallCheckers;
 
   std::vector<CheckEndOfTranslationUnit> EndOfTranslationUnitCheckers;
 
+  std::vector<CheckInitialStateFunc> InitialStateCheckers;
+
   struct EventInfo {
     SmallVector<CheckEventFunc, 4> Checkers;
     bool HasDispatcher;
     EventInfo() : HasDispatcher(false) { }
   };
   
   typedef llvm::DenseMap<EventTag, EventInfo> EventsTy;
   EventsTy Events;
 };
 
 } // end ento namespace
 
 } // end clang namespace
 
 #endif

include/clang/StaticAnalyzer/Core/PathSensitive/EntryPointInfo.h

+//== EntryPointInfo.h - Entry point information ------------*- C++ -*--=//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+//  This file defines EntryPointInfo.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_STATICANALYZER_CORE_PATHSENSITIVE_ENTRYPOINTINFO_H
+#define LLVM_CLANG_STATICANALYZER_CORE_PATHSENSITIVE_ENTRYPOINTINFO_H
+
+#include "clang/StaticAnalyzer/Core/PathSensitive/ProgramState_Fwd.h"
+#include "llvm/Support/Casting.h"
+
+namespace clang {
+class Decl;
+class FunctionDecl;
+
+namespace ento {
+
+class EntryPointInfo {
+  const Decl *D;
+  ProgramStateRef State;
+
+  friend class CheckerManager;
+
+public:
+  explicit EntryPointInfo(const Decl* D_, ProgramStateRef S_) : D(D_), State(S_) {}
+
+  const ProgramStateRef &getState() const { return State; }
+
+  const Decl *getDecl() const { return D; }
+
+  template <typename TDecl>
+  const TDecl *getDeclAs() const { return llvm::dyn_cast<TDecl>(D); }
+
+  const FunctionDecl *getDeclAsFunction() const {
+    return getDeclAs<FunctionDecl>();
+  }
+};
+
+} // end namespace enzo
+} // end namespace clang
+
+#endif

lib/StaticAnalyzer/Core/CheckerManager.cpp

 //===--- CheckerManager.cpp - Static Analyzer Checker Manager -------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // Defines the Static Analyzer Checker Manager.
 //
 //===----------------------------------------------------------------------===//
 
+#include "clang/StaticAnalyzer/Core/PathSensitive/EntryPointInfo.h"
 #include "clang/StaticAnalyzer/Core/CheckerManager.h"
 #include "clang/AST/DeclBase.h"
 #include "clang/Analysis/ProgramPoint.h"
 #include "clang/StaticAnalyzer/Core/Checker.h"
 #include "clang/StaticAnalyzer/Core/PathSensitive/CallEvent.h"
 #include "clang/StaticAnalyzer/Core/PathSensitive/CheckerContext.h"
 
 using namespace clang;
return !StmtCheckers.empty()              ||
          !BindCheckers.empty()              ||
          !EndAnalysisCheckers.empty()       ||
          !EndFunctionCheckers.empty()           ||
          !BranchConditionCheckers.empty()   ||
          !LiveSymbolsCheckers.empty()       ||
          !DeadSymbolsCheckers.empty()       ||
          !RegionChangesCheckers.empty()     ||
          !EvalAssumeCheckers.empty()        ||
-         !EvalCallCheckers.empty();
+         !EvalCallCheckers.empty()               ||
+         !InitialStateCheckers.empty();
 }
 
 void CheckerManager::finishedCheckerRegistration() {
 #ifndef NDEBUG
   // Make sure that for every event that has listeners, there is at least
   // one dispatcher registered for it.
   for (llvm::DenseMap<EventTag, EventInfo>::iterator
          I = Events.begin(), E = Events.end(); I != E; ++I)

 void CheckerManager::runCheckersForPrintState(raw_ostream &Out,
                                               ProgramStateRef State,
                                               const char *NL, const char *Sep) {
   for (llvm::DenseMap<CheckerTag, CheckerRef>::iterator
         I = CheckerTags.begin(), E = CheckerTags.end(); I != E; ++I)
     I->second->printState(Out, State, NL, Sep);
 }
 
+ProgramStateRef
+CheckerManager::runCheckersForInitialState(const Decl *D, ProgramStateRef State) {
+  EntryPointInfo EntryInfo(D, State);
+
+  for (unsigned i = 0, e = InitialStateCheckers.size(); i != e; ++i) {
+    if (!EntryInfo.State) return nullptr; // exit early in case of unfeasible state [TODO: revise]
+    EntryInfo.State = InitialStateCheckers[i](EntryInfo);
+  }
+
+  return EntryInfo.State;
+}
+
 //===----------------------------------------------------------------------===//
 // Internal registration functions for AST traversing.
 //===----------------------------------------------------------------------===//
 
 void CheckerManager::_registerForDecl(CheckDeclFunc checkfn,
                                       HandlesDeclFunc isForDeclFn) {
   DeclCheckerInfo info = { checkfn, isForDeclFn };
   DeclCheckers.push_back(info);
void CheckerManager::_registerForEvalCall(EvalCallFunc checkfn) {
   EvalCallCheckers.push_back(checkfn);
 }
 
 void CheckerManager::_registerForEndOfTranslationUnit(
                                             CheckEndOfTranslationUnit checkfn) {
   EndOfTranslationUnitCheckers.push_back(checkfn);
 }
 
+void CheckerManager::_registerForInitialState(CheckInitialStateFunc checkfn) {
+  InitialStateCheckers.push_back(checkfn);
+}
+
 //===----------------------------------------------------------------------===//
 // Implementation details.
 //===----------------------------------------------------------------------===//
 
 const CheckerManager::CachedStmtCheckers &
 CheckerManager::getCachedStmtCheckersFor(const Stmt *S, bool isPreVisit) {
   assert(S);
 

lib/StaticAnalyzer/Core/CoreEngine.cpp

if (G.num_roots() == 0) { // Initialize the analysis by constructing
 
     // Construct an edge representing the
     // starting location in the function.
     BlockEdge StartLoc(Entry, Succ, L);
 
     // Set the current block counter to being empty.
     WList->setBlockCounter(BCounterFactory.GetEmptyCounter());
 
-    if (!InitState)
-      // Generate the root.
-      generateNode(StartLoc, SubEng.getInitialState(L), nullptr);
-    else
-      generateNode(StartLoc, InitState, nullptr);
+    if (!InitState) {
+      InitState = SubEng.getInitialState(L);
+
+      // it's possible for a checker to return NULL state, in which case getInitialState()
+      // above will return NULL - this means that we don't want this code path checked;
+      // return early
+      // TODO: correct? (seems to work)
+      if (!InitState) {
+        SubEng.processEndWorklist(hasWorkRemaining());
+        return WList->hasWork();
+      }
+    }
+
   }
 
   // Check if we have a steps limit
   bool UnlimitedSteps = Steps == 0;
 
   while (WList->hasWork()) {
     if (!UnlimitedSteps) {
       if (Steps == 0) {

lib/StaticAnalyzer/Core/ExprEngine.cpp

if (!MD->isStatic()) {
         if (Optional<Loc> LV = V.getAs<Loc>()) {
           state = state->assume(*LV, true);
           assert(state && "'this' cannot be null");
         }
       }
     }
   }
 
-  return state;
+  return getCheckerManager().runCheckersForInitialState(D, state);
 }
 
 ProgramStateRef
 ExprEngine::createTemporaryRegionIfNeeded(ProgramStateRef State,
                                           const LocationContext *LC,
                                           const Expr *Ex,
                                           const Expr *Result) {
   SVal V = State->getSVal(Ex, LC);