[analyzer] Improve valist check

[xazax.hun]

Summary

This patch makes the valist check more robust to the different kind of ASTs that are generated on different platforms:

Generated on x86_64-pc-linux-gnu:

|-TypedefDecl 0x1d07d08 <<invalid sloc>> <invalid sloc> implicit referenced __builtin_va_list 'struct __va_list_tag [1]'
| `-ConstantArrayType 0x1d07cb0 'struct __va_list_tag [1]' 1 
|   `-RecordType 0x1d07b20 'struct __va_list_tag'
|     `-Record 0x1d07a90 '__va_list_tag'

Generated on hexagon-unknown-linux:

-TypedefDecl 0x6c47e0 <<invalid sloc>> <invalid sloc> implicit referenced __builtin_va_list 'char *'
| `-PointerType 0x6c47a0 'char *'
|   `-BuiltinType 0x6c4020 'char'

This patch also manages to fix one of the FIXMEs in the tests.

Note that, some tests are only there for x86_64-pc-linux-gnu. The reason is that, the analyzer manages to notice the uninitializedness of va_list on hexagon-unknown-linux, so it generates a sink node before this check could be triggered. Also this patch moves this check out of alpha stage.

[NoQ]

lib/StaticAnalyzer/Checkers/ValistChecker.cpp

178

I suspect that UnknownVal should also be handled before that, otherwise we'd have null dereference on the next line.

test/Analysis/valist-uninitialized-no-undef.c

6

Hmm, where's the previous one?

20

As the patch tries to handle symbolic va_list regions, i wonder what's so particularly hard about this false positive (apart from its being obviously rare, by the way did you actually see such code?).

[xazax.hun]

• Address some review comments.
• Add some additional tests.
• Fixed some false positives (checking for symbolic values for va_copy and more robust detection of which valist model is used by the platform)
• I have run the checker on https://github.com/rathena/rathena, there were no false positives or crashes using the current state. It is a 170KLOC C project.

[xazax.hun]

lib/StaticAnalyzer/Checkers/ValistChecker.cpp

178

Indeed.

test/Analysis/valist-uninitialized-no-undef.c

6

Tha calling function is after this one.

20

What is strange, this case does work with the hexagon AST variant.

[xazax.hun]

• Fixed a comment.

[xazax.hun]

test/Analysis/valist-uninitialized-no-undef.c

20

Also, I did not see such code inproduction yet

[danielmarjamaki]

lib/StaticAnalyzer/Checkers/ValistChecker.cpp

189

I would personally recommend parentheses around EReg and VaListModelledAsArray to highlight the precedence.

[xazax.hun]

• Minor style improvement.

[xazax.hun]

• Move the check out of alpha.

[danielmarjamaki]

I am running this checker right now on various projects. Here are currently seen results.. https://drive.google.com/open?id=0BykPmWrCOxt2STZMOXZ5OGlwM3c

Feel free to look at it and see if there are FPs or TPs.

[zaks.anna]

test/Analysis/valist-uninitialized-no-undef.c

26

Please, split the long "expected" lines into multiple lines - one per note. It will improve readability in non-wrapping editors. Thanks!

[xazax.hun]

• Improve the readability of test files.
• Rebased on latest ToT.

[xazax.hun]

In D30157#689374, @danielmarjamaki wrote:

I am running this checker right now on various projects. Here are currently seen results.. https://drive.google.com/open?id=0BykPmWrCOxt2STZMOXZ5OGlwM3c

Feel free to look at it and see if there are FPs or TPs.

Thank you for running this check on those projects! I did not do a deep review of the findings but some of the results I checked are true positives. Also, the number of results are not scary, none of the projects generated tons of results.

[NoQ]

It's great to see things move out of alpha and finally become actually accessible to the majority of users. Thanks!

lib/StaticAnalyzer/Checkers/ValistChecker.cpp

public:
   DefaultBool ChecksEnabled[CK_NumCheckKinds];
   CheckName CheckNames[CK_NumCheckKinds];
 
   void checkPreStmt(const VAArgExpr *VAA, CheckerContext &C) const;
   void checkPreCall(const CallEvent &Call, CheckerContext &C) const;
   void checkDeadSymbols(SymbolReaper &SR, CheckerContext &C) const;
 
 private:
-  const MemRegion *getVAListAsRegion(SVal SV, CheckerContext &C) const;
+  const MemRegion *getVAListAsRegion(SVal SV, const Expr *VAExpr,
+                                     bool &IsSymbolic, CheckerContext &C) const;
   StringRef getVariableNameFromRegion(const MemRegion *Reg) const;
   const ExplodedNode *getStartCallSite(const ExplodedNode *N,
-                                       const MemRegion *Reg,
-                                       CheckerContext &C) const;
+                                       const MemRegion *Reg) const;
 
   void reportUninitializedAccess(const MemRegion *VAList, StringRef Msg,
                                  CheckerContext &C) const;
   void reportLeakedVALists(const RegionVector &LeakedVALists, StringRef Msg1,
                            StringRef Msg2, CheckerContext &C, ExplodedNode *N,
                            bool ForceReport = false) const;
 
   void checkVAListStartCall(const CallEvent &Call, CheckerContext &C,
void ValistChecker::checkPreCall(const CallEvent &Call,
   else if (Call.isCalled(VaCopy))
     checkVAListStartCall(Call, C, true);
   else if (Call.isCalled(VaEnd))
     checkVAListEndCall(Call, C);
   else {
     for (auto FuncInfo : VAListAccepters) {
       if (!Call.isCalled(FuncInfo.Func))
         continue;
+      bool Symbolic;
       const MemRegion *VAList =
-          getVAListAsRegion(Call.getArgSVal(FuncInfo.VAListPos), C);
+          getVAListAsRegion(Call.getArgSVal(FuncInfo.VAListPos),
+                            Call.getArgExpr(FuncInfo.VAListPos), Symbolic, C);
       if (!VAList)
         return;
 
       if (C.getState()->contains<InitializedVALists>(VAList))
         return;
 
+      // We did not see va_start call, but the source of the region is unknown.
+      // Be conservative and assume the best.
+      if (Symbolic)
+        return;
+
       SmallString<80> Errmsg("Function '");
       Errmsg += FuncInfo.Func.getFunctionName();
       Errmsg += "' is called with an uninitialized va_list argument";
       reportUninitializedAccess(VAList, Errmsg.c_str(), C);
       break;
     }
   }
 }
 
+const MemRegion *ValistChecker::getVAListAsRegion(SVal SV, const Expr *E,
+                                                  bool &IsSymbolic,
+                                                  CheckerContext &C) const {
+  // FIXME: on some platforms CallAndMessage checker finds some instances of
+  // the uninitialized va_list usages. CallAndMessage checker is disabled in
+  // the tests so they can verify platform independently those issues. As a
+  // side effect, this check is required here.
+  if (SV.isUnknownOrUndef())
+    return nullptr;
+  // TODO: In the future this should be abstracted away by the analyzer.
+  bool VaListModelledAsArray = false;
+  if (const auto *Cast = dyn_cast<CastExpr>(E)) {
+    QualType Ty = Cast->getType();
+    VaListModelledAsArray =
+        Ty->isPointerType() && Ty->getPointeeType()->isRecordType();
+  }
+  const MemRegion *Reg = SV.getAsRegion();
+  if (const auto *DeclReg = Reg->getAs<DeclRegion>()) {
+    if (isa<ParmVarDecl>(DeclReg->getDecl()))
+      Reg = C.getState()->getSVal(SV.castAs<Loc>()).getAsRegion();
+  }
+  IsSymbolic = Reg && Reg->getAs<SymbolicRegion>();
+  // Some VarRegion based VA lists reach here as ElementRegions.
+  const auto *EReg = dyn_cast_or_null<ElementRegion>(Reg);
+  return EReg && VaListModelledAsArray ? EReg->getSuperRegion() : Reg;
+}
+
 void ValistChecker::checkPreStmt(const VAArgExpr *VAA,
                                  CheckerContext &C) const {
   ProgramStateRef State = C.getState();
-  SVal VAListSVal = State->getSVal(VAA->getSubExpr(), C.getLocationContext());
-  const MemRegion *VAList = getVAListAsRegion(VAListSVal, C);
+  const Expr *VASubExpr = VAA->getSubExpr();
+  SVal VAListSVal = State->getSVal(VASubExpr, C.getLocationContext());
+  bool Symbolic;
+  const MemRegion *VAList =
+      getVAListAsRegion(VAListSVal, VASubExpr, Symbolic, C);
   if (!VAList)
     return;
+  if (Symbolic)
+    return;
   if (!State->contains<InitializedVALists>(VAList))
     reportUninitializedAccess(
         VAList, "va_arg() is called on an uninitialized va_list", C);
 }
 
 void ValistChecker::checkDeadSymbols(SymbolReaper &SR,
                                      CheckerContext &C) const {
   ProgramStateRef State = C.getState();
   InitializedVAListsTy TrackedVALists = State->get<InitializedVALists>();
   RegionVector LeakedVALists;
   for (auto Reg : TrackedVALists) {
     if (SR.isLiveRegion(Reg))
       continue;
     LeakedVALists.push_back(Reg);
     State = State->remove<InitializedVALists>(Reg);
   }
   if (ExplodedNode *N = C.addTransition(State))
     reportLeakedVALists(LeakedVALists, "Initialized va_list", " is leaked", C,
                         N);
 }
 
-const MemRegion *ValistChecker::getVAListAsRegion(SVal SV,
-                                                  CheckerContext &C) const {
-  const MemRegion *Reg = SV.getAsRegion();
-  const auto *TReg = dyn_cast_or_null<TypedValueRegion>(Reg);
-  // Some VarRegion based VLAs reach here as ElementRegions.
-  const auto *EReg = dyn_cast_or_null<ElementRegion>(TReg);
-  return EReg ? EReg->getSuperRegion() : TReg;
-}
-
 // This function traverses the exploded graph backwards and finds the node where
 // the va_list is initialized. That node is used for uniquing the bug paths.
 // It is not likely that there are several different va_lists that belongs to
 // different stack frames, so that case is not yet handled.
-const ExplodedNode *ValistChecker::getStartCallSite(const ExplodedNode *N,
-                                                    const MemRegion *Reg,
-                                                    CheckerContext &C) const {
+const ExplodedNode *
+ValistChecker::getStartCallSite(const ExplodedNode *N,
+                                const MemRegion *Reg) const {
   const LocationContext *LeakContext = N->getLocationContext();
   const ExplodedNode *StartCallNode = N;
 
   bool FoundInitializedState = false;
 
   while (N) {
     ProgramStateRef State = N->getState();
     if (!State->contains<InitializedVALists>(Reg)) {
if (!(ChecksEnabled[CK_Unterminated] ||
     return;
   for (auto Reg : LeakedVALists) {
     if (!BT_leakedvalist) {
       BT_leakedvalist.reset(new BugType(CheckNames[CK_Unterminated],
                                         "Leaked va_list", "Memory Error"));
       BT_leakedvalist->setSuppressOnSink(true);
     }
 
-    const ExplodedNode *StartNode = getStartCallSite(N, Reg, C);
+    const ExplodedNode *StartNode = getStartCallSite(N, Reg);
     PathDiagnosticLocation LocUsedForUniqueing;
 
     if (const Stmt *StartCallStmt = PathDiagnosticLocation::getStmt(StartNode))
       LocUsedForUniqueing = PathDiagnosticLocation::createBegin(
           StartCallStmt, C.getSourceManager(), StartNode->getLocationContext());
 
     SmallString<100> Buf;
     llvm::raw_svector_ostream OS(Buf);
for (auto Reg : LeakedVALists) {
     R->markInteresting(Reg);
     R->addVisitor(llvm::make_unique<ValistBugVisitor>(Reg, true));
     C.emitReport(std::move(R));
   }
 }
 
 void ValistChecker::checkVAListStartCall(const CallEvent &Call,
                                          CheckerContext &C, bool IsCopy) const {
-  const MemRegion *VAList = getVAListAsRegion(Call.getArgSVal(0), C);
-  ProgramStateRef State = C.getState();
+  bool Symbolic;
+  const MemRegion *VAList =
+      getVAListAsRegion(Call.getArgSVal(0), Call.getArgExpr(0), Symbolic, C);
   if (!VAList)
     return;
 
+  ProgramStateRef State = C.getState();
+
   if (IsCopy) {
-    const MemRegion *Arg2 = getVAListAsRegion(Call.getArgSVal(1), C);
+    const MemRegion *Arg2 =
+        getVAListAsRegion(Call.getArgSVal(1), Call.getArgExpr(1), Symbolic, C);
     if (Arg2) {
       if (ChecksEnabled[CK_CopyToSelf] && VAList == Arg2) {
         RegionVector LeakedVALists{VAList};
         if (ExplodedNode *N = C.addTransition(State))
           reportLeakedVALists(LeakedVALists, "va_list",
                               " is copied onto itself", C, N, true);
         return;
-      } else if (!State->contains<InitializedVALists>(Arg2)) {
+      } else if (!State->contains<InitializedVALists>(Arg2) && !Symbolic) {
         if (State->contains<InitializedVALists>(VAList)) {
           State = State->remove<InitializedVALists>(VAList);
           RegionVector LeakedVALists{VAList};
           if (ExplodedNode *N = C.addTransition(State))
             reportLeakedVALists(LeakedVALists, "Initialized va_list",
                                 " is overwritten by an uninitialized one", C, N,
                                 true);
         } else {
void ValistChecker::checkVAListStartCall(const CallEvent &Call,
   }
 
   State = State->add<InitializedVALists>(VAList);
   C.addTransition(State);
 }
 
 void ValistChecker::checkVAListEndCall(const CallEvent &Call,
                                        CheckerContext &C) const {
-  const MemRegion *VAList = getVAListAsRegion(Call.getArgSVal(0), C);
+  bool Symbolic;
+  const MemRegion *VAList =
+      getVAListAsRegion(Call.getArgSVal(0), Call.getArgExpr(0), Symbolic, C);
   if (!VAList)
     return;
 
+  // We did not see va_start call, but the source of the region is unknown.
+  // Be conservative and assume the best.
+  if (Symbolic)
+    return;
+
   if (!C.getState()->contains<InitializedVALists>(VAList)) {
     reportUninitializedAccess(
         VAList, "va_end() is called on an uninitialized va_list", C);
     return;
   }
   ProgramStateRef State = C.getState();
   State = State->remove<InitializedVALists>(VAList);
   C.addTransition(State);

test/Analysis/valist-uninitialized-no-undef.c

+// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -analyze -analyzer-checker=core,alpha.valist.Uninitialized,alpha.valist.CopyToSelf -analyzer-output=text -analyzer-store=region -verify %s
+
+#include "Inputs/system-header-simulator-for-valist.h"
+
+// This is the same function as the previous one, but it is called in call_inlined_uses_arg(),
+// and the warning is generated during the analysis of call_inlined_uses_arg().
+void inlined_uses_arg(va_list arg) {
+  (void)va_arg(arg, int); //expected-warning{{va_arg() is called on an uninitialized va_list}} expected-note{{va_arg() is called on an uninitialized va_list}}
+}
+
+void call_inlined_uses_arg(int fst, ...) {
+  va_list va;
+  inlined_uses_arg(va); // expected-note{{Calling 'inlined_uses_arg'}}
+}
+
+void f6(va_list *fst, ...) {
+  va_start(*fst, fst);
+  // FIXME: There should be no warning for this.
+  (void)va_arg(*fst, int); // expected-warning{{va_arg() is called on an uninitialized va_list}} expected-note{{va_arg() is called on an uninitialized va_list}}
+  va_end(*fst);
+} 
+
+void call_vprintf_bad(int isstring, ...) {
+  va_list va;
+  vprintf(isstring ? "%s" : "%d", va); //expected-warning{{Function 'vprintf' is called with an uninitialized va_list argument}} expected-note{{Function 'vprintf' is called with an uninitialized va_list argument}} expected-note{{Assuming 'isstring' is 0}} expected-note{{'?' condition is false}}
+}
+
+void call_vsprintf_bad(char *buffer, ...) {
+  va_list va;
+  va_start(va, buffer); // expected-note{{Initialized va_list}}
+  va_end(va); // expected-note{{Ended va_list}}
+  vsprintf(buffer, "%s %d %d %lf %03d", va); //expected-warning{{Function 'vsprintf' is called with an uninitialized va_list argument}} expected-note{{Function 'vsprintf' is called with an uninitialized va_list argument}}
+}
+

test/Analysis/valist-uninitialized.c

-// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -analyze -analyzer-checker=core,alpha.valist.Uninitialized,alpha.valist.CopyToSelf -analyzer-output=text -analyzer-store=region -verify %s
+// RUN: %clang_cc1 -triple hexagon-unknown-linux -analyze -analyzer-checker=core,alpha.valist.Uninitialized,alpha.valist.CopyToSelf -analyzer-disable-checker=core.CallAndMessage -analyzer-output=text -analyzer-store=region -verify %s
+// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -analyze -analyzer-checker=core,alpha.valist.Uninitialized,alpha.valist.CopyToSelf -analyzer-disable-checker=core.CallAndMessage -analyzer-output=text -analyzer-store=region -verify %s
 
 #include "Inputs/system-header-simulator-for-valist.h"
 
 void f1(int fst, ...) {
   va_list va;
   (void)va_arg(va, int); //expected-warning{{va_arg() is called on an uninitialized va_list}} expected-note{{va_arg() is called on an uninitialized va_list}}
 }
 

 }
 
 void f5(va_list fst, ...) {
   va_start(fst, fst);
   (void)va_arg(fst, int);
   va_end(fst);
 } // no-warning
 
-//FIXME: this should not cause a warning
-void f6(va_list *fst, ...) {
-  va_start(*fst, fst);
-  (void)va_arg(*fst, int); //expected-warning{{va_arg() is called on an uninitialized va_list}} expected-note{{va_arg() is called on an uninitialized va_list}}
-  va_end(*fst);
-}
-
 void f7(int *fst, ...) {
   va_list x;
   va_list *y = &x;
   va_start(*y,fst);
   (void)va_arg(x, int);
   va_end(x);
 } // no-warning
 

 void ends_arg(va_list arg) {
   va_end(arg);
 } //no-warning
 
 void uses_arg(va_list arg) {
   (void)va_arg(arg, int);
 } //no-warning
 
-// This is the same function as the previous one, but it is called in call_uses_arg2(),
-// and the warning is generated during the analysis of call_uses_arg2().
-void inlined_uses_arg(va_list arg) {
-  (void)va_arg(arg, int); //expected-warning{{va_arg() is called on an uninitialized va_list}} expected-note{{va_arg() is called on an uninitialized va_list}}
-}
-
-void call_inlined_uses_arg(int fst, ...) {
-  va_list va;
-  inlined_uses_arg(va); // expected-note{{Calling 'inlined_uses_arg'}}
-}
-
 void call_vprintf_ok(int isstring, ...) {
   va_list va;
   va_start(va, isstring);
   vprintf(isstring ? "%s" : "%d", va);
   va_end(va);
 } //no-warning
 
-void call_vprintf_bad(int isstring, ...) {
+void call_some_other_func(int n, ...) {
   va_list va;
-  vprintf(isstring ? "%s" : "%d", va); //expected-warning{{Function 'vprintf' is called with an uninitialized va_list argument}} expected-note{{Function 'vprintf' is called with an uninitialized va_list argument}} expected-note{{Assuming 'isstring' is 0}} expected-note{{'?' condition is false}}
-}
+  some_library_function(n, va);
+} //no-warning
 
-void call_vsprintf_bad(char *buffer, ...) {
-  va_list va;
-  va_start(va, buffer); // expected-note{{Initialized va_list}}
-  va_end(va); // expected-note{{Ended va_list}}
-  vsprintf(buffer, "%s %d %d %lf %03d", va); //expected-warning{{Function 'vsprintf' is called with an uninitialized va_list argument}} expected-note{{Function 'vsprintf' is called with an uninitialized va_list argument}}
+void inlined_uses_arg_good(va_list arg) {
+  (void)va_arg(arg, int);
 }
 
-void call_some_other_func(int n, ...) {
+void call_inlined_uses_arg_good(int fst, ...) {
   va_list va;
-  some_library_function(n, va);
-} //no-warning
+  va_start(va, fst);
+  inlined_uses_arg_good(va);
+  va_end(va);
+}
 
+void va_copy_test(va_list arg) {
+  va_list dst;
+  va_copy(dst, arg);
+  va_end(dst);
+}

test/Analysis/valist-unterminated.c

+// RUN: %clang_cc1 -triple hexagon-unknown-linux -analyze -analyzer-checker=core,alpha.valist.Unterminated,alpha.valist.CopyToSelf -analyzer-output=text -analyzer-store=region -verify %s
 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -analyze -analyzer-checker=core,alpha.valist.Unterminated,alpha.valist.CopyToSelf -analyzer-output=text -analyzer-store=region -verify %s
 
 #include "Inputs/system-header-simulator-for-valist.h"
 
 void f1(int fst, ...) {
   va_list va;
   va_start(va, fst); // expected-note{{Initialized va_list}}
   return; // expected-warning{{Initialized va_list 'va' is leaked}} expected-note{{Initialized va_list 'va' is leaked}}

 }
 
 void f4(va_list *fst, ...) {
   va_start(*fst, fst); // expected-note{{Initialized va_list}}
   return; // expected-warning{{Initialized va_list is leaked}} expected-note{{Initialized va_list is leaked}}
 }
 
 void f5(va_list fst, ...) {
-  va_start(fst, fst);
-  //FIXME: this should cause a warning
-} // no-warning
+  va_start(fst, fst); // expected-note{{Initialized va_list}}
+} // expected-warning{{Initialized va_list}} expected-note{{Initialized va_list}}
 
 void f6(va_list *fst, ...) {
   va_start(*fst, fst); // expected-note{{Initialized va_list}}
   (void)va_arg(*fst, int);
   //FIXME: this should NOT cause a warning
   va_end(*fst); // expected-warning{{Initialized va_list is leaked}} expected-note{{Initialized va_list is leaked}}
 }