This is an archive of the discontinued LLVM Phabricator instance.

Differential D81592

[analyzer] SATest: Add a set of initial projects for testing
ClosedPublic

Authored by vsavchenko on Jun 10 2020, 10:31 AM.

Details

Reviewers
NoQ
dcoughlin
Commits
rG8dc2802773ce: [analyzer] SATest: Add a set of initial projects for testing

Diff Detail

Unit TestsFailed

TimeTest
6,870 mslinux > lldb-api.commands/expression/import-std-module/basic::Unknown Unit Message ("")
2,090 mslinux > lldb-api.commands/expression/import-std-module/conflicts::Unknown Unit Message ("")
3,460 mslinux > lldb-api.commands/expression/import-std-module/deque-basic::Unknown Unit Message ("")
3,350 mslinux > lldb-api.commands/expression/import-std-module/deque-dbg-info-content::Unknown Unit Message ("")
2,810 mslinux > lldb-api.commands/expression/import-std-module/forward_list::Unknown Unit Message ("")
View Full Test Results (45 Failed)

Event Timeline

vsavchenko created this revision.Jun 10 2020, 10:31 AM
Herald added a project: Restricted Project. · View Herald TranscriptJun 10 2020, 10:32 AM
Herald added subscribers: cfe-commits, ASDenysPetrov, Charusso and 8 others. · View Herald Transcript
Harbormaster failed remote builds in B59833: Diff 269897!Jun 10 2020, 11:06 AM
vsavchenko added a child revision: D81593: [analyzer] SATest: Make docker interfaces transparent.Jun 11 2020, 3:06 AM
vsavchenko added a parent revision: D81572: [analyzer] SATest: Add convenience 'docker' command.
NoQ added inline comments.Jun 18 2020, 7:12 AM
clang/utils/analyzer/.dockerignore
2

NO NeWlINE At EnD oF fILE

clang/utils/analyzer/projects/box2d/run_static_analyzer.cmd
2

Is there a flag to enable assertions or something like that? Usually we recommend people to enable assertions during analysis. It's also valuable to test that assertions are actually working.

Disabling assertions is in fact relatively good for reference results testing because all the false positives that they suppress are suddenly visible, and therefore the signal we receive from such testing is amplified significantly. But i wouldn't force such workflow on our solution-for-everyone because that requires a certain level of masochism in order to distinguish between "my patch introduced a false positive and it's bad" and "my patch introduced a false positive but it would have been suppressed by assertions and we do actually want to emit the warning in absence of assertions so it's good".

NoQ mentioned this in D81599: [analyzer] SATest: Add 5 more projects for testing.Jun 18 2020, 7:26 AM
vsavchenko marked an inline comment as done.Jun 18 2020, 7:58 AM
vsavchenko added inline comments.
clang/utils/analyzer/projects/box2d/run_static_analyzer.cmd
2

CMake doesn't have a unified solution for "Release with assertions" type of build. It is usually done on per-project basis (like in LLVM). I guess rather than searching for such a flag for every project, we can indeed change it to Debug.

NoQ added inline comments.Jun 19 2020, 7:48 AM
clang/utils/analyzer/projects/box2d/run_static_analyzer.cmd
2

Maybe something like -DCMAKE_C_FLAGS=-UNDEBUG could also work (or use scan-build --force-analyze-debug-code which is basically the same thing) but choosing Debug indeed sounds like the sanest solution.

vsavchenko updated this revision to Diff 273051.Jun 24 2020, 8:43 AM

Change build type to Debug

vsavchenko updated this revision to Diff 273083.Jun 24 2020, 9:58 AM

Add a new line in .dockerignore

Harbormaster failed remote builds in B61562: Diff 273051!Jun 24 2020, 10:15 AM
Harbormaster failed remote builds in B61578: Diff 273083!Jun 24 2020, 11:55 AM
NoQ accepted this revision.Jun 24 2020, 4:54 PM

Thx!

This revision is now accepted and ready to land.Jun 24 2020, 4:54 PM
Closed by commit rG8dc2802773ce: [analyzer] SATest: Add a set of initial projects for testing (authored by vsavchenko). · Explain WhyJun 25 2020, 2:31 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 273051

.gitignore

Show First 20 Lines • Show All 49 Lines • ▼ Show 20 Lines
#==============================================================================# #==============================================================================#
# Directories to ignore (do not add trailing '/'s, they skip symlinks). # Directories to ignore (do not add trailing '/'s, they skip symlinks).
#==============================================================================# #==============================================================================#
# VS2017 and VSCode config files. # VS2017 and VSCode config files.
.vscode .vscode
.vs .vs
# clangd index # clangd index
.clangd .clangd
# static analyzer regression testing project files
/clang/utils/analyzer/projects/*/CachedSource
/clang/utils/analyzer/projects/*/PatchedSource
/clang/utils/analyzer/projects/*/ScanBuildResults
/clang/utils/analyzer/projects/*/RefScanBuildResults
No newline at end of file

clang/utils/analyzer/.dockerignore

  • This file was added.
./projects
No newline at end of file
NoQUnsubmitted
Not Done
ReplyInline Actions

NO NeWlINE At EnD oF fILE

NoQ: NO NeWlINE At EnD oF fILE

clang/utils/analyzer/projects/box2d/cleanup_run_static_analyzer.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
rm -rf ./build

clang/utils/analyzer/projects/box2d/run_static_analyzer.cmd

  • This file was added.
cmake . -DCMAKE_BUILD_TYPE=Debug -Bbuild -GNinja
cmake --build build
NoQUnsubmitted
Not Done
ReplyInline Actions

Is there a flag to enable assertions or something like that? Usually we recommend people to enable assertions during analysis. It's also valuable to test that assertions are actually working.

Disabling assertions is in fact relatively good for reference results testing because all the false positives that they suppress are suddenly visible, and therefore the signal we receive from such testing is amplified significantly. But i wouldn't force such workflow on our solution-for-everyone because that requires a certain level of masochism in order to distinguish between "my patch introduced a false positive and it's bad" and "my patch introduced a false positive but it would have been suppressed by assertions and we do actually want to emit the warning in absence of assertions so it's good".

NoQ: Is there a flag to enable assertions or something like that? Usually we recommend people to…
vsavchenkoAuthorUnsubmitted
Done
ReplyInline Actions

CMake doesn't have a unified solution for "Release with assertions" type of build. It is usually done on per-project basis (like in LLVM). I guess rather than searching for such a flag for every project, we can indeed change it to Debug.

vsavchenko: CMake doesn't have a unified solution for "Release with assertions" type of build. It is…
NoQUnsubmitted
Not Done
ReplyInline Actions

Maybe something like -DCMAKE_C_FLAGS=-UNDEBUG could also work (or use scan-build --force-analyze-debug-code which is basically the same thing) but choosing Debug indeed sounds like the sanest solution.

NoQ: Maybe something like `-DCMAKE_C_FLAGS=-UNDEBUG` could also work (or use `scan-build --force…

clang/utils/analyzer/projects/cxxopts/cleanup_run_static_analyzer.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
rm -rf ./build

clang/utils/analyzer/projects/cxxopts/run_static_analyzer.cmd

  • This file was added.
cmake . -DCMAKE_BUILD_TYPE=Debug -Bbuild -GNinja
cmake --build build

clang/utils/analyzer/projects/libsoundio/cleanup_run_static_analyzer.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
rm -rf ./build

clang/utils/analyzer/projects/libsoundio/run_static_analyzer.cmd

  • This file was added.
cmake . -DCMAKE_BUILD_TYPE=Debug -Bbuild -GNinja
cmake --build build

clang/utils/analyzer/projects/oatpp/cleanup_run_static_analyzer.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
rm -rf ./build

clang/utils/analyzer/projects/oatpp/run_static_analyzer.cmd

  • This file was added.
cmake . -DCMAKE_BUILD_TYPE=Debug -Bbuild -GNinja
cmake --build build

clang/utils/analyzer/projects/projects.json

  • This file was added.
[
{
"name": "cxxopts",
"mode": 1,
"source": "git",
"origin": "https://github.com/jarro2783/cxxopts.git",
"commit": "794c975"
},
{
"name": "box2d",
"mode": 1,
"source": "git",
"origin": "https://github.com/erincatto/box2d.git",
"commit": "1025f9a"
},
{
"name": "tinyexpr",
"mode": 1,
"source": "git",
"origin": "https://github.com/codeplea/tinyexpr.git",
"commit": "ffb0d41"
},
{
"name": "symengine",
"mode": 1,
"source": "git",
"origin": "https://github.com/symengine/symengine.git",
"commit": "4f669d59"
},
{
"name": "termbox",
"mode": 1,
"source": "git",
"origin": "https://github.com/nsf/termbox.git",
"commit": "0df1355"
},
{
"name": "tinyvm",
"mode": 1,
"source": "git",
"origin": "https://github.com/jakogut/tinyvm.git",
"commit": "10c25d8"
},
{
"name": "tinyspline",
"mode": 1,
"source": "git",
"origin": "https://github.com/msteinbeck/tinyspline.git",
"commit": "f8b1ab7"
},
{
"name": "oatpp",
"mode": 1,
"source": "git",
"origin": "https://github.com/oatpp/oatpp.git",
"commit": "d3e60fb"
},
{
"name": "libsoundio",
"mode": 1,
"source": "git",
"origin": "https://github.com/andrewrk/libsoundio.git",
"commit": "b810bf2"
},
{
"name": "zstd",
"mode": 1,
"source": "git",
"origin": "https://github.com/facebook/zstd.git",
"commit": "2af4e073"
}
]

clang/utils/analyzer/projects/symengine/cleanup_run_static_analyzer.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
rm -rf ./build

clang/utils/analyzer/projects/symengine/run_static_analyzer.cmd

  • This file was added.
cmake . -DCMAKE_BUILD_TYPE=Debug -DWITH_COTIRE=OFF -Bbuild -GNinja
cmake --build build

clang/utils/analyzer/projects/termbox/cleanup_run_static_analyzer.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
./waf clean
exit 0

clang/utils/analyzer/projects/termbox/run_static_analyzer.cmd

  • This file was added.
./waf configure
./waf

clang/utils/analyzer/projects/tinyexpr/cleanup_run_static_analyzer.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
make clean

clang/utils/analyzer/projects/tinyexpr/run_static_analyzer.cmd

  • This file was added.
make

clang/utils/analyzer/projects/tinyspline/cleanup_run_static_analyzer.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
rm -rf ./build

clang/utils/analyzer/projects/tinyspline/run_static_analyzer.cmd

  • This file was added.
cmake . -DCMAKE_BUILD_TYPE=Debug -Bbuild -GNinja
cmake --build build

clang/utils/analyzer/projects/tinyvm/cleanup_run_static_analyzer.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
make clean

clang/utils/analyzer/projects/tinyvm/run_static_analyzer.cmd

  • This file was added.
make

clang/utils/analyzer/projects/zstd/cleanup_run_static_analyzer.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
rm -rf ./build_analyzer

clang/utils/analyzer/projects/zstd/run_static_analyzer.cmd

  • This file was added.
cmake ./build/cmake -DCMAKE_BUILD_TYPE=Debug -Bbuild_analyzer -GNinja
cmake --build build_analyzer