This is an archive of the discontinued LLVM Phabricator instance.

[analyzer] Added support of scan-build and exploded-graph-rewriter regression tests for Windows
ClosedPublic

Authored by ASDenysPetrov on Mar 25 2020, 6:09 AM.

Details

Summary

Regression tests scan-build and exploded-graph-rewriter was tagged as not supporting Windows platform. The actual reason was that nobody checked that fact before running test on Windows. But the tests looks like they can work well relying on the way they are written.

Removed tag UNSUPPORTED from scan-build tests.
Removed tag UNSUPPORTED from exploded-graph-rewriter tests.
Fixed path to exploded-graph-rewriter.py in local.cfg.
Fixed JsonSupport in terms of handling Windows specific path delimiters.

Now all these tests locally pass on Windows. Shall be checked on other platforms.

Diff Detail

Event Timeline

ASDenysPetrov created this revision.Mar 25 2020, 6:09 AM
ASDenysPetrov edited the summary of this revision. (Show Details)Mar 27 2020, 3:08 AM
NoQ added inline comments.Mar 27 2020, 9:38 AM
clang/test/Analysis/exploded-graph-rewriter/macros.c
6–7

Nice!

clang/test/Analysis/scan-build/exclude_directories.test
1–2

Well, let's hope this actually works :)

clang/utils/analyzer/exploded-graph-rewriter.py
390

Interesting, so you're making less substitutions now than there were before. What was the test that you've noticed start failing that inspired you to make this change? I should have tested this properly too :D

ASDenysPetrov added inline comments.Mar 27 2020, 2:01 PM
clang/utils/analyzer/exploded-graph-rewriter.py
390

The test is
clang/test/Analysis/exploded-graph-rewriter/escapes.c
You can check before and after my changes. Some function "foo" looks like \\"\\\\"foo\\\\"\\" in dot file and were parsed abnormaly.

NoQ accepted this revision.Mar 28 2020, 1:07 PM

Aha, ok, got it, thanks! I'll try to land this and see what happens.

clang/utils/analyzer/exploded-graph-rewriter.py
390

Damn that's interesting. For me this test passes both before and after the patch (note that UNSUPPORTED: windows only disables the test on windows - it was working just fine on all other machines). So it looks like something's not quite cross-platform within our script, and the correct solution would be to address this portability problem.

I don't mind doing this separately later.

This revision is now accepted and ready to land.Mar 28 2020, 1:07 PM
ASDenysPetrov marked an inline comment as done.Mar 29 2020, 2:54 AM

If it helps I've attached my escapes.c.tmp.dot file which you can try to reproduce with.

clang/utils/analyzer/exploded-graph-rewriter.py
390

Sorry, I fogot to mention that the reason is not just in "foo" but also in the escape characters for Windows specific path.
Thus there were two problems.
Actually I removed the line .replace('\\\\', '\\') \ (which parsed Win path incorrectly) and added the line .replace('\\\\"', '\\"') \ (which started parsing 'foo' escapes correctly)

@NoQ could you, please, commit the change on my behalf.
@dcoughlin , please, make any note or accept the change.

NoQ requested changes to this revision.Mar 30 2020, 11:18 AM
NoQ added a reviewer: Charusso.
NoQ added inline comments.
clang/utils/analyzer/exploded-graph-rewriter.py
390

Aha, that explains, thanks! Let's try to come up with a less scary fix then?

In fact we discard the full path later anyway, so we could do this on the C++ side instead; that'd avoid the problem entirely.

Also we can try to properly escape it on the C++ side.

+@Charusso because he implemented a lot of this.

This revision now requires changes to proceed.Mar 30 2020, 11:18 AM

@Charusso Could you point me where I can find correspondent code which creates dot (graph) files?

@Charusso Could you point me where I can find correspondent code which creates dot (graph) files?

Each class has its own dumpJson method which builds up the graph together. The actual dot creation happens in ExprEngine.cpp's struct DOTGraphTraits<ExplodedGraph*>, changed in: D62346

I believe you are interested in escaping strings in the C++ side as @NoQ suggested. The usual way of digging into the past:
escapes.c -> GitHub blame -> recent commits -> commit message's review -> D63462 (where the escapes happen)

ASDenysPetrov edited the summary of this revision. (Show Details)

@NoQ please, review my new patch. I've made some changes from C++ side. It regards replacing backslash to forward-slash in case of Win platform defined.

Charusso accepted this revision.Apr 3 2020, 4:26 AM

@NoQ please, review my new patch. I've made some changes from C++ side. It regards replacing backslash to forward-slash in case of Win platform defined.

I have totally forgotten that. Awesome! Please note that, most likely the build-bots are going to break, because we do not support Windows very well. If that happens, please revert the patch. I think it is ready for the trial.

clang/test/Analysis/exploded-graph-rewriter/lit.local.cfg
15

I like the raw os.path.join. Could you flatten this formatting out or something?

clang/test/Analysis/scan-build/exclude_directories.test
1–2

+1 beware.

ASDenysPetrov marked an inline comment as done.

Changed tabs to spaces in lit.config.cfg.

clang/test/Analysis/scan-build/exclude_directories.test
1–2

As I figured out this might protect us from crashes (from lit.config.cfg):

use_lit_shell = os.environ.get("LIT_USE_INTERNAL_SHELL")
config.test_format = lit.formats.ShTest(use_lit_shell == "0")
NoQ accepted this revision.Apr 6 2020, 2:44 AM

Aha, fantastic! It's all understandable and obvious now. I'll try to land.

This revision is now accepted and ready to land.Apr 6 2020, 2:44 AM
This revision was automatically updated to reflect the committed changes.
Herald added a project: Restricted Project. · View Herald TranscriptApr 6 2020, 10:52 AM
Herald added a subscriber: cfe-commits. · View Herald Transcript
NoQ added a comment.EditedApr 6 2020, 11:32 AM

I landed it without removing REQUIRES: shell first and buildbots seem to be fine. That said, it's likely that REQUIRES: shell still disables them on Windows buildbots. I'll try to remove REQUIRES: shell tomorrow and see what happens.

thakis added a subscriber: thakis.Apr 7 2020, 6:05 AM

This seems to break tests on Windows: http://45.33.8.238/win/12320/step_7.txt

NoQ added a comment.Apr 7 2020, 6:47 AM

This seems to break tests on Windows: http://45.33.8.238/win/12320/step_7.txt

Thank you! Reverted.

clang/test/Analysis/scan-build/exclude_directories.test
1–2

Ok, so i only reverted the part where REQUIRES: shell is removed (rGcfd388d344ab) because that's when the buildbot problems started. The part that removes UNSUPPORTED: system-windows (rG2ddd3325c4d7) is still in. So i guess REQUIRES: shell can't be entirely removed. The error doesn't look like it's about not being able to run scan-build though; scan-build started just fine and produced some output. So it's likely a smaller bug.

@ASDenysPetrov any immediate thoughts? Do tests actually run on your machine when REQUIRES: shell is still in? I.e., do we get at least some testing on user's machines on the current trunk after my latest revert?

Here's a snapshot of the failure:

ninja: Entering directory `out/gn'
[0/2] ACTION //clang/test:check-clang(//llvm/utils/gn/build/toolchain:win)
llvm-lit.py: C:/src/llvm-project/llvm\utils\lit\lit\llvm\config.py:343: note: using clang: c:\src\llvm-project\out\gn\bin\clang.exe
-- Testing: 17064 tests, 32 workers --
Testing:  0
FAIL: Clang :: Analysis/scan-build/html_output.test (974 of 17064)
******************** TEST 'Clang :: Analysis/scan-build/html_output.test' FAILED ********************
Script:
--
: 'RUN: at line 1';   rm -rf C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir && mkdir C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir
: 'RUN: at line 2';   'c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat' --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe  -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir c:\src\llvm-project\out\gn\bin\clang.exe -S C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\html_output.test -check-prefix CHECK-STDOUT
: 'RUN: at line 12';   ls C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir/*/ | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\html_output.test -check-prefix CHECK-FILENAMES
: 'RUN: at line 21';   cat C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir/*/index.html      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\html_output.test -check-prefix CHECK-INDEX-HTML
: 'RUN: at line 27';   cat C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir/*/report-*.html      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\html_output.test -check-prefix CHECK-REPORT-HTML
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "rm" "-rf" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir"
$ "mkdir" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir"
$ ":" "RUN: at line 2"
$ "c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat" "--use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe" "-o" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir" "c:\src\llvm-project\out\gn\bin\clang.exe" "-S" "C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c"
$ "c:\src\llvm-project\out\gn\bin\filecheck.exe" "C:\src\llvm-project\clang\test\Analysis\scan-build\html_output.test" "-check-prefix" "CHECK-STDOUT"
# command stderr:
C:\src\llvm-project\clang\test\Analysis\scan-build\html_output.test:8:15: error: CHECK-STDOUT: expected string not found in input
CHECK-STDOUT: scan-build: 1 bug found.
              ^
<stdin>:4:1: note: scanning from here
scan-build: Removing directory '/c/src/llvm-project/out/gn/obj/clang/test/Analysis/scan-build/Output/html_output.test.tmp.output_dir/2020-04-07-090114-54704-1' because it contains no reports.
^
<stdin>:5:1: note: possible intended match here
scan-build: No bugs found.
^

error: command failed with exit status: 1

--

********************
Testing:  0
FAIL: Clang :: Analysis/scan-build/plist_html_output.test (977 of 17064)
******************** TEST 'Clang :: Analysis/scan-build/plist_html_output.test' FAILED ********************
Script:
--
: 'RUN: at line 1';   rm -rf C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir && mkdir C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir
: 'RUN: at line 2';   'c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat' --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe  -plist-html -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir c:\src\llvm-project\out\gn\bin\clang.exe -S C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\plist_html_output.test -check-prefix CHECK-STDOUT
: 'RUN: at line 14';   ls C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir/*/ | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\plist_html_output.test -check-prefix CHECK-FILENAMES
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "rm" "-rf" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir"
$ "mkdir" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir"
$ ":" "RUN: at line 2"
$ "c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat" "--use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe" "-plist-html" "-o" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir" "c:\src\llvm-project\out\gn\bin\clang.exe" "-S" "C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c"
$ "c:\src\llvm-project\out\gn\bin\filecheck.exe" "C:\src\llvm-project\clang\test\Analysis\scan-build\plist_html_output.test" "-check-prefix" "CHECK-STDOUT"
# command stderr:
C:\src\llvm-project\clang\test\Analysis\scan-build\plist_html_output.test:10:15: error: CHECK-STDOUT: expected string not found in input
CHECK-STDOUT: scan-build: 1 bug found.
              ^
<stdin>:6:1: note: scanning from here
scan-build: Removing directory '/c/src/llvm-project/out/gn/obj/clang/test/Analysis/scan-build/Output/plist_html_output.test.tmp.output_dir/2020-04-07-090114-54706-1' because it contains no reports.
^
<stdin>:7:1: note: possible intended match here
scan-build: No bugs found.
^

error: command failed with exit status: 1

--

********************
Testing:  0
FAIL: Clang :: Analysis/scan-build/exclude_directories.test (1014 of 17064)
******************** TEST 'Clang :: Analysis/scan-build/exclude_directories.test' FAILED ********************
Script:
--
: 'RUN: at line 1';   rm -rf C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir && mkdir C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir
: 'RUN: at line 2';   'c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat' --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe  -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir c:\src\llvm-project\out\gn\bin\clang.exe -S      C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory1/file1.c      C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory2/file2.c      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\exclude_directories.test -check-prefix CHECK-NO-EXCLUDE
: 'RUN: at line 17';   rm -rf C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir && mkdir C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir
: 'RUN: at line 18';   'c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat' --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe  -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir --exclude directory1 c:\src\llvm-project\out\gn\bin\clang.exe -S      C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory1/file1.c      C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory2/file2.c      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\exclude_directories.test -check-prefix CHECK-EXCLUDE1
: 'RUN: at line 27';   rm -rf C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir && mkdir C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir
: 'RUN: at line 28';   'c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat' --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe  -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir --exclude directory1 --exclude directory2 c:\src\llvm-project\out\gn\bin\clang.exe -S      C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory1/file1.c      C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory2/file2.c      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\exclude_directories.test -check-prefix CHECK-EXCLUDE-BOTH
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "rm" "-rf" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir"
$ "mkdir" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir"
$ ":" "RUN: at line 2"
$ "c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat" "--use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe" "-o" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir" "c:\src\llvm-project\out\gn\bin\clang.exe" "-S" "C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory1/file1.c" "C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory2/file2.c"
$ "c:\src\llvm-project\out\gn\bin\filecheck.exe" "C:\src\llvm-project\clang\test\Analysis\scan-build\exclude_directories.test" "-check-prefix" "CHECK-NO-EXCLUDE"
# command stderr:
C:\src\llvm-project\clang\test\Analysis\scan-build\exclude_directories.test:13:19: error: CHECK-NO-EXCLUDE: expected string not found in input
CHECK-NO-EXCLUDE: scan-build: 2 bugs found.
                  ^
<stdin>:2:1: note: scanning from here
C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build>perl -S scan-build --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir c:\src\llvm-project\out\gn\bin\clang.exe -S C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory1/file1.c C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory2/file2.c 
^
<stdin>:5:1: note: possible intended match here
scan-build: No bugs found.
^

error: command failed with exit status: 1

--

********************
Testing:  0.
FAIL: Clang :: Analysis/scan-build/plist_output.test (1116 of 17064)
******************** TEST 'Clang :: Analysis/scan-build/plist_output.test' FAILED ********************
Script:
--
: 'RUN: at line 1';   rm -rf C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir && mkdir C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir
: 'RUN: at line 2';   'c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat' --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe  -plist -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir c:\src\llvm-project\out\gn\bin\clang.exe -S C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\plist_output.test -check-prefix CHECK-STDOUT
: 'RUN: at line 12';   ls C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir/*/ | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\plist_output.test -check-prefix CHECK-FILENAMES
: 'RUN: at line 17';   cat C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir/*/report-*.plist      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\plist_output.test -check-prefix CHECK-REPORT-PLIST-CONTENTS
--
Exit Code: 2

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "rm" "-rf" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir"
$ "mkdir" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir"
$ ":" "RUN: at line 2"
$ "c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat" "--use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe" "-plist" "-o" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir" "c:\src\llvm-project\out\gn\bin\clang.exe" "-S" "C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c"
$ "c:\src\llvm-project\out\gn\bin\filecheck.exe" "C:\src\llvm-project\clang\test\Analysis\scan-build\plist_output.test" "-check-prefix" "CHECK-STDOUT"
$ ":" "RUN: at line 12"
$ "ls" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir/*/"
$ "c:\src\llvm-project\out\gn\bin\filecheck.exe" "C:\src\llvm-project\clang\test\Analysis\scan-build\plist_output.test" "-check-prefix" "CHECK-FILENAMES"
# command stderr:
FileCheck error: '<stdin>' is empty.
FileCheck command line:  c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\plist_output.test -check-prefix CHECK-FILENAMES

error: command failed with exit status: 2

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60
FAIL: Clang :: SemaObjC/parameterized_classes_subst.m (11256 of 17064)
******************** TEST 'Clang :: SemaObjC/parameterized_classes_subst.m' FAILED ********************
Script:
--
: 'RUN: at line 1';   c:\src\llvm-project\out\gn\bin\clang.exe -cc1 -internal-isystem c:\src\llvm-project\out\gn\lib\clang\11.0.0\include -nostdsysteminc -fblocks -fsyntax-only -Wnullable-to-nonnull-conversion C:\src\llvm-project\clang\test\SemaObjC\parameterized_classes_subst.m -verify
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "c:\src\llvm-project\out\gn\bin\clang.exe" "-cc1" "-internal-isystem" "c:\src\llvm-project\out\gn\lib\clang\11.0.0\include" "-nostdsysteminc" "-fblocks" "-fsyntax-only" "-Wnullable-to-nonnull-conversion" "C:\src\llvm-project\clang\test\SemaObjC\parameterized_classes_subst.m" "-verify"
# command stderr:
error: 'error' diagnostics seen but not expected: 
  File C:\src\llvm-project\clang\test\SemaObjC\parameterized_classes_subst.m Line 479: type argument 'T' (aka 'id') does not satisfy the bound ('id<NSCopying>') of type parameter 'T'
error: 'note' diagnostics seen but not expected: 
  File C:\src\llvm-project\clang\test\SemaObjC\parameterized_classes_subst.m Line 475: type parameter 'T' declared here
2 errors generated.

error: command failed with exit status: 1

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 

Testing Time: 173.99s
********************
Failing Tests (5):
    Clang :: Analysis/scan-build/exclude_directories.test
    Clang :: Analysis/scan-build/html_output.test
    Clang :: Analysis/scan-build/plist_html_output.test
    Clang :: Analysis/scan-build/plist_output.test
    Clang :: SemaObjC/parameterized_classes_subst.m

  Expected Passes    : 16885
  Expected Failures  : 28
  Unsupported Tests  : 146
  Unexpected Failures: 5
FAILED: gen/clang/test/run-lit 
C:/src/depot_tools/bootstrap-3_8_0_chromium_8_bin/python/bin/python.exe bin/llvm-lit.py -sv ../../clang/test
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "../hack/llvmgnsyncbot/syncbot.py", line 233, in <module>
    sys.exit(main())
  File "../hack/llvmgnsyncbot/syncbot.py", line 228, in main
    run(args.last_exit)
  File "../hack/llvmgnsyncbot/syncbot.py", line 191, in run
    subprocess.check_call(['ninja', '-C', 'out/gn', test])
  File "C:\python_27_amd64\files\lib\subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja', '-C', 'out/gn', 'check-clang']' returned non-zero exit status 1
ASDenysPetrov added a comment.EditedApr 7 2020, 6:57 AM

@NoQ
Yes, I have some thoughts. I am on revision master from [02.04.2020 10:29:20] (5 days ago). I've check these test and all passes on my PC.

As you can see in the log (No bugs found):

scan-build: Removing directory '/c/src/llvm-project/out/gn/obj/clang/test/Analysis/scan-build/Output/plist_html_output.test.tmp.output_dir/2020-04-07-090114-54706-1' because it contains no reports.
^
<stdin>:7:1: note: possible intended match here
scan-build: No bugs found.

That means that Inputs/single_null_dereference.c is somewhat differs from mine or clang behavior was changed since my revision. I will try to update sources and reproduce on the latest ones.

Do tests actually run on your machine when REQUIRES: shell is still in?

No they don't. They are marked as unsuported if REQUIRES: shell is in.

But what about SemaObjC/parameterized_classes_subst.m, I strongly doubt the reason is in this patch.

ASDenysPetrov added a comment.EditedApr 7 2020, 7:55 AM

@NoQ
I've just pulled all from github and make a build. Then I ran the tests, and they successfully passed.
Namely, scan-build analyzed inputs and create stdout with diagnostics. Then filecheck checked without fails. But in case of buildbot it didn't create any diagnostic stdout.

According to buildbot's logs I assume that smth is wrong on its side.

This comment was removed by Charusso.

I believe it is very strange on a Windows system to have multiple dots in a file. The other issue could be the wildcard /*/ in a path full of \s. The LLVM lit (https://llvm.org/docs/CommandGuide/lit.html) has tons of Windows-related shortcuts, which I have never seen being used, but could be useful.

thakis added a comment.Apr 7 2020, 9:17 AM

Thanks for checking. Let me try the tests locally over here tonight. I'll get back to you in a few hours.

But what about SemaObjC/parameterized_classes_subst.m, I strongly doubt the reason is in this patch.

That's a known unrelated flake, see thread for "[clang] a8c8b62 - [ObjC generics] Fix not inheriting type bounds in categories/extensions".

I believe it is very strange on a Windows system to have multiple dots in a file. The other issue could be the wildcard /*/ in a path full of \s. The LLVM lit (https://llvm.org/docs/CommandGuide/lit.html) has tons of Windows-related shortcuts, which I have never seen being used, but could be useful.

I've checked. There is no problem with dots and /*/. My command promt (Win10) handles them perfectly well. Mixing / and \ in paths is also acceptable and handles correctly.
For instance, Visual Studio creates file with multiple dots for projects as "Project1.vcxproj.filters". Also you can easely create the same directory.
Wildcard /*/ is a part of input syntax of unix utility cat, thus it works as well.

Is there any way to get to buildbot file system to compare test files with what is in the master branch? Particularly C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c

NoQ added a comment.Apr 7 2020, 8:27 PM

Is there any way to get to buildbot file system to compare test files with what is in the master branch? Particularly C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c

If the source was checked out incorrectly then most likely everything would have been broken, not just our test. I'd rather suspect write permissions or something, but then again, a lot of tests create directories and then write into them. Dunno.

@thakis, do you know what is the setup of this buildbot? Like, what's so special about it? 'Cause i weren't seeing any failures from other windows buildbots. Can we discriminate between this buildbot and other buildbots with any UNSUPPORTED: bang (even if we have to add one)?

If the source was checked out incorrectly then most likely everything would have been broken, not just our test. I'd rather suspect write permissions or something, but then again, a lot of tests create directories and then write into them. Dunno.

No, I mean the source itself could be different.
I don't think permissions is the case, because we saw the log when scan-build reported: No bugs found..

thakis added a comment.Apr 8 2020, 6:13 PM

Sorry for the slow reply. I repro'd this locally now, and here I see this error:

'''

command stderr:

'perl' is not recognized as an internal or external command,
operable program or batch file.
'''

perl wasn't required to run tests on Win for a long time. If we do want to run these tests on Windows, could we make the config lit file check if perl is available and set a feature and require that in the tests?

@thakis

'perl' is not recognized as an internal or external command,

I think you got another problem, that Perl isn't just presented in your PATH. It is not the case for buildbot logs.

perl wasn't required to run tests on Win for a long time. If we do want to run these tests on Windows, could we make the config lit file check if perl is available and set a feature and require that in the tests?

This is a good idea, to make tests more robust and friendly verbal.

@thakis

'perl' is not recognized as an internal or external command,

I think you got another problem, that Perl isn't just presented in your PATH. It is not the case for buildbot logs.

I tried again in a git bash shell where perl is present. With that, I get this error: http://codepad.org/t6ZhaCXI

"Can't find scan-build on PATH."

I guess that means that the bat file runs, but the perl -s scan-build therein fails? scan-build is right next to the bat file, but the perl process can't know the directory the .bat file is in. With this change it gets a bit farther:

-perl -S scan-build %*
+perl -S %~dp0\scan-build %*

(Since it works for others I probably misunderstood how things are supposed to work, and this change might be wrong.)

New error:

"""

command stderr:

Can't find c:\src\llvm-project\clang\tools\scan-build\bin\\scan-build on PATH, '.' not in PATH.
"""

If I also remove the -S switch, that goes away (what's supposed to add clang\tools\scan-build\bin to path? Or is it supposed to be copied to bin\ in the out dir? It looks like clang\tools\scan-build\CMakeList.txt has logic to copy the script around, but clang\test\CMakeLists.txt doesn't have a dependency on the scan-build target, so building check-clang doesn't guarantee that the script is copied. Maybe it's just this dependency that's missing?

If I add scan-build to clang/test/CMakeLists.txt and undo my other changes, the test passes for me. I guess you always build that target, which is why you didn't see this problem. But I wonder why this isn't needed on non-Windows.

(Also, that dep should only be added if CLANG_INSTALL_SCANBUILD isn't set to OFF, and I suppose the tests should only run if it isn't off too.)

If I add scan-build to clang/test/CMakeLists.txt and undo my other changes, the test passes for me. I guess you always build that target, which is why you didn't see this problem. But I wonder why this isn't needed on non-Windows.

On non-Windows, %scan-build expands to clang/tools/scan-build/bin/scan-build directly, and so non-Windows doesn't require scan-build on the PATH (i.e. in bin/ in the build dir) since there's no scan-build.bat trampoline.

I wonder if %scan-build should expand to "perl clang/tools/scan-build/bin/scan-build" on Windows, instead of to scan-build.bat, so that the Windows setup is more similar to the non-Windows setup.

ASDenysPetrov added a comment.EditedApr 10 2020, 6:37 AM

@thakis
Let me explain what works for me.
This is in my PATH:

  • C:\Perl\c\bin
  • C:\Perl\perl\site\bin
  • C:\Perl\perl\bin
  • D:\llvm-project\buildn\bin

This is what in D:\llvm-project\buildn\bin (all this is a product of ninja+gcc):

  • clang.exe
  • FileCheck.exe
  • scan-build.bat
  • scan-build
  • llvm-lit.py

No additional changes required (CMakeList.txt, bat editing, etc.).
Than I run command promt from arbitrary dir and run next: llvm-lit.py D:/llvm-project/clang/test/Analysis/scan-build/exclude_directories.test
Test passes. Done.
I hope it will help.

@thakis
Let me explain what works for me.
This is in my PATH:

  • C:\Perl\c\bin
  • C:\Perl\perl\site\bin
  • C:\Perl\perl\bin
  • D:\llvm-project\buildn\bin

^ This is not usually true. (But I think lit adds build/bin to the PATH for tests, so it's possibly true for tests, which would be enough.)

This is what in D:\llvm-project\buildn\bin (all this is a product of ninja+gcc):

  • clang.exe
  • FileCheck.exe
  • scan-build.bat
  • scan-build
  • llvm-lit.py

No additional changes required (CMakeList.txt, bat editing, etc.).
Than I run command promt from arbitrary dir and run next: llvm-lit.py D:/llvm-project/clang/test/Analysis/scan-build/exclude_directories.test
Test passes. Done.

If you run del bin\* followed by make check-clang (or ninja check-clang or what have you), then I think bin/scan-build won't be built since it's not a dependency of the check-clang target. It doesn't have to be a dependency on non-Win, but on Win the execution flow is different due to the bat trampoline.

Please try that and confirm that you see a test failure in that scenario as well.

@thakis,

^ This is not usually true. (But I think lit adds build/bin to the PATH for tests, so it's possibly true for tests, which would be enough.)

AFAIK lit do not add anything to PATH. You should do it by yourself.

If you run del bin\* followed by make check-clang (or ninja check-clang or what have you), then I think bin/scan-build won't be built since it's not a dependency of the check-clang target. It doesn't have to be a dependency on non-Win, but on Win the execution flow is different due to the bat trampoline.

I did as you ask. Confirm your suggestion: scan-build have not appeared after ninja check-clang. All the tests except one passed. Log attached

.
Intresting but scan-build tests haven't been run through ninja check-clang.