[analyzer] Added support of scan-build and exploded-graph-rewriter regression tests for Windows

[ASDenysPetrov]

Summary

Regression tests scan-build and exploded-graph-rewriter was tagged as not supporting Windows platform. The actual reason was that nobody checked that fact before running test on Windows. But the tests looks like they can work well relying on the way they are written.

Removed tag UNSUPPORTED from scan-build tests.
Removed tag UNSUPPORTED from exploded-graph-rewriter tests.
Fixed path to exploded-graph-rewriter.py in local.cfg.
Fixed JsonSupport in terms of handling Windows specific path delimiters.

Now all these tests locally pass on Windows. Shall be checked on other platforms.

[NoQ]

clang/test/Analysis/exploded-graph-rewriter/macros.c

6–7

Nice!

clang/test/Analysis/scan-build/exclude_directories.test

1–2

Well, let's hope this actually works :)

clang/utils/analyzer/exploded-graph-rewriter.py

390

Interesting, so you're making less substitutions now than there were before. What was the test that you've noticed start failing that inspired you to make this change? I should have tested this properly too :D

[ASDenysPetrov]

clang/utils/analyzer/exploded-graph-rewriter.py

390

The test is
clang/test/Analysis/exploded-graph-rewriter/escapes.c
You can check before and after my changes. Some function "foo" looks like \\"\\\\"foo\\\\"\\" in dot file and were parsed abnormaly.

[NoQ]

Aha, ok, got it, thanks! I'll try to land this and see what happens.

clang/utils/analyzer/exploded-graph-rewriter.py

390

Damn that's interesting. For me this test passes both before and after the patch (note that UNSUPPORTED: windows only disables the test on windows - it was working just fine on all other machines). So it looks like something's not quite cross-platform within our script, and the correct solution would be to address this portability problem.

I don't mind doing this separately later.

[ASDenysPetrov]

If it helps I've attached my escapes.c.tmp.dot file which you can try to reproduce with.

escapes.c.tmp.dot15 KBDownload

clang/utils/analyzer/exploded-graph-rewriter.py

390

Sorry, I fogot to mention that the reason is not just in "foo" but also in the escape characters for Windows specific path.
Thus there were two problems.
Actually I removed the line .replace('\\\\', '\\') \ (which parsed Win path incorrectly) and added the line .replace('\\\\"', '\\"') \ (which started parsing 'foo' escapes correctly)

[ASDenysPetrov]

@NoQ could you, please, commit the change on my behalf.
@dcoughlin , please, make any note or accept the change.

[NoQ]

clang/utils/analyzer/exploded-graph-rewriter.py

390

Aha, that explains, thanks! Let's try to come up with a less scary fix then?

In fact we discard the full path later anyway, so we could do this on the C++ side instead; that'd avoid the problem entirely.

Also we can try to properly escape it on the C++ side.

+@Charusso because he implemented a lot of this.

[ASDenysPetrov]

@Charusso Could you point me where I can find correspondent code which creates dot (graph) files?

[Charusso]

In D76768#1952674, @ASDenysPetrov wrote:

@Charusso Could you point me where I can find correspondent code which creates dot (graph) files?

Each class has its own dumpJson method which builds up the graph together. The actual dot creation happens in ExprEngine.cpp's struct DOTGraphTraits<ExplodedGraph*>, changed in: D62346

I believe you are interested in escaping strings in the C++ side as @NoQ suggested. The usual way of digging into the past:
escapes.c -> GitHub blame -> recent commits -> commit message's review -> D63462 (where the escapes happen)

[ASDenysPetrov]

@NoQ please, review my new patch. I've made some changes from C++ side. It regards replacing backslash to forward-slash in case of Win platform defined.

[Charusso]

In D76768#1959285, @ASDenysPetrov wrote:

@NoQ please, review my new patch. I've made some changes from C++ side. It regards replacing backslash to forward-slash in case of Win platform defined.

I have totally forgotten that. Awesome! Please note that, most likely the build-bots are going to break, because we do not support Windows very well. If that happens, please revert the patch. I think it is ready for the trial.

clang/test/Analysis/exploded-graph-rewriter/lit.local.cfg

15

I like the raw os.path.join. Could you flatten this formatting out or something?

clang/test/Analysis/scan-build/exclude_directories.test

1–2

+1 beware.

[ASDenysPetrov]

Changed tabs to spaces in lit.config.cfg.

clang/test/Analysis/scan-build/exclude_directories.test

1–2

As I figured out this might protect us from crashes (from lit.config.cfg):

use_lit_shell = os.environ.get("LIT_USE_INTERNAL_SHELL")
config.test_format = lit.formats.ShTest(use_lit_shell == "0")

[NoQ]

Aha, fantastic! It's all understandable and obvious now. I'll try to land.

[NoQ]

I landed it without removing REQUIRES: shell first and buildbots seem to be fine. That said, it's likely that REQUIRES: shell still disables them on Windows buildbots. I'll try to remove REQUIRES: shell tomorrow and see what happens.

[thakis]

This seems to break tests on Windows: http://45.33.8.238/win/12320/step_7.txt

[NoQ]

In D76768#1966813, @thakis wrote:

This seems to break tests on Windows: http://45.33.8.238/win/12320/step_7.txt

Thank you! Reverted.

clang/test/Analysis/scan-build/exclude_directories.test

1–2

Ok, so i only reverted the part where REQUIRES: shell is removed (rGcfd388d344ab) because that's when the buildbot problems started. The part that removes UNSUPPORTED: system-windows (rG2ddd3325c4d7) is still in. So i guess REQUIRES: shell can't be entirely removed. The error doesn't look like it's about not being able to run scan-build though; scan-build started just fine and produced some output. So it's likely a smaller bug.

@ASDenysPetrov any immediate thoughts? Do tests actually run on your machine when REQUIRES: shell is still in? I.e., do we get at least some testing on user's machines on the current trunk after my latest revert?

Here's a snapshot of the failure:

ninja: Entering directory `out/gn'
[0/2] ACTION //clang/test:check-clang(//llvm/utils/gn/build/toolchain:win)
llvm-lit.py: C:/src/llvm-project/llvm\utils\lit\lit\llvm\config.py:343: note: using clang: c:\src\llvm-project\out\gn\bin\clang.exe
-- Testing: 17064 tests, 32 workers --
Testing:  0
FAIL: Clang :: Analysis/scan-build/html_output.test (974 of 17064)
******************** TEST 'Clang :: Analysis/scan-build/html_output.test' FAILED ********************
Script:
--
: 'RUN: at line 1';   rm -rf C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir && mkdir C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir
: 'RUN: at line 2';   'c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat' --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe  -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir c:\src\llvm-project\out\gn\bin\clang.exe -S C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\html_output.test -check-prefix CHECK-STDOUT
: 'RUN: at line 12';   ls C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir/*/ | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\html_output.test -check-prefix CHECK-FILENAMES
: 'RUN: at line 21';   cat C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir/*/index.html      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\html_output.test -check-prefix CHECK-INDEX-HTML
: 'RUN: at line 27';   cat C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir/*/report-*.html      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\html_output.test -check-prefix CHECK-REPORT-HTML
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "rm" "-rf" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir"
$ "mkdir" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir"
$ ":" "RUN: at line 2"
$ "c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat" "--use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe" "-o" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\html_output.test.tmp.output_dir" "c:\src\llvm-project\out\gn\bin\clang.exe" "-S" "C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c"
$ "c:\src\llvm-project\out\gn\bin\filecheck.exe" "C:\src\llvm-project\clang\test\Analysis\scan-build\html_output.test" "-check-prefix" "CHECK-STDOUT"
# command stderr:
C:\src\llvm-project\clang\test\Analysis\scan-build\html_output.test:8:15: error: CHECK-STDOUT: expected string not found in input
CHECK-STDOUT: scan-build: 1 bug found.
              ^
<stdin>:4:1: note: scanning from here
scan-build: Removing directory '/c/src/llvm-project/out/gn/obj/clang/test/Analysis/scan-build/Output/html_output.test.tmp.output_dir/2020-04-07-090114-54704-1' because it contains no reports.
^
<stdin>:5:1: note: possible intended match here
scan-build: No bugs found.
^

error: command failed with exit status: 1

--

********************
Testing:  0
FAIL: Clang :: Analysis/scan-build/plist_html_output.test (977 of 17064)
******************** TEST 'Clang :: Analysis/scan-build/plist_html_output.test' FAILED ********************
Script:
--
: 'RUN: at line 1';   rm -rf C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir && mkdir C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir
: 'RUN: at line 2';   'c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat' --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe  -plist-html -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir c:\src\llvm-project\out\gn\bin\clang.exe -S C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\plist_html_output.test -check-prefix CHECK-STDOUT
: 'RUN: at line 14';   ls C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir/*/ | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\plist_html_output.test -check-prefix CHECK-FILENAMES
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "rm" "-rf" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir"
$ "mkdir" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir"
$ ":" "RUN: at line 2"
$ "c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat" "--use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe" "-plist-html" "-o" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_html_output.test.tmp.output_dir" "c:\src\llvm-project\out\gn\bin\clang.exe" "-S" "C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c"
$ "c:\src\llvm-project\out\gn\bin\filecheck.exe" "C:\src\llvm-project\clang\test\Analysis\scan-build\plist_html_output.test" "-check-prefix" "CHECK-STDOUT"
# command stderr:
C:\src\llvm-project\clang\test\Analysis\scan-build\plist_html_output.test:10:15: error: CHECK-STDOUT: expected string not found in input
CHECK-STDOUT: scan-build: 1 bug found.
              ^
<stdin>:6:1: note: scanning from here
scan-build: Removing directory '/c/src/llvm-project/out/gn/obj/clang/test/Analysis/scan-build/Output/plist_html_output.test.tmp.output_dir/2020-04-07-090114-54706-1' because it contains no reports.
^
<stdin>:7:1: note: possible intended match here
scan-build: No bugs found.
^

error: command failed with exit status: 1

--

********************
Testing:  0
FAIL: Clang :: Analysis/scan-build/exclude_directories.test (1014 of 17064)
******************** TEST 'Clang :: Analysis/scan-build/exclude_directories.test' FAILED ********************
Script:
--
: 'RUN: at line 1';   rm -rf C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir && mkdir C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir
: 'RUN: at line 2';   'c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat' --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe  -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir c:\src\llvm-project\out\gn\bin\clang.exe -S      C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory1/file1.c      C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory2/file2.c      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\exclude_directories.test -check-prefix CHECK-NO-EXCLUDE
: 'RUN: at line 17';   rm -rf C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir && mkdir C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir
: 'RUN: at line 18';   'c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat' --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe  -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir --exclude directory1 c:\src\llvm-project\out\gn\bin\clang.exe -S      C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory1/file1.c      C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory2/file2.c      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\exclude_directories.test -check-prefix CHECK-EXCLUDE1
: 'RUN: at line 27';   rm -rf C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir && mkdir C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir
: 'RUN: at line 28';   'c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat' --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe  -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir --exclude directory1 --exclude directory2 c:\src\llvm-project\out\gn\bin\clang.exe -S      C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory1/file1.c      C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory2/file2.c      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\exclude_directories.test -check-prefix CHECK-EXCLUDE-BOTH
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "rm" "-rf" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir"
$ "mkdir" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir"
$ ":" "RUN: at line 2"
$ "c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat" "--use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe" "-o" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir" "c:\src\llvm-project\out\gn\bin\clang.exe" "-S" "C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory1/file1.c" "C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory2/file2.c"
$ "c:\src\llvm-project\out\gn\bin\filecheck.exe" "C:\src\llvm-project\clang\test\Analysis\scan-build\exclude_directories.test" "-check-prefix" "CHECK-NO-EXCLUDE"
# command stderr:
C:\src\llvm-project\clang\test\Analysis\scan-build\exclude_directories.test:13:19: error: CHECK-NO-EXCLUDE: expected string not found in input
CHECK-NO-EXCLUDE: scan-build: 2 bugs found.
                  ^
<stdin>:2:1: note: scanning from here
C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build>perl -S scan-build --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\exclude_directories.test.tmp.output_dir c:\src\llvm-project\out\gn\bin\clang.exe -S C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory1/file1.c C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/multidirectory_project/directory2/file2.c 
^
<stdin>:5:1: note: possible intended match here
scan-build: No bugs found.
^

error: command failed with exit status: 1

--

********************
Testing:  0.
FAIL: Clang :: Analysis/scan-build/plist_output.test (1116 of 17064)
******************** TEST 'Clang :: Analysis/scan-build/plist_output.test' FAILED ********************
Script:
--
: 'RUN: at line 1';   rm -rf C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir && mkdir C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir
: 'RUN: at line 2';   'c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat' --use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe  -plist -o C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir c:\src\llvm-project\out\gn\bin\clang.exe -S C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\plist_output.test -check-prefix CHECK-STDOUT
: 'RUN: at line 12';   ls C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir/*/ | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\plist_output.test -check-prefix CHECK-FILENAMES
: 'RUN: at line 17';   cat C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir/*/report-*.plist      | c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\plist_output.test -check-prefix CHECK-REPORT-PLIST-CONTENTS
--
Exit Code: 2

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "rm" "-rf" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir"
$ "mkdir" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir"
$ ":" "RUN: at line 2"
$ "c:\src\llvm-project\clang\tools\scan-build\bin\scan-build.bat" "--use-analyzer=c:\src\llvm-project\out\gn\bin\clang.exe" "-plist" "-o" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir" "c:\src\llvm-project\out\gn\bin\clang.exe" "-S" "C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c"
$ "c:\src\llvm-project\out\gn\bin\filecheck.exe" "C:\src\llvm-project\clang\test\Analysis\scan-build\plist_output.test" "-check-prefix" "CHECK-STDOUT"
$ ":" "RUN: at line 12"
$ "ls" "C:\src\llvm-project\out\gn\obj\clang\test\Analysis\scan-build\Output\plist_output.test.tmp.output_dir/*/"
$ "c:\src\llvm-project\out\gn\bin\filecheck.exe" "C:\src\llvm-project\clang\test\Analysis\scan-build\plist_output.test" "-check-prefix" "CHECK-FILENAMES"
# command stderr:
FileCheck error: '<stdin>' is empty.
FileCheck command line:  c:\src\llvm-project\out\gn\bin\filecheck.exe C:\src\llvm-project\clang\test\Analysis\scan-build\plist_output.test -check-prefix CHECK-FILENAMES

error: command failed with exit status: 2

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60
FAIL: Clang :: SemaObjC/parameterized_classes_subst.m (11256 of 17064)
******************** TEST 'Clang :: SemaObjC/parameterized_classes_subst.m' FAILED ********************
Script:
--
: 'RUN: at line 1';   c:\src\llvm-project\out\gn\bin\clang.exe -cc1 -internal-isystem c:\src\llvm-project\out\gn\lib\clang\11.0.0\include -nostdsysteminc -fblocks -fsyntax-only -Wnullable-to-nonnull-conversion C:\src\llvm-project\clang\test\SemaObjC\parameterized_classes_subst.m -verify
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "c:\src\llvm-project\out\gn\bin\clang.exe" "-cc1" "-internal-isystem" "c:\src\llvm-project\out\gn\lib\clang\11.0.0\include" "-nostdsysteminc" "-fblocks" "-fsyntax-only" "-Wnullable-to-nonnull-conversion" "C:\src\llvm-project\clang\test\SemaObjC\parameterized_classes_subst.m" "-verify"
# command stderr:
error: 'error' diagnostics seen but not expected: 
  File C:\src\llvm-project\clang\test\SemaObjC\parameterized_classes_subst.m Line 479: type argument 'T' (aka 'id') does not satisfy the bound ('id<NSCopying>') of type parameter 'T'
error: 'note' diagnostics seen but not expected: 
  File C:\src\llvm-project\clang\test\SemaObjC\parameterized_classes_subst.m Line 475: type parameter 'T' declared here
2 errors generated.

error: command failed with exit status: 1

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 

Testing Time: 173.99s
********************
Failing Tests (5):
    Clang :: Analysis/scan-build/exclude_directories.test
    Clang :: Analysis/scan-build/html_output.test
    Clang :: Analysis/scan-build/plist_html_output.test
    Clang :: Analysis/scan-build/plist_output.test
    Clang :: SemaObjC/parameterized_classes_subst.m

  Expected Passes    : 16885
  Expected Failures  : 28
  Unsupported Tests  : 146
  Unexpected Failures: 5
FAILED: gen/clang/test/run-lit 
C:/src/depot_tools/bootstrap-3_8_0_chromium_8_bin/python/bin/python.exe bin/llvm-lit.py -sv ../../clang/test
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "../hack/llvmgnsyncbot/syncbot.py", line 233, in <module>
    sys.exit(main())
  File "../hack/llvmgnsyncbot/syncbot.py", line 228, in main
    run(args.last_exit)
  File "../hack/llvmgnsyncbot/syncbot.py", line 191, in run
    subprocess.check_call(['ninja', '-C', 'out/gn', test])
  File "C:\python_27_amd64\files\lib\subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja', '-C', 'out/gn', 'check-clang']' returned non-zero exit status 1

[ASDenysPetrov]

@NoQ
Yes, I have some thoughts. I am on revision master from [02.04.2020 10:29:20] (5 days ago). I've check these test and all passes on my PC.

As you can see in the log (No bugs found):

scan-build: Removing directory '/c/src/llvm-project/out/gn/obj/clang/test/Analysis/scan-build/Output/plist_html_output.test.tmp.output_dir/2020-04-07-090114-54706-1' because it contains no reports.
^
<stdin>:7:1: note: possible intended match here
scan-build: No bugs found.

That means that Inputs/single_null_dereference.c is somewhat differs from mine or clang behavior was changed since my revision. I will try to update sources and reproduce on the latest ones.

Do tests actually run on your machine when REQUIRES: shell is still in?

No they don't. They are marked as unsuported if REQUIRES: shell is in.

But what about SemaObjC/parameterized_classes_subst.m, I strongly doubt the reason is in this patch.

[ASDenysPetrov]

@NoQ
I've just pulled all from github and make a build. Then I ran the tests, and they successfully passed.
Namely, scan-build analyzed inputs and create stdout with diagnostics. Then filecheck checked without fails. But in case of buildbot it didn't create any diagnostic stdout.

According to buildbot's logs I assume that smth is wrong on its side.

[Charusso]

This comment was removed by Charusso.

[Charusso]

I believe it is very strange on a Windows system to have multiple dots in a file. The other issue could be the wildcard /*/ in a path full of \s. The LLVM lit (https://llvm.org/docs/CommandGuide/lit.html) has tons of Windows-related shortcuts, which I have never seen being used, but could be useful.

[thakis]

Thanks for checking. Let me try the tests locally over here tonight. I'll get back to you in a few hours.

But what about SemaObjC/parameterized_classes_subst.m, I strongly doubt the reason is in this patch.

That's a known unrelated flake, see thread for "[clang] a8c8b62 - [ObjC generics] Fix not inheriting type bounds in categories/extensions".

[ASDenysPetrov]

In D76768#1967129, @Charusso wrote:

I believe it is very strange on a Windows system to have multiple dots in a file. The other issue could be the wildcard /*/ in a path full of \s. The LLVM lit (https://llvm.org/docs/CommandGuide/lit.html) has tons of Windows-related shortcuts, which I have never seen being used, but could be useful.

I've checked. There is no problem with dots and /*/. My command promt (Win10) handles them perfectly well. Mixing / and \ in paths is also acceptable and handles correctly.
For instance, Visual Studio creates file with multiple dots for projects as "Project1.vcxproj.filters". Also you can easely create the same directory.
Wildcard /*/ is a part of input syntax of unix utility cat, thus it works as well.

Is there any way to get to buildbot file system to compare test files with what is in the master branch? Particularly C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c

[NoQ]

In D76768#1968023, @ASDenysPetrov wrote:

Is there any way to get to buildbot file system to compare test files with what is in the master branch? Particularly C:\src\llvm-project\clang\test\Analysis\scan-build/Inputs/single_null_dereference.c

If the source was checked out incorrectly then most likely everything would have been broken, not just our test. I'd rather suspect write permissions or something, but then again, a lot of tests create directories and then write into them. Dunno.

@thakis, do you know what is the setup of this buildbot? Like, what's so special about it? 'Cause i weren't seeing any failures from other windows buildbots. Can we discriminate between this buildbot and other buildbots with any UNSUPPORTED: bang (even if we have to add one)?

[ASDenysPetrov]

If the source was checked out incorrectly then most likely everything would have been broken, not just our test. I'd rather suspect write permissions or something, but then again, a lot of tests create directories and then write into them. Dunno.

No, I mean the source itself could be different.
I don't think permissions is the case, because we saw the log when scan-build reported: No bugs found..

[thakis]

Sorry for the slow reply. I repro'd this locally now, and here I see this error:

'''

command stderr:

'perl' is not recognized as an internal or external command,
operable program or batch file.
'''

perl wasn't required to run tests on Win for a long time. If we do want to run these tests on Windows, could we make the config lit file check if perl is available and set a feature and require that in the tests?

[ASDenysPetrov]

@thakis

'perl' is not recognized as an internal or external command,

I think you got another problem, that Perl isn't just presented in your PATH. It is not the case for buildbot logs.

perl wasn't required to run tests on Win for a long time. If we do want to run these tests on Windows, could we make the config lit file check if perl is available and set a feature and require that in the tests?

This is a good idea, to make tests more robust and friendly verbal.

[thakis]

In D76768#1971420, @ASDenysPetrov wrote:

@thakis

'perl' is not recognized as an internal or external command,

I think you got another problem, that Perl isn't just presented in your PATH. It is not the case for buildbot logs.

I tried again in a git bash shell where perl is present. With that, I get this error: http://codepad.org/t6ZhaCXI

"Can't find scan-build on PATH."

I guess that means that the bat file runs, but the perl -s scan-build therein fails? scan-build is right next to the bat file, but the perl process can't know the directory the .bat file is in. With this change it gets a bit farther:

-perl -S scan-build %*
+perl -S %~dp0\scan-build %*

(Since it works for others I probably misunderstood how things are supposed to work, and this change might be wrong.)

New error:

"""

command stderr:

Can't find c:\src\llvm-project\clang\tools\scan-build\bin\\scan-build on PATH, '.' not in PATH.
"""

If I also remove the -S switch, that goes away (what's supposed to add clang\tools\scan-build\bin to path? Or is it supposed to be copied to bin\ in the out dir? It looks like clang\tools\scan-build\CMakeList.txt has logic to copy the script around, but clang\test\CMakeLists.txt doesn't have a dependency on the scan-build target, so building check-clang doesn't guarantee that the script is copied. Maybe it's just this dependency that's missing?

[thakis]

If I add scan-build to clang/test/CMakeLists.txt and undo my other changes, the test passes for me. I guess you always build that target, which is why you didn't see this problem. But I wonder why this isn't needed on non-Windows.

[thakis]

(Also, that dep should only be added if CLANG_INSTALL_SCANBUILD isn't set to OFF, and I suppose the tests should only run if it isn't off too.)

[thakis]

In D76768#1974044, @thakis wrote:

If I add scan-build to clang/test/CMakeLists.txt and undo my other changes, the test passes for me. I guess you always build that target, which is why you didn't see this problem. But I wonder why this isn't needed on non-Windows.

On non-Windows, %scan-build expands to clang/tools/scan-build/bin/scan-build directly, and so non-Windows doesn't require scan-build on the PATH (i.e. in bin/ in the build dir) since there's no scan-build.bat trampoline.

I wonder if %scan-build should expand to "perl clang/tools/scan-build/bin/scan-build" on Windows, instead of to scan-build.bat, so that the Windows setup is more similar to the non-Windows setup.

[ASDenysPetrov]

@thakis
Let me explain what works for me.
This is in my PATH:

• C:\Perl\c\bin
• C:\Perl\perl\site\bin
• C:\Perl\perl\bin
• D:\llvm-project\buildn\bin

This is what in D:\llvm-project\buildn\bin (all this is a product of ninja+gcc):

• clang.exe
• FileCheck.exe
• scan-build.bat
• scan-build
• llvm-lit.py

No additional changes required (CMakeList.txt, bat editing, etc.).
Than I run command promt from arbitrary dir and run next: llvm-lit.py D:/llvm-project/clang/test/Analysis/scan-build/exclude_directories.test
Test passes. Done.
I hope it will help.

[thakis]

In D76768#1974255, @ASDenysPetrov wrote:

@thakis
Let me explain what works for me.
This is in my PATH:

• C:\Perl\c\bin
• C:\Perl\perl\site\bin
• C:\Perl\perl\bin
• D:\llvm-project\buildn\bin

^ This is not usually true. (But I think lit adds build/bin to the PATH for tests, so it's possibly true for tests, which would be enough.)

This is what in D:\llvm-project\buildn\bin (all this is a product of ninja+gcc):

• clang.exe
• FileCheck.exe
• scan-build.bat
• scan-build
• llvm-lit.py

No additional changes required (CMakeList.txt, bat editing, etc.).
Than I run command promt from arbitrary dir and run next: llvm-lit.py D:/llvm-project/clang/test/Analysis/scan-build/exclude_directories.test
Test passes. Done.

If you run del bin\* followed by make check-clang (or ninja check-clang or what have you), then I think bin/scan-build won't be built since it's not a dependency of the check-clang target. It doesn't have to be a dependency on non-Win, but on Win the execution flow is different due to the bat trampoline.

Please try that and confirm that you see a test failure in that scenario as well.

[ASDenysPetrov]

@thakis,

^ This is not usually true. (But I think lit adds build/bin to the PATH for tests, so it's possibly true for tests, which would be enough.)

AFAIK lit do not add anything to PATH. You should do it by yourself.

If you run del bin\* followed by make check-clang (or ninja check-clang or what have you), then I think bin/scan-build won't be built since it's not a dependency of the check-clang target. It doesn't have to be a dependency on non-Win, but on Win the execution flow is different due to the bat trampoline.

I did as you ask. Confirm your suggestion: scan-build have not appeared after ninja check-clang. All the tests except one passed. Log attached

log_ninja_check-clang.txt23 KBDownload

.
Intresting but scan-build tests haven't been run through ninja check-clang.

clang/test/Analysis/exploded-graph-rewriter/dynamic_types.cpp

 // FIXME: Figure out how to use %clang_analyze_cc1 with our lit.local.cfg.
 // RUN: %clang_cc1 -analyze -triple x86_64-unknown-linux-gnu \
 // RUN:                     -analyzer-checker=core \
 // RUN:                     -analyzer-dump-egraph=%t.dot %s
 // RUN: %exploded_graph_rewriter %t.dot | FileCheck %s
 // REQUIRES: asserts
 
-// FIXME: Substitution doesn't seem to work on Windows.
-// UNSUPPORTED: system-windows
-
 struct S {};
 
 void test() {
   // CHECK: Dynamic Types:
   // CHECK-SAME: <tr><td align="left"><table border="0"><tr>
   // CHECK-SAME:   <td align="left">HeapSymRegion\{conj_$1\{struct S *, LC1,
   // CHECK-SAME:       S{{[0-9]*}}, #1\}\}</td>
   // CHECK-SAME:   <td align="left">struct S</td>
   // CHECK-SAME: </tr></table></td></tr>
   new S;
 }

clang/test/Analysis/exploded-graph-rewriter/escapes.c

 // FIXME: Figure out how to use %clang_analyze_cc1 with our lit.local.cfg.
 // RUN: %clang_cc1 -analyze -triple x86_64-unknown-linux-gnu \
 // RUN:                     -analyzer-checker=core \
 // RUN:                     -analyzer-dump-egraph=%t.dot %s
 // RUN: %exploded_graph_rewriter %t.dot | FileCheck %s
 // REQUIRES: asserts
 
-// FIXME: Substitution doesn't seem to work on Windows.
-// UNSUPPORTED: system-windows
-
 void escapes() {
   // CHECK: <td align="left"><b>Store: </b> <font color="gray">(0x{{[0-9a-f]*}})</font></td>
   // CHECK-SAME: <td align="left">foo</td><td align="left">0</td>
   // CHECK-SAME: <td align="left">&amp;Element\{"foo",0 S64b,char\}</td>
   // CHECK: <td align="left"><b>Expressions: </b></td>
   // CHECK-SAME: <td align="left">"foo"</td>
   // CHECK-SAME: <td align="left">&amp;Element\{"foo",0 S64b,char\}</td>
   const char *const foo = "\x66\x6f\x6f";
 
   // CHECK: <font color="cyan4">BinaryOperator</font>
   // CHECK-SAME: <td align="left">1 \| 2</td>
   // CHECK-SAME: <td align="left">3 S32b</td>
   int x = 1 | 2;
 }

clang/test/Analysis/exploded-graph-rewriter/initializers_under_construction.cpp

 // FIXME: Figure out how to use %clang_analyze_cc1 with our lit.local.cfg.
 // RUN: %clang_cc1 -analyze -triple x86_64-unknown-linux-gnu \
 // RUN:                     -analyzer-checker=core \
 // RUN:                     -analyzer-dump-egraph=%t.dot %s
 // RUN: %exploded_graph_rewriter %t.dot | FileCheck %s
 // REQUIRES: asserts
 
-// FIXME: Substitution doesn't seem to work on Windows.
-// UNSUPPORTED: system-windows
-
 struct A {
   A() {}
 };
 
 struct B {
   A a;
   B() : a() {}
 };
 
 void test() {
   // CHECK: (construct into member variable)
   // CHECK-SAME: <td align="left">a</td>
   // CHECK-SAME: <td align="left">&amp;b.a</td>
   B b;
 }

clang/test/Analysis/exploded-graph-rewriter/lit.local.cfg

 # -*- Python -*-
 
 import lit.util
 import lit.formats
 import os
 
 use_lit_shell = os.environ.get("LIT_USE_INTERNAL_SHELL")
 config.test_format = lit.formats.ShTest(use_lit_shell == "0")
 
 config.substitutions.append(('%exploded_graph_rewriter',
                              '\'%s\' %s --dump-dot-only' % (
-                                 config.python_executable,
-                                 lit.util.which('exploded-graph-rewriter.py',
-                                                os.path.join(
-                                                    config.clang_src_dir,
-                                                    'utils', 'analyzer')))))
+                os.path.join(
+                  config.clang_src_dir,
+                  'utils', 'analyzer', 'exploded-graph-rewriter.py'))))
 
 config.suffixes.add('.dot')

clang/test/Analysis/exploded-graph-rewriter/macros.c

+#define NULL 0
+void *foo() {
+  return NULL;
+}
+
+// The code above shall go first, because check tags below are sensetive to the line numbers on which the code is placed.
+// You can change lines below in the way you need.
+
 // FIXME: Figure out how to use %clang_analyze_cc1 with our lit.local.cfg.
 // RUN: %clang_cc1 -analyze -triple x86_64-unknown-linux-gnu \
 // RUN:                     -analyzer-checker=core \
 // RUN:                     -analyzer-dump-egraph=%t.dot %s
 // RUN: %exploded_graph_rewriter %t.dot | FileCheck %s
 // REQUIRES: asserts
 
-// FIXME: Substitution doesn't seem to work on Windows.
-// UNSUPPORTED: system-windows
-
-// CHECK: macros.c:<b>17</b>:<b>10</b>
+// CHECK: macros.c:<b>3</b>:<b>10</b>
 // CHECK-SAME: <font color="royalblue1">
-// CHECK-SAME:   (<i>spelling at </i> macros.c:<b>15</b>:<b>14</b>)
+// CHECK-SAME:   (<i>spelling at </i> macros.c:<b>1</b>:<b>14</b>)
 // CHECK-SAME: </font>
-#define NULL 0
-void *foo() {
-  return NULL;
-}

clang/test/Analysis/exploded-graph-rewriter/objects_under_construction.cpp

 // FIXME: Figure out how to use %clang_analyze_cc1 with our lit.local.cfg.
 // RUN: %clang_cc1 -analyze -triple x86_64-unknown-linux-gnu \
 // RUN:                     -analyze-function "test()" \
 // RUN:                     -analyzer-checker=core \
 // RUN:                     -analyzer-dump-egraph=%t.dot %s
 // RUN: %exploded_graph_rewriter %t.dot | FileCheck %s
 // REQUIRES: asserts
 
-// FIXME: Substitution doesn't seem to work on Windows.
-// UNSUPPORTED: system-windows
-
 struct S {
   S() {}
 };
 
 void test() {
   // CHECK: Objects Under Construction:
   // CHECK-SAME: <tr>
   // CHECK-SAME:   <td align="left"><b>#0 Call</b></td>

clang/test/Analysis/scan-build/exclude_directories.test

-// FIXME: Actually, "perl".
-REQUIRES: shell
-
-// FIXME: Should ideally work on Windows.
-UNSUPPORTED: system-windows
-
 RUN: rm -rf %t.output_dir && mkdir %t.output_dir
 RUN: %scan-build -o %t.output_dir %clang -S \
 RUN:     %S/Inputs/multidirectory_project/directory1/file1.c \
 RUN:     %S/Inputs/multidirectory_project/directory2/file2.c \
 RUN:     | FileCheck %s -check-prefix CHECK-NO-EXCLUDE
 
 // The purpose of this test is to ensure that the --exclude command line option
 // actually excludes reports from inside the specified directories.

clang/test/Analysis/scan-build/help.test

-// FIXME: Actually, "perl".
-REQUIRES: shell
-
-// FIXME: Should ideally work on Windows.
-UNSUPPORTED: system-windows
-
 RUN: %scan-build -h | FileCheck %s
 RUN: %scan-build --help | FileCheck %s
 
 Test for help output from scan-build.
 
 
 CHECK: USAGE: scan-build [options] <build command> [build options]
 

clang/test/Analysis/scan-build/html_output.test

-// FIXME: Actually, "perl".
-REQUIRES: shell
-
-// FIXME: Should ideally work on Windows.
-UNSUPPORTED: system-windows
-
 RUN: rm -rf %t.output_dir && mkdir %t.output_dir
 RUN: %scan-build -o %t.output_dir %clang -S %S/Inputs/single_null_dereference.c \
 RUN:     | FileCheck %s -check-prefix CHECK-STDOUT
 
 // Test html output
 
 CHECK-STDOUT: scan-build: Using '{{.*}}' for static analysis
 CHECK-STDOUT: scan-build: 1 bug found.

clang/test/Analysis/scan-build/plist_html_output.test

-// FIXME: Actually, "perl".
-REQUIRES: shell
-
-// FIXME: Should ideally work on Windows.
-UNSUPPORTED: system-windows
-
 RUN: rm -rf %t.output_dir && mkdir %t.output_dir
 RUN: %scan-build -plist-html -o %t.output_dir %clang -S %S/Inputs/single_null_dereference.c \
 RUN:     | FileCheck %s -check-prefix CHECK-STDOUT
 
 // Test combined plist and html output with -plist-html
 
 CHECK-STDOUT: scan-build: Using '{{.*}}' for static analysis
 CHECK-STDOUT: scan-build: Analysis run complete.

clang/test/Analysis/scan-build/plist_output.test

-// FIXME: Actually, "perl".
-REQUIRES: shell
-
-// FIXME: Should ideally work on Windows.
-UNSUPPORTED: system-windows
-
 RUN: rm -rf %t.output_dir && mkdir %t.output_dir
 RUN: %scan-build -plist -o %t.output_dir %clang -S %S/Inputs/single_null_dereference.c \
 RUN:     | FileCheck %s -check-prefix CHECK-STDOUT
 
 // Test plist output
 
 CHECK-STDOUT: scan-build: Using '{{.*}}' for static analysis
 CHECK-STDOUT: scan-build: Analysis run complete.

clang/utils/analyzer/exploded-graph-rewriter.py

def add_raw_line(self, raw_line):
                 self.root_id = node_id
             # Note: when writing tests you don't need to escape everything,
             # even though in a valid dot file everything is escaped.
             node_label = result.group(2).replace('\\l', '') \
                                         .replace(' ', '') \
                                         .replace('\\"', '"') \
                                         .replace('\\{', '{') \
                                         .replace('\\}', '}') \
-                                        .replace('\\\\', '\\') \
+                                        .replace('\\\\"', '\\"') \
                                         .replace('\\|', '|') \
                                         .replace('\\<', '\\\\<') \
                                         .replace('\\>', '\\\\>') \
                                         .rstrip(',')
             logging.debug(node_label)
             json_node = json.loads(node_label)
             self.nodes[node_id].construct(node_id, json_node)
             return