This is an archive of the discontinued LLVM Phabricator instance.

[analyzer] NFC: Update test infrastructure to support multiple constraint managers
ClosedPublic

Authored by ddcc on Feb 24 2017, 10:48 PM.

Details

Summary

Replace calls to %clang/%clang_cc1 with %clang_analyze_cc1 when invoking static analyzer, and perform runtime substitution to select the appropriate constraint manager, per D28952.

Diff Detail

Repository
rL LLVM

Event Timeline

ddcc created this revision.Feb 24 2017, 10:48 PM
dcoughlin accepted this revision.Feb 26 2017, 6:42 PM

LGTM! Thanks!

This revision is now accepted and ready to land.Feb 26 2017, 6:42 PM
This revision was automatically updated to reflect the committed changes.
ddcc reopened this revision.Feb 26 2017, 8:10 PM

Looks like there were four latent test failures that had been masked due to the way they were written. Here's the new version.

This revision is now accepted and ready to land.Feb 26 2017, 8:10 PM
ddcc updated this revision to Diff 89833.Feb 26 2017, 8:10 PM

Fix four test failures

This revision was automatically updated to reflect the committed changes.
ddcc reopened this revision.EditedFeb 27 2017, 7:36 PM

It looks like there's some sort of pickling-related test failure when lit tries to fork, because it can't find the class lit.TestingConfig.AnalyzerTest: http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/6154/steps/test/logs/stdio . I'm guessing that this has something to do with the AnalyzerTest class being loaded dynamically from lit.local.cfg , but short of moving it out into llvm/utils/lit/lit/formats, I have no idea how to fix this. Also, I can't reproduce this problem locally with ninja -j4 check-clang. @dcoughlin do you have any suggestions?

This revision is now accepted and ready to land.Feb 27 2017, 7:36 PM

Saving the relevant portion of the build bot log output before it disappears from the bot:

0.000 [0/1/0] Running all regression tests
-- Testing: 31894 tests, 80 threads --
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 381, in main
    self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1384, in load
    return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 864, in load
    dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 886, in load_eof
    raise EOFError
EOFError
lit.py: C:/Buildbot/Slave/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/llvm.src/tools/clang/test/lit.cfg:200: note: using clang: 'C:/Buildbot/Slave/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/llvm.obj/./bin/clang.EXE'
Traceback (most recent call last):
  File "C:/Buildbot/Slave/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/llvm.src/utils/lit/lit.py", line 6, in <module>
    main()
  File "C:\Buildbot\Slave\llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast\llvm.src\utils\lit\lit\main.py", line 161, in main
    main_with_tmp(builtinParameters)
  File "C:\Buildbot\Slave\llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast\llvm.src\utils\lit\lit\main.py", line 484, in main_with_tmp
    opts.useProcesses)
  File "C:\Buildbot\Slave\llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast\llvm.src\utils\lit\lit\run.py", line 272, in execute_tests
    queuer.start()
  File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 425, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 554, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 606, in save_list
    self._batch_appends(iter(obj))
  File "C:\Python27\lib\pickle.py", line 639, in _batch_appends
    save(x)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 731, in save_inst
    save(stuff)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 731, in save_inst
    save(stuff)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 396, in save_reduce
    save(cls)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 754, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <class 'lit.TestingConfig.AnalyzerTest'>: it's not found as lit.TestingConfig.AnalyzerTest

I'm not sure what is going on.

One thing to try is defining the class in clang/test/lit.cfg -- which seems to work for Swift. (See https://github.com/apple/swift/blob/master/test/lit.cfg#L137) We'd still want to only use that class in Analysis/lit.local.cfg.

Another possibility is to not support running with Z3 on Windows, at least for now. So in Analysis/lit.local.cfg we would only set the custom test_format if (not sys.platform in ['win32']) is true.

ddcc added a comment.Mar 1 2017, 7:37 AM

With the first method, I'm not sure how to refer to the AnalyzerTest class defined in lit.cfg from lit.local.cfg. It doesn't seem to be in scope, so unless I store an instantiation in the config object, I don't think it's accessible.

In D30373#689613, @ddcc wrote:

With the first method, I'm not sure how to refer to the AnalyzerTest class defined in lit.cfg from lit.local.cfg. It doesn't seem to be in scope, so unless I store an instantiation in the config object, I don't think it's accessible.

Rather than creating and storing an instance of AnalyzerTest in lit.cfg you could store the class itself (that is, an instance of the metatype) and then instantiate it using that when needed in lit.local.cfg. This would avoid creating an long-held, global instance of AnalyzerTest when running other tests.

ddcc updated this revision to Diff 90386.Mar 2 2017, 1:43 PM

Move AnalyzerTest to lit.cfg

Thanks for sticking with this!!

This looks good to me, although I would prefer if the definition of the AnalyzerTest class were moved to the bottom of the lit.cfg file. The other configuration and substitutions are far more important to the rest of clang -- people shouldn't have to scan through analyzer-specific stuff to get to configuration of broad relevance.

I also think you may end up needing to conditionalize setting config.analyzer_format to only occur on non-win32 -- but you can commit and see if it triggers the bots.

ddcc updated this revision to Diff 90403.Mar 2 2017, 2:55 PM

Move definition to bottom of lit.cfg

This revision was automatically updated to reflect the committed changes.
ddcc reopened this revision.Mar 2 2017, 3:10 PM

You're right about the failure being specific to Windows, I'll roll back to the original implementation in lit.local.cfg and just skip on Windows.

This revision is now accepted and ready to land.Mar 2 2017, 3:10 PM
ddcc updated this revision to Diff 90408.Mar 2 2017, 3:14 PM

Rollback to lit.local.cfg implementation, disable on Windows

This revision was automatically updated to reflect the committed changes.
ddcc reopened this revision.Mar 2 2017, 3:49 PM

Made a mistake with the last commit, I believe this should be fine now?

This revision is now accepted and ready to land.Mar 2 2017, 3:49 PM
ddcc updated this revision to Diff 90411.Mar 2 2017, 3:49 PM

Fix Windows check

Looks great. Thanks!!

This revision was automatically updated to reflect the committed changes.