[lldb] Catch all exceptions when checking if simulator exists

[augusto2112]

Summary

Some lldb tests are failing intermittently due to the check for simulators failing for some reason unrelated to the test. Catch any exceptions thrown by calling the xcodebuild subprocess invoked, report them to the user, and skip the test

[teemperor]

Do we know what the exception that is being thrown here? FWIW, except Exception will catch everything, even completely bogus things such as type errors, calling functions with the wrong arguments and things that should probably be a hard failures instead of marking the test as unsupported (which is completely silent).

[augusto2112]

The error that happens in xcodebuild is:

2021-08-30 16:54:51.925 xcodebuild[95554:469108]  DVTDownloadable: Failed to get authorization with error Error Domain=DVTDownloadableErrors Code=3 "The authorization was denied since no user interaction was possible." UserInfo={AuthorizationErrorCode=-60007, NSLocalizedDescription=The authorization was denied since no user interaction was possible.}
xcodebuild: error: The authorization was denied since no user interaction was possible.

But I'm not sure what python exception is being thrown (full logs here). I could add a try/except block around only the check_output call so we don't anything extra, or I could try changing the exception we catch from subprocess.CalledProcessError to subprocess.SubprocessError, which is the base class of the subprocess exceptions, although I'm not sure if this would fix it.

[teemperor]

Weird, in the test log it seems xcodebuild fails, but that would just raise a CalledProcessError (which we already catch here and cause the test to skip). And this would also not print anything to stderr as we set the stderr output in the check_output call to /dev/null? But instead of aborting or skipping the test, we're actually running the test and then failing to parse the simctl output in the test it self (and that's why the test fails):

ERROR: test_simulator_ostype_ios_debugserver (TestAppleSimulatorOSType.TestAppleSimulatorOSType)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildnode/jenkins/workspace/oss-lldb-5.5-incremental-macos-cmake/llvm-project/lldb/packages/Python/lldbsuite/test/tools/lldb-server/gdbremote_testcase.py", line 52, in test_method
    return attrvalue(self)
  File "/Users/buildnode/jenkins/workspace/oss-lldb-5.5-incremental-macos-cmake/llvm-project/lldb/packages/Python/lldbsuite/test/decorators.py", line 136, in wrapper
    return func(*args, **kwargs)
  File "/Users/buildnode/jenkins/workspace/oss-lldb-5.5-incremental-macos-cmake/llvm-project/lldb/test/API/tools/lldb-server/TestAppleSimulatorOSType.py", line 120, in test_simulator_ostype_ios
    self.check_simulator_ostype(sdk='iphonesimulator',
  File "/Users/buildnode/jenkins/workspace/oss-lldb-5.5-incremental-macos-cmake/llvm-project/lldb/test/API/tools/lldb-server/TestAppleSimulatorOSType.py", line 22, in check_simulator_ostype
    sim_devices = json.loads(sim_devices_str)['devices']
  File "/Applications/Xcode-beta.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/Applications/Xcode-beta.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Applications/Xcode-beta.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Config=x86_64-/Users/buildnode/jenkins/workspace/oss-lldb-5.5-incremental-macos-cmake/Ninja-ReleaseAssert+stdlib-Release/llvm-macosx-x86_64/bin/clang

Not sure if I'm looking at a different implementation than we're running on jenkins, but I don't see how we can end up with this xcodebuild error that would end up in stderr and that would still allow the test to run. Any idea?

[augusto2112]

My initial thought was that xcodebuild is throwing some other error (maybe, for some reason, it times out, which is a subprocess.TimeoutExpired exception), which is why my solution was to catch any exception thrown by subprocess.check_output.

Many the security agent is the one outputting the error, and xcodebuild just returns as if everything is fine? I could change the affected test and add a try/catch around the json parsing, and abort the test if that fails.

[aprantl]

What do you think about putting the json.loads(sim_devices_str)['devices'] into the decorator, so we know that xcodebuild has an SDK *and* the simulator has devices available?

lldb/packages/Python/lldbsuite/test/decorators.py

def should_skip_simulator_test():
             DEVNULL = open(os.devnull, 'w')
             output = subprocess.check_output(["xcodebuild", "-showsdks"], stderr=DEVNULL).decode("utf-8")
             if re.search('%ssimulator' % platform, output):
                 return None
             else:
                 return "%s simulator is not supported on this system." % platform
         except subprocess.CalledProcessError:
             return "Simulators are unsupported on this system (xcodebuild failed)"
+        except Exception as e:
+            return "Simulator check failed with exception: %s" % e
 
     return skipTestIfFn(should_skip_simulator_test)
 
 
 def debugserver_test(func):
     """Decorate the item as a debugserver test."""
     return add_test_categories(["debugserver"])(func)