Revert "Follow up to e05c10380ce7: add aarch64 to test XFails"
This reverts commit 4fd29e4fd3c5490b658f4b467d94aeed6b3d1ac3.
There's a report in D95617 that this is failing on what (I think?) is an
aarch64 bot, which should be covered by the XFail list... reverting this
follow-up and the base patch until I work out what's wrong here.