This is an archive of the discontinued LLVM Phabricator instance.

[ARM64] Add QEMU testing environment setup guide for SVE testing
ClosedPublic

Authored by omjavaid on Jun 17 2020, 5:17 PM.

Details

Summary

This patch adds a HowTo document to lldb docs which gives instruction on how to setup QEMU environment for modern Arm64 features not released in publicly available hardware till date.

More generally this howto can be applied to other architecture types supported by QEMU emulator. This may help setup a lldb test environment for all supported architecture/platform types.

Diff Detail

Event Timeline

omjavaid created this revision.Jun 17 2020, 5:17 PM
rovka added a subscriber: rovka.Jun 25 2020, 4:19 AM

Hi Omair,

If this is intended to be more generally useful, I wonder if we should rename it to something else (e.g. lldb-qemu-howto.txt) and only use AArch64 SVE as a case study.

In any case, thanks for writing this up!

lldb/docs/lldb-qemu-aarch64-sve-howto.txt
5 ↗(On Diff #271536)

Nitpick: "in an emulation environment in the absence of actual [...]

7 ↗(On Diff #271536)

Isn't this guide still useful after the patches are merged?

9 ↗(On Diff #271536)

Typo: all the instructions

19 ↗(On Diff #271536)

Would it be better for this to be an actual script in lldb/utils or somewhere similar?

78 ↗(On Diff #271536)

Is this difficult to infer from QEMU's own docs? If not, maybe just add a link and mention only what's specific to this guide (I'm guessing the target-list).

96 ↗(On Diff #271536)

Same as above, could this be just a link to the official docs?

124 ↗(On Diff #271536)

Same as step 1: can this live in utils?

176 ↗(On Diff #271536)

Nitpick: For Internet access [...]

217 ↗(On Diff #271536)

For consistency, maybe there should be some commands here, or a link to the docs for cross-compiling.

Hi Omair,

If this is intended to be more generally useful, I wonder if we should rename it to something else (e.g. lldb-qemu-howto.txt) and only use AArch64 SVE as a case study.

In any case, thanks for writing this up!

Thanks Rovka, I ll generalize this doc and correct the pointed issues in next update.

omjavaid updated this revision to Diff 278128.Jul 15 2020, 3:21 AM

Updated after incorporating review comments.

@rovka I gave your idea a thought about moving included scripts to lldb/utils. IMO these scripts have no relationship with LLDB and no such precedent exist where we uploaded such tools or utilities in LLDB source tree. Lets have other opinions and finalize.

Updated after incorporating review comments.

@rovka I gave your idea a thought about moving included scripts to lldb/utils. IMO these scripts have no relationship with LLDB and no such precedent exist where we uploaded such tools or utilities in LLDB source tree. Lets have other opinions and finalize.

The relationship to lldb is that it can be used to setup lldb for cross-testing. And I personally don't see a difference between a standalone bash script stashed somewhere inside the repo, and having that script embedded inside a .txt file. The bar for stashing a script into the repo is fairly low, and this seems useful enough to cross it, particularly if it can handle architectures other than arm (I do like the idea of making the description generic, and then mention arm/sve specific in a separate section or file).

lldb/docs/lldb-qemu-aarch64-sve-howto.txt
39–40 ↗(On Diff #278128)

Why not just RFS_IMG_SIZE_GB=$2, etc., without all the shifts?

omjavaid updated this revision to Diff 286161.Aug 17 2020, 3:50 PM

In light of review comments I have come up with this new sceheme for setting up QEMU VM based testing environment for LLDB.

We now have three scripts:

  1. setup.sh is used to build Linux kernel images for Arm and AArch64. Also it builds QEMU from latest sources.
  2. rootfs.sh can generate Ubuntu rootf file system images for our QEMU VMs
  3. run-qemu.sh puts it all together and uses kernel image, rootfs image and arch information to run a QEMU VM. We currently support Arm and AArch64 architectures and more architectures can be added later.

This can generates a QEMU testing setup in less than 10 minutes on a host machine with reasonable resources.

@labath @rovka thoughts?

You can buy a PRIMEHPC FX700 from Fujitsu. It has a A64FX CPU with SVE.

I like this. Apart from the inline comments (which are mostly about removing excessive mentions of arm(64) -- I want to make that text feel generic, since the file is in a generic place), I have one high-level comment. Given the current layout of the repo, it seems like these scripts would be better placed under lldb/scripts than lldb/tools. E.g., this scripts are fairly similar to the macos-setup-codesign.sh script, which also lives in that folder.

lldb/tools/lldb-test-qemu/README.txt
2 ↗(On Diff #286161)
9–14 ↗(On Diff #286161)
16 ↗(On Diff #286161)
20 ↗(On Diff #286161)
22 ↗(On Diff #286161)
lldb/tools/lldb-test-qemu/rootfs.sh
57–65 ↗(On Diff #286161)

You're only passing these to qemu-debootstrap. It sounds like things could "just work" for a bunch of other distro/arch combinations if we just removed these checks.

lldb/tools/lldb-test-qemu/run-qemu.sh
83–87 ↗(On Diff #286161)

It would be nice if this searched for the qemu binary on the PATH if it was not explicitly provided. Something like:

if [ -d qemu.git ]; then
  QEMU_BIN=$(pwd)/qemu.git/arm-softmmu/qemu-system-arm
else
  QEMU_BIN=qemu-system-arm
fi
99–101 ↗(On Diff #286161)

How about we check for this in the parsing while loop? That would mean the --arch argument has to come before --sve, but that does not seem like a bad thing

I didn't look at the scripts in detail but I have a few general (high level) comments:

  • I also feel this should go under lldb/scripts.
  • Can we put the documentations as RST under lldb/docs so it's rendered on the website? We already have bunch of developer resources there. That will certainly help with discovery.
  • LLVM generally encourages scripts to be written in Python. While it doesn't look like this will work out of the box on Windows, I can imagine someone might be interested in that down the line. I also think the bar is lower for new contributors to write Python than bash. I'm kind of on the fence here because there's a reasonable balance between just running commands and control flow, but then I've been scarred by the build-script-impl disaster in Swift so I lean towards better safe than sorry.

I didn't look at the scripts in detail but I have a few general (high level) comments:

  • I also feel this should go under lldb/scripts.
  • Can we put the documentations as RST under lldb/docs so it's rendered on the website? We already have bunch of developer resources there. That will certainly help with discovery.
  • LLVM generally encourages scripts to be written in Python. While it doesn't look like this will work out of the box on Windows, I can imagine someone might be interested in that down the line. I also think the bar is lower for new contributors to write Python than bash. I'm kind of on the fence here because there's a reasonable balance between just running commands and control flow, but then I've been scarred by the build-script-impl disaster in Swift so I lean towards better safe than sorry.

These scripts fairly optional and are more like developer help for someone who wants to give a quick test to any of LLVM tools on some target in absence of hardware and I agree someone may wanna use them on windows and quite likely I might be that someone in very near future. But I ll hold moving to python for now. I ll write an rst doc and arrange the scripts under lldb/scripts as suggested.

omjavaid updated this revision to Diff 290189.Sep 7 2020, 12:00 AM

This diff addresses issues highlighted in review.

@labath any further thoughts?

JDevlieghere added inline comments.Sep 9 2020, 9:38 AM
lldb/docs/resources/test.rst
436

What's the reason you chose to refer to the README in the source tree instead of converting it to an RST page and linking to that page from here? I think it's good to have an entry here rather than the whole document, but I still think this should be a hyperlink to its own page.

omjavaid added inline comments.Sep 14 2020, 12:20 AM
lldb/docs/resources/test.rst
436

Well honestly this was more of me going lazy I ll make it a separate rst page if that sounds more appropriate.

omjavaid updated this revision to Diff 293136.Sep 21 2020, 4:15 AM

This adds a rst page qemu-testing.rst that describes content which was previously written in README.txt.

Ping!

LGTM or any further comments or suggestions?

PS: I have been on holidays so apologies for late responses in past few week.

labath accepted this revision.Oct 5 2020, 2:37 AM

This looks fine to me. Please wait a bit to give Jonas the opportunity to comment. (FWIW, I think these are fine as shell scripts right now, but that may change if windows comes into play.)

This revision is now accepted and ready to land.Oct 5 2020, 2:37 AM