[ARM64] Add QEMU testing environment setup guide for SVE testing

[omjavaid]

Summary

This patch adds a HowTo document to lldb docs which gives instruction on how to setup QEMU environment for modern Arm64 features not released in publicly available hardware till date.

More generally this howto can be applied to other architecture types supported by QEMU emulator. This may help setup a lldb test environment for all supported architecture/platform types.

[rovka]

Hi Omair,

If this is intended to be more generally useful, I wonder if we should rename it to something else (e.g. lldb-qemu-howto.txt) and only use AArch64 SVE as a case study.

In any case, thanks for writing this up!

lldb/docs/lldb-qemu-aarch64-sve-howto.txt

6

Nitpick: "in an emulation environment in the absence of actual [...]

8

Isn't this guide still useful after the patches are merged?

10

Typo: all the instructions

20

Would it be better for this to be an actual script in lldb/utils or somewhere similar?

79

Is this difficult to infer from QEMU's own docs? If not, maybe just add a link and mention only what's specific to this guide (I'm guessing the target-list).

97

Same as above, could this be just a link to the official docs?

125

Same as step 1: can this live in utils?

177

Nitpick: For Internet access [...]

218

For consistency, maybe there should be some commands here, or a link to the docs for cross-compiling.

[omjavaid]

In D82064#2113851, @rovka wrote:

Hi Omair,

If this is intended to be more generally useful, I wonder if we should rename it to something else (e.g. lldb-qemu-howto.txt) and only use AArch64 SVE as a case study.

In any case, thanks for writing this up!

Thanks Rovka, I ll generalize this doc and correct the pointed issues in next update.

[omjavaid]

Updated after incorporating review comments.

@rovka I gave your idea a thought about moving included scripts to lldb/utils. IMO these scripts have no relationship with LLDB and no such precedent exist where we uploaded such tools or utilities in LLDB source tree. Lets have other opinions and finalize.

[labath]

In D82064#2152807, @omjavaid wrote:

Updated after incorporating review comments.

@rovka I gave your idea a thought about moving included scripts to lldb/utils. IMO these scripts have no relationship with LLDB and no such precedent exist where we uploaded such tools or utilities in LLDB source tree. Lets have other opinions and finalize.

The relationship to lldb is that it can be used to setup lldb for cross-testing. And I personally don't see a difference between a standalone bash script stashed somewhere inside the repo, and having that script embedded inside a .txt file. The bar for stashing a script into the repo is fairly low, and this seems useful enough to cross it, particularly if it can handle architectures other than arm (I do like the idea of making the description generic, and then mention arm/sve specific in a separate section or file).

lldb/docs/lldb-qemu-aarch64-sve-howto.txt

39–40

Why not just RFS_IMG_SIZE_GB=$2, etc., without all the shifts?

[omjavaid]

In light of review comments I have come up with this new sceheme for setting up QEMU VM based testing environment for LLDB.

We now have three scripts:

1. setup.sh is used to build Linux kernel images for Arm and AArch64. Also it builds QEMU from latest sources.
2. rootfs.sh can generate Ubuntu rootf file system images for our QEMU VMs
3. run-qemu.sh puts it all together and uses kernel image, rootfs image and arch information to run a QEMU VM. We currently support Arm and AArch64 architectures and more architectures can be added later.

This can generates a QEMU testing setup in less than 10 minutes on a host machine with reasonable resources.

@labath @rovka thoughts?

[tschuett]

You can buy a PRIMEHPC FX700 from Fujitsu. It has a A64FX CPU with SVE.

[labath]

I like this. Apart from the inline comments (which are mostly about removing excessive mentions of arm(64) -- I want to make that text feel generic, since the file is in a generic place), I have one high-level comment. Given the current layout of the repo, it seems like these scripts would be better placed under lldb/scripts than lldb/tools. E.g., this scripts are fairly similar to the macos-setup-codesign.sh script, which also lives in that folder.

lldb/tools/lldb-test-qemu/README.txt

2 ↗

(On Diff #286161)

9–14 ↗

(On Diff #286161)

16 ↗

(On Diff #286161)

20 ↗

(On Diff #286161)

22 ↗

(On Diff #286161)

lldb/tools/lldb-test-qemu/rootfs.sh

57–65 ↗

(On Diff #286161)

You're only passing these to qemu-debootstrap. It sounds like things could "just work" for a bunch of other distro/arch combinations if we just removed these checks.

lldb/tools/lldb-test-qemu/run-qemu.sh

83–87 ↗

(On Diff #286161)

It would be nice if this searched for the qemu binary on the PATH if it was not explicitly provided. Something like:

if [ -d qemu.git ]; then
  QEMU_BIN=$(pwd)/qemu.git/arm-softmmu/qemu-system-arm
else
  QEMU_BIN=qemu-system-arm
fi

99–101 ↗

(On Diff #286161)

How about we check for this in the parsing while loop? That would mean the --arch argument has to come before --sve, but that does not seem like a bad thing

[JDevlieghere]

I didn't look at the scripts in detail but I have a few general (high level) comments:

• I also feel this should go under lldb/scripts.
• Can we put the documentations as RST under lldb/docs so it's rendered on the website? We already have bunch of developer resources there. That will certainly help with discovery.
• LLVM generally encourages scripts to be written in Python. While it doesn't look like this will work out of the box on Windows, I can imagine someone might be interested in that down the line. I also think the bar is lower for new contributors to write Python than bash. I'm kind of on the fence here because there's a reasonable balance between just running commands and control flow, but then I've been scarred by the build-script-impl disaster in Swift so I lean towards better safe than sorry.

[omjavaid]

In D82064#2223991, @JDevlieghere wrote:

I didn't look at the scripts in detail but I have a few general (high level) comments:

• I also feel this should go under lldb/scripts.
• Can we put the documentations as RST under lldb/docs so it's rendered on the website? We already have bunch of developer resources there. That will certainly help with discovery.
• LLVM generally encourages scripts to be written in Python. While it doesn't look like this will work out of the box on Windows, I can imagine someone might be interested in that down the line. I also think the bar is lower for new contributors to write Python than bash. I'm kind of on the fence here because there's a reasonable balance between just running commands and control flow, but then I've been scarred by the build-script-impl disaster in Swift so I lean towards better safe than sorry.

These scripts fairly optional and are more like developer help for someone who wants to give a quick test to any of LLVM tools on some target in absence of hardware and I agree someone may wanna use them on windows and quite likely I might be that someone in very near future. But I ll hold moving to python for now. I ll write an rst doc and arrange the scripts under lldb/scripts as suggested.

[omjavaid]

This diff addresses issues highlighted in review.

@labath any further thoughts?

[JDevlieghere]

lldb/docs/resources/test.rst

427 ↗

(On Diff #290189)

What's the reason you chose to refer to the README in the source tree instead of converting it to an RST page and linking to that page from here? I think it's good to have an entry here rather than the whole document, but I still think this should be a hyperlink to its own page.

[omjavaid]

lldb/docs/resources/test.rst

427 ↗

(On Diff #290189)

Well honestly this was more of me going lazy I ll make it a separate rst page if that sounds more appropriate.

[omjavaid]

This adds a rst page qemu-testing.rst that describes content which was previously written in README.txt.

[omjavaid]

Ping!

LGTM or any further comments or suggestions?

PS: I have been on holidays so apologies for late responses in past few week.

[labath]

This looks fine to me. Please wait a bit to give Jonas the opportunity to comment. (FWIW, I think these are fine as shell scripts right now, but that may change if windows comes into play.)

lldb/docs/lldb-qemu-aarch64-sve-howto.txt

+################################################################################
+LLDB Testing on AArch64 SVE Linux using QEMU system mode emulation
+################################################################################
+
+QEMU can be used to test LLDB in an emulation environment in the absence of
+actual hardware. This write up will help setup a QEMU environment for testing
+AArch64 features like SVE, MTE, Pointer Authentication etc. This write can be
+used as reference to setup similar testing environment for other architectures
+supported by QEMU emulator.
+
+Ubuntu Bionic/Focal x86_64 host was used to test all the instructions in this
+document. Please update it according to your host distribution/architecture.
+################################################################################
+# STEP 1
+################################################################################
+# Bash script for creating Debian/Ubuntu Linux RootFS for QEMU system emulation
+# Example usage:
+# To create an img file of size 8 GB containing Ubuntu Bionic arm64 rootFS
+# > bash create-qemu-rootfs.sh bionic-arm64 8G bionic arm64
+################################################################################
+#!/bin/bash
+
+# Prerequisites:
+sudo apt-get install debootstrap qemu-user-static schroot qemu-utils
+
+if [ $# -gt 3 ]; then
+  echo "Your command line contains $# arguments"
+else
+  echo "Invalid or no arguments"
+  echo "Usage example: create-rootfs-qemu.sh focal-arm64 8G focal arm64"
+  echo "focal-arm64 is image name"
+  echo "8G is image size in Gigabytes"
+  echo "focal is distro name"
+  echo "arm64 is rootFS architecture"
+  exit
+fi
+
+RFS_IMG_NAME=$1
+shift
+RFS_IMG_SIZE_GB=$1
+shift
+BOOTSTRAP_DISTRO=$1
+shift
+BOOTSTRAP_ARCH=$1
+
+echo "Root FS image name ... $RFS_IMG_NAME"
+echo "Root FS size in GB ... $RFS_IMG_SIZE_GB"
+echo "Root FS bootstrap distro name ... $BOOTSTRAP_DISTRO"
+echo "Root FS bootstrap distro architecture... $BOOTSTRAP_ARCH"
+
+qemu-img create $RFS_IMG_NAME.img $RFS_IMG_SIZE_GB
+
+mkfs.ext4 $RFS_IMG_NAME.img
+mkdir $RFS_IMG_NAME.dir
+sudo mount -o loop $RFS_IMG_NAME.img $RFS_IMG_NAME.dir
+
+sudo qemu-debootstrap --arch $BOOTSTRAP_ARCH $BOOTSTRAP_DISTRO $RFS_IMG_NAME.dir
+
+sudo chroot $RFS_IMG_NAME.dir locale-gen en_US.UTF-8
+
+sudo chroot $RFS_IMG_NAME.dir sed -i \
+'s/main/main restricted multiverse universe/g' /etc/apt/sources.list
+
+sudo chroot $RFS_IMG_NAME.dir sed -i '$ a\nameserver 8.8.8.8' /etc/resolv.conf
+
+sudo chroot $RFS_IMG_NAME.dir apt update
+sudo chroot $RFS_IMG_NAME.dir apt -y install ssh bash-completion
+sudo chroot $RFS_IMG_NAME.dir adduser --gecos "" $USER
+sudo chroot $RFS_IMG_NAME.dir adduser $USER sudo
+sudo umount $RFS_IMG_NAME.dir
+rmdir $RFS_IMG_NAME.dir
+
+################################################################################
+#  End of STEP 1
+################################################################################
+
+################################################################################
+# STEP 2
+#
+# Build QEMU from source for AArch64 system-mode emulation
+#
+# Please visit https://wiki.qemu.org/Hosts/Linux for instructions
+# on how to build QEMU from source.
+#
+# In order to enable AArch64 system by QEMU please run configure QEMU resources
+# with following arguments: ./configure --target-list=aarch64-softmmu
+#
+# Please run ./configure --help inside QEMU source directory for a complete
+# list of supported emulation targets.
+################################################################################
+
+# Install dependencies
+sudo apt-get build-dep qemu
+sudo apt install libsdl2-dev
+
+# Checkout source code
+git clone git://git.qemu.org/qemu.git qemu.git
+
+# Configure and build
+cd qemu.git
+./configure --target-list=aarch64-softmmu
+make
+
+################################################################################
+# STEP 3
+#
+# Cross compile Linux kernel
+################################################################################
+
+# Install dependencies
+sudo apt install gcc-arm-linux-gnueabihf gcc-aarch64-linux-gnu
+
+# Checkout source code, select branch and clean up source directory
+git clone  \
+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux.git
+
+cd linux.git
+make mrproper
+
+# kernel_branch=master arch=arm64 config=defconfig
+make O=../linux.build/master/arm64 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
+defconfig
+
+make O=../linux.build/master/arm64 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
+
+
+################################################################################
+# STEP 4
+#
+# AArch64 SVE System Mode Emulation
+#
+# Bash script given below boots our newly built AArch64 Linux kernel using
+# RootFS img file created in step 1.
+################################################################################
+#!/bin/bash
+
+if [ $# -gt 2 ]; then
+  echo "Your command line contains $# arguments"
+else
+  echo "Invalid or no arguments"
+  echo "Usage example: sudo bash run-qemu-emulation.sh \
+./qemu.git/aarch64-softmmu/qemu-system-aarch64 \
+./linux.build/master/arm64/ ./bionic-arm64.img"
+  echo "First argument is .. qemu-system-aarch64"
+  echo "Second argument is .. Linux kernel build directory"
+  echo "Third argument is .. RootFS image file"
+  exit
+fi
+
+QEMU_AARCH64=$1
+shift
+LINUX_BUILD=$1
+shift
+QEMU_RFS_IMG=$1
+
+echo "QEMU AArch64 system emulation executable ... $QEMU_AARCH64"
+echo "Root FS image name ... $QEMU_RFS_IMG"
+echo "Linux ARM64 build directory ... $LINUX_BUILD"
+
+LINUX_DTBS=$LINUX_BUILD/arch/arm64/boot/dts/
+
+QEMU_MACHINE=virt
+QEMU_CPU=max
+QEMU_CORES=2
+QEMU_MEMORY=2048
+QEMU_GDBSERVER_HOST_PORT=54321
+QEMU_GDBSERVER_VM_PORT=54321
+QEMU_SSH_HOST_PORT=5555
+
+
+$QEMU_AARCH64 \
+-kernel $LINUX_BUILD/arch/arm64/boot/Image \
+-M $QEMU_MACHINE \
+-cpu $QEMU_CPU \
+-machine type=$QEMU_MACHINE \
+-smp $QEMU_CORES \
+-m $QEMU_MEMORY \
+-drive file=$QEMU_RFS_IMG,index=0,media=disk,format=raw \
+-append "root=/dev/vda rw ip=dhcp mem=1024M raid=noautodetect  \
+crashkernel=128M rootwait console=ttyAMA0 devtmpfs.mount=0" \
+-nographic \
+-net nic -net user \
+-nic user,hostfwd=tcp::$QEMU_GDBSERVER_HOST_PORT-:$QEMU_GDBSERVER_VM_PORT,hostfwd=tcp::$QEMU_SSH_HOST_PORT-:22
+
+###############################################################################
+# Your default username will be the one with whom you created RootFS image.
+# For internet access inside your emulation environment run: sudo dhclient enp0s2.
+################################################################################
+# Alternatively setup a tap device on host computer by following this link:
+# https://wiki.qemu.org/Documentation/Networking/NAT
+#
+# Run QEMU emulation using commandline given below:
+
+$QEMU_AARCH64 \
+-kernel $LINUX_BUILD/arch/arm64/boot/Image \
+-M $QEMU_MACHINE \
+-cpu $QEMU_CPU,sve-max-vq=$QEMU_SVE_MAX_VQ \
+-machine type=$QEMU_MACHINE \
+-smp $QEMU_CORES \
+-m $QEMU_MEMORY \
+-drive file=$QEMU_RFS_IMG,index=0,media=disk,format=raw \
+-append "root=/dev/vda rw ip=dhcp mem=1024M raid=noautodetect crashkernel=128M rootwait console=ttyAMA0 devtmpfs.mount=0" \
+-nographic \
+-net tap -net nic
+
+################################################################################
+#  End of STEP 4
+################################################################################
+
+################################################################################
+# STEP 5
+# Running lldb-server in QEMU emulation environment
+################################################################################
+
+# Login emulation environment and install dependencies
+sudo apt install python-dev libedit-dev libncurses5-dev libexpat1-dev
+
+# Setup ssh access between host machine and emulation environment
+# Make sure br0 is enabled on host machine by running:
+ip add
+
+# Find out ip address assigned to eth0 in emulation environment
+ip add
+
+# Verify ssh access
+ssh username@ip-address-of-emulation-environment
+
+# Cross compile LLDB server for AArch64 Linux
+# Please visit https://lldb.llvm.org/resources/build.html for instructions
+# on how to cross compile LLDB server.
+
+# Transfer LLDB server executable to emulation environment
+scp lldb-server username@ip-address-of-emulation-environment:/home/username