AArch64

The AArch64 Port of OSv is ongoing, initially targeting the QEMU Mach-virt platform, running on the ARM Foundation Model v8 or on the APM X-Gene Mustang development board.

Functional Status

mainline contains AArch64 support for the loader image (loader.img), which means it is possible to embed simple programs inside the loader itself. Manual modification of the bootfs.manifest.skeleton is necessary for non-tools (tools/ are added automatically).

Only one CPU is bootable, and there are some limitations, mostly in the libc support but also elsewhere, and you can read the details below.

The latest AArch64 work is available in the aarch64-next branch at:

https://github.com/hw-claudio/osv_aarch64 "aarch64-next"

https://github.com/hw-claudio/osv_aarch64/tree/aarch64-next

The AArch64 branch still requires QEMU patches to be useful, until PCI support for ARM reaches upstream QEMU. There are two patchsets floating on qemu-devel, both functional and working with this OSv branch. One of them is from Alexander Graf, and includes PCI-Express ECAM support, another is from Alvise Rigo, which instead supports standard CAM.

You can find a working QEMU tree which includes all necessary patches at:

https://github.com/hw-claudio/qemu-aarch64-queue "pci-on-machvirt"

https://github.com/hw-claudio/qemu-aarch64-queue/tree/pci-on-machvirt

In order to build the final usr.img there are limitations of the current build system which need to be addressed first.

Most of the problems are due to the build step which requires to run an OSv VM in order to build the OSv with the ZFS file system, but there other major issues,

including the framework loosely defined by the python scripts in scripts/ , which contain functionality which is used both as part of the build (and thus meant for the build host), and as part of the runtime (and thus meant for the target). The framework therefore ends up being x86-specific and cross-compilation unfriendly.

To address these challenges, a user-space ZFS image creation tool is being worked on, which will enable building the ZFS-based image without running OSv,

and then the scripts should be reworked (or avoided as much as possible) for the usr.img creation.

For development purposes, you can find a pre-built tentative usr.img image to use for virtio-blk and ZFS mounting tests at:

https://github.com/hw-claudio/osv_aarch64.git "usr.img"

In the usr.img branch, look for a file in the top source directory called "usr.img.aarch64"

Component Status

build system: the first pass (loader.img) mostly works through cross-compilation, but there are issues to proceed any further as mentioned. Work is ongoing on the ZFS part of the picture.

qemu-system-aarch64 tcg software system emulation: currently the AArch64 image runs on the Foundation Model but also successfully on the QEMU tcg software system emulation. This will require some changes in the near future as/if QEMU starts supporting EL2/EL3 and starts the machine by default at that exception level. The changes will involve mostly setting up some registers and then dropping to EL1.

tests with available hardware (development boards): preliminary tests with the hardware currently available has been done, in particular with the APM X-Gene Mustang board.

external dependencies: Avi has successfully added the Fedora packages for AArch64, and they seem ok, although they still contain broken stap information, causing warnings.

devices support: preliminary support for PCI is available in the aarch64-next branch. virtio-net, virtio-rng and virtio-blk are enabled in this experimental branch. To merge with mainline we need QEMU to support PCI for ARM, which is slowly progressing.

smp support: currently only one CPU can boot. Having multiple cpus boot already requires many dependencies (possibly ACPI/PSCI) and work. Running smp will trigger bugs and missing functionality which will need to be sorted out.

libc: We need to implement setjmp, longjmp, ucontext and architecture signal code, and also the broken/missing floating point support for the libc math functions which depend on floating point representation.

musl: also related to the preceeding point, as libc is implemented partly with musl, partly with own code for the OSv-specific parts. We need a true full AArch64 port for musl. There have been two attempts already, but nothing has landed in mainline musl yet. This would need to target latest musl, as currently used musl version does not include support for the needed ld128 floating point representation. Integrating latest musl needs work due to interface changes, for both x64 and AArch64.

hardware information passing from the host to the guest is currently based on device trees, with fallback defaults for the mach-virt platform. For the long term we Need: ACPI from QEMU. This is slowly progressing thanks to the efforts of Shannon Zhao (from Huawei) and Red Hat, but it will still take quite some time before it lands in mainline.

ELF64: initial relocations for get_init() are supported, plus basic relocs necessary to run applications. Additional relocations will be implemented if/when we hit missing ones while enabling more and more applications for AArch64.

console: pl011 UART output and input is now available, no FIFO (or, FIFO depth = 1).

backtrace: basic functionality now available.

power: currently stubbed, depends on the ACPI and PSCI work.

exception handling: basic infrastructure there, but need work for sync exceptions to differentiate page_faults from other sync aborts.

MMU: done, but need to revisit for the sigfault detection speedup feature added to x64 (and stubbed on AArch64). We also need to clean up parameter passing through the MMU code, and remove some code duplication.

page fault and VMA handling: basic functionality now available.

interrupts: basic functionality now available.

GIC: more or less done, v2. Note that we don't support GICv2m or GICv3, so we cannot have MSI or MSI-X at the moment.

Generic Timers: basic functionality available, needs a patch for the host bootwrapper for the foundation models (CNTFRQ_EL0 needs to be set to the correct value in the Makefile => 100Mhz)

scheduling: basic support for task switching available (switch-to-first, switch-to)

signals: work started.

arch trace: nothing available.

sampler: sampler support missing for AArch64.

scripts: most scripts have not even been looked at for AArch64

management tools: management tools have not been looked at yet for AArch64

tests: some tests build but most don't because of other missing components. No attempt to run any tests beside tst-hello.so.

string optimizations: imported from newlib (based on BSD-licensed Linaro patches). This includes memcpy, memset and memcpy_backwards, slightly changed from the original due to different API entry point.

hypervisor and firmware detection: not a priority, not implemented.

Build instructions

These are some brief instructions about how to cross-compile OSv's loader.img (very incomplete still) on an X86-64 build host machine.

At the time of writing this, the available functionality is minimal: the loader image boots, gic is initialized, timers are initialized, etc, and a simple hello world application is started on top of OSv (in the case of aarch64-next), or you will get an abort with a backtrace in zfs (for master).

Crosscompiler Tools and Host Image from Linaro

You can find a 32bit (needs multilib 😟) crosscompiler from Linaro, in particular the package

gcc-linaro-aarch64-linux-gnu-4.8-2013.12_linux, which is not distro-specific 😃, and includes all tools needed for building.

http://releases.linaro.org/13.12/components/toolchain/binaries

For the host root filesystem for AArch64, a good option is the Linaro LEG Image

linaro-image-leg-java-genericarmv8-20131215-598.rootfs.tar.gz

http://releases.linaro.org/13.12/openembedded/aarch64/

You can experiment with other images and compilers from Linaro, but those I am using right now.

Crosscompiler Tools for Ubuntu

For Ubuntu there are AArch64 crosscompilers available in the official repositories as well. Packages are named g++-4.8-aarch64-linux-gnu and gcc-4.8-aarch64-linux-gnu.

Ubuntu is also used over here, and works ok.

Preparing the AArch64 Host

You will need to have or build an AArch64 linux kernel for the Host, which will be run on top of the Foundation v8 Model. In addition to that, you will need the bootwrapper, which you can get from:

http://git.kernel.org/pub/scm/linux/kernel/git/cmarinas/boot-wrapper-aarch64.git

Use the foundation-v8.dts, and my suggestion is to use nfsroot to mount the root filesystem (the linaro LEG image). The boot wrapper takes as input the linux kernel Image and produces linux-system.axf, which will be the input for the Foundation Model.

Running ARMv8 Foundation Model

Start the Foundation model:

./Foundation_v8 --image=linux-system.axf --cores=1 --network=nat --network-nat-ports=1234=1234

The latter option will expose the 1234 port on the host side to the same port number in the guest running inside the model. You can add additional mappings as desired/needed.

If you are skipping the user space initialization via something like init=/bin/sh for speedup (edit the bootwrapper Makefile), in Foundation model you might need to run:

/sbin/udhcpc eth0

Preparing the guest: External dependencies

Nothing to do anymore, since they are now part of the mainline tree.

Preparing the guest: Environment Variables for make

In addition to the general requirements for building OSv (see README.md),

note that the simple build system recognizes the ARCH and CROSS_PREFIX environment variables, and looks for the following build tools:

CXX=$(CROSS_PREFIX)g++
CC=$(CROSS_PREFIX)gcc
LD=$(CROSS_PREFIX)ld
STRIP=$(CROSS_PREFIX)strip
OBJCOPY=$(CROSS_PREFIX)objcopy
HOST_CXX=g++

In order to build AArch64, countrary to the past when the target architecture was automatically detected by running the supplied compiler, you need to explicitly say make ARCH=aarch64, otherwise the build system will try to detect ARCH running uname on the host machine, and try to build x64.

At the beginning of the build process, look for this message:

build.mk:
build.mk: building arch=aarch64, override with ARCH env
build.mk:

If the message does not say arch=aarch64, the crosscompiler could not be found or run correctly. In this case, check the CROSS_PREFIX variable, or the compiler binary name if it's not canonical (do you need to add a symlink for example from g++-4.8.3 to g++ ?).

Running the guest

An example of command line for QEMU which works running on top of Foundation Model with kvm with an AArch64 qemu-system-aarch64 binary is:

$ qemu-system-aarch64 -nographic -M virt -enable-kvm \
    -kernel ./loader.img -cpu host -m 1024M

An example of command line for QEMU running on system emulation on an x86_64 host with an x86_64 qemu-system-aarch64 binary is:

$ qemu-system-aarch64 -nographic -M virt \
    -kernel ./loader.img -cpu cortex-a57 -m 1024M

Jani Kokkonen <[email protected]>
Claudio Fontana <[email protected]>