VM Setup

Overview

This guide describes how to set up a testing environment for Popcorn Linux using qemu virtual machines. Basically, we will run two VMs - one for x86 and one for arm - on a x86 host, where the host and VMs are connected over Ethernet. To ease the testing, we will compile the kernel at the host (not inside of the VMs) and boot VMs using the kernel on the host's file system. The examples and commands are based on Debian 8 Jessie, but you may use Ubuntu as well.

--------------   --------------
|   Node 0   |   |   Node 1   |
|            |   |            |
|    x86     |   |    arm     |
| 10.4.4.100 |   | 10.4.4.101 |
--------------   --------------
|  qem-x86   |   |  qemu-arm  |
-------------------------------
|  linux-x86       linux-arm  |
|                             |   ------------
|         host (x86)          |   | Gateway  |
|          10.4.4.10          |---| 10.4.4.1 |--- Internet
-------------------------------   ------------

Prepare for the setup

The machine should have;
- At least 20 GB of storage
- 8 GB of RAM (qemu scripts below initiate VMs with 4 GB each. You can change -m option in the scripts to use different size of memory)
Let's start with updating apt repository and install some utilities:

$ sudo apt-get update
$ sudo apt-get install build-essential libssl-dev libncursesw5-dev git curl bc

Retrieve the popcorn kernel source code and download the VM image files. Place them in the same directory on the host. Rename the kernel source file accordingly after downloading.

$ git clone --depth=1 -b main --single-branch https://github.com/ssrg-vt/popcorn-kernel.git linux-x86
$ cp -r linux-x86 linux-arm/

Item	Filename	URL
x86 VM image	x86.img	https://drive.google.com/file/d/0B7RfKPGm2YZsaURxTVh5ZTMyTk0/view?usp=sharing&resourcekey=0-9mYvVcbEQdJ6lpavETXVVA
arm VM image	arm.img	https://drive.google.com/file/d/0B7RfKPGm2YZsaF9rdkZTLVAxX3M/view?usp=sharing&resourcekey=0-yFetJmokHSER3bBcAWcMRg

The directory should be as follows:

$ ls
arm.img   x86.img    linux-arm    linux-x86

Set up the host network interface

To provide network interfaces to the host and VMs at the same time, we make the host's NIC as a bridged interface and tap VMs' NICs to the bridged interface. We assume the host uses 10.4.4.10, x86 VM 10.4.4.100, and ARM VM 10.4.4.101 in 10.4.4.0/24 subnet. Change address, netmask, gateway and dns-nameservers fields accordingly if you want to use other IP addresses/subnet.

(Note if you are running VMs on a laptop with wireless NIC, the bridge may not work well. You may want to set up the VM network using tap.)

Install bridge-related utilities:

$ sudo apt-get install bridge-utils

Edit /etc/network/interfaces to setup the host's eth0 as a bridged interface:

$ sudo vim /etc/network/interfaces
        ...
auto eth0
iface eth0 inet manual

auto br0
iface br0 inet static
    address 10.4.4.10
    netmask 255.255.255.0
    gateway 10.4.4.1
    dns-nameservers 8.8.8.8
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0
        ...

Allow IP forwarding at the host

$ sudo vi /etc/sysctl.conf
...
net.ipv4.ip_forward = 1
...

$ sysctl -p /etc/sysctl.conf

Reload the NIC configuration.

$ sudo /etc/init.d/networking restart

NIC configuration should be like:

$ brctl show
 bridge name        bridge id                STP enabled       interfaces
 br0                8000.0cc47adec9ca        no                eth0

$ sudo ifconfig
br0       Link encap:Ethernet  HWaddr 0c:c4:7a:de:c9:ca
          inet addr:10.4.4.10  Bcast:10.4.4.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
        ...

eth0      Link encap:Ethernet  HWaddr 0c:c4:7a:de:c9:ca
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
        ...

Also, the host should be able to connect to the Internet as it was.

Set up the host network interface for Netplan

If you are using the new netplan daemon, you can update your file on /etc/netplan/*.yaml, add the ethernet and the bridge. You will need to do sudo netplan generate and sudo netplan apply after that.

Set up x86 virtual machine

Firstly, install the qemu package to emulate x86 systems:

$ sudo apt-get install qemu-system-x86

Qemu -kernel and -append options allows to boot VMs with a kernel image on the host's filesystem. This is very handy during the testing stage since we can instantly test a kernel by compiling it on the host (using all cores!) and booting VMs using the newly built kernel without additional copy/setup/update. Let's build the x86 kernel first at the host.
Create a working directory for x86 kernel and set to use the pre-made kernel configuration:

$ git clone https://github.com/ssrg-vt/popcorn-kernel.git linux-x86
$ cp linux-x86/kernel/popcorn/configs/config-x86_64-qemu linux-x86/.config

You can use your own configuration file, but MAKE SURE that the following configurations are disabled:
- CONFIG_SWAP
- CONFIG_TRANSPARENT_HUGEPAGE
- CONFIG_CMA
- CONFIG_MIGRATION
- CONFIG_COMPACTION
- CONFIG_KSM
- CONFIG_MEM_SOFT_DIRTY
- Page should be 4 KB.
  - ARM should select ARM64_4K_PAGES
  - PPC should use CONFIG_PPC_4K_PAGES.

For this purpose, you can run:

$./update_config.sh

Build the kernel:

$ make -C linux-x86 -j 8

Start the x86 virtual machine with:

$ sudo qemu-system-x86_64 \
    -enable-kvm -cpu host -smp 2 -m 4096 -no-reboot -nographic \
    -drive id=root,media=disk,file=x86.img \
    -net nic,macaddr=00:da:bc:de:00:13 -net tap \
    -kernel linux-x86/arch/x86/boot/bzImage \
    -append "root=/dev/sda1 console=ttyS0"

A VM will be started using the kernel image in linux-x86 directory and using the current terminal as its console. If the host does not support KVM, remove -enable-kvm option. The VM will automatically login into popcorn account. The password for popcorn and root are "popcorn" by default. Also, popcorn account is in the sudoer list.
Check that the kernel is the one that we built:

popcorn@x86:~$ uname -a
Linux x86 4.4.55-popcorn+ #1779 SMP Thu Nov 2 11:20:13 EDT 2017 x86_64 GNU/Linux

Check the network status:

popcorn@x86:~$ sudo ifconfig
eth0      Link encap:Ethernet  HWaddr 00:da:bc:de:00:13
          inet addr:10.4.4.100  Bcast:10.4.4.255  Mask:255.255.255.0
        ...

If the network interface is not shown, check /etc/udev/rules.d/70-persistent-net.rules so that ATTR{address} field matches to the MAC address given as the qemu option (macaddr=00:da:bc:de:00:13 in above). Also, you can edit /etc/network/interface to change the IP address.
You should be able to ssh to the VM from the host (the default password for popcorn/root is "popcorn"):

$ ssh [email protected]
        ...
popcorn@x86:~$

Now, the x86 VM is up and ready to use!

Set up ARM virtual machine

The overall procedure for ARM is very similar to that on x86. The only difference is, we need to cross-compile the ARM kernel from the host. To this end, install the crossbuild suite first, cross-compile the kernel, and boot the ARM VM using the kernel. https://wiki.debian.org/CrossToolchains describes the in-depth guide to install crossbuild suite in various Debian distributions. We will here follow the steps for Debian Jessie. Please refer to the Internet documents for installing the crossbuild toolchain on your Linux distribution.

On Ubuntu (x86-64), you can simply install the gcc-aarch64-linux-gnu package to get the toolchain. Congratulations!
Note: at the time of this note, emdebian.org is down. Thanks to cifranix for pointing this out.
Installing on Debian currently requires downloading archived packages from here
You can then install the .deb file. At the time of this writing the filename was crossbuild-essential-arm64_12.8_all.deb

$ sudo dpkg -i crossbuild-essential-arm64_12.8_all.deb
$ sudo apt-get install -f

Depreciated Instructions (for now,let us know if emdebian comes back online)

Installing on Debian is not that simple as Debian removed the suites from its repository. Firstly, create /etc/apt/sources.list.d/crosstools.list with the following repository info:

$ sudo echo "deb http://emdebian.org/tools/debian/ jessie main" > /etc/apt/sources.list.d/crosstools.list

Configure apt-get and install the suite:

$ curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | sudo apt-key add -
$ sudo dpkg --add-architecture arm64
$ sudo apt-get update
$ sudo apt-get install crossbuild-essential-arm64

OK, let's prepare for the kernel source and config for ARM:

$ git clone https://github.com/ssrg-vt/popcorn-kernel.git linux-arm
$ cp linux-arm/kernel/popcorn/configs/config-arm64-qemu linux-arm/.config

Crossbuild the kernel with ARCH="arm64". If it fails to locate the compiler, edit CROSS_CROSS_COMPILE entry in .config (currently it is set to aarch64-linux-gnu-)

$ ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" make menuconfig
$ ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" make -C linux-arm -j 8

In the meantime, let's install the qemu for arm:

$ sudo apt-get install qemu-system-arm

After the kernel build is completed, starting the VM:

$ sudo qemu-system-aarch64 \
    -machine virt -cpu cortex-a57 -m 4096 -nographic \
    -drive id=root,if=none,media=disk,file=arm.img \
    -device virtio-blk-device,drive=root \
    -netdev type=tap,id=net0 \
    -device virtio-net-device,netdev=net0,mac=00:da:bc:de:02:11 \
    -kernel linux-arm/arch/arm64/boot/Image \
    -append "root=/dev/vda console=ttyAMA0"

The VM will automatically login as popcorn. Check the kernel status:

popcorn@arm:~$ uname -a
Linux arm 4.4.55-popcorn+ #266 SMP Thu Nov 2 10:41:16 EDT 2017 aarch64 GNU/Linux

popcorn@arm:~$ sudo ifconfig
eth0      Link encap:Ethernet  HWaddr 00:da:bc:de:02:11
          inet addr:10.4.4.101  Bcast:10.4.4.255  Mask:255.255.255.0
        ...

Likewise x86, check MAC addresses in /etc/udev/rules.d/70-persistent-net.rules and the qemu option if the network interface is not shown up.

Set up PPC virtual machine

It is same to the ARM's case; crossbuild PPC kernel on x86 and run qemu with a set of proper configurations. PPC VM image is available at here

$ sudo apt-get install qemu-system-ppc
$ git clone [email protected]:ssrg-vt/popcorn-kernel.git linux-ppc
$ cp linux-ppc/kernel/popcorn/configs/config-ppc64le-qemu linux-ppc/.config
$ ARCH="powerpc" make -C linux-ppc -j 8

You can start the VM with:

$ sudo qemu-system-ppc64 -machine pseries-2.1 -smp cores=1,threads=1 -m 2048 \
	-nodefaults -nographic -serial mon:stdio \
	-kernel linux-ppc/arch/powerpc/boot/zImage \
	-append root=/dev/sda2 \
	-device spapr-vscsi \
	-drive id=root,media=disk,file=ppc.img \
	-netdev type=tap,id=net0 \
	-device e1000,netdev=net0,mac=00:da:bc:de:02:20

Setup the Popcorn messaging layer

Popcorn Linux is comprised of many subsystems and a messaging module. The messaging module contains the network and system configurations (in this example, a machine at 10.4.4.100 is node 0, a machine at 10.4.4.101 is node 1), and the rest of Popcorn subsystems are initialized only after the messaging module is successfully loaded. When we insert the messaging module into the kernel, Popcorn identifies itself (e.g., I am node 0 on x86 as my IP address is 10.4.4.100) and establishes connections to all nodes in the cluster (node 1 in this case). Currently such configurations are hardcoded in the ip_addresses[] array in msg_layer/config.h in the source code in Popcorn4.4. Pocorn5.2's configuration is in /etc/popcorn/nodes. All kernels in the same Popcorn "cluster" should have the identical configuration.

(Popcorn Linux4.4 only) Let check the configuration from linux-x86/msg_layer/config.h. Edit the IP addresses accordingly if you are using your IP address set:

        ...
const char *ip_addresses[] = {
    "10.4.4.100", /* x86 address */
    "10.4.4.101", /* arm address */
}
        ...

(Popcorn Linux5.2 only) Create a config file called "nodes" under "/etc/popcorn" on x86 server.

popcorn@x86:~$ cat /etc/popcorn/nodes
10.4.4.100
10.4.4.101

Compile the message layer module for x86:

$ make -C linux-x86 -j 8 msg_layer/msg_socket.ko
        ...
  LD [M]  msg_layer/msg_socket.ko

Copy the module to the x86 VM:

$ scp linux-x86/msg_layer/msg_socket.ko [email protected]:

Load the module at the x86 VM:

popcorn@x86:~$ sudo insmod ./msg_socket.ko
[   22.155240]  *  0: 10.4.4.100
[   22.156029]     1: 10.4.4.101
[   22.156744]

The message implies the Popcorn cluster is comprised of two nodes; node 0 at 10.4.4.100 and node 1 at 10.4.4.101, and the * in front of 0 indicates this is node 0.
Let's do the same for the ARM VM:

$ cp linux-x86/msg_layer/config.h linux-arm/msg_layer/
$ ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" make -C linux-arm -j 8 msg_layer/msg_socket.ko
$ scp linux-arm/msg_layer/msg_socket.ko [email protected]:

(Popcorn Linux5.2 only) Create a config file called "nodes" under "/etc/popcorn" on ARM server.

popcorn@arm:~$ cat /etc/popcorn/nodes
10.4.4.100
10.4.4.101

Load the module on the ARM VM:

popcorn@arm:~$ sudo insmod ./msg_socket.ko
[   48.433928]     0: 10.4.4.100
[   48.434146]  *  1: 10.4.4.101
[   48.434278]
[   48.465160]   1 identified as arm
[   48.465382]   0 identified as x86
popcorn@arm:~$

You could see the similar message from x86 console

[  132.347344]   0 identified as x86
[  132.348349]   1 identified as arm

Congratulation! Now, the VMs are connected over the messaging layer, and ready to migrate threads!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly