Filesystems

OSv supports a variety of filesystems which are described in the below paragraphs. Layers-wise, it comes with the VFS layer (see fs/vfs/*) and particular filesystem implementations found under fs/**/* with the exception of ZFS.

Root Filesystem

During boot time, OSv initially mounts the BootFS filesystem (see vfs_init() and mount_rootfs()) and then proceeds to mount and pivot to a 'real' filesystem like RoFS, ZFS or VirtioFS (for details see this code in loader.cc) unless the --nomount kernel option was specified. The root filesystem can be explicitly selected using the --rootfs option, otherwise, the loader will try to discover it by trying RoFS, then Virtio-FS, and eventually ZFS.

Please note that OSv also supports the /etc/fstab file where one could add an extra filesystem mount point. In addition, one can mount an extra filesystem by supplementing appropriate options to the command line like so:

./scripts/run.py --execute='--rootfs=rofs --mount-fs=zfs,/dev/vblk0.2,/data /hello

In order to build an image with a specific type of filesystem, you need to specify the fs option (which defaults to zfs) like so:

./scripts/build image=tests fs=rofs

RamFS

RoFS

ZFS

The ZFS code has been based on the FreeBSD implementation as of circa 2014 and since adapted to work in OSv. The ZFS is a sophisticated filesystem that traces its roots in Solaris and you can find some resources about it on this Wiki page. The majority of the ZFS code can be found under the subtree bsd/sys/cddl/. The ZFS filesystem driver has been fairly recently extracted from the kernel as a separate shared library libsolaris.so which is dynamically loaded during boot time from a different filesystem (most likely BootFS or RoFS) before ZFS filesystem can be mounted.

There are three ways ZFS can be mounted on OSv:

1. The first and the original one assumes mounting ZFS at the root (/) from the 1st partition of the 1st disk - /dev/vblk0.1.
2. The second one involves mounting ZFS from the 2nd partition of the 1st disk - /dev/vblk0.2 at an arbitrary non-root mount point, for example /data.
3. Similarly, the third way involves mounting ZFS from the 1st partition of the 2nd or higher disk - for example /dev/vblk1.1 at an arbitrary non-root mount point as well. Please note that both the second and third options assume that the root filesystem is non-ZFS - most likely RoFS or Virtio-FS.

The disadvantage of the 1st option is that the code and data live in the same read-write filesystem, whereas the other two options allow one to isolate code from mutable data. Ideally one would put all code and configuration on RoFS partition, colocated on the same disk (2) or not (3), and mutable data on the separate partition on the same disk (2) or different (3). It has been shown that booting and mounting ZFS from a separate disk is also slightly faster (by 30-40ms) compared to the original option 1.

Below are the examples of building and running OSv with ZFS

ZFS mounted at /

This is the original and default method. Please note the libsolaris.so is part of the loader.elf and loaded from BootFS which makes the kernel larger.

./scripts/build image=native-example fs=zfs #The fs defaults to zfs

./scripts/run.py

OSv v0.56.0-152-gfd716a77
...
devfs: created device vblk0.1 for a partition at offset:4194304 with size:532676608
virtio-blk: Add blk device instances 0 as vblk0, devsize=536870912
...
zfs: driver has been initialized!
VFS: mounting zfs at /zfs
zfs: mounting osv/zfs from device /dev/vblk0.1
...

ZFS mounted from 2nd partition

This is a fairly new method that allows ZFS to be mounted at a non-root mount point like /data for example and mixed with another filesystem on the same disk. Please note that libsolaris.so is placed on a root filesystem (typically RoFS) under /usr/lib/fs/ and loaded from it automatically. However, to mount the ZFS filesystem one has to use the --mount option or add an appropriate entry to the /etc/fstab.

./scripts/build image=native-example,zfs fs=rofs_with_zfs #Has to add zfs module that adds /usr/lib/fs/libsolaris.so to RoFS 

./scripts/run.py --execute='--rootfs=rofs --mount-fs=zfs,/dev/vblk0.2,/data /hello'

OSv v0.56.0-152-gfd716a77
...
devfs: created device vblk0.1 for a partition at offset:4194304 with size:191488
devfs: created device vblk0.2 for a partition at offset:4385792 with size:532676608
virtio-blk: Add blk device instances 0 as vblk0, devsize=537062400
...
VFS: mounting rofs at /rofs
zfs: driver has been initialized!
VFS: initialized filesystem library: /usr/lib/fs/libsolaris.so
VFS: mounting devfs at /dev
VFS: mounting procfs at /proc
VFS: mounting sysfs at /sys
VFS: mounting ramfs at /tmp
VFS: mounting zfs at /data
zfs: mounting osv/zfs from device /dev/vblk0.2
...

ZFS mounted from different disk

This fairly new method is similar to the above, in that it also allows ZFS to be mounted at a non-root mount point like /data but this time from a different disk. Please note that libsolaris.so is placed on a root filesystem (typically RoFS) under /usr/lib/fs/ and loaded from it automatically as well. Similar to the above, to mount the ZFS filesystem one has to use the --mount option or add an appropriate entry to the /etc/fstab.

./scripts/build image=native-example,zfs fs=rofs --create-zfs-disk #Creates empty disk at build/last/zfs_disk.img with ZFS filesystem

./scripts/run.py --execute='--rootfs=rofs --mount-fs=zfs,/dev/vblk1.1,/data /hello' --second-disk-image build/last/zfs_disk.img

OSv v0.56.0-152-gfd716a77
...
devfs: created device vblk0.1 for a partition at offset:4194304 with size:1010688
virtio-blk: Add blk device instances 0 as vblk0, devsize=5204992
devfs: created device vblk1.1 for a partition at offset:512 with size:536870400
virtio-blk: Add blk device instances 1 as vblk1, devsize=536870912
...
VFS: mounting rofs at /rofs
zfs: driver has been initialized!
VFS: initialized filesystem library: /usr/lib/fs/libsolaris.so
VFS: mounting devfs at /dev
VFS: mounting procfs at /proc
VFS: mounting sysfs at /sys
VFS: mounting ramfs at /tmp
VFS: mounting zfs at /data
zfs: mounting osv/zfs from device /dev/vblk1.1
...

Creating ZFS disk on host

Please note that in the examples above, the ZFS pool and filesystem are created using the zfs_loader.elf version of OSv that executes zpool.so, zfs.so and cpiod.so among others. This is actually quite fast and efficient but recently we have enhanced the build mechanism to create ZFS disks using the zpool and zfs on a Linux host provided you have OpenZFS installed (see more info here). To that end, there is a fairly new script zfs-image-on-host.sh that can be used to either mount an existing OSv ZFS disk or create a new one. The latter actually can be orchestrated by the build script if one passes the option --use-openzfs like so:

./scripts/build image=native-example fs=zfs -j$(nproc) --use-openzfs

Some help from the zfs-image-on-host.sh:

Manipulate ZFS images on host using OpenZFS - mount, unmount and build.

Usage: zfs-image-on-host.sh mount <image_path> <partition> <pool_name> <filesystem> |
                            build <image_path> <partition> <pool_name> <filesystem> <populate_image> |
                            unmount <pool_name>

Where:
  image_path      path to a qcow2 or raw ZFS image; defaults to build/last/usr.img
  partition       partition of disk above; defaults to 1
  pool_name       name of ZFS pool; defaults to osv
  filesystem      name of ZFS filesystem; defaults to zfs
  populate_image  boolean value to indicate if the image should be populated with content
                  from build/last/usr.manifest; defaults to true but only used with 'build' command

Examples:
  zfs-image-on-host.sh mount                                     # Mount OSv image from build/last/usr.img under /zfs
  zfs-image-on-host.sh mount build/last/zfs_disk.img 1           # Mount OSv image from build/last/zfs_disk.img 2nd partition under /zfs
  zfs-image-on-host.sh unmount                                   # Unmount OSv image from /zfs

VirtioFS

NFS