-
-
Notifications
You must be signed in to change notification settings - Fork 605
Modularization
By design, OSv has always been a "fat" unikernel and by default has provided a large subset of glibc functionality and has included full standard C++ library (libstdc++), the ZFS implementation, drivers for many devices, and has supported many hypervisors. On one hand, it makes running arbitrary applications on any hypervisor very easy using a single universal kernel. On another hand, such universality comes with the price of the bloated kernel with many symbols and drivers and possibly ZFS unused, thus causing inefficient memory usage, longer boot time, and potential security vulnerabilities. In addition, the C++ applications linked against a version of libstdc++ different than the version the kernel was linked against, may simply not work.
To address these issues, release 0.57 introduced a new experimental build mode to hide the non-glibc symbols and libstdc++ and extracted ZFS code out of the kernel in form of a dynamically linked library. It also provided another build option to tailor the kernel to a set of specific device drivers - 'driver profiles', and finally allowed building a version of the kernel with a subset of glibc symbols needed to support a specific application. The following paragraphs explain these changes and features in more detail.
As issue 97 explains, OSv by default exports all symbols including full standard C++ library. This has many negative implications - overly large kernel ELF, thus slower boot time, and potential for a collision of the kernel symbols with the application ones. On top of this, the C++ applications may not work if linked with the version of libstdc++ different than the OSv kernel.
To address these, release 0.57 added a new build option conf_hide_symbols that would hide most non-glibc and the libstdc++ symbols if enabled. In essence, most source files except the ones under the musl/ and libc/ directories would be compiled with the special compiler flags - -fvisibility=hidden and -fvisibility-inlines-hidden (C++ only) if conf_hide_symbols enabled. On another hand, the symbols to be exposed as public (like the glibc ones), would be annotated with OSV_***_API macros that translate to __attribute__ ((visibility ("default"))). Finally, in order to remove all unneeded code - "garbage", all files would be compiled with the flags -ffunction-sections -fdata-sections and linked with the flag --gc-sections. For more details about symbol visibility please read this article.
./scripts/build image=native-example fs=rofs conf_hide_symbols=1 -j$(nproc)
The kernel ELF file, built with most symbols hidden as above is roughly 3.6M in size, compared to 6M - a reduction of ~ 40%. This great reduction stems from the fact that the libstdc++ is no longer linked with --whole-archive, the symbol table is much smaller and unsued code is garbage collected. Please note the resulting kernel is still universal as it exports all glibc symbols and includes all device drivers.
The conf_hide_symbols is disabled by default because there are still some OSv modules like httpserver-api and more importantly some unit tests that depend on some internal C++ ABI of OSv. For more details look at this issue.
The list of the public symbols exported by the kernel is actually enforced during the build process based on the symbol list files for each advertised library like libc.so.6 located and maintained under the directory /exported_symbols (for more details look at this readme). These files are concatenated using scripts/generate_version_script.sh to the $(out)/version_script and fed to the linker as an argument to --version-script flag.
For more insight into the technical details, please look at this commit and that one.
ZFS is one of the three filesystems that OSv implements. The original code comes from FreeBSD circa 2014 and has been since integrated and adapted to work within OSv. ZFS is one of the best file systems for persistent mutable data and what one needs to run mysql or elasticsearch on OSv for example. Although a lot of effort has gone into integrating ZFS into OSv, its sheer code base volume made OSv kernel bigger than it needs to be and there are a plethora of stateless use cases like redis, memcached, microservices, etc where ZFS is not needed and instead we can use ROFS now.
To allow to optionally include ZFS, we extracted the ZFS code out of the kernel in form of a dynamically shared library (aka module) as part of the 0.57 release. The ZFS-less kernel is now roughly 800K smaller, thus making OSv images smaller, using less memory, and booting slightly faster. The ZFS library - libsolaris.so - can be loaded from the ROFS filesystem and the ZFS filesystem mounted on the same or different disk as explained in this wiki and illustrated by this example:
# include zfs module that puts /usr/lib/fs/libsolaris.so on RoFS filesystem
./scripts/build image=native-example,zfs fs=rofs_with_zfs
Please look at this commit to better understand how ZFS code got extracted from the kernel and this one to get insight into the logic behind loading the libsolaris.so and mounting the ZFS filesystem.
This patch introduces new build mechanism that allows creating custom kernel with specific list of device drivers intended to target given hypervisor. Such kernel benefits from smaller size and better security as all unneeded code is removed.
In essence, we introduce new build script and makefile parameter:
drivers_profile. This new parameter is intended to specify a
drivers profile which is simply a list of device drivers to be linked
into kernel with some extra functionality like PCI or ACPI these drivers
depend on. Each profile is specified in a tiny make include file (.mk)
under new conf/profiles/$(arch) directory and included by the main
makefile as requested by drivers_profile parameter. The main makefile
has number of new ifeq conditions that add given driver object file
to the linked objects list depending on the value (0 or 1) of given
conf_drivers_ variable specified in the relevant profile file.
Sometimes it is necessary to conditionally enable/disable given
code depending on the drivers selected. The good example of it is
arch-setup.cc which actually registers individual drivers and this
is where we need some kind of #if-way of registering given driver.
To that end, this patch adds new script gen-drivers-config-header and
new rule to the makefile, which automatically generates driver-config.h
header file under build/$(mode)/gen/include/osv. The driver-config.h
is comprised of the #define CONF_drivers_* macros that specify if given
driver is enabled or not (1, 0) and is included by relatively few source
file like arch-setup.cc. The extra benefit of this approach is that
every time we change value of drivers_profile, all relevant files are
recompiled and new kernel linked.
See https://github.com/cloudius-systems/osv/commit/d19ccb1cde100ab4a5c8e6db9a0d69560cabbd04
This patch introduces another new build mechanism that allows creating custom kernel exporting only symbols required by specific application. Such kernel benefits from smaller size and better security as all unneeded code is removed. This patch addresses remaining part of the modularization/librarization functionality as explained by the issue https://github.com/cloudius-systems/osv/issues/1110 and this part of the roadmap - https://github.com/cloudius-systems/osv/wiki/Roadmap#modularizationlibrarization. This idea was also mentioned in the P99 OSv presentation - see slide 12.