Skip to content

design

Jump to bottom
Baptiste Daroussin edited this page Sep 19, 2014 · 7 revisions

Design of poudriere

Poudriere is a port builder tool, it has 2 main frontends:

  • testport
  • bulk

Both are designed to build packages in a clean room, and are able to handle both the legacy pkg_install and the new pkgng.

The code is written 100% in shell script and has very few dependencies out of base.

List of dependencies that should be installed on the box running poudriere:

Port Optional Description
devel/subversion yes Used to be able to get the sources and ports from the FreeBSD svn
devel/git yes Used to be able to get the ports from the FreeBSD github mirror
ports-mgmt/portlint yes Used for testport or bulk -t to run portlint prior to building the packages
ports-mgmt/pkg yes Used for sanity checking a pkgng repository (incremental building)

Poudriere is composed of jails and ports tree, it associates them to build packages.

Data

Jails

The poudriere jail is designed to handle the jails themself.

Configuration of a jail is stored in /usr/local/etc/poudriere.d/jails/name/*

With the following files in the directory

File Description
arch Architecture of the jail
fs zfs dataset path if zfs is in use
method the method used to get the "sources"
mnt the path to the root of the jail on the host filesystem
version the version of the jail

Creating a jail

Creating a jail can be done using different methods: ftp, allbsd, gjb, svn, svn+http, svn+ssh, svn+https, svn+file, csup

Whatever method is used the principle is the same: getting the binaries and the sources for a given version/architecture of FreeBSD, and prepare a minimal setup so that it results in a usable jail.

The jails will never be touched again except when specifically requested for an update via "poudriere jail -u".

The jails created will then be considered as a reference of a clean vanilla jail

the minimal setup consists in doing the following:

Modify the login.conf to add the following environment variables:

  • UNAME_r
  • UNAME_v
  • OSVERSION

which will set what is excepted on the jail in order to make sure the builds correctly discovers the release, version and OSVERSION of the jail they are building on.

Modify the make.conf to add the following variables:

  • USE_PACKAGE_DEPENDS=yes
  • BATCH=yes
  • WRKDIRPREFIX=/wrkdirs

Only on i386 jail when the host is amd64:

  • ARCH=i386
  • MACHINE=i386
  • MACHINE_ARCH=i386

If the jail is created on top of a zfs filesystem, then a @clean snapshot will be created

method ftp

This is the default method, it fetches the prebuilt sets from the FreeBSD ftp's (default host can be tweaked via poudriere.conf)

This following set will be installed:

For version lower than 9+ jails:

  • base
  • dict
  • src
  • games
  • lib32 if the jail is a amd64 one

For version upper than 9+ jails:

  • base
  • src
  • games
  • lib32 if the jail is a amd64 one

method gjb/allbsd

This method is the same has ftp except that it fetches the sets out of the gjb's snapshot hosting service or the allbsd one.

method svn (+*)

This method will build the jails from sources fetching them using svn (all svn possible methods are supported)

Only world is built, and the following features are supported: ccache and parallel (make -j) building.

The built jail can be tuned using src.conf: if a src.con files exists in /usr/local/etc/poudriere.d/ on the host, it will used as the src.conf for the jail. if a <jailname>-src.conf file exists in /usr/local/etc/poudriere.d/ on the host, it will be appended to the src.conf used to built the jail.

The rest is just "normal" building of world:

make buildworld
make installworld
make distrib-dirs
make distribution

method csup

This method is considered as deprecated and will be removed soon, it works the same way as the svn method except the sources are fetched using csup.

Updating/Upgrading jails

method ftp

The update of the jail is done by calling freebsd-update from inside the jail. A parameter can be passed to specify a release the user wants to jump to, in that case freebsd-update will be called to perform an upgrade to the said version.

method gjb/allbsd

No update/upgrade is supported if the jail as been created using the method.

method svn/csup

Only updates on the same branch are supported, sources will be updated, the world rebuilt and installed, once the new world is in the old files will be cleaned out:

make delete-old
make delete-old-libs

Ports trees

The ports tree management can be done using the following methods:

  • portsnap
  • svn+*
  • git

the list of ports tree configured is located in /usr/local/poudriere.d/ports/<ptname>

With the following files in it

File Description
fs zfs dataset path if zfs is in use
method the method used to get the ports
mnt the path to the root of the ports tree on the host filesystem

Creating a ports tree

Ports trees are named to allow having different ports tree for example one per patch to test.

if no name is specified then the "default" one is used.

method portsnap

This is the default method as it requires no external dependencies. A directory which will contain the ports tree is created, in this directory a .snap hidden directory is created to receive all the portsnap temporary files.

Creation is done using the following command:

/usr/sbin/portsnap -d ${PTMNT}/.snap -p ${PTMNT} fetch extract

PTMNT being the path to the ports tree that has been created

method svn

As for the jails, the svn method can support the following submethods:

  • svn
  • svn+http
  • svn+https
  • svn+ssh
  • svn+file

the following variable can be tweak in poudriere.conf: SVN_HOST which by default is svn.FreeBSD.org

method git

Same as the two previous methods except that the ports tree will be fetched out of the git

The following variable can be tweak in poudriere.conf: GIT_URL which by default is: git://github.com/freebsd/freebsd-ports.git

Updating the ports tree

The following command allows to update the ports tree

poudriere ports -u -p \<name\>

according to the method used to create the command it will run the following commands:

Using portsnap

portsnap -d ${PTMNT}/.snap -p ${PTMNT} fetch update

or if run from crontab

portsnap -d ${PTMNT}/.snap -p ${PTMNT} cron update

Using svn

svn -q update

Using git

git pull

The build process

General principles

Whatever build is run, the principle is the same.

First a reference jail is created by copying the original jail (chosen via the -j option) into the reference destination.

If the original jail has been created on top of zfs then the reference jail is created as a zfs clone of the @clean snapshot of the original jail.

Otherwise the jail is duplicated using pax(1)

if USE_TMPFS is set to "all" in poudriere.conf then the reference jail is mounted as tmpfs and the original jail duplicated using pax(1)

Once the reference jail is created:

The following directories are created inside the reference jail:

  • /proc
  • /dev
  • /compat/linux/proc
  • /usr/ports
  • /wrkdirs
  • /usr/local (or custom localbase is set)
  • /distfiles
  • /packages
  • /new_packages
  • /ccache
  • /var/db/ports

Then the following filesystems are mounted in it:

  • devfs is mounted in /dev with the following rules applied:
    • everything hidden
    • null unhide
    • zero unhide
    • random unhide
    • urandom unhide
    • stdin unhide
    • stdout unhide
    • stderr unhide
    • fd unhide
    • fd/* unhide
  • the ports dir is nullfs mounted in /usr/ports in readonly
  • the packages dir is nullfs mounted in /packages in readonly
  • the distfiles dir is nullfs mounted in /distfiles in readonly

The option directories that can be found under /usr/local/etc/poudriere.d will be mounted if they exist respecting that order:

  1. /usr/local/etc/poudriere.d/<jailname>-<portstreename>-<setname>-options
  2. /usr/local/etc/poudriere.d/<jailname>-<portstreename>-options
  3. /usr/local/etc/poudriere.d/<jailname>-options
  4. /usr/local/etc/poudriere.d/options

It will be nullfs mounted in readonly as /var/db/ports inside of the jail directory

The reference jail is now populated:

  1. WITH_CCACHE_BUILD=yes is added to the jail's make.conf if CCACHE_DIR is defined
  2. MAKE_ENV+= CCACHE_DIR=/ccache is added to the jail's make.conf if CCACHE_DIR is defined
  3. PACKAGES=/packages is appended
  4. DISTDIR=/distfiles is appended

The content of the following is appended is they exists:

  1. /usr/local/etc/poudriere.d/make.conf
  2. /usr/local/etc/poudriere.d/<jailname>-<portstreename>-make.conf
  3. /usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf
  4. /usr/local/etc/poudriere.d/<jailname>-<portstreename>-<setname>-make.conf

To finish the file corresponding to the variable RESOLV_CONF in poudriere.conf is copied inside the jail as /etc/resolv.conf

The jail is now ready to be used

dependency/sanitycheck

The queue calculation is done on the reference jail directly. inside the /poudriere directory

if TMPFS_DATA is set or USE_TMPFS is set to all then /poudriere will be a tmpfs directory (highly recommanded)

The following directories are created inside the /poudriere directory:

  • /poudriere/building # this is the pool of currently building ports
  • /poudriere/pool # this is the pool of ready to build ports (no dependencies left to built)
  • /poudriere/deps # this is the pool of left ports which at least depend on one port not already built
  • /poudriere/rdeps # this is a reference of reverse dependencies. Each package dir contains a link to each package depending on it
  • /poudriere/var/run # This will receive the pids of the running builders
  • /poduriere/var/cache/origin-pkgname #cache to quickly figure out package name out of an origin
  • /poudriere/var/cache/pkgname-origin #cache to quickly figure out origin out of a packagename

It initializes the log directory in

  • ${POUDRIERE_DATA}/logs/bulk/<MASTERNAME>/<STARTTIME>

POUDRIERE_DATA being /usr/local/poduriere/data by default

MASTERNAME being if setname is set: <jailname>-<portstreename>-<setname> otherwise <jailname>-<portstreename> STARTTIME begin the time of the beginning of the build is the following format: date +%Y-%m-%d_%H:%M:%S

The different statistic files are initialized in the log directory:

file default value
.poudriere.stats_queued 0
.poudriere.stats_built 0
.poudriere.stats_failed 0
.poudriere.stats_ignored 0
.poudriere.stats_skipped 0
.poudriere.ports.built ""
.poudriere.ports.failed ""
.poudriere.ports.ignored ""
.poudriere.ports.skipped ""

The dependency calculation itself is done in parallel (using one process per core by default)

For each ports listed to be built en entry is created in /poudriere/deps corresponding to the package name

All the dependency are extracted from the port via: make -VPKG_DEPENDS -VBUILD_DEPENDS -VEXTRACT_DEPENDS -VLIB_DEPENDS -VPATCH_DEPENDS -VFETCH_DEPENDS -VRUN_DEPENDS

For each dependency an entry corresponding to the dependency pkgname is added to /poudriere/deps/<pkgname>/<depname> And also in an entry is added to /poudriere/rdeps/<depname>/<pkgname> (reverse dependency tracking)

The above process is also applied to the depname

The above will populate /poudriere/deps with all the packages that needs to be built each package will have entry for each dependency it depends on.

Now that the full dependency tree is populated, poudriere will sanity check the repository of packages if there are already packages in it In parallel (one process per core) each package is checked and deleted if:

  1. the origin of the package doesnt exists any more in the ports tree
  2. the version in ports is newer than the version of the package
  3. if CHECK_CHANGED_OPTIONS is set, the package is delete is the options activated in the ports tree for the given port are different from the one for the built package

After that poudriere will process all existing packages and will delete each one for which a dependency is missing and loop on this process until, no package is deleted.

To finish with the sanity check, all the dead symlinks are deleted.

The dependency calculation process ends by looping over all the entries in /poudriere/deps, and each time it figures out the package already exists in the package directory then the entry is removed from the queue.

All the ports with no dependency to be built (empty directories in /poudriere/deps) are now moved to /poudriere/pool/unbalanced as they are ready to be built straight away.

Bulks

Before starting the bulk itself, the reference jail is marked a prepkg, On zfs (no tmpfs) jails a new zfs snapshot @prepkg is created On other filesystems a mtree file tracking all files of the jail is created.

The build is now started in parallel: one building jail per core will be created.

The reference jail is cloned using zfs clone or duplicated via pax(1) into new jails.

For each jail the following filesystems are mounted in it:

  • devfs is mounted in /dev with the following rules applied:
    • everything hidden
    • null unhide
    • zero unhide
    • random unhide
    • urandom unhide
    • stdin unhide
    • stdout unhide
    • stderr unhide
    • fd unhide
    • fd/* unhide
  • the ports dir is nullfs mounted in /usr/ports in readonly
  • the packages dir is nullfs mounted in /packages in readonly
  • the distfiles dir is nullfs mounted in /distfiles
  • fdescfs is mounted in /dev/fd
  • procfs is mounted in /proc
  • linprocfs is mounted in /compat/linux/proc
  • the specified ccache dir if any is nullfs mounted in /ccache
  • /wrkdirs is mounted as tmpfs or mdfs according to configuration if needed
  • the option directory is nullfs mounted readonly using the same rule as for the reference jail

For each port in /poudriere/pool a job is started on one of the above builders:

  1. the /poudriere/pool/<pkgname> dir is moved to /poudriere/building/<pkgname>
  2. umount all tmpfs within the builders to cleanup their content
  3. rollback the jail file system to the @prepkg state (using zfs rollback or using mtree -r + some pax hack)
  4. cleanup /wrkdirs from any previous build
  5. install all pkg, fetch, extract, patch, build and lib dependencies if any inside a jail(8) with no network
  6. run make check-config in a jail(8) with no network
  7. run make fetch in a jail(8) with network access
  8. run make checksum in a jail(8) with network access
  9. copy the distfiles required by the port and only them into /portdistfiles and modify /etc/make.conf so that the port will pickup its distfiles only in this directory
  10. run make extract in a jail(8) with no network
  11. run make patch in a jail(8) with no network
  12. run make configure in a jail(8) with no network
  13. run make build in a jail(8) with no network
  14. run make run-depends in a jail(8) with no network
  15. run make install-mtree in a jail(8) with no network
  16. if in test mode mark the filesystem as preinst (mtree file created)
  17. run make install in a jail(8) with no network
  18. modify /etc/make.conf to that is will create its packages in /new_packages and create that directory
  19. run make package in a jail(8) with no network
  20. if in test mode run make deinstall in a jail(8) with no network
  21. if in test mode check for leftovers by comparing the state of the filesystem with the one in the preinst mtree
  22. if all the above went ok copy the created packages from /new_packages to the final package directory
  23. remove all the /poudriere/deps/*/<pkgname> and move all the new empty directoris into /poudriere/pool/unbalanced so that they can be picked up in a jail for building.
  24. the /poudriere/building/<pkgname> is removed

Pool priority

The pool queue is designed to prioritize building packages that are depended on first. This allows the ready-to-build pool to more quickly get all packages in a ready-to-build state. This can help avoid bottlenecks where large dependencies must build before moving on, which may be a problem with large CPU sets.

  1. a /poudriere/pool/N pool is created for the number of POOL_BUCKETS (default is 10)
  2. packages are initially added into /poudriere/pool/unbalanced
  3. they are then moved to /poudriere/pool/N where N is the number of packages that directly depend on it. This is capped at POOL_BUCKETS-1
  4. the build queue reads from the highest priority pool down to 0
  5. as builds finish, they add new packages ready-to-build into /poudriere/pool/unbalanced and then call balance_pool to prioritize those

Once finished (all packages built) if the built packages are pkgng "pkg repo" in the repository (with a key to sign it is specified in poudriere.conf) if the built packages are pkg_install a INDEX file is created out of the packages in the repository

Clone this wiki locally