GUB the Grand Unified Builder

  • Home
  • Basics
  • Applications
  • History
  • Links

GUB -- Grand Unified Builder

WHAT IS GUB

Grand Unified Builder (GUB, http://lilypond.org/gub) is a mini source-based
distribution and packaging system. It cross-compiles several packages with
their dependencies and assembles them into a single installation package.

GUB aims at correctness ie, be dependable and reproducible, while avoiding
gratituous rebuilds. It was originally developed for turnkey building of all
LilyPond binary installers. Currently, GUB supports GNU/Linux, FreeBSD, MacOS-X
and Microsoft Windows installation packages. All packages are fully relocatable
and can be installed with user privileges.

GUB uses plain Python classes to describe package build specifics. For
reasonably well behaved packages, individual build specifications can stay very
simple by extending the generic build specification classes.

USAGE

Run these programs with --help argument for instructions

  bin/gpkg              - Gub Package Manager
  bin/gub               - the Gub Universal Builder
  bin/gib               - Gub (platform dependent) Installer Builder
  bin/gub-tester        - build tester; integrates with
                             Bazaar,CVS,Darcs,Git,SVN and gub
  bin/gupdate           - check for and show or update new upstream releases
  bin/cygwin-packager   - roll cygwin package

BUGS

Send bug reports and patches to [email protected].

Supported installer packages

GNU Denemo
GNU LilyPond
with all essential dependencies. Currently, 207 package build specifications
are included in GUB.

Supported installer platforms

darwin-ppc
   tar.bz2 file for Darwin 7 (MacOS 10.3)/PowerPC

darwin-x86
   tar.bz2 file for Darwin 8 (MacOS 10.4)/x86

mingw
   mingw executable for Windows32

linux-x86
   shar archive for Linux/x86

linux-64
   shar archive for Linux/x86_64

linux-ppc
   shar archive for Linux/PowerPC

freebsd-x86
   shar archive for FreeBSD 6/x86

freebsd-64
   shar archive for FreeBSD 6/x86_64

cygwin
   .tar.bz2 packages for Cygwin/Windows32

arm
   shar archive for Linux/ARM (largely untested)

debian
   shar archive for Debian (largely untested)

LICENSE

GUB is distributed under the terms of the GNU General Public License, version 2
or later. Gub comes with NO WARRANTY.

ACQUIRING

The Git repository is at

  https://github.com/gperciva/gub

You can get it by doing

  git clone git://github.com/gperciva/gub.git

or download the latest tarball doing

  wget https://github.com/gperciva/gub/tarball/master

GUB home page

  http://lilypond.org/gub

INSTALLING

* You need
  - about 9 GB of free space (for all platforms)
  - standard unix shell utilities: cat, cp, install, mv, rm, sed, ...
  - Python 2.4 or newer (2.5, 2.6, 3.0 are known to work)
  - a standard unix development environment with GCC and G++,
    including:
    - 32-bit and 64-bit libraries
    - the file(1) utility

* Try
  bin/gub --help
  make help

Examples

  * Prepare a cross build environment for darwin-x86

      bin/gub darwin-x86::cross/gcc

  * Cross build bison for FreeBSD x86_64

      bin/gub freebsd-64::ftp://ftp.gnu.org/pub/gnu/bison/bison-2.3.tar.gz

  * Make a LilyPond installer for your platform

      bin/gub lilypond-installer

    this leaves the installer in uploads/

  * Cross build Denemo (http://denemo.org) for Windows (mingw32)

      make denemo

    this leaves the Windows installer in uploads/

                           

GUB the Grand Unified Builder

  • Home
  • Basics
  • Applications
  • History
  • Links

GUB BASICS

ROADMAP

       directory                            contents and use
. (GNUmakefile, *.make)  make drivers (try: make help)
bin/                     programs (gub, gpkg, gib, ...; try: bin/gub --help)
gub/                     python library
gub/specs/               build specifications (gub/specs/
                         hello.py,bison.py,denemo.py,lilypond.py...)
patches/                 patch files used in build specifications
target/                  build trees
log/ (gub.log, ...)      (build) log files
uploads/                 installers
target/<platform>/       working tree for <platform>
target/linux-x86/src/    linux-x86 unpacked sources
target/linux-64/build/   linux-64 build output
target/darwin-x86/log/   darwin-x86 most recent lilypond build log
lilypond.log
target/mingw/packages/   .gup packages for mingw
target/darwin-ppc/log/   status of partial darwin-ppc builds
status/
target/freebsd-x86/root/ freebsd-x86 build and install root
target/tools/root/usr/   native tools used for build
bin/
sourcefiles/             non-patch files used in build specifications
nsis/                    nsis windows installer scripts

BASIC WORKINGS

Checksums and rebuilds

GUB seeks to minimise unnecessary rebuilds while being strict about important
changes. Before starting the actual building of a package, the build recipe is
executed as a dry-run and a checksum is calculated from these build commands.
This checksum includes the environment variables DISPLAY, FTP_PROXY,
GUB_TOOLS_PREFIX, HOME, HTTP_PROXY, LIBRESTRICT, PATH [no other environment
variables are allowed through].

The packages' build checksum is matched with the previous build and if it
changes, the package is rebuilt. Otherwise, only the binary package is
reinstalled, if necessary. This means that the .py build specs can be changed
and refactored at will and as long as no actual build commands change, nothing
will be rebuilt.

This makes that forcing rebuilds can can be tricky: gub refuses to do so if it
deems rebuilding unnecessary.

SHARING DIRECTORIES

GUB uses an ABI environment variable to work around some build bugs in some
packages. This variable is not checked by the environment-changed function, so
problems will probably occur if you share your GUB build directory between a
32-bit and 64-bit OS. If you change your operating system, delete the target/
directory of GUB before trying to build it again.

LIBRESTRICT

A common problem with [making reproducible cross] builds is using files or
features that are not controlled by the [GUB] build process, but just happen to
be on the build host and happen to differ between most build hosts.

GUB uses the LD_PRELOAD feature with the gub/specs/librestrict.py package and
code from ./librestrict/restrict.c to disallow reading from build host files.
This makes for much better reprocucibility, especially between different build
hosts.

The downside to this LIBRESTRICT feature is that quit a few packages are not
prepared for cross building. LIBRESTRICT will catch these errors and they have
to be fixed in the package's spec file.

This is what such a LIBRESTRICT assertion could like

    librestrict:error:/home/janneke/vc/gub/target/mingw/root/usr/cross/libexec/gcc/i686-mingw32/4.1.1/cc1plus: tried to open () file /home/janneke/vc/lilypond/lily/main.cc
    librestrict:allowed:
      /home/janneke/vc/gub/target/mingw
      /tmp
      /dev/null
      /dev/urandom
      /proc/self
    cc1plus: internal compiler error: Aborted
    Please submit a full bug report,
    with preprocessed source if appropriate.
    See  for instructions.
    [1]10:42:08 janneke@peder:~/vc/gub
    $

you often find these in config.log files when package-dependant code blondly
looks in /usr/include or /usr/lib.

stat-restriction

LIBRESTRICT has an experimental strict mode enabled by setting the environment
variable

    LIBRESTRICT=open:stat

This will even disallow any STATting outside the target's build tree. It would
be nice to get this to work and some effort has been done. However, this is
potentially a lot of work as not only upstream packages but also build system
authors [autoconf, automake, libtool etc.] look at the build host root file
system while cross building.

Simplifying package builds or avoiding code duplication

Another idea of GUB is to identify, fix and generalise build quircks and move
them from a package's spec file into one of the generic build classes. This
increases maintainability by avoiding code duplication, makes spec files
smaller, simpler and also increases the chances of a new package's spec file to
only need a source url to specify the build.

BASIC USAGE

Build package hello for platform linux-x86

    bin/gub linux-x86::hello

Examine why package hello wants to rebuild

    less target/linux-x86/log/build.log
        search from end back to `mismatch:' [type: FC-c?mismatch:RET]

Force a rebuild of package hello for platform linux-x86

    rm -rf target/linux-x86/*/hello-*

Force a rebuild of package hello for all platforms

    rm -rf target/*/*/hello-*

Prepare to rebuild entire linux-x86 platform

    rm -rf target/linux-x86

Prepare to rebuild everything, all platforms

    rm -rf target

SPEC FILES

Plain url builds

<--no spec-->

Well behaved packages can be built without spec file, straight from an url. For
example, to build bison-2.2 as a build tool, do

    bin/gub tools::http://ftp.gnu.org/pub/gnu/bison/bison-2.2.tar.gz


This produces

    target/tools/packages/bison-2.2.tools.gup
    target/tools/packages/bison.tools.hdr
    target/tools/packages/bison.checksum


Check files installed under target/tools/root/ by doing

    bin/gpkg -p tools files bison


To build the very latest findutils for your architecture, do

    bin/gub git://git.savannah.gnu.org/findutils.git


Simple spec

gub/specs/hello.py

    from gub import target

    class Hello (target.AutoBuild):
        source = 'http://lilypond.org/downloads/gub-sources/hello-1.0.tar.gz'


build it for mingw by doing

    bin/gub mingw::hello


__TOOLS package with dependencies

gub/specs/autoconf.py

    from gub import tools

    class Autoconf__tools (tools.AutoBuild):
        source = 'http://ftp.gnu.org/pub/gnu/autoconf/autoconf-2.63.tar.gz'
        parallel_build_broken = True
        dependencies = [
            'm4',
            'perl',
            ]


build it by doing

    bin/gub tools::autoconf


configure flags

gub/specs/icoutils.py

    from gub import tools

    class Icoutils__tools (tools.AutoBuild):
        dependencies = ['libpng-devel']
        configure_flags = (tools.AutoBuild.configure_flags
                           + ' --with-libintl-prefix=%(system_prefix)s'
                           + ' --disable-nls')


__<platform>__<arch>-specific instructions

For platform specific instructions, append "__<platform>" to the class name.
For platform+architecture-specific instructions, "__<platform>__<arch>" to the
class name.

gub/specs/pixman.py

    from gub import target

    class Glib (target.AutoBuild):
        ...

    class Glib__freebsd (Glib):
        dependencies = Glib.dependencies + ['libiconv-devel']
        configure_variables = Glib.configure_variables + ' CFLAGS=-pthread'

    class Glib__freebsd__x86 (Glib__freebsd):
        # Must include -pthread in lib flags, because our most beloved
        # libtool (2.2.6a) thinks it knows best and blondly strips -pthread
        # if it thinks it's a compile flag.
        # FIXME: should add fixup to update_libtool ()
        compile_flags = ' G_THREAD_LIBS=-pthread G_THREAD_LIBS_FOR_GTHREAD=-pthread '


String expansion

In all build commands, standard Python string expansion

    %(string-name)s


can be used. These strings are taken from class Settings in gub/settings.py and
from the package's build class and its parent classes.

Adding string expansion variables to a spec build class can be done in several
ways

  • class variable

        class Foo (target.Autobuild):
            common_configure_flags = ' --without-java'


  • instance variable

        class Foo__mingw (Foo):
            def __init__ (self, settings, source):
                Foo.__init__ (self, settings, source)
                self.target_gcc_flags = '-mms-bitfields'


  • decorated funtion returning a string

        class Bar__mingw (target.Autobuild):
            @context.subst_method
            def target_os (self):
                return 'windows'


  • Manipulating the substitution_dict using gub/build.py's class Change_dict
    or overriding the get_substitution_dict method.

Pre-defined strings

Typical variables defined by settings are

    build_architecture=x86_64-linux
    build_os=linux
    build_platform=linux-64
    platform=target
    target_architecture=x86_64-linux
    target_bits=64
    target_cpu=x86_64
    target_os=linux
    target_platform=target


Typical variables defined by the build class

    compile_flags=
    builddir=.../target/build/...
    config_cache_overrides=
    configure_binary= .../configure
    configure_command= sh .../configure ...
    configure_flags= --prefix=... --enable-shared --enable-static
    configure_variables= CFLAGS=-I.../target/root/usr/include
    install_command=
    install_flags=
    install_prefix=
    install_root=
    make_flags=
    srcdir=.../target/src/...


More examples

See gub/specs/*.py for some more examples.

                           

GUB -- LilyPond

INSTALLING

Get GUB

    git clone git://github.com/gperciva/gub.git
or
    wget https://github.com/gperciva/gub/tarball/master

In addition to the requirements for GUB, you need

  • A TeX installation
  • texinfo
  • zip

Build everything

  make lilypond

this leaves installers and documentation in

  uploads/lilypond-VERSION-RELEASE.darwin-ppc.tar.bz2
  uploads/lilypond-VERSION-RELEASE.darwin-x86.tar.bz2
  uploads/lilypond-VERSION-RELEASE.freebsd-64.shar.sh
  uploads/lilypond-VERSION-RELEASE.freebsd-x86.shar.sh
  uploads/lilypond-VERSION-RELEASE.linux-64.shar.sh
  uploads/lilypond-VERSION-RELEASE.linux-ppc.shar.sh
  uploads/lilypond-VERSION-RELEASE.linux-x86.shar.sh
  uploads/lilypond-VERSION-RELEASE.mingw.exe

  uploads/lilypond-VERSION-RELEASE.documentation.tar.bz2
  uploads/lilypond-VERSION-RELEASE.webdoc.tar.bz2
  uploads/webdoc/VERSION

  uploads/lilypond-VERSION-RELEASE.test-output.tar.bz2
  uploads/wbtest/VERSION

More examples

  * Make a LilyPond installer for your platform

      bin/gub lilypond-installer

    OR (old style)

      bin/gub lilypond  # package with dependencies
      bin/gib --branch=lilypond=git.sv.gnu.org--lilypond.git-master lilypond #installer

  * Build all LilyPond installers

      make lilypond

    this leaves installers for GNU/Linux (x86, x86_64, powerpc),
    Windows, MacOSX (x86 and powerpc), FreeBSD (x86, x86_64) and
    documentation in uploads/

  * Optional: Prepare build environment

      make -f lilypond.make bootstrap

TESTING

The test-lily directory contains scripts for running various tests. Run without
arguments for instructions

  cron-builder.py - entry point from continuous crontab testing
  test-binary.py  - test binary builds
  with-lock.py    - run a command with a lockfile
  rsync-lily-doc.py
                  - prepare documentation for upload
  dist-check.py   - check tarball for anomalies
  cygwin-packager.py
                  - repackage GUB packages into Cygwin tarballs.

CRON USAGE

I have the following in my crontab

  • Hourly build of LilyPond subpackage for all platforms except native (native
    is darwin-pcc?)

    test-lily/cron-builder.py --package\
      --test-options "--to [email protected] --quiet --from [email protected]
                      --smtp smtp.xs4all.nl --repository downloads/lilypond.git"\
      darwin-x86 freebsd-x86 freebsd-64 mingw linux-x86 linux-64

  • Hourly build of native installer, to run documentation and dist check.

    test-lily/cron-builder.py\
      --package --installer --tarball --docs\
      --test-options "--to [email protected] --dependent --quiet\
                      --from [email protected] --smtp smtp.xs4all.nl --repository downloads/lilypond.git"\
      --unversioned\
      darwin-ppc

  • Nightly build of all installers, to be ready for shipping binaries in the
    morning.

    test-lily/cron-builder.py\
      --installer\
      --test-options "--to [email protected] --quiet --from [email protected]\
                      --smtp smtp.xs4all.nl --repository downloads/lilypond.git"\
      darwin-x86 freebsd-x86 freebsd-64 mingw linux-x86 linux-64 darwin-ppc

                           

GUB -- GNU Denemo

INSTALLING

Get GUB

    git clone git://github.com/gperciva/gub.git
or
    wget https://github.com/gperciva/gub/tarball/master

Build everything

  make denemo

this produces a windows installer in

  uploads/denemo-x.y.z-b.mingw.exe

More examples

  * Build main Denemo installers

      make PLATFORMS='mingw linux-x86 linux-64' denemo

  * Build all Denemo installers

      make 'PLATFORMS=mingw linux-x86 linux-64 linux-ppc darwin-ppc darwin-x86 freebsd-x86 freebsd-64' denemo

    this leaves installers for GNU/Linux (x86, x86_64, powerpc),
    FreeBSD (x86, x86_64) and Windows, and documentation in uploads/

    OR (old style)

      bin/gub denemo
      bin/gib denemo --branch=lilypond=git.sv.gnu.org--lilypond.git-master --branch=denemo=git.savannah.gnu.org--denemo.git-master

  * Optional: Prepare build environment

      make -f lilypond.make bootstrap

                           


GUB the Grand Unified Builder

  • Home
  • Basics
  • Applications
  • History
  • Links

GUB -- Grand Unified Builder -- HISTORY

HISTORY

The story starts June 1999 with a crazy guy with an itch to run LilyPond on
Windows. To get a feel for the times, this was LilyPond-1.1.47, requiring Egcs
1.1, Python 1.5, Guile 1.3, discussing on [email protected]. Two years
later Cygwin cross, our first cross build environment is released.

This is evolved into a patch set for mknetrel. Spring 2005: In a bold move a
set of patches for MinGW are created to build LilyPond and all its [17]
dependencies. Simultaniously, a cross build environment and installer for
MacOSX is created.

Creating a cross build environment is not too difficult, esp. not if you know
how to use Google. The problem is that they always break and never work on
other machines. GUB starts as an effort to unify the Windows and MacOS builders
and to make the process somewhat dependable and reproducible, i.e.: even more
free time for the developers (if there is such a thing) as the builds always
succeed. It uses Python instead of make and bash and adds checksums for the
package build specs. The unification of cross builds greatly facilitates adding
of new architectures and before long GUB supports cross building lilypond
binaries for 9 different platforms.

GUB2 adds serialization of the Python build code before execution. This makes
the build checksumming less coarse, ie, less gratuitous rebuilds and more
reliable. Also, all Python code is executed before the build process starts,
which means that there is less chance of silly run time errors. Lots of
packages are added in an effort to replace the OpenEmbedded-based build system
for the Samco 3G video phone at SES, Philips Research Eindhoven.

In Novell's Hackweek-iii GUB enters into its Third Life with GUB3, adding
cross-platform build dependencies and patches for an OpenOffice.org MinGW cross
build, including all essential libraries. Later, Inkscape is added, exploring
and implementing x.org dependencies. After that, building a windows installer
for GNU Denemo took only one day, including all dependencies.

                           

GUB the Grand Unified Builder

  • Home
  • Basics
  • Applications
  • History
  • Links

Links

autopackage.org
openSUSE build service
codesourcery.com
pre-GUB LilyPond installers
emdebian.org
GUB build specifications
GUB patches
lilypond.org/blog/janneke/gub3-ooo-mingw-cross-build
openembedded.org
scratchbox.org
Scratbox2; sb2 Dan Kegel's crosstool