feat!: better batch-building images

[xynydev]

Currently startingpoint can build multiple images based on multiple recipe's.

It would be great, though, to be able to share some settings between images, like package lists.

As building multiple images from a single recipe is not possible, I think this should be done via "includes". We'd have a "recipes" directory which gets copied in full into the Containerfile somewhere. All the included files would have to be in that directory as well. Then all the include's would somehow be parsed and compiled, or interpreted when come accross them when reading the files.

Now, this is not a YAML feature, but prior art should be investigated. The possibility of using another config language, like Nix(?), could be considered.

This feature should be worked on only after a "1.0" release.

[tulilirockz]

We could also make it so there could be a "shared" recipe -> shared.recipe.yaml would be read if it exists, and it would contain a subset of the raw recipe, with just the packages and scripts that all the other recipes would use!

[xynydev]

We could also make it so there could be a "shared" recipe -> shared.recipe.yaml would be read if it exists, and it would contain a subset of the raw recipe, with just the packages and scripts that all the other recipes would use!

I think it should be more modular than that, but that'd work. Could be a key in the main recipe.yml that chooses a (list of?) recipe.shared.yml which would contain the same settings except for the base image and name. Good idea!

[scottames]

I'm attempting to add multi-images to my startingpoint fork (I need main & nvidia). My initial inspiration is to mostly duplicate what bluefin does with the workflow matrix & the env args. Is that something you all would consider a good path here, or are you hoping to be more general and/or focused on the recipe(s)?

The reason I ask is if what I do for myself works well and would be of interest to bring back in I'd be down to attempt it.

[xynydev]

In Bluefin a lot of the config is in build.yml and the Containerfile, while startingpoint is specifically made to abstract that away into the recipe, so that you don't need to learn two different DSLs to do something somewhat dependent on them.

I really like the idea of shared recipes, ie. making a base recipe with a set of configuration, declared in a main recipe to use it as a fallback. This would allow us to keep the current model of build matrix with just declaring the recipes.

[scottames]

Following along the Vanilla work, this is potentially a tool that could help solve this: https://github.com/Vanilla-OS/Vib

[xynydev]

Vib

Ooh, Vanilla's doing this too!
Theirs is based around the concept of modules that are declared in the recipe (a modular structure like some wanted here, but one I couldn't will into existence). I think our system is simpler, but it could be adapted towards a modular thing like that like so; split the sections like yafti and rpm to different modules, allow people to create their own modules, and then declare them in the recipe. Modules, in this case, would probably just be shell scripts that parse and execute the actions of a specific module, which is just the declaration of what you want.

Like Nix, for example, here's a module in my home-manager config for the starship prompt.

programs.starship = {
  enable = true;
  enableFishIntegration = true;
  enableZshIntegration = true;
};

Could this be a better, more maintainable approach to what we're trying to do? The original version was just kind of hacked together, not much design went into it.

I guess I could do this;

React with 🚀 if you would prefer a modular startingpoint.

• This would be a major breaking change that would make you break up your recipe.yml into different files for different things.

React with 🎉 if you would prefer a nesting version of the current recipe.yml system.

• recipe.yml would just be extended with an option allowing you to include the options from another recipe as a base to your own.
• This would be a somewhat minor change, but also allow easier maintenance of multi-image repos.

[xynydev]

More modular brainstorming.

I'm thinking it'd be kind of similar to the Github Actions yml format.

# these are specific to the image, not modular
name: xyz
description: lorem ipsum
base-image: ghcr.io/...
fedora-version: 38

modules:
  - uses: scripts
    type: pre # given as argument to script
    run:
      - autorun.sh

  # making the pkg install modular would technically make it possible to reuse sp for any OS
  - uses: rpm-ostree
    repos:
      - copr.fedora...
    install:
      - python3-pip
      - libadwaita
    remove:
      - firefox
      - firefox-langpacks

  - uses: scripts
    type: post
    run:
      - autorun.sh

  - uses: yafti
    config: yafti.yml # if not specified, would use a default config
    custom-flatpaks:
      - io.github.celluloid_player.Celluloid
      - org.kde.krita

This would also make it possible to pull in module configurations from different files.

modules:
  - from-file: packages.yml

And it could be possible to extend this to pull in modules from external repos

modules:
  - uses: ublue-os/bling@font-installer
    fonts:
      - Work Sans
      - Space Mono Nerd Font

Or pull them in via the Containerfile

COPY --from=ghcr.io/ublue-os/bling:latest /modules/font-installer /tmp/modules/font-installer

modules:
  - uses: font-installer
    fonts:
      - Work Sans
      - Space Mono Nerd Font

[xynydev]

How I envision the modules to work is a modules directory with subfolders for all the modules, which are used with their "id" aka the name of the subfolder. The module folder would have a README for the modules docs and a shell script that is executed.
The module scripts should probably only be provided with the options scoped to the module via some temporary file in /tmp/moduleinputs or smth.

The modules can still execute arbitrary code in the image build and, by extension, your machine, so they probably shouldn't/can't be made "trustless". They would have to be trusted, and people should be cautioned to only use third-party modules on their own risk and after reading their code.

[xynydev]

Working on modularity #135, comments appreciated.

[scottames]

@EinoHR I like what you've got going in #135! Merge that with some of the concepts that you have in the previous comment and I think you're on the right track! Namely, being able to extend and pull in modules locally or remote and pull them in from the containerfile, though I am not sure how you're going to accomplish the latter.

It is probably a bit overkill for anyone that's new to the ecosystem and just wants one single image built in a simple way without relying on magic, but to your point, could make it a bit more approachable.

Personally for my image, I ended up moving more in the direction that bluefin has in being able to leverage the matrix and 'if' features in the raw github actions workflow. It would be awesome to be able to leverage that directly. (I'll give it some thought)

[xynydev]

Sure, thanks!

Personally for my image, I ended up moving more in the direction that bluefin has in being able to leverage the matrix and 'if' features in the raw github actions workflow.

That's totally intended! I originally started this codebase with two goals; easy to get started with, and easy to understand, si that one could extend as needed and learn new things. I'll be trying to continue that with thorough documentation of the modules. I also have a vague plan for not a template, but an advanced minimum viable product/buildsystem that could get you started on a more custom OCI-building path without needing to learn startingpoint or any other repository's custom system.