feat: startingpoint modules (#33)

* feat: add startingpoint modules
some modules authored by @gerblesh
original source: blue-build/legacy-template#135

* docs: better readme for modules dir

* docs: yafti deps

* docs: sp more information

* feat: startingpoint modules in container

* docs: sentence structure, check sidebar

Containerfile

@@ -18,3 +18,4 @@ FROM scratch
 COPY --from=builder /tmp/ublue-os/files /files
 COPY --from=builder /tmp/ublue-os/rpms /rpms
 COPY --from=ghcr.io/ublue-os/ublue-update:latest /rpms/ublue-update.noarch.rpm /rpms
+COPY modules /modules

modules/README.md

@@ -0,0 +1,5 @@
+# Modules for Startingpoint
+
+Here (check the sidebar if you're on the website) is documentation for every default module for [Startingpoint](https://github.com/ublue-os/startingpoint/). Source code is inside the [bling](https://github.com/ublue-os/bling/) repository, which is intended for hosting more static and shared parts of custom images.
+
+For more information about Startingpoint and modules, refer to the README inside the `config/` directory.

modules/bling/README.md

@@ -0,0 +1,20 @@
+# [`bling`](https://github.com/ublue-os/bling) Module for Startingpoint
+
+The `bling` module allows you to easily declare which general parts of `ublue-os/bling` to pull in to your custom image. It requires the `rpms` and `files` directories from the `bling` container to already exist inside `/tmp/bling/` (pulled inside the Containerfile by default).
+
+The blingbling to pull in is declared under `install:`, and the code for installing them is all in simple named scripts under the `installers/` directory. The basic code for the `bling` module is very similar to the code of the `script` module.
+
+## Example configuration:
+
+```yml
+type: bling # configure what to pull in from ublue-os/bling
+install:
+    - fonts # selection of common good free fonts
+    - justfiles # add "!include /usr/share/ublue-os/just/bling.just"
+                # in your custom.just (added by default) or local justfile
+    - nix-installer # these are the silverblue nix installer scripts from dnkmmr69420
+    - ublue-os-wallpapers
+    # - ublue-update # https://github.com/ublue-os/ublue-update
+    # - dconf-update-service # a service unit that updates the dconf db on boot
+    # - devpod # https://devpod.sh/ as an rpm
+```

modules/bling/bling.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+
+# Tell build process to exit if there are any errors.
+set -oue pipefail
+
+get_yaml_array INSTALL '.install[]' "$1"
+
+export BLING_DIRECTORY="/tmp/bling"
+
+cd "/tmp/modules/bling/installers"
+
+# Make every bling installer executable
+find "$PWD" -type f -exec chmod +x {} \;
+
+for ITEM in "${INSTALL[@]}"; do
+    echo "Pulling from bling: $ITEM"
+    # The trainling newline from $ITEM is removed
+    eval "$PWD/${ITEM%$'\n'}.sh"
+done

modules/bling/installers/dconf-update-service.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+# Tell build process to exit if there are any errors.
+set -oue pipefail
+
+cp -r "$BLING_DIRECTORY/files/usr/etc/systemd/system/dconf-update.service" "/usr/etc/systemd/system/dconf-update.service"
+systemctl enable dconf-update.services

modules/bling/installers/devpod.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+# Tell build process to exit if there are any errors.
+set -oue pipefail
+
+rpm-ostree install "$BLING_DIRECTORY"/rpms/devpod*.rpm

modules/bling/installers/fonts.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+# Tell build process to exit if there are any errors.
+set -oue pipefail
+
+cp -r "$BLING_DIRECTORY"/files/usr/share/fonts/* "/usr/share/fonts"

modules/bling/installers/justfiles.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+# Tell build process to exit if there are any errors.
+set -oue pipefail
+
+cp -r "$BLING_DIRECTORY"/files/usr/share/ublue-os/just/* "/usr/share/ublue-os/just"

modules/bling/installers/nix-installer.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+# Tell build process to exit if there are any errors.
+set -oue pipefail
+
+cp "$BLING_DIRECTORY/files/usr/bin/ublue-nix-install" "/usr/bin/ublue-nix-install"
+cp "$BLING_DIRECTORY/files/usr/bin/ublue-nix-uninstall" "/usr/bin/ublue-nix-uninstall"

modules/bling/installers/ublue-os-wallpapers.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+# Tell build process to exit if there are any errors.
+set -oue pipefail
+
+rpm-ostree install "$BLING_DIRECTORY"/rpms/ublue-os-wallpapers*.rpm

modules/bling/installers/ublue-update.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+# Tell build process to exit if there are any errors.
+set -oue pipefail
+
+# Check if ublue-os-update-services rpm is installed, these services conflict with ublue-update
+if rpm -q ublue-os-update-services > /dev/null; then
+    rpm-ostree override remove ublue-os-update-services
+fi
+
+# Change the conflicting update policy for rpm-ostreed
+RPM_OSTREE_CONFIG="/usr/etc/rpm-ostreed.conf"
+
+if [[ -f $RPM_OSTREE_CONFIG ]]; then
+    if [[ "$(get_config_value AutomaticUpdatePolicy $RPM_OSTREE_CONFIG)" == "stage" ]]; then
+        set_config_value AutomaticUpdatePolicy none $RPM_OSTREE_CONFIG
+    fi
+fi
+
+rpm-ostree install "$BLING_DIRECTORY"/rpms/ublue-update*.rpm

modules/files/README.md

@@ -0,0 +1,16 @@
+# `files` Module for Startingpoint
+
+The `files` module simplifies the process of copying files to the image during the build time. These files are sourced from the `config/files` directory, which is located at `/tmp/config/files` inside the image.
+
+> **Warning**
+> If you want to place anything in `/etc` of the final image, you MUST place them in `/usr/etc` in your repo, so that they're written to `/usr/etc` on the final system. That is the proper directory for "system" configuration templates on immutable Fedora distros, whereas the normal `/etc` is meant for manual overrides and editing by the machine's admin AFTER installation! See issue https://github.com/ublue-os/startingpoint/issues/28.
+
+## Example Configuration:
+
+```yaml
+type: files
+files:
+  usr: /usr
+```
+
+In the example above, `usr` represents the directory located inside the `config/files` in the repository, while `/usr` designates the corresponding destination within the image.

modules/files/files.sh

@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+
+# Tell build process to exit if there are any errors.
+set -oue pipefail
+
+get_yaml_array FILES '.files[]' "$1"
+
+cd "$CONFIG_DIRECTORY/files"
+
+if [[ ${#FILES[@]} -gt 0 ]]; then
+    echo "Adding files to image"
+    for pair in "${FILES[@]}"; do
+        FILE="$PWD/$(echo $pair | yq 'to_entries | .[0].key')"
+        DEST=$(echo $pair | yq 'to_entries | .[0].value')
+        if [ -d "$FILE" ]; then
+            if [ ! -d "$DEST" ]; then
+                mkdir -p "$DEST"
+            fi
+            echo "Copying $FILE to $DEST"
+            cp -r "$FILE"/* $DEST
+        elif [ -f "$FILE" ]; then
+            DEST_DIR=$(dirname "$DEST")
+            if [ ! -d "$DEST_DIR" ]; then
+                mkdir -p "$DEST_DIR"
+            fi
+            echo "Copying $FILE to $DEST"
+            cp $FILE $DEST
+        else
+            echo "File or Directory $FILE Does Not Exist in $CONFIG_DIRECTORY/files"
+            exit 1
+        fi
+    done
+fi

modules/rpm-ostree/README.md

@@ -0,0 +1,33 @@
+# [`rpm-ostree`](https://coreos.github.io/rpm-ostree/) Module for Startingpoint
+
+The `rpm-ostree` module offers pseudo-declarative package and repository management using `rpm-ostree`.
+
+The module first downloads the repository files from repositories declared under `repos:` into `/etc/yum.repos.d/`. The magic string `%OS_VERSION%` is substituted with the current VERSION_ID (major Fedora version), which can be used, for example, for pulling correct versions of repositories from [Fedora's Copr](https://copr.fedorainfracloud.org/).
+
+Then the module installs the packages declared under `install:` using `rpm-ostree install`, it removes the packages declared under `remove:` using `rpm-ostree override remove`. If there are packages declared under both `install:` and `remove:` a hybrid command `rpm-ostree remove <packages> --install <packages>` is used, which should allow you to switch required packages for other ones.
+
+Additionally, the `rpm-ostree` module supports a temporary (waiting for `rpm-ostree` issue [#233](https://github.com/coreos/rpm-ostree/issues/233)) fix for packages that install into `/opt/`. Installation for packages that install into folder names declared under `optfix:` are fixed using some symlinks.
+
+## Example Configuration:
+
+```yml
+type: rpm-ostree
+repos:
+  - https://copr.fedorainfracloud.org/coprs/atim/starship/repo/fedora-%OS_VERSION%/atim-starship-fedora-%OS_VERSION%.repo
+install:
+  - python3-pip
+  - libadwaita
+remove:
+  - firefox
+  - firefox-langpacks
+```
+
+
+## Known issues
+
+When removing certain packages, some problem probably in upstream `rpm-ostree` causes a `depsolve` issue similar to below. [Removed packages are still present in the underlying ostree repository](https://coreos.github.io/rpm-ostree/administrator-handbook/#removing-a-base-package), what `remove` does is "hide" them from the system, it doesn't reclaim disk space.
+```
+Resolving dependencies...done
+error: Could not depsolve transaction; 1 problem detected:
+Problem: conflicting requests
+```

modules/rpm-ostree/rpm-ostree.sh

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+
+# Tell build process to exit if there are any errors.
+set -oue pipefail
+
+# Pull in repos
+get_yaml_array REPOS '.repos[]' "$1"
+if [[ ${#REPOS[@]} -gt 0 ]]; then
+    echo "Adding repositories"
+    for REPO in "${REPOS[@]}"; do
+        REPO="${REPO//%OS_VERSION%/${OS_VERSION}}"
+        wget "${REPO//[$'\t\r\n ']}" -P "/etc/yum.repos.d/"
+    done
+fi
+
+# Create symlinks to fix packages that create directories in /opt
+get_yaml_array OPTFIX '.optfix[]' "$1"
+if [[ ${#OPTFIX[@]} -gt 0 ]]; then
+    echo "Creating symlinks to fix packages that install to /opt"
+    # Create symlink for /opt to /var/opt since it is not created in the image yet
+    mkdir -p "/var/opt"
+    ln -s "/var/opt"  "/opt"
+    # Create symlinks for each directory specified in recipe.yml
+    for OPTPKG in "${OPTFIX[@]}"; do
+        OPTPKG="${OPTPKG%\"}"
+        OPTPKG="${OPTPKG#\"}"
+        OPTPKG=$(printf "$OPTPKG")
+        mkdir -p "/usr/lib/opt/${OPTPKG}"
+        ln -s "../../usr/lib/opt/${OPTPKG}" "/var/opt/${OPTPKG}"
+        echo "Created symlinks for ${OPTPKG}"
+    done
+fi
+
+get_yaml_array INSTALL '.install[]' "$1"
+get_yaml_array REMOVE '.remove[]' "$1"
+
+# The installation is done with some wordsplitting hacks
+# because of errors when doing array destructuring at the installation step.
+# This is different from other ublue projects and could be investigated further.
+INSTALL_STR=$(echo "${INSTALL[*]}" | tr -d '\n')
+REMOVE_STR=$(echo "${REMOVE[*]}" | tr -d '\n')
+
+# Install and remove RPM packages
+if [[ ${#INSTALL[@]} -gt 0 && ${#REMOVE[@]} -gt 0 ]]; then
+    echo "Installing & Removing RPMs"
+    echo "Installing: ${INSTALL_STR[*]}"
+    echo "Removing: ${REMOVE_STR[*]}"
+    # Doing both actions in one command allows for replacing required packages with alternatives
+    rpm-ostree override remove $REMOVE_STR $(printf -- "--install=%s " $INSTALL_STR)
+elif [[ ${#INSTALL[@]} -gt 0 ]]; then
+    echo "Installing RPMs"
+    echo "Installing: ${INSTALL_STR[*]}"
+    rpm-ostree install $INSTALL_STR
+elif [[ ${#INSTALL[@]} -gt 0 ]]; then
+    echo "Removing RPMs"
+    echo "Removing: ${REMOVE_STR[*]}"
+    rpm-ostree override remove $REMOVE_STR
+fi

modules/script/README.md

@@ -0,0 +1,26 @@
+# `script` Module for Startingpoint
+
+The `script` module can be used to run arbitrary scripts at image build time that take no or minimal external configuration (in the form of command line arguments).
+The scripts, which are run from the `config/scripts` directory, are declared under `scripts:`.
+
+## Example Configuration
+
+```yml
+type: script
+scripts:
+    - signing.sh 
+```
+
+## Creating a Script
+
+Look at `example.sh` for an example shell script. You can rename and copy the file for your own purposes. In order for the script to be executed, declare it in the recipe
+
+When creating a script, please make sure
+
+- ...its filename ends with `.sh`.
+  - This follows convention for (especially bash) shell scripts.
+  - `autorun.sh` only executes files that match `*.sh`.
+- ...it starts with a [shebang](<https://en.wikipedia.org/wiki/Shebang_(Unix)>) like `#!/usr/bin/env bash`.
+  - This ensures the script is ran with the correct interpreter / shell.
+- ...it contains the command `set -oue pipefail` near the start.
+  - This will make the image build fail if your script fails. If you do not care if your script works or not, you can omit this line.

modules/script/script.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+
+# Tell build process to exit if there are any errors.
+set -oue pipefail
+
+get_yaml_array SCRIPTS '.scripts[]' "$1"
+
+cd "$CONFIG_DIRECTORY/scripts"
+
+# Make every script executable
+find "$PWD" -type f -exec chmod +x {} \;
+
+for SCRIPT in "${SCRIPTS[@]}"; do
+    echo "Running script $SCRIPT"
+    eval "$PWD/$SCRIPT"
+done

modules/systemd/README.md

@@ -0,0 +1,41 @@
+# `systemd` Module for Startingpoint
+
+The `systemd` module streamlines the management of systemd units during image building. Units are divided into `system` and `user` categories, with `system` units managed directly using `systemctl` and `user` units using `systemctl --user`. You can specify which units to enable or disable under each category.
+
+## Example Configuration:
+
+```yaml
+type: systemd
+system:
+  enable:
+    - example.service
+  disable:
+    - example.target
+user:
+  enable:
+    - example.timer
+  disable:
+    - example.service
+```
+
+In this example:
+
+### System Units
+- `example.service`: Enabled (runs on system boot)
+- `example.target`: Disabled (does not run on system boot)
+
+### User Units
+- `example.timer`: Enabled (runs for the user)
+- `example.service`: Disabled (does not run for the user)
+
+This configuration achieves the same results as the following commands:
+
+```sh
+# System Units
+systemctl enable example.service
+systemctl disable example.target
+
+# User Units
+systemctl --user enable example.timer
+systemctl --user disable example.service
+```

modules/systemd/systemd.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+# Tell build process to exit if there are any errors.
+set -oue pipefail
+
+get_yaml_array ENABLED '.system.enabled[]' "$1"
+get_yaml_array DISABLED '.system.disabled[]' "$1"
+get_yaml_array USER_ENABLED '.user.enabled[]' "$1"
+get_yaml_array USER_DISABLED '.user.disabled[]' "$1"
+
+
+if [[ ${#ENABLED[@]} -gt 0 ]]; then
+    for unit in "${ENABLED[@]}"; do
+        unit=$(printf "$unit")
+        systemctl enable $unit
+    done
+fi
+if [[ ${#DISABLED[@]} -gt 0 ]]; then
+    for unit in "${DISABLED[@]}"; do
+        unit=$(printf "$unit")
+        systemctl disable $unit
+    done
+fi
+if [[ ${#USER_ENABLED[@]} -gt 0 ]]; then
+    for unit in "${ENABLED[@]}"; do
+        unit=$(printf "$unit")
+        systemctl --user enable $unit
+    done
+fi
+if [[ ${#USER_DISABLED[@]} -gt 0 ]]; then
+    for unit in "${DISABLED[@]}"; do
+        unit=$(printf "$unit")
+        systemctl --user disable $unit
+    done
+fi