Enable transient (composefs) root (again)

This reverts commit 7977ead and
effectively migrates the change from CentOS/centos-bootc-dev@8f5be09
to here.

See the linked PR for much more debate, but in a nutshell this
flips around a lot of tradeoffs around the filesystem layout
and will result in a much more easily explained state that
is intended to somewhat more closely match what happens with
a default `podman run --rm` invocation of the container.

Signed-off-by: Colin Walters <[email protected]>

docs/usage.md

@@ -119,27 +119,30 @@ when --apply is used.
 
 ## Filesystem interaction and layout
 
-At "build" time, this image runs the same as any other OCI image where
-the default filesystem setup is an `overlayfs` for `/` that captures all
-changes written - to anywhere.
-
-However, the default runtime (when booted on a virtual or physical host system,
-with systemd as pid 1) there are some rules around persistence and writability.
-
-The reason for this is that the primary goal is that base operating system
-changes (updating kernels, binaries, configuration) are managed in your container
-image and updated via `bootc upgrade`.
+The base images produced by this project run exactly the same as the
+"application" base images (e.g. `quay.io/centos/centos:stream9`) when
+run under a container runtime.
+
+However, when installed to a physical or virtual maachine (via `bootc install`
+or equivalent tooling), the filesystem layout has two major differences:
+
+- The `/etc` directory is persistent by default, and a "3 way merge" process
+  is performed across upgrades using the underlying [ostree tooling](https://ostreedev.github.io/ostree/deployment/#contents-of-a-deployment).
+- The `/var` directory acts as if a `VOLUME /var` instruction was provided; the contents
+  of the `/var` from the container image are written to a persistent directory, but
+  *are not changed or merged across upgrades*.
+- Every other directory (e.g. `/usr`, `/opt`) are backed by a [composefs](https://github.com/containers/composefs/)
+  read-only filesystem tree, with a *transient writable overlay*.  Changes made here
+  are discarded across reboots or upgrades.  This is intended to match the semantics
+  of a default `podman|docker run --rm` invocation.
+
+The primary goal is that base operating system changes (updating kernels, binaries,
+configuration) are managed in your container image and updated via `bootc upgrade`.
 
 In general, aim for most content in your container image to be underneath
 the `/usr` filesystem.  This is mounted read-only by default, and this
 matches many other "immutable infrastructure" operating systems.
 
-The `/etc` filesystem defaults to persistent and writable - and is the expected
-place to put machine-local state (static IP addressing, hostnames, etc).
-
-All other machine-local persistent data should live underneath `/var` by default;
-for example, the default is for systemd to persist the journal to `/var/log/journal`.
-
 ### Understanding `root.transient``
 
 At a technical level today, the base image uses the

tier-0/ostree.yaml

@@ -7,9 +7,10 @@ postprocess:
     #!/usr/bin/env bash
     mkdir -p /usr/lib/ostree
     cat > /usr/lib/ostree/prepare-root.conf << EOF
-    # TODO disabled due to https://github.com/osbuild/bootc-image-builder/issues/149
-    #[root]
-    #transient = true
+    [root]
+    transient = true
+    [composefs]
+    enabled = true
     [sysroot]
     readonly = true
     EOF