Update-Flow Breakpoints / Required Updates

[w4tsn]

A question to you all. I noticed here and there for some changes in my rpm-ostree remote configuration that I'd like to force a client to do Update A before attempting to update B. E.g. when I do a key-exchange of a public key that is shipped with the ostree ref. If the new key is required to update to C, D and so forth than I'd like the client to update to A first.

Surely I could just play with time and say the key-exchange lasts for 6 months, so both public keys are present for 6 months. Fairly enough time for every client to download one of the updates with both keys. But it still gets me wondering, isn't this somewhat unreliable and hacky to assume that I just have to give clients enough time to update before removing old public keys?

Is there a usual flow of things in a system like rpm-ostree? Or should the public keys for the ostree remote just not be handled by the remote itself but a distinct PKI service? What is typcial experience with this?

The public key topic is rather specific. In general there are updates that introduce the necessity of "migrations". I suppose that usually one would handle those migrations on a per RPM level and don't care much about it in rpm-ostree?

[lucab]

With regards to "forcing clients through migrations", Fedora CoreOS has some additional upgrade concepts layered on top of the usual rpm-ostree flow, restricting the matrix of possible upgrades that a client can take:

• an upgrade graph (DAG), so that clients can discover which updates they should go through and in which order
• some forced chokepoints in the DAG (upgrade barriers), so that clients can be forced to apply some update containing "migration logic", before proceeding into further updates

One way that this is used is exactly to introduce new signing keys: https://docs.fedoraproject.org/en-US/fedora-coreos/update-barrier-signing-keys/
The resulting update graph looks more or less like this https://builds.coreos.fedoraproject.org/graph?stream=stable

[w4tsn]

Thanks for pointing that out! A bit more context: I'm working with Fedora IoT and did not yet try to use cosa and CoreOS tools / flows for Fedora IoT

[jlebon]

Yeah, ostree/rpm-ostree itself has no concept of checkpoint updates, though it has been proposed. If you have a tool driving updates, then you can go the FCOS/OKD/OCP route where you provide out-of-band information on checkpoints. An in-band approach is to use versioned branches + commit metadata (like end-of-life) which the tool detects and then rebases to the new branch.

[cgwalters]

I've struggled the whole time of ostree/rpm-ostree's existence for whether it should be "high level" or "low level". In this case though I can't imagine anyone from the CoreOS side trying to push for something that's not zincati based.

So indeed IMO IoT should be using zincati.

But, it doesn't rule out doing something in ostree for this but it'd need to have a strong driver. (And perhaps...in theory zincati could work on ostree-but-not-rpm-ostree systems)

[cgwalters]

The public key topic is rather specific.

Actually specifically on this you may want to pile onto ostreedev/ostree#2260

[w4tsn]

Hmm. I'll probably give it a try to include zincati in Fedora IoT and see how that works out. If this functionality is not to be included in rpm-ostree or even libostree I guess a section in the docs how to set this up is reasonable. At least when using rpm-ostree as a vendor it's a common use-case to have support for this - common enough I would argue for a native implementation in libostree / rpm-ostree but since there is extensive tooling already including it in the docs is the least we can do