[Tracking] OKD 4.6 Stable

[vrutkovs]

Features in this release:

• samples operator is using UBI / CentOS images
  Some imagestreams may be removed
• NetworkManager-ovs, glusterfs and open-vm-tools are being installed via OS extensions mechanism
  This ensures post-install actions are being run correctly on the machine. -ovs is required for OVN to function.
• Fedora 33 content

Blocking bugs:

• Bug 1882136 - created toolbox fails to start: open /proc/sys/net/ipv4/ping_group_range: Permission denied: OCI runtime permission denied error - blocks 4.6 machine-os-content promotion
  Fixed in F33
• Bug 1888467 - Repos should be disabled in -firstboot.service before OS extensions are applied
  FCOS would use Fedora repos when OS extensions are being installed, may fail on disconnected installation
• 4.5 -> 4.6 upgrade fails on CI due to OVN issues (needs verification, may be fixed by now)
  Needs MCO PR to land

Nice to have:

• Bug 1844198 - the openvswitch service triggers SELinux denials on netlink_netfilter_socket
• Bug 1879099 - Extra reboot during 4.5 -> 4.6 upgrade
  This may prevent nodes to start during machine update (network is already 4.6, but nm-ovs is not yet installed) - needs verification, may be not necessary

TODO:

• Bug 1894477: Fix bash in nodeip-configuration.service openshift/machine-config-operator#2199
  Fixes baremetal UPI install
• [release-4.6] Bug 1894483: Fix bash in nodeip-configuration.service openshift/machine-config-operator#2200
  release-4.6 cherry-pick of ^
• Bug 1895360: pkg/daemon: don't delete a file if its replaced with a dropin openshift/machine-config-operator#2196
  Handle kubelet proxy config from plain file to a systemd dropin in Ignition
• [release-4.6] Bug 1899735: pkg/daemon: don't delete a file if its replaced with a dropin openshift/machine-config-operator#2246
  release-4.6 cherry-pick of ^
• GCP Builds Fail since 4.6.0-0.okd-2020-10-08-055226 #352
  Incorrect hostname set (GCP only)
• Static IP config with nameserver is broken due to systemd-resolved
  OKD 4.6: Missing /etc/resolv.conf when using static ip configuration #380. Workaround: initial FCOS image should be F32

Post-release todolist:

• Move manifests installed by installer to okd-machine-os manifest
• Update installer and okd-machine-os to use stable FCOS once F33 content lands there
• CRI-O 1.19
• samples operator is using UBI / CentOS images - compile a list of differences

[vrutkovs]

State as of Oct 15:

Managed to update 4.5 to custom 4.6 with the following PRs

• [release-4.6] Bug 1888467: firstboot.service: disable existing repos before pivot openshift/machine-config-operator#2161
• Bug 1880591: Adds missing check for ovs-config-executed openshift/cluster-network-operator#836
• Bug 1880591: Fixes race during 4.5->4.6 upgrade with ovn node and master openshift/ovn-kubernetes#307
• Bug 1887040: OVS config: check if OVS is installed openshift/machine-config-operator#2154
• Bug 1887607: OVS config: check if OVS is installed openshift/machine-config-operator#2164
• Bug 1888853: daemon: allow an extension to install group of packages openshift/machine-config-operator#2187
• MCO mistakenly removes a file for kubelet proxy settings, although its converted into a systemd dropin

Nice-to-have:

• Bug 1879099: Inject version into controllerconfig, refuse mismatches openshift/machine-config-operator#2112 - speeds up upgrades by making less reboots
• release-4.6 cherrypick of ^

GCP hostname workaround:

• Update hostname when transient hostname is fedora openshift/machine-config-operator#2160
• release-4.6 cherry-pick of ^

PRs to build OKD-specific images:

• MCO: add OKD tests for release-4.6 and release-4.7 branches openshift/release#12683
• Samples: add OKD test configs for 4.6/4.7 branches openshift/release#12686
• operator-marketplace: OKD updates for release-4.x branches openshift/release#12691

Machine-os-content build repo:

• Okd-machine-os repo onboarding openshift/release#13076
• Add initial machine-os-content for OKD 4.7 openshift/okd-machine-os#1 - 4.7 changes
• OKD 4.6: add tests to okd-machine-os release-4.6 openshift/release#13145 + PR to land 4.6 changes

[vrutkovs]

Remaining list of PRs to merge:

• Bug 1894477: Fix bash in nodeip-configuration.service openshift/machine-config-operator#2199
  Fixes baremetal UPI install
• [release-4.6] Bug 1894483: Fix bash in nodeip-configuration.service openshift/machine-config-operator#2200
  release-4.6 cherry-pick of ^
• WIP [fcos] pkg/daemon: don't delete a file if its replaced with a dropin openshift/machine-config-operator#2198
  Handle kubelet proxy config from plain file to a systemd dropin in Ignition
• default hostname now is fedora, used to be localhost coreos/fedora-coreos-tracker#649
  Incorrect hostname set (GCP only)

[LorbusChris]

We might need openshift/machine-config-operator#2160 as a temporary fix after all until https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/572 is implemented

[willmtemple]

samples operator is using UBI / CentOS images
Some imagestreams may be removed

I manually one-time provisioned the okd-x86_64 Templates and CentOS-based imagestreams from the operator into the openshift namespace on my OKD 4.5.0 lab cluster, will there be any action during the update for me or will the operator simply take over managing them?

Really looking forward to this update.

[vrutkovs]

will there be any action during the update for me or will the operator simply take over managing them?

The operator will replace the managed imagestreams (like python, wildfly etc.). The rest will be untouched

[jomeier]

It looks like that almost all open tasks are resolved 👍

[LorbusChris]

Update

Open Issues:

• templates: use Afterburn for setting GCP hostnames openshift/machine-config-operator#2217
  Worked around by [release-4.6] Run afterburn in gcp-hostname.service openshift/okd-machine-os#9, not blocking at this stage
• OKD 4.6: Missing /etc/resolv.conf when using static ip configuration #380
  Workaround: Use F32 as initial image

Pending Backports:

• [release-4.6] Bug 1888467: firstboot.service: disable existing repos before pivot openshift/machine-config-operator#2161
• [release-4.6] Bug 1901187: templates: Disable SSH keys lookup from authorized_keys.d on FCOS openshift/machine-config-operator#2245 (not blocking)
• [release-4.6] Bug 1899735: pkg/daemon: don't delete a file if its replaced with a dropin openshift/machine-config-operator#2246
• Backport of templates: use Afterburn for setting GCP hostnames openshift/machine-config-operator#2217
  Worked around by [release-4.6] Run afterburn in gcp-hostname.service openshift/okd-machine-os#9

[bdurrow]

Will 4.6 RC hold for FCOS 33 to land in stable or would that only be a requirement for GA? If so it seems that will be the critical path.

[Conan-Kudo]

There's also an issue where Fedora CoreOS has the wrong iptables implementation: coreos/fedora-coreos-tracker#676

[LorbusChris]

Will 4.6 RC hold for FCOS 33 to land in stable or would that only be a requirement for GA? If so it seems that will be the critical path.

We won't hold RC for that

There's also an issue where Fedora CoreOS has the wrong iptables implementation: coreos/fedora-coreos-tracker#676

I'm not sure that's actually an issue for OKD - in 4.5 it works with the older implementation. Still a good idea to track progress on it here, too :)

[dustymabe]

There's also an issue where Fedora CoreOS has the wrong iptables implementation: coreos/fedora-coreos-tracker#676

I'm not sure that's actually an issue for OKD - in 4.5 it works with the older implementation. Still a good idea to track progress on it here, too :)

The complicated part is if you start 4.6 with iptables legacy and then it switches to nft. We're going to try to make sure existing nodes don't get switched on upgrade, but new nodes would have nft. Just something to consider.

[ibotty]

Why is it a problem to switch? Can't it be switched to nft when the node gets started with the new compose? What's the failure case?

[dustymabe]

Maybe it's no problem. I just don't know. It looks like we're going to hold off for a while on the migration so just assume iptables legacy for now (which is working fine).

[vrutkovs]

https://amd64.origin.releases.ci.openshift.org/releasestream/4-stable/release/4.6.0-0.okd-2020-11-27-200126
https://github.com/openshift/okd/releases/tag/4.6.0-0.okd-2020-11-27-200126

[titou10titou10]

Is there a migration path from 4.5.0-0.okd-2020-10-15-235428 to 4.6.0-0.okd-xxx ?

[vrutkovs]

Is there a migration path from 4.5.0-0.okd-2020-10-15-235428 to 4.6.0-0.okd-xxx ?

Not yet, should be available shortly once tests finish. See "Upgrades from" section in https://amd64.origin.releases.ci.openshift.org/releasestream/4-stable/release/4.6.0-0.okd-2020-11-27-200126

[marshallford]

Not yet, should be available shortly once tests finish. See "Upgrades from" section in https://amd64.origin.releases.ci.openshift.org/releasestream/4-stable/release/4.6.0-0.okd-2020-11-27-200126

I'm getting:

Unable to show full changelog: could not generate a changelog: error: Could not load commits for https://github.com/openshift/installer: from commit 63200c80c431b8dbaa06c0cc13282d819bd7e5f8 does not exist