Stabilize spec 3.4.0

[prestist]

Marking an experimental spec as stable

When an experimental version of the Ignition config spec (e.g.: 3.1.0-experimental) is to be declared stable (e.g. 3.1.0), there are a handful of changes that must be made to the code base. These changes should have the following effects:

• Any configs with a version field set to the previously experimental version will no longer pass validation. For example, if 3.1.0-experimental is being marked as stable, any configs written for 3.1.0-experimental should have their version fields changed to 3.1.0, for Ignition will no longer accept them.
• A new experimental spec version will be created. For example, if 3.1.0-experimental is being marked as stable, a new version of 3.2.0-experimental (or 4.0.0-experimental if backwards incompatible changes are being made) will now be accepted, and start to accumulate new changes to the spec.
• The new stable spec and the new experimental spec will be identical except for the accepted versions. The new experimental spec is a direct copy of the old experimental spec, and no new changes to the spec have been made yet, so initially the two specs will have the same fields and semantics.
• The HTTP Accept header that Ignition uses whenever fetching a config will be updated to advertise the new stable spec.
• New features will be documented in the Upgrading Configs documentation.

The changes that are required to achieve these effects are typically the following:

Making the experimental package stable

• Rename config/vX_Y_experimental to config/vX_Y, and update the golang package statements
• Drop _experimental from all imports in config/vX_Y
• Update MaxVersion in config/vX_Y/types/config.go to delete the PreRelease field
• Update config/vX_Y/config.go to update the comment block on ParseCompatibleVersion
• Update config/vX_Y/config_test.go to test that the new stable version is valid and the old experimental version is invalid
• Update the Accept header in internal/resource/url.go to specify the new spec version.

Creating the new experimental package

• Copy config/vX_Y into config/vX_(Y+1)_experimental, and update the golang package statements
• Update all config/vX_Y imports in config/vX_(Y+1)_experimental to config/vX_(Y+1)_experimental
• Update config/vX_(Y+1)_experimental/types/config.go to set MaxVersion to the correct major/minor versions with PreRelease set to "experimental"
• Update config/vX_(Y+1)_experimental/config.go to point the prev import to the new stable vX_Y package and update the comment block on ParseCompatibleVersion
• Update config/vX_(Y+1)_experimental/config_test.go to test that the new stable version is invalid and the new experimental version is valid
• Update config/vX_(Y+1)_experimental/translate/translate.go to translate from the previous stable version. Update the old_types import, delete all functions except translateIgnition and Translate, and ensure translateIgnition translates the entire Ignition struct.
• Update config/vX_(Y+1)_experimental/translate/translate_test.go to point the old import to the new stable vX_Y/types package
• Update config/config.go imports to point to the experimental version.
• Update config/config_test.go to add the new experimental version to TestConfigStructure.
• Update generate to generate the new stable and experimental versions, and rerun generate.

Update all relevant places to use the new experimental package

• All places that imported config/vX_Y_experimental should be updated to config/vX_(Y+1)_experimental.
• Update tests/register/register.go in the following ways:
  • Add import config/vX_Y/types
  • Update import config/vX_Y_experimental/types to config/vX_(Y+1)_experimental/types
  • Add config/vX_Y/types's identifier to configVersions in Register()

Update the blackbox tests

• Bump the invalid -experimental version in the relevant VersionOnlyConfig test in tests/negative/general/config.go.
• Find all tests using X.Y.0-experimental and alter them to use X.Y.0.
• Update the Accept header checks in tests/servers/servers.go to specify the new spec version.

Update docs

• Rename docs/configuration-vX_Y-experimental.md to docs/configuration-vX_Y.md and make a copy as docs/configuration-vX_(Y+1)_experimental.md.
• In docs/configuration-vX_Y.md, drop -experimental from the version number in the heading and the ignition.version field, and drop the prerelease warning. Update the nav_order field in the Jekyll front matter to be one less than the nav_order of the previous stable spec.
• In docs/configuration-vX_(Y+1)_experimental.md, update the version of the experimental spec in the heading and the ignition.version field.
• Add a section to docs/migrating-configs.md.
• In docs/specs.md, update the list of stable and experimental spec versions (listing the latest stable release first) and update the table listing the Ignition release where a spec has been marked as stable.
• Note the stabilization in docs/release-notes.md, following the format of previous stabilizations. Drop the -exp version suffix from any notes for the upcoming release.

External tests

If there are any external kola tests that were using the now stabilized experimental spec that are not part of the Ignition repo (e.g. tests in the fedora-coreos-config repo), CI will fail for the spec stabilization PR.

For tests using experimental Ignition configs: (none)

• Uncomment the commented-out workaround for this in .cci.jenkinsfile.
• When bumping the Ignition package in fedora-coreos-config, you'll need to update the external test in that repo to make CI green.
• Comment out the workaround.

For tests using experimental Butane configs:

see #1553 (comment)

• Snooze the affected tests in kola-denylist.yaml.
• Stabilize the Butane spec and revendor into coreos-assembler.
• Drop the snoozes.

Other packages

• Add a stable spec to ignition-config-rs and regenerate schema.
  • Put out a new release.
• Bump ignition-config-rs in coreos-installer to support the new spec in iso customize and pxe customize. Update release notes.
  • Put out a new coreos-installer release - New release 0.17.0 coreos-installer#1123
• Add a new downgrade translation to ign-converter.
• Stabilize Butane specs. - see Stabilize spec 3.4.0 #1553 (comment)
  • Put out a new release.
• Drop -experimental from configs in FCOS docs and remove colocated experimental-config warnings - Stabilize Butane spec 1.5.0 fedora-coreos-docs#509
• Revendor Ignition and Butane into coreos-assembler and update mantle/platform/conf/conf.go and conf_test.go - see Stabilize spec 3.4.0 #1553 (comment)
• Ask the Machine Config Operator to support the new spec.

[bgilbert]

Reopening to track followup changes in other projects.

[bgilbert]

To avoid snoozing tests for a long time, we've now broken the external test ratchet into two pieces:

Stabilize Ignition spec

• Snooze the affected tests in kola-denylist.yaml
• Stabilize Ignition in Butane (but not the Butane spec) - Stabilize Ignition spec 3.4.0 butane#435
• Revendor Ignition and Butane into coreos-assembler and update mantle/platform/conf/conf.go and conf_test.go - mantle: stabilize Ignition 3.4.0 spec coreos-assembler#3387
• Drop the snoozes - denylist: drop snoozes for Ignition spec bump fedora-coreos-config#2286

Stabilize Butane spec

• Stabilize Butane spec - fcos|r4e|flatcar spec stabilization butane#442
• Put out a new Butane release - Release 0.18.0 butane#430
• Disable affected tests - tests/kola: hard-disable tests using Butane experimental spec fedora-coreos-config#2332
• Revendor Butane into coreos-assembler - go.mod: update Butane to 0.18.0 coreos-assembler#3409
• Update tests to drop -experimental suffix; re-enable tests - Stabilize Butane 1.5.0 spec and re-enable tests fedora-coreos-config#2329

[bgilbert]

The stabilization ratchet is now complete. The only remaining step is finishing the Butane release, but that's routine, and is tracked separately in coreos/butane#430. Closing this out.