⚠ kOps 1.24 has not been released yet! ⚠
This is a document to gather the release notes prior to the release.
By enabling the Karpenter
feature flag, users can now create InstanceGroups managed by (https://karpenter.sh)[Karpenter]:
spec:
manager: Karpenter
You can also start a Karpenter-only cluster with kops create cluster --instance-manager=karpenter ...
kOps will directly manage the Karpenter Provisioner resources. Read more about how Karpenter works on kOps in the Karpenter docs.
- The minimum version for the Terraform AWS Provider has been bumped to 4.0.0 to address the deprecation of the aws_s3_bucket_object resource and its replacement with the aws_s3_object resource. Such resources will be destroyed and recreated without downtime when applying the changes.
As of Kubernetes version 1.24, the control plane (formerly master) nodes no longer have the deprecated node-role.kubernetes.io/master
label.
The deprecated node-role.kubernetes.io/master
taint has been replaced by node-role.kubernetes.io/control-plane
. If you run your own workload on the control plane, you have to adjust your Pod spec to accommodate for this change.
The following shows a node affinity/node label selector and tolerations that works with both new and old control plane nodes:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: Exists
- matchExpressions:
- key: node-role.kubernetes.io/master
operator: Exists
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
- key: node-role.kubernetes.io/master
operator: Exists
The deprecated kubernetes.io/role
label has been removed for all roles as of Kubernetes version 1.24. Use node-role.kubernetes.io/<role>
label instead.
-
Support for Kubernetes version 1.18 has been removed.
-
Support for Aliyun/Alibaba Cloud has been removed.
-
Support for Docker has been removed for Kubernetes 1.24+. See https://kubernetes.io/blog/2020/12/02/dockershim-faq
-
Cert Manager upgraded from 1.6 to 1.8. This has backwards-breaking changes. See upgrading from 1.6 to 1.7 and [1.1.7 to 1.8.
-
Support for Kubernetes version 1.19 is deprecated and will be removed in kOps 1.25.
-
Support for Kubernetes version 1.20 is deprecated and will be removed in kOps 1.26.
-
All legacy addons are deprecated in favor of managed addons, including the metrics server addon and the autoscaler addon.
-
Due to lack of maintainers, the CloudFormation support has been deprecated. The current implementation will be left as-is until the implementation needs updates or otherwise becomes incompatible. At that point, it will be removed. We very much welcome anyone willing to contribute to this target.
-
Support for Docker has been removed for Kubernetes 1.24+. See https://kubernetes.io/blog/2020/12/02/dockershim-faq
- Update release notes and minimum k8s version @hakman #12929
- kops auth-plugin: need to clear any existing password / key @justinsb #12921
- Add integration test for k8s 1.24 @olemarkus #12930
- Only shellcheck files @olemarkus #12931
- Do not set insecure-port as of k8s 1.20 @olemarkus #12926
- tests: Improve logging on test failure @justinsb #12933
- nodeup: store the CloudProvider in the context @justinsb #12923
- bazel: always build with pure (CGO_ENABLED=0) @justinsb #12934
- nodeup: print more info on hash mismatches @justinsb #12935
- PKI library: Add initial support for EC keys @justinsb #12936
- Recognize debian bullseye as having "broken" resolv.conf @justinsb #12937
- Remove code for now-unsupported Kubernetes 1.18 @johngmyers #12939
- Add missing k8s 1.18 relnote @johngmyers #12938
- Remove obsolete, redundant secrets.md @johngmyers #12942
- Drop support for Weave as of k8s 1.23 @johngmyers #12941
- Remove support for Aliyun/Alibaba Cloud @johngmyers #12944
- Document CoreDNS configuration settings @recollir #12914
- Update name of kubernetes-ca keypair in documentation @johngmyers #12943
- Revert "Recognize debian bullseye as having "broken" resolv.conf" @olemarkus #12947
- Set the default LT version to the new LT version @olemarkus #12932
- Make service topology for cilium configurable @olemarkus #12918
- gce: ServiceAccount task @justinsb #12950
- Update Calico and Canal to v3.21.2 @hakman #12951
- Update Go to v1.17.5 @hakman #12954
- Skip IPv6 LB test in the k/s e2e @hakman #12953
- GCE: Task for StorageBucket IAM @justinsb #12958
- GCE: Project IAM Binding task @justinsb #12959
- add verify-golangci-lint.sh script @rlankfo #12892
- Hubble relay should not tolerate anything @olemarkus #12963
- Do not explicitly skip Dashboard tests @hakman #12962
- Do not skip NodePort tests for Calico @hakman #12960
- Remove verify-staticcheck @rifelpet #12965
- wait for instances to drain from classic LB @heybronson #12902
- Support Karpenter @olemarkus #12906
- Update containerd to v1.6.0-beta.4 @hakman #12968
- Update controller-runtime to v0.11.0 @hakman #12967
- Add missing permissions @olemarkus #12977
- Do not skip HPA tests @hakman #12972
- Do not skip RuntimeClass tests @hakman #12974
- gce: Use ServiceAccount task when building model @justinsb #12978
- Quote values and remove limits in karpenter provisioners @olemarkus #12979
- Promote alpha with December releases @olemarkus #12984
- gce: map multiple serviceaccounts @justinsb,@hakman #12982
- Defend against nil containerd @justinsb #12990
- Remove unused TemplateResource interface @justinsb #12989
- Avoid double-encoding templates @justinsb #12991
- Refactor nodeup script to avoid action-at-a-distance @justinsb #12993
- gce: use per InstanceGroup serviceaccounts @justinsb #12988
- dep: update github.com/pkg/sftp @justinsb #12996
- Create helper functions for parsing public keys @justinsb #12999
- Use terraform literals in GCP service account references @rifelpet #12995
- kops-controller: use controller-runtime manager @justinsb #12997
- gce: clean up networking objects by reference @justinsb #12987
- componentconfig: expose advertise-address flag for kube-apiserver @justinsb #12998
- Do not allow docker on k8s 1.24+ @olemarkus #12927
- Ignore images hosted in private ECR repositories as containerd cannot pull these @olemarkus #13000
- Skip RuntimeClass tests for older Kubernetes versions @hakman #13003
- Various nill pointer fixes for karpenter @olemarkus #12973
- Set Resource Based Naming on managed subnets @johngmyers #12864
- Add kubetest2-kops flags for overriding instance group fields @rifelpet #13005
- Support creating dualstack internal NLBs @johngmyers #13006
- Skip SCTP check for all versions of k8s 1.23/1.24 @olemarkus #13008
- Use spread constraints rather than affinity to spread pods @olemarkus #12961
- Bump karpenter to 0.5.3 and RBN support @olemarkus #13002
- Validate IGs more strictly after defaults have applied @olemarkus #12660
- Karpenter template fix @olemarkus #13009
- staticcheck cleanup: fixup nodeup/pkg/model @justinsb #13013
- nodeup bash script: use explicit return code @justinsb #13012
- Prevent creation of unsupported etcd clusters @olemarkus #13011
- Create cgroups for kube and runtime if configured @olemarkus #12917
- Do not install ClusterRole and binding used by in-tree volume provider if CSI is used @olemarkus #13010
- kubetest2 - Use the same binary path and env when fetching IGs @rifelpet #13018
- Use fi.Keyset instead of passing tasks around @justinsb #12992
- add instance connection draining for NLBs @heybronson #12966
- Use kubelet --non-masquerade-cidr only for Docker with kubenet @hakman #13007
- Fix dangling ENIs from AWS VPC CNI @olemarkus #13021
- Update k8s dependencies to v1.23.1 @hakman #13022
- Improve HA for various addons @olemarkus #13027
- Add a CLI flag for creating one karpenter-managed IG for worker nodes instead of ASG-managed ones @olemarkus #12975
- Allow IPv6-only subnets @johngmyers #13026
- Support specifying instance requirements per IG @olemarkus #13019
- Remove TerraformJSON feature flag @rifelpet #13029
- LBC has to run on the control plane, so set replicas accordingly @olemarkus #13033
- Fix various typos related to karpenter @olemarkus #13035
- Kube components log to stdout @olemarkus #13038
- Identify pending instances @olemarkus #13040
- Add managed-by label to static kube-proxy pods @olemarkus #13039
- Prefix karpenter logging-config name @olemarkus #13037
- gce: don't set per-IG permissions when using shared account @justinsb #13043
- Add documentation on karpenter @olemarkus #13036
- external CCM for GCE @jiahuif #13017
- Migrate to GCE CCM in k8s 1.24 @johngmyers #13045
- Fix OpenStack SecurityGroupRule/LB When CIDR is IPv6 @iGene #13032
- update deps @zetaab #13047
- Bump Cluster Autoscaler and update manifest @olemarkus #13050
- Use instance requirements with Karpenter @olemarkus #13031
- force update dependencies @zetaab #13055
- Enhance AddHostPathMapping to support a fluent style @justinsb #13062
- addons: support for kopeio-networking addon @justinsb #12727
- Use latest GCP CCM for k8s 1.24 @johngmyers #13066
- Add action for automatically tagging releases @johngmyers #12805
- Bump external-snapshotted to v5.0.0 @olemarkus #13067
- Release 1.24.0-alpha.2 @johngmyers #13069
- Release notes for 1.24.0-alpha.2 @johngmyers #13070
- Update release process for automatic tagging @johngmyers #13075
- Remove temporary restrictions on automatically tagging releases @johngmyers #13071
- add flatcar note related to additionalUserData @shubhindia #13061
- Drain OpenStack loadbalancers @zetaab #12983
- Extend terraform support for IPv6 @rifelpet #13028
- Update containerd to v1.6.0-beta.5 @hakman #13084
- Release notes for 1.22.3 @johngmyers #13085
- Spotinst: Update
spotinst/ocean-controller
to v1.0.81 @liranp #13086 - Support price and priority cluster-autoscaler expanders @danports #13081
- Update containerd to v1.6.0-rc.0 @hakman #13098
- decrease the openstack monitoring default timeout @zetaab #13097
- Don't try to add node name to instances without node object @olemarkus #13106
- fix ipv4+ipv6 sec groups/listeners in OpenStack @zetaab #13093
- Do not create an IAM role for dns-controller on gossip clusters @olemarkus #13110
- Add ipv6 to relnotes @olemarkus #13088
- Use IPv6-only subnets for worker nodes in private IPv6 topology @johngmyers #13030
- Remove networking flags as of k8s 1.24 @olemarkus #13120
- Create helper function for ec2 create/tag-on-create IAM permissions @olemarkus #13104
- Add DescribeRegions to nodeup privs @olemarkus #13114
- Remove featureflag for creating IPv6 clusters @hakman #12788
- Preload channel versions from namespaces @olemarkus #13049
- Don't set unsupported configs by default @olemarkus #13111
- Update pause image to v3.6 @hakman #13125
- Clean up kubelet networking flags for dockershim @hakman #13128
- January bump of channels @olemarkus #13130
- expose external ccm metrics for OpenStack @zetaab #13131
- Update to aws-sdk-go to v1.42.37 @jinhong- #13132
- Fix recommended kops versions in channels @olemarkus #13134
- Tag on create for remaining CCM privileges @olemarkus #12911
- Bump metrics-server to 0.6.0 and enable HA mode @olemarkus #13135
- OpenStack - Add loadbalancer pool monitor to API LB @zetaab #13096
- Bump CCM images @olemarkus #13143
- Bump karpenter to 0.5.6 @olemarkus #13151
- Promote alpha AMIs to stable @yurrriq #13152
- Bump 1.23 version in alpha channel @olemarkus #13153
- Add missing v prefix to default upgrade test version @olemarkus #13155
- Bump cert-manager and related godep to 1.6.2 @olemarkus #13154
- add node-drain-timeout flag to rolling-update @heybronson #13103
- Bump etcd-manager to v3.0.20220128 @olemarkus #13158
- Replace deprecated aws.BackgroundContext with context.Background @justinsb #13162
- Fix nil pointer when IAM not populated @justinsb #13167
- JWKS / IRSA: Expose public ACLs to terraform @justinsb #13166
- [DigitalOcean] update ccm version to 0.1.36 @srikiz #13175
- Bump Ubuntu AMI in alpha @olemarkus #13177
- Use etcd-manager pre-release until final release has been cut @olemarkus #13183
- Bump karpenter to 0.6.0 @olemarkus #13185
- More descriptive error message when public key file can't be opened @nckturner #13186
- update GCE default images @zetaab #13181
- Fix etcd-manager for ipv6 @olemarkus #13191
- Update Calico and Canal to v3.21.4 @hakman #13189
- Update to etcd-manager v3.0.20220203 @justinsb #13196
- Pull k8s-custom-iptables from k8s.gcr.io @justinsb #13194
- Add support for AB tests starting out with released kops version @olemarkus #13174
- Update containerd to v1.6.0-rc.2 @hakman #13198
- tests: ensure that we use ACLs with memfs @justinsb #13165
- Karpenter fixes @olemarkus #13207
- Always enable Leader Election for cloud-controller-manager @jiahuif #13187
- Use short commit sha for default stage location instead of git-describe @olemarkus #13208
- use 1.23.1 ccm for openstack @zetaab #13136
- Document download of test versions @olemarkus #13209
- Remove snapshot controller dependency on ebs csi driver @olemarkus #13213
- fix KCM LogLevel setting not honored @jiahuif #13218
- Fix CSI migration feature gates @olemarkus #13203
- CCM: use flagbuilder instead of manually building argv @jiahuif #13219
- Update containerd to v1.6.0-rc.3 @hakman #13224
- Promote alpha to stable @MoShitrit #13227
- always enable Leader Election for openstack CCM @jiahuif #13220
- Update aws node termination handler to 1.14.0 @ryan-dyer-sp,@ryan-dyer #13092
- [Issue-12293] Fix json output to keep it consistent for single or multiple objects @srikiz #13188
- Fix irsa for k8s < 1.20 @olemarkus #13212
- enable pruning for CCM @jiahuif #13235
- Add support for graceful node shutdown @olemarkus #12994
- allow specify GCP project via env. @jiahuif #13237
- KCM should not run with leader migraton when aws ccm is enabled @olemarkus #13241
- Do not enable graceful shutdown if k8s version < 1.21 @olemarkus #13242
- Update metrics-server e2e test for 0.6.0 @olemarkus #13243
- Install runc from opencontainers/runc @hakman #13240
- Fix nilpointer when graceful shutdown is not configured @olemarkus #13246
- Install contained from the release package @hakman #13248
- CCM: allow setting Controllers for cloudControllerManagerConfig @jiahuif #13252
- CCM: add livenessProbe for GCP CCM @jiahuif #13253
- E2E HA Upgrade/Rollback for Leader Migration @jiahuif #13251
- Bump AWS CNI to 1.10.2 @MoShitrit #13228
- Update supported distros for IPv6 @hakman #13256
- Karpenter on kOps will now use approperiate max pods @olemarkus #13178
- Allow PrefixList for sshAccess and kubernetesApiAccess @hierynomus #13113
- service account workaround for gce @jiahuif #13261
- GCP API health checks @zetaab #13199
- Update containerd to v1.6.0 @hakman #13262
- re-organize Leader Migration test with exec tester @jiahuif #13265
- Update LBC to 2.4.0 @olemarkus #13267
- Enable RBN with AWS CCM 1.22.0-alpha.1 @johngmyers #13268
- Disable some flags in kube-apiserver when logging-format is not text @h3poteto #13264
- kops: Leader Migration testing: run with pure kubetest2 @jiahuif #13276
- Bump k8s versions in alpha with Feb 2022 releases @MoShitrit #13275
- Validate taints in IG spec @olemarkus #13266
- test: use
T.TempDir
to create temporary test directory @Juneezee #13283 - Do not create a cert-manager namespace @olemarkus #13284
- Add missing permissions to aws lbc for irsa @olemarkus #13280
- [DigitalOcean] Implement new VPC if network-cidr flag is specified @srikiz #13060
- Use current tree in presubmit upgrade jobs if version B is latest @olemarkus #13290
- Release notes for 1.22.4 @justinsb #13294
- alpha channel: recommend kOps 1.22.4 @justinsb #13296
- docs: add hubble ui helm chart deployment @eddycharly #13299
- cleanup GCP Cluster Service Accounts @zetaab #13201
- docs for release process shouldn't assume remotes @justinsb #13295
- Release notes for 1.23.0-beta.2 @hakman #13303
- Add support to install EKS Pod Identity Webhook @h3poteto,@olemarkus #13176
- Update kubetest2 deps @olemarkus #13314
- use own function to define CSI image version @zetaab #13311
- Add support for ed25519 keys in AWS @aclevername #13304
- Bump AWS SDK to v1.43.11 @olemarkus #13322
- Make cloudProvider a struct in v1alpha3 API @johngmyers #13059
- Update containerd to v1.6.1 @hakman #13325
- Fix GCE service account creation @zetaab #13310
- Use proper image and add health check @olemarkus #13328
- Update stable and alpha channels @olemarkus #13334
- Release notes for 1.21.5 @hakman #13336
- Add e2e for pod identity webhook @olemarkus #13335
- Add webhook notes + some docs changes @olemarkus #13338
- Only delete node object on GCE @olemarkus #13289
- Release notes for 1.23.0 @hakman #13340
- Bump AWS CCM to 1.22.0-alpha.2 @olemarkus #13342
- Bump CCM 1.22 image. Use the 1.23 image for 1.24 due to latest being broken @olemarkus #13357
- Update channels @hakman #13356
- Recommend enabling IRSA for new clusters @olemarkus #12976
- Post 1.23.0 release doc updates @johngmyers #13359
- Add user to container securityContext and remove command @olemarkus #13343
- [Digital Ocean] e2e tests - Fix seeding for generating random zones @srikiz #13362
- wait for all targetGroups to drain @heybronson #13363
- Support GPU in OpenStack @zetaab #13330
- Add missing permissions to aws lbc for IP targeting @olemarkus #13369
- If kubetest2 fails cluster validation, we run down before exiting @olemarkus #13373
- If image is empty, have kops upgrade fill it in @olemarkus #13374
- Update channels @hakman #13379
- Update HPA docs @ddelange #13367
- Clean up nodeup targets @olemarkus #13370
- Upgrade aws-iam-authenticator to v0.5.5 @glebiller #13381
- Add protocol explicitly to services @olemarkus #13383
- Allow duplicate taint keys @olemarkus #13366
- [Digital Ocean] Remove sfo2 region from the list of supported DO regions @srikiz #13382
- Fix long role names @olemarkus #13364
- Migrate to registry.k8s.io @hakman #13380
- Remove oss-upload target since aliyun support has been removed @olemarkus #13389
- dev: create scripts to make it easier to run e2e tests @justinsb #13161
- Remove pr target @olemarkus #13392