[Help] Unable to upgrade a managed nodegroup

[ybykov-a9s]

Hello!

I can’t upgrade a managed nodegroup using eksctl

Following document was used for the procedure:
https://docs.aws.amazon.com/eks/latest/userguide/update-managed-node-group.html#mng-update

Steps to reproduce:

Create a cluster using following manifest

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: yby-test
  region: eu-central-1
  version: "1.28"
managedNodeGroups:
  - name: mng-medium
    instanceType: t3a.medium
    desiredCapacity: 2
    minSize: 1
    maxSize: 2
    volumeSize: 10
    iam:
      withAddonPolicies:
        ebs: true

It gets created successfully

Next I upgrade a control plane's kubernetes version using following command:

eksctl upgrade cluster --name yby-test --region eu-central-1 --approve

Everything works fine:

2024-09-27 15:05:31 [ℹ]  will upgrade cluster "yby-test" control plane from current version "1.28" to "1.29"
2024-09-27 15:14:54 [✔]  cluster "yby-test" control plane has been upgraded to version "1.29"
2024-09-27 15:14:54 [ℹ]  you will need to follow the upgrade procedure for all of nodegroups and add-ons
2024-09-27 15:14:55 [ℹ]  re-building cluster stack "eksctl-yby-test-cluster"
2024-09-27 15:14:55 [✔]  all resources in cluster stack "eksctl-yby-test-cluster" are up-to-date
2024-09-27 15:14:55 [ℹ]  checking security group configuration for all nodegroups
2024-09-27 15:14:55 [ℹ]  all nodegroups have up-to-date cloudformation templates

And then I try to upgrade nodegroup to the target version using:

eksctl upgrade nodegroup --cluster yby-test --region eu-central-1 --name mng-medium --kubernetes-version=1.29

Here is the log:

2024-09-27 15:17:45 [ℹ]  will upgrade nodes to release version: 1.29.8-20240917
2024-09-27 15:17:45 [ℹ]  upgrading nodegroup version
2024-09-27 15:17:45 [ℹ]  updating nodegroup stack
2024-09-27 15:17:46 [ℹ]  waiting for CloudFormation changeset "eksctl-update-nodegroup-1727443065" for stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:18:16 [ℹ]  waiting for CloudFormation changeset "eksctl-update-nodegroup-1727443065" for stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:18:16 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:18:46 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:19:28 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:20:28 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:21:36 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:23:25 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:24:58 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:25:34 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:26:44 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:28:41 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:30:12 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:31:48 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
2024-09-27 15:33:39 [ℹ]  waiting for CloudFormation stack "eksctl-yby-test-nodegroup-mng-medium"
Error: error updating nodegroup stack: waiter state transitioned to Failure

If I check Cloudformation console I see a following event:

ManagedNodeGroup
Resource handler returned message: "Requested release version 1.29.8-20240917 is not valid for kubernetes version 1.28. (Service: Eks, Status Code: 400, Request ID: 00c5f96d-c686-42a6-98e8-06abde8621d6)" (RequestToken: 38c80d12-e9e8-12b7-ab49-6c7cf2a65b6c, HandlerErrorCode: InvalidRequest)

If I try to upgrade a node pool using AWS web console everything works fine, but without any changes in Cloudformation logs. Therefore I suppose it doesn't use Cloudformation.

eksctl version
0.190.0-dev+3fccc8ed8.2024-09-04T12:58:57Z

What help do you need?

Please point me if I misunderstood the documentation or if it's a bug.
Maybe there are other actions which nave to be done.

Tell me if I should provide more information or tests.

Thanks in advance.

--
Eugene Bykov

[github-actions]

Hello ybykov-a9s 👋 Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

[TreeKat71]

I am also facing this issue. And I checked the CloudFormation, it mentioned the Resource handler returned message: "Volume of size 10GB is smaller than snapshot 'snap-0145xxxxxx10a66e4', expect size>= 20GB

But I can do that (less than 20 GB) in my another account. They are both in the same region. The only difference I can think of is k8s cluster version. I can create node with 10 GB in 1.29, can't in 1.30

The eksctl version I am using is 0.184

[roman5595]

Any update on this ? Im facing same issue, Resource handler returned message: "Requested release version 1.31.0-20241024 is not valid for kubernetes version 1.30. (Service: Eks, Status Code: 400, Request ID: 15e2fb73-4134-4763-94d4-6b1ffc6d04b3)" (RequestToken: 1565436d-5bbc-7be1-7081-7a0631cf5842, HandlerErrorCode: InvalidRequest)

After I upgraded successfully control plane to 1.31, i cannot upgrade managed node group to 1.31.

[TreeKat71]

Requested release version 1.31.0-20241024 is not valid for kubernetes version 1.30

Are you sure your control plane is already updated?

---

And I found the reason why I can not upgrade my managed nodegroup.

I am also facing this issue. And I checked the CloudFormation, it mentioned the Resource handler returned message: "Volume of size 10GB is smaller than snapshot 'snap-0145xxxxxx10a66e4', expect size>= 20GB

But I can do that (less than 20 GB) in my another account. They are both in the same region. The only difference I can think of is k8s cluster version. I can create node with 10 GB in 1.29, can't in 1.30

The eksctl version I am using is 0.184

I am using two different AMI and OS. So AmazonLinux2 is able to reduce the disk size to 8 but AmazonLinux2023 can not.
That is what I currently know... I am not sure if it is documented.