Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DO NOT MERGE] Test if updated kubespawner resolves culler failure #3149

Closed

Conversation

sgibson91
Copy link
Member

ref #3130

I have opened this for visibility into what I have done, but shouldn't be merged. If updating kubespawner fixes the issue, then we should aim for a new release and update it in the main hub image, not maintain another image.

We are hoping that with the deployment of this image, some long-lived pods will be successfully culled.

@github-actions
Copy link

Merging this PR will trigger the following deployment actions.

Support and Staging deployments

Cloud Provider Cluster Name Upgrade Support? Reason for Support Redeploy Upgrade Staging? Reason for Staging Redeploy
gcp 2i2c No Yes Following prod hubs require redeploy: ucmerced

Production deployments

Cloud Provider Cluster Name Hub Name Reason for Redeploy
gcp 2i2c ucmerced Following helm chart values files were modified: ucmerced.values.yaml

Copy link
Member

@yuvipanda yuvipanda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've undeployed this by deploying from master temporarily! We can try again later

@@ -27,8 +27,8 @@ jupyterhub:
url: http://www.ucmerced.edu/
singleuser:
image:
name: quay.io/2i2c/2i2c-hubs-image
tag: "14107b8a85fb"
name: quay.io/2i2c/culler-failure
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should actually be setting the image under hub.image rather than singleuser.image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened #3150 about having a separate staging hub for ucmerced perhaps, given they're still on the old user image and there may be other things we need to test.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woof, is it too late in the game to change that image name? I was definitely lead to believe I was changing the right thing since it was the "hubs-image"

Copy link
Member

@consideRatio consideRatio Sep 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That name has confused me also, but since we are phasing it out I think we shouldn't work on changing the name at this point.

I wonder if we have this failure-to-cull result associated with using that singleuser image and/or configuring a specific jupyter server application. It could be that the user servers incorrectly reports they are active when they arent, and that is fixed by a modern jupyter server, but we are stuck with old versions in this hub because we use this old image?

@NessieCanCode
Copy link

Hi @sgibson91, I'm an admin at UC Merced, and I think this Test image is causing disruption to our users, which I would like to bring to your attention. https://2i2c.freshdesk.com/support/tickets/980 I was able to reproduce the issue as well.

@yuvipanda
Copy link
Member

Hey @NessieCanCode! I've undeployed this change, and things should be working now! I also replied on the ticket :)

@yuvipanda
Copy link
Member

@NessieCanCode also, welcome to our github :)

Investigating culler fialures
@sgibson91 sgibson91 force-pushed the culler-failure-experimental-image branch from 05ea398 to 96f2a27 Compare September 18, 2023 11:38
@sgibson91
Copy link
Member Author

I reverted this and redeployed from master again

@sgibson91 sgibson91 closed this Sep 19, 2023
@sgibson91 sgibson91 deleted the culler-failure-experimental-image branch April 13, 2024 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Status: Done 🎉
Development

Successfully merging this pull request may close these issues.

4 participants