Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

if experiment name is too long the suggestion service can't start #2454

Open
garymm opened this issue Nov 12, 2024 · 4 comments
Open

if experiment name is too long the suggestion service can't start #2454

garymm opened this issue Nov 12, 2024 · 4 comments
Assignees
Labels

Comments

@garymm
Copy link

garymm commented Nov 12, 2024

What happened?

I provided an experiment name that was 57 characters long.
It got stuck waiting for trials to be created because the suggestion service couldn't be started because the name was more than 63 characters long.

What did you expect to happen?

Katib to pick a valid name for the service.

Environment

Kubernetes version:

$ kubectl version
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.3

Katib controller version:

$ `kubectl get pods -n kubeflow -l katib.kubeflow.org/component=controller -o jsonpath="{.items[*].spec.containers[*].image}"
docker.io/kubeflowkatib/katib-controller:v0.17.0

Katib Python SDK version:

$ pip show kubeflow-katib
Name: kubeflow-katib
Version: 0.17.0

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

@garymm
Copy link
Author

garymm commented Nov 12, 2024

I may be interested in contributing a fix if it would be welcome and some guidance could be provided as to how to go about it.

@andreyvelich
Copy link
Member

Thank you for creating this @garymm!
Yes, it's a great idea, we can just add the Experiment validation here: https://github.com/kubeflow/katib/blob/master/pkg/webhook/v1beta1/experiment/validator/validator.go#L81

/assign @garymm

Feel free to reach out if you have any questions.

@andreyvelich
Copy link
Member

/remove-label lifecycle/needs-triage

@garymm
Copy link
Author

garymm commented Nov 15, 2024

What exactly should the validaton be though? I think the better fix is to name the experiment service in a way that is guaranteed to be legal. Where might that happen in the code?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants