Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bitnami/mlflow] Add support for Google Cloud Storage as artifact store #28938

Merged
merged 12 commits into from
Sep 6, 2024

Conversation

mautini
Copy link
Contributor

@mautini mautini commented Aug 20, 2024

Description of the change

This PR follows this one #26461 and integration of google-cloud-storage library in bitnami mlflow Docker image: bitnami/containers#67246

It adds support to Google Cloud Storage as backend store for mlflow artifacts. The use could specify credentials via workload identity or application credential.

Benefits

User will be able to use Google Storage as backend for artifact storage in mlflow tracking. This is useful for other that migrate from another mlflow instance (running with GCS as backend) or that cannot use other cloud provider.

Possible drawbacks

More complexity to the helm chart / configuration? But it's a feature supported by mlflow so I think we have to expose it.

Checklist

  • Chart version bumped in Chart.yaml according to semver. This is not necessary when the changes only affect README.md files.
  • Variables are documented in the values.yaml and added to the README.md using readme-generator-for-helm
  • Title of the pull request follows this pattern [bitnami/<name_of_the_chart>] Descriptive title
  • All commits signed off and in agreement of Developer Certificate of Origin (DCO)

@github-actions github-actions bot added mlflow triage Triage is needed labels Aug 20, 2024
@github-actions github-actions bot requested a review from carrodher August 20, 2024 16:21
Signed-off-by: Bitnami Containers <[email protected]>
@carrodher carrodher added verify Execute verification workflow for these changes in-progress labels Aug 21, 2024
@github-actions github-actions bot removed the triage Triage is needed label Aug 21, 2024
@github-actions github-actions bot removed the request for review from carrodher August 21, 2024 06:47
@github-actions github-actions bot requested a review from andresbono August 21, 2024 06:47
Signed-off-by: Bitnami Containers <[email protected]>
carrodher and others added 2 commits September 2, 2024 09:50
Signed-off-by: Carlos Rodríguez Hernández <[email protected]>
Signed-off-by: Bitnami Containers <[email protected]>
@andresbono
Copy link
Contributor

Thank you @mautini for this contribution! The CI failed for some reason. Let me retry it and I will review the PR. Thanks!

Copy link
Contributor

@andresbono andresbono left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution!! 🎉 I left some suggestions and questions. Could you please look at them?

Also, could you share if you performed some specific validation of the changes on your own env? To better understand the use case as well.

bitnami/mlflow/templates/_helpers.tpl Outdated Show resolved Hide resolved
bitnami/mlflow/templates/tracking/deployment.yaml Outdated Show resolved Hide resolved
bitnami/mlflow/templates/tracking/deployment.yaml Outdated Show resolved Hide resolved
@@ -273,6 +288,11 @@ spec:
{{- if .Values.tracking.persistence.subPath }}
subPath: {{ .Values.tracking.persistence.subPath }}
{{- end }}
{{- if and (include "mlflow.v0.gcs.enabled" .) .Values.externalGCS.useCredentialsInSecret .Values.externalGCS.existingSecret }}
- name: google-application-credentials
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no volume called like that, should it be gcs-key? Or the other way around.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. I took this part from the original MR there: #26461 as I don't use this authentication method. Let's rename it to gcs-key

bitnami/mlflow/values.yaml Outdated Show resolved Hide resolved
mautini and others added 2 commits September 4, 2024 12:43
Co-authored-by: Andrés Bono <[email protected]>
Signed-off-by: Nicolas Mauti <[email protected]>
Signed-off-by: Nicolas Mauti <[email protected]>
@mautini
Copy link
Contributor Author

mautini commented Sep 4, 2024

Hi @andresbono, thanks a lot for your careful review. I applied your suggested changes, responded to your questions and pushed a fix for the mounting of key.json. Let me know if there is something left.

Signed-off-by: Bitnami Containers <[email protected]>
Copy link
Contributor

@andresbono andresbono left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the additional context!

I just added a few minor suggestions. If you agree with them, you can apply them and the PR is ready to go 🚀

bitnami/mlflow/templates/_helpers.tpl Outdated Show resolved Hide resolved
bitnami/mlflow/templates/tracking/deployment.yaml Outdated Show resolved Hide resolved
bitnami/mlflow/templates/tracking/deployment.yaml Outdated Show resolved Hide resolved
bitnami/mlflow/values.yaml Outdated Show resolved Hide resolved
bitnami/mlflow/values.yaml Outdated Show resolved Hide resolved
Co-authored-by: Andrés Bono <[email protected]>
Signed-off-by: Nicolas Mauti <[email protected]>
@mautini
Copy link
Contributor Author

mautini commented Sep 5, 2024

All good for me, thanks!

Signed-off-by: Bitnami Containers <[email protected]>
Copy link
Contributor

@andresbono andresbono left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you so much!

@andresbono andresbono enabled auto-merge (squash) September 6, 2024 15:04
@andresbono andresbono merged commit 2c6d076 into bitnami:main Sep 6, 2024
11 checks passed
mike12806 pushed a commit to mike12806/charts that referenced this pull request Oct 30, 2024
…re (bitnami#28938)

* Add support for GCS artifact storage

Signed-off-by: Nicolas Mauti <[email protected]>

* Generate README from values file

Signed-off-by: Nicolas Mauti <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Bitnami Containers <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Bitnami Containers <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Bitnami Containers <[email protected]>

* Apply suggestions from code review

Co-authored-by: Andrés Bono <[email protected]>
Signed-off-by: Nicolas Mauti <[email protected]>

* Fix keys.json mounting

Signed-off-by: Nicolas Mauti <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Bitnami Containers <[email protected]>

* Apply suggestions from code review

Co-authored-by: Andrés Bono <[email protected]>
Signed-off-by: Nicolas Mauti <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Bitnami Containers <[email protected]>

* Update README.md with readme-generator-for-helm

Signed-off-by: Bitnami Containers <[email protected]>

---------

Signed-off-by: Nicolas Mauti <[email protected]>
Signed-off-by: Bitnami Containers <[email protected]>
Signed-off-by: Carlos Rodríguez Hernández <[email protected]>
Co-authored-by: Bitnami Containers <[email protected]>
Co-authored-by: Carlos Rodríguez Hernández <[email protected]>
Co-authored-by: Andrés Bono <[email protected]>
Signed-off-by: Michael Faherty <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mlflow solved verify Execute verification workflow for these changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants