Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(jobs): MLOps example #64

Merged
merged 23 commits into from
Dec 14, 2023
Merged

feat(jobs): MLOps example #64

merged 23 commits into from
Dec 14, 2023

Conversation

redanrd
Copy link
Contributor

@redanrd redanrd commented Nov 27, 2023

Summary

Machine Learning workflow example using:

  • Serverless containers for ML serving.
  • Serverless jobs for ML training.
  • Container registry for docker images.
  • Object storage for creating a data store, a model registry, and a performance monitoring store.

Checklist

  • I have reviewed this myself.
  • I have attached a README to my example. You can use this template as reference.
  • I have updated the project README to link my example.

@redanrd redanrd self-assigned this Nov 27, 2023
@redanrd redanrd marked this pull request as ready for review November 28, 2023 08:56
@redanrd redanrd changed the title MLOps Example Serverless MLOps Example Nov 28, 2023
README.md Outdated Show resolved Hide resolved
Copy link
Contributor

@cyclimse cyclimse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great example! You were really quick ^^ Would love if you did a quick demo of it!! Or maybe even a blog post :D

My main remark is with secret management (I recommend using serverless container secrets instead of a .env in the image) + unifying the Terraform setup. The rest of the comments can be ignored.

jobs/ml-ops/container/inference-api/data_processing.py Outdated Show resolved Hide resolved
jobs/ml-ops/container/README.md Outdated Show resolved Hide resolved
jobs/ml-ops/container/inference-api/Dockerfile Outdated Show resolved Hide resolved
jobs/ml-ops/container/inference-api/Dockerfile Outdated Show resolved Hide resolved
jobs/ml-ops/container/inference-api/data_processing.py Outdated Show resolved Hide resolved
jobs/ml-ops/container/inference-api/main.py Outdated Show resolved Hide resolved
jobs/ml-ops/container/terraform/providers.tf Outdated Show resolved Hide resolved
jobs/ml-ops/registry/README.md Outdated Show resolved Hide resolved
jobs/ml-ops/s3/data-store/main.py Outdated Show resolved Hide resolved
jobs/ml-ops/job/ml_training.py Outdated Show resolved Hide resolved
@Shillaker Shillaker changed the title Serverless MLOps Example feat(jobs): MLOps example Dec 1, 2023
Copy link
Contributor

@cyclimse cyclimse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! It's super clean, and I love using the Terraform provider to create the JobDef 😍

You can ignore my comments, it's all nitpicks.

jobs/ml-ops/inference/main.py Outdated Show resolved Hide resolved
jobs/ml-ops/inference/main.py Outdated Show resolved Hide resolved
jobs/ml-ops/terraform/providers.tf Show resolved Hide resolved
jobs/ml-ops/terraform/jobs.tf Outdated Show resolved Hide resolved
@redanrd redanrd merged commit 3b63d8f into main Dec 14, 2023
4 checks passed
@redanrd redanrd deleted the jobs-ml-example branch December 14, 2023 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants