Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

k8s #1

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions .github/workflows/ci-cd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,12 @@ jobs:
- name: Test with pytest
run: |
pytest -v
# - name: Deploy to Heroku
# env:
# HEROKU_API_TOKEN: ${{ secrets.HEROKU_API_TOKEN }}
# HEROKU_APP_NAME: ${{ secrets.HEROKU_APP_NAME }}
# if: github.ref == 'refs/heads/main' && job.status == 'success'
# run: |
# echo '-----------HEROKU_APP_NAME------------'
# git remote add heroku https://heroku:[email protected]/$HEROKU_APP_NAME.git
# git push heroku HEAD:main -f
- name: Deploy to Heroku
env:
HEROKU_API_TOKEN: ${{ secrets.HEROKU_API_TOKEN }}
HEROKU_APP_NAME: ${{ secrets.HEROKU_APP_NAME }}
if: github.ref == 'refs/heads/main' && job.status == 'success'
run: |
echo '-----------HEROKU_APP_NAME------------'
git remote add heroku https://heroku:[email protected]/$HEROKU_APP_NAME.git
git push heroku HEAD:main -f
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,4 @@ RUN pip install -r requirements.txt

# Run the streamlit on container startup
# CMD [ "streamlit", "run","--server.enableCORS","false","imgwebapp.py" ]
CMD ["app.py"]
CMD ["python", "app.py"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
artifact_path: model
flavors:
python_function:
env: conda.yaml
loader_module: mlflow.sklearn
model_path: model.pkl
python_version: 3.9.6
sklearn:
pickled_model: model.pkl
serialization_format: cloudpickle
sklearn_version: 1.0.1
model_uuid: fcec9392ebc64cd79ac53c6ea8246889
run_id: c62a732627a44abdaea195e7df7365f3
utc_time_created: '2022-04-06 17:43:34.425149'
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
channels:
- conda-forge
dependencies:
- python=3.9.6
- pip
- pip:
- mlflow
- cloudpickle==2.0.0
- psutil==5.9.0
- scikit-learn==1.0.1
name: mlflow-env
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
mlflow
cloudpickle==2.0.0
psutil==5.9.0
scikit-learn==1.0.1
15 changes: 15 additions & 0 deletions ElasticNet Regression/0/c62a732627a44abdaea195e7df7365f3/meta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
artifact_uri: ElasticNet Regression/0/c62a732627a44abdaea195e7df7365f3/artifacts
end_time: 1649267016976
entry_point_name: ''
experiment_id: '0'
lifecycle_stage: active
name: ''
run_id: c62a732627a44abdaea195e7df7365f3
run_uuid: c62a732627a44abdaea195e7df7365f3
source_name: ''
source_type: 4
source_version: ''
start_time: 1649267014388
status: 4
tags: []
user_id: sanand
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1649267014416 0.65671588782232 0
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1649267014418 0.009995540913424361 0
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1649267014415 0.804346762796031 0
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.7
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.6
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
42
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[{"run_id": "c62a732627a44abdaea195e7df7365f3", "artifact_path": "model", "utc_time_created": "2022-04-06 17:43:34.425149", "flavors": {"python_function": {"model_path": "model.pkl", "loader_module": "mlflow.sklearn", "python_version": "3.9.6", "env": "conda.yaml"}, "sklearn": {"pickled_model": "model.pkl", "sklearn_version": "1.0.1", "serialization_format": "cloudpickle"}}, "model_uuid": "fcec9392ebc64cd79ac53c6ea8246889"}]
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ElasticNetWineModel
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
e62a81860e2f9ec863a8867270843e1f5bd5a0c1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
src/models/train_and_evaluate.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
LOCAL
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
sanand
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
artifact_path: model
flavors:
python_function:
env: conda.yaml
loader_module: mlflow.sklearn
model_path: model.pkl
python_version: 3.9.6
sklearn:
pickled_model: model.pkl
serialization_format: cloudpickle
sklearn_version: 1.0.1
model_uuid: c7cb88be9fd348ce81ee7e1f01c131ad
run_id: c92363bfed564b87910fc53950a43a2e
utc_time_created: '2022-04-06 19:29:29.121091'
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
channels:
- conda-forge
dependencies:
- python=3.9.6
- pip
- pip:
- mlflow
- cloudpickle==2.0.0
- psutil==5.9.0
- scikit-learn==1.0.1
name: mlflow-env
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
mlflow
cloudpickle==2.0.0
psutil==5.9.0
scikit-learn==1.0.1
15 changes: 15 additions & 0 deletions ElasticNet Regression/0/c92363bfed564b87910fc53950a43a2e/meta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
artifact_uri: ElasticNet Regression/0/c92363bfed564b87910fc53950a43a2e/artifacts
end_time: 1649273371592
entry_point_name: ''
experiment_id: '0'
lifecycle_stage: active
name: ''
run_id: c92363bfed564b87910fc53950a43a2e
run_uuid: c92363bfed564b87910fc53950a43a2e
source_name: ''
source_type: 4
source_version: ''
start_time: 1649273369082
status: 4
tags: []
user_id: sanand
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1649273369113 0.65671588782232 0
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1649273369114 0.009995540913424361 0
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1649273369111 0.804346762796031 0
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.7
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.6
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
42
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[{"run_id": "c92363bfed564b87910fc53950a43a2e", "artifact_path": "model", "utc_time_created": "2022-04-06 19:29:29.121091", "flavors": {"python_function": {"model_path": "model.pkl", "loader_module": "mlflow.sklearn", "python_version": "3.9.6", "env": "conda.yaml"}, "sklearn": {"pickled_model": "model.pkl", "sklearn_version": "1.0.1", "serialization_format": "cloudpickle"}}, "model_uuid": "c7cb88be9fd348ce81ee7e1f01c131ad"}]
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ElasticNetWineModel
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
e62a81860e2f9ec863a8867270843e1f5bd5a0c1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
src/models/train_and_evaluate.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
LOCAL
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
sanand
4 changes: 4 additions & 0 deletions ElasticNet Regression/0/meta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
artifact_location: ElasticNet Regression/0
experiment_id: '0'
lifecycle_stage: active
name: Default
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
artifact_path: model
flavors:
python_function:
env: conda.yaml
loader_module: mlflow.sklearn
model_path: model.pkl
python_version: 3.9.6
sklearn:
pickled_model: model.pkl
serialization_format: cloudpickle
sklearn_version: 1.0.1
model_uuid: 8b7d64fae99e4e18ac42ea03ec6e3d15
run_id: 8333c552040b40a19faabc0ef5c77dfc
utc_time_created: '2022-04-06 17:49:37.722370'
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
channels:
- conda-forge
dependencies:
- python=3.9.6
- pip
- pip:
- mlflow
- cloudpickle==2.0.0
- psutil==5.9.0
- scikit-learn==1.0.1
name: mlflow-env
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
mlflow
cloudpickle==2.0.0
psutil==5.9.0
scikit-learn==1.0.1
15 changes: 15 additions & 0 deletions ElasticNetRegression/0/8333c552040b40a19faabc0ef5c77dfc/meta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
artifact_uri: ElasticNetRegression/0/8333c552040b40a19faabc0ef5c77dfc/artifacts
end_time: 1649267380172
entry_point_name: ''
experiment_id: '0'
lifecycle_stage: active
name: ''
run_id: 8333c552040b40a19faabc0ef5c77dfc
run_uuid: 8333c552040b40a19faabc0ef5c77dfc
source_name: ''
source_type: 4
source_version: ''
start_time: 1649267377685
status: 4
tags: []
user_id: sanand
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1649267377713 0.65671588782232 0
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1649267377714 0.009995540913424361 0
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1649267377711 0.804346762796031 0
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.7
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.6
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
42
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[{"run_id": "8333c552040b40a19faabc0ef5c77dfc", "artifact_path": "model", "utc_time_created": "2022-04-06 17:49:37.722370", "flavors": {"python_function": {"model_path": "model.pkl", "loader_module": "mlflow.sklearn", "python_version": "3.9.6", "env": "conda.yaml"}, "sklearn": {"pickled_model": "model.pkl", "sklearn_version": "1.0.1", "serialization_format": "cloudpickle"}}, "model_uuid": "8b7d64fae99e4e18ac42ea03ec6e3d15"}]
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ElasticNetWineModel
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
e62a81860e2f9ec863a8867270843e1f5bd5a0c1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
src/models/train_and_evaluate.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
LOCAL
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
sanand
4 changes: 4 additions & 0 deletions ElasticNetRegression/0/meta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
artifact_location: ElasticNetRegression/0
experiment_id: '0'
lifecycle_stage: active
name: Default
24 changes: 20 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,17 @@
mlops_demo
==============================
Model Development and Productionization
=======================================
|-- Project Organization
|  | <- Cookiecutter `https://cookiecutter.readthedocs.io/en/stable/`
|-- Data Version Control
|  | <- DVC `https://dvc.org/`
|-- CI+CD - Github
|-- Model Productionization/Deployment
|  |-- UI <- Flask
|  |-- Scalability and high availability <- k8
|-- FeedBack and Retrain



Mlops demo

Project Organization
------------
Expand Down Expand Up @@ -73,4 +83,10 @@ Project Organization
`6. search for cloud build >> trigger`</br>
```7. gcloud projects add-iam-policy-binding hft-demo-v1 --member=serviceAccount:[email protected] --role=roles/container.developer```</br>
`8. k8s deploy`</br>
`9. k8s failing due to permission issue and time out`</br>
`9. k8s failing due to permission issue and time out`</br>
`10. Configure terraform for k8s creation: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/guides/getting-started`


`Ml flow execute--`

`mlflow server --backend-store-uri sqlite:///mlflow.db --default-artifact-root ./artifact --host 0.0.0.0 -p 123`
2 changes: 1 addition & 1 deletion app.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,4 @@ def index():


if __name__ == "__main__":
app.run(host='0.0.0.0', port=5001, debug=True)
app.run(host='0.0.0.0', port=5000, debug=True)
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
artifact_path: model
flavors:
python_function:
env: conda.yaml
loader_module: mlflow.sklearn
model_path: model.pkl
python_version: 3.9.6
sklearn:
pickled_model: model.pkl
serialization_format: cloudpickle
sklearn_version: 1.0.1
model_uuid: ef2527ce53194b8d97def52f652735a9
run_id: 12d1df0e748445e3827d2c5bca9a255b
utc_time_created: '2022-04-06 19:51:30.072091'
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
channels:
- conda-forge
dependencies:
- python=3.9.6
- pip
- pip:
- mlflow
- cloudpickle==2.0.0
- psutil==5.9.0
- scikit-learn==1.0.1
name: mlflow-env
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
mlflow
cloudpickle==2.0.0
psutil==5.9.0
scikit-learn==1.0.1
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
artifact_path: model
flavors:
python_function:
env: conda.yaml
loader_module: mlflow.sklearn
model_path: model.pkl
python_version: 3.9.6
sklearn:
pickled_model: model.pkl
serialization_format: cloudpickle
sklearn_version: 1.0.1
model_uuid: 96b3a88e1fc24b838149a9bd8d7c829e
run_id: c97263c68b474462b4a0532fb88c042c
utc_time_created: '2022-04-06 19:50:48.751639'
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
channels:
- conda-forge
dependencies:
- python=3.9.6
- pip
- pip:
- mlflow
- cloudpickle==2.0.0
- psutil==5.9.0
- scikit-learn==1.0.1
name: mlflow-env
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
mlflow
cloudpickle==2.0.0
psutil==5.9.0
scikit-learn==1.0.1
Loading