Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MLFlow Upload Fix #10

Merged
merged 6 commits into from
Feb 20, 2024
Merged

MLFlow Upload Fix #10

merged 6 commits into from
Feb 20, 2024

Conversation

klemen1999
Copy link
Collaborator

  • Fixed ExportOnTrainEnd callback's ability to upload to mlflow (also some fixes in Exporter related to this)
  • Edited UploadCheckpoint callback (previously UploadCheckpointOnTrainEnd) so it now uploads currently best checkpoint (based on validation loss) after every validation interval. This way we have some intermediate checkpoint in case training crashes before it is finished.

Copy link

github-actions bot commented Feb 16, 2024

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
4430 3455 78% 0% 🟢

New Files

File Coverage Status
luxonis_train/callbacks/upload_checkpoint.py 39% 🟢
TOTAL 39% 🟢

Modified Files

File Coverage Status
luxonis_train/callbacks/init.py 100% 🟢
luxonis_train/callbacks/export_on_train_end.py 38% 🟢
luxonis_train/core/exporter.py 55% 🟢
TOTAL 64% 🟢

updated for commit: 0abec0e by action🐍

Copy link

github-actions bot commented Feb 16, 2024

Test Results

  6 files    6 suites   36m 39s ⏱️
 31 tests  31 ✅ 0 💤 0 ❌
186 runs  186 ✅ 0 💤 0 ❌

Results for commit 0abec0e.

♻️ This comment has been updated with latest results.

Copy link
Collaborator

@conorsim conorsim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - I did a quick manual test. It appears a couple workflows are failing though

Copy link
Contributor

@jkbmrz jkbmrz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works as expected on my end!

Copy link
Collaborator

@kozlov721 kozlov721 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kozlov721 kozlov721 assigned kozlov721 and unassigned kozlov721 Feb 18, 2024
@kozlov721 kozlov721 added bug Something isn't working fix Fixing a bug and removed bug Something isn't working labels Feb 18, 2024
@kozlov721 kozlov721 changed the title Fix/mlflow upload MLFlow Upload Fix Feb 18, 2024
@kozlov721 kozlov721 merged commit 8e35f25 into dev Feb 20, 2024
10 checks passed
@kozlov721 kozlov721 deleted the fix/mlflow-upload branch February 20, 2024 02:44
kozlov721 pushed a commit that referenced this pull request Oct 9, 2024
* fixed incorrect class property call

* fixed exporter uploading

* uploadCheckpoint uploads on every checkpoint epoch

* fix temp files names

* updated callback readme

* pre-commit run
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix Fixing a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants