Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADJUST io.py for logging mlflow; Credits: Blirona #1

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
c0a51a2
ADJUST io.py for logging mlflow; Credits: Blirona
FeU-aKlos Sep 5, 2023
fe3737d
MADE ajustments as commented by PR
FeU-aKlos Sep 7, 2023
f884c88
ADD rclone utils file; ADJUST trainer, so that intermediate results a…
FeU-aKlos Sep 7, 2023
40bb9ca
ADJUST trainer file
FeU-aKlos Sep 7, 2023
5a85257
ADJUST trainer/io.py to have the imports at the beginning of the file
FeU-aKlos Sep 8, 2023
9a6e0e1
ADJUST files
FeU-aKlos Sep 8, 2023
8de4be8
ADJUST imports again circular import issue
FeU-aKlos Sep 8, 2023
098bc7e
ADJUST output_path name
FeU-aKlos Sep 8, 2023
bce3f88
DELETE intermediate saving of the results to lakefs <- otherwise, sto…
FeU-aKlos Sep 10, 2023
51df431
ADJUST rclone methods and trainer.py, so that intermediate results ar…
FeU-aKlos Sep 11, 2023
ee37597
FIX minor bug
FeU-aKlos Sep 11, 2023
bf41074
ADJUST somethiong :D
FeU-aKlos Sep 11, 2023
36d5b18
ADJUST somethiong :D
FeU-aKlos Sep 11, 2023
6958fda
ADJUST stuff
FeU-aKlos Sep 11, 2023
84999c2
ADJUST stuff
FeU-aKlos Sep 11, 2023
4904235
run_eval does not have any influence on the training anymore. eval is…
FeU-aKlos Sep 12, 2023
e1acc82
delete mlflow logging stuff. this will be done at the end of the trai…
FeU-aKlos Sep 12, 2023
e23a011
ADD clone dir from minio
FeU-aKlos Sep 14, 2023
8a1a751
SIMPLIFIED copy data from s3bucket function in trainer rclone.py
FeU-aKlos Sep 14, 2023
f1725a8
ADJUST copy method
FeU-aKlos Sep 14, 2023
6e69c68
ADJUST to only copy from lakefs ... sync2copy
FeU-aKlos Sep 14, 2023
a4118fb
FIX potential bug
FeU-aKlos Sep 20, 2023
a3d12c8
FIX bug loss value, if not float
FeU-aKlos Sep 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion trainer/io.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from pathlib import Path
from typing import Any, Callable, Dict, List, Tuple, Union
from urllib.parse import urlparse

import mlflow
import fsspec
import torch
from coqpit import Coqpit
Expand Down Expand Up @@ -161,6 +161,15 @@ def save_checkpoint(
save_func=save_func,
**kwargs,
)
from TTS.tts.models.modelWrapper import MyModel

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Add the imports at the start of the file

Copy link
Collaborator Author

@FeU-aKlos FeU-aKlos Sep 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if they are at the beginning of the file, then there might be a circular import error.

from TTS.utils.synthesizer import Synthesizer
synthesizer = Synthesizer(checkpoint_path, os.path.join(output_folder, 'config.json'))
model = MyModel(synthesizer=synthesizer)
# For MLflow logging, , artifacts={'checkpoint': file_name, "output": output_folder}
FeU-aKlos marked this conversation as resolved.
Show resolved Hide resolved

run_name = os.getenv("RUN_NAME", "vits_janika_de")

mlflow.pyfunc.log_model(python_model=model, artifact_path="models/TTS",registered_model_name=run_name, code_path=[output_folder])
FeU-aKlos marked this conversation as resolved.
Show resolved Hide resolved
if save_n_checkpoints is not None:
keep_n_checkpoints(output_folder, save_n_checkpoints)

Expand Down