Skip to content
This repository has been archived by the owner on Oct 9, 2024. It is now read-only.

[BUG] Error training large dataset #12

Open
jaaaamIron opened this issue Jun 13, 2022 · 0 comments
Open

[BUG] Error training large dataset #12

jaaaamIron opened this issue Jun 13, 2022 · 0 comments
Labels
bug Something isn't working model training Issues related to training/using custom models

Comments

@jaaaamIron
Copy link

On ColabPro+ I am getting an error when training on a large (3 hour+) dataset. Fairly sure it isn't a RAM/memory issue, but could be wrong. Here is what I get in the console:

Creating dataset...
This usually takes around 2-3 minutes for each minute of audio
(10 minutes of training audio -> 20-30 minutes)
/bin/bash: line 1: 6256 Killed ddsp_prepare_tfrecord --input_audio_filepatterns=audio/* --output_tfrecord_path=data//train.tfrecord --num_shards=10 --sample_rate=16000 --frame_rate=50 --example_secs=4.0 --hop_secs=1.0 --viterbi=True --center=True &> /dev/null

Training...
I0613 13:24:12.203999 139645850212224 ddsp_run.py:179] Restore Dir: /content/gdrive/MyDrive/1/ddsp-training-2022-06-13-1303
I0613 13:24:12.204447 139645850212224 ddsp_run.py:180] Save Dir: /content/gdrive/MyDrive/1/ddsp-training-2022-06-13-1303
I0613 13:24:12.244054 139645850212224 resource_reader.py:50] system_path_file_exists:optimization/base.gin
E0613 13:24:12.244385 139645850212224 resource_reader.py:55] Path not found: optimization/base.gin
I0613 13:24:12.250916 139645850212224 resource_reader.py:50] system_path_file_exists:eval/basic.gin
E0613 13:24:12.251169 139645850212224 resource_reader.py:55] Path not found: eval/basic.gin
I0613 13:24:12.350420 139645850212224 ddsp_run.py:152] Operative config not found in /content/gdrive/MyDrive/1/ddsp-training-2022-06-13-1303
I0613 13:24:12.350906 139645850212224 resource_reader.py:50] system_path_file_exists:models/vst/vst.gin
E0613 13:24:12.351280 139645850212224 resource_reader.py:55] Path not found: models/vst/vst.gin
I0613 13:24:12.357154 139645850212224 resource_reader.py:50] system_path_file_exists:datasets/tfrecord.gin
E0613 13:24:12.357398 139645850212224 resource_reader.py:55] Path not found: datasets/tfrecord.gin
I0613 13:24:12.358336 139645850212224 resource_reader.py:50] system_path_file_exists:datasets/base.gin
E0613 13:24:12.358542 139645850212224 resource_reader.py:55] Path not found: datasets/base.gin
I0613 13:24:12.367111 139645850212224 ddsp_run.py:184] Operative Gin Config:
import ddsp
import ddsp.training as ddsp2

@jaaaamIron jaaaamIron added the bug Something isn't working label Jun 13, 2022
@wilzh40 wilzh40 added the model training Issues related to training/using custom models label Nov 1, 2022
@wilzh40 wilzh40 changed the title [BUG]Error training large dataset [BUG] Error training large dataset Nov 1, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working model training Issues related to training/using custom models
Projects
None yet
Development

No branches or pull requests

2 participants