Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data class holds all samples in memory #2

Open
kevinhughes27 opened this issue Dec 28, 2016 · 0 comments
Open

Data class holds all samples in memory #2

kevinhughes27 opened this issue Dec 28, 2016 · 0 comments

Comments

@kevinhughes27
Copy link
Owner

This won't scale much longer since if I add any more training data it won't all fit in memory. In this project they have a list of all the training images and only load what is required for each training batch.

My current way might be a bit quicker to train since the IO is front loaded but eventually something like this might be required.
@zhangchunsheng zhangchunsheng mentioned this issue Aug 3, 2017
Closed
@mgagvani mgagvani mentioned this issue Jun 3, 2021
Merged
kevinhughes27 added a commit that referenced this issue Jun 4, 2021
@mgagvani @kevinhughes27
Changed prepare samples to use Numpy array for X data (#65)
6ab288d 
* Changed prepare samples to use np array for X

The previous method, with a list, took too much memory. With the old method, using 20 GB worth of 1080p images, memory usage would exceed my computer's 16 GB. With the updated code it does not exceed 1 GB. Kind of solved this issue: #2

* Update utils.py

Co-authored-by: Kevin Hughes <[email protected]>

* Update with suggestions

Co-authored-by: Kevin Hughes <[email protected]>
kevinhughes27 added a commit that referenced this issue Nov 6, 2021
@mgagvani @kevinhughes27
Put back load_imgs function (#67)
798848e 
* Changed prepare samples to use np array for X

The previous method, with a list, took too much memory. With the old method, using 20 GB worth of 1080p images, memory usage would exceed my computer's 16 GB. With the updated code it does not exceed 1 GB. Kind of solved this issue: #2

* Update utils.py

Co-authored-by: Kevin Hughes <[email protected]>

* Update with suggestions

* Put back load_imgs

* Fix with upstream

Co-authored-by: Kevin Hughes <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kevinhughes27