Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

investigate high resources consumption #41

Closed
poolpOrg opened this issue Sep 13, 2024 · 2 comments
Closed

investigate high resources consumption #41

poolpOrg opened this issue Sep 13, 2024 · 2 comments
Assignees
Milestone

Comments

@poolpOrg
Copy link
Collaborator

It seems that recent changes have improved RAM consumption on initial push...

... but secondary push and some commands, like check, exhaust RAM resources.

This is counter-intuitie since first push is supposedly the most costly operation and check is supposedly very cheap, so something needs to be investigated here.

@poolpOrg poolpOrg self-assigned this Sep 13, 2024
@poolpOrg
Copy link
Collaborator Author

identified as side effects from ram -> ldb switch and a missing tmp cleanup: leads mfs tmp directories to retain objects in memory.

as an additional improvement I made sure we don't keep too many vfs and indexes in memory simultaneously, the refactor is committed in branch ldb-cleanup pending some testing before merging to main.

@poolpOrg poolpOrg added this to the v1.0.0-rc1 milestone Nov 18, 2024
@poolpOrg
Copy link
Collaborator Author

Lots of rework was done in the VFS layer, the only remaining part using memory is states which is the topic of issue #90

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant