New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

new Prodigy optimizer version: signigicantly less VRAM #3022

Open

dxqbYD opened this issue Dec 29, 2024 · 0 comments

dxqbYD commented Dec 29, 2024 •

edited

Loading

Hey,

just a heads-up, there is a new version of the Prodigy optimizer that requires significantly less VRAM.

new hyperparameter slice_p==11:
saves 45% of VRAM overhead. There should be no downside. VRAM requirement on the level of AdamW
set beta1 to 0:
saves another 25% - this can have a quality impact, similar to Adafactor compared to AdamW

Details:
konstmish/prodigy#22
konstmish/prodigy#23
konstmish/prodigy#32

OneTrainer PR:
Nerogar/OneTrainer#550

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment