Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for 24GB VRAM fine tuning via 8bit optimizers #162

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

youngmae
Copy link

Caveats: Open to feedback on configuration - wasn't entirely clear how to separate it, but made it generic enough to pull in any optimizer from bnb.

Summary of changes

  • Introduces dependency on bitsandbytes
  • Introduces an optimizer flag to configure bnb usage
  • Documentation

Tests done
5k test wav dataset, stock model config, and stock pretrained model:
| Name | Type | Params

0 | diffusion | ConditionedDiffusionModelWrapper | 1.2 B
1 | diffusion_ema | EMA | 1.1 B
2 | losses | MultiLoss | 0

1.1 B Trainable params
1.2 B Non-trainable params
2.3 B Total params
9,080.665 Total estimated model params size (MB)

With VRAM usage observed:
Device 0 [NVIDIA GeForce RTX 3090] PCIe GEN 4@16x RX: 47.85 MiB/s TX: 7.812 MiB/s
GPU 1635MHz MEM 9501MHz TEMP 77°C FAN 97% POW 315 / 350 W
GPU[|||||||||||||||||||||||||||||||100%] MEM[||||||||||||||||||22.367Gi/24.000Gi]

youngmae and others added 3 commits December 13, 2024 15:14
@rsxdalv
Copy link

rsxdalv commented Dec 23, 2024

Great work! Question - is it possible to make it an optional dependency? To divide the training and inference.

Edit: this PR seems to address just that https://github.com/Stability-AI/stable-audio-tools/pull/139/files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants