New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[QUESTION] SGD support in distrib_optimizer.py #1287

Open

zstreeter opened this issue Nov 13, 2024 · 0 comments

zstreeter commented Nov 13, 2024

I see the following assert in the distributed optimizer:
https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/optimizer/distrib_optimizer.py#L470-L472

I would like to use SGD for better performance. Why is this assert here? What is needed to support SGD in general?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment