Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for tensor parallelism and add OLMo2-26B model config / train script #117

Merged
merged 43 commits into from
Dec 3, 2024
Merged

Add support for tensor parallelism and add OLMo2-26B model config / train script #117

merged 43 commits into from
Dec 3, 2024

Commits on Nov 27, 2024

  1. Add OLMo2-26B config

    @epwalsh
    epwalsh committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    0ef2a8a View commit details
    Browse the repository at this point in the history

  2. make model wider, shorter

    @epwalsh
    epwalsh committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    d4c65ac View commit details
    Browse the repository at this point in the history

  3. finalize config

    @epwalsh
    epwalsh committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    3d23a2d View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into epwalsh/olmo2-13B

    @epwalsh
    epwalsh committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    d4ac791 View commit details
    Browse the repository at this point in the history

  5. update

    @epwalsh
    epwalsh committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    32c520c View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2024

  1. Add max and abs to selective ops AC 

    See pytorch/torchtitan#701
    @epwalsh
    epwalsh committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    20eca35 View commit details
    Browse the repository at this point in the history

  2. Add methods to apply TP to various modules

    @epwalsh
    epwalsh committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    07ae12f View commit details
    Browse the repository at this point in the history

  3. ensure we don't f-up applying float8 to output again

    @epwalsh
    epwalsh committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    ef5e695 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2024

  1. only print config from local rank 0

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    eb710a1 View commit details
    Browse the repository at this point in the history

  2. fix

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    4ebd618 View commit details
    Browse the repository at this point in the history

  3. fix?

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    365f179 View commit details
    Browse the repository at this point in the history

  4. fix?

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    2aeed48 View commit details
    Browse the repository at this point in the history

  5. fix

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    9f31745 View commit details
    Browse the repository at this point in the history

  6. fix

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    d37d45d View commit details
    Browse the repository at this point in the history

  7. more fixes

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    9a6dc33 View commit details
    Browse the repository at this point in the history

  8. add test for tensor parallel model

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    ed6a02e View commit details
    Browse the repository at this point in the history

  9. fixes

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    17359c0 View commit details
    Browse the repository at this point in the history

  10. expand test

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    01634ff View commit details
    Browse the repository at this point in the history

  11. fix GQA attention

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    527c443 View commit details
    Browse the repository at this point in the history

  12. updates

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    7e8cbf2 View commit details
    Browse the repository at this point in the history

  13. update train example to allow tensor parallel

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    8ecc741 View commit details
    Browse the repository at this point in the history

  14. updates

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    b89236e View commit details
    Browse the repository at this point in the history

  15. fix log

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    9fec486 View commit details
    Browse the repository at this point in the history

  16. more fixes

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    8da6f8e View commit details
    Browse the repository at this point in the history

  17. disable async tp by default

    @epwalsh
    epwalsh committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    b411c2b View commit details
    Browse the repository at this point in the history

Commits on Dec 3, 2024

  1. log

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    c01a374 View commit details
    Browse the repository at this point in the history

  2. fix

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    b6a0981 View commit details
    Browse the repository at this point in the history

  3. update size

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    6348fb5 View commit details
    Browse the repository at this point in the history

  4. try next nightly

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    de20020 View commit details
    Browse the repository at this point in the history

  5. comment out

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    2863fe9 View commit details
    Browse the repository at this point in the history

  6. try flash, no compile

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    5858777 View commit details
    Browse the repository at this point in the history

  7. fix

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    739aa89 View commit details
    Browse the repository at this point in the history

  8. try compile again

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    3dfdedf View commit details
    Browse the repository at this point in the history

  9. adjust hidden size, NCCL log level

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    355f61a View commit details
    Browse the repository at this point in the history

  10. ok try newer nightly again

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    6e581db View commit details
    Browse the repository at this point in the history

  11. ensure mock batch is the same for each DP rank

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    4e1a2c3 View commit details
    Browse the repository at this point in the history

  12. try something else

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    81adb75 View commit details
    Browse the repository at this point in the history

  13. clean up

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    95c41ff View commit details
    Browse the repository at this point in the history

  14. fix speed monitor for TP

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    a0005cd View commit details
    Browse the repository at this point in the history

  15. clean up

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    bed869f View commit details
    Browse the repository at this point in the history

  16. fix lint

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    c1f7d44 View commit details
    Browse the repository at this point in the history

  17. changelog

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    d552a3a View commit details
    Browse the repository at this point in the history

  18. try compile again

    @epwalsh
    epwalsh committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    8ba7f58 View commit details
    Browse the repository at this point in the history