Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warc Backoff #160

Open
wants to merge 184 commits into
base: main
Choose a base branch
from
Open

Warc Backoff #160

wants to merge 184 commits into from

Commits on May 7, 2024

  1. better listing

    @soldni
    soldni committed May 7, 2024
    Configuration menu
    Copy the full SHA
    3231c83 View commit details
    Browse the repository at this point in the history

  2. better resolvers

    @soldni
    soldni committed May 7, 2024
    Configuration menu
    Copy the full SHA
    7da8e1f View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. better list

    @soldni
    soldni committed May 8, 2024
    Configuration menu
    Copy the full SHA
    6859981 View commit details
    Browse the repository at this point in the history

  2. parsing from stdin

    @soldni
    soldni committed May 8, 2024
    Configuration menu
    Copy the full SHA
    6e8f31d View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. fixed behavior with old omegaconf

    @soldni
    soldni committed May 10, 2024
    Configuration menu
    Copy the full SHA
    6395416 View commit details
    Browse the repository at this point in the history

  2. math parsers

    @soldni
    soldni committed May 10, 2024
    Configuration menu
    Copy the full SHA
    f6db05e View commit details
    Browse the repository at this point in the history

  3. math parsers

    @soldni
    soldni committed May 10, 2024
    Configuration menu
    Copy the full SHA
    8f282cf View commit details
    Browse the repository at this point in the history

  4. science

    @soldni
    soldni committed May 10, 2024
    Configuration menu
    Copy the full SHA
    acf4ce5 View commit details
    Browse the repository at this point in the history

  5. pipeline

    @soldni
    soldni committed May 10, 2024
    Configuration menu
    Copy the full SHA
    cbc448d View commit details
    Browse the repository at this point in the history

  6. errors

    @soldni
    soldni committed May 10, 2024
    Configuration menu
    Copy the full SHA
    db2491c View commit details
    Browse the repository at this point in the history

  7. added backoff

    @soldni
    soldni committed May 10, 2024
    Configuration menu
    Copy the full SHA
    149fea8 View commit details
    Browse the repository at this point in the history

  8. added backoff

    @soldni
    soldni committed May 10, 2024
    Configuration menu
    Copy the full SHA
    3bde00f View commit details
    Browse the repository at this point in the history

  9. logging

    @soldni
    soldni committed May 10, 2024
    Configuration menu
    Copy the full SHA
    3903c7b View commit details
    Browse the repository at this point in the history

  10. unused

    @soldni
    soldni committed May 10, 2024
    Configuration menu
    Copy the full SHA
    875b8cb View commit details
    Browse the repository at this point in the history

Commits on May 11, 2024

  1. url fixes

    @soldni
    soldni committed May 11, 2024
    Configuration menu
    Copy the full SHA
    302e0a5 View commit details
    Browse the repository at this point in the history

  2. speedup

    @soldni
    soldni committed May 11, 2024
    Configuration menu
    Copy the full SHA
    289d973 View commit details
    Browse the repository at this point in the history

  3. speedup

    @soldni
    soldni committed May 11, 2024
    Configuration menu
    Copy the full SHA
    38ca143 View commit details
    Browse the repository at this point in the history

Commits on May 12, 2024

  1. data

    @soldni
    soldni committed May 12, 2024
    Configuration menu
    Copy the full SHA
    0f848d0 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. fix

    @soldni
    soldni committed May 13, 2024
    Configuration menu
    Copy the full SHA
    6351595 View commit details
    Browse the repository at this point in the history

  2. extensions

    @soldni
    soldni committed May 13, 2024
    Configuration menu
    Copy the full SHA
    802a4de View commit details
    Browse the repository at this point in the history

  3. debug

    @soldni
    soldni committed May 13, 2024
    Configuration menu
    Copy the full SHA
    967fcbd View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. backoff

    @soldni
    soldni committed May 14, 2024
    Configuration menu
    Copy the full SHA
    0e86ef9 View commit details
    Browse the repository at this point in the history

  2. missed a continue

    @soldni
    soldni committed May 14, 2024
    Configuration menu
    Copy the full SHA
    71159cc View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. more

    @soldni
    soldni committed May 15, 2024
    Configuration menu
    Copy the full SHA
    31cfc7d View commit details
    Browse the repository at this point in the history

  2. more

    @soldni
    soldni committed May 15, 2024
    Configuration menu
    Copy the full SHA
    57587e3 View commit details
    Browse the repository at this point in the history

  3. logging backoff

    @soldni
    soldni committed May 15, 2024
    Configuration menu
    Copy the full SHA
    673d3f5 View commit details
    Browse the repository at this point in the history

  4. removed unused deps

    @soldni
    soldni committed May 15, 2024
    Configuration menu
    Copy the full SHA
    478b387 View commit details
    Browse the repository at this point in the history

  5. match

    @soldni
    soldni committed May 15, 2024
    Configuration menu
    Copy the full SHA
    01f796f View commit details
    Browse the repository at this point in the history

  6. removed profiler

    @soldni
    soldni committed May 15, 2024
    Configuration menu
    Copy the full SHA
    256c3fb View commit details
    Browse the repository at this point in the history

  7. config

    @soldni
    soldni committed May 15, 2024
    Configuration menu
    Copy the full SHA
    ccc2aa2 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'soldni/backoff' of https://github.com/allenai/dolma int… 

    …o soldni/backoff
    @soldni
    soldni committed May 15, 2024
    Configuration menu
    Copy the full SHA
    78ad625 View commit details
    Browse the repository at this point in the history

  9. fixed api

    @soldni
    soldni committed May 15, 2024
    Configuration menu
    Copy the full SHA
    cae4084 View commit details
    Browse the repository at this point in the history

  10. fixed test

    @soldni
    soldni committed May 15, 2024
    Configuration menu
    Copy the full SHA
    a6fffe7 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2024

  1. batching

    @soldni
    soldni committed May 17, 2024
    Configuration menu
    Copy the full SHA
    4988f62 View commit details
    Browse the repository at this point in the history

  2. added support for batching

    @soldni
    soldni committed May 17, 2024
    Configuration menu
    Copy the full SHA
    7306fcd View commit details
    Browse the repository at this point in the history

  3. saving owm for later

    @soldni
    soldni committed May 17, 2024
    Configuration menu
    Copy the full SHA
    8e57c2f View commit details
    Browse the repository at this point in the history

  4. loosening reqs

    @soldni
    soldni committed May 17, 2024
    Configuration menu
    Copy the full SHA
    456f831 View commit details
    Browse the repository at this point in the history

  5. old python compatible syntax

    @soldni
    soldni committed May 17, 2024
    Configuration menu
    Copy the full SHA
    8ec9880 View commit details
    Browse the repository at this point in the history

  6. html spans

    @soldni
    soldni committed May 17, 2024
    Configuration menu
    Copy the full SHA
    33a725c View commit details
    Browse the repository at this point in the history

  7. fixed

    @soldni
    soldni committed May 17, 2024
    Configuration menu
    Copy the full SHA
    5f4ffc1 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'main' into soldni/backoff

    @soldni
    soldni authored May 17, 2024
    Configuration menu
    Copy the full SHA
    b5e2b0d View commit details
    Browse the repository at this point in the history

Commits on May 20, 2024

  1. wip

    @soldni
    soldni committed May 20, 2024
    Configuration menu
    Copy the full SHA
    28be895 View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. Merge branch 'main' into soldni/backoff

    @soldni
    soldni authored May 21, 2024
    Configuration menu
    Copy the full SHA
    d3efee4 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2024

  1. adding support for batching

    @soldni
    soldni committed May 22, 2024
    Configuration menu
    Copy the full SHA
    aee8ba6 View commit details
    Browse the repository at this point in the history

  2. better eval

    @soldni
    soldni committed May 22, 2024
    Configuration menu
    Copy the full SHA
    ff1e496 View commit details
    Browse the repository at this point in the history

  3. fixed minor failure

    @soldni
    soldni committed May 22, 2024
    Configuration menu
    Copy the full SHA
    ed28a7a View commit details
    Browse the repository at this point in the history

  4. merge

    @soldni
    soldni committed May 22, 2024
    Configuration menu
    Copy the full SHA
    0953e80 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'soldni/backoff' of https://github.com/allenai/dolma int… 

    …o soldni/backoff
    @soldni
    soldni committed May 22, 2024
    Configuration menu
    Copy the full SHA
    3c4f17d View commit details
    Browse the repository at this point in the history

  6. small fix in math for processors

    @soldni
    soldni committed May 22, 2024
    Configuration menu
    Copy the full SHA
    2cc4084 View commit details
    Browse the repository at this point in the history

  7. using native types when possible

    @soldni
    soldni committed May 22, 2024
    Configuration menu
    Copy the full SHA
    eda41c3 View commit details
    Browse the repository at this point in the history

  8. indent

    @soldni
    soldni committed May 22, 2024
    Configuration menu
    Copy the full SHA
    d93a54f View commit details
    Browse the repository at this point in the history

  9. copyright

    @soldni
    soldni committed May 22, 2024
    Configuration menu
    Copy the full SHA
    29dca70 View commit details
    Browse the repository at this point in the history

  10. better string

    @soldni
    soldni committed May 22, 2024
    Configuration menu
    Copy the full SHA
    dc2fa98 View commit details
    Browse the repository at this point in the history

  11. comment

    @soldni
    soldni committed May 22, 2024
    Configuration menu
    Copy the full SHA
    97b3bd2 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. progressbar

    @soldni
    soldni committed May 23, 2024
    Configuration menu
    Copy the full SHA
    845072e View commit details
    Browse the repository at this point in the history

  2. added support for old-style retries_on_error

    @soldni
    soldni committed May 23, 2024
    Configuration menu
    Copy the full SHA
    35719fc View commit details
    Browse the repository at this point in the history

  3. added support for retries_on_error

    @soldni
    soldni committed May 23, 2024
    Configuration menu
    Copy the full SHA
    67b3bda View commit details
    Browse the repository at this point in the history

  4. data

    @soldni
    soldni committed May 23, 2024
    Configuration menu
    Copy the full SHA
    155319c View commit details
    Browse the repository at this point in the history

  5. deps

    @soldni
    soldni committed May 23, 2024
    Configuration menu
    Copy the full SHA
    d8cb681 View commit details
    Browse the repository at this point in the history

  6. get_annotations not available

    @soldni
    soldni committed May 23, 2024
    Configuration menu
    Copy the full SHA
    e6270dc View commit details
    Browse the repository at this point in the history

  7. fixes

    @soldni
    soldni committed May 23, 2024
    Configuration menu
    Copy the full SHA
    75a5b0d View commit details
    Browse the repository at this point in the history

  8. quoting type aliases

    @soldni
    soldni committed May 23, 2024
    Configuration menu
    Copy the full SHA
    86371d6 View commit details
    Browse the repository at this point in the history

  9. 3.8 compatibility

    @soldni
    soldni committed May 23, 2024
    Configuration menu
    Copy the full SHA
    73aad08 View commit details
    Browse the repository at this point in the history

  10. more style

    @soldni
    soldni committed May 23, 2024
    Configuration menu
    Copy the full SHA
    b9ec3eb View commit details
    Browse the repository at this point in the history

  11. pyi

    @soldni
    soldni committed May 23, 2024
    Configuration menu
    Copy the full SHA
    e42f9fc View commit details
    Browse the repository at this point in the history

  12. Merge branch 'soldni/pbar2' into soldni/backoff

    @soldni
    soldni committed May 23, 2024
    Configuration menu
    Copy the full SHA
    d9cbac0 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. progress

    @soldni
    soldni committed May 24, 2024
    Configuration menu
    Copy the full SHA
    09aa96a View commit details
    Browse the repository at this point in the history

  2. viz pbar

    @soldni
    soldni committed May 24, 2024
    Configuration menu
    Copy the full SHA
    be6c984 View commit details
    Browse the repository at this point in the history

  3. fixes

    @soldni
    soldni committed May 24, 2024
    Configuration menu
    Copy the full SHA
    5c90e9f View commit details
    Browse the repository at this point in the history

  4. fixing small regression in tests

    @soldni
    soldni committed May 24, 2024
    Configuration menu
    Copy the full SHA
    f5c696c View commit details
    Browse the repository at this point in the history

  5. order from user

    @soldni
    soldni committed May 24, 2024
    Configuration menu
    Copy the full SHA
    e941f05 View commit details
    Browse the repository at this point in the history

  6. same order

    @soldni
    soldni committed May 24, 2024
    Configuration menu
    Copy the full SHA
    99264b0 View commit details
    Browse the repository at this point in the history

  7. tests

    @soldni
    soldni committed May 24, 2024
    Configuration menu
    Copy the full SHA
    8f86b62 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'soldni/pbar2' into soldni/backoff

    @soldni
    soldni committed May 24, 2024
    Configuration menu
    Copy the full SHA
    88a9e55 View commit details
    Browse the repository at this point in the history

  9. older

    @soldni
    soldni committed May 24, 2024
    Configuration menu
    Copy the full SHA
    18a3f71 View commit details
    Browse the repository at this point in the history

  10. note for common runtime

    @soldni
    soldni committed May 24, 2024
    Configuration menu
    Copy the full SHA
    f0e8af4 View commit details
    Browse the repository at this point in the history

  11. removing attempts

    @soldni
    soldni committed May 24, 2024
    Configuration menu
    Copy the full SHA
    36c18d2 View commit details
    Browse the repository at this point in the history

Commits on May 25, 2024

  1. progressbar

    @soldni
    soldni committed May 25, 2024
    Configuration menu
    Copy the full SHA
    708affc View commit details
    Browse the repository at this point in the history

  2. progressbar

    @soldni
    soldni committed May 25, 2024
    Configuration menu
    Copy the full SHA
    824e11e View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. adding linearizers

    @soldni
    soldni committed May 28, 2024
    Configuration menu
    Copy the full SHA
    bb054ca View commit details
    Browse the repository at this point in the history

  2. Merge branch 'soldni/backoff' of https://github.com/allenai/dolma int… 

    …o soldni/backoff
    @soldni
    soldni committed May 28, 2024
    Configuration menu
    Copy the full SHA
    c2315e7 View commit details
    Browse the repository at this point in the history

  3. license script

    @soldni
    soldni committed May 28, 2024
    Configuration menu
    Copy the full SHA
    3cec13a View commit details
    Browse the repository at this point in the history

  4. script

    @soldni
    soldni committed May 28, 2024
    Configuration menu
    Copy the full SHA
    81c473a View commit details
    Browse the repository at this point in the history

  5. science

    @soldni
    soldni committed May 28, 2024
    Configuration menu
    Copy the full SHA
    42e1514 View commit details
    Browse the repository at this point in the history

  6. added better tests

    @soldni
    soldni committed May 28, 2024
    Configuration menu
    Copy the full SHA
    4336a46 View commit details
    Browse the repository at this point in the history

  7. added tests

    @soldni
    soldni committed May 28, 2024
    Configuration menu
    Copy the full SHA
    e01b408 View commit details
    Browse the repository at this point in the history

  8. types

    @soldni
    soldni committed May 28, 2024
    Configuration menu
    Copy the full SHA
    46135ab View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. sorting

    @soldni
    soldni committed May 29, 2024
    Configuration menu
    Copy the full SHA
    a8803e8 View commit details
    Browse the repository at this point in the history

  2. name

    @soldni
    soldni committed May 29, 2024
    Configuration menu
    Copy the full SHA
    002611f View commit details
    Browse the repository at this point in the history

  3. send

    @soldni
    soldni committed May 29, 2024
    Configuration menu
    Copy the full SHA
    b090d25 View commit details
    Browse the repository at this point in the history

  4. typo

    @soldni
    soldni committed May 29, 2024
    Configuration menu
    Copy the full SHA
    486ef25 View commit details
    Browse the repository at this point in the history

  5. spacing

    @soldni
    soldni committed May 29, 2024
    Configuration menu
    Copy the full SHA
    275eb95 View commit details
    Browse the repository at this point in the history

  6. skipping big tests

    @soldni
    soldni committed May 29, 2024
    Configuration menu
    Copy the full SHA
    26b791b View commit details
    Browse the repository at this point in the history

  7. optional tests w large download

    @soldni
    soldni committed May 29, 2024
    Configuration menu
    Copy the full SHA
    c9f5888 View commit details
    Browse the repository at this point in the history

  8. corner case failure

    @soldni
    soldni committed May 29, 2024
    Configuration menu
    Copy the full SHA
    628dd14 View commit details
    Browse the repository at this point in the history

  9. quantized

    @soldni
    soldni committed May 29, 2024
    Configuration menu
    Copy the full SHA
    bc61e05 View commit details
    Browse the repository at this point in the history

  10. s3 destination

    @soldni
    soldni committed May 29, 2024
    Configuration menu
    Copy the full SHA
    36b4275 View commit details
    Browse the repository at this point in the history

  11. commit

    @soldni
    soldni committed May 29, 2024
    Configuration menu
    Copy the full SHA
    d7d629f View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. style

    @soldni
    soldni committed May 30, 2024
    Configuration menu
    Copy the full SHA
    47b00dc View commit details
    Browse the repository at this point in the history

  2. owm

    @soldni
    soldni committed May 30, 2024
    Configuration menu
    Copy the full SHA
    af2f820 View commit details
    Browse the repository at this point in the history

  3. science v2

    @soldni
    soldni committed May 30, 2024
    Configuration menu
    Copy the full SHA
    19c13cf View commit details
    Browse the repository at this point in the history

  4. science v2

    @soldni
    soldni committed May 30, 2024
    Configuration menu
    Copy the full SHA
    2678191 View commit details
    Browse the repository at this point in the history

  5. science v1

    @soldni
    soldni committed May 30, 2024
    Configuration menu
    Copy the full SHA
    851e767 View commit details
    Browse the repository at this point in the history

  6. science v1

    @soldni
    soldni committed May 30, 2024
    Configuration menu
    Copy the full SHA
    5401ae5 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2024

  1. edit

    @soldni
    soldni committed May 31, 2024
    Configuration menu
    Copy the full SHA
    7dc4ad9 View commit details
    Browse the repository at this point in the history

  2. config

    @soldni
    soldni committed May 31, 2024
    Configuration menu
    Copy the full SHA
    5763cc3 View commit details
    Browse the repository at this point in the history

  3. fixes total

    @soldni
    soldni committed May 31, 2024
    Configuration menu
    Copy the full SHA
    493a18f View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2024

  1. minor fix

    @soldni
    soldni committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    c436c0d View commit details
    Browse the repository at this point in the history

  2. linersizer

    @soldni
    soldni committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    537b7a7 View commit details
    Browse the repository at this point in the history

  3. fallback

    @soldni
    soldni committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    7221a0b View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2024

  1. added compression

    @soldni
    soldni committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    5d09315 View commit details
    Browse the repository at this point in the history

  2. adding test data

    @soldni
    soldni committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    937ccc0 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. wip

    @soldni
    soldni committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    e93cc48 View commit details
    Browse the repository at this point in the history

  2. tests

    @soldni
    soldni committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    b7c5c59 View commit details
    Browse the repository at this point in the history

  3. tests

    @soldni
    soldni committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    f6bbf23 View commit details
    Browse the repository at this point in the history

  4. fixed tests

    @soldni
    soldni committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    33da8dc View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. fixes

    @soldni
    soldni committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    fd041c1 View commit details
    Browse the repository at this point in the history

  2. added flags in config

    @soldni
    soldni committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    68d6b35 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'soldni/zst' into soldni/backoff

    @soldni
    soldni committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    84370ca View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. better error handling

    @soldni
    soldni committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    928b85a View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into soldni/backoff

    @soldni
    soldni committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    f5767c2 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'soldni/mixer-fix' into soldni/backoff

    @soldni
    soldni committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    592b7a8 View commit details
    Browse the repository at this point in the history

  4. added files (to be removed)

    @soldni
    soldni committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    a3ddbd2 View commit details
    Browse the repository at this point in the history

  5. new stuff

    @soldni
    soldni committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    1c6a62a View commit details
    Browse the repository at this point in the history

  6. other names

    @soldni
    soldni committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    16b8dc9 View commit details
    Browse the repository at this point in the history

  7. update

    @soldni
    soldni committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    f182d0d View commit details
    Browse the repository at this point in the history

  8. configs

    @soldni
    soldni committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    3227a11 View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2024

  1. small fix gopher tagger

    @soldni
    soldni committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    75d2938 View commit details
    Browse the repository at this point in the history

  2. addding configs

    @soldni
    soldni committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    f8b771b View commit details
    Browse the repository at this point in the history

  3. wip

    @soldni
    soldni committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    cea2d1e View commit details
    Browse the repository at this point in the history

  4. optional

    @soldni
    soldni committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    426ef1a View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2024

  1. para

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    ed9cf90 View commit details
    Browse the repository at this point in the history

  2. new resolver

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    8a4dbfc View commit details
    Browse the repository at this point in the history

  3. random delay

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    5f5feeb View commit details
    Browse the repository at this point in the history

  4. delay

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    2b20cd4 View commit details
    Browse the repository at this point in the history

  5. jitter log

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    6e9fad2 View commit details
    Browse the repository at this point in the history

  6. feix

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    76c4f6b View commit details
    Browse the repository at this point in the history

  7. dedup

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    9a86e09 View commit details
    Browse the repository at this point in the history

  8. test

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    a1fb0e2 View commit details
    Browse the repository at this point in the history

  9. new steps

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    cd3f38d View commit details
    Browse the repository at this point in the history

  10. fixes

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    1ee4bed View commit details
    Browse the repository at this point in the history

  11. all

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    d818395 View commit details
    Browse the repository at this point in the history

  12. all

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    ffd7ccd View commit details
    Browse the repository at this point in the history

  13. indent

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    06ecd9c View commit details
    Browse the repository at this point in the history

  14. keyword

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    6102ef8 View commit details
    Browse the repository at this point in the history

  15. wip

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    e42b7c3 View commit details
    Browse the repository at this point in the history

  16. discarding fields

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    5107c34 View commit details
    Browse the repository at this point in the history

  17. w

    @soldni
    soldni committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    eaca238 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. sizes

    @soldni
    soldni committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    ca29511 View commit details
    Browse the repository at this point in the history

  2. lciense

    @soldni
    soldni committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    965c053 View commit details
    Browse the repository at this point in the history

  3. fixing paths

    @soldni
    soldni committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    4af5f21 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. scripts to get labels

    @soldni
    soldni committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    1c27e33 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. exp

    @soldni
    soldni committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    def2027 View commit details
    Browse the repository at this point in the history

  2. test, stats

    @soldni
    soldni committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    f1c877a View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. optional id

    @soldni
    soldni committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    3e07e51 View commit details
    Browse the repository at this point in the history

  2. reverted

    @soldni
    soldni committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    38a3122 View commit details
    Browse the repository at this point in the history

  3. ext

    @soldni
    soldni committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    b487104 View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2024

  1. missed configs

    @soldni
    soldni committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    7462aa8 View commit details
    Browse the repository at this point in the history

  2. added function to count top k tokens

    @soldni
    soldni committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    c8e4d7c View commit details
    Browse the repository at this point in the history

  3. missing

    @soldni
    soldni committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    ba66c91 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2024

  1. count

    @soldni
    soldni committed Jul 6, 2024
    Configuration menu
    Copy the full SHA
    d7558db View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2024

  1. added option for tokenizer to split on special tokens

    @soldni
    soldni committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    a6e74a7 View commit details
    Browse the repository at this point in the history

  2. added configs

    @soldni
    soldni committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    a576020 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'soldni/tiktoken' into soldni/backoff

    @soldni
    soldni committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    f31662a View commit details
    Browse the repository at this point in the history

  4. encoding special tokens

    @soldni
    soldni committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    c31fab3 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. more paths

    @soldni
    soldni committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    f405d47 View commit details
    Browse the repository at this point in the history

  2. configs

    @soldni
    soldni committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    3498632 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. Merge branch 'main' into soldni/backoff

    @soldni
    soldni authored Aug 8, 2024
    Configuration menu
    Copy the full SHA
    e560e99 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. cc-news-new

    @soldni
    soldni committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    58a84d4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into soldni/backoff

    @soldni
    soldni authored Aug 21, 2024
    Configuration menu
    Copy the full SHA
    ab586a1 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. version

    @soldni
    soldni committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    d7998ff View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2024

  1. adding new lengths

    @soldni
    soldni committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    5dd7611 View commit details
    Browse the repository at this point in the history

  2. script

    @soldni
    soldni committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    bd46c36 View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2024

  1. partitions

    @soldni
    soldni committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    04277c4 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. small

    @soldni
    soldni committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    1768ff0 View commit details
    Browse the repository at this point in the history

  2. 100 chars

    @soldni
    soldni committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    a50fcaa View commit details
    Browse the repository at this point in the history

  3. datasets

    @soldni
    soldni committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    de42c1a View commit details
    Browse the repository at this point in the history

  4. reformatted

    @soldni
    soldni committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    d34012e View commit details
    Browse the repository at this point in the history

  5. Merge branch 'soldni/backoff' of https://github.com/allenai/dolma int… 

    …o soldni/backoff
    @soldni
    soldni committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    15c3ca6 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. .

    @soldni
    soldni committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    af13c63 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. science

    @soldni
    soldni committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    15608e1 View commit details
    Browse the repository at this point in the history

  2. added option to provide compression during tagging

    @soldni
    soldni committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    7ea0862 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'soldni/compression2' into soldni/backoff

    @soldni
    soldni committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    4c3cab6 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2024

  1. configs

    @soldni
    soldni committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    da4957c View commit details
    Browse the repository at this point in the history