Code

Add segment ids/sequence masking#827

Code

Merged

dlwh merged 14 commits intomainstanford-crfm/levanter:mainfrom segment_idsstanford-crfm/levanter:segment_idsCopy head branch name to clipboardDec 3, 2024

+202-38

Commits

Commits on Dec 1, 2024

• add segment ids

  dlwh

  committed
  039f319

• better test, combine segment masks

  dlwh

  committed
  9131078

Commits on Dec 2, 2024

• add with_segment_ids

  dlwh

  committed
  039ecd7

• actually use segment_ids

  dlwh

  committed
  8bf34bd

• comments

  dlwh

  committed
  92247ac

• cleanup

  dlwh

  committed
  381ce3a

• sigh

  dlwh

  committed
  de04991

• tol

  dlwh

  committed
  bf14a2b

• sigh

  dlwh

  committed
  df0f0eb

• kk

  dlwh

  committed
  c29e071

• better vmap

  dlwh

  committed
  1bbb411

• this?

  dlwh

  committed
  4fff4c3

• this?

  dlwh

  committed
  5052f36

Commits on Dec 3, 2024

• enough device puts and we're good

  dlwh

  committed
  1a7729a