Code

Faster tokenizer#137

Code

Open

LoganDark wants to merge 7 commits intoBlinkDL:mainBlinkDL/ChatRWKV:mainfrom LoganDark:fast-tokenizerLoganDark/ChatRWKV:fast-tokenizerCopy head branch name to clipboard

+80-5

Commits

Commits on Jun 4, 2023

• Faster tokenizer

  Show description for 6359bfe
  LoganDark

  committed
  6359bfe

• Back to dictionary

  Show description for 8ccb10a
  LoganDark

  committed
  8ccb10a

Commits on Jun 5, 2023

• Hint Generator type

  LoganDark

  committed
  c01f3fb

• Remove now unused tok2len

  LoganDark

  committed
  a8e9a26

• Make tokenizer compile with mypyc

  LoganDark

  committed
  44b52ba

• Typecheck more with mypyc

  LoganDark

  committed
  3a89a5b

Commits on Jul 22, 2023

• Merge branch 'main' into fast-tokenizer

  LoganDark

  authored
  2b15367