Code

feat: Init tokenizer from filehandle#76

Code

Merged

asafgardin merged 7 commits intomainAI21Labs/ai21-tokenizer:mainfrom init-tokenizer-from-filehandleAI21Labs/ai21-tokenizer:init-tokenizer-from-filehandleCopy head branch name to clipboardJan 2, 2024

+100-4

Commits

Commits on Jan 2, 2024

• feat: allow creating JurassicTokenizer from model file handle

  tomeras91

  committed
  59c9c5e

• fix: Add default for model_path and model_file_handle

  tomeras91

  committed
  e934f11

• feat: Add JurassicTokenizer.from_file_path classmethod

  tomeras91

  committed
  adee09e

• fix: remove model_path=None in JurassicTokenizer.from_file_handle

  tomeras91

  committed
  5bb5938

• fix: rename _assert_exactly_one to _validate_init and make it not static

  tomeras91

  committed
  5384fd4

• refactor: semantics

  asafgardin

  committed
  e565a62

• test: Added tests

  asafgardin

  committed
  57a25f7