What is the difference between 50256-consistent-oneword and 50256-consistent?
#22
Replies: 1 comment
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hey guys,
Specifically, I would like to ask which flags are set to train these 2 tokenizers? I can't find any flags that force tokenizer to have only 1 word tokens.
Thank you.
Beta Was this translation helpful? Give feedback.
All reactions