yoctol 乂卍oO煞氣ㄟtokenizerOo卍乂
Tokenizers have the same interface of Jieba:
from tokenizer_hub import XXX_tokenizer
tokenizer = XXX_tokenizer()
tokenizer.lcut('我来到北京清华大学')
> ['我', '来到', '北京', '清华大学']-
Notifications
You must be signed in to change notification settings - Fork 0
yoctol 乂卍oO煞氣ㄟtokenizerOo卍乂
License
Yoctol/tokenizer-hub
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
yoctol 乂卍oO煞氣ㄟtokenizerOo卍乂
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published