You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Would it be possible to expose the function getEncodingNameForModel that's available in the javascript version also in the wasm version of the library?
The wasm version currently exposes encoding_for_model and get_encoding and both create a tokenizer instance immediately. For our uses, we'd like to first translate the model name to the underlying encoding and then instantiate the tokenizer with get_encoding ourselves. This would allow us to cache and reuse a single tokenizer across multiple models that use the same encoding.
The text was updated successfully, but these errors were encountered:
Would it be possible to expose the function
getEncodingNameForModel
that's available in thejavascript
version also in the wasm version of the library?The wasm version currently exposes
encoding_for_model
andget_encoding
and both create a tokenizer instance immediately. For our uses, we'd like to first translate the model name to the underlying encoding and then instantiate the tokenizer withget_encoding
ourselves. This would allow us to cache and reuse a single tokenizer across multiple models that use the same encoding.The text was updated successfully, but these errors were encountered: