Crash on React Native: SyntaxError: Invalid RegExp: Invalid escape

[julian-hecker]

I'm working on integrating LangChain and OpenAI in Expo React Native. I've run into quite a number of issues, but it's getting there.

Now, I'm facing the following issue:

When a response from OpenAI has streamed and completed, LangChain uses TikToken to try to count the total number of tokens used.

When the response is getting encoded in TikToken, it tries to create the following RegExp on this line: /('s|'S|'t|'T|'re|'rE|'Re|'RE|'ve|'vE|'Ve|'VE|'m|'M|'ll|'lL|'Ll|'LL|'d|'D)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+/gu. Then plug that into http://regexr.com/, and it'll tell you the \p feature might not be supported in all browsers. Pretty sure this isn't supported in the React Native Hermes runtime.

Let me know what we can do about this, maybe I can help :)

Edit: I have also raised an issue in the langchain-js repo.

[asp3]

having the same issue, any workarounds? @julian-hecker

[julian-hecker]

@asp3 My workaround until the issue was fixed in Langchain was to change the code inside node_modules using patch-package. I believe my solution of wrapping the encode method in a try/catch throws out the token count, however.