Raise error when concurrent to call new Tiktoken

[qkxie]

The context code is https://github.com/mckaywrigley/chatbot-ui/blob/main/pages/api/chat.ts#L21

The key problem is here:

In my frontend, I use promise.all to send 3 requests simultaneously to this backend API. I have noticed the following phenomena:

1. When running this backend with "npm run dev", everything works fine.
2. When running with "npm run build" and then "npm run start":
3. If I make a separate request to this API after the backend is started, and then the frontend sends concurrent requests, everything works fine.
4. If the frontend immediately sends concurrent requests to this backend API after it is started, it will result in the error shown in the image below.

[qkxie]

I think the problem is that init need some time to finish its process. Only after init finished, then can I call new Tiktoken. But when I promise.all three calls to the api after the backend launch, the first call will trigger init. the second call find that the init process is running, so it just skip it and run new Tiktoken immediately. But this time, the init process is not finished yet. so Tiktoken raise error.

I think the second call should wait until the first call finished the init process install of skip at once.

[qkxie]

This is a demo code to trigger this error:

const tiktokenModel = require("tiktoken/encoders/cl100k_base.json");
const { Tiktoken, init } = require("tiktoken/lite/init");
const fs = require('fs')

const wasm = fs.readFileSync("./node_modules/tiktoken/lite/tiktoken_bg.wasm");


async function test() {
    await init((imports) => WebAssembly.instantiate(wasm, imports));
    encoding = new Tiktoken(
        tiktokenModel.bpe_ranks,
        tiktokenModel.special_tokens,
        tiktokenModel.pat_str
      )
    const tokens = encoding.encode("hello world");
    console.log(tokens)
    encoding.free();
}


async function main() {
    a = test()
    b = test()
    await Promise.all([a, b])
}

main()

the error message is:

/Users/qkxie/Project/test_tiktoken/node_modules/tiktoken/lite/tiktoken_bg.cjs:393
    throw new Error(getStringFromWasm0(arg0, arg1));
          ^

Error: `unwrap_throw` failed
    at module.exports.__wbindgen_throw (/Users/qkxie/Project/test_tiktoken/node_modules/tiktoken/lite/tiktoken_bg.cjs:393:11)
    at wasm://wasm/003425c2:wasm-function[863]:0x78e3c
    at wasm://wasm/003425c2:wasm-function[567]:0x6f758
    at wasm://wasm/003425c2:wasm-function[8]:0x3aaa
    at new Tiktoken (/Users/qkxie/Project/test_tiktoken/node_modules/tiktoken/lite/tiktoken_bg.cjs:179:26)
    at test (/Users/qkxie/Project/test_tiktoken/index.js:10:16)
    at async Promise.all (index 0)
    at async main (/Users/qkxie/Project/test_tiktoken/index.js:24:5)

Node.js v20.5.1