Embedding Error: Requested 9064 tokens instead of 8192 tokens

[githubdebugger]

After uploading a document, lobe-chat did the chunking and when it was embedding the document it failed with this error:

embeddingChunks error {
message: {"endpoint":"https://api.openai.com/v1","error":{"message":"This model's maximum context length is 8192 tokens, however you requested 9064 tokens (9064 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.","type":"invalid_request_error","param":null,"code":null},"errorType":"ProviderBizError","provider":"openai"},
name: 'EmbeddingError'
}

I have not modified any token limits for the models, I do not even know how to make any changes to embedding models.

Any idea how do I fix this error, so I would be able to upload the document?

[arvinxx]

what's your document? It seems didn't chunk correctly

[githubdebugger]

what's your document? It seems didn't chunk correctly

It is a book in PDF format.

[githubdebugger]

what's your document? It seems didn't chunk correctly

I tried this file https://github.com/PlamenStilyianov/FinMathematics/blob/master/Paul%20Wilmott%20on%20Quantitative%20Finance.pdf (just a random file, I am not the uploader of this file) and I am able to reproduce it every time.

I tried other smaller PDF's and they are working fine.

I am using the latest lobe-chat version, in a docker container.

[githubdebugger]

@arvinxx not sure if you got a chance to look at the reply here, I tried this out with the latest 1.29.6 and I am able to reproduce this error with the above document every time.

embeddingChunks error {
message: {"endpoint":"https://api.openai.com/v1","error":{"message":"This model's maximum context length is 8192 tokens, however you requested 9064 tokens (9064 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.","type":"invalid_request_error","param":null,"code":null},"errorType":"ProviderBizError","provider":"openai"},
name: 'EmbeddingError'
}

[arvinxx]

I think it maybe a bug. I will try to address it

[githubdebugger]

Thanks! You can use the document in the thread to easily reproduce the bug, in case you want to see the issue.

Appreciate your help.