Skip to content
This repository has been archived by the owner on Jul 11, 2023. It is now read-only.

Rate limit #8

Open
metallicusdev opened this issue Feb 14, 2023 · 8 comments
Open

Rate limit #8

metallicusdev opened this issue Feb 14, 2023 · 8 comments

Comments

@metallicusdev
Copy link

Running into rate limit

Rate limit reached for default-global-with-image-limits in organization XXX on requests per min. Limit: 60.000000 / min. Current: 70.000000 / min. Contact [email protected] if you continue to have issues.
@cfortuner
Copy link
Owner

What are you using in promptable? Can you share your usage?

@metallicusdev
Copy link
Author

metallicusdev commented Feb 14, 2023

import { OpenAI, FileLoader, CharacterTextSplitter, Embeddings } from "promptable";

// Create a model provider!
const provider = new OpenAI(process.env.API_KEY);

// Create documents
// Load documents
const loader = new FileLoader('../output.md');
let docs = await loader.load();

// Split documents into chunks
const splitter = new CharacterTextSplitter("\n");
docs = splitter.splitDocuments(docs, {
  chunk: true,
  chunkSize: 1000, // tokens :)
});

// Create embeddings
const embeddings = new Embeddings(
    "proton-ts-embeddings",
    provider,
    docs,
    { cacheDir: "." }
);
await embeddings.index();

It should be able to handle API ratelimits and retry

@cfortuner
Copy link
Owner

What are your thoughts on using something like this??

#12

@ymansurozer
Copy link

I've just discovered this repo and it is very promising. It would sooo good to have an NPM alternative of GPT Index. Thanks for the great work!

My humble view: #12 looks great. But in embedMany, is there a reason why you're not sending the documents to the embeddings endpoint as an array input (string[]) and instead sending them in separate requests as single input (string)?

@cfortuner
Copy link
Owner

Whoops! Yeah i'll update that @ymansurozer

Thanks for pointing that out.

Also, I'd love your input on the api you think we should make for Indexing. Want to DM me in Discord? https://discord.gg/PGVadrjm

@ymansurozer
Copy link

Happy to help! Now we need to somehow handle embedding requests that exceed 250,000 tokens in total to overcome rate limits. :) I'll DM on Discord now. @cfortuner

@hanrelan
Copy link
Collaborator

@cfortuner if openai returns a retry-after header in their 429 status code we should use that for the rate limiting

@cfortuner
Copy link
Owner

Hey just a quick update.

we're still working on the right implementation here, but coming soon!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants