Nothing to be done as far as OpenAI, hopefully soon they will relax it but no public timeline. On your end you can implement exponential back off and retries and add user caps/rate limits. Good to know what % of users are being rate limited and distribution of # of requests by user. - ThreadSky

danielransom.com • 664 days ago

Nothing to be done as far as OpenAI, hopefully soon they will relax it but no public timeline. On your end you can implement exponential back off and retries and add user caps/rate limits. Good to know what % of users are being rate limited and distribution of # of requests by user.

Comments

martinshkreli.bsky.social•664 days ago

Yeah we’re engineering it but if you think about it, not a whole lot of capacity… 2.4m tokens an hour. If you’re using 3k tokens in a query, you’re looking at serving 800 users an hour with just that one 3k request

danielransom.com•664 days ago

Rate limits on their other APIs are much more reasonable. Problem is GPT-4 is still brand new. For what you guys are doing there’s only one company with only one API good enough. They’ve got you by the short hairs. For love or money you can’t get more access. While you wait, train your own model.

bryan.oh-hi.org•664 days ago

I really don't understand why openAI doesn't just scale up to cash in on their lead. Their backend must be limited in some terrible way.

martinshkreli.bsky.social•664 days ago

I think you CAN get dedicated hardware. Running gpt-4 around the clock is around $1.3m bill: 40k/min=$2.40/min, $144/hr, 3500/day, $1.3m a year.

We’re close to capacity with one product and started hitting rate limits today.

danielransom.com•664 days ago

They’ve already scaled harder and faster than anyone has before. They hit 1B users in March. They’ve probably got the best team money can buy and paid use is profitable but they’re still hitting limits. Scaling takes time even if everything is going your way. Plus these models aren’t cheap to run.

lukeingalls.bsky.social•664 days ago

Why not do queueing instead and just run jobs until you hit a rate limit? That ought to maximize usage whereas exponential backoff might not

martinshkreli.bsky.social•664 days ago

I’m worried we have demand for 24-7 uninterrupted GPT-4 for the whole year. Eg capacity constrained.

danielransom.com•664 days ago

Depends on whether traffic is peaky. May be a good alternative but you would need to expire jobs in the queue - after a short period people will probably leave or move on to another question. Doesn’t work well for chat. If you did conversations over email or so this could work really well.

Comments

Posting Rules

Reply