OS3’s models
All ids together. A minute’s tokens count each request’s whole prompt, cached part included, plus its output.- Requests in flight are counted at each of OS3’s gateway servers on its own, so an organisation whose requests reach several can have more in all.
- It is a cap, not reserved capacity: one organisation holds at most half of the normal lane on a GPU server, the same for every organisation.
- One agent that sends a full 1M-token prompt every 12 seconds uses about 5.3M tokens a minute.
Your own provider’s models
When OS3 sends your calls on to your own provider (OpenAI, Anthropic, Gemini and others), each key has its plan’s limit, apart from the ones above:
OS3 can set other limits on a key, or one limit over all your organisation’s keys, on request.
Past a limit
- Over requests or tokens a minute:
429 rate_limit_exceeded, withretry-after-msandretry-afterheaders. The OpenAI SDKs wait and retry by themselves. - Over requests in flight:
429 tenant_busy, withRetry-After(1 to 5 seconds). - A stream that has started is never cut. Nothing refused is billed.