Skip to main content

OS3’s models

All ids together. A minute’s tokens count each request’s whole prompt, cached part included, plus its output.
  • Requests in flight are counted at each of OS3’s gateway servers on its own, so an organisation whose requests reach several can have more in all.
  • It is a cap, not reserved capacity: one organisation holds at most half of the normal lane on a GPU server, the same for every organisation.
  • One agent that sends a full 1M-token prompt every 12 seconds uses about 5.3M tokens a minute.

Your own provider’s models

When OS3 sends your calls on to your own provider (OpenAI, Anthropic, Gemini and others), each key has its plan’s limit, apart from the ones above: OS3 can set other limits on a key, or one limit over all your organisation’s keys, on request.

Past a limit

  • Over requests or tokens a minute: 429 rate_limit_exceeded, with retry-after-ms and retry-after headers. The OpenAI SDKs wait and retry by themselves.
  • Over requests in flight: 429 tenant_busy, with Retry-After (1 to 5 seconds).
  • A stream that has started is never cut. Nothing refused is billed.
See Errors.