Serverless rate limits
Serverless Inference runs models on shared, multi-tenant deployments. To keep capacity fair and latency consistent, Crusoe enforces rate limits that cap how many tokens and requests a project can send to a Serverless Inference endpoint each minute. Limits apply per project and per model, and only to traffic handled by Serverless Inference endpoints.
How rate limits work
Two limits are tracked each minute:
- Tokens Per Minute (TPM): The total input and output tokens your project can process for a given model in a one-minute window.
- Requests Per Minute (RPM): The total requests your project can send to a given model in a one-minute window.
Each limit is enforced independently. When your project exceeds either one, the endpoint returns a 429 Too Many Requests response.
Default limits
Your default limits depend on whether your organization has a payment method on file:
| Account status | TPM per model | RPM per model |
|---|---|---|
| No payment method | 500,000 | 30 |
| Payment method added | 2,000,000 | 600 |
To automatically raise both limits to 2,000,000 TPM and 600 RPM, add a payment method. If your product needs higher limits than that before launching, contact us to discuss your expected volume.
A 2,000,000 TPM limit doesn't guarantee a successful response. Serverless Inference deployments are shared, so you can still receive a 503 Service Unavailable while a model scales to meet aggregate traffic, even when you're within your limits.
Responses when a limit is reached
Serverless Inference returns different status codes depending on whether your project exceeded its limit or the shared endpoint is experiencing unusually high load:
| HTTP status | Description |
|---|---|
429 Too Many Requests | Your project exceeded its TPM or RPM limit for the model. |
503 Service Unavailable | Your request is within your limits, but the shared endpoint is experiencing unusually high load and can't accept it right now. |
Both 429 and 503 responses are transient, so you can retry the request with exponential backoff rather than failing immediately. If you're consistently hitting your rate limit or seeing repeated 503s, contact us so we can help raise your limits or advise on capacity planning.
Need higher rate limits or reserved capacity?
- Contact us for a rate limit increase. This is recommended if you expect your initial launch traffic to exceed the default limits.
- If you need predictable response rates or reserved inference capacity, use Self-Serve Deployments. They run on dedicated hardware with reserved capacity, which gives you predictable performance and often better economics at scale.
Retry with exponential backoff
Your rate limit resets every minute, and shared-endpoint load fluctuates continuously, so retrying immediately only adds to the contention. Back off exponentially between attempts instead:
- Wait a short initial delay (for example, 1 second) before your first retry.
- Double the delay after each subsequent failed attempt (1s, 2s, 4s, 8s, and so on).
- Add a small amount of random jitter to each delay so that multiple clients don't retry simultaneously.
- Cap the delay (for example, at 30–60 seconds) and the number of retries, then surface the error if the request still hasn't succeeded.
Because TPM and RPM limits reset every minute, a request that fails with 429 will typically succeed within the next window. 503 responses usually clear even sooner, as soon as the shared endpoint scales to meet demand.
Rate limit headers
Every Serverless Inference response includes headers that report your current limit and remaining allowance, so you can throttle client-side without waiting for a 429:
| Header | Example | Description |
|---|---|---|
x-ratelimit-limit-tokens | 2000000 | Maximum tokens permitted before the TPM limit is exhausted. |
x-ratelimit-remaining-tokens | 1700000 | Remaining tokens permitted before the TPM limit is exhausted. |
x-ratelimit-limit-requests | 600 | Maximum requests permitted before the RPM limit is exhausted. |
x-ratelimit-remaining-requests | 59 | Remaining requests permitted before the RPM limit is exhausted. |
Considerations
Rate limits are designed to protect the shared endpoint from overload, not to limit consumption. Because of this, there might be instances where actual throughput exceeds your configured TPM limits.