Rate Limits

Rate limits control how many API requests can be made per key within a time window. They help prevent abuse, manage costs, and ensure fair usage.

Understanding Rate Limits

What Gets Limited

How Limits Work

1. Each key has configured limits
2. Requests count against the limit
3. Limits reset at the end of each period
4. Exceeding limits returns 429 errors

Default Limits by Plan

Configuring Limits

Per-Key Limits

Set limits when creating or editing a key:

1. Go to Settings > API Keys
2. Click on a key to edit
3. Adjust rate limit settings
4. Save changes

Available Settings

Rate Limit Headers

API responses include rate limit information:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1234567890

Handling Rate Limits

429 Response

When you exceed limits:

{
  "error": {
    "message": "Rate limit exceeded. Please slow down.",
    "type": "rate_limit_error",
    "code": "rate_limit_exceeded"
  }
}

Implementing Backoff

import time
import random
 
def request_with_backoff(make_request, max_retries=5):
    for attempt in range(max_retries):
        try:
            return make_request()
        except RateLimitError as e:
            if attempt < max_retries - 1:
                # Exponential backoff with jitter
                wait = (2 ** attempt) + random.uniform(0, 1)
                time.sleep(wait)
            else:
                raise

Best Practices

1. Check headers: Monitor remaining quota
2. Implement backoff: Handle 429s gracefully
3. Queue requests: Smooth out traffic spikes
4. Use batching: Combine requests where possible

Monitoring Usage

In Studio

View rate limit usage:

1. Go to Analytics > Usage
2. Select your key
3. View requests over time
4. Check for rate limit events

Alerts

Set up alerts for:

• Approaching rate limits (80% threshold)
• Rate limit exceeded events
• Unusual traffic patterns

Optimizing for Limits

Reduce Request Volume

• Cache responses where appropriate
• Batch similar requests
• Use efficient prompts

Reduce Token Usage

• Use concise prompts
• Set appropriate max_tokens
• Choose smaller models for simple tasks

Spread Traffic

• Implement request queuing
• Use multiple keys for different services
• Schedule batch jobs during off-peak times

Upgrading Limits

If you consistently hit limits:

1. Review your usage patterns
2. Optimize where possible
3. Upgrade your plan for higher limits
4. Contact support for Enterprise options

Related

• Virtual Keys Overview - Key management
• Creating Keys - Create new keys
• Error Handling - Handle rate limit errors