Rate Limiting

Rate limiting is a technique for controlling the frequency of requests a client can make to an API or service within a specified time window. It protects services from abuse, ensures fair resource allocation among users, and prevents downstream services (like LLM APIs) from being overwhelmed. Common algorithms include fixed window, sliding window, token bucket, and leaky bucket. In AI applications, rate limiting is applied at multiple layers: API gateway (requests per minute), LLM proxy (tokens per minute), and application level (agent runs per user per day).