Circuit Breakers

Enterprise Edition

Circuit breakers prevent cascading failures by short-circuiting requests when upstream services are overwhelmed. When connection or request thresholds are exceeded, Envoy immediately returns errors instead of queuing more requests.

Why Circuit Breakers?

Envoy’s default circuit breaker limits (1024 connections, 1024 pending requests) may be too low for high-traffic environments. Without proper tuning:

  • Legitimate requests get rejected during traffic spikes
  • Slow upstream services cause request queues to build up
  • A single failing upstream can exhaust connection pools

Circuit breakers allow you to set appropriate limits based on your traffic patterns and upstream capacity.

Configuration Levels

Circuit breakers can be configured at two levels:

  1. Global (Config CRD): Applies to all tenants as the default
  2. Tenant (Tenant CRD): Overrides global config for specific tenants

Tenant-level configuration takes precedence over global configuration.

Configuration Fields

FieldTypeDefaultDescription
maxConnectionsint641024Maximum connections to all upstream endpoints
maxPendingRequestsint641024Maximum requests queued waiting for a connection
maxParallelRequestsint641024Maximum parallel requests (HTTP/2, gRPC multiplexing)
maxParallelRetriesint643Maximum parallel retry attempts
maxRequestsPerConnectionint64-Maximum requests per connection before closing
perEndpoint.maxConnectionsint64-Maximum connections per individual endpoint

Global Configuration

Configure circuit breakers globally in the Config CRD under spec.circuitBreaker:

apiVersion: kubelb.k8c.io/v1alpha1
kind: Config
metadata:
  name: default
  namespace: kubelb
spec:
  circuitBreaker:
    maxConnections: 10000
    maxPendingRequests: 5000
    maxParallelRequests: 10000
    maxParallelRetries: 10
    maxRequestsPerConnection: 1000
    perEndpoint:
      maxConnections: 500

Tenant Configuration

Override global settings for specific tenants in the Tenant CRD:

apiVersion: kubelb.k8c.io/v1alpha1
kind: Tenant
metadata:
  name: high-traffic-tenant
  namespace: kubelb
spec:
  circuitBreaker:
    maxConnections: 50000
    maxPendingRequests: 25000
    maxParallelRequests: 50000
    maxParallelRetries: 20

Circuit Breaker Behavior

When thresholds are exceeded:

  1. Envoy adds x-envoy-overloaded header to responses
  2. New requests receive HTTP 503 (Service Unavailable)
  3. Existing in-flight requests continue to completion
  4. The circuit opens immediately, with no gradual degradation

Monitor for x-envoy-overloaded headers to detect when circuit breakers are triggering.

Sizing Examples

The same fields are tuned per scenario; only the values change:

  • High-traffic global defaults (Config): maxConnections: 100000, maxPendingRequests: 50000, maxParallelRequests: 100000, maxParallelRetries: 50, perEndpoint.maxConnections: 2000.
  • Resource-constrained tenant (Tenant): maxConnections: 500, maxPendingRequests: 250, maxParallelRequests: 500, maxParallelRetries: 3, maxRequestsPerConnection: 100.
  • gRPC/HTTP2 tenant (Tenant): keep maxConnections low (for example 1000) and maxParallelRequests high (for example 50000), because requests are multiplexed over few connections; maxPendingRequests: 10000, maxParallelRetries: 10.

Monitoring

Track circuit breaker metrics to tune your configuration:

  • envoy_cluster_upstream_cx_overflow: Connections rejected due to maxConnections
  • envoy_cluster_upstream_rq_pending_overflow: Requests rejected due to maxPendingRequests
  • envoy_cluster_upstream_rq_retry_overflow: Retries rejected due to maxParallelRetries

Further Reading