Cluster Tuning and Limits
Cluster tuning baseline
The published numbers are not reproducible on stock component limits. The table below lists
what was tuned and why.
| Component | Stock | Tuned | Why |
|---|
| ovn-central (deployment) | cpu 4 / mem 8 Gi | cpu 4 / mem 16 Gi | headroom for NB and SB database growth at high object counts. The limit must never exceed control-plane node RAM: an oversized limit lets ovn-central grow past physical memory, and the kernel then OOM-kills the control-plane node, apiserver and etcd included. 16 Gi is safe on 24 GiB control-plane nodes; scale the limit to your own nodes |
| ovn-central probes | timeout 45 s | timeout 300 s, failureThreshold 10 | database replay after a pod restart exceeds 45 s at high object counts, so the longer timeout prevents kubelet from killing pods mid-replay |
| kube-ovn-controller | cpu 2 / mem 4 Gi | cpu 8 / mem 8 Gi | workqueue drain at high object counts (runs on 251 GiB workers) |
| ovs-ovn (DaemonSet, every node) | cpu 2 / mem 1 Gi | cpu 4 / mem 8 Gi | each node’s ovn-controller holds the whole cluster’s logical-flow set; the 1 Gi stock limit OOM-killed it at about 7.7 k subnets, which was the real per-node datapath wall before the bump |
| kube-ovn-cni (DaemonSet) | mem 1 Gi | mem 2 Gi | headroom for pod-wiring churn at scale |
| kube-multus-ds (DaemonSet) | mem 256 Mi | mem 1 Gi | at the stock limit multus OOM-crash-looped (92 to 514 restarts over 102 days), which blocked all pod wiring on affected nodes and surfaced as a false 50-service ceiling |
First binding factor per capability
| Resource | First binding factor |
|---|
| Network tenants (VPCs) | etcd database size (77 % of its danger line at 10 k; everything else below 50 %) |
| Subnets | network-controller programming throughput; before tuning, per-node ovs-ovn memory (1 Gi OOM at about 7.7 k) |
| Firewall policies | none observed up to 355 k ACLs cluster-wide (cap-bound) |
| Security groups | kube-ovn-controller stability (a concurrent-map data race at the 20 k-SG load, an upstream bug) |
| Services | pod-wiring throughput (every functional VIP needs a Ready backend pod) |
| Static routes | controller programming cadence (drift between requested and programmed) |
| Combined tenants (degradation) | per-VM datapath state (port binding, conntrack, and NAT); empty objects are nearly free |
Interpretation caveats
- Cap-bound versus wall-bound. Cap-bound means the run reached its configured cap while
healthy, so the value is a lower bound. Wall-bound means something real stopped it (named in the
tables). Only security groups and static routes are wall-bound; every other number published here
is cap-bound or window-bound.
- Catch-up-window-bound numbers understate the ceiling. The subnetsPerCluster figure of 11,822
is where programming settled within a 3-hour window, not a wall. A longer window or a faster
controller raises it.
- Security groups carry an upstream-bug caveat. The ceiling is a kube-ovn 1.14.30 stability
bound, not an OVN capacity bound.
- QoS policies and network-attachment templates have no standalone row by design. Both are
binding-triggered: a bare CR programs nothing until a pod or EIP references it, so a bare-object
ceiling would measure only API storage, which is meaningless under the programmed-and-functional
definition.
- Static routes used the earlier methodology (no functional verify, no catch-up settle).
- One cluster shape. All numbers come from the reference cluster listed in the
section overview; the tuning baseline above is part of the
result.