Kubernetes Control Plane Sizing Research

Resources per control plane node - recommended values, not minimums. Sources: kubernetes.io, etcd.io, AWS EKS best practices, Azure AKS best practices, OpenShift docs, K3s docs.

---

Overview: CP Node Sizing by Cluster Scale

Cluster size	CP nodes	CPU / node	RAM / node	etcd disk	etcd topology
3–5 nodes (dev / home lab)	1 (or 3 HA)	2 vCPU	4–8 GB	20 GB SSD, 50 seq IOPS	Stacked, co-located with CP
10–20 nodes (small prod)	3 HA	4 vCPU	8–16 GB	20–40 GB SSD, 50 seq IOPS	Stacked, SSD mandatory
50–100 nodes (medium prod)	3 HA	4–8 vCPU	16–32 GB	40–100 GB SSD, 500 seq IOPS	Stacked or External
250–500 nodes (large prod)	3–5 HA+	8 vCPU	32–64 GB	100 GB NVMe, 500+ seq IOPS	External, 5-node etcd
1000 nodes (enterprise)	5 HA+	8–16 vCPU	64 GB	200 GB NVMe, 500+ seq IOPS	External + Events separate
5000 nodes (K8s max)	5+ per zone	16–32 vCPU	64–128 GB	500 GB NVMe, 10GbE between etcd	External, Events + main separate

Stacked vs External etcd

• Stacked - etcd runs directly on the CP node. Simpler to operate, fewer machines. Fine for clusters up to ~100 nodes.
• External - etcd on dedicated machines, separate from CP. Better fault isolation, recommended for ≥ 100 nodes in production or whenever etcd needs independent scaling.

---

etcd Hardware Recommendations

etcd is the most sensitive component. Disk latency matters more than CPU or RAM.

Scenario	CPU	RAM	Disk	Sequential IOPS	Network
Dev / test	2 cores	4–8 GB	20 GB SSD	50 IOPS	1 GbE
Small prod (< 100 nodes)	2–4 cores	8 GB	40 GB SSD	50 IOPS	1 GbE
Medium prod (100–500 nodes)	4–8 cores	8–16 GB	100 GB NVMe	500 IOPS	1 GbE
Large prod (500–2000 nodes)	8–16 cores	16–32 GB	200 GB NVMe	500+ IOPS	10 GbE
Heavy load (thousands of clients)	16 cores	32–64 GB	500 GB NVMe	500+ IOPS	10 GbE

Key etcd rules

• fsync latency p99 < 10 ms - mandatory. If the disk cannot sustain this, heartbeat timeouts trigger leader elections, destabilizing the cluster.
• Sequential IOPS, not concurrent. Cloud providers advertise concurrent IOPS, which can be ~10x higher than sequential. Always benchmark with fio:

fio --rw=write --ioengine=sync --fdatasync=1 \
    --filename=/var/lib/etcd/test.file \
    --size=22m --bs=2300 --name=etcd-bench

• Prefer SSD/NVMe. Spinning disks (7200 RPM) can work at 50 seq IOPS minimum, but any variance spikes latency. NVMe is strongly preferred for production.
• Dedicated disk for etcd. Do not share the etcd volume with logs, container images, or other I/O-heavy workloads.
• Avoid network-attached storage (iSCSI, NFS, Ceph) for etcd data - latency is unpredictable.
• etcd quota. Default is 2 GB. For large clusters with many objects, increase to 8 GB max. Defragment periodically.
• etcd memory. 8 GB is typically sufficient. For heavy deployments with thousands of watchers and millions of keys: 16–64 GB.
• etcd cluster size. 3 nodes is standard for production (tolerates 1 failure). 5 nodes for higher fault tolerance (tolerates 2 failures). Do not exceed 7 nodes - write performance degrades as quorum requires more agreement.
• Events in a separate etcd instance. For clusters >= 500–1000 nodes, Event objects generate significant write amplification. A dedicated etcd instance for Events improves stability and performance of the main cluster datastore.

---

kube-apiserver Scaling

The apiserver is stateless and horizontally scalable - you can run multiple replicas behind a load balancer.

Cluster size	Instances	--max-requests-inflight	--max-mutating-requests-inflight	Notes
<= 20 nodes	1	400 (default)	200 (default)	Default settings work fine
20–100 nodes	2–3	400–800	200–400	Watch instead of List; monitor APF metrics
100–500 nodes	3	800–1500	400–750	Use Informers in controllers; ResourceVersion=0 for cache reads
500–1000 nodes	3–5	2000+	1000+	APF FlowSchema isolation; tune nginx/LB watch timeout
1000–5000 nodes	5+ per zone	2000+ per instance	1000+	Scale vertically first, then horizontally

API server load reduction patterns

• Use Watch instead of List. Append ?watch=true to API requests or use informers. LIST with no arguments forces a quorum read from etcd and is the most expensive operation.
• Paginate large lists. Add limit=500 to list requests. Use fieldSelector and namespace-scoped paths.
• Use Informers (client-go, controller-runtime) - they combine LIST + WATCH and maintain in-memory caches, eliminating redundant API calls.
• ResourceVersion=0 for non-critical reads. Serves from apiserver cache instead of etcd, significantly reducing etcd load.
• API Priority and Fairness (APF). Use custom FlowSchema + PriorityLevelConfiguration to isolate noisy clients (monitoring agents, operators) from critical control plane traffic.
• DaemonSet rollouts on large clusters. Use RollingUpdate strategy. A new DaemonSet on a 1000-node cluster can spike apiserver load significantly. Add nodes in batches of <= 50.

---

kube-controller-manager and kube-scheduler

Both are single-instance (leader election), so they scale only vertically.

Cluster size	controller-manager RAM	scheduler RAM	Notes
<= 100 nodes	~200 MB	~100 MB	Defaults fine
100–500 nodes	~300–500 MB	~200 MB	Monitor reconcile loop latency
500–1000 nodes	~500 MB – 1 GB	~300–500 MB	Watch for scheduling queue depth
1000+ nodes	1–2 GB	500 MB – 1 GB	Complex affinity rules increase scheduler CPU significantly

• kube-scheduler CPU spikes under complex anti-affinity rules. At 1000+ nodes, poorly written PodAntiAffinity rules with requiredDuringSchedulingIgnoredDuringExecution can cause seconds-long scheduling decisions.
• controller-manager can be split into individual controller processes via the --controllers flag, allowing independent scaling per controller type (feature gate ControllerManagerLeaderMigration).

---

Kubernetes Hard Limits (v1.35)

Limit	Value
Maximum nodes per cluster	5,000
Maximum pods per cluster	150,000
Maximum containers per cluster	300,000
Maximum pods per node (default)	110 (configurable up to 250)

---

Scaling Anti-patterns to Avoid

Anti-pattern	Impact	Fix
Using etcd for application data	Blows up DB size, slow compaction	etcd is for K8s state only
Unbounded LIST calls in controllers	High apiserver + etcd load, potential OOM	Use Informers and Watch
Shared disk between etcd and workloads	fsync contention -> leader elections	Dedicated etcd volume
Adding hundreds of nodes at once	Apiserver spike, cloud API rate limiting	Batch additions, 10–20% of current size
7+ etcd members	Write performance degradation	3 or 5 nodes maximum
iSCSI/NFS for etcd storage	Unpredictable latency, instability	Local SSD/NVMe only
Large Secrets/ConfigMaps in etcd	Inflated DB size, slow snapshots	Split or move to external secret store

---

Practical Sizing Formula (rough estimate)

etcd DB size ~ (pods x 2 KB) + (services x 1 KB) + (configmaps/secrets x avg size)

Example: 5,000 pods + 500 services + 1,000 configmaps @ 5 KB avg:

(5000 x 2) + (500 x 1) + (1000 x 5) = 10 MB + 0.5 MB + 5 MB ~ 16 MB active data

The actual etcd DB will be larger due to MVCC revision history. Run etcdctl defrag and etcdctl compact regularly. Monitor etcd_db_total_size_in_bytes and etcd_mvcc_db_total_size_in_use_in_bytes.

---

Recommended Monitoring Metrics

Metric	Alert threshold	Component
etcd_disk_wal_fsync_duration_seconds p99	> 10 ms	etcd
etcd_db_total_size_in_bytes	> 75% of quota	etcd
etcd_server_leader_changes_seen_total (rate)	> 3 / hour	etcd
apiserver_request_duration_seconds p99	> 1s (mutating), > 60s (watch)	apiserver
apiserver_flowcontrol_rejected_requests_total	> 0	apiserver
scheduler_pending_pods	growing trend	scheduler
workqueue_depth (controller-manager)	growing trend	controller-manager

---

References

• Kubernetes: Considerations for large clusters
• etcd: Hardware recommendations
• AWS EKS: Scaling the control plane
• Azure AKS: Performance and scaling best practices
• OpenShift: Recommended etcd practices
• K3s: Requirements and resource profiling