Why the clusters ran out of IPs
On EKS, the Amazon VPC CNI gives every pod a real address from the VPC. By default that address comes from the same subnet as the node, so pods draw on the same pool as the nodes, load balancers and everything else in that subnet. When the subnet runs short, the CNI can’t hand out an address and new pods can’t start, even on nodes with CPU and memory to spare. Adding nodes doesn’t help, because new nodes draw from the same subnets.
The usual fix of a bigger VPC range is often not available. Private address space (RFC 1918) is shared with offices, VPNs and other VPCs, and a larger block has to be carved out without colliding with any of them.
Moving pods into the CGNAT range
Custom networking separates where nodes live from where pods get their addresses:
- A second address range for the VPC. The VPC gets a secondary CIDR from the shared address space set aside for carrier-grade NAT (RFC 6598); here,
100.126.0.0/16. AWS recommends this space because corporate networks rarely use it, so it is unlikely to collide. - Pod subnets in every zone. Subnets are carved from that range in each Availability Zone the cluster uses.
- One ENIConfig per zone. An ENIConfig resource points the VPC CNI at the pod subnet (and security groups) for a zone. Naming each one after its zone lets nodes pick up the right one automatically.
- Custom networking switched on in the CNI. The CNI then gives pods addresses only from extra network interfaces in the ENIConfig subnet. The node keeps its own address on the routable subnet.
- Traffic out uses the node’s address. By default, traffic to anything outside the VPC leaves through the node’s own interface, so the rest of the network sees routable node addresses, never the CGNAT range.
The result: pods stop competing with everything else for routable addresses, and the routable subnets only have to hold the nodes.
AWS documents the setup in Deploy Pods in alternate subnets with custom networking and its custom networking best practices.
What it costs
- Fewer pods per node. The node’s main network interface no longer carries pod addresses, so each instance type fits fewer pods. Prefix delegation, which hands out blocks of addresses instead of single ones, can win that back.
- Nodes have to be replaced. The change only applies to new nodes. Existing nodes have to be cordoned and drained so their pods reschedule onto nodes that use the new setup, which makes the change disruptive.
- More to operate. There are now two address ranges, a set of ENIConfig resources and a CNI setting that all have to stay in step.
AWS recommends IPv6 clusters instead where IPv6 is an option, and a private NAT gateway where the real problem is overlapping address ranges.
Planning the cutover
Because existing nodes have to be drained and replaced, the move is planned around downtime. The migration is coordinated with stakeholders to keep downtime inside scheduled maintenance periods and to keep services communicating with the clusters throughout.
Making every change reviewable
The change touches two layers: the VPC side (the secondary range and its subnets) and the cluster side (the ENIConfig resources and the CNI setting). All changes were committed with Terraform via Spacelift, and manifests deployed through ArgoCD.