How Ruckos LB moves a packet: XDP, direct return, IP-in-IP and BGP
One stable address for a service, spread across its replicas on any host, for any TCP or UDP protocol. Here is the path a packet takes, header by header.
Ruckos LB is the load balancer built into Ruckos Linux. A service gets one address from a range you set once for the cluster. Its replicas can run on any host, and a failed replica or a lost host drops out without the address moving. This post follows one connection through it.
1. The decision is made at the network card
Each host loads a small eBPF program at the XDP hook of its network card. XDP runs on a packet as soon as the driver has it, so the choice of replica is made before the packet goes any further into the host. The node API loads the program and fills its lookup tables; the program only ever reads those tables.
The program does two lookups. First a connection cache of 65,536 entries, keyed on the connection's source and destination addresses, ports and protocol. If the connection is already there, the packet goes to the replica it has been using. An open connection stays on its replica for as long as that replica lives.
2. A new connection is hashed onto a ring
A connection that is not in the cache is hashed into a ring of 4,093 slots for that address, filled with the service's healthy replicas using Maglev consistent hashing. The ring is built over container names, so every host builds the same ring and makes the same choice for the same connection. It does not matter which host a packet arrives at.
When a replica fails or a new one starts, most of the ring stays as it was. Together with the connection cache, that means a change disturbs only the connections that were on the replica that went away.
3. Delivery, on this host or another
If the chosen replica is on the same host, the program rewrites the packet's destination MAC to the container's and passes it on; the host's bridge delivers it. If the replica is on another host, the program puts an outer IPv4 header in front of the packet, addressed to that host, and sends it back out of the card. The receiving host removes the outer header and delivers the original packet to its own replica.
In both cases the original packet is untouched: it still goes from the client to the service address. Each container behind an address holds that address on its loopback interface and is set never to answer ARP for it, so it accepts the packet as its own.
The outer header costs 20 bytes. For TCP, the host caps the segment size a replica offers in its handshake (1,440 bytes on a 1,500-byte network), so wrapped packets always fit. UDP has no handshake to adjust, so a UDP service needs jumbo frames between hosts or datagrams small enough to fit.
4. The reply goes straight back
The replica answers with the service address as its source, out through the host's bridge to the gateway. The reply never passes back through the load balancer, and nothing is translated on either path, so the application sees the real client address.
This is direct server return. The load balancer does work only on the request, which is usually the smaller half of a connection.
5. How the address reaches your network
There are two ways, chosen per cluster to fit the network you have.
BGP. Each host runs a BGP speaker inside the node API and announces a service address only while it has a healthy replica of it. Your switches install equal-cost routes and spread flows across the hosts. BFD on each session detects a lost host in well under a second. When a host is drained, it withdraws its routes first, while IP-in-IP keeps serving the connections that still arrive. Hosts can share one AS number or have one each, and can peer by address or over unnumbered interfaces.
Layer 2. On a network without BGP, each address has one owner host, held by a lease. The owner answers ARP for the address and sends gratuitous ARP when it takes over. Everything for that address enters through the owner, which still spreads it across every replica, locally or over IP-in-IP. If the owner is lost, another host takes the lease within seconds. To spread arrivals across hosts in this mode, use more addresses: different hosts own different ones.
For BGP, the network needs to peer with the hosts and use equal-cost routes, allow BFD, accept replies whose source is a service address (loose reverse-path filtering), and pass requests and replies through the same firewall.
6. Health checks keep the ring honest
Each host checks the replicas it runs. A TCP connect every second marks a replica down after three failures in a row and back up after two passes. An optional HTTP check on a path you name runs every five seconds. A host acts on its own results at once and tells the others only when a replica changes state. The cluster's store is never in the packet path.
7. Three ways to balance
Spread hashes each connection on its own, so one client's connections land on different replicas. Sticky hashes on the client address, so one client always reaches the same replica. Primary puts a single replica in every slot of the ring, so all traffic goes to it, and a healthy standby takes over if it fails.
Try it
rctl weir range 192.168.1.11 10.100.0.1 10.100.0.254
rctl weir vip create 192.168.1.11 --service web --port 80 --port 443 --tier frontend
The test opens connections from your machine and reports which replica took each one. Replies come straight from the replica, so the client cannot tell from the reply alone; the test asks each host's connection cache instead. Here is a real run against a DNS service with three replicas on three hosts, set to spread:
$ rctl weir test 192.168.1.11 dns --requests 20
Successful: Spread, the connections went to 3 containers.
20 connections to 10.100.0.1:53 from this PC (Spread)
1 dns-2 on host-3
2 dns-2 on host-3
3 dns-2 on host-3
4 dns-3 on host-2
5 dns-2 on host-3
6 dns-3 on host-2
7 dns-2 on host-3
8 dns-1 on host-1
9 dns-3 on host-2
10 dns-3 on host-2
11 dns-1 on host-1
12 dns-1 on host-1
13 dns-2 on host-3
14 dns-3 on host-2
15 dns-1 on host-1
16 dns-1 on host-1
17 dns-1 on host-1
18 dns-3 on host-2
19 dns-2 on host-3
20 dns-2 on host-3
8 x dns-2 on host-3
6 x dns-3 on host-2
6 x dns-1 on host-1
Twenty connections from one machine landed on all three replicas, eight, six and six. The same test is a button in Ruckos-V Manager, under the cluster's Load Balancer settings. More on the Ruckos LB page.