Skip to content

Relay fleet

How nexus records relays, hands them to gateways, drains them and removes them. Code: internal/relay (decisions), internal/relayconnect (the relay.v1 handlers), internal/store/pgstore/relay*.go (storage). Tables: Database → Relays. Architecture: Relay architecture.

Operator RPCs

Both on PortalGatewayService, platform admin only. The portal page is /admin/relays.

ListRelays

Returns every relay row as a RelaySummary, including relays that have stopped heartbeating.

Field Notes
id Minted at enrolment, rl-<8 hex>
hostname, region From the relay's enrolment info
address_v4, address_v6 Reported public addresses
version Daemon version
healthy Self-reported; nothing clears it, so read it with fresh
fresh Last heartbeat within 90 s (relay.StaleAfter)
draining Excluded from new assignments
last_seen_at Last heartbeat
forward_sessions / max_forward Current and ceiling (default ceiling 512)
rendezvous_sessions / max_rendezvous Current and ceiling
attached_gateways Gateways assigned now
tunnel_url Rung-3 endpoint (wss://tunnel.<id>.<zone>/t); empty when the relay offers no rung 3

SetRelayDraining

SetRelayDraining({ relay_id: "rl-…", draining: true | false })

Draining stops new assignments and moves attached gateways as each one asks again; nothing is cut mid-flight, so emptying a relay takes a few minutes.

The flag is sticky against heartbeats. The heartbeat update is

UPDATE relays SET draining = draining OR $1, …

so a relay can volunteer to drain but a heartbeat can never clear an operator's drain. SetRelayDraining writes the column directly and is the only way to undrain. The heartbeat response also carries should_drain, the control plane's request in the other direction.

Heartbeat

RelayService.Heartbeat updates forward_sessions, rendezvous_sessions, draining (OR'd), tunnel_url (written every beat, including empty, so a relay that lost its certificate stops being offered rung 3), last_seen_at and updated_at.

The response carries the whole SNI route table and the whole WireGuard peer set for that relay. The relay replaces its routes and reconciles its peers against the set, removing anything absent; a revoked gateway or withdrawn service disappears on the next beat. There is no delta protocol, so a missed beat cannot leave the relay inconsistent.

Relay heartbeats also drive failover for other relays: on each beat, nexus looks for relays that went stale within the last 15 minutes (relay.LostWindow) and sends relay_reassess to the gateways still attached to them, a bounded number per beat and at most every two minutes per gateway. There is no timer for this; nexus runs scale-to-zero.

Assignment

RequestRelay (from the gateway) calls relay.Assign:

  1. Prefer the gateway's current relay while it is still available — so a gateway does not hop between equally free relays. A current relay without a tunnel is given up once another available relay offers one.
  2. Otherwise Choose from the available relays, ordered by: same region first, then relays offering rung 3, then most free capacity, then id (deterministic). The gateway gets a list, best first, and walks it itself.
  3. If the gateway names its current relay as unreachable and another relay can take it, it is moved. Only that gateway moves; the relay is not marked.

A relay is available for a role when it is healthy, fresh (90 s), not draining, and has free capacity for that role.

relay_sessions.attached_at changes only when the gateway moves to a different relay; a repeat assignment to the same relay keeps it.

When nexus answers RequestRelay it writes the relay_sessions row and the gateway's DNS record (pointing at the chosen relay's address). The Cloudflare client compares the existing record and skips the write when the content is already correct. RelayService.ReportAttach / ReportDetach would do the same from the relay side, but relayd does not call them; a detach is honoured only if the gateway is still recorded on the relay sending it.

Reaping

Part of the hourly housekeeping sweep (relay.Reap). A relay is abandoned when:

  • it has heartbeated at least once, and its last heartbeat is older than 7 days (AbandonedAfter), and
  • no gateway is attached to it.

For each abandoned relay, nexus first withdraws its tunnel.<relay-id>.<zone> DNS name, then deletes the row. If the DNS withdrawal fails, the row is kept and retried on the next sweep. Host records written by terraform are not touched.

GetRelayLink (device read) returns the control plane's half of a gateway's relay path: role, tunnel address, attached_at, relay freshness, drain state and capacity. Relay id, host, region, addresses, tunnel_url and the forward capacity figures are filled for platform admins only; a customer sees the gateway's own names, never the relay behind them. The device's half (interface, handshake, which rung carries the link) comes from its state report; hasHandshake and lastHandshakeSeconds are read from there.