Relay Architecture¶
Relays are stateless forwarding nodes. They let a gateway with no public address keep a tunnel up and receive inbound connections. A relay reads the plaintext SNI of each TLS ClientHello and splices the connection, without decrypting it, to the assigned gateway through the gateway's WireGuard tunnel. The gateway terminates TLS with its own certificate, so a relay cannot read the traffic it carries.
One relay address serves every gateway and every published service, because routing is by SNI rather than by address or port.
The bring-up and attach sequences are drawn in How It Works.
relayd (edge/relay)¶
| Function | Detail |
|---|---|
| Enrolment | Spends the bootstrap token from cloud-init (/etc/digilan/relay.env) once, through RelayEnrollmentService.Enroll. The durable credential is written under /var/lib/digilan. |
| Heartbeat | RelayService.Heartbeat every 30 s (internal/relayd/heartbeat.go). It reports health, capacity and sessions, the version, and tunnel_url. The response carries the whole set of WireGuard peers, SNI routes and rung-3 tunnel tokens. |
| SNI forwarder | TCP 443. A known name is spliced to <gateway tunnel address>:443. An unknown name is closed without a TLS alert. |
| WireGuard | wg0, one peer per assigned gateway, reconciled from every heartbeat. A peer missing from the response is removed, which is how revocation reaches the relay. |
| Rung 3 | WebSocket tunnel (udpws) at wss://tunnel.<relay-id>.<suffix>/t. Only that name is terminated, and it is checked before the routing table. |
| Tunnel certificate | RelayService.RequestCertificate with a CSR that names nothing. Nexus answers with the chain and the names it signed, which is how the relay learns its own name. |
| Probe echo | Packets starting with PRBE on UDP 51820 or 443 are steered by nftables to port 51821 and echoed back. The gateway uses this to test a UDP path without touching its live tunnel. |
The unit runs relayd -wg with only CAP_NET_BIND_SERVICE and CAP_NET_ADMIN, and logs to the serial console as well as the journal. There is no SSH unless relay_debug_ssh_public_key is set.
Tunnel certificate retries¶
tunnel_url is reported only once a certificate has been obtained and is being served. After a failed request, relayd retries after 1 minute, doubling up to 30 minutes. When nexus answers that it offers no certificates (no NEXUS_RELAY_ZONE_SUFFIX), relayd checks again after 60 minutes. A relay booted before nexus had the zone configured therefore gets rung 3 within the hour, or immediately after systemctl restart relayd.
Nexus side¶
| RPC | Caller | Purpose |
|---|---|---|
GatewayInternalService.RegisterRelayToken |
terraform (scripts/relay-token.sh, X-Internal-Key) |
Accept a relay's first enrolment |
RelayEnrollmentService.Enroll |
relayd | Bootstrap token in exchange for relay_id + api_token |
RelayService.Heartbeat |
relayd, every 30 s | Health in, peers/routes/tokens out |
RelayService.RequestCertificate |
relayd | Certificate for tunnel.<relay-id>.<suffix> |
GatewayDeviceService.RequestRelay |
gateway | Assignment: relay address, WireGuard key and port, tunnel_url, tunnel token |
PortalGatewayService.ListRelays, SetRelayDraining |
portal, platform admin | Fleet view and drain (Relay Fleet) |
PortalGatewayService.GetRelayLink |
portal | One gateway's link, for the path view |
RelayService.ReportAttach and ReportDetach exist in the API but relayd does not call them. The assignment is recorded, and DNS is written, when nexus answers RequestRelay.
Assignment¶
RequestRelay takes the gateway's identity from its token, never from the request.
- A relay is a candidate when it is healthy, its last heartbeat is less than 90 s old, it is not draining, and it has a free slot for the role.
- The gateway's current relay is kept if it is still a candidate. A gateway moves only when its relay is unfit, or when it reports that relay unreachable, or when it has no tunnel and another relay offers one.
- Otherwise the emptiest candidate is chosen.
- The session and the gateway's tunnel address are recorded before the answer is sent.
- Nexus upserts the gateway's DNS records. If no relay is available, the answer is empty rather than an error, and the gateway asks again later.
DNS¶
| Record | Written by | TTL |
|---|---|---|
<gateway-id>.<suffix> and *.<gateway-id>.<suffix> to the relay's IP |
nexus, on assignment (internal/gwdns) |
60 s, unproxied |
tunnel.<relay-id>.<suffix> |
nexus, for the relay certificate | |
<relay_subdomain>N.<domain> (host record) |
terraform (modules/relay) |
300 s |
A Cloudflare write is skipped when the record already holds the right content. Gateways receive the relay's address in the assignment, so the host record is not on the failover path.
ReapAbandonedRelays runs from nexus's hourly sweep. It deletes relays that have been gone for a week with no attached gateway, and withdraws their tunnel names first.
Transport ladder¶
The agent tries each rung in order and keeps the first that completes a handshake. It re-probes faster rungs without disturbing the current one. See Relay Link.
| Rung | Transport | Port | Gets through |
|---|---|---|---|
| 1 | WireGuard over UDP | 51820 | Nothing in the way |
| 2 | WireGuard over UDP | 443, redirected by nftables to the same interface | Port filtering |
| 3 | WireGuard in WebSocket/TLS (udpws) |
TCP 443 | Networks that allow only TCP |
A relay with no tunnel certificate reports an empty tunnel_url, and gateways treat rung 3 as absent on it.
Identity and replacement¶
A relay holds nothing worth keeping, so it is replaced rather than updated:
- its credential comes from enrolment
- its WireGuard key is generated at boot
- its certificate is obtained again
- its routing table arrives with the first heartbeat
The public IP (scaleway_instance_ip) and the host record are separate resources and survive a replacement. The procedure is Relay Replacement, and it uses scripts/relay-replace.sh.
Releases¶
relayd is built on a tag (linux/amd64, -trimpath) and published to the project's generic package registry as relayd/<tag>. Each environment's relayd_package_url pins the version that new instances install. A cloud-init deploy token with read_package_registry fetches it. There is no channel and no self-update: changing the version means editing the URL and replacing relays one by one.
Design constraints¶
- No fleet-wide key on a relay. Rung-3 tokens are minted by nexus per peer and sent on the heartbeat. A relay compares them and never derives one.
- One terminated name, no wildcard. A wildcard certificate on a relay could impersonate every service it forwards.
tunnel_urlis reported, never derived. If the URL were derived, a failed TLS handshake would look to the ladder like a broken rung rather than an absent one.- The whole set on every heartbeat. Removal is how revocation propagates within 30 s.
- Nexus brokers certificates. No relay and no gateway holds a DNS credential.