Skip to content

Replacing a Relay

Relays are immutable — they are replaced, not updated in place. A relay holds nothing worth keeping: its credential is re-derived at enrolment, its WireGuard key is generated fresh, its certificate is obtained from Let's Encrypt, and its routing table arrives on the first heartbeat. Replacement updates the OS kernel, base image, and every package alongside the daemon. There is no OTA for relays.

The relay's public IP address and DNS record are separate terraform resources and survive replacement. The new relay gets the same address. Gateways and visitors see no address change.

The replacement script

scripts/relay-replace.sh in infra/terraform wraps the full procedure: drain, wait for empty, replace the instance and bootstrap token together, and verify the new relay.

Prerequisites

Tools: curl, jq, tofu. Environment:

Variable Value
NEXUS_URL The environment's nexus, e.g. https://dev.nexus.digilan-edge.com
AUTH_TOKEN A platform-admin bearer token (the script calls ListRelays and SetRelayDraining)

Run it from the environment directory (environments/dev, environments/prod) or pass --tofu-dir. The apply plans against that directory's tfvars:

  • relay_count must be set there. With the default 0, the targeted relay is in no configuration and tofu plans to destroy it and create nothing.
  • relayd_package_url in tfvars chooses the version the new instance installs. --version only checks what the new relay reports. To upgrade, edit tfvars first, then run the script.

Usage

cd infra/terraform/environments/<env>

# Capacity check and the exact tofu command; drains nothing, applies nothing:
../../scripts/relay-replace.sh relay1 --dry-run

# Drain and wait, then print the tofu command and stop:
../../scripts/relay-replace.sh relay1

# Drain, wait, replace and verify:
../../scripts/relay-replace.sh relay1 --yes --version v0.3.2

relay1 is the hostname nexus lists; the Terraform index is its number minus one, or --index N. Other options: --timeout (drain wait, default 600 s), --verify-timeout (default 1500 s), --module (default module.relay).

When several rows share the hostname (earlier replaced instances), the script acts on the live one: fresh first, then the most recently seen.

What the script does

  1. Capacity check: the remaining fresh, non-draining relays must have enough free slots for this relay's gateways, or it refuses.
  2. Drain: SetRelayDraining(draining: true). New gateways go elsewhere; existing ones move at their next check-in.
  3. Wait for empty: polls ListRelays every 15 seconds until attachedGateways, forwardSessions and rendezvousSessions are 0. On timeout it undrains the relay and exits non-zero; nothing is destroyed.

  4. Replace (only with --yes; otherwise printed). For relay2:

    tofu apply \
  '-target=module.relay.scaleway_instance_server.relay[1]' \
  '-target=module.relay.random_password.bootstrap[1]' \
  '-target=module.relay.null_resource.bootstrap_token[1]' \
  '-replace=module.relay.random_password.bootstrap[1]' \
  '-replace=module.relay.scaleway_instance_server.relay[1]' \
  -auto-approve

    The plan reads 3 to add, 0 to change, 3 to destroy: the server, a fresh bootstrap token, and its registration with nexus. A spent token cannot enrol a new machine, so the token is replaced with the server. The relay's IP and DNS record are not replaced.

  5. Verify: polls until a relay with the same hostname and a new id is fresh, healthy and not draining, and prints its id, version and tunnel URL.

Target the three resources, never the whole module

-target=module.relay applies every pending change in the module: after a tfvars edit such as a new relayd_package_url it would also rewrite the other relays' user_data. The narrow targets leave the other relays alone; they move to the new version when they are replaced in turn. Targeting still includes the server's dependencies (IP, security group), so after editing the module itself, read the plan before the first --yes.

Replace a whole fleet one relay at a time, starting the next only after the current one is verified.

Timing

A replaced relay takes approximately 16 minutes from tofu apply to appearing healthy in the fleet page. Most of that time is cloud-init installing packages and the Let's Encrypt certificate exchange, not the control plane.

The default verification timeout is 25 minutes

The script waits up to 1500 seconds for the new relay to enrol and become healthy. If the relay has not appeared by then, it reports a timeout warning — but the relay may still be booting. Check the fleet page manually.

Rollback

Before destroy (steps 1–3): click Put back in service on the fleet page. The relay is back in service and the fleet is as it was. Gateways that already migrated stay on their new relay.

After destroy (step 4 complete): the old relay is gone. The new relay is already serving. If the new version has a problem, drain the new relay, wait for gateways to leave, and replace it again with the old version by setting relayd_package_url back to the previous tag in tfvars.

Important considerations

The deploy token must never appear in version control

TF_VAR_relayd_package_token should be read from a file with mode 0600, never stored in terraform.tfvars or committed to the repository. The relay package is hosted on a private GitLab registry and the token grants read access to it.

The new relay gets a new ID

A replaced relay enrols as a new identity with a new relay ID. The old relay's row goes quiet within 90 seconds, appears under Replaced instances on the fleet page, and is reaped after 7 days with no attached gateways, together with its tunnel.<old-id>.<zone> DNS record.

Let's Encrypt rate limits

Each replaced relay obtains a new certificate for its tunnel name. Let's Encrypt allows 50 new certificates per registered domain per week. With a fleet of 3, this is not a concern. At scale, replacing many relays in a short window could hit the limit.

See also