Skip to content

Troubleshooting

Agent won't connect

doas rcctl check edge_agent
doas tail -50 /var/log/daemon | grep edge-agent
nc -z <nexus-host> 443
nc -z <realtime-host> 443

Common causes: clock drift on a freshly installed box (doas rdate -n time.cloudflare.com), and an identity.json pointing at another environment's URLs.

Realtime returns 503 "nexus unavailable"

The Durable Object's call back to nexus failed. Two common causes:

  1. Trailing newline in the INTERNAL_API_KEY Worker secret. workerd rejects the X-Internal-Key header; wrangler tail shows Invalid header value. Re-set it without a newline (Terraform → Secrets).
  2. Wrong NEXUS_URL in the realtime deploy. Realtime CI passes it with --var NEXUS_URL:... per job; prod must be https://prod.nexus.digilan-edge.com.

Portal shows "Unable to load organizations"

Every portal RPC is failing, usually because the bundle was built with a wrong VITE_NEXUS_URL, or the user's token was issued for another environment's Zitadel project (each nexus accepts only its own environment's tokens). Check a ConnectRPC request in the browser's network tab; rebuild with the right variables if the host is wrong.

"No organization yet" after signing in

The account has no grant. See Authz → "No organization yet".

Config push reports "offline" immediately

The gateway's WebSocket is not attached to its Durable Object. Nexus keeps the desired config version; the agent fetches it on reconnect (GatewayDeviceService.GetConfig).

If the device should be online, check the agent (rcctl check, log tail) and probe https://<realtime-host>/gateway/none?token=bogus with WebSocket headers: a 401 "invalid token" means the path is healthy.

Config rolled back

The agent did not receive a confirmation before its rollback timer expired. It restored the previous config and reported rolled_back (rollback timer expired). Re-apply and confirm.

Nexus fails to start: JWKS

portal jwks: ...

Nexus loads the Zitadel JWKS at startup and exits if it cannot. Check that auth.digilan-edge.com responds and that NEXUS_ZITADEL_ISSUER is right. A Zitadel cold start can cause this; the next start usually succeeds.

A device reads as the wrong version

  • edge-agent version prints dev: a stale copy at /usr/local/bin/edge-agent is ahead on PATH. The service runs /usr/local/sbin/edge-agent; use the absolute path.
  • A version means nothing without its channel. Tags publish to beta; stable is a separate manual promotion. A fleet that looks behind is usually on stable while the build sits on beta. See Agent Releases.
  • Do not compare the agent's pf hash with nexus's config hash. They are computed over different bytes and never match.

An OTA update never lands

Scaleway object storage resets any HTTP/2 response body past about 10 MiB, and Go negotiates HTTP/2 over TLS by default. The updater pins its download client to HTTP/1.1 (downloadTransport() in internal/selfupdate). When reproducing a download by hand, force HTTP/1.1 or you reproduce the failure instead:

curl --http1.1 -o /tmp/agent '<presigned-url>'

Also check that the device is assigned (unassigned devices are not offered updates) and that a release exists for its channel and arch.

A gateway went silent after an update

The watchdog restores the previous binary if the new one does not confirm within 90 seconds. If a box still ends up without a working agent and you have the portal Console or a local login, install a known-good build by hand (docs/manual-agent-recovery.md in edge/gateway). Stop the service first and replace the binary with an atomic rename on the same filesystem; never cp over a running binary (ETXTBSY).

The portal looks stale after a deploy

An automated fetch of the portal can receive a Cloudflare bot-challenge page, which contains none of the new strings and reads like a deploy that did not land. Check in a real browser, or look for a string from the old build in the same response: if that is missing too, you are looking at the challenge page.

Two agent processes

Config flickering between versions can mean two edge-agent processes:

ps aux | grep edge-agent

Stop the service, kill any leftover process, and start it once with doas rcctl start edge_agent.

Scaleway container stuck in error

scw container container list name=<container> region=fr-par
scw container container get <id> region=fr-par

image was not found in container registry means the image tag in tfvars does not match what CI pushed: nexus and authz push :dev, :stg and :prd (not :prod).

For a scaleway_container_domain in error, see Terraform → Recovering a container_domain.