Skip to content

Setting Up a New Environment

The full procedure, in order: shared resources, the environment, its deployments, the first admin and the first device. The architecture is described in How It Works.

Everything here is named by variable, output or CI variable. Never paste a value from terraform.tfvars, tofu output, .zitadel-apps.json or .zitadel-service-user.json into a ticket, a chat or these docs.

Before you start

  • Tools: glab (signed in to the work account), tofu, jq, curl, node/npx (for wrangler).
  • A Scaleway API key for the Digilan project. An apply that creates IAM applications needs a key with IAMManager. Both the shared root (the SMTP identity) and every environment root (the OTA reader and publisher identities) create them. Each root's provider pins organization_id to scaleway_organization_id.
  • A Cloudflare API token that can edit DNS in the main zone and manage Pages projects on the account.
  • Maintainer access to the GitLab projects, to set CI variables and run manual jobs.

All commands run from the infra/terraform checkout.

State backend

State lives in GitLab's HTTP backend on digilan/infra/terraform. One state per root:

glab opentofu init -R 'digilan/infra/terraform' '<state>' -d ./environments/<state>

<state> is shared, dev, stg or prod. The token that init writes is short-lived: run init again before every plan or apply session. A plan that fails on the backend with an authentication error means the token has expired, not that the state is broken.

1. Shared resources (once for all environments)

environments/shared creates:

  • the container registry
  • the Zitadel container with its own RDB instance and DNS (auth.digilan-edge.com)
  • the shared Postgres instance digilan-shared
  • transactional email (TEM) with its SPF, DKIM and DMARC records

  • the OTA ed25519 signing key

  • Create environments/shared/terraform.tfvars from environments/tfvars.example. It needs these keys: scaleway_region, scaleway_project_id, scaleway_organization_id, scaleway_access_key, scaleway_secret_key, registry_name, cloudflare_api_token, cloudflare_zone_id, and zitadel_pat (empty at first).

  • init, tofu plan, read the plan, tofu apply.
  • Sign in to the Zitadel console at https://auth.digilan-edge.com/ui/console as the admin named by the output zitadel_admin_login. Its password is the output zitadel_admin_password.
  • Create a service account with IAM_OWNER at instance level, generate a personal access token for it, and set it as zitadel_pat.
  • Apply again. With zitadel_pat set, the shared root:
    • configures SMTP through TEM
    • sets the login policy, with self-registration and external-IdP registration closed on the instance and on every org
    • applies the branding
    • creates the nexus-service machine user (ORG_USER_MANAGER), writing its id and token to environments/shared/.zitadel-service-user.json (git-ignored). The outputs are nexus_service_user_id and nexus_service_pat.

Upload logos by hand in the Zitadel console under Default Settings > Branding. The REST API cannot upload them.

2. Code changes for a new environment name

dev, stg and prod are already wired everywhere except where noted. For any other name, make these changes first and merge them:

Repo Change
infra/terraform Copy environments/<existing> to environments/<env>, with the defaults in variables.tf: environment, the subdomains, db_name. For relays, also copy relay.tf, relay-zone.tf and relay-variables.tf, from prod (stg has none).
platform/nexus Add build_<env> and deploy_<env> jobs to .gitlab-ci.yml (CONTAINER_NAME: nexus-<env>). Add the portal origin to portalOrigins in internal/app/app.go: CORS is a fixed list, and a portal not in it gets no answers.
platform/authz Add build_<env> and deploy_<env> jobs (CONTAINER_NAME: authz-<env>).
platform/realtime Add [env.<env>] to wrangler.toml (Worker name, custom-domain route, DO binding and migration), and a deploy_<env> job that passes NEXUS_URL.
platform/portal Add build_<env> (its VITE_* values) and deploy_<env> (PAGES_PROJECT_NAME: digilan-portal-<env>) jobs.
edge/gateway Add a .target:<env> block and publish:<env>:<arch> / promote:stable:<env>:<arch> jobs. Only dev and prod are targets today; stg receives no agent releases.

3. Images

The environment root points nexus and authz at registry images (nexus_image, authz_image). Push a tag on platform/nexus and on platform/authz before the first apply, so that their build_<env> jobs publish rg.fr-par.scw.cloud/digilan/nexus:<tag> and authz:<tag>. The tags are dev, stg and prd. The deploy jobs fail or wait until the containers exist; that is expected.

4. Environment root

  1. Create environments/<env>/terraform.tfvars. It needs these keys:
    • Scaleway: scaleway_project_id, scaleway_access_key, scaleway_secret_key (plus scaleway_zone and scaleway_region if not the defaults).
    • Cloudflare: cloudflare_api_token, cloudflare_zone_id, cloudflare_account_id.
    • Images: nexus_image, authz_image.
    • Shared database, copied from the shared outputs rdb_shared_*: shared_rdb_instance_id, shared_rdb_host, shared_rdb_port, shared_rdb_admin_user, shared_rdb_admin_password.
    • Zitadel: zitadel_pat (the shared PAT, so the root can create the environment's project and apps) and nexus_zitadel_service_token (the shared output nexus_service_pat, which nexus needs to invite users).
    • Optional: nexus_min_scale, nexus_max_scale, nexus_log_level, authz_log_level, and the relay keys in step 8.
  2. init, then tofu plan and tofu apply. This creates:
    • the database <db_name> on the shared instance
    • the container namespace digilan-<env> with nexus-<env> and authz-<env> and their DNS
    • the JWT keypair in Secret Manager
    • the shared keys (authz_api_key, internal_api_key)
    • the Pages project digilan-portal-<env> and its domain
    • the OTA bucket digilan-agent-releases-<env> with its reader and publisher identities
    • the release publisher key
  3. During that apply, null_resource.zitadel_apps runs scripts/zitadel-apps.sh. The script creates or reconciles the Zitadel project digilan-<env> and the portal app (plus the Monsieur Wifi and radius apps when their URLs are set), and writes environments/<env>/.zitadel-apps.json.
  4. Plan and apply once more. NEXUS_ZITADEL_PROJECT_ID is read from .zitadel-apps.json at plan time. On the first apply the file did not exist yet, so nexus got an empty project id and rejects every portal token.

To run the script by hand instead, set its inputs (ZITADEL_URL = the shared output zitadel_container_url, ZITADEL_PAT, ZITADEL_HOST, ENVIRONMENT, PORTAL_URL, MRWIFI_URL, RADIUS_URL, OUTPUT_FILE, DEV_MODE) and run scripts/zitadel-apps.sh. The script is idempotent.

Pages project settings (build_config, deployment_configs, source) belong to the portal's deploys. Terraform manages only the project and its name, so a portal deploy does not show up as drift.

5. Realtime

  1. Store the internal key as the Worker secret. Run this from environments/<env>:

    tofu output -raw internal_api_key | npx wrangler secret put INTERNAL_API_KEY --env <wrangler env>

    Run it in a platform/realtime checkout so wrangler finds wrangler.toml. The wrangler env for prod is production.

  2. Push a tag on platform/realtime and run deploy_<env>. It is automatic for dev and manual otherwise. CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID come from the digilan/platform group variables. The custom domain in wrangler.toml creates <realtime_subdomain>.digilan-edge.com.

6. Nexus, authz, portal

  1. Nexus and authz: run deploy_<env> in each project's tag pipeline. It is automatic for nexus dev and manual everywhere else. The job looks the container up by name and redeploys it, so it pulls the image again. On start, nexus runs its migrations and seeds the roles into authz. Check https://<nexus>/healthz.
  2. Portal: set the CI variables <ENV>_ZITADEL_CLIENT_ID and <ENV>_ZITADEL_PROJECT_ID on platform/portal from the outputs portal_client_id and zitadel_project_id. The prefixes are DEV, STG and PRD. Push a tag and run deploy_<env>. It is automatic for dev and manual for stg and prod.

A change that terraform makes to nexus's environment, such as the relay suffix or a new key, reaches the running container only when the container is redeployed.

7. Agent releases (OTA)

Set these on edge/gateway (Settings > CI/CD > Variables). Mask the secrets.

Scope Variables Source
All targets OTA_S3_ENDPOINT, OTA_S3_REGION s3.<region>.scw.cloud, the region
All targets OTA_SIGNING_PRIVATE_KEY, OTA_SIGNING_PUBLIC_KEY (file type) shared outputs ota_signing_private_key, ota_signing_public_key
dev OTA_S3_BUCKET, OTA_PUBLISHER_ACCESS_KEY, OTA_PUBLISHER_SECRET_KEY, NEXUS_URL, RELEASE_PUBLISHER_KEY dev outputs ota_bucket_name, ota_publisher_access_key, ota_publisher_secret_key, release_publisher_key, and dev's nexus URL
prod OTA_S3_BUCKET_PROD, OTA_PUBLISHER_ACCESS_KEY_PROD, OTA_PUBLISHER_SECRET_KEY_PROD, NEXUS_URL_PROD, RELEASE_PUBLISHER_KEY_PROD the same outputs from prod

Dev uses the unsuffixed names; every other target uses _<ENV>, as declared in its .target:<env> block. A publish job fails loudly if any of its variables is missing.

The next tag publishes to beta in each target. Promote with the manual promote:stable:<env>:<arch> jobs (Agent Releases). A device takes updates only once it is assigned.

8. Relays

Relays and per-gateway names are optional. Without them, gateways run, but there are no published services and no relay link.

  1. Turn on the nexus side first. Set relay_zone_suffix in tfvars (for example r.<env>.digilan-edge.com). Prod's variable refuses dev's suffix r.digilan-edge.com and anything inside it. Apply, then redeploy nexus. Nexus now has NEXUS_RELAY_ZONE_SUFFIX, the zone id, a DNS token and the ACME account.
    • For a zone of its own, set relay_zone_name (needs a Cloudflare token that can create zones), or relay_dns_zone_id and relay_dns_api_token for a zone made by hand.
    • Otherwise the names live in the main zone.

  2. Then create the relays. Export the package token, then set relay_count and relay_subdomain:

    export TF_VAR_relayd_package_token="$(cat ~/.config/digilan/relayd-package-token)"
    • relay_subdomain must not be relay, which is dev's.
    • relayd_package_url pins the relayd version.
    • relay_count above 0 is refused without the package token.

    Apply. For each relay, terraform registers its bootstrap token with nexus through scripts/relay-token.sh, so nexus must be up and reachable. Terraform then creates the instance, whose cloud-init installs relayd, which enrols. 3. Check Admin > Relays in the portal: each relay is fresh, healthy, and has a tunnel URL.

9. First admin

Sign-up is closed, so the first account is made in Zitadel and the first grant in SQL. Nobody exists yet who could grant it through the API.

  1. In the Zitadel console, create a human user in the organization the portal signs into, with a verified email. Copy its user id: that is the token's sub.

  2. Connect to the environment database (DSN from tofu output -raw database_connection_string in environments/<env>; it is a live secret) and grant platform admin:

    INSERT INTO acl (subject_id, role_id, target, target_id)
VALUES ('<zitadel user id>', 3, 'portal:system', '*')
ON CONFLICT DO NOTHING;

    target_id must be *: platform admin is the wildcard action on portal:system.

  3. Sign in to the portal. A second factor can be enrolled; it is not required. Admin appears in the navigation.

10. Organizations and accounts

  • Organizations: Admin > Organizations > Create. Only a platform admin can create one. Signing in never creates a personal organization.
  • Accounts: Account > Team > Invite. Nexus creates the Zitadel user, which sends the invitation email, and the role grant follows on the same screen. ORG_OWNER manages the org and its members. ORG_MEMBER operates its gateways without the console or member management. Inviting requires grant authority: a platform admin, an ORG_OWNER, or write/transfer on a device.

A user with no grant who signs in sees the "no organization" screen. That almost always means a missing grant, not a Zitadel problem. See Authz.

11. First device

  1. Admin > Enrollment: create a bootstrap token. Name it, and set a use cap and an expiry if you want. The token is shown once.
  2. Install the gateway:
    • with the provisioning USB, which bakes the token in (Provisioning);
    • or on an installed OpenBSD box, with edge-agent setup --addr https://<nexus> --realtime wss://<realtime> --token <token>.
  3. The device appears in Admin > Devices in the holding zone. Assign it to an organization, and from then on it is managed and receives agent updates.

Gotchas

  • Re-run glab opentofu init before each plan or apply session. The backend token it stores expires.
  • Creating IAM applications needs IAMManager on the Scaleway key. Otherwise the apply fails with "insufficient permissions".
  • Apply twice on a new environment. The first apply writes .zitadel-apps.json; the second gives nexus the project id.
  • .zitadel-apps.json and .zitadel-service-user.json are local files, not state. They are never committed. Keep a copy outside the repo (~/.config/digilan/<env>-zitadel-apps.json, mode 0600) and copy it in before planning. Without them a plan sets the Zitadel ids to empty strings. Re-running the script recovers the ids but not the Monsieur Wifi client secret, which Zitadel shows only once: if it is lost, regenerate the secret of mrwifi-<env> in the Zitadel console.
  • A relay booted before nexus offers certificates reports no tunnel URL, so gateways get no rung 3 from it. relayd checks again every 60 minutes. systemctl restart relayd on the relay (serial console), or rebooting the instance (scw instance server reboot <id>), gets it at once. Avoid this by setting relay_zone_suffix and redeploying nexus before raising relay_count.
  • relay_count defaults to 0. Any plan or apply run against a tfvars that does not set it plans to destroy every relay. Read the plan.
  • Nexus CORS is in code. A new portal hostname must be added to internal/app/app.go.
  • Serverless containers scale to zero. The first request after idle is slow, and nexus's hourly sweep (log pruning, relay reaping) runs only while an instance is up.