Skip to content

Agent Releases

The release catalogue in each environment's nexus is what devices update to. A build has to be published to an environment before any device there can be pointed at it.

How a build is published

Tagging edge/gateway runs the release pipeline. Merging to main publishes nothing.

cd edge/gateway
git tag vX.Y.Z-beta && git push origin vX.Y.Z-beta

Tags are named vX.Y.Z-beta. The tag is the version string the agent reports.

For each arch (amd64, arm64) CI:

  1. cross-compiles the OpenBSD agent with the version and the OTA public key baked in,
  2. signs the binary with the shared ed25519 key and verifies the signature,
  3. for each of dev and prod: uploads it to that environment's private bucket (digilan-agent-releases-<env>) and registers it with that environment's nexus (ReleasePublisherService.PublishAgentRelease, gated by X-Release-Key) on the beta channel.

Each (environment, arch) pair is its own job (publish:dev:amd64, publish:prod:arm64, ...), so a prod failure leaves dev's release in place and can be retried alone. Every environment receives the same bytes, sha256, signature and object key. stg has no release jobs.

Channels

Channel How a build gets there
beta Automatically, when a tag is published
stable Only by a manual promote:stable:<env>:<arch> job on the tag's pipeline

Devices track stable unless moved. Promotion registers the same object under stable (no rebuild, no re-upload), so stable receives exactly the bytes that were tested on beta. Each environment and arch is promoted by its own job: promote:stable:dev:amd64, promote:stable:dev:arm64, promote:stable:prod:amd64, promote:stable:prod:arm64.

Nexus serves the newest release per (channel, arch) by registration time.

Install images must embed a tag-built agent (AGENT_VERSION or AGENT_BIN; the image builders refuse anything else), or the devices they install refuse OTA updates; see Updating the agent.

Canarying a release

  1. Tag. The build appears on beta in dev and prod.
  2. On a canary device's page, set the Agent version card's Channel to beta. This changes which builds the device is offered and nudges it to check; it does not install anything by itself.
  3. Update agent on the canary and watch the phase timeline complete.
  4. Leave it long enough to cover what the change touches (a reboot, a config apply, a WAN failover).
  5. Run the promote:stable:<env>:<arch> jobs. Stable devices then see Update available.

Never read a device's version without its channel

A fleet that is current on beta and a fleet that has fallen behind stable look the same in a version column. Read version and channel together.

An update runs the installed updater

The update that delivers a fix to the updater or watchdog is installed by the version being replaced. The fix protects the update after it, not the one that delivers it. Treat the first hop onto such a release as the risky one.

Backfilling a tag into prod

A manual pipeline on main with BACKFILL_TAG=<existing tag> takes that tag's build artifact, checks it, re-derives sha256 and signature, and publishes it to prod on beta; the prod promote jobs then put it on stable. Backfill several tags oldest first, since the newest registration wins.

CI variables

Set on digilan/edge/gateway from tofu output (also listed at the top of its .gitlab-ci.yml):

Scope Variables
shared OTA_S3_ENDPOINT, OTA_S3_REGION, OTA_SIGNING_PRIVATE_KEY (file), OTA_SIGNING_PUBLIC_KEY (file)
dev OTA_S3_BUCKET, OTA_PUBLISHER_ACCESS_KEY, OTA_PUBLISHER_SECRET_KEY, NEXUS_URL, RELEASE_PUBLISHER_KEY
prod OTA_S3_BUCKET_PROD, OTA_PUBLISHER_ACCESS_KEY_PROD, OTA_PUBLISHER_SECRET_KEY_PROD, NEXUS_URL_PROD, RELEASE_PUBLISHER_KEY_PROD

Secret values are masked. The signing pair comes from environments/shared (ota_signing_private_key, ota_signing_public_key); the others from the environment's outputs (ota_bucket_name, ota_s3_endpoint, ota_publisher_access_key, ota_publisher_secret_key, release_publisher_key).

Keys and storage

  • Signing key: one ed25519 key for all environments (environments/shared/ota-signing.tf). A gateway keeps its binary when it moves between environments, so a per-environment key would strand it. Only CI holds the private key; nexus never sees it.
  • Bucket: private, one per environment. Nexus holds a read-only identity and hands devices short-lived presigned URLs; CI holds a write-only identity. Neither can delete. Old binaries are kept, so a pinned or long-offline device can still fetch its version.
  • Publisher key: per environment, separate from the nexus/realtime internal key.

Admin API

PortalAdminService (platform admin):

  • ListAgentReleases lists the catalogue.
  • CreateAgentRelease registers a release by hand.
  • SetAgentReleaseYanked withdraws a release as a target.

A yank is not a rollback

Yanking stops devices being offered the release. Devices already running it keep running it. If a release breaks devices, publish a known-good release and move the fleet forward onto it.

Verifying the fleet

The agent reports its running version in its state report, so the device page shows what is installed, not what was requested. A device whose requested and reported versions disagree for long is stuck, not slow; see Troubleshooting.

See also Updating the agent (OTA) and the on-device mechanism in OTA Updates.