Skip to content

OTA Agent Updates

Gateways update their own edge-agent binary over their outbound connection, without SSH. A signed build is published per environment and selected per device channel. An out-of-process watchdog reverts a binary that does not come up. For the operator procedure see Agent Releases and Updating the agent. The sequence is drawn in How It Works.

Two reconciled axes

The agent converges config and binary the same way: nexus holds the desired state, and pushes only nudge the device to ask sooner.

Axis Desired Actual Nudge Safety net
Config devices.desired_config_version devices.config_version apply_config push 60 s in-process rollback
Agent pin (devices.desired_agent_version) or latest on devices.agent_channel devices.agent_version (from state reports) update_agent command out-of-process watchdog

GatewayDeviceService.GetDesiredState returns both axes. The agent calls it:

  • at startup
  • on every connect
  • every 30 minutes
  • when it receives an update_agent command

Release pipeline (edge/gateway/.gitlab-ci.yml)

A pipeline that publishes runs only on a tag. Release tags are named vX.Y.Z-beta.

  1. Build: build:amd64 and build:arm64 run GOOS=openbsd go build -trimpath with -X main.version=<tag>, and bake the public key in with -X main.otaSigningPublicKey=…. Each binary is signed with ed25519 (cmd/otasign) and verified against the public key before anything is published.
  2. Publish: publish:<env>:<arch> uploads to s3://digilan-agent-releases-<env>/agent/<tag>/edge-agent-openbsd-<arch>. After the upload, it registers the release with that environment's nexus on channel beta (ReleasePublisherService.PublishAgentRelease, X-Release-Key).
  3. Promote: promote:stable:<env>:<arch> is a manual job. It registers the same object, sha256 and signature on channel stable. Each environment is promoted on its own.

Every tag publishes to beta on both targets, dev and prod. stg receives no releases. A manual pipeline with BACKFILL_TAG set registers the build of an existing tag in prod without rebuilding it.

Nexus refuses a second row for the same (version, arch, channel). Within a channel, the newest registration wins.

Selection in nexus (internal/release)

DesiredFor returns nothing in any of these cases:

  • the device is not assigned, or has no reported architecture
  • nothing is published for its channel and architecture
  • the device already runs the selected version
  • the selected channel release is older than what the device runs (channel downgrades are refused; a pin may go backwards)
  • the object store is not configured

Otherwise it returns the release with a short-lived presigned GET from the environment bucket. Nexus signs it with its read-only key.

Portal action RPC Effect
Update (device Agent card) UpdateGatewayAgent Pins desired_agent_version (or clears the pin to follow the channel), records a scheduled phase, and sends update_agent
Channel selector SetAgentChannel Sets devices.agent_channel (stable by default, or beta)
Admin release list ListAgentReleases, SetAgentReleaseYanked A yanked release is never selected

On the device (internal/selfupdate)

  1. Download to the data directory using the presigned URL. An expired URL is not retried, because the next reconcile mints a fresh one.
  2. Verify the sha256 and the ed25519 signature against the key baked into the running binary. Abort on any mismatch. A binary built outside the release pipeline has no baked-in key and refuses every update, so an install image must embed a release build of the agent.
  3. Back up the running binary, write an update-pending marker, and arm the watchdog.
  4. Swap the binary and restart the edge_agent service.
  5. The new agent confirms after it starts and reconnects. Each phase is reported over the WebSocket as agent_update, and the DO relays it to nexus as ReportAgentUpdate (Agent Update Progress).

Watchdog

The watchdog is a generated sh script, launched detached so that it outlives the restart. It sleeps through the probation window, then checks for the confirmation marker. If the marker is missing, it rolls back:

  1. Stop the new agent, force-killing it if needed, so nothing keeps the binary mapped.
  2. Restore the backup with a staged copy and an atomic mv. A direct cp over a running binary fails with ETXTBSY.
  3. Remove the pending marker, and only then restart the service.

Keys and storage

Item Where Used by
ed25519 signing key environments/shared/ota-signing.tf, one key for all environments CI signs; agents verify with the baked-in public key
Release bucket digilan-agent-releases-<env>, private nexus presigns reads; CI writes
Reader key IAM app digilan-ota-reader-<env> (ObjectStorageObjectsRead) nexus
Publisher key IAM app digilan-ota-publisher-<env> (ObjectStorageObjectsWrite) CI
Release publisher key random_password.release_publisher_key, secret digilan-release-publisher-key-<env> CI, as X-Release-Key

The signing key is shared so that a gateway moved from one environment to another, which keeps its binary, still verifies releases there. The bucket expires objects under agent/nightly/ after 30 days and keeps everything else.