Self-Heal

internal/selfheal: the standing guard against a device that has gone quiet and cannot be reached to fix. The rollback guard covers a config apply for 60 s and the self-update watchdog covers a new binary for 90 s; this covers everything after.

It runs as a generated shell script, selfheal.sh in the data directory, from a managed block in root's crontab every 5 minutes, so it works when the agent is dead.

Contact and known-good

• The agent stamps last-contact on every realtime connect.
• Once a build has held a connection for 15 minutes, the agent snapshots the running binary and config into known-good/. A build that passes the 90 s update probation but wedges shortly after is never snapshotted.

Contact is stale after 20 minutes without one, comfortably longer than a platform redeploy.

What a run does

Situation	Action
Never had contact	Nothing: a box that has not enrolled has nothing to roll back to
An update is on probation	Nothing: the update watchdog owns that
Agent not running, box matches known-good	Start the agent
In contact	Reset the isolation ladder
Stale, box differs from known-good	Restore known-good binary and config, restart the agent
Stale, box matches known-good, internet reachable	Nothing
Stale, box matches known-good, no internet	Climb the isolation ladder

The difference check is what makes it safe to run forever. An internet or platform outage makes contact stale on a box that already matches its snapshot, so nothing is restored and nothing loops. Only a box that changed and then went quiet is rolled back. The restore stops the agent first and uses a staged copy plus atomic rename, for the same ETXTBSY reason as the update watchdog.

Reachable internet means the fault is on the platform's side. That would be true on every device at once during a platform outage, so acting on it would be a fleet-wide reboot storm. The internet probe is ping -c 1 -w 3 9.9.9.9, the same target the uplink failover uses, so the two never disagree about the same network.

The isolation ladder

For a box that is stale, unchanged and has no internet at all. One rung per run, cheapest first, each given 10 minutes to work before the next:

1. Bounce the uplink: ifconfig <if> down; sleep 2; ifconfig <if> up on the interface holding the default route at that moment. This is what unplugging the cable does for the upstream port, and a software restart does not reliably reproduce it.
2. Renew the lease: rcctl restart dhcpleased.
3. Restart the agent.
4. Reboot, once.

The rung and the time it was tried are kept on disk in isolation/, so the reboot does not restart the ladder from the bottom and loop. After the reboot rung nothing more happens until the device makes contact again, which resets the ladder. A genuinely isolated device reaches the reboot after roughly half an hour.

Actions are logged to syslog with the tag edge-agent-selfheal.