Uplink Failover

Which uplink carries traffic is decided on the device. The agent writes the configuration and gets out of the way: the decision runs in ifstated(8), driven by a generated shell helper, so it keeps working when the agent is dead and does not depend on reaching the platform.

Source: internal/wanfailover. Uplinks and their priority come from the config's WANPriority, through internal/dhcpleased. With fewer than two uplinks there is no failover: the files are removed and ifstated is not used.

The pieces

Thing	Where	Job
/etc/ifstated.conf	generated	One state per uplink (wan_<if>); transitions guarded by carrier and probe, evaluated every 10 seconds
/usr/local/libexec/digilan-wan	generated shell	probe <if>, activate <if>, uplinks
/var/db/digilan/wan/<if>.verdict	on disk	up or down: the debounced answer. No file reads as up
/var/db/digilan/wan/<if>.streak	on disk	Evidence accumulated against the current verdict
/var/db/digilan/wan/dhcpleased.conf.<if>	generated	The dhcpleased.conf variant to use while <if> carries (ignore routes on the others)
/var/db/digilan/wan/transitions, uplink-health	on disk	Switch and health records the agent reports

The agent reads verdicts through wanfailover.ProbeVerdict, which is how internal/wwan knows whether a cellular link is worth rebuilding.

The state machine's transitions are mutually exclusive, and there is no transition when no uplink is healthy: the device stays where it is rather than switching to something equally dead.

Probing

The probe target is 9.9.9.9. probe_once asks a different question depending on whether the uplink is carrying traffic:

• Carrying: ping the target over the default route. A genuine end-to-end test.
• Dormant point-to-point (a cellular link): add a host route for the target through the peer, ping, remove the route. A cellular peer address is a placeholder the carrier neither routes nor answers, so a first-hop test would fail on every healthy link.
• Dormant with an on-link gateway: ping the first hop (the static gateway, or the DHCP server dhcpleasectl reports), sourced from this interface's own address. The source is what makes it a test of this uplink.

All uplinks are probed continuously. standby in the state report means not carrying and not judged down.

The debounce is a decay

ifstated cannot count, so the helper does. A probe that disagrees with the standing verdict adds to the streak; FailuresToFail (3) flips the verdict to down, SuccessesToRecover (6) flips it back. A probe that agrees with the verdict decrements the streak rather than clearing it, and the streak resets to zero when the verdict flips.

Resetting instead of decaying hides a link that is mostly broken

If an agreeing probe cleared the streak, one reply in three would cancel the evidence of the other two, and an uplink losing most of its packets would keep its up verdict indefinitely. It would also suppress the cellular rebuild, which acts only on a down verdict.

Recovery needs more evidence than failure, so a marginal line does not drag the default route back and forth.

Verdicts and streaks are files, so they survive an ifstated restart; only a teardown (fewer than two uplinks) removes the directory.

Activating an uplink

activate <if> returns at once if the uplink is already fully in place. Otherwise it:

1. installs that uplink's dhcpleased.conf variant and restarts dhcpleased, so only this uplink's lease installs a default route;
2. removes every other default route (dhcpleased can leave one per lease) and installs this one (for a point-to-point link, route add default <peer>);
3. for a point-to-point uplink, sets fallback nameservers with route nameserver <if> 9.9.9.9 149.112.112.112 and withdraws them from the other uplinks (DHCP uplinks get DNS from dhcpleased/resolvd);
4. waits up to 10 seconds for this to be the only default route. On success it puts the carrying interface in the egress group and removes every other, so egress means exactly one interface, and records the switch in transitions (capped, oldest lines trimmed). On failure it restores the previous uplink.

The egress group is what NAT and firewall rules are written against: nat-to (egress) follows a failover; a rule naming a physical port stops matching when traffic moves. See PF Generation.

Applying

Applier.Apply writes the variants, then the helper, then ifstated.conf; checks it with ifstated -n -f (a rejection is logged, not fatal); enables and restarts ifstated; and re-activates a non-primary uplink that was carrying, so a config apply does not snap traffic back to a dead primary. Files that do not carry the generated marker are never overwritten.

Testing a failover without a cable

• Applying a config that blocks the probe target does not work: the rollback window (60 s) cannot contain the consecutive failures a failover needs, and the restart re-runs activate.
• A pf rule added by hand does nothing on its own: pf skips rule evaluation for packets matching an existing state, and the probe's ICMP state is refreshed every 10 seconds. Kill it with pfctl -k 0.0.0.0/0 -k 9.9.9.9 (a single -k matches the source, which is the device). A pass ... quick rule can still win over one appended at the end; an anchor is the supported way to inject a rule.

The reliable test is unplugging the cable, judged from a client behind the gateway. The gateway's own traffic takes the new uplink regardless, so it looks healthy in exactly the case where everything behind it has lost the internet.