Automatic node switching, shipped and then turned off

What it was

In July 2025 I shipped a service that checked the health of your current node every thirty seconds and on app resume, and if it stopped answering, moved you to another node you had marked as trusted. Health meant fetching real data for the chain, a balance or a height, not a ping, because a ping calls a node healthy while it returns nothing useful. Switching was limited to trusted nodes, about thirty defaults migrated on, so the wallet never moved you onto a node you had not sanctioned.

What went wrong

A slow client and a dead node look the same from the app. Being offline was the worst case: every node looked dead and the service cycled you through every trusted node you had. Three corrective changes followed. One attempt at the slow-network case was closed in review because its extra checks created more edge cases than they closed, which was the right call. The next two were narrower: check connectivity before blaming the node, cap at five attempts with a cooldown, reset on a healthy check, and probe a candidate before switching to it so a dead node does not burn an attempt.

One disagreement in review is worth recording. The reviewer wanted the attempt counters cleared in a finally block after a switch. I held that they should clear on a healthy check, because the health check is not a ping, it only succeeds against a connected working node, so that is exactly the moment the state is known good. I took the reviewer’s two other suggestions and kept this one, with the mechanism written out, and they agreed.

What I did about it

In January 2026 I turned it off behind a feature flag. Twenty-eight lines added, eighteen removed, nothing deleted. The health checks, the per-node trusted flag, the settings and the service are all still in the tree, and turning it back on is a one-line change. The reason is the one above: it could not reliably tell a slow or offline client from a dead node, and the fixes were adding surface area rather than removing the cause. Anything more specific than that would be me guessing, so I will not say it.

Two months later the same idea came back small: a three-line fallback on broadcast for the EVM chains, inside one call, which is about the size the problem deserved.