High availability
Does your second
DNS server work?
Your router hands out two DNS servers, but a device only uses its second one when the first stops answering. So you find out it doesn't work on the day the first one dies. A failover drill finds out now: one node goes quiet for chosen devices for a few seconds, and the report says who switched, how fast, and who was stuck.
Why it matters
Two servers you've never tested
Most systems ask their first DNS server until it stops answering, and only then their second. On a healthy network the second server sees nothing from them, so a missing entry, a typo in the router, or a device that only takes one server stays invisible. Watching normal traffic can't tell you; a short, deliberate silence can.
Who followed, how fast
Per device: the queries the drilled node left unanswered, the first query another node answered, and the time between them. Coverage is followed / (followed + stuck), with the median and p90 time to failover.
Who was stuck
A device that kept asking the silent node and nobody else has no working second server. The report lists them first, with their names and groups, so you know which router setting or device to fix.
Who came back
Fifteen minutes later the report adds whether each device returned to its first server or stayed on the second. Windows stays for about 15 minutes; that's normal, and worth knowing.
Small and safe by default
15 seconds to 5 minutes, never longer. Refused when no other node is ready, a drill just ran, a rollout is baking, or health is severe. A restart cancels it. Start with your own laptop.
The rest keeps working
The node's own probes, its infrastructure, the other cluster nodes, and excluded devices are always answered. The API, the cluster channel, and metrics stay up. It's a maintenance window too, so alerts don't page you.
Kubernetes, honestly
A resolver pod behind one balancer address measures the balancer, not your devices: they never see a server change. The report says so and how long the Service kept sending to the silent pod.
How it works
A maintenance window with a drop
- Preview: on the Cluster page, a node's Drill…: how long, whom (some devices, a group, everyone), why. The preview runs every check and lists the devices that used the node in the last hour and would go unanswered.
- The drill: the node tells every other node to start recording, then stops answering its targets: UDP queries get no answer, TCP and DoT connections close, each counted as
drilled. It reports not ready, like a node in maintenance. - The report: a minute after the end (late switches count), the drilled node gathers what every node saw and works out the verdicts. It's on the Drills tab, in
GET /api/v1/drills/{id}, as a marker on the charts, and in the metrics.
Devices without a working second server lose DNS for the drill's length. Start with one device for 30 seconds, then a group, then everyone. The cluster guide covers giving devices two servers in the first place.
For agents
Planned, approved, reported
An AI agent plans a drill with plan_failover_drill: the preview carries every precondition and the devices, and an operator can approve it in the agent inbox. It needs the ops:drill scope, which no token gets by default. drill_report reads the results. See AI agents.