Decisions ADR-151 to ADR-151

Context

CLU-014 (owner request 2026-10-18). Routers hand devices two DNS servers (the Pi and the homelab's .112), but a stub resolver uses its first server until it stops answering, so on a healthy network nothing shows whether the second one works for every device; a passive "single-homed" finding would be guesswork. Design: docs/design/failover-drills.md.

Decision

• A drill is a maintenance window with a drop (reuses OPS-010/ADR-118: readiness off, alerts muted, out of elections and the health level, the window file, forwarding, handover on the primary). For its targets (addresses or names, a group, or everyone) the node discards UDP queries after client identification and before rate limiting, and closes TCP, DoT, DoH and DoQ connections without an answer: the nearest thing to a dead node. Each counts as a new query status drilled.
• Query path: one relaxed load of the drill's end time (0: none); only during a drill, one ArcSwap load of the target set and a lookup. Never dropped: [clients] infrastructure, loopback, the node's own and other cluster nodes' addresses, and excluded devices (OBS-022).
• Bounds: 15 to 300 s, default 60; the cap is not configurable (a long drill is an outage). Ends on expiry or DELETE. A restart cancels a drill (unlike maintenance, which resumes).
• Preconditions (409 drill_refused with the reason): another node ready and answering in the last 5 minutes; no maintenance on the node and no drill anywhere in the cluster in the last 10 minutes; no pinned or baking rollout (CLU-013); health not severe; targets that resolve to a device seen on the node in the last hour (unless everyone). Not a configuration change: allowed under GitOps authority, audited, operator role; agents need the scope ops:drill (not in the defaults) and a reason.
• Coverage: a per-client seen table on the telemetry thread (baseline, affinity; 4,096 cap, persisted); a per-client window map on every node while any node drills (learned from the heartbeat's new drill_id), frozen 60 s after the end; the drilled node merges the parts (federated read, retried 5 minutes) into a saved report (last 50 kept) with verdicts followed (answered elsewhere after the first drop, by 5 s after the end) / stuck (≥ 3 dropped, no such answer) / quiet / elsewhere, time to failover, and returned vs sticky after 15 minutes.
• Interfaces: POST/DELETE/GET /api/v1/nodes/{id}/drill (dryRun), GET /api/v1/drills[/{id}], GET /api/v1/clients/affinity; MCP plan_failover_drill, drill_report; the Cluster page's Drill… and Drills list; metrics; annotations drill.started/drill.ended. Never a health reason.

Consequences

• The first real drill should be on the owner's own laptop for 30 s, then a group, then everyone; never run against the live Pi without the owner's go.
• Scheduled drills and a REFUSED mode were open questions; the owner said yes to both (below), and both are built.
• Devices that never queried during the window are quiet: a drill can't tell an idle device from one without DNS.

Amendments at implementation (T15.2, 2026-10-19):

• Closing connections: a new Response::Close in telltale-net. TCP and DoT close the connection; UDP drops; DoH and DoQ answer nothing but keep the connection, because HTTP/2 and QUIC multiplex other queries on it.
• "Loopback" is the node itself: 127.0.0.1 and ::1 are never dropped, nor held by FLT-017's analytics hold. Other 127.0.0.0/8 addresses are ordinary clients: the cluster e2e's stub resolvers bind them. Masking's infrastructure list (which has all of 127/8) is used without its loopback entries.
• Who's in the report: the targets seen on the node in the hour before (the baseline's targets), any device the node dropped, and any device another node answered during the window. A non-target the drilled node answered as usual is left out. A target with 1 or 2 unanswered queries and nothing elsewhere is quiet: too little to tell.
• The window starts at once: the drilled node calls every peer (drill peer call, 1 s deadline) before it drops anything, and the heartbeat's drill_id is the fallback. A new heartbeat flag drills tells an older peer (missing) from one that recorded nothing.
• Returned: checked from the drilled node's seen table (to the second) 15 minutes after the end.
• Seen table size: each entry keeps 7 daily counts for the affinity view, about 70 bytes rather than the design's 40 (under 300 KB at the 4,096 cap).
• Markers: drill.started and drill.ended are new marker kinds; a drill's maintenance window leaves no maintenance.* markers of its own.
• Test hooks: TELLTALE_DRILL_GAP_SECS, TELLTALE_DRILL_RETURN_SECS, and TELLTALE_DRILL_RETRY_SECS shorten the 10-minute gap, the 15-minute check, and the 5-minute retry for the cluster e2e. TELLTALE_DRILL_IGNORE_HEALTH passes the health check (shown as ignored in the checks): the e2e's nodes have no working upstream, so their health is severe.
• CLI: telltale ctl drill start|stop|report over the API (the design's local telltale drill subcommands are the same calls; only ctl is built).

Owner decisions (2026-10-19, the open questions):

• Scheduled drills: yes. [drill] schedule = "monthly" with nodes (the nodes that drill themselves), the targets (group or clients, else everyone), for_secs (default 30), hour (default 3) and tz, and mode. The day is in the first week, picked from the node's label and the month (FNV-1a), so it moves from month to month; the node tries once a minute through the hour, skips the month if a precondition fails all hour, and never drills within 7 days of another drill. The report goes out through a new one-off alert kind drill_report (any drill, scheduled or not, reports ended in the last 2 hours) to whatever destinations the rule names, email included. Shared configuration, so nodes says where it runs.
• REFUSED mode: yes. mode: "refused" (API, form, MCP, ctl --refused, [drill] mode): the node answers REFUSED with extended error 14 (Not Ready, "failover drill") at once, still counted as drilled. Drop stays the default; the preview says the REFUSED drill is faster but less realistic.
• The live Pi: yes, laptop first for 30 s, then a group, then everyone.

After the first live drills (2026-10-19, owner go):

• What ran: the laptop was skipped: on Wi-Fi its only DNS server is the homelab's 192.168.5.112, so the Pi never sees it (only the agent's own test lookups). The IOT group for 30 s; then everyone except the homelab server 192.168.5.2, for 30 s. The homelab server was left out because its only DNS server is the Pi and k3s's CoreDNS forwards through it, so dropping it would cut external DNS for the host and its pods.
• Two verdict rules were too loose, and are tightened (owner yes): an answer elsewhere counts as following only by 5 s after the end (FOLLOW_GRACE_MS): after that the drilled node answers again, so it proves nothing (the IOT drill had a "48.6 s failover" in a 30 s drill). And a target with nothing dropped is quiet, not followed, even when it asked another node (the second drill dropped nothing at all, yet said 5 followed and 100 % coverage). A device that asked the other node both before and after its first drop still follows, with no time. The windows still record 60 s after the end.
• A drill on a resolver pod (owner go): the pod left the EndpointSlice about 14 s after the start (balancerSwitchMs 12,652; 3 of the first 13 probes on .112 timed out, none after). Two changes (owner yes): the chart's readiness probe is every 2 s with 2 failures (about 4 s, for maintenance and real failures too), and the primary relays: a node reaches only the peers it has a stream to (a resolver pod only the primary; the Pi no resolver pod), so the report had missed every node but the drilled one and the primary, on the Pi's drills too. A new peer call relay on the primary passes a window notice or a read (part, list, report, active, seen) on to its peers that the caller didn't reach, and answers with theirs; never a start or end, never a relay. A node that learns of a drill this way has no heartbeat from the drilled node, so its window runs to the drill's end (the cutoff comes from the drilled node's own end), and it holds the one-drill-at-a-time rule until then plus the gap. An older primary refuses the call: the report is as before.

Requirements CLU-013 CLU-014 FLT-017 OBS-022 OPS-010

Builds on ADR-118

Link to this decision · In the specification