For the technical audience
Decisions ADR-051 to ADR-100
Every architecture decision record (ADR) behind TelltaleDNS: what we knew, what we chose, and what it costs. This page is built from the specification on every change, so it's always the current record.
ADR-051 Manual failover v1: roles and epochs, key sharing with eligible nodes, fencing, orphaned versionsT5.4 implements CLU-005 under spec/12 §5 and ADR-048.
Context
T5.4 implements CLU-005 under spec/12 §5 and ADR-048. The owner's topology is two eligible nodes and no witness, so the manual mode comes first: T5.4 is split into T5.4a (manual promotion, fencing, conflicts, config authority) and T5.4b (witness and quorum election with leases, plus the telltale-sim partition checker). T5.1–T5.2 tied "primary" to "holds the CA key", which makes promotion impossible.
Decision
- Role is explicit:
cluster.jsongainsrole(primaryorreplica) and theepochthe role belongs to.- A node publishes only while it's
primaryin the highest epoch it has seen. - Every Hello, heartbeat and manifest carries the sender's epoch.
- Signed member registry: each manifest carries the node registry (ID, site, eligible, advertise URLs), signed with everything else. The primary records nodes as they join (
nodes.json). Every node then knows every eligible node and its URLs, so replicas can find a new primary without a new token. - Eligible nodes hold the signing key (
spec/12§4).- The primary sends the CA key over the mTLS stream to peers that the registry marks eligible and that are connected with a certificate for that node ID.
- It's stored owner-only like the primary's.
- Non-eligible nodes (Kubernetes resolver pods, CLU-009) never receive it.
- A promoted node can then sign manifests and issue certificates. Compromising an eligible node compromises the cluster, which is the same trust the spec gives eligible nodes. Rotation comes with the CA rotation in T5.4b.
- Who dials whom:
- Replicas dial the current primary: the highest-epoch node known as primary first, then the other eligible URLs.
- Eligible nodes also keep a stream to every other eligible node, so a returning old primary learns about a newer epoch right away.
- Manual promotion:
- How:
telltale cluster promoteon an eligible node, or the Cluster page's "Promote this node" (admin; T5.7 adds TOTP confirmation). - What it does: sets
epoch = max seen + 1androle = primary, then starts publishing at once without a restart. The first manifest of the new epoch records its base: the(epoch, seq)it had applied. - Refused when:
- the node isn't eligible;
- it lacks the key;
- it's not authority-matching under ADR-048: a
gitopscluster needsconfig_source = "gitops"on the node, unless--emergency.
- Emergency primaries coordinate but publish no new configuration: every node keeps the last authoritative version.
- How:
- Fencing: a node that sees a higher epoch (in a Hello, heartbeat or manifest) stops publishing and becomes a replica of that epoch's primary at once. Snapshots and manifests from a lower epoch than a replica's applied one are rejected.
- Orphaned versions (
spec/12§5 rule 3):- When a fenced old primary had published versions in its epoch after the new primary's base, it keeps the last one under
cluster/conflicts/(manifest, config blob, and the changed paths compared with the new primary's version). - It shows them on the Cluster page under Conflicts, and in
GET /api/v1/cluster. They are never applied silently. - Re-applying a conflict on the current primary arrives with write forwarding (T5.7); until then the page shows what changed, and the owner re-makes the change on the primary.
- When a fenced old primary had published versions in its epoch after the new primary's base, it keeps the last one under
- Config authority (ADR-048):
cluster init --config-authority gitops|apiis recorded in the signed registry (defaultapi).[cluster] config_source = "gitops" | "file"is node-local (the Helm chart setsgitops), reported in Hello, and shown per node on the Cluster page.- Under
gitopsauthority, API config writes return409 gitops_managedon every node.
Consequences
- With the owner's homelab node as the GitOps primary, the Pi becomes an emergency primary candidate: if k8s is down for long, promoting the Pi keeps the cluster coordinated while configuration stays frozen. ADR-049 (Git source) later makes the Pi a full candidate.
- Existing clusters migrate in place: the CA holder becomes
role = primaryat epoch 1, and eligible replicas receive the key on their next connection.
Amended 2026-10-08 (review 05-01, 05-02; amendment Proposed): standing is the primary's decision, not the member's. A join with the bootstrap secret (ADR-058) is always an ephemeral member, never eligible or a witness, so the key share never reaches a resolver pod; a Hello can't raise a known member's eligibility (a member the registry doesn't know is still taken at its word once). Fencing by epoch (Hello, heartbeat, and the election's view of the highest epoch) counts only voters in the signed registry, so a member that can't vote can't make the primary step down with an unsigned frame. A stricter follow-up would fence only on a verified, signed manifest.
Requirements CLU-005 CLU-009
ADR-052 Node-local overrides: node-only records, and warn (not refuse) on shared settings in a replica's fileT5.5 (CLU-006).
Context
T5.5 (CLU-006). spec/12 §4 says a node's own overrides allow listen addresses, cache size, query-log retention, worker count, site, and local-only records, and that overriding anything else is "rejected at startup with a clear error". ADR-047 already makes the node-local sections (node, cluster, listen, api, auth, telemetry, cache) stay with each node, and replaces everything else with the primary's.
Decision
- Node-only records:
[[record]] node_only = truekeeps a record on the node whose file has it. A primary leaves it out of the shared configuration, and a replica keeps its own node-only records next to the cluster's. - Shared settings in a replica's file are ignored and reported, not refused.
- At every load the replica logs a warning naming the sections.
- The Cluster page gains a failing
node_settingscheck with the fix (remove them, or mark node-only records). - The spec's "reject at startup" would make a node's upgrade, or a cluster join, refuse to start over settings that have no effect. DNS would stop over a cosmetic problem, against rule 5 and CLU-004. The owner's Pi is exactly this case: its file predates joining.
- No separate
node.tomlfile is needed: a node's own config file plays that role, and the node-local sections are already defined by ADR-047.
Consequences
the owner's Pi shows a failing node_settings check until its file is trimmed to node-local settings. Promoting to the spec's strict refusal stays possible later, as an opt-in [cluster] strict_overrides = true.
Requirements CLU-004 CLU-006
Builds on ADR-047
ADR-053 Federated reads v1: what's exact, what's approximate, and which scopes existT5.6 (CLU-002, OBS-012).
Context
T5.6 (CLU-002, OBS-012). spec/12 §6 says analytics reads fan out to the peers in scope and merge, with HDR histograms merged exactly, query logs k-way merged with per-node cursors, a 2 s per-peer timeout, and missing_nodes. It names the scopes node:<id>, site:<name> and cluster.
Decision
- Transport: reads are RPCs over the existing cluster streams (frame types
RpcRequest/RpcResponse, kindapi.read, JSON body). No new port, connection, or certificate. Peers are asked concurrently with a 2 s deadline. A peer without a stream is reported missing at once. - Merges:
- counters sum per bucket, across every breakdown;
- top lists add counts and error bounds per key (the true count stays within
[count − errorBound, count]); - query-log pages interleave newest first, with a cursor (
fed:+ base64url JSON) that holds each node's "before" time, or nothing once that node has run out.
- Latency is approximate in v1. Each node sends its percentiles, and they are weighted by query count; the maximum is exact. Shipping histograms for an exact merge needs a serialized HDR format on the wire. It's deferred until a dashboard needs exact cluster-wide tails.
- Scopes:
cluster(the default) andnode:local.node:<id>andsite:<name>still answer400 unsupported_scope. The live tail, configuration reads, and Explain stay local. - Edge cases (accepted):
- Rows that share the exact microsecond with a page's last row from the same node may be skipped on the next page.
- A node that joins while someone is paging appears only from the next fresh search.
missingNodesreports the latest federated read on the node; concurrent reads may each see the other's list.
- Failure: a peer error, timeout, or undecodable answer drops that node from the result and names it in
missingNodes. If this node's own query log is off, but peers answered, their rows are returned and this node is listed as missing. DNS answering is never involved (CLU-004).
Consequences
any node's UI is a full management view for reads. Exact cluster-wide latency, named-node/site scopes and a federated live tail are follow-ups.
Requirements CLU-002 CLU-004 OBS-012
ADR-054 Write forwarding v1: the entry node vouches for the user over mTLST5.7 (CLU-002).
Context
T5.7 (CLU-002). spec/12 §6 says a write on any node is forwarded to the primary with the user's identity, "signed by the receiving node", so the audit log shows both the user and the entry node. Users aren't replicated yet (ADR-045 is still to come), so a user exists only on the node they signed in to.
Decision
- What's forwarded: the configuration writes the API offers (devices, local names, forwarded domains), including dry runs. They go to the primary as the RPC
api.writeover the existing cluster stream, with a 10 s deadline. - Who vouches:
- The entry node authenticates and authorizes the user with its own users and roles, exactly as for a local write.
- The primary trusts the entry node's claim because the stream is mutually authenticated with the cluster CA: the peer ID comes from the certificate, never from the request.
- No extra signature is added: the mTLS channel already proves which node sent it, and a member node is trusted to replicate configuration anyway.
- What's recorded:
- The primary stores the entry as made by
<user> via <site>and audits it with actor kindclusterandremote = node <id>. - The entry node audits it under the user's name, with the primary's answer.
- The primary stores the entry as made by
- Versions: a replica reports the primary's configuration version (ETag; 1 s deadline, falling back to its own), so
If-Matchworks through any node. The primary's answer, including its error code (409, 404, 400), is returned unchanged. - Unreachable primary: the write is refused with 503 and a hint. It is never queued: a queued write could apply long after the user saw it fail.
- Not forwarded: under a GitOps authority, writes are refused on every node (409
gitops_managed, ADR-048). Promotion and the node's own users, tokens and sessions stay local until ADR-045 replicates identities.
Consequences
the owner can manage the cluster from the Pi's UI as well as the homelab node's, as long as both are on the API authority. Today's cluster is GitOps-managed, so there it stays read-only, by design.
Amended 2026-10-08 (review 05-03; amendment Proposed): "a member node is trusted to replicate configuration anyway" no longer covers every member since ADR-058. Forwarded writes and the reads that change the answering node (pause, resume, cache flush) are refused from members the registry marks ephemeral: they serve no UI, so no user's change arrives through them.
Requirements CLU-002
ADR-055 Query-log ship mode v1: closed segment files, delivered once, searched where they landT5.8 (CLU-007).
Context
T5.8 (CLU-007). spec/12 §7 describes ship mode: raw events are streamed to a target in batches of columnar blocks, spilled to a 64 MiB buffer when the target is unreachable, and replayed; rollups are shipped as per-minute aggregates; both mode does both; receivers keep per-source-node segments.
Decision
- Unit of shipping: whole closed segment parts, not blocks.
- In ship mode the writer closes a part at least every
[telemetry.ship] interval_secs(default 300 s). - The shipper sends each closed part (every part except the newest) to the target in 512 KiB chunks over the cluster channel (RPC
qlog.put). - The receiver checks the whole file's BLAKE3, then files it unchanged under
qlog-nodes/<sender-id>/. Nothing is re-encoded, and the sender's footer and bloom filters stay valid. - Only after the receiver confirms does the sender delete its copy. A lost confirmation means the file is sent again and overwritten, never duplicated.
- In ship mode the writer closes a part at least every
- Buffer: the local log is the store-and-forward buffer, bounded by
buffer_bytes(default 64 MiB) via the existing retention, which drops the oldest file first. No separate in-memory spill: a RAM disk underdata_dirgives the memory variant without new code. - Exactly one copy: a row is either in the sender's buffer or on the receiver. The receiver's search includes shipped logs, labelled by node, and federated reads reach the sender's unshipped rows, so a cluster-wide search sees each row once.
- Target:
[telemetry.ship] to(a node ID or site), defaulting to the primary. A primary in ship mode with no other target keeps its log (logged once). - Deferred:
- Rollup shipping. Per-minute counts stay on each node: they're small, the dashboard already federates them, and shipping them only matters for ephemeral pods (T5.10). Done in T9.3: ship-mode nodes send their last 30 minutes of minutes every
interval_secs(≤ 60 s); the target keeps them per node for 7 days (shipped_minute), and the cluster view adds only those of nodes that didn't answer live, so nothing counts twice. bothmode. It would need dedup between a live sender and its shipped copy. Not needed for the hybrid Pi + k8s setup.
- Rollup shipping. Per-minute counts stay on each node: they're small, the dashboard already federates them, and shipping them only matters for ephemeral pods (T5.10). Done in T9.3: ship-mode nodes send their last 30 minutes of minutes every
- Safety:
- Receivers accept only from cluster members (mTLS), only hex node IDs become paths, and segments over 256 MiB are refused.
- Shipping is a background task, and receiving runs on blocking threads, so DNS is never involved (CLU-004).
- The peer RPC handler is now installed whether or not the node runs the API, so a node without an API can still receive logs and answer federated reads.
Consequences
the owner's Pi can set mode = "ship" to move its query log to the homelab node's volume. Rows then appear with up to interval_secs of delay in the homelab node's own search; cluster-wide search shows them at once.
Requirements CLU-004 CLU-007
ADR-056 Automatic failover v1: one vote per epoch, voter-granted leases, pre-vote, a vote-only witnessT5.4b (CLU-005).
Context
T5.4b (CLU-005). spec/12 §5 has witness and quorum modes: a candidate needs a majority of eligible nodes plus witness for epoch + 1, and promotes only after the old primary's lease (15 s, renewed every 5 s) has expired; "a primary accepts writes only while it holds a valid lease". The AC is a simulator: 10k randomized partition schedules, never two writers in one epoch, orphaned writes always surfaced.
Decision
- One protocol for both modes (
telltale_cluster::election, pure, no I/O):- Voters are the eligible nodes plus witnesses, from the signed registry.
- A voter grants each epoch once (persisted, fsync, before answering). It refuses a newer epoch while the lease it granted to another node runs (15 s on its own clock), and refuses candidates whose applied version is behind its own.
- The primary renews with every voter every 5 s. It treats its lease as ending 15 s − 2 s after it sent the round a majority granted, so it stops publishing before any voter of that majority can elect someone else, for clock rates within ±1 %.
- Pre-vote: a node first asks whether it would win. A node cut off from the cluster never inflates epochs, and never deposes a healthy primary when the partition heals.
witnessandquorumare the same thing with different voters, so the mode is justmanualorauto.
- Mode:
telltale cluster set-failover auto|manualon the primary. It's carried in the signed manifest, like the authority.autoruns elections only with three or more voters, this node one of them; otherwise it behaves asmanual, and the Cluster page'sfailovercheck says why.- Manual
promoteis refused while elections run, because it would bypass the votes.
- Witness:
telltale cluster join <token> --witness --advertise <url>, thentelltale cluster witness.- It runs only the cluster channel and the vote handler: no DNS, lists, API or pipeline. It learns the registry and mode from verified manifests.
- It never becomes primary and never receives the CA key.
- Transport:
- Votes are the
elect.askRPC on cluster streams. The candidate is bound to the stream's certificate: a node can only ask for itself. - Voters keep a full mesh in
automode: of each pair, the lower node ID dials.
- Votes are the
- Gating:
Cluster::may_publish()is false for a primary without a valid lease. The publish loop waits, and API writes get 503 with a hint.- A primary starting in
automode begins with publishing paused until its first renewal. - A restarted primary resumes its epoch if its own ballot is for itself; otherwise it takes part in a normal election.
- Only nodes holding the CA key stand as candidates.
- Under a
gitopsauthority, an elected node that isn't Git-managed becomes an emergency primary (ADR-048).
Verification
tests/sim.rscovers 2 + witness, 3, and 4 + witness voter sets under random asymmetric partitions, 0–15 % loss, heavy-tailed delays to 3 s, crashes with persisted ballots and histories, and ±1 % clock drift.- It checks:
- one primary per epoch;
- never two writable primaries at the same real instant;
- every write kept, or reported as orphaned;
- a primary within 60 s of healing.
- Results at 10,000 schedules (release, in CI): 43,744 elections, 1.49 M writes, 176,933 orphaned and reported, worst recovery 24.75 s.
- It's mutation-checked: removing the lease check, or timing the lease from the reply instead of the request, is caught.
deploy/cluster/failover-e2e.sh(real processes, CI): kill the primary, and the replica is elected in about 14 s with 100 % of DNS queries answered; the old primary returns and follows.
Deferred
CA rotation, the other part of T5.4b's title, is now T5.4c. Clocks are assumed not to jump by more than the margin; NTP slews rather than steps in normal operation.
Amended 2026-10-08 (review 05-05; amendment Proposed): the clock assumption above didn't hold for the hosts this runs on. A Raspberry Pi has no RTC: it starts on the time it shut down with, and NTP steps it, by minutes or hours, once DNS works. With leases as wall-clock times, a forward step made every lease a voter had granted look expired at once. So:
- Leases are timed on the election clock (
telltale_cluster::clock):CLOCK_BOOTTIMEon Linux (read throughrustix, nounsafe), which never steps and counts time spent suspended;Instantelsewhere. Nothing on the wire changes:AskandReplycarry no times. - A ballot records which process's clock its lease is on (
clock). A ballot from another process (a restart) or from an older build (wall clock,clock = 0) is taken as granting a full lease from now (Ballot::adopt), and stored that way at once so later reads don't extend it again. That is never shorter than promised, and costs at most 15 s of extra waiting after a voter restarts. - The simulator restarts nodes on a new, unrelated clock (anywhere from far behind to far ahead) and adopts their ballots the same way. Mutation-checked: forgetting the lease on adoption is caught ("two writers") within 2,000 schedules.
Consequences
the owner's Pi + homelab cluster stays manual until a witness exists. A witness on any third device (for example the NAS) makes failover automatic, with the Pi as an emergency primary while the configuration authority is GitOps.
Requirements CLU-005
Builds on ADR-048
ADR-057 Node certificate renewal over the cluster channel, same keyT5.4c (CLU-001).
Context
T5.4c (CLU-001). Node certificates last 90 days (ADR-044), and nothing renewed them, so a cluster would lose its links about 90 days after its nodes joined. The owner's cluster joined on 2026-10-04. spec/12 §3 says certificates renew automatically; the CA rotation half of T5.4c is rarer, because the CA lasts 10 years.
Decision
- When: each node checks every 6 hours (first one minute after start) and renews when fewer than 30 days remain, so two thirds of the lifetime has been used. It retries every 10 minutes on failure.
- Same key: the node sends a CSR for its existing key, so its node ID (derived from the key) never changes.
- Who signs:
- A node holding the cluster key (the primary, eligible nodes) signs its own.
- Others ask the primary with the
cert.renewRPC on their stream. - The primary signs only when the CSR's key belongs to the peer the stream's certificate names. It takes the certificate's names (advertise hosts) from the signed registry, never from the request.
- Checks before use: the new certificate must name this node and verify against the cluster CA (Ed25519). It's written atomically.
- Switch-over: a certificate generation counter makes the dialer and the listener rebuild their TLS settings for the next connection. Open streams keep their session, since certificates are checked at handshake only.
- CA rotation (new CA, cross-signing, key re-share) stays in T5.4c, still open.
Consequences
clusters keep their links indefinitely while a CA holder is reachable at least once every 30 days. A witness or replica cut off from every CA holder for 90 days loses its certificate and must rejoin with a token.
Requirements CLU-001
Builds on ADR-044
Referenced by ADR-066
ADR-058 Kubernetes resolver pods: a shared bootstrap secret with proof of possession, ephemeral members, sync-gated readinessT5.10 (CLU-009).
Context
T5.10 (CLU-009). spec/12 §2 says resolver pods join with a long-lived token from a Secret, get a fresh identity per start, expire after ephemeral_ttl (10 min) without heartbeats, and display under their site. A join token pins the CA fingerprint, which only exists after the controller creates the cluster. So a chart can't put a token in a Secret without the controller writing to the Kubernetes API, which needs RBAC and a client library.
Decision
- Bootstrap secret instead of a pre-made token:
- The chart generates a random secret in a Secret (kept with
lookupandresource-policy: keep, or the user's own viaexistingSecret), mounted in every pod. - The controller accepts it like a join token that never expires:
[cluster] bootstrap_secret_file, re-read at every join, so rotations apply without a restart.
- The chart generates a random secret in a Secret (kept with
- Proof of possession instead of trust on first use:
- A pod with
join_urlfirst callsPOST /cluster/v1/cawith a fresh nonce. - It trusts the returned CA only if the server answers HMAC-SHA256(secret, nonce : CA fingerprint), proving it holds the same secret.
- It then pins that CA and joins exactly as with a token.
- A man in the middle without the secret can't pass this.
- A pod with
- First start creates or joins:
[cluster.init](advertise URLs, authority) creates the cluster on the controller's first start, the same astelltale cluster init.join_urljoins on a pod's first start, retrying for 2 minutes. An ephemeral pod that can't join exits, so Kubernetes retries it (CrashLoopBackOff while the controller starts).
- Ephemeral members:
[cluster] ephemeral = truejoins as ephemeral: never eligible, never a voter, no CA key.- The registry records
ephemeral. The primary drops ephemeral members not connected and not heard from forephemeral_ttl_secs, checked every minute. A pod that never connected is measured from when it joined, or from the controller's start. - The Cluster page groups them by site with counts (up, ready, q/s, behind).
- Readiness: an ephemeral node is ready only once it has applied the cluster's configuration. A new pod never serves an unfiltered answer, and the Service routes to it only after sync.
- Helm
mode: scaled:- The allInOne StatefulSet becomes the controller and keeps its labels, so switching modes needs no reinstall.
- Adds a resolver Deployment (
emptyDirdata, ship-mode query log,maxUnavailable: 0), a-clusterService for the cluster port, the join Secret, and a NetworkPolicy rule for the cluster port. - The DNS Service selects every pod; the API and metrics Services select the controller.
hostNetworkwithscaledis refused, since pods would compete for port 53; that'sdaemonSet, later.
- Found on the way: a replica's follower ignored a manifest that arrived before it subscribed, and waited for the primary's next change. It now processes the current one at start.
Consequences
deploy/helm/scaled-e2e.sh(CI, this build's binary on kind) proves the controller and two pods join and sync, a pod answers DNS, an outside node joins through the cluster port, and a deleted pod is replaced and expires.- A pod restart is a new member, by design. A registry of long-gone pods is cleaned within
ttl + 1 min.
Amended 2026-10-08 (review 05-01, 05-06; amendment Proposed): what makes a pod untrusted is now enforced by the primary: a join with the bootstrap secret is ephemeral whatever it asks for. The primary's POST /cluster/v1/ca proof answers anyone, so a weak secret could be brute-forced offline from collected proofs: a secret shorter than 32 bytes is warned about at startup (not refused, so existing deployments keep joining), and the docs ask for 32 random bytes or more. The chart generates 48 characters.
Requirements CLU-009
ADR-059 Version compatibility v1: a protocol window, a schema stamp, and refusal over guessingT5.11 (CLU-010).
Context
T5.11 (CLU-010). spec/12 §9 says RPC carries a protocol version and manifests a schema version; nodes accept N and N−1; the primary refuses to emit features the oldest connected node doesn't support, and warns. Until now, peers needed the exact same protocol, and manifests carried no schema. The shared configuration is parsed strictly (deny_unknown_fields), so an older replica would reject a newer primary's new settings.
Decision
- Protocol window: a stream is accepted when the peers' protocols differ by at most one. Frames are protobuf, and fields are only ever added, so unknown fields are ignored.
PROTOCOLis raised only when a message changes meaning. - Schema stamp: manifests carry
schema, the configuration schema the primary writes (now 1; 0 means from before stamping). It's raised when the shared configuration gains settings an older build would reject. - No down-converting: the primary does not try to strip settings an older replica wouldn't know. It publishes what it has.
- A replica that can't read it keeps serving its last version, so DNS is unaffected.
- The replica records a sync error naming both schemas and saying to upgrade.
- Per-field feature gating is deferred: the documented upgrade order (replicas first) makes it rarely matter, and guessing which settings are safe to drop could silently change filtering.
- Visibility: each node's protocol is on the Cluster page, and a
versionscheck fails while versions or protocols are mixed, with the upgrade order as the fix. Build versions become fully distinguishable with T6.9's stamped build identity; today edge builds all say 0.1.0.
Verification
deploy/cluster/upgrade-e2e.sh (CI) runs the published edge binary as N−1 against this build:
- both nodes on N−1;
- the replica on N following the N−1 primary;
- the primary on N;
- the replica back on N−1 following the N primary.
An edit reaches the replica at every stage, and a client with both servers configured gets an answer to every query.
Addendum (2026-10-05): the shared configuration leaves out top-level sections that are at their defaults, and a replica reads a missing section as the default. Without this, adding any new section (DNSSEC, ADR-060) broke N−1 replicas even when no one used it: CI's upgrade test caught it. A new setting now reaches an older replica only once someone actually sets it. That is the case the "no down-converting" rule is for.
Requirements CLU-010
Builds on ADR-060
Referenced by ADR-064
ADR-060 DNSSEC validation v1: hickory's validator over our upstream groups, off by defaultT6.1 (DNS-011).
Context
T6.1 (DNS-011). spec/03 §5 asks for:
- modes
off,validate(the default once stable) andvalidate_permissive; - DO=1 upstream, validation from the root trust anchor, and CD=1 clients getting unvalidated data;
- AD only when validated and the client asked;
- bogus answers as SERVFAIL with EDE 6;
- negative trust anchors;
hickory-proto's verification primitives behind aValidator.
In hickory 0.26 the chain-of-trust logic (DnssecDnsHandle, with NSEC/NSEC3 denial proofs, wildcard checks, NSEC3 iteration limits and a validation cache) lives in hickory-net. The primitives in hickory-proto alone would mean writing that ourselves.
Decision
- Use
hickory-net'sDnssecDnsHandle(dnssec-ring,tokio; it adds no TLS or HTTP stacks). It wraps aDnsHandlewe implement over one of our upstream groups.- The question, and every DNSKEY and DS lookup the proof needs, go to the same upstreams with DO=1 and CD=1 (we check, so the upstream mustn't drop what it thinks is bogus).
- There's one handle per group, so its validation cache lasts across queries.
- It's wrapped as
telltale_upstream::dnssec::Validator.
- Verdict: the weakest proof among the answer records, or among the authority records for NXDOMAIN and NODATA. Errors from the validator other than upstream failures count as bogus.
- Serving:
- Secure gets AD for clients that asked; insecure and indeterminate get no AD.
- Bogus becomes SERVFAIL with EDE 6, never a stale answer, and is never cached; in permissive mode it's served and counted.
- RRSIG, NSEC and NSEC3 records are stripped for clients without DO.
- Validation runs on cache misses only, and validated answers are cached with their AD bit. The cache key already separates DO and CD.
- Negative trust anchors:
[dnssec] negative_trust_anchorsplus every route withdnssec_nta. - Mode names and default:
off,validate,permissive(spec'svalidate_permissive). The default staysoffuntil it has run on the owner's nodes for a while; then a follow-up makesvalidatethe default. - Timing: a cold chain (root, TLD and zone keys) takes several sequential lookups. Validation gets three times the query budget, so a slow first lookup still lands in the cache even if that client gave up.
- Found on the way (all queries): a truncated UDP answer's TCP retry shared the adaptive per-attempt timeout calibrated on small answers (about 3× average round trip), so large answers (DNSKEY sets with signatures, long TXT) timed out. The TCP retry now gets the configured timeout of its own.
Verification
unit tests for verdicts, NTA coverage and stripping. deploy/dnssec-e2e.sh (CI, real DNS tree, forwarding to 1.1.1.1 and 9.9.9.10) checks:
- ietf.org and cloudflare.com are secure with AD;
- google.com is insecure without AD;
- dnssec-failed.org is SERVFAIL with EDE 6, and answered with CD;
- signed NXDOMAIN and NODATA validate;
- verdicts are counted.
Deferred
- RFC 5011 trust-anchor updates (the built-in root keys cover KSK-2017 and KSK-2024);
- aggressive NSEC caching (RFC 8198, with recursion);
- per-reason EDE codes (7, 9, 10, 12) instead of 6;
- the
dnssecfield in query events and the Settings UI.
Requirements DNS-011
Referenced by ADR-059
ADR-061 Pi-hole import v1: a reviewed starting configuration, faithful group semanticsT6.3 (API-007; spec/08 §8) asks for an importer of Pi-hole v5 and v6 Teleporter archives covering adlists, domain lists, groups, clients, local DNS/CNAME records, upstreams, and conditional forwarding, with a report of every unmapped setting.
Context
T6.3 (API-007; spec/08 §8) asks for an importer of Pi-hole v5 and v6 Teleporter archives covering adlists, domain lists, groups, clients, local DNS/CNAME records, upstreams, and conditional forwarding, with a report of every unmapped setting. The spec doesn't say whether the result is applied or reviewed, how Pi-hole's per-entry group assignments map onto our per-group list sets, or what happens to Pi-hole behaviors we don't have.
Decision
- Output, not apply:
telltale import pihole PATHwrites TOML for review (likeimport zone, ADR-041), validated to load before it's written. Applying through the API (and the UI's import page) waits for T6.7's restore path. - Inputs: v6 zip (
pihole.tomland agravity.dbof the group tables), v5 tar.gz (JSON per table plussetupVars.conf,pihole-FTL.conf,custom.list,dnsmasq.d), a directory, or a baregravity.db. Readers are ours: zip and tar.gz in about 200 lines onminiz_oxide, with CRC checks and size limits. SQLite goes throughrusqlite, already in the tree. Archived databases are read from a private temporary copy. - Groups stay exact: each group gets exactly the lists Pi-hole gave it, so
defaultdoesn't fall back to "every list". Domain entries become inline lists, one per (type, set of groups). Disabled groups get no lists. Clients in no group go to apihole-no-groupgroup with no lists, as Pi-hole blocks nothing for them. - Matching:
- exact entries use
match = "exact"; - a regex our parser would read as a plain name is wrapped as
/regex/; - entries our engine can't run are reported, not dropped silently.
- exact entries use
- Upstreams: they go in the
defaultupstream group with strategyfastest, matching Pi-hole's preference for the fastest server. - Conditional forwarding: each rev server becomes its own upstream group and route. The route covers the domain and the network's reverse zones: non-octet prefixes are expanded, so a /23 is two /24 zones, never widened. It gets a negative trust anchor.
- Devices: DHCP reservations become named devices (MAC and IP join a matching client, or a new one named by the host name). The DHCP server is not imported.
- Report:
- v6: every setting Pi-hole marks
### CHANGEDthat isn't mapped; - v5: every unmapped key in
setupVars.confandpihole-FTL.conf, except the old install's own plumbing (interface, addresses, web server); - names only, never values, since they include password hashes.
- v6: every setting Pi-hole marks
- Not imported: clients matched by host name or interface (we match by address); the
IPblocking modes; query history; passwords and tokens.
Verification
- Unit tests on hand-written inputs in both formats.
deploy/pihole-import-e2e.sh(CI) configures the officialpihole/piholeimages (v6 2026.09.0 = FTL v6.7.1, v5 2024.07.0) through their own tools, exports Teleporter archives, imports them, and serves the result: local names, a CNAME's TTL, exact and regex denies (;querytype), the blocking mode, and group scoping.- Pi-hole-generated files aren't committed (their comments are Pi-hole's text, NFR-006).
Consequences
- Merging into an existing configuration is manual (both define
default). - The importer's e2e showed that the first blocking snapshot waited for every list's first fetch, including retries, so an unreachable adlist delayed all blocking at first start. Fixed: the fetcher signals the compiler at most 10 s (
settle) after the first update in a round, so lists that arrive together still compile once.
Requirements API-007 NFR-006
Builds on ADR-041
Referenced by ADR-062
ADR-062 Technitium import v1: read the API, not the backupT6.4 (API-007, P1).
Context
T6.4 (API-007, P1). spec/08 §8 says "backup zip → blocked/allowed zones, block-list URLs, forwarders + protocols, Advanced Blocking app config → groups". Technitium's backup holds dns.config, zone files, and the allowed/blocked lists in its own versioned binary serialization. That format is undocumented and changes between releases. Its HTTP API returns the same data as documented JSON.
Decision
- Source:
telltale import technitium URLreads a running server through the API with an API token, which comes fromTECHNITIUM_TOKENor--token-file, never argv. - Calls, all read-only:
settings/get;zones/list, thenzones/records/getper zone;allowed/export,blocked/export;apps/list, andapps/config/getfor Advanced Blocking;dhcp/scopes/listanddhcp/scopes/get(optional).
- Output: reviewed TOML like ADR-061, validated before it's written.
- Mapping:
- Forwarders go in the
defaultgroup: concurrent forwarding becomesparallelwith the same fan-out, but only with two or more usable forwarders.name (ip:port)pins the address, with the name as TLS name. - Forwarder zones: forwarders by priority, a route, and an NTA unless every forwarder of the zone validated.
- Primary zones: records of the types we serve, with their TTLs.
- Blocked and allowed zones become subtree inline lists.
- Advanced Blocking groups become groups. The catch-all network's group becomes
default, and other networks becomenetworks(ADR-050). URL lists are shared across groups by URL. Regexes are wrapped as/re/. Server-wide lists are added to every group, since Technitium applies both. Disabled groups get no lists. The bypass list becomes a group with no lists. - Blocking answer: NXDOMAIN, or custom addresses, become the group's
block_mode. - Rate limit: the per-/32 limit (queries per minute) becomes
[ratelimit]. - Other settings:
dnssecValidationandsaveCachecarry over. DHCP reservations become named devices.
- Forwarders go in the
- Not imported (reported): secondary and stub zones; QUIC forwarders; forwarders with no address; listener- and host-based group selection; per-list answers; the DHCP server; encrypted listeners; recursion ACLs, TSIG, zone transfers, the proxy, and other apps. Listeners, logging, users, and tokens are not imported either.
Verification
- Unit tests on hand-written API responses.
deploy/technitium-import-e2e.sh(CI) runs againsttechnitium/dns-server:15.6.0. It configures everything above through Technitium's API, installs Advanced Blocking from Technitium's store, and creates an API token. It then imports, serves the result, and checks the answers.
Consequences
- Importing needs the old server running, which matches the migration path (import, compare, switch).
- A backup-zip reader can follow if the format gets documented.
- The e2e found a real-data bug: concurrent forwarding with a single usable forwarder produced an invalid
parallelgroup. Fixed and tested.
Requirements API-007
ADR-063 Backups v1: a checked archive of config and local data, without the cluster keyT6.7 (API-007, P0) asks for Teleporter parity: one archive with config and local data, optionally with query history.
Context
T6.7 (API-007, P0) asks for Teleporter parity: one archive with config and local data, optionally with query history. spec/08 §8 sketches telltale ctl backup create [--include-qlog] writing a .ttbk (tar.zst + manifest + signature) restored onto a new primary. ctl (the API-wrapping CLI, API-008) doesn't exist yet. The spec doesn't say what the signature is signed with or whether cluster identity is included.
Decision
- Commands:
telltale backup create | show | restore, local commands that work on the data directory.createis safe while the server runs: SQLite databases are copied consistently withVACUUM INTO, and query-log segments only grow, so the size seen at open is copied.
- Contents:
- config files under
config/; state.db(users, tokens, UI-made entries, audit), with sessions and idempotency keys removed;rollups.dbandanomaly.json;- the query log (
qlog/,qlog-nodes/) only with--include-qlog; - lists, snapshots, the cache, and
setup-tokenare left out, since they're rebuilt or transient.
- config files under
- Format: a tar stream compressed with zstd (frame checksum on), with
manifest.jsonlast. It holds the format version, source node and paths, and each entry's size and BLAKE3 hash. The file is written owner-only and renamed into place when complete. - No signature in v1: a signature by the source node's key proves nothing to the fresh node it's restored onto, which doesn't know that key. Integrity comes from the manifest hashes and zstd's checksum. Confidentiality is the file's permissions, as with Pi-hole's Teleporter. Encryption with a passphrase is the natural next step.
- No cluster identity by default:
cluster/, with the CA key, isn't included. A portable file with the CA key could impersonate the cluster. A restored member starts standalone, and the manifest records that it was a member so restore can say what to do. - Restore:
- streams into a staging directory under the data directory;
- checks every entry against the manifest, and anything unexpected, a path outside
config/ordata/, or.., fails before any file is moved; - refuses to replace existing files without
--force; - chowns restored files to the data directory's owner.
Verification
- Unit tests: contents, permissions, round trip, session stripping,
--force, corruption, path containment, and the query log. deploy/backup-e2e.sh(CI) backs up a running node with an admin and an API-made name, restores it onto a second node, and checks the following on that node:- the name answers;
- the admin signs in, and the old session doesn't carry over;
- the audit log is present;
- damaged archives are refused.
Consequences
- Download:
GET /api/v1/backup(admin, auditedbackup.create, without the query log) and a Settings → System button. Restore through the API is deferred: restoring under a running server is risky. --include-cluster-key(for disaster recovery of a lone primary) and passphrase encryption are follow-ups.
Requirements API-007 API-008
ADR-064 Agent principals v1: scoped tokens, deny by default, a reason for every changeT6.5 (AGT-001..005, AGT-009).
Context
T6.5 (AGT-001..005, AGT-009). spec/13 §2 asks for:
- agent tokens with an owner, scopes, an optional group restriction, rate limits, and expiry;
- attribution
agent:<token> (owner: <user>)with the client and a required reason; - dry runs with impact estimates on every mutation;
- guardrails and a cluster-wide kill switch;
- an OpenAPI document written for agents and linted.
Tokens so far had a role-like scope (read/write/admin), and the audit log already recorded token:<name> (owner: <user>) (ADR-038).
Decision
- Agent tokens:
POST /tokenswithkind: "agent".- The scopes are AGT-004's list:
analytics:read,querylog:read,config:read,config:write:{clients,records,forwards}(*for all),ops:pause,ops:cache,cluster:admin. - The default is
analytics:read+config:read: read-only, without the query log. - Optional
groupandratePerMinute. - Each scope needs a role from the creator (
config:write:*andops:*need operator,cluster:adminneeds admin). A token's effective role is the lower of its owner's and its scopes'. - Stored in three new
tokenscolumns (migration 5). User tokens are unchanged.
- The scopes are AGT-004's list:
- Deny by default: one table maps method and path to a scope. Anything not in it is refused for agents: users, tokens, passwords, backups, the audit log, and routes added later until they're mapped. The role checks still apply on top.
- Group restriction: honored where a group view exists.
- The query log and top lists: the middleware replaces the
groupparameter. - Devices: listed for the group only; writes only into, and only on devices already in, the group.
- Everything else is refused for restricted tokens rather than shown unfiltered.
- The query log and top lists: the middleware replaces the
- Every change needs a reason: agents' non-GET requests without
X-Telltale-Reasonget 400. - Attribution: the actor is
agent:<token> (owner: <user>) via <X-Telltale-Client>, with kindagent. - Rate limits: a per-token token bucket in memory, refilled per minute, default 120 (
[agents] rate_per_minute). Over the limit: 429 withRetry-After. Per node, so a cluster allows N× in total; documented. - Kill switch:
[agents] enabled, a shared section, so it replicates to every node (and, since ADR-059's addendum, doesn't trouble N−1 replicas while unset). It's applied on start and every reload. - Impact estimates: device, name, and forward changes return
recentQueriesandimpact, a sentence with the numbers. They're counted from the in-memory top lists for the current and previous hour (a lower bound, cheap, no log scan). Promotion gets?dryRun=true, returning aPromotePlan(epoch, emergency, impact) from the same checks without acting. - Lint:
.spectral.yaml(spectral:oas plus operation summary, description, tags, operationId, success response, and parameter descriptions) runs in CI. Every response now has a description.
Verification
- Unit tests:
- the scope map denies by default;
- scope parsing and the roles scopes imply;
- the kill switch, rate limit, required reason, and group refusal;
- the forced group parameter.
- API tests: agent tokens over HTTP (scopes, refusals, reason, attribution, kill switch); group-restricted tokens; the promote dry run.
deploy/agent-e2e.sh(CI) on a running node:- a dry run of every mutation reports an impact and changes nothing;
- a real change is attributed with its reason;
- a reload with
enabled = falsestops the agent and not the admin.
Consequences
- MCP (T6.6) builds on these principals.
ops:pauseandops:cacheexist ahead of their endpoints.- Per-group analytics views beyond top lists would let restricted agents see more.
- OAuth for agents (AGT-008) stays P1.
Amended 2026-10-08 (review 06-02, 06-09; amendment Proposed): an agent's pre-save list check returns how many lines couldn't be used and why, not their text (the node would otherwise read any URL it can reach for the agent). Viewers list only their own plans; operators and admins see all.
Requirements AGT-001 AGT-004 AGT-008 AGT-009
ADR-065 MCP server v1: read-only tools over the REST routes, one implementation for two transportsT6.6 (AGT-006, AGT-008 bearer part, AGT-009; ADR-010).
Context
T6.6 (AGT-006, AGT-008 bearer part, AGT-009; ADR-010). spec/13 §3.1 lists the read-only tools, and §4 suggests a telltale-mcp crate with schemas generated from the Rust types (schemars) and a catalog snapshot test. The AC asks for:
- an MCP conformance test;
- scope and privacy authorization tests;
- a scripted agent completing the "why is the TV slow" story.
Decision
- Where: a module of
telltale-api(mcp.rs), mounted at/mcpon every node's API listener behind the same authentication. A separate crate would add nothing at this size.POST /mcptakes JSON-RPC 2.0 (single or batch) and answers JSON (Streamable HTTP without SSE;GET /mcpis 405). Protocol2025-06-18, also answering2025-03-26and2024-11-05. Methods:initialize,ping,tools/list,tools/call. Notifications get 202. - Tools call REST in-process: each tool builds one or more
GET /api/v1/...requests and sends them through the REST router with the caller's ownAuthorization/cookie. Scopes, group restriction, rate limits (one per inner call), privacy levels, and the kill switch apply exactly as for REST, with no second authorization model. The REST problem+json becomes the tool's error text. - Catalog (read-only):
get_overview,top_items,search_queries,explain_decision,get_client_profile(the device, then its queries by IP),latency_breakdown,upstream_health,list_effectiveness,find_anomalies,cluster_status,get_config(one section).- Each description starts "Read-only.", and each tool carries
readOnlyHint. test_resolutionis deferred: it needs resolution without cache or log, which the pipeline doesn't offer yet.
- Each description starts "Read-only.", and each tool carries
- Schemas: hand-written JSON Schemas, frozen in the committed
docs/api/mcp-tools.jsonand diffed by a test (UPDATE_MCP=1to regenerate). Generating them from the REST parameter types would tie tool arguments to query-string shapes; the snapshot still prevents drift. - Guardrails: results are capped at 32 KiB (with a note) and 200 rows. Each
initializestarts a session (Mcp-Session-Id), and the client name/version from it is passed on asX-Telltale-Client, so agent actions are attributed to the MCP client (AGT-005). - stdio:
telltale mcp --stdiorelays newline-delimited JSON-RPC between stdin/stdout and a node's/mcpwith a token, so there's one implementation of the tools. The node address comes from--urlor the config files; nothing but protocol goes to stdout. - Agents and
/mcp:POST /mcpneeds no scope of its own and no reason header (the tools are read-only; the inner calls are checked). Group-restricted tokens may use it, since their inner calls are narrowed.
Verification
- Unit tests: the catalog snapshot; side-effect statements and caps; arguments become queries.
- An API test: initialize with a session, list, call, scope error inside a tool, unknown method, no credentials.
deploy/mcp-e2e.sh(CI) runs the official@modelcontextprotocol/sdkclient against a live node. Over Streamable HTTP, the listed tools must match the committed catalog. The test then plays the "why is the TV slow?" story (device profile with recent queries, upstream health, explain on a blocked name), repeats it overtelltale mcp --stdio, and checks that a token withoutquerylog:readgets an error fromsearch_queries.
Consequences
- Write tools with plan/apply and approvals (§3.2) are P1.
- OAuth for MCP clients (AGT-008) is P1.
- MCP resources and prompts are later.
- An SSE stream (
GET /mcp) would be needed only for server-initiated messages.
Amended 2026-10-08 (review 06-04, 06-07; amendment Proposed): the REST calls behind a tool carry the /mcp request's client address (OnBehalfOf, an extension only in-process requests can carry), so changes made through MCP are audited from the agent's address, not 0.0.0.0. They don't take from the agent's rate-limit bucket again: the rate counts tool calls, and each message of a JSON-RPC batch, which holds at most 16 messages. A resource cut at 32 KiB says so.
Requirements AGT-005 AGT-006 AGT-008 AGT-009
Builds on ADR-010
ADR-066 CA rotation: two trusted CAs, three member-gated phasesT5.4c (CLU-001, CLU-005).
Context
T5.4c (CLU-001, CLU-005). spec/12 and the roadmap ask for:
- CA rotation: a new key, a cross-signed transition, node certificate renewal over the channel, and the old CA retired;
- re-sharing the key with eligible nodes.
ADR-057 already renews node certificates (same key) over the channel. A CA rotation changes the trust anchor every TLS handshake and every manifest signature depends on.
Decision
- Two trusted CAs instead of cross-signing: during a rotation
ca.crtis a bundle of both CAs. TLS trusts every certificate in it, so certificates from either CA verify. Manifest signatures and issued certificates are checked against any CA in the bundle. The bundle's first CA is the one that signs, andca.keyis always its key. Cross-signing adds nothing in a cluster whose members all receive the bundle. - Distribution:
- The primary's signed manifest carries
ca_bundle. A node adopts a new bundle only if it keeps at least one CA the node already trusts, so trust moves step by step and a stolen CA alone can't replace it. - Witnesses follow manifests for this alone.
- Eligible nodes get the next CA's key with the existing key share, now re-sent whenever it changes. A key that arrives before its CA is trusted is held (
ca-pending.key) until a verified bundle includes it.
- The primary's signed manifest carries
- Phases, driven by the primary, each gated on every non-ephemeral registry member (heartbeats now carry
trust_fp, the CAs a node trusts, andissuer_fp, the CA of its certificate):trust: the bundle is old + new; the old CA signs.switch: once all trust both, the new key becomesca.key(the old is kept asca-old.key) and the bundle becomes new + old. Nodes see that their certificate isn't from the signing CA and renew within a minute (self-issued with the new key, or overcert.renew). A node whoseca.keydoesn't match the signing CA asks the primary rather than sign with the old key.finish: once every certificate is from the new CA, the bundle is the new CA alone, and the old key and the rotation state are deleted.
- Operator interface:
telltale cluster rotate-castarts a rotation on the primary while it runs, and the running primary carries it out (it re-reads its identity every cycle).--statusand aca_rotationCluster check show the phase and pending members. Each new CA's name carries a label, so the two trusted CAs never share a subject. - Fixed on the way:
- Configuration fetches used the TLS settings from start-up, so they'd fail after a rotation. They'd also present an old certificate after a renewal. They now read the current identity.
- Witnesses never renewed their certificates (ADR-057 gap); they do now.
- Cluster files were made owner-only only after being written, and two streams could collide on one temporary name. Each write now uses its own temporary name, and secrets are created owner-only.
Verification
- Unit tests:
- bundles: verify, issuer, order-independent fingerprint;
- start: refuses a second rotation;
- adoption: refuses unrelated trust, promotes a held key, switches, then retires.
deploy/cluster/rotate-e2e.sh(CI) with a primary, an eligible replica, and a witness under DNS load:- all three phases complete, and every node ends trusting only the new CA, with a certificate from it;
- the replica holds the new key, and no old, next, or pending key is left;
- every query was answered during the rotation;
- a configuration change replicates afterwards (signed with the new key), and a restarted replica reconnects and syncs.
Consequences
- A member offline for the whole rotation must rejoin.
- Join tokens pin the CA fingerprint, so tokens from before a rotation stop working.
- A forced rotation (with members down) is a possible follow-up.
Corrected 2026-10-08 (review 05-10): "a stolen CA alone can't replace it" overstates the rule. The rule stops a CA the node doesn't trust from replacing trust; it doesn't contain a stolen current CA key, whose holder can rotate trust to a CA of their own in three manifests, exactly as the legitimate primary does. That is inherent (the key signs everything): a suspected compromise of the cluster key means re-creating the cluster, as docs/running.md now says.
Requirements CLU-001 CLU-005
Builds on ADR-057
ADR-067 Quick rules: their own layer, the most specific scope wins, expiry on the serverT6.12 (owner request 2026-10-05) wants everyday per-person rules: The engine can already express device-scoped rules ($client in the manual-rules overlay, ADR-003), but the overlay uses the lists' four tiers (important allow > important block > allow > block).
Context
T6.12 (owner request 2026-10-05) wants everyday per-person rules:
- unblock a game's server for one phone for two hours;
- block a site for the kids' devices until tomorrow.
The engine can already express device-scoped rules ($client in the manual-rules overlay, ADR-003), but the overlay uses the lists' four tiers (important allow > important block > allow > block). Those can't say "this device's rule beats its group's rule beats the lists": a group-wide allow would override a parent's block for one child. Nothing expires on its own.
Decision
- A quick rule:
- allow or block a domain and its subdomains;
- for listed devices (by name, IP, or MAC, as
[[client]]keys), listed groups, or everyone; - an optional expiry (an absolute time; the UI offers durations);
- a note, who made it (user or agent), and when.
- Storage: in
state.dbas a managed entry of kindrule(ADR-040). In a cluster, rules travel with the shared configuration the primary publishes (ADR-047), and a replica's UI forwards the write to the primary (ADR-054). Rules are capped at 1,000 per cluster. - Matching: quick rules are their own layer, checked before the lists:
- Among rules matching the query name (any suffix of it) whose scope includes the client, a device rule beats a group rule, which beats an everyone rule.
- Within one scope, the longer (more specific) domain wins, then allow beats block.
- A match decides allow or block outright. Otherwise the lists decide as before.
CNAME inspection (FLT-007) uses the same decision, so a block also catches a CNAME to the blocked name.
- Hot path: the rules sit in a read-mostly table behind
ArcSwap, keyed by the hash of each rule's reversed domain. A lookup probes the query's suffixes, at most one probe per label, with no allocation. With no rules it costs one branch. - Expiry: each node ignores a rule from the moment its expiry passes, by its own clock (T6.11 shows clock offsets). A sweep every 5 s on the primary deletes expired rules and audit-logs each expiry. A replica never needs the sweep to stop applying a rule.
- Explanations: an answer decided by a quick rule is attributed to it ("allowed by a quick rule for Mom's phone, expires 21:30"). The query log records it, and "Why?" and explain show it above the list matches.
- Agents: the scope
config:write:rules, with the usual reason header, dry run, and impact estimate (recent queries the rule would change).
Consequences
- A new rule takes effect on the next query, with no list recompile.
- Cost: about 52 ns per query with 1,000 rules, about 4 ns with none (xxh3 suffix keys, measured 2026-10-05).
- Rules are deliberately simpler than list syntax: no regex, no query types. Anything fancier belongs in a list.
- Device scoping is only as good as recognising the device; the docs say so.
- Expiry depends on node clocks being roughly right. A node whose clock is off keeps or drops a rule early or late by that offset, and the Cluster page flags offsets of 2 s or more.
- Mixed versions (CLU-010): the config schema rejects unknown keys, and an empty
rulelist is left out of the shared configuration. So an N−1 replica follows the cluster as long as nobody has made a quick rule. Once one exists, the replica refuses that version and keeps serving its last good one until it's upgraded. Quick rules therefore wait for a cluster-wide upgrade, as any new shared section does.
Requirements CLU-010 FLT-007
Builds on ADR-003 ADR-040 ADR-047 ADR-054
Referenced by ADR-003
ADR-068 One process per data directory: an exclusive lock held for the process's lifeT6.14 (owner request 2026-10-05).
Context
T6.14 (owner request 2026-10-05). Scaling a single-volume Deployment (or starting a second telltale run with the same configuration) makes two processes share one data directory: the node identity and cluster certificates, state.db, the query-log segments, and the cache dump. Nothing stopped it, and the damage is silent: two nodes with one identity, interleaved query-log writes, and SQLite contention.
Decision
telltale runtakes an exclusive advisory lock (flock, throughstd::fs::File::try_lock) on<data_dir>/telltale.lockbefore anything else touches the directory, and holds it until the process exits.- If another process holds it, the new one retries for 10 s (a previous process may still be shutting down), then exits non-zero with an error. The error names the directory and the holder (pid, host or pod name, and start time, which the holder writes into the file), and points to
mode: scaled. - If the lock file can't be opened at all (a read-only or unusual file system), the node logs a warning and starts anyway: the lock protects against a mistake and must never stop DNS (CLU-004).
telltale backup restoretakes the same lock without waiting, so it refuses to restore under a running server.- Read-only commands (
qlog search,explain,backup create) don't take it.
Consequences
- The operating system releases the lock when the process exits, even after a crash or
SIGKILL, so there is no stale-lock cleanup. - A rolling update of a single-volume Deployment would wait for the old pod forever. The chart uses a StatefulSet and the homelab Deployment uses
Recreate, and the docs tell people with their own manifests to do the same. - On network file systems
flockmay be emulated or local-only. That is acceptable for a guard against mistakes.
Requirements CLU-004
ADR-069 Editing upstreams, lists, and groups from the UI: overrides in state.db, the files untouchedT7.5 (owner report 2026-10-06).
Context
T7.5 (owner report 2026-10-06). On a node installed with install.sh, the upstreams and lists come in the starter /etc/telltale/telltale.toml. ADR-040 makes anything the files define read-only through the API ("files win"), so none of them could be changed in the UI. Rewriting the file isn't an option either: Docker and Kubernetes mount it read-only, and on Git-managed nodes the next deploy would replace it.
Decision
- Upstreams, upstream groups, lists, and client groups made or changed through the API are managed entries in
state.db(like ADR-040's), by name. An entry can:- add a new name;
- override a file entry of the same name (the whole entry is replaced);
- hide a file entry (a tombstone: the name is left out).
- The files are never rewritten. The UI marks each entry as file, UI, or UI override of the file, and "Revert to the file" deletes the override or tombstone.
- Every change is validated on the merged configuration before it's stored (a group can't lose its last upstream; a list can't be removed while a group uses it; the default upstream group must keep a member), with dry run,
If-Match, idempotency, and audit as for the other managed kinds. - In a cluster, the merged configuration is what the primary publishes (ADR-047), so changes reach every node.
- On a node whose configuration comes from Git (
config_source = "gitops"), each change response carries the TOML to add to the repository or the Helm values to make it permanent ("Keep it in Git"), because an override lives only in that node'sstate.db. - Devices, local names, forwarded domains, and quick rules keep ADR-040's rule (files win), since the starter config doesn't define them.
Consequences
- A standalone node can be run entirely from the UI after
install.sh. - Two sources of truth for the same name are possible; the UI and the
GETendpoints show which one is in effect (source:file,api, oroverride). - A file change to an overridden entry has no effect until the override is reverted; the UI says so on the entry.
Referenced by ADR-111
ADR-070 Agents' plans: a dry run kept in memory on the node that made it, applied with If-MatchT7.1 (AGT-007).
Context
T7.1 (AGT-007). A plan_* MCP tool must show what a change would do and let the agent apply it later, only if nothing changed meanwhile and, with [agents] require_approval, only after an operator approves. The spec doesn't say where plans live, how long they last beyond "10 minutes", or how a cluster shares them.
Decision
- A plan is the REST write (method, path, body) plus its dry-run answer, the config version it saw, the reason, and who asked (the caller's token or session). Plan tools call the REST dry run in-process with the agent's own credentials, so scopes, group restrictions, rate limits, and the kill switch apply unchanged.
apply_planreplays the same write withIf-Match: <version>and the plan ID as theIdempotency-Key. A 412 marks the planstale. Only the caller that made a plan can apply or discard it.- Plans live in the memory of the node that made them, at most 500, open for 10 minutes and listed for a day. They aren't stored or replicated: a restart drops them, and the agent plans again.
- With
[agents] require_approval = truea new plan ispendinguntil a person with the operator role approves or rejects it (POST /api/v1/plans/{id}/approve|reject, the "Agent changes" page). Agent tokens can never approve or reject. Both decisions are audited. - Approval is checked when the plan is made: turning the setting on doesn't hold plans already made, and turning it off doesn't release pending ones.
- Immediate operations (
flush_cache,pause_blocking,resume_blocking) have no plan, asspec/13§3.2 says; they're audited like any agent write. plan_set_schedulewaits for schedules (FLT-010). Alert destinations for pending plans wait for OBS-010.
Consequences
- Nothing new is persisted, and a plan can't outlive the configuration it was checked against.
- In a cluster, an operator approves on the node the agent used. Agents should use the node whose UI people use (normally the primary). If agents and people use different nodes, plans would need to move to the replicated
state.db.
Requirements AGT-007 FLT-010 OBS-010
ADR-071 MCP over OAuth: JWT access tokens from the OIDC provider, for existing users onlyT7.4 (AGT-008, P1).
Context
T7.4 (AGT-008, P1). MCP clients sign in with OAuth 2.1: the server publishes RFC 9728 metadata naming an authorization server, and accepts that server's access tokens. The spec says to use the configured OIDC provider but not how tokens become principals, or which tokens are accepted.
Decision
- One provider is the authorization server for MCP (
[auth.oidc] mcp_provider). TelltaleDNS is only a resource server: client registration, consent screens, and refresh are the provider's. - Accepted tokens are JWTs signed with the provider's published keys (asymmetric algorithms only), with its issuer, the audience
mcp_audience(default<public_url>/mcp), and a current expiry (60 s leeway). There's no introspection, so opaque tokens are refused. - The token's
submust match a user who has signed in to the web UI with that provider (ADR-034). An unknown subject is refused rather than creating an account, because access tokens rarely carry the group claims that decide roles. - The principal is an agent of that user: the TelltaleDNS scopes in
scopeorscp(or the read-only default when there are none), never above the user's stored role; audited asagent:oauth:<azp> (owner: <user>). Rate limits, the kill switch, plans, and approvals apply as for agent tokens. - An unauthenticated or refused
/mcprequest getsWWW-Authenticate: Bearer resource_metadata="…"(anderror="invalid_token"when a token was sent).
Consequences
- Assistants connect with SSO and the user's consent, and no long-lived token is stored in the assistant.
- Each request verifies a signature (keys cached with discovery, refreshed on rotation); no per-token cache is kept.
- Providers that only issue opaque access tokens need a token per agent instead, until introspection is added.
Requirements AGT-008
Builds on ADR-034
ADR-072 Event sinks: one bounded buffer per sink, API-shaped JSON, no disk spillT7.13 (OBS-010; 06 §5).
Amended by
ADR-088. Syslog over TLS and a capped disk spill for webhook sinks were added later.
Context
T7.13 (OBS-010; 06 §5). The spec asks for JSON-lines, syslog, and webhook sinks "with retry and a disk spill cap", but not the event format, how a slow sink is kept off the query path, or how retries are bounded.
Decision
- An event is the API's query row (
QueryRow: the same fields and names asGET /api/v1/queries, names resolved when the event is written, the query log's privacy level applied), so a sink's output and the API's agree. - The aggregator thread formats each event once and offers it to every sink with
try_sendon a bounded channel (max_buffer); each sink writes from its own thread. A full buffer drops the event, counted and logged once a minute. DNS and the query log never wait for a sink. - The webhook posts batches (
batchevents orflush_secs), retries a failed batch twice (1 s, 2 s), then drops it. Events buffer in memory only; the "disk spill cap" isn't implemented: the query log already keeps every event on disk. - Syslog is RFC 5424 over UDP or TCP (RFC 6587 octet counting); TLS isn't implemented yet. Severity is notice for blocks and informational otherwise.
- Sinks are started once at startup (like the query log's writer); a reload doesn't change them.
Consequences
- A collector outage loses events beyond the buffer; the query log, which sinks don't replace, still has them.
- Syslog over TLS needs a forwarder (rsyslog, Vector) until it's added.
Requirements OBS-010
Builds on ADR-088
Referenced by ADR-076
ADR-073 DGA scoring and NXDOMAIN storms: fixed formulas, shipped bigrams, absolute storm barT7.14 (OBS-009; 06 §7).
Context
T7.14 (OBS-009; 06 §7). The spec names the signals (entropy, consonant runs, bigram log-likelihood) and the storm rule shape ("> X/min and > Y%") but not the weights, thresholds, the bigram source, or how findings are grouped.
Decision
- The bigram table is 27 × 27 signed bytes (
presets/bigrams.bin,round(4 × log2 P(next | previous))with start and end symbols) built from the public-domain dwyl/english-words list, embedded in the binary. - The score of a registrable domain's own label (8 to 63 characters, not
xn--) is0.7 × max(bigram, mix) + 0.2 × consonants + 0.1 × entropy, each clamped to 0..1: bigram = (mean bits per letter pair − 6) / 2; mix = (letter/digit switches − 2) / 4; consonants = (longest run − 5) / 3; entropy = (normalized entropy − 0.85) / 0.15. Calibrated on ~200 real service domains (none reach 0.2) and random-letter and hex labels (0.8 to 1.0). Logs come from tables, so the score is identical on every architecture. - Only first-seen domains of devices past their learning period are scored for findings; a
dgafinding needs at least 2 at or above the sensitivity's threshold (0.75, 0.6, 0.5) in one hour, and is one finding per device-hour with samples. - An NXDOMAIN storm is absolute (no baseline): at least
nxdomain_per_minute(30) NXDOMAIN answers andnxdomain_percent(50 %) of the device's queries in a calendar minute; once per device per hour. - The first-seen feed starts after a device's first day (its initial learning would flood it) and keeps the last 2000 entries, persisted with the engine.
- List overlap is counted at compile time (pairs of lists per shared name);
hitsare this node's counts since start, not 24 h / 7 d windows from the query log.
Consequences
- Dictionary-word DGAs and short generated labels aren't caught; some vendor IDs may score high (the UI shows the score, not a verdict).
- A restart resets list hits; overlap is exact for the active snapshot.
Requirements OBS-009
ADR-074 Our own iterative resolver, conservative defaultsT7.15 (DNS-012, UPS-012; 03 §6).
Context
T7.15 (DNS-012, UPS-012; 03 §6). 02 §2 allows building on hickory-recursor. The spec fixes the limits (16 CNAME hops, 32 referrals) and asks for QNAME minimization, 0x20 (configurable), SRTT server selection, glue in bailiwick, and RFC 8198, but not the defaults, the timeouts, or how the resolver meets the per-query budget.
Decision
telltale-recursoris our own iterative resolver ontelltale-proto(no new dependencies): exact control over minimization, bailiwick rules, and limits, and nothing added to the image beyond its code.- It's a transport of
telltale-upstream(recursive://), so groups, hedging, health, the answer cache, and DNSSEC validation (which then fetches DNSKEY and DS through it) work unchanged. Its attempt timeout is at least 3 s (not the 1 s cap of forwarders); the query's overall budget (2 s) still bounds a client's wait, and zone cuts learned meanwhile stay cached, so a retry continues from there. - Defaults: QNAME minimization on, relaxed (RFC 9156 §2.3: the full name after NXDOMAIN or failure on a minimized query; at most 10 minimized steps); 0x20 off (a few authoritative servers don't echo case; when on, such servers are remembered and asked plainly); IPv6 off (many home networks lack an IPv6 route); root hints built in, no priming query.
- Bailiwick: answers only for names under the zone being asked; glue only for server names under the referring zone; upward or sideways referrals are lame. A server name inside its own zone without glue is skipped.
- Caches: zone cuts with addresses and server-name addresses (20,000 entries each, TTLs clamped to 30 s..1 day), server SRTT (EWMA). Final answers are cached by the answer cache only.
- Limits: 16 CNAME/DNAME hops, 32 referrals per lookup, 4 nested lookups for server names, 96 queries per client question, 4 servers tried per question, 1.2 s per server at most.
- RFC 8198 (aggressive NSEC) is deferred: it needs validated NSEC records kept below the validator.
Consequences
- The common "Pi-hole + unbound" setup becomes one upstream; imports from Technitium without forwarders use it directly.
- A cold resolution in a zone with slow or broken servers can exceed the 2 s budget the first time (the client gets SERVFAIL or a stale answer; the next query succeeds from the learned cuts).
Requirements DNS-012 UPS-012
ADR-075 Proxies, socket plugins, pins, and client certificatesT7.16 (UPS-010, UPS-011; 04 §2, §6).
Context
T7.16 (UPS-010, UPS-011; 04 §2, §6). The spec names the features but not where a proxied hostname is resolved, how an exec:// plugin learns its socket, how plugins restart, or how pins interact with tls_insecure_skip_verify.
Decision
- Proxies (SOCKS5 with RFC 1929 credentials, HTTP CONNECT) carry
tcp://,tls://, andhttps://upstreams; others are a configuration error. A hostname URL goes to the proxy by name (SOCKS5 ATYP 3,CONNECT host:port), so no bootstrap is needed and no lookup leaks (Tor). Proxy addresses are IP literals (orlocalhost). UDP through SOCKS5 (UDP ASSOCIATE) stays P2. - Socket plugins speak RFC 7766 framing over a Unix socket, on the same pipelined connection pool as TCP upstreams.
exec://plugins getTELLTALE_PLUGIN_SOCKET(<data_dir>/plugins/<upstream>.sock) andTELLTALE_UPSTREAMin their environment,argsas arguments, stdin closed, stdout and stderr logged per line; they restart with backoff (1 s doubling to 60 s, reset after a minute up) and are killed when their upstream is dropped (each reload rebuilds upstreams, so a reload restarts plugins). - Pins are base64 SHA-256 of the leaf's
SubjectPublicKeyInfo, checked after the normal WebPKI verification; withtls_insecure_skip_verifyonly the pin is checked.tls_caadds roots for one upstream;tls_client_cert/tls_client_key(PEM) enable mTLS. Files are read at load and reload. - Deferred: DoH GET, the WASM upstream ABI (P2).
Consequences
- Tor users point an encrypted upstream at the Tor SOCKS port with a hostname URL and nothing else.
- A plugin that needs state across reloads must keep it itself (it's restarted on configuration changes).
Requirements UPS-010 UPS-011
ADR-076 OTLP over HTTP with JSON, metrics converted from our expositionT7.17 (OBS-006; 06 §5).
Context
T7.17 (OBS-006; 06 §5). The spec asks for "the same metric set via OTLP/HTTP" and optional QueryEvent logs, without an encoding or a mapping.
Decision
- OTLP/HTTP with the JSON encoding (the protobuf JSON mapping: 64-bit integers as strings). No OpenTelemetry SDK and no generated protobuf types: the binary stays small, and every OTLP/HTTP receiver accepts JSON.
- Metrics are converted from our own Prometheus exposition every
interval_secs: counters → cumulative monotonic sums (start time = process start), gauges → gauges, histograms → explicit-bucket histograms (per-bucket counts from the cumulative ones). Names and labels are kept as in/metrics, so dashboards and docs apply to both. - Query events as logs are an event-sink format (
otlp_logs, ADR-072): batching, buffering, and retries are the sink's. Body: a one-line summary; attributes use OpenTelemetry DNS-style keys where they exist (dns.question.name,client.address),telltale.*otherwise. - No OTLP/gRPC.
Consequences
- Collectors that convert back to Prometheus see
_totalalready in counter names (most keep it as is). - A counter reset (restart) shows as a new start time, as OTLP expects.
Requirements OBS-006
Builds on ADR-072
ADR-077 dnstap: a sampled copy of the wire messages, off by defaultT7.18 (OBS-007; 06 §5).
Context
T7.18 (OBS-007; 06 §5). dnstap needs the query and response messages, which query events don't carry (02 §3: no allocation on the hot path).
Decision
- The tap is a
OnceLockon the pipeline set at startup: with dnstap off, the query path pays one atomic load. With it on, one query insample_every(a relaxed counter) is copied, both messages, into a bounded channel withtry_send; full means dropped and counted. Measured with bench-smoke: no change with it off (within WSL run-to-run noise). - The writer thread speaks bidirectional Frame Streams (READY, ACCEPT, START; STOP at the end) to a Unix socket or TCP reader, reconnects with backoff (1 s to 30 s), and writes
CLIENT_QUERYandCLIENT_RESPONSE(address, family, transport including DoT/DoH/DoQ, times, messages; identity = node name, version = TelltaleDNS version). Protobuf is encoded by hand: the schema is small and fixed. - Deferred:
FORWARDER_QUERY/FORWARDER_RESPONSE(they'd need a tap in the upstream exchange), and the client's port (not on the event path).
Consequences
- At
sample_every = 1every query costs two small allocations and a channel send; the docs say to sample on busy resolvers.
Requirements OBS-007
ADR-078 DHCPv4: authoritative, broadcast replies, leases name devicesT7.19 (OPS-008; 08 §7).
Superseded by
ADR-091. The DHCP server was removed on 2026-10-06; this record stays for history.
Context
T7.19 (OPS-008; 08 §7). The spec lists the options, static leases, and a lease file, but not the server's stance toward clients asking for foreign addresses, how replies reach clients without an address, or how leases "feed client naming".
Decision
- A module of the binary (
crates/telltale/src/dhcp.rs), its own task and socket (socket2:SO_BROADCAST,SO_REUSEADDR, optionalSO_BINDTODEVICE), never on the DNS path. - Authoritative for its subnet (like
dhcp-authoritativein dnsmasq): REQUEST for an address this client may not have (someone else's, outside the pool, another subnet) gets NAK; INIT-REBOOT for a free pool address is granted. Offers are held 60 s; DECLINEd addresses are skipped for 10 minutes; no ping-before-offer. - Replies (RFC 2131 §4.1): to
giaddr:67via a relay, tociaddr:68when the client has an address, else broadcast to255.255.255.255:68(no raw-socket unicast to an unconfigured client: broadcast is allowed and needs no extra capability). - Options 1, 3, 6 (default: the server's address), 15, 28, 42, 51, 54, 58, 59, 119 (RFC 3397, uncompressed; split per RFC 3396 when long).
- Leases persist to
<data_dir>/dhcp-leases.jsonafter every change; the live view (address → lease) is published on the pipeline, so device names fall back to the lease host name (a configured client name wins), andGET /api/v1/dhcp/leasesreads it. - In a cluster, one node runs it (documented); other nodes don't see its leases yet.
Consequences
- A Pi can replace the router's DHCP; unnamed devices get readable names immediately.
- Without failover, a second DHCP node needs split pools (not supported in config yet).
Builds on ADR-091
Referenced by ADR-091
ADR-079 DNSCrypt with RustCrypto's crypto_box; stamps map to existing transportsT7.24 (UPS-003; 04 §2).
Context
T7.24 (UPS-003; 04 §2). DNSCrypt v2 needs X25519 and libsodium's box constructions; 02 §7 lists no crate for them. Stamps may name DNSCrypt, DoH, DoT, or DoQ servers.
Decision
- New dependency
crypto_box0.9 (RustCrypto; its SalsaBox and ChaChaBox are libsodium'scrypto_boxandcrypto_box_curve25519xchacha20poly1305, on thecurve25519-dalekalready in the tree) anded25519-dalek(already in the tree) for certificates. The IETF XChaCha20-Poly1305 AEAD is a different construction and wouldn't interoperate. The detached API is used and the tag written first (libsodium's_easylayout); verified live against five providers' servers. - One X25519 key per upstream for its lifetime (like dnscrypt-proxy's default); a fresh random nonce per query; ISO 7816-4 padding to 64 bytes, at least 256 over UDP; TCP when the answer is truncated.
- Certificates: TXT records of the provider name fetched in clear, each checked with the stamp's Ed25519 key (
verify_strict) and its validity dates; the newest construction, then the highest serial, wins; refreshed after an hour or at expiry. - DoH/DoT/DoQ stamps become those transports (host name as the TLS name, an address in the stamp pinned); the stamp's certificate hashes aren't enforced yet.
- Not supported: anonymized DNSCrypt relays, DNSCrypt stamps'
propsflags (DNSSEC, no-log, no-filter are informational).
Consequences
- Users of dnscrypt-proxy's server list can paste stamps directly.
- About 100 KiB more in the binary.
Requirements UPS-003
ADR-080 Router integrations: read-only polling of UniFi and OPNsenseT8.2 (M8 "router integrations"; API-010 lists DHCP leases as a naming source).
Context
T8.2 (M8 "router integrations"; API-010 lists DHCP leases as a naming source). Most homes run DHCP on the router, so TelltaleDNS's own DHCP (T7.19) doesn't see those leases.
Decision
[[router]]entries are polled read-only (interval_secs, 300; 15 s after a failure). UniFi:stat/sta(connected clients) with an API key (X-API-KEY), or a local user's login (UniFi OS/api/auth/loginand/proxy/network/..., falling back to the classic/api/login); the UniFi alias wins over the host name. OPNsense: the lease search of Dnsmasq, then Kea, then ISC, with the API key and secret (basic auth).- The result replaces that router's part of a lease view next to the DHCP server's; device names fall back to configured clients, then DHCP leases, then router leases.
GET /api/v1/dhcp/leaseslists both with asource. - Self-signed consoles:
tls_ca(the console's certificate), ortls_insecure_skip_verifywith a configuration warning. Secrets come from files. - Not done: OPNsense static mappings without a lease, UniFi's offline known clients (
rest/user), pfSense, OpenWrt.
Consequences
- Device names appear on networks where the router runs DHCP, with no change on the router beyond a read-only account or key.
Requirements API-010
ADR-081 mDNS device naming: passive, A records of <name>.localT8.3 (M8 "mDNS client naming"; API-010 lists mDNS as a naming source).
Context
T8.3 (M8 "mDNS client naming"; API-010 lists mDNS as a naming source). Many devices announce their host name over mDNS (RFC 6762) whether or not anyone asks.
Decision
[clients] mdns(off by default): join 224.0.0.251 on the mDNS port (5353, address and port reuse) and read A records owned by a two-label<name>.localin the answer and additional sections of responses others send. Never send queries or responses (no probing, no load on the network). IPv4 only, like the DHCP and router names.- The latest name per address is kept in memory (4096 at most, the oldest tenth dropped when full) and published every 5 s; device names fall back to configured clients, DHCP leases, router leases, then mDNS.
- Service instance names (
Living Room._airplay._tcp.local), link-local and unspecified addresses are ignored. - Not done: asking (
QUqueries to fill the gaps), IPv6 (AAAA owners would need the neighbor table to match clients), persisting across restarts, showing the names in the leases API.
Consequences
- Devices get readable names without a router integration or DHCP, on networks where TelltaleDNS can hear multicast.
Requirements API-010
ADR-082 vqlog: a pipe language compiled to qlog searches, aggregated in memoryT8.4, AGT-012: "a constrained, safe query DSL (vqlog) over the query log and rollups (filter, group by, top-K, percentiles, time bucket), with a cost estimate, exposed as one tool".
Context
T8.4, AGT-012: "a constrained, safe query DSL (vqlog) over the query log and rollups (filter, group by, top-K, percentiles, time bucket), with a cost estimate, exposed as one tool". The spec doesn't fix the syntax, the limits, or the scope.
Decision
- Syntax: stages separated by
|—from,where(conditions joined byand;in (…)lists instead ofor),bucket,by,stats,top,sort,limit. A fixed grammar parsed into a plan intelltale-api(syntax errors are 400s with hints); no expressions, functions, or joins. The name kept from the spec (vqlog) although the product was renamed. - Execution reads the query log only, not the rollups: rollups can't filter by name or client, and percentiles over them would be approximations of approximations. The conditions the qlog index can use are pushed into the search (time, one name, one client, group, statuses, qtypes, rcodes, one upstream, minimum latency); every condition is also checked on each row. Grouping is exact, in memory; percentiles are exact (nearest rank).
- Cost: a header-only estimate (segments, blocks, and rows that could match, using the same block skipping as search) before the scan, returned with every answer;
dryRunreturns only that. Above 100 million estimated rows the query is refused. The scan stops at 2 million matches, 100,000 groups, or 25 seconds and reportstruncatedwith the reason; results then cover the newest queries. - Access:
querylog:read(it reads per-device, per-name rows); not group-aware, so group-restricted agents can't use it. Scope: this node's log plus the logs shipped to it (CLU-007), so the primary of a shipping cluster sees everything;missingNodesas elsewhere. - Not done:
orbetween fields, rollup-backed long windows, per-cluster fan-out with mergeable sketches, a UI editor.
Consequences
- One MCP tool answers most analytics questions with exact numbers and a visible cost; the narrow tools stay for the common ones.
- Long windows on busy networks are slow (it scans), and the limits say so instead of degrading DNS (search threads run at background priority).
Requirements AGT-012 CLU-007
Referenced by ADR-089
ADR-083 io_uring UDP: measured, not adoptedM8 lists io_uring; 02 §3 calls it "a P2 experiment behind a feature flag".
Context
M8 lists io_uring; 02 §3 calls it "a P2 experiment behind a feature flag". The UDP listener already batches with recvmmsg/sendmmsg on one SO_REUSEPORT socket per worker, so the syscall cost per query is already amortized.
Measured (2026-10-06): a standalone spike, not in the repo: one server thread doing the same per-datagram work both ways (64-slot RecvMsg/SendMsg io_uring ring vs 64-message recvmmsg/sendmmsg), loopback, WSL2 kernel 6.18, 8 cores, alternating 5-second rounds.
- With 4 client threads: 530.8k vs 526.3k answered per round (io_uring −0.9%).
- With 7 client threads (server about 90% CPU): 456.8k vs 471.6k (+3%).
- The spread between rounds was about ±15%, so neither difference is meaningful.
Decision
keep recvmmsg/sendmmsg; don't add an io_uring listener or feature. Reasons:
- No measured gain.
- More
unsafein the hot path (buffers that must outlive submissions). - A second listener to test.
- io_uring is often unavailable where TelltaleDNS runs: Docker's default seccomp profile blocks it, and some distributions disable it (
kernel.io_uring_disabled).
Revisit if a profile on real hardware shows syscalls dominating at the owner's target rates. Multishot receive with provided buffers would be the variant to try.
Consequences
- One UDP code path.
- The M8 item is closed with data rather than code.
ADR-084 WASM upstream plugins: deferred, and wasmi rather than wasmtime if builtM8 lists WASM upstream plugins; 04 §3 and ADR-007 name wasmtime behind a feature flag.
Context
M8 lists WASM upstream plugins; 04 §3 and ADR-007 name wasmtime behind a feature flag. T7.16 shipped the out-of-process plugins (unix:// and exec://, DNS wire format), which already let any language implement an upstream.
Measured (2026-10-06): stripped, LTO, panic = "abort" release binaries of a program that compiles one module, against an empty program, on x86_64 Linux:
| Engine | Adds (raw) | Adds (gzip -9) |
|---|---|---|
| wasmtime 49 (Cranelift JIT, component model) | 9.4 MiB | 3.3 MiB |
| wasmi 2.0 (interpreter, pure Rust) | 1.7 MiB | 0.6 MiB |
The release binaries are 19–23 MiB raw today, and the image budget is 15 MiB compressed (OPS-001).
Decision
- Don't build WASM plugins now. There's no user request; the process plugins cover the use case; and a sandboxed host networking API is a design of its own.
- If they're built, use wasmi behind an off-by-default
wasmfeature. It costs about a fifth of wasmtime's size, is pure Rust (no JIT, so it runs on armv7 and under seccomp profiles that forbid executable memory), and is fast enough for a per-query function that mostly waits on the network. - Revisit on a concrete request, for example a resolver protocol that people want to ship as one file.
Consequences
- M8 closes without code here.
- The plugin story stays the process boundary: crash isolation, and any language.
Requirements OPS-001
Builds on ADR-007
ADR-085 Alert email: a small built-in SMTP submission clientT9.4 (OBS-010 lists email as a destination; deferred in T7.12).
Context
T9.4 (OBS-010 lists email as a destination; deferred in T7.12). The owner has no relay of their own.
Decision
- A minimal client in the binary (
smtp.rs, written from RFC 5321/3207/4954/5322) over the rustls stack already shipped, instead of a mail library: one message per connection, STARTTLS (smtp://, 587) or implicit TLS (smtps://, 465), AUTH PLAIN or LOGIN, the public roots plustls_ca, base64 text bodies, RFC 2047 subjects. - A password is only sent encrypted: STARTTLS can't fall back to plain, and
smtp+insecure://(for local catchers) refuses a username in the configuration check. - Validation: CI runs a scripted SMTP server (STARTTLS with a private CA, implicit TLS, a wrong password reported as 535); Mailpit was checked by hand as an independent implementation; real delivery is the owner's choice of provider (a Gmail app password needs no relay).
- Not done: HTML mail, DKIM (the provider signs), a queue (a failed alert is logged like any destination).
Consequences
email works with any provider that offers SMTP submission, for about 30 KiB of code and no new dependency.
Requirements OBS-010
ADR-086 Recursion and DNSSEC follow-ups: priming, hedging, EDE reasons, an anchors file; RFC 5011 and 8198 deferredT9.8 picks up the deferrals of T6.3 (DNSSEC) and T7.15 (recursion).
Amended by
ADR-092. RFC 8198 (aggressive NSEC) was implemented later; RFC 5011 stays deferred.
Context
T9.8 picks up the deferrals of T6.3 (DNSSEC) and T7.15 (recursion).
Decision
- Root priming (RFC 8109): on first use, the hints are asked for
. NS; the answer's glue (or, without it, lookups of the names) becomes the cached root zone cut, renewed when it expires (TTL capped at a day). A failed attempt waits a minute before the next. - Hedged queries: a server that hasn't answered after 1.5 × its smoothed RTT (at least 150 ms; 400 ms when unknown) is joined by the next server; the first usable answer wins. It costs at most one extra query per slow step, inside the 96-query budget.
- EDE reasons: a bogus answer's RRSIGs give the code: none for the records that answer → 10, all expired → 7, all not yet valid → 8 (serial-number time comparison), else 6. The library's validator doesn't expose its reason, so this reads the signatures it returns.
- Trust anchors: both current root KSKs (20326, 38696) ship built in (a test pins them).
[dnssec] trust_anchors_filereplaces them with a file of DNSKEY records; a bad file keeps the built-in ones. - Deferred — RFC 5011 automated trust-anchor updates: they need a persistent state machine with a 30-day hold-down for a root KSK change that happens every several years, which builds already carry. The anchors file covers the gap.
- Deferred — RFC 8198 aggressive NSEC caching: it needs a cache of validated NSEC/NSEC3 ranges next to the library's validator and answer synthesis on the query path; worth doing with measurements, not as a follow-up.
Consequences
- Recursive lookups ride out a slow server and a moved root server.
- SERVFAILs say why a signature failed.
- Validation survives the root key rollover with no action.
Builds on ADR-092
Referenced by ADR-092
ADR-087 Upstream follow-ups: DoH GET, UDP over SOCKS5, DNSCrypt relays, per-client ECST9.9 picks up the deferrals of T7.8 (DoH), T7.16 (proxies), T7.23 (ECS), and T7.24 (DNSCrypt).
Context
T9.9 picks up the deferrals of T7.8 (DoH), T7.16 (proxies), T7.23 (ECS), and T7.24 (DNSCrypt).
Decision
- DoH GET:
doh_method = "get"puts the query in?dns=(base64url, no padding) with ID 0; POST stays the default (smaller requests, no URL length concerns). - UDP through SOCKS5: a
udp://upstream with asocks5://proxy uses UDP ASSOCIATE (RFC 1928 §7). Up to four associations are kept for reuse; one whose control connection or relay fails is dropped. The TCP retry for a truncated answer goes through the proxy too. HTTP proxies still refuseudp://in the configuration check. - Anonymized DNSCrypt:
relay(a relay stamp, type 0x81, orip:port) on a DNSCrypt upstream prefixes every packet with the anonymized-DNSCrypt header (8 × 0xff, 0x0000, the server's address as IPv6, its port), the certificate query included, over UDP and TCP. One relay per upstream; choosing relays at random from a list is left to configuring several upstreams in a group. - Per-client ECS (
ecs = "client"): the client's source address cut to /24 or /56 (RFC 7871 §11.1). Gaps resolved conservatively:- nothing is sent for non-public sources (private, loopback, link-local, CGNAT, ULA);
- ECS options sent by clients are still never forwarded or echoed, so a downstream forwarder never caches an answer as if it were scoped;
- the cache key gains the packed subnet (8 bytes per key, 0 when unused), but only for groups that have a
"client"member, which is a flag checked once per miss; - entries are keyed by the source prefix, not the upstream's SCOPE: never shared more widely than the subnet sent, at the cost of fewer hits;
- scoped entries aren't written to the cache file, whose format has no subnet.
Consequences
- DNSCrypt users get the anonymity dnscrypt-proxy's relays give, and SOCKS5 users keep UDP's latency.
ecs = "client"multiplies cache entries by the number of client subnets; it's meant for resolvers with public clients, and does nothing on a home network.
ADR-088 DNS64 exclusions and reverse names; syslog over TLS; a webhook spillT9.10 and T9.11 pick up the deferrals of T7.21 (DNS64) and T7.13 (event sinks).
Context
T9.10 and T9.11 pick up the deferrals of T7.21 (DNS64) and T7.13 (event sinks).
Decision
- DNS64 exclusion set (RFC 6147 §5.1.4):
dns64_excludeon the group takes networks of either family. An excluded AAAA counts as missing. With all of a name's AAAA records excluded, AAAA records are made up for it. With some left, the excluded ones are removed, along with signatures, since the set they signed changed; DNS64 already never applies to CD clients. An excluded A record is never made into AAAA.::ffff:0:0/96is always excluded, as the RFC recommends. - DNS64 reverse names (§5.3.1): a PTR question for an address inside the client group's /96 gets a CNAME to the embedded IPv4 address's
in-addr.arpaname, resolved like a safe-search target. A private IPv4 address keeps RFC 6303's local NXDOMAIN, so it never goes upstream. Of the RFC's two options, the CNAME was chosen over making up a PTR record: it needs no state and shows where the answer comes from. - Syslog over TLS (RFC 5425):
tls://host:port, with octet-counted frames as on TCP, rustls with the public roots plustls_ca, and the certificate checked against the host in the address. No client certificates (none asked for yet). - Webhook spill: off unless
spill_max_bytesis set (at least 1 MiB).- Refused batches are appended as JSON lines to
<data_dir>/sinks/<name>.spill, with the send offset in a.posfile, so a restart doesn't resend what was sent. The file is compacted when it needs room and removed when it's empty. - Replay runs up to 20 batches after a delivered batch or on a quiet tick, and stops at the first failure. Events that arrive meanwhile wait in the sink's normal buffer.
- Refused batches are appended as JSON lines to
Consequences
- IPv6-only networks get working reverse names and can route around unreachable IPv6 ranges.
- A SIEM can take events over TLS.
- A collector outage no longer loses events, up to the cap. The disk writes happen only during an outage, on the sink's own thread.
Referenced by ADR-072
ADR-089 vqlog or and rollup windows; pre-save checksT9.12 picks up ADR-082's "not done" (or, rollup-backed long windows) and the pre-save checks deferred from T7.5.
Context
T9.12 picks up ADR-082's "not done" (or, rollup-backed long windows) and the pre-save checks deferred from T7.5.
Decision
or:andbinds tighter. A parenthesized group is one term of theandlist, one level deep. A bareormakes the whole stage one group, and mixing it with parentheses is refused as ambiguous rather than guessed. The plan keeps the plain conditions (still pushed into the index search) and addsany_ofgroups checked per row. At most 32 alternatives in all;in (…)covers many values of one field.- Rollup windows:
- When: a query whose
fromis older thanretention_days, or any query while the query log is off, is answered from the rollups when it fits them. - What fits:
countonly, noor, buckets of whole hours, and at most one of status, qtype (the named ones), rcode, proto, and group, used as the key, the filter, or both. The rollups keep each breakdown on its own, so two can't be combined. - Levels: day buckets use the day rollups; everything else uses the hour rollups.
- Labels:
sourcesays which store answered. A query that doesn't fit runs over the log as before, with anotesaying the log only coversretention_days. - Not chosen: answering partly from each store. Its numbers would mix two sources, with no line between them.
- When: a query whose
- Pre-save checks:
POST /api/v1/checks/upstreamand/checks/listtake the same body as the PUT.- The upstream check builds the upstream through the normal router (a configuration holding only it) and asks for
. NSwithin 5 s. The list check downloads with[filter]'s limits, or takes inline rules, and parses as the compiler does. pathlists aren't checked: reading server files on request would be a new capability.- Both need the operator role and the entry's write scope (they reach out from the node), and both are audited. The paths sit under
/checks/so they can't collide with an upstream or list namedtest.
- The upstream check builds the upstream through the normal router (a configuration holding only it) and asks for
Consequences
- An agent can ask "blocked per day for six months" and get exact counts.
- Questions that need names or clients stay limited to what the log keeps, and the answer says so.
- A typo in an upstream URL or a dead list URL shows up before it's saved.
Builds on ADR-082
ADR-090 DoH load from h2load; restart counts in the data directoryT9.13 picks up the deferrals of T0.3 (DoT/DoH load, a realistic profile) and T6.14 (restart counts, Grafana panels).
Context
T9.13 picks up the deferrals of T0.3 (DoT/DoH load, a realistic profile) and T6.14 (restart counts, Grafana panels).
Decision
- Transports bench: dnsperf (
-m tcp|dot) measures TCP and DoT.- DoH uses h2load, a bench-only tool that isn't shipped. dnsperf's DoH client ran at about 40 ms a query against a listener that answers curl in 175 µs, so it would have measured itself.
- DoH requests are GETs built from the same corpus, so the names match the other transports.
- The existing
realistic-homecorpus stays with the swap test. The new mode uses cache hits, because the Python stub upstream would saturate first and hide the transports' cost.
- Restart counts:
<data_dir>/runs.jsonholdsstarts,unclean, andrunning. A start incrementsstarts, and adds touncleanifrunningis still set, then setsrunning. A clean stop clears it, even when startup fails.- It's written only by the process holding the data-directory lock.
- An unreadable file starts over, rather than failing startup.
- These are counters, so
increase()over a range works across restarts. The heartbeat-observedrestartscolumn in the cluster view stays as it is (what peers saw).
- Grafana: new rows for transports (queries by protocol, TLS/DoH/DoQ/PROXY problems) and the process (uptime, restarts, unclean starts, OOM kills, CPU, temperature, data-disk free). Every expression is checked with
promtool check rulesin the Prometheus image.
Consequences
- What DoT and DoH cost compared with UDP is measured, and repeatable on any machine with h2load.
- A crash loop shows up as a number on the dashboard instead of a short uptime.
ADR-091 No DHCP server: routers hand out addressesT7.19 added an optional DHCPv4 server (OPS-008, ADR-078).
Context
T7.19 added an optional DHCPv4 server (OPS-008, ADR-078). The owner (2026-10-06): most people let their router serve DHCP, and DHCP sits outside what a DNS product should own. Keeping it means a privileged port (67), a lease state machine, and a failover question in clusters, all for few users.
Decision
- The server goes: the code,
[dhcp], its end-to-end test, and its documentation. OPS-008 is descoped, and ADR-078 is superseded. - Device names keep coming from outside DHCP sources:
- routers' DHCP clients (
[[router]]: UniFi, OPNsense); - mDNS announcements;
- the importers, which still turn Pi-hole's and Technitium's DHCP reservations into named devices.
- routers' DHCP clients (
GET /api/v1/dhcp/leasesstays: these are the router's DHCP leases. It dropsreservedandsource = dhcp.- A configuration that still has
[dhcp]fails to load with the usual unknown-field error rather than being ignored. Neither of the owner's nodes used it.
Consequences
- About 1,100 lines and a privileged port are gone. The scope stays DNS.
- Anyone who wants TelltaleDNS to serve DHCP needs their router, or dnsmasq beside it.
Builds on ADR-078
Referenced by ADR-078
ADR-092 Aggressive NSEC (RFC 8198): NSEC ranges per upstream group, NSEC3 laterADR-086 deferred RFC 8198.
Context
ADR-086 deferred RFC 8198. The owner asked for the deferred features that make TelltaleDNS a more complete DNS product. Unbound and BIND both do this by default, and on a home network the root zone's NSEC records cover the steady trickle of made-up TLDs (.lan, .home, .localdomain).
Decision
- What's kept: after the validator proves a negative answer secure, its NSEC records are cached, keyed by owner in canonical order, per upstream group. Each is kept only if:
- it's proven secure;
- it's inside the zone of the SOA that came with it;
- it's signed by that zone. The SOA and all RRSIGs are kept too.
- How long: the NSEC's TTL, capped by the SOA's negative TTL (RFC 8198 §5.4). At most 20,000 ranges and 2,000 zones per group; past that, the ones expiring first go.
- Answering on a miss, before going upstream:
- NXDOMAIN when one range covers the name and one covers the wildcard at the closest encloser (RFC 4035 §5.4).
- NODATA when an NSEC at the name lacks the type and CNAME, and isn't a delegation (unless the question is DS).
- Nothing below a delegation's or a DNAME's owner. The child zone, or the alias, decides there.
- The answer: NXDOMAIN or NODATA with the SOA and NSEC records, plus their signatures for DO clients, and AD as for any secure answer. TTLs are counted down to the earliest expiry. It's counted as synthesized, not as a validation.
- Switch:
[dnssec] aggressive_nsec, on by default, as in Unbound. - Deferred — NSEC3: synthesis needs the zone's hash parameters and a hash per lookup, and opt-out spans prove nothing for unsigned delegations. Worth doing with measurements.
Consequences
- Repeated junk lookups under NSEC-signed zones (the root above all) stop reaching upstreams, and are answered in microseconds instead of a round trip.
- Answers still carry their proof, so a validating client downstream can check them.
- A config reload starts the cache empty, like the validator's own cache.
Builds on ADR-086
Referenced by ADR-086
ADR-093 DNS stamps: enforce their certificate hashesT7.24 parsed DoH/DoT/DoQ stamps but skipped their hashes.
Context
T7.24 parsed DoH/DoT/DoQ stamps but skipped their hashes. A stamp's hashes are SHA-256 digests of the tbsCertificate of certificates in the server's chain: the publisher's way of saying "only these CAs (or this certificate) may vouch for it". dnscrypt-proxy enforces them.
Decision
- Hashes are parsed. Each must be 32 bytes, or the stamp is malformed.
- When a stamp has hashes, the normal verification runs first, then some certificate in the presented chain must match one. With
tls_insecure_skip_verifythe match is still required, as for SPKI pins. - No opt-out switch. Someone who doesn't want the pinning uses the plain URL instead of the stamp, which is clearer than a flag that silently weakens what the stamp says.
- Failures name the cause.
Consequences
- Stamps mean what they say.
- When a provider rotates its intermediate CA before republishing the stamp, that upstream fails until the stamp is updated. A group with another member fails over meanwhile.
- No built-in preset uses stamps.
ADR-094 Local zones answer SOA and NS at their apex; no zone transfersT7.22's zones answered their records, but not SOA or NS at the apex.
Context
T7.22's zones answered their records, but not SOA or NS at the apex. Their negative answers carried a placeholder SOA owned by the question name. Tools (dig SOA, monitoring checks, some resolvers' zone-cut detection) expect real ones.
Decision
- From the zone file: if it has SOA and NS at its apex, they're served as written. Relative names are completed with the origin.
- Otherwise, made up as Unbound does for local zones:
- NS
localhost.; - SOA
localhost. hostmaster.<zone>; - serial = load time;
- refresh 3600, retry 600, expire 86400;
- MINIMUM =
negative_ttl.
- NS
- Negative answers (NXDOMAIN and NODATA) carry that SOA owned by the apex, with TTL
negative_ttl(RFC 2308). - Not done: zone transfers (AXFR/IXFR). They need TSIG and secondary servers, which home and lab installs rarely have. The zone file, or Git, is the source to copy instead.
Consequences
dig SOA/NSand negative caching downstream behave as with any authoritative server.- The made-up
localhost.name server can't be queried from outside, which is the point: these zones are served by the resolver itself.
ADR-095 $dnsrewrite in lists: AdGuard's precedence, a supported subsetT2.3 kept $dnsrewrite rules in the snapshot but didn't apply them (T7.20 put rewrites in the configuration).
Context
T2.3 kept $dnsrewrite rules in the snapshot but didn't apply them (T7.20 put rewrites in the configuration). AdGuard-format lists and personal rule sets use them for local names, safe-search CNAMEs, and blanking names out. Without them those lists lose part of their meaning.
Decision
- Values supported:
- an address, either family (A/AAAA);
- a name (CNAME);
NXDOMAIN,REFUSED, orSERVFAIL;- the full form for
NOERRORwith A, AAAA, or CNAME, and any rcode with an empty answer. Other record types (MX, TXT, SRV, HTTPS, …) make the line unsupported (counted, skipped), so a list author's intent is never half-applied.
- Precedence, as AdGuard does it:
- a matching rewrite wins over blocking;
- an exception (
@@…$dnsrewrite, with or without a value) turns rewrites off for the names it covers; $client,$dnstype, and$denyallownarrow a rewrite like any rule;- schedules and quick rules still come first.
- Combining: an rcode wins, then the first CNAME, then every address of the asked family. When the name is rewritten only for the other family, the answer is an empty NOERROR.
- On the query path: a matcher flag skips everything when no list has rewrites. Otherwise the modifier index is walked once more for the name. The answer has TTL 60, is attributed to the list in the event, and a CNAME is resolved through the normal path (cache, upstreams, filtering of the target).
Consequences
- AdGuard-style lists and personal rules work as their authors meant.
- Configuration rewrites (
[[group.rewrite]]) stay the per-group tool. List rewrites follow the lists a group uses.
ADR-096 A client group's own upstream groupT9.25 (UPS-007, owner request 2026-10-06).
Context
T9.25 (UPS-007, owner request 2026-10-06). UPS-007 already routes by client group ([[route]] match_group), but that lives in routing, not on the group, and the UI had no way to set it. The spec doesn't say how it ranks against domain routes, which group wins for a device in several, or what removing an upstream group a group uses does.
Decision
[[group]] upstreams = "<upstream group>", validated like a route'supstream_group. The router turns it into a route for the group's clients that matches any name, placed after the explicit[[route]]entries. Precedence: a domain route (deeper suffix), then an explicit route of equal depth (amatch_grouproute, which comes first), then the group'supstreams, thendefault.- A device in several groups: the derived routes are ordered by group priority (highest first), then configuration order, so its highest-priority group with
upstreamsdecides. - Removing an upstream group that a client group or a route still uses is refused with a hint, rather than clearing the choice. Clearing it would quietly send, say, the kids' questions to
defaultinstead of a family resolver. Reverting an API override of the files' entry is allowed (the group stays).
Consequences
- No hot-path cost: routes are consulted only on a cache miss, and the cache is already keyed by upstream group, so groups never share answers from different resolvers.
[[route]] match_groupstays for narrower cases (one group'sTXTquestions, say).
Requirements UPS-007
ADR-097 Releases: one workflow, checked before and verified after; main moves on at onceT9.26 (OPS-004, owner request 2026-10-07: cut v0.1.0, and make releases deterministic for future agents).
Context
T9.26 (OPS-004, owner request 2026-10-07: cut v0.1.0, and make releases deterministic for future agents). Tag builds already published signed binaries, versioned images, and a chart (ADR-038, ADR-046), but cutting one was a manual tag push, nothing checked the version against Cargo.toml or the chart, nothing confirmed the result, and edge builds kept the released version's number.
Decision
- A
releaseworkflow (manual, inputX.Y.Z) runsdeploy/release/preflight.sh: X.Y.Z with no suffix, equal toCargo.toml's and the chart's version, a new tag, a notes filedeploy/release/notes/vX.Y.Z.md, and a passing CI run onmain's head. Only then does it tagmain's head. - A tag pushed with the workflow's token starts no workflows, so the release workflow starts the image workflow on the tag itself (
workflow_dispatch). The image workflow's publish jobs run for pushes and dispatches onmainorv*tags. - The tag's build ends with
verify(deploy/release/verify.sh): signatures, the binary's version and channel, "latest release", the four image tags as one multi-arch image, and the chart's version and appVersion. The release is done when it passes. - Right after tagging,
mainmoves to the next version (default: next minor) withdeploy/release/bump.sh, because semver sortsX.Y.Z-edge.NbelowX.Y.Z. The workflow then starts CI and edge for that commit. - A published tag is never moved or reused; a broken release is followed by a patch release.
- Release notes are hand-written per version; GitHub's generated changelog is appended.
Consequences
- Releases are one button with the same checks every time;
docs/releasing.mdis the procedure for maintainers and agents. - The workflow pushes one commit to
mainper release. If branch protection is added later, the bump step needs an exception or becomes a pull request.
Requirements OPS-004
ADR-098 DNSSEC: library limits are indeterminate; bogus names are logged while trying validationRunning [dnssec] mode = "permissive" on the live cluster (2026-10-07) counted 14 bogus answers in five minutes.
Context
Running [dnssec] mode = "permissive" on the live cluster (2026-10-07) counted 14 bogus answers in five minutes. hickory 0.26's validator returns an error both for a broken proof and for its own limit ("exceeded max validation depth", logged at ERROR); TelltaleDNS treated every non-network error as bogus, so validate mode would have answered SERVFAIL for such names. Its log named none of them. On prod.ftl.netflix.com (unsigned zone, CNAMEs) hickory repeats NS and DS lookups for the same name until the limit. Raising the limit from 26 to 40 overflowed a runtime thread's stack in a debug build.
Decision
- The depth limit stays at hickory's default (26). Hitting it is indeterminate: the answer is fetched again unvalidated and served without AD, in both modes. Other validation errors stay bogus.
- In permissive mode every bogus answer is logged at INFO with its name, type, and EDE code (
DNSSEC: bogus answer served (permissive mode)), so a trial shows which names would fail. - hickory's
hickory_net::dnsseclog target is off: TelltaleDNS counts and logs each verdict itself (that module warned on every unsigned NODATA).
- Unsigned zones proven by TelltaleDNS (T10.9). The trial also showed hickory judging names in unsigned zones bogus when reached through CNAMEs (
www.netflix.com,www.amazon.com,www.bbc.co.uk: EDE 10), though it validated the parent's denial of their DS correctly. Before accepting a bogus verdict, an error, or a validation that timed out, TelltaleDNS proves the zone unsigned itself: walking down from the domain under the TLD, the first zone apex (it has its own SOA, from an ordinary query) whose DS the parent validly denies (secure NSEC, or NSEC3 opt-out, which validates as insecure) is unsigned. If every owner in the answer and the question is in such a zone, the answer is insecure. Proven zones are remembered for 15 minutes (at most 4,096), and questions under them aren't handed to hickory again (which also avoids its NS/DS loop). - Only the DS denial is trusted: whether a name is an apex comes from an unvalidated query, but a signed zone's DS can't be validly denied, so a spoofed apex answer can't make a signed zone insecure. A name inside a signed zone that isn't a cut (
www.isc.org) is skipped because it has no SOA of its own. - Correction (2026-10-07, review 03-01): the paragraph above is wrong for every name in a signed zone except its apex. A validated denial of a DS proves only that the name has no DS, and every ordinary name in a signed zone has none; the SOA comes from an unvalidated query, so a forger supplies it. Together they let a spoofed answer under a signed zone pass as insecure (and be remembered for 15 minutes). The denial must also show that the name is a delegation: the NSEC owned by the name (or the NSEC3 whose owner is the name's hash) lists NS and lists neither SOA nor DS (RFC 4035 §5.2, RFC 5155 §8.9; the check hickory applies in its own insecure-delegation case). An NSEC3 opt-out denial is still read as unsigned, as hickory has already checked it. The SOA query stays as a cheap pre-filter, never as evidence. The text above is kept as the record of what was first decided.
- Deadline and limits (2026-10-07, review 03-05, owner's answer: give up at twice the query budget and serve the conservative verdict). A validation holds one of the pipeline's in-flight permits (
spec/02§8.4), and the proof walk could run for far longer than the client's budget (up to 3 × budget for the lookup, then up to 12 more queries of one budget each). Now the whole validation, the lookup and any unsigned-zone proof, ends within two query budgets; the lookup may use three quarters of that, so a lookup that loops (the Netflix case) is cut off in time for the proof to rescue it, and each step of the proof gets only what the deadline leaves. At most 8 proofs run at once; one that finds no slot is "not proven". A name whose zone could not be proven unsigned is not walked again for one minute. All three can only make the answer more conservative: not proven means the bogus verdict stands, or, if the lookup produced no answer, stale or SERVFAIL as for any upstream failure. The cold-chain allowance for the lookup drops from three budgets to one and a half.
Consequences
- Switching to
validatecan't SERVFAIL a name only because hickory gave up or misjudged an unsigned CNAME chain. Checked live (Quad9, 2026-10-07): invalidatemode the Netflix, Amazon, BBC, and Apple names resolve (insecure),isc.organdexample.comvalidate with AD, anddnssec-failed.orgis refused (SERVFAIL); repeat questions are answered from the cache. - A bogus verdict costs a few DS and SOA queries the first time a zone is seen (cached after). The first question under a looping name can still take about 1.5 s.
- hickory reports a negative answer whose proof isn't secure as an error carrying the proof (
DnsError::Nsec); for a DS,Insecureis an NSEC3 opt-out denial, read as unsigned like a secure one. - Open (cold starts): right after a start, Quad9 was seen closing DoT connections with queries unanswered (logged as
upstream connection ended why="closed by the server"); a validation that loses. DNSKEYthat way caches failures and calls signed zones bogus for a while. The pool now retries such a query once on a fresh connection, which fixed most cases in testing but not every one. Until the cause is known,permissivestays the recommended mode.
ADR-099 Connection limits per listener, and per client addressCode review of v0.2.0, pass 01 (owner's answer to question 2, 2026-10-07: a per-address cap of 32, configurable).
Context
Code review of v0.2.0, pass 01 (owner's answer to question 2, 2026-10-07: a per-address cap of 32, configurable). Every stream listener capped connections globally at 1,024 and nothing per client, so one host could take every slot (TCP, DoT, DoH) until a restart; the QUIC listeners had no cap at all (01-02). The spec doesn't say how a client is told apart, or what a balancer does to the count.
Decision
[[listen]] max_connections(default 1,024) andmax_connections_per_address(default 32,0= no limit) ontcp,dot,doh,doh3, anddoqlisteners. Over a limit, a connection is refused or closed at once and counted (telltale_{tcp,doh,doq}_rejected_totalfor the overall limit,…_rejected_per_address_totalfor the per-address one).- The address counted is the client's: after the PROXY protocol v2 header when
proxy_protocolis on (a balancer's own address is never counted for everyone), and after the QUIC handshake fordoqanddoh3(before it, the source address can be spoofed, which would let an attacker spend a victim's budget). An IPv4 client is one address; an IPv6 client is its /64, because a host rotates through its prefix; IPv4-mapped IPv6 addresses count as IPv4. - No address is exempt (loopback included): a limit of 32 is far above what a client needs, and operators who proxy many clients through one address set the limit higher or
0.
Consequences
- Behind a load balancer that hides client addresses (no PROXY protocol, source NAT, Kubernetes
externalTrafficPolicy: Cluster), every client shares one count, so the default 32 can starve legitimate TCP/DoT/DoH clients.docs/running.mdsays so next to the key; the Helm chart's default exposure (externalTrafficPolicy: Local) keeps client addresses. - Changing a limit restarts that listener (the existing replace-on-change rule); open connections finish first.
- The count is one short lock per accepted connection, off the UDP path.
ADR-100 The advertised EDNS payload size is [dns] edns_payloadCode review of v0.2.0, finding 01-12 (owner's answer, 2026-10-07: a configuration key).
Context
Code review of v0.2.0, finding 01-12 (owner's answer, 2026-10-07: a configuration key). DNS-005 says "a configurable advertised UDP payload size (default 1232)", but the value was a constant in the pipeline's settings and no key reached it. The config has no section for answer behaviour, and the spec doesn't name one.
Decision
A new [dns] section with one key for now, edns_payload (default 1232, 512 to 4096: the UDP workers' send buffer is 4,096 bytes, crates/telltale-net/src/sys.rs). It needs a restart (listed with the other restart-only sections); other answer-shaping settings can join the section later.
Consequences
Operators on a jumbo-frame or fragmentation-safe network can raise the size; the default and the DNS Flag Day reasoning are unchanged. Upstream-facing payload sizes (edns_payload per upstream, spec/04) are separate and untouched.
Requirements DNS-005
No decision matches. Try fewer words, or another status or area.