For the technical audience

Decisions ADR-051 to ADR-100

Every architecture decision record (ADR) behind TelltaleDNS: what we knew, what we chose, and what it costs. This page is built from the specification on every change, so it's always the current record.

ADR-051 Manual failover v1: roles and epochs, key sharing with eligible nodes, fencing, orphaned versionsAcceptedCLUT5.4 implements CLU-005 under spec/12 §5 and ADR-048.

Context

T5.4 implements CLU-005 under spec/12 §5 and ADR-048. The owner's topology is two eligible nodes and no witness, so the manual mode comes first: T5.4 is split into T5.4a (manual promotion, fencing, conflicts, config authority) and T5.4b (witness and quorum election with leases, plus the telltale-sim partition checker). T5.1–T5.2 tied "primary" to "holds the CA key", which makes promotion impossible.

Decision

  • Role is explicit:
    • cluster.json gains role (primary or replica) and the epoch the role belongs to.
    • A node publishes only while it's primary in the highest epoch it has seen.
    • Every Hello, heartbeat and manifest carries the sender's epoch.
  • Signed member registry: each manifest carries the node registry (ID, site, eligible, advertise URLs), signed with everything else. The primary records nodes as they join (nodes.json). Every node then knows every eligible node and its URLs, so replicas can find a new primary without a new token.
  • Eligible nodes hold the signing key (spec/12 §4).
    • The primary sends the CA key over the mTLS stream to peers that the registry marks eligible and that are connected with a certificate for that node ID.
    • It's stored owner-only like the primary's.
    • Non-eligible nodes (Kubernetes resolver pods, CLU-009) never receive it.
    • A promoted node can then sign manifests and issue certificates. Compromising an eligible node compromises the cluster, which is the same trust the spec gives eligible nodes. Rotation comes with the CA rotation in T5.4b.
  • Who dials whom:
    • Replicas dial the current primary: the highest-epoch node known as primary first, then the other eligible URLs.
    • Eligible nodes also keep a stream to every other eligible node, so a returning old primary learns about a newer epoch right away.
  • Manual promotion:
    • How: telltale cluster promote on an eligible node, or the Cluster page's "Promote this node" (admin; T5.7 adds TOTP confirmation).
    • What it does: sets epoch = max seen + 1 and role = primary, then starts publishing at once without a restart. The first manifest of the new epoch records its base: the (epoch, seq) it had applied.
    • Refused when:
      • the node isn't eligible;
      • it lacks the key;
      • it's not authority-matching under ADR-048: a gitops cluster needs config_source = "gitops" on the node, unless --emergency.
    • Emergency primaries coordinate but publish no new configuration: every node keeps the last authoritative version.
  • Fencing: a node that sees a higher epoch (in a Hello, heartbeat or manifest) stops publishing and becomes a replica of that epoch's primary at once. Snapshots and manifests from a lower epoch than a replica's applied one are rejected.
  • Orphaned versions (spec/12 §5 rule 3):
    • When a fenced old primary had published versions in its epoch after the new primary's base, it keeps the last one under cluster/conflicts/ (manifest, config blob, and the changed paths compared with the new primary's version).
    • It shows them on the Cluster page under Conflicts, and in GET /api/v1/cluster. They are never applied silently.
    • Re-applying a conflict on the current primary arrives with write forwarding (T5.7); until then the page shows what changed, and the owner re-makes the change on the primary.
  • Config authority (ADR-048):
    • cluster init --config-authority gitops|api is recorded in the signed registry (default api).
    • [cluster] config_source = "gitops" | "file" is node-local (the Helm chart sets gitops), reported in Hello, and shown per node on the Cluster page.
    • Under gitops authority, API config writes return 409 gitops_managed on every node.

Consequences

  • With the owner's homelab node as the GitOps primary, the Pi becomes an emergency primary candidate: if k8s is down for long, promoting the Pi keeps the cluster coordinated while configuration stays frozen. ADR-049 (Git source) later makes the Pi a full candidate.
  • Existing clusters migrate in place: the CA holder becomes role = primary at epoch 1, and eligible replicas receive the key on their next connection.

Amended 2026-10-08 (review 05-01, 05-02; amendment Proposed): standing is the primary's decision, not the member's. A join with the bootstrap secret (ADR-058) is always an ephemeral member, never eligible or a witness, so the key share never reaches a resolver pod; a Hello can't raise a known member's eligibility (a member the registry doesn't know is still taken at its word once). Fencing by epoch (Hello, heartbeat, and the election's view of the highest epoch) counts only voters in the signed registry, so a member that can't vote can't make the primary step down with an unsigned frame. A stricter follow-up would fence only on a verified, signed manifest.

Requirements CLU-005 CLU-009

Builds on ADR-048 ADR-049 ADR-058

ADR-052 Node-local overrides: node-only records, and warn (not refuse) on shared settings in a replica's fileAcceptedCLUT5.5 (CLU-006).

Context

T5.5 (CLU-006). spec/12 §4 says a node's own overrides allow listen addresses, cache size, query-log retention, worker count, site, and local-only records, and that overriding anything else is "rejected at startup with a clear error". ADR-047 already makes the node-local sections (node, cluster, listen, api, auth, telemetry, cache) stay with each node, and replaces everything else with the primary's.

Decision

  • Node-only records: [[record]] node_only = true keeps a record on the node whose file has it. A primary leaves it out of the shared configuration, and a replica keeps its own node-only records next to the cluster's.
  • Shared settings in a replica's file are ignored and reported, not refused.
    • At every load the replica logs a warning naming the sections.
    • The Cluster page gains a failing node_settings check with the fix (remove them, or mark node-only records).
    • The spec's "reject at startup" would make a node's upgrade, or a cluster join, refuse to start over settings that have no effect. DNS would stop over a cosmetic problem, against rule 5 and CLU-004. The owner's Pi is exactly this case: its file predates joining.
  • No separate node.toml file is needed: a node's own config file plays that role, and the node-local sections are already defined by ADR-047.

Consequences

the owner's Pi shows a failing node_settings check until its file is trimmed to node-local settings. Promoting to the spec's strict refusal stays possible later, as an opt-in [cluster] strict_overrides = true.

Requirements CLU-004 CLU-006

Builds on ADR-047

ADR-053 Federated reads v1: what's exact, what's approximate, and which scopes existAcceptedCLUOBST5.6 (CLU-002, OBS-012).

Context

T5.6 (CLU-002, OBS-012). spec/12 §6 says analytics reads fan out to the peers in scope and merge, with HDR histograms merged exactly, query logs k-way merged with per-node cursors, a 2 s per-peer timeout, and missing_nodes. It names the scopes node:<id>, site:<name> and cluster.

Decision

  • Transport: reads are RPCs over the existing cluster streams (frame types RpcRequest/RpcResponse, kind api.read, JSON body). No new port, connection, or certificate. Peers are asked concurrently with a 2 s deadline. A peer without a stream is reported missing at once.
  • Merges:
    • counters sum per bucket, across every breakdown;
    • top lists add counts and error bounds per key (the true count stays within [count − errorBound, count]);
    • query-log pages interleave newest first, with a cursor (fed: + base64url JSON) that holds each node's "before" time, or nothing once that node has run out.
  • Latency is approximate in v1. Each node sends its percentiles, and they are weighted by query count; the maximum is exact. Shipping histograms for an exact merge needs a serialized HDR format on the wire. It's deferred until a dashboard needs exact cluster-wide tails.
  • Scopes: cluster (the default) and node:local. node:<id> and site:<name> still answer 400 unsupported_scope. The live tail, configuration reads, and Explain stay local.
  • Edge cases (accepted):
    • Rows that share the exact microsecond with a page's last row from the same node may be skipped on the next page.
    • A node that joins while someone is paging appears only from the next fresh search.
    • missingNodes reports the latest federated read on the node; concurrent reads may each see the other's list.
  • Failure: a peer error, timeout, or undecodable answer drops that node from the result and names it in missingNodes. If this node's own query log is off, but peers answered, their rows are returned and this node is listed as missing. DNS answering is never involved (CLU-004).

Consequences

any node's UI is a full management view for reads. Exact cluster-wide latency, named-node/site scopes and a federated live tail are follow-ups.

Requirements CLU-002 CLU-004 OBS-012

ADR-054 Write forwarding v1: the entry node vouches for the user over mTLSAcceptedCLUT5.7 (CLU-002).

Context

T5.7 (CLU-002). spec/12 §6 says a write on any node is forwarded to the primary with the user's identity, "signed by the receiving node", so the audit log shows both the user and the entry node. Users aren't replicated yet (ADR-045 is still to come), so a user exists only on the node they signed in to.

Decision

  • What's forwarded: the configuration writes the API offers (devices, local names, forwarded domains), including dry runs. They go to the primary as the RPC api.write over the existing cluster stream, with a 10 s deadline.
  • Who vouches:
    • The entry node authenticates and authorizes the user with its own users and roles, exactly as for a local write.
    • The primary trusts the entry node's claim because the stream is mutually authenticated with the cluster CA: the peer ID comes from the certificate, never from the request.
    • No extra signature is added: the mTLS channel already proves which node sent it, and a member node is trusted to replicate configuration anyway.
  • What's recorded:
    • The primary stores the entry as made by <user> via <site> and audits it with actor kind cluster and remote = node <id>.
    • The entry node audits it under the user's name, with the primary's answer.
  • Versions: a replica reports the primary's configuration version (ETag; 1 s deadline, falling back to its own), so If-Match works through any node. The primary's answer, including its error code (409, 404, 400), is returned unchanged.
  • Unreachable primary: the write is refused with 503 and a hint. It is never queued: a queued write could apply long after the user saw it fail.
  • Not forwarded: under a GitOps authority, writes are refused on every node (409 gitops_managed, ADR-048). Promotion and the node's own users, tokens and sessions stay local until ADR-045 replicates identities.

Consequences

the owner can manage the cluster from the Pi's UI as well as the homelab node's, as long as both are on the API authority. Today's cluster is GitOps-managed, so there it stays read-only, by design.

Amended 2026-10-08 (review 05-03; amendment Proposed): "a member node is trusted to replicate configuration anyway" no longer covers every member since ADR-058. Forwarded writes and the reads that change the answering node (pause, resume, cache flush) are refused from members the registry marks ephemeral: they serve no UI, so no user's change arrives through them.

Requirements CLU-002

Builds on ADR-045 ADR-048 ADR-058

Referenced by ADR-067 ADR-103

ADR-055 Query-log ship mode v1: closed segment files, delivered once, searched where they landAcceptedCLUT5.8 (CLU-007).

Context

T5.8 (CLU-007). spec/12 §7 describes ship mode: raw events are streamed to a target in batches of columnar blocks, spilled to a 64 MiB buffer when the target is unreachable, and replayed; rollups are shipped as per-minute aggregates; both mode does both; receivers keep per-source-node segments.

Decision

  • Unit of shipping: whole closed segment parts, not blocks.
    • In ship mode the writer closes a part at least every [telemetry.ship] interval_secs (default 300 s).
    • The shipper sends each closed part (every part except the newest) to the target in 512 KiB chunks over the cluster channel (RPC qlog.put).
    • The receiver checks the whole file's BLAKE3, then files it unchanged under qlog-nodes/<sender-id>/. Nothing is re-encoded, and the sender's footer and bloom filters stay valid.
    • Only after the receiver confirms does the sender delete its copy. A lost confirmation means the file is sent again and overwritten, never duplicated.
  • Buffer: the local log is the store-and-forward buffer, bounded by buffer_bytes (default 64 MiB) via the existing retention, which drops the oldest file first. No separate in-memory spill: a RAM disk under data_dir gives the memory variant without new code.
  • Exactly one copy: a row is either in the sender's buffer or on the receiver. The receiver's search includes shipped logs, labelled by node, and federated reads reach the sender's unshipped rows, so a cluster-wide search sees each row once.
  • Target: [telemetry.ship] to (a node ID or site), defaulting to the primary. A primary in ship mode with no other target keeps its log (logged once).
  • Deferred:
    • Rollup shipping. Per-minute counts stay on each node: they're small, the dashboard already federates them, and shipping them only matters for ephemeral pods (T5.10). Done in T9.3: ship-mode nodes send their last 30 minutes of minutes every interval_secs (≤ 60 s); the target keeps them per node for 7 days (shipped_minute), and the cluster view adds only those of nodes that didn't answer live, so nothing counts twice.
    • both mode. It would need dedup between a live sender and its shipped copy. Not needed for the hybrid Pi + k8s setup.
  • Safety:
    • Receivers accept only from cluster members (mTLS), only hex node IDs become paths, and segments over 256 MiB are refused.
    • Shipping is a background task, and receiving runs on blocking threads, so DNS is never involved (CLU-004).
    • The peer RPC handler is now installed whether or not the node runs the API, so a node without an API can still receive logs and answer federated reads.

Consequences

the owner's Pi can set mode = "ship" to move its query log to the homelab node's volume. Rows then appear with up to interval_secs of delay in the homelab node's own search; cluster-wide search shows them at once.

Requirements CLU-004 CLU-007

ADR-056 Automatic failover v1: one vote per epoch, voter-granted leases, pre-vote, a vote-only witnessAcceptedCLUT5.4b (CLU-005).

Context

T5.4b (CLU-005). spec/12 §5 has witness and quorum modes: a candidate needs a majority of eligible nodes plus witness for epoch + 1, and promotes only after the old primary's lease (15 s, renewed every 5 s) has expired; "a primary accepts writes only while it holds a valid lease". The AC is a simulator: 10k randomized partition schedules, never two writers in one epoch, orphaned writes always surfaced.

Decision

  • One protocol for both modes (telltale_cluster::election, pure, no I/O):
    • Voters are the eligible nodes plus witnesses, from the signed registry.
    • A voter grants each epoch once (persisted, fsync, before answering). It refuses a newer epoch while the lease it granted to another node runs (15 s on its own clock), and refuses candidates whose applied version is behind its own.
    • The primary renews with every voter every 5 s. It treats its lease as ending 15 s − 2 s after it sent the round a majority granted, so it stops publishing before any voter of that majority can elect someone else, for clock rates within ±1 %.
    • Pre-vote: a node first asks whether it would win. A node cut off from the cluster never inflates epochs, and never deposes a healthy primary when the partition heals.
    • witness and quorum are the same thing with different voters, so the mode is just manual or auto.
  • Mode:
    • telltale cluster set-failover auto|manual on the primary. It's carried in the signed manifest, like the authority.
    • auto runs elections only with three or more voters, this node one of them; otherwise it behaves as manual, and the Cluster page's failover check says why.
    • Manual promote is refused while elections run, because it would bypass the votes.
  • Witness:
    • telltale cluster join <token> --witness --advertise <url>, then telltale cluster witness.
    • It runs only the cluster channel and the vote handler: no DNS, lists, API or pipeline. It learns the registry and mode from verified manifests.
    • It never becomes primary and never receives the CA key.
  • Transport:
    • Votes are the elect.ask RPC on cluster streams. The candidate is bound to the stream's certificate: a node can only ask for itself.
    • Voters keep a full mesh in auto mode: of each pair, the lower node ID dials.
  • Gating:
    • Cluster::may_publish() is false for a primary without a valid lease. The publish loop waits, and API writes get 503 with a hint.
    • A primary starting in auto mode begins with publishing paused until its first renewal.
    • A restarted primary resumes its epoch if its own ballot is for itself; otherwise it takes part in a normal election.
    • Only nodes holding the CA key stand as candidates.
  • Under a gitops authority, an elected node that isn't Git-managed becomes an emergency primary (ADR-048).

Verification

  • tests/sim.rs covers 2 + witness, 3, and 4 + witness voter sets under random asymmetric partitions, 0–15 % loss, heavy-tailed delays to 3 s, crashes with persisted ballots and histories, and ±1 % clock drift.
  • It checks:
    • one primary per epoch;
    • never two writable primaries at the same real instant;
    • every write kept, or reported as orphaned;
    • a primary within 60 s of healing.
  • Results at 10,000 schedules (release, in CI): 43,744 elections, 1.49 M writes, 176,933 orphaned and reported, worst recovery 24.75 s.
  • It's mutation-checked: removing the lease check, or timing the lease from the reply instead of the request, is caught.
  • deploy/cluster/failover-e2e.sh (real processes, CI): kill the primary, and the replica is elected in about 14 s with 100 % of DNS queries answered; the old primary returns and follows.

Deferred

CA rotation, the other part of T5.4b's title, is now T5.4c. Clocks are assumed not to jump by more than the margin; NTP slews rather than steps in normal operation.

Amended 2026-10-08 (review 05-05; amendment Proposed): the clock assumption above didn't hold for the hosts this runs on. A Raspberry Pi has no RTC: it starts on the time it shut down with, and NTP steps it, by minutes or hours, once DNS works. With leases as wall-clock times, a forward step made every lease a voter had granted look expired at once. So:

  • Leases are timed on the election clock (telltale_cluster::clock): CLOCK_BOOTTIME on Linux (read through rustix, no unsafe), which never steps and counts time spent suspended; Instant elsewhere. Nothing on the wire changes: Ask and Reply carry no times.
  • A ballot records which process's clock its lease is on (clock). A ballot from another process (a restart) or from an older build (wall clock, clock = 0) is taken as granting a full lease from now (Ballot::adopt), and stored that way at once so later reads don't extend it again. That is never shorter than promised, and costs at most 15 s of extra waiting after a voter restarts.
  • The simulator restarts nodes on a new, unrelated clock (anywhere from far behind to far ahead) and adopts their ballots the same way. Mutation-checked: forgetting the lease on adoption is caught ("two writers") within 2,000 schedules.

Consequences

the owner's Pi + homelab cluster stays manual until a witness exists. A witness on any third device (for example the NAS) makes failover automatic, with the Pi as an emergency primary while the configuration authority is GitOps.

Requirements CLU-005

Builds on ADR-048

ADR-057 Node certificate renewal over the cluster channel, same keyAcceptedCLUT5.4c (CLU-001).

Context

T5.4c (CLU-001). Node certificates last 90 days (ADR-044), and nothing renewed them, so a cluster would lose its links about 90 days after its nodes joined. The owner's cluster joined on 2026-10-04. spec/12 §3 says certificates renew automatically; the CA rotation half of T5.4c is rarer, because the CA lasts 10 years.

Decision

  • When: each node checks every 6 hours (first one minute after start) and renews when fewer than 30 days remain, so two thirds of the lifetime has been used. It retries every 10 minutes on failure.
  • Same key: the node sends a CSR for its existing key, so its node ID (derived from the key) never changes.
  • Who signs:
    • A node holding the cluster key (the primary, eligible nodes) signs its own.
    • Others ask the primary with the cert.renew RPC on their stream.
    • The primary signs only when the CSR's key belongs to the peer the stream's certificate names. It takes the certificate's names (advertise hosts) from the signed registry, never from the request.
  • Checks before use: the new certificate must name this node and verify against the cluster CA (Ed25519). It's written atomically.
  • Switch-over: a certificate generation counter makes the dialer and the listener rebuild their TLS settings for the next connection. Open streams keep their session, since certificates are checked at handshake only.
  • CA rotation (new CA, cross-signing, key re-share) stays in T5.4c, still open.

Consequences

clusters keep their links indefinitely while a CA holder is reachable at least once every 30 days. A witness or replica cut off from every CA holder for 90 days loses its certificate and must rejoin with a token.

Requirements CLU-001

Builds on ADR-044

Referenced by ADR-066

ADR-058 Kubernetes resolver pods: a shared bootstrap secret with proof of possession, ephemeral members, sync-gated readinessAcceptedCLUT5.10 (CLU-009).

Context

T5.10 (CLU-009). spec/12 §2 says resolver pods join with a long-lived token from a Secret, get a fresh identity per start, expire after ephemeral_ttl (10 min) without heartbeats, and display under their site. A join token pins the CA fingerprint, which only exists after the controller creates the cluster. So a chart can't put a token in a Secret without the controller writing to the Kubernetes API, which needs RBAC and a client library.

Decision

  • Bootstrap secret instead of a pre-made token:
    • The chart generates a random secret in a Secret (kept with lookup and resource-policy: keep, or the user's own via existingSecret), mounted in every pod.
    • The controller accepts it like a join token that never expires: [cluster] bootstrap_secret_file, re-read at every join, so rotations apply without a restart.
  • Proof of possession instead of trust on first use:
    • A pod with join_url first calls POST /cluster/v1/ca with a fresh nonce.
    • It trusts the returned CA only if the server answers HMAC-SHA256(secret, nonce : CA fingerprint), proving it holds the same secret.
    • It then pins that CA and joins exactly as with a token.
    • A man in the middle without the secret can't pass this.
  • First start creates or joins:
    • [cluster.init] (advertise URLs, authority) creates the cluster on the controller's first start, the same as telltale cluster init.
    • join_url joins on a pod's first start, retrying for 2 minutes. An ephemeral pod that can't join exits, so Kubernetes retries it (CrashLoopBackOff while the controller starts).
  • Ephemeral members:
    • [cluster] ephemeral = true joins as ephemeral: never eligible, never a voter, no CA key.
    • The registry records ephemeral. The primary drops ephemeral members not connected and not heard from for ephemeral_ttl_secs, checked every minute. A pod that never connected is measured from when it joined, or from the controller's start.
    • The Cluster page groups them by site with counts (up, ready, q/s, behind).
  • Readiness: an ephemeral node is ready only once it has applied the cluster's configuration. A new pod never serves an unfiltered answer, and the Service routes to it only after sync.
  • Helm mode: scaled:
    • The allInOne StatefulSet becomes the controller and keeps its labels, so switching modes needs no reinstall.
    • Adds a resolver Deployment (emptyDir data, ship-mode query log, maxUnavailable: 0), a -cluster Service for the cluster port, the join Secret, and a NetworkPolicy rule for the cluster port.
    • The DNS Service selects every pod; the API and metrics Services select the controller.
    • hostNetwork with scaled is refused, since pods would compete for port 53; that's daemonSet, later.
  • Found on the way: a replica's follower ignored a manifest that arrived before it subscribed, and waited for the primary's next change. It now processes the current one at start.

Consequences

  • deploy/helm/scaled-e2e.sh (CI, this build's binary on kind) proves the controller and two pods join and sync, a pod answers DNS, an outside node joins through the cluster port, and a deleted pod is replaced and expires.
  • A pod restart is a new member, by design. A registry of long-gone pods is cleaned within ttl + 1 min.

Amended 2026-10-08 (review 05-01, 05-06; amendment Proposed): what makes a pod untrusted is now enforced by the primary: a join with the bootstrap secret is ephemeral whatever it asks for. The primary's POST /cluster/v1/ca proof answers anyone, so a weak secret could be brute-forced offline from collected proofs: a secret shorter than 32 bytes is warned about at startup (not refused, so existing deployments keep joining), and the docs ask for 32 random bytes or more. The chart generates 48 characters.

Requirements CLU-009

Referenced by ADR-051 ADR-054

ADR-059 Version compatibility v1: a protocol window, a schema stamp, and refusal over guessingAcceptedCLUT5.11 (CLU-010).

Context

T5.11 (CLU-010). spec/12 §9 says RPC carries a protocol version and manifests a schema version; nodes accept N and N−1; the primary refuses to emit features the oldest connected node doesn't support, and warns. Until now, peers needed the exact same protocol, and manifests carried no schema. The shared configuration is parsed strictly (deny_unknown_fields), so an older replica would reject a newer primary's new settings.

Decision

  • Protocol window: a stream is accepted when the peers' protocols differ by at most one. Frames are protobuf, and fields are only ever added, so unknown fields are ignored. PROTOCOL is raised only when a message changes meaning.
  • Schema stamp: manifests carry schema, the configuration schema the primary writes (now 1; 0 means from before stamping). It's raised when the shared configuration gains settings an older build would reject.
  • No down-converting: the primary does not try to strip settings an older replica wouldn't know. It publishes what it has.
    • A replica that can't read it keeps serving its last version, so DNS is unaffected.
    • The replica records a sync error naming both schemas and saying to upgrade.
    • Per-field feature gating is deferred: the documented upgrade order (replicas first) makes it rarely matter, and guessing which settings are safe to drop could silently change filtering.
  • Visibility: each node's protocol is on the Cluster page, and a versions check fails while versions or protocols are mixed, with the upgrade order as the fix. Build versions become fully distinguishable with T6.9's stamped build identity; today edge builds all say 0.1.0.

Verification

deploy/cluster/upgrade-e2e.sh (CI) runs the published edge binary as N−1 against this build:

  • both nodes on N−1;
  • the replica on N following the N−1 primary;
  • the primary on N;
  • the replica back on N−1 following the N primary.

An edit reaches the replica at every stage, and a client with both servers configured gets an answer to every query.

Addendum (2026-10-05): the shared configuration leaves out top-level sections that are at their defaults, and a replica reads a missing section as the default. Without this, adding any new section (DNSSEC, ADR-060) broke N−1 replicas even when no one used it: CI's upgrade test caught it. A new setting now reaches an older replica only once someone actually sets it. That is the case the "no down-converting" rule is for.

Requirements CLU-010

Builds on ADR-060

Referenced by ADR-064

ADR-060 DNSSEC validation v1: hickory's validator over our upstream groups, off by defaultAcceptedDNST6.1 (DNS-011).

Context

T6.1 (DNS-011). spec/03 §5 asks for:

  • modes off, validate (the default once stable) and validate_permissive;
  • DO=1 upstream, validation from the root trust anchor, and CD=1 clients getting unvalidated data;
  • AD only when validated and the client asked;
  • bogus answers as SERVFAIL with EDE 6;
  • negative trust anchors;
  • hickory-proto's verification primitives behind a Validator.

In hickory 0.26 the chain-of-trust logic (DnssecDnsHandle, with NSEC/NSEC3 denial proofs, wildcard checks, NSEC3 iteration limits and a validation cache) lives in hickory-net. The primitives in hickory-proto alone would mean writing that ourselves.

Decision

  • Use hickory-net's DnssecDnsHandle (dnssec-ring, tokio; it adds no TLS or HTTP stacks). It wraps a DnsHandle we implement over one of our upstream groups.
    • The question, and every DNSKEY and DS lookup the proof needs, go to the same upstreams with DO=1 and CD=1 (we check, so the upstream mustn't drop what it thinks is bogus).
    • There's one handle per group, so its validation cache lasts across queries.
    • It's wrapped as telltale_upstream::dnssec::Validator.
  • Verdict: the weakest proof among the answer records, or among the authority records for NXDOMAIN and NODATA. Errors from the validator other than upstream failures count as bogus.
  • Serving:
    • Secure gets AD for clients that asked; insecure and indeterminate get no AD.
    • Bogus becomes SERVFAIL with EDE 6, never a stale answer, and is never cached; in permissive mode it's served and counted.
    • RRSIG, NSEC and NSEC3 records are stripped for clients without DO.
    • Validation runs on cache misses only, and validated answers are cached with their AD bit. The cache key already separates DO and CD.
  • Negative trust anchors: [dnssec] negative_trust_anchors plus every route with dnssec_nta.
  • Mode names and default: off, validate, permissive (spec's validate_permissive). The default stays off until it has run on the owner's nodes for a while; then a follow-up makes validate the default.
  • Timing: a cold chain (root, TLD and zone keys) takes several sequential lookups. Validation gets three times the query budget, so a slow first lookup still lands in the cache even if that client gave up.
  • Found on the way (all queries): a truncated UDP answer's TCP retry shared the adaptive per-attempt timeout calibrated on small answers (about 3× average round trip), so large answers (DNSKEY sets with signatures, long TXT) timed out. The TCP retry now gets the configured timeout of its own.

Verification

unit tests for verdicts, NTA coverage and stripping. deploy/dnssec-e2e.sh (CI, real DNS tree, forwarding to 1.1.1.1 and 9.9.9.10) checks:

  • ietf.org and cloudflare.com are secure with AD;
  • google.com is insecure without AD;
  • dnssec-failed.org is SERVFAIL with EDE 6, and answered with CD;
  • signed NXDOMAIN and NODATA validate;
  • verdicts are counted.

Deferred

  • RFC 5011 trust-anchor updates (the built-in root keys cover KSK-2017 and KSK-2024);
  • aggressive NSEC caching (RFC 8198, with recursion);
  • per-reason EDE codes (7, 9, 10, 12) instead of 6;
  • the dnssec field in query events and the Settings UI.

Requirements DNS-011

Referenced by ADR-059

ADR-061 Pi-hole import v1: a reviewed starting configuration, faithful group semanticsAcceptedAPINFRT6.3 (API-007; spec/08 §8) asks for an importer of Pi-hole v5 and v6 Teleporter archives covering adlists, domain lists, groups, clients, local DNS/CNAME records, upstreams, and conditional forwarding, with a report of every unmapped setting.

Context

T6.3 (API-007; spec/08 §8) asks for an importer of Pi-hole v5 and v6 Teleporter archives covering adlists, domain lists, groups, clients, local DNS/CNAME records, upstreams, and conditional forwarding, with a report of every unmapped setting. The spec doesn't say whether the result is applied or reviewed, how Pi-hole's per-entry group assignments map onto our per-group list sets, or what happens to Pi-hole behaviors we don't have.

Decision

  • Output, not apply: telltale import pihole PATH writes TOML for review (like import zone, ADR-041), validated to load before it's written. Applying through the API (and the UI's import page) waits for T6.7's restore path.
  • Inputs: v6 zip (pihole.toml and a gravity.db of the group tables), v5 tar.gz (JSON per table plus setupVars.conf, pihole-FTL.conf, custom.list, dnsmasq.d), a directory, or a bare gravity.db. Readers are ours: zip and tar.gz in about 200 lines on miniz_oxide, with CRC checks and size limits. SQLite goes through rusqlite, already in the tree. Archived databases are read from a private temporary copy.
  • Groups stay exact: each group gets exactly the lists Pi-hole gave it, so default doesn't fall back to "every list". Domain entries become inline lists, one per (type, set of groups). Disabled groups get no lists. Clients in no group go to a pihole-no-group group with no lists, as Pi-hole blocks nothing for them.
  • Matching:
    • exact entries use match = "exact";
    • a regex our parser would read as a plain name is wrapped as /regex/;
    • entries our engine can't run are reported, not dropped silently.
  • Upstreams: they go in the default upstream group with strategy fastest, matching Pi-hole's preference for the fastest server.
  • Conditional forwarding: each rev server becomes its own upstream group and route. The route covers the domain and the network's reverse zones: non-octet prefixes are expanded, so a /23 is two /24 zones, never widened. It gets a negative trust anchor.
  • Devices: DHCP reservations become named devices (MAC and IP join a matching client, or a new one named by the host name). The DHCP server is not imported.
  • Report:
    • v6: every setting Pi-hole marks ### CHANGED that isn't mapped;
    • v5: every unmapped key in setupVars.conf and pihole-FTL.conf, except the old install's own plumbing (interface, addresses, web server);
    • names only, never values, since they include password hashes.
  • Not imported: clients matched by host name or interface (we match by address); the IP blocking modes; query history; passwords and tokens.

Verification

  • Unit tests on hand-written inputs in both formats.
  • deploy/pihole-import-e2e.sh (CI) configures the official pihole/pihole images (v6 2026.09.0 = FTL v6.7.1, v5 2024.07.0) through their own tools, exports Teleporter archives, imports them, and serves the result: local names, a CNAME's TTL, exact and regex denies (;querytype), the blocking mode, and group scoping.
  • Pi-hole-generated files aren't committed (their comments are Pi-hole's text, NFR-006).

Consequences

  • Merging into an existing configuration is manual (both define default).
  • The importer's e2e showed that the first blocking snapshot waited for every list's first fetch, including retries, so an unreachable adlist delayed all blocking at first start. Fixed: the fetcher signals the compiler at most 10 s (settle) after the first update in a round, so lists that arrive together still compile once.

Requirements API-007 NFR-006

Builds on ADR-041

Referenced by ADR-062

ADR-062 Technitium import v1: read the API, not the backupAcceptedAPIT6.4 (API-007, P1).

Context

T6.4 (API-007, P1). spec/08 §8 says "backup zip → blocked/allowed zones, block-list URLs, forwarders + protocols, Advanced Blocking app config → groups". Technitium's backup holds dns.config, zone files, and the allowed/blocked lists in its own versioned binary serialization. That format is undocumented and changes between releases. Its HTTP API returns the same data as documented JSON.

Decision

  • Source: telltale import technitium URL reads a running server through the API with an API token, which comes from TECHNITIUM_TOKEN or --token-file, never argv.
  • Calls, all read-only:
    • settings/get;
    • zones/list, then zones/records/get per zone;
    • allowed/export, blocked/export;
    • apps/list, and apps/config/get for Advanced Blocking;
    • dhcp/scopes/list and dhcp/scopes/get (optional).
  • Output: reviewed TOML like ADR-061, validated before it's written.
  • Mapping:
    • Forwarders go in the default group: concurrent forwarding becomes parallel with the same fan-out, but only with two or more usable forwarders. name (ip:port) pins the address, with the name as TLS name.
    • Forwarder zones: forwarders by priority, a route, and an NTA unless every forwarder of the zone validated.
    • Primary zones: records of the types we serve, with their TTLs.
    • Blocked and allowed zones become subtree inline lists.
    • Advanced Blocking groups become groups. The catch-all network's group becomes default, and other networks become networks (ADR-050). URL lists are shared across groups by URL. Regexes are wrapped as /re/. Server-wide lists are added to every group, since Technitium applies both. Disabled groups get no lists. The bypass list becomes a group with no lists.
    • Blocking answer: NXDOMAIN, or custom addresses, become the group's block_mode.
    • Rate limit: the per-/32 limit (queries per minute) becomes [ratelimit].
    • Other settings: dnssecValidation and saveCache carry over. DHCP reservations become named devices.
  • Not imported (reported): secondary and stub zones; QUIC forwarders; forwarders with no address; listener- and host-based group selection; per-list answers; the DHCP server; encrypted listeners; recursion ACLs, TSIG, zone transfers, the proxy, and other apps. Listeners, logging, users, and tokens are not imported either.

Verification

  • Unit tests on hand-written API responses.
  • deploy/technitium-import-e2e.sh (CI) runs against technitium/dns-server:15.6.0. It configures everything above through Technitium's API, installs Advanced Blocking from Technitium's store, and creates an API token. It then imports, serves the result, and checks the answers.

Consequences

  • Importing needs the old server running, which matches the migration path (import, compare, switch).
  • A backup-zip reader can follow if the format gets documented.
  • The e2e found a real-data bug: concurrent forwarding with a single usable forwarder produced an invalid parallel group. Fixed and tested.

Requirements API-007

Builds on ADR-050 ADR-061

ADR-063 Backups v1: a checked archive of config and local data, without the cluster keyAcceptedAPIT6.7 (API-007, P0) asks for Teleporter parity: one archive with config and local data, optionally with query history.

Context

T6.7 (API-007, P0) asks for Teleporter parity: one archive with config and local data, optionally with query history. spec/08 §8 sketches telltale ctl backup create [--include-qlog] writing a .ttbk (tar.zst + manifest + signature) restored onto a new primary. ctl (the API-wrapping CLI, API-008) doesn't exist yet. The spec doesn't say what the signature is signed with or whether cluster identity is included.

Decision

  • Commands: telltale backup create | show | restore, local commands that work on the data directory.
    • create is safe while the server runs: SQLite databases are copied consistently with VACUUM INTO, and query-log segments only grow, so the size seen at open is copied.
  • Contents:
    • config files under config/;
    • state.db (users, tokens, UI-made entries, audit), with sessions and idempotency keys removed;
    • rollups.db and anomaly.json;
    • the query log (qlog/, qlog-nodes/) only with --include-qlog;
    • lists, snapshots, the cache, and setup-token are left out, since they're rebuilt or transient.
  • Format: a tar stream compressed with zstd (frame checksum on), with manifest.json last. It holds the format version, source node and paths, and each entry's size and BLAKE3 hash. The file is written owner-only and renamed into place when complete.
  • No signature in v1: a signature by the source node's key proves nothing to the fresh node it's restored onto, which doesn't know that key. Integrity comes from the manifest hashes and zstd's checksum. Confidentiality is the file's permissions, as with Pi-hole's Teleporter. Encryption with a passphrase is the natural next step.
  • No cluster identity by default: cluster/, with the CA key, isn't included. A portable file with the CA key could impersonate the cluster. A restored member starts standalone, and the manifest records that it was a member so restore can say what to do.
  • Restore:
    • streams into a staging directory under the data directory;
    • checks every entry against the manifest, and anything unexpected, a path outside config/ or data/, or .., fails before any file is moved;
    • refuses to replace existing files without --force;
    • chowns restored files to the data directory's owner.

Verification

  • Unit tests: contents, permissions, round trip, session stripping, --force, corruption, path containment, and the query log.
  • deploy/backup-e2e.sh (CI) backs up a running node with an admin and an API-made name, restores it onto a second node, and checks the following on that node:
    • the name answers;
    • the admin signs in, and the old session doesn't carry over;
    • the audit log is present;
    • damaged archives are refused.

Consequences

  • Download: GET /api/v1/backup (admin, audited backup.create, without the query log) and a Settings → System button. Restore through the API is deferred: restoring under a running server is risky.
  • --include-cluster-key (for disaster recovery of a lone primary) and passphrase encryption are follow-ups.

Requirements API-007 API-008

ADR-064 Agent principals v1: scoped tokens, deny by default, a reason for every changeAcceptedAGTT6.5 (AGT-001..005, AGT-009).

Context

T6.5 (AGT-001..005, AGT-009). spec/13 §2 asks for:

  • agent tokens with an owner, scopes, an optional group restriction, rate limits, and expiry;
  • attribution agent:<token> (owner: <user>) with the client and a required reason;
  • dry runs with impact estimates on every mutation;
  • guardrails and a cluster-wide kill switch;
  • an OpenAPI document written for agents and linted.

Tokens so far had a role-like scope (read/write/admin), and the audit log already recorded token:<name> (owner: <user>) (ADR-038).

Decision

  • Agent tokens: POST /tokens with kind: "agent".
    • The scopes are AGT-004's list: analytics:read, querylog:read, config:read, config:write:{clients,records,forwards} (* for all), ops:pause, ops:cache, cluster:admin.
    • The default is analytics:read + config:read: read-only, without the query log.
    • Optional group and ratePerMinute.
    • Each scope needs a role from the creator (config:write:* and ops:* need operator, cluster:admin needs admin). A token's effective role is the lower of its owner's and its scopes'.
    • Stored in three new tokens columns (migration 5). User tokens are unchanged.
  • Deny by default: one table maps method and path to a scope. Anything not in it is refused for agents: users, tokens, passwords, backups, the audit log, and routes added later until they're mapped. The role checks still apply on top.
  • Group restriction: honored where a group view exists.
    • The query log and top lists: the middleware replaces the group parameter.
    • Devices: listed for the group only; writes only into, and only on devices already in, the group.
    • Everything else is refused for restricted tokens rather than shown unfiltered.
  • Every change needs a reason: agents' non-GET requests without X-Telltale-Reason get 400.
  • Attribution: the actor is agent:<token> (owner: <user>) via <X-Telltale-Client>, with kind agent.
  • Rate limits: a per-token token bucket in memory, refilled per minute, default 120 ([agents] rate_per_minute). Over the limit: 429 with Retry-After. Per node, so a cluster allows N× in total; documented.
  • Kill switch: [agents] enabled, a shared section, so it replicates to every node (and, since ADR-059's addendum, doesn't trouble N−1 replicas while unset). It's applied on start and every reload.
  • Impact estimates: device, name, and forward changes return recentQueries and impact, a sentence with the numbers. They're counted from the in-memory top lists for the current and previous hour (a lower bound, cheap, no log scan). Promotion gets ?dryRun=true, returning a PromotePlan (epoch, emergency, impact) from the same checks without acting.
  • Lint: .spectral.yaml (spectral:oas plus operation summary, description, tags, operationId, success response, and parameter descriptions) runs in CI. Every response now has a description.

Verification

  • Unit tests:
    • the scope map denies by default;
    • scope parsing and the roles scopes imply;
    • the kill switch, rate limit, required reason, and group refusal;
    • the forced group parameter.
  • API tests: agent tokens over HTTP (scopes, refusals, reason, attribution, kill switch); group-restricted tokens; the promote dry run.
  • deploy/agent-e2e.sh (CI) on a running node:
    • a dry run of every mutation reports an impact and changes nothing;
    • a real change is attributed with its reason;
    • a reload with enabled = false stops the agent and not the admin.

Consequences

  • MCP (T6.6) builds on these principals.
  • ops:pause and ops:cache exist ahead of their endpoints.
  • Per-group analytics views beyond top lists would let restricted agents see more.
  • OAuth for agents (AGT-008) stays P1.

Amended 2026-10-08 (review 06-02, 06-09; amendment Proposed): an agent's pre-save list check returns how many lines couldn't be used and why, not their text (the node would otherwise read any URL it can reach for the agent). Viewers list only their own plans; operators and admins see all.

Requirements AGT-001 AGT-004 AGT-008 AGT-009

Builds on ADR-038 ADR-059

ADR-065 MCP server v1: read-only tools over the REST routes, one implementation for two transportsAcceptedAGTT6.6 (AGT-006, AGT-008 bearer part, AGT-009; ADR-010).

Context

T6.6 (AGT-006, AGT-008 bearer part, AGT-009; ADR-010). spec/13 §3.1 lists the read-only tools, and §4 suggests a telltale-mcp crate with schemas generated from the Rust types (schemars) and a catalog snapshot test. The AC asks for:

  • an MCP conformance test;
  • scope and privacy authorization tests;
  • a scripted agent completing the "why is the TV slow" story.

Decision

  • Where: a module of telltale-api (mcp.rs), mounted at /mcp on every node's API listener behind the same authentication. A separate crate would add nothing at this size. POST /mcp takes JSON-RPC 2.0 (single or batch) and answers JSON (Streamable HTTP without SSE; GET /mcp is 405). Protocol 2025-06-18, also answering 2025-03-26 and 2024-11-05. Methods: initialize, ping, tools/list, tools/call. Notifications get 202.
  • Tools call REST in-process: each tool builds one or more GET /api/v1/... requests and sends them through the REST router with the caller's own Authorization/cookie. Scopes, group restriction, rate limits (one per inner call), privacy levels, and the kill switch apply exactly as for REST, with no second authorization model. The REST problem+json becomes the tool's error text.
  • Catalog (read-only): get_overview, top_items, search_queries, explain_decision, get_client_profile (the device, then its queries by IP), latency_breakdown, upstream_health, list_effectiveness, find_anomalies, cluster_status, get_config (one section).
    • Each description starts "Read-only.", and each tool carries readOnlyHint.
    • test_resolution is deferred: it needs resolution without cache or log, which the pipeline doesn't offer yet.
  • Schemas: hand-written JSON Schemas, frozen in the committed docs/api/mcp-tools.json and diffed by a test (UPDATE_MCP=1 to regenerate). Generating them from the REST parameter types would tie tool arguments to query-string shapes; the snapshot still prevents drift.
  • Guardrails: results are capped at 32 KiB (with a note) and 200 rows. Each initialize starts a session (Mcp-Session-Id), and the client name/version from it is passed on as X-Telltale-Client, so agent actions are attributed to the MCP client (AGT-005).
  • stdio: telltale mcp --stdio relays newline-delimited JSON-RPC between stdin/stdout and a node's /mcp with a token, so there's one implementation of the tools. The node address comes from --url or the config files; nothing but protocol goes to stdout.
  • Agents and /mcp: POST /mcp needs no scope of its own and no reason header (the tools are read-only; the inner calls are checked). Group-restricted tokens may use it, since their inner calls are narrowed.

Verification

  • Unit tests: the catalog snapshot; side-effect statements and caps; arguments become queries.
  • An API test: initialize with a session, list, call, scope error inside a tool, unknown method, no credentials.
  • deploy/mcp-e2e.sh (CI) runs the official @modelcontextprotocol/sdk client against a live node. Over Streamable HTTP, the listed tools must match the committed catalog. The test then plays the "why is the TV slow?" story (device profile with recent queries, upstream health, explain on a blocked name), repeats it over telltale mcp --stdio, and checks that a token without querylog:read gets an error from search_queries.

Consequences

  • Write tools with plan/apply and approvals (§3.2) are P1.
  • OAuth for MCP clients (AGT-008) is P1.
  • MCP resources and prompts are later.
  • An SSE stream (GET /mcp) would be needed only for server-initiated messages.

Amended 2026-10-08 (review 06-04, 06-07; amendment Proposed): the REST calls behind a tool carry the /mcp request's client address (OnBehalfOf, an extension only in-process requests can carry), so changes made through MCP are audited from the agent's address, not 0.0.0.0. They don't take from the agent's rate-limit bucket again: the rate counts tool calls, and each message of a JSON-RPC batch, which holds at most 16 messages. A resource cut at 32 KiB says so.

Requirements AGT-005 AGT-006 AGT-008 AGT-009

Builds on ADR-010

ADR-066 CA rotation: two trusted CAs, three member-gated phasesAcceptedCLUT5.4c (CLU-001, CLU-005).

Context

T5.4c (CLU-001, CLU-005). spec/12 and the roadmap ask for:

  • CA rotation: a new key, a cross-signed transition, node certificate renewal over the channel, and the old CA retired;
  • re-sharing the key with eligible nodes.

ADR-057 already renews node certificates (same key) over the channel. A CA rotation changes the trust anchor every TLS handshake and every manifest signature depends on.

Decision

  • Two trusted CAs instead of cross-signing: during a rotation ca.crt is a bundle of both CAs. TLS trusts every certificate in it, so certificates from either CA verify. Manifest signatures and issued certificates are checked against any CA in the bundle. The bundle's first CA is the one that signs, and ca.key is always its key. Cross-signing adds nothing in a cluster whose members all receive the bundle.
  • Distribution:
    • The primary's signed manifest carries ca_bundle. A node adopts a new bundle only if it keeps at least one CA the node already trusts, so trust moves step by step and a stolen CA alone can't replace it.
    • Witnesses follow manifests for this alone.
    • Eligible nodes get the next CA's key with the existing key share, now re-sent whenever it changes. A key that arrives before its CA is trusted is held (ca-pending.key) until a verified bundle includes it.
  • Phases, driven by the primary, each gated on every non-ephemeral registry member (heartbeats now carry trust_fp, the CAs a node trusts, and issuer_fp, the CA of its certificate):
    1. trust: the bundle is old + new; the old CA signs.
    2. switch: once all trust both, the new key becomes ca.key (the old is kept as ca-old.key) and the bundle becomes new + old. Nodes see that their certificate isn't from the signing CA and renew within a minute (self-issued with the new key, or over cert.renew). A node whose ca.key doesn't match the signing CA asks the primary rather than sign with the old key.
    3. finish: once every certificate is from the new CA, the bundle is the new CA alone, and the old key and the rotation state are deleted.
  • Operator interface: telltale cluster rotate-ca starts a rotation on the primary while it runs, and the running primary carries it out (it re-reads its identity every cycle). --status and a ca_rotation Cluster check show the phase and pending members. Each new CA's name carries a label, so the two trusted CAs never share a subject.
  • Fixed on the way:
    • Configuration fetches used the TLS settings from start-up, so they'd fail after a rotation. They'd also present an old certificate after a renewal. They now read the current identity.
    • Witnesses never renewed their certificates (ADR-057 gap); they do now.
    • Cluster files were made owner-only only after being written, and two streams could collide on one temporary name. Each write now uses its own temporary name, and secrets are created owner-only.

Verification

  • Unit tests:
    • bundles: verify, issuer, order-independent fingerprint;
    • start: refuses a second rotation;
    • adoption: refuses unrelated trust, promotes a held key, switches, then retires.
  • deploy/cluster/rotate-e2e.sh (CI) with a primary, an eligible replica, and a witness under DNS load:
    • all three phases complete, and every node ends trusting only the new CA, with a certificate from it;
    • the replica holds the new key, and no old, next, or pending key is left;
    • every query was answered during the rotation;
    • a configuration change replicates afterwards (signed with the new key), and a restarted replica reconnects and syncs.

Consequences

  • A member offline for the whole rotation must rejoin.
  • Join tokens pin the CA fingerprint, so tokens from before a rotation stop working.
  • A forced rotation (with members down) is a possible follow-up.

Corrected 2026-10-08 (review 05-10): "a stolen CA alone can't replace it" overstates the rule. The rule stops a CA the node doesn't trust from replacing trust; it doesn't contain a stolen current CA key, whose holder can rotate trust to a CA of their own in three manifests, exactly as the legitimate primary does. That is inherent (the key signs everything): a suspected compromise of the cluster key means re-creating the cluster, as docs/running.md now says.

Requirements CLU-001 CLU-005

Builds on ADR-057

ADR-067 Quick rules: their own layer, the most specific scope wins, expiry on the serverAcceptedCLUFLTT6.12 (owner request 2026-10-05) wants everyday per-person rules: The engine can already express device-scoped rules ($client in the manual-rules overlay, ADR-003), but the overlay uses the lists' four tiers (important allow &gt; important block &gt; allow &gt; block).

Context

T6.12 (owner request 2026-10-05) wants everyday per-person rules:

  • unblock a game's server for one phone for two hours;
  • block a site for the kids' devices until tomorrow.

The engine can already express device-scoped rules ($client in the manual-rules overlay, ADR-003), but the overlay uses the lists' four tiers (important allow > important block > allow > block). Those can't say "this device's rule beats its group's rule beats the lists": a group-wide allow would override a parent's block for one child. Nothing expires on its own.

Decision

  • A quick rule:
    • allow or block a domain and its subdomains;
    • for listed devices (by name, IP, or MAC, as [[client]] keys), listed groups, or everyone;
    • an optional expiry (an absolute time; the UI offers durations);
    • a note, who made it (user or agent), and when.
  • Storage: in state.db as a managed entry of kind rule (ADR-040). In a cluster, rules travel with the shared configuration the primary publishes (ADR-047), and a replica's UI forwards the write to the primary (ADR-054). Rules are capped at 1,000 per cluster.
  • Matching: quick rules are their own layer, checked before the lists:
    1. Among rules matching the query name (any suffix of it) whose scope includes the client, a device rule beats a group rule, which beats an everyone rule.
    2. Within one scope, the longer (more specific) domain wins, then allow beats block.
    3. A match decides allow or block outright. Otherwise the lists decide as before.

CNAME inspection (FLT-007) uses the same decision, so a block also catches a CNAME to the blocked name.

  • Hot path: the rules sit in a read-mostly table behind ArcSwap, keyed by the hash of each rule's reversed domain. A lookup probes the query's suffixes, at most one probe per label, with no allocation. With no rules it costs one branch.
  • Expiry: each node ignores a rule from the moment its expiry passes, by its own clock (T6.11 shows clock offsets). A sweep every 5 s on the primary deletes expired rules and audit-logs each expiry. A replica never needs the sweep to stop applying a rule.
  • Explanations: an answer decided by a quick rule is attributed to it ("allowed by a quick rule for Mom's phone, expires 21:30"). The query log records it, and "Why?" and explain show it above the list matches.
  • Agents: the scope config:write:rules, with the usual reason header, dry run, and impact estimate (recent queries the rule would change).

Consequences

  • A new rule takes effect on the next query, with no list recompile.
  • Cost: about 52 ns per query with 1,000 rules, about 4 ns with none (xxh3 suffix keys, measured 2026-10-05).
  • Rules are deliberately simpler than list syntax: no regex, no query types. Anything fancier belongs in a list.
  • Device scoping is only as good as recognising the device; the docs say so.
  • Expiry depends on node clocks being roughly right. A node whose clock is off keeps or drops a rule early or late by that offset, and the Cluster page flags offsets of 2 s or more.
  • Mixed versions (CLU-010): the config schema rejects unknown keys, and an empty rule list is left out of the shared configuration. So an N−1 replica follows the cluster as long as nobody has made a quick rule. Once one exists, the replica refuses that version and keeps serving its last good one until it's upgraded. Quick rules therefore wait for a cluster-wide upgrade, as any new shared section does.

Requirements CLU-010 FLT-007

Builds on ADR-003 ADR-040 ADR-047 ADR-054

Referenced by ADR-003

ADR-068 One process per data directory: an exclusive lock held for the process's lifeAcceptedCLUT6.14 (owner request 2026-10-05).

Context

T6.14 (owner request 2026-10-05). Scaling a single-volume Deployment (or starting a second telltale run with the same configuration) makes two processes share one data directory: the node identity and cluster certificates, state.db, the query-log segments, and the cache dump. Nothing stopped it, and the damage is silent: two nodes with one identity, interleaved query-log writes, and SQLite contention.

Decision

  • telltale run takes an exclusive advisory lock (flock, through std::fs::File::try_lock) on <data_dir>/telltale.lock before anything else touches the directory, and holds it until the process exits.
  • If another process holds it, the new one retries for 10 s (a previous process may still be shutting down), then exits non-zero with an error. The error names the directory and the holder (pid, host or pod name, and start time, which the holder writes into the file), and points to mode: scaled.
  • If the lock file can't be opened at all (a read-only or unusual file system), the node logs a warning and starts anyway: the lock protects against a mistake and must never stop DNS (CLU-004).
  • telltale backup restore takes the same lock without waiting, so it refuses to restore under a running server.
  • Read-only commands (qlog search, explain, backup create) don't take it.

Consequences

  • The operating system releases the lock when the process exits, even after a crash or SIGKILL, so there is no stale-lock cleanup.
  • A rolling update of a single-volume Deployment would wait for the old pod forever. The chart uses a StatefulSet and the homelab Deployment uses Recreate, and the docs tell people with their own manifests to do the same.
  • On network file systems flock may be emulated or local-only. That is acceptable for a guard against mistakes.

Requirements CLU-004

ADR-069 Editing upstreams, lists, and groups from the UI: overrides in state.db, the files untouchedAcceptedT7.5 (owner report 2026-10-06).

Context

T7.5 (owner report 2026-10-06). On a node installed with install.sh, the upstreams and lists come in the starter /etc/telltale/telltale.toml. ADR-040 makes anything the files define read-only through the API ("files win"), so none of them could be changed in the UI. Rewriting the file isn't an option either: Docker and Kubernetes mount it read-only, and on Git-managed nodes the next deploy would replace it.

Decision

  • Upstreams, upstream groups, lists, and client groups made or changed through the API are managed entries in state.db (like ADR-040's), by name. An entry can:
    • add a new name;
    • override a file entry of the same name (the whole entry is replaced);
    • hide a file entry (a tombstone: the name is left out).
  • The files are never rewritten. The UI marks each entry as file, UI, or UI override of the file, and "Revert to the file" deletes the override or tombstone.
  • Every change is validated on the merged configuration before it's stored (a group can't lose its last upstream; a list can't be removed while a group uses it; the default upstream group must keep a member), with dry run, If-Match, idempotency, and audit as for the other managed kinds.
  • In a cluster, the merged configuration is what the primary publishes (ADR-047), so changes reach every node.
  • On a node whose configuration comes from Git (config_source = "gitops"), each change response carries the TOML to add to the repository or the Helm values to make it permanent ("Keep it in Git"), because an override lives only in that node's state.db.
  • Devices, local names, forwarded domains, and quick rules keep ADR-040's rule (files win), since the starter config doesn't define them.

Consequences

  • A standalone node can be run entirely from the UI after install.sh.
  • Two sources of truth for the same name are possible; the UI and the GET endpoints show which one is in effect (source: file, api, or override).
  • A file change to an overridden entry has no effect until the override is reverted; the UI says so on the entry.

Builds on ADR-040 ADR-047

Referenced by ADR-111

ADR-070 Agents' plans: a dry run kept in memory on the node that made it, applied with If-MatchAcceptedAGTFLTOBST7.1 (AGT-007).

Context

T7.1 (AGT-007). A plan_* MCP tool must show what a change would do and let the agent apply it later, only if nothing changed meanwhile and, with [agents] require_approval, only after an operator approves. The spec doesn't say where plans live, how long they last beyond "10 minutes", or how a cluster shares them.

Decision

  • A plan is the REST write (method, path, body) plus its dry-run answer, the config version it saw, the reason, and who asked (the caller's token or session). Plan tools call the REST dry run in-process with the agent's own credentials, so scopes, group restrictions, rate limits, and the kill switch apply unchanged.
  • apply_plan replays the same write with If-Match: <version> and the plan ID as the Idempotency-Key. A 412 marks the plan stale. Only the caller that made a plan can apply or discard it.
  • Plans live in the memory of the node that made them, at most 500, open for 10 minutes and listed for a day. They aren't stored or replicated: a restart drops them, and the agent plans again.
  • With [agents] require_approval = true a new plan is pending until a person with the operator role approves or rejects it (POST /api/v1/plans/{id}/approve|reject, the "Agent changes" page). Agent tokens can never approve or reject. Both decisions are audited.
  • Approval is checked when the plan is made: turning the setting on doesn't hold plans already made, and turning it off doesn't release pending ones.
  • Immediate operations (flush_cache, pause_blocking, resume_blocking) have no plan, as spec/13 §3.2 says; they're audited like any agent write.
  • plan_set_schedule waits for schedules (FLT-010). Alert destinations for pending plans wait for OBS-010.

Consequences

  • Nothing new is persisted, and a plan can't outlive the configuration it was checked against.
  • In a cluster, an operator approves on the node the agent used. Agents should use the node whose UI people use (normally the primary). If agents and people use different nodes, plans would need to move to the replicated state.db.

Requirements AGT-007 FLT-010 OBS-010

ADR-071 MCP over OAuth: JWT access tokens from the OIDC provider, for existing users onlyAcceptedAGTT7.4 (AGT-008, P1).

Context

T7.4 (AGT-008, P1). MCP clients sign in with OAuth 2.1: the server publishes RFC 9728 metadata naming an authorization server, and accepts that server's access tokens. The spec says to use the configured OIDC provider but not how tokens become principals, or which tokens are accepted.

Decision

  • One provider is the authorization server for MCP ([auth.oidc] mcp_provider). TelltaleDNS is only a resource server: client registration, consent screens, and refresh are the provider's.
  • Accepted tokens are JWTs signed with the provider's published keys (asymmetric algorithms only), with its issuer, the audience mcp_audience (default <public_url>/mcp), and a current expiry (60 s leeway). There's no introspection, so opaque tokens are refused.
  • The token's sub must match a user who has signed in to the web UI with that provider (ADR-034). An unknown subject is refused rather than creating an account, because access tokens rarely carry the group claims that decide roles.
  • The principal is an agent of that user: the TelltaleDNS scopes in scope or scp (or the read-only default when there are none), never above the user's stored role; audited as agent:oauth:<azp> (owner: <user>). Rate limits, the kill switch, plans, and approvals apply as for agent tokens.
  • An unauthenticated or refused /mcp request gets WWW-Authenticate: Bearer resource_metadata="…" (and error="invalid_token" when a token was sent).

Consequences

  • Assistants connect with SSO and the user's consent, and no long-lived token is stored in the assistant.
  • Each request verifies a signature (keys cached with discovery, refreshed on rotation); no per-token cache is kept.
  • Providers that only issue opaque access tokens need a token per agent instead, until introspection is added.

Requirements AGT-008

Builds on ADR-034

ADR-072 Event sinks: one bounded buffer per sink, API-shaped JSON, no disk spillAcceptedOBST7.13 (OBS-010; 06 §5).

Amended by

ADR-088. Syslog over TLS and a capped disk spill for webhook sinks were added later.

Context

T7.13 (OBS-010; 06 §5). The spec asks for JSON-lines, syslog, and webhook sinks "with retry and a disk spill cap", but not the event format, how a slow sink is kept off the query path, or how retries are bounded.

Decision

  • An event is the API's query row (QueryRow: the same fields and names as GET /api/v1/queries, names resolved when the event is written, the query log's privacy level applied), so a sink's output and the API's agree.
  • The aggregator thread formats each event once and offers it to every sink with try_send on a bounded channel (max_buffer); each sink writes from its own thread. A full buffer drops the event, counted and logged once a minute. DNS and the query log never wait for a sink.
  • The webhook posts batches (batch events or flush_secs), retries a failed batch twice (1 s, 2 s), then drops it. Events buffer in memory only; the "disk spill cap" isn't implemented: the query log already keeps every event on disk.
  • Syslog is RFC 5424 over UDP or TCP (RFC 6587 octet counting); TLS isn't implemented yet. Severity is notice for blocks and informational otherwise.
  • Sinks are started once at startup (like the query log's writer); a reload doesn't change them.

Consequences

  • A collector outage loses events beyond the buffer; the query log, which sinks don't replace, still has them.
  • Syslog over TLS needs a forwarder (rsyslog, Vector) until it's added.

Requirements OBS-010

Builds on ADR-088

Referenced by ADR-076

ADR-073 DGA scoring and NXDOMAIN storms: fixed formulas, shipped bigrams, absolute storm barAcceptedOBST7.14 (OBS-009; 06 §7).

Context

T7.14 (OBS-009; 06 §7). The spec names the signals (entropy, consonant runs, bigram log-likelihood) and the storm rule shape ("> X/min and > Y%") but not the weights, thresholds, the bigram source, or how findings are grouped.

Decision

  • The bigram table is 27 × 27 signed bytes (presets/bigrams.bin, round(4 × log2 P(next | previous)) with start and end symbols) built from the public-domain dwyl/english-words list, embedded in the binary.
  • The score of a registrable domain's own label (8 to 63 characters, not xn--) is 0.7 × max(bigram, mix) + 0.2 × consonants + 0.1 × entropy, each clamped to 0..1: bigram = (mean bits per letter pair − 6) / 2; mix = (letter/digit switches − 2) / 4; consonants = (longest run − 5) / 3; entropy = (normalized entropy − 0.85) / 0.15. Calibrated on ~200 real service domains (none reach 0.2) and random-letter and hex labels (0.8 to 1.0). Logs come from tables, so the score is identical on every architecture.
  • Only first-seen domains of devices past their learning period are scored for findings; a dga finding needs at least 2 at or above the sensitivity's threshold (0.75, 0.6, 0.5) in one hour, and is one finding per device-hour with samples.
  • An NXDOMAIN storm is absolute (no baseline): at least nxdomain_per_minute (30) NXDOMAIN answers and nxdomain_percent (50 %) of the device's queries in a calendar minute; once per device per hour.
  • The first-seen feed starts after a device's first day (its initial learning would flood it) and keeps the last 2000 entries, persisted with the engine.
  • List overlap is counted at compile time (pairs of lists per shared name); hits are this node's counts since start, not 24 h / 7 d windows from the query log.

Consequences

  • Dictionary-word DGAs and short generated labels aren't caught; some vendor IDs may score high (the UI shows the score, not a verdict).
  • A restart resets list hits; overlap is exact for the active snapshot.

Requirements OBS-009

ADR-074 Our own iterative resolver, conservative defaultsAcceptedDNSUPST7.15 (DNS-012, UPS-012; 03 §6).

Context

T7.15 (DNS-012, UPS-012; 03 §6). 02 §2 allows building on hickory-recursor. The spec fixes the limits (16 CNAME hops, 32 referrals) and asks for QNAME minimization, 0x20 (configurable), SRTT server selection, glue in bailiwick, and RFC 8198, but not the defaults, the timeouts, or how the resolver meets the per-query budget.

Decision

  • telltale-recursor is our own iterative resolver on telltale-proto (no new dependencies): exact control over minimization, bailiwick rules, and limits, and nothing added to the image beyond its code.
  • It's a transport of telltale-upstream (recursive://), so groups, hedging, health, the answer cache, and DNSSEC validation (which then fetches DNSKEY and DS through it) work unchanged. Its attempt timeout is at least 3 s (not the 1 s cap of forwarders); the query's overall budget (2 s) still bounds a client's wait, and zone cuts learned meanwhile stay cached, so a retry continues from there.
  • Defaults: QNAME minimization on, relaxed (RFC 9156 §2.3: the full name after NXDOMAIN or failure on a minimized query; at most 10 minimized steps); 0x20 off (a few authoritative servers don't echo case; when on, such servers are remembered and asked plainly); IPv6 off (many home networks lack an IPv6 route); root hints built in, no priming query.
  • Bailiwick: answers only for names under the zone being asked; glue only for server names under the referring zone; upward or sideways referrals are lame. A server name inside its own zone without glue is skipped.
  • Caches: zone cuts with addresses and server-name addresses (20,000 entries each, TTLs clamped to 30 s..1 day), server SRTT (EWMA). Final answers are cached by the answer cache only.
  • Limits: 16 CNAME/DNAME hops, 32 referrals per lookup, 4 nested lookups for server names, 96 queries per client question, 4 servers tried per question, 1.2 s per server at most.
  • RFC 8198 (aggressive NSEC) is deferred: it needs validated NSEC records kept below the validator.

Consequences

  • The common "Pi-hole + unbound" setup becomes one upstream; imports from Technitium without forwarders use it directly.
  • A cold resolution in a zone with slow or broken servers can exceed the 2 s budget the first time (the client gets SERVFAIL or a stale answer; the next query succeeds from the learned cuts).

Requirements DNS-012 UPS-012

ADR-075 Proxies, socket plugins, pins, and client certificatesAcceptedUPST7.16 (UPS-010, UPS-011; 04 §2, §6).

Context

T7.16 (UPS-010, UPS-011; 04 §2, §6). The spec names the features but not where a proxied hostname is resolved, how an exec:// plugin learns its socket, how plugins restart, or how pins interact with tls_insecure_skip_verify.

Decision

  • Proxies (SOCKS5 with RFC 1929 credentials, HTTP CONNECT) carry tcp://, tls://, and https:// upstreams; others are a configuration error. A hostname URL goes to the proxy by name (SOCKS5 ATYP 3, CONNECT host:port), so no bootstrap is needed and no lookup leaks (Tor). Proxy addresses are IP literals (or localhost). UDP through SOCKS5 (UDP ASSOCIATE) stays P2.
  • Socket plugins speak RFC 7766 framing over a Unix socket, on the same pipelined connection pool as TCP upstreams. exec:// plugins get TELLTALE_PLUGIN_SOCKET (<data_dir>/plugins/<upstream>.sock) and TELLTALE_UPSTREAM in their environment, args as arguments, stdin closed, stdout and stderr logged per line; they restart with backoff (1 s doubling to 60 s, reset after a minute up) and are killed when their upstream is dropped (each reload rebuilds upstreams, so a reload restarts plugins).
  • Pins are base64 SHA-256 of the leaf's SubjectPublicKeyInfo, checked after the normal WebPKI verification; with tls_insecure_skip_verify only the pin is checked. tls_ca adds roots for one upstream; tls_client_cert/tls_client_key (PEM) enable mTLS. Files are read at load and reload.
  • Deferred: DoH GET, the WASM upstream ABI (P2).

Consequences

  • Tor users point an encrypted upstream at the Tor SOCKS port with a hostname URL and nothing else.
  • A plugin that needs state across reloads must keep it itself (it's restarted on configuration changes).

Requirements UPS-010 UPS-011

ADR-076 OTLP over HTTP with JSON, metrics converted from our expositionAcceptedOBST7.17 (OBS-006; 06 §5).

Context

T7.17 (OBS-006; 06 §5). The spec asks for "the same metric set via OTLP/HTTP" and optional QueryEvent logs, without an encoding or a mapping.

Decision

  • OTLP/HTTP with the JSON encoding (the protobuf JSON mapping: 64-bit integers as strings). No OpenTelemetry SDK and no generated protobuf types: the binary stays small, and every OTLP/HTTP receiver accepts JSON.
  • Metrics are converted from our own Prometheus exposition every interval_secs: counters → cumulative monotonic sums (start time = process start), gauges → gauges, histograms → explicit-bucket histograms (per-bucket counts from the cumulative ones). Names and labels are kept as in /metrics, so dashboards and docs apply to both.
  • Query events as logs are an event-sink format (otlp_logs, ADR-072): batching, buffering, and retries are the sink's. Body: a one-line summary; attributes use OpenTelemetry DNS-style keys where they exist (dns.question.name, client.address), telltale.* otherwise.
  • No OTLP/gRPC.

Consequences

  • Collectors that convert back to Prometheus see _total already in counter names (most keep it as is).
  • A counter reset (restart) shows as a new start time, as OTLP expects.

Requirements OBS-006

Builds on ADR-072

ADR-077 dnstap: a sampled copy of the wire messages, off by defaultAcceptedOBST7.18 (OBS-007; 06 §5).

Context

T7.18 (OBS-007; 06 §5). dnstap needs the query and response messages, which query events don't carry (02 §3: no allocation on the hot path).

Decision

  • The tap is a OnceLock on the pipeline set at startup: with dnstap off, the query path pays one atomic load. With it on, one query in sample_every (a relaxed counter) is copied, both messages, into a bounded channel with try_send; full means dropped and counted. Measured with bench-smoke: no change with it off (within WSL run-to-run noise).
  • The writer thread speaks bidirectional Frame Streams (READY, ACCEPT, START; STOP at the end) to a Unix socket or TCP reader, reconnects with backoff (1 s to 30 s), and writes CLIENT_QUERY and CLIENT_RESPONSE (address, family, transport including DoT/DoH/DoQ, times, messages; identity = node name, version = TelltaleDNS version). Protobuf is encoded by hand: the schema is small and fixed.
  • Deferred: FORWARDER_QUERY/FORWARDER_RESPONSE (they'd need a tap in the upstream exchange), and the client's port (not on the event path).

Consequences

  • At sample_every = 1 every query costs two small allocations and a channel send; the docs say to sample on busy resolvers.

Requirements OBS-007

ADR-078 DHCPv4: authoritative, broadcast replies, leases name devicesSupersededT7.19 (OPS-008; 08 §7).

Superseded by

ADR-091. The DHCP server was removed on 2026-10-06; this record stays for history.

Context

T7.19 (OPS-008; 08 §7). The spec lists the options, static leases, and a lease file, but not the server's stance toward clients asking for foreign addresses, how replies reach clients without an address, or how leases "feed client naming".

Decision

  • A module of the binary (crates/telltale/src/dhcp.rs), its own task and socket (socket2: SO_BROADCAST, SO_REUSEADDR, optional SO_BINDTODEVICE), never on the DNS path.
  • Authoritative for its subnet (like dhcp-authoritative in dnsmasq): REQUEST for an address this client may not have (someone else's, outside the pool, another subnet) gets NAK; INIT-REBOOT for a free pool address is granted. Offers are held 60 s; DECLINEd addresses are skipped for 10 minutes; no ping-before-offer.
  • Replies (RFC 2131 §4.1): to giaddr:67 via a relay, to ciaddr:68 when the client has an address, else broadcast to 255.255.255.255:68 (no raw-socket unicast to an unconfigured client: broadcast is allowed and needs no extra capability).
  • Options 1, 3, 6 (default: the server's address), 15, 28, 42, 51, 54, 58, 59, 119 (RFC 3397, uncompressed; split per RFC 3396 when long).
  • Leases persist to <data_dir>/dhcp-leases.json after every change; the live view (address → lease) is published on the pipeline, so device names fall back to the lease host name (a configured client name wins), and GET /api/v1/dhcp/leases reads it.
  • In a cluster, one node runs it (documented); other nodes don't see its leases yet.

Consequences

  • A Pi can replace the router's DHCP; unnamed devices get readable names immediately.
  • Without failover, a second DHCP node needs split pools (not supported in config yet).

Builds on ADR-091

Referenced by ADR-091

ADR-079 DNSCrypt with RustCrypto's crypto_box; stamps map to existing transportsAcceptedUPST7.24 (UPS-003; 04 §2).

Context

T7.24 (UPS-003; 04 §2). DNSCrypt v2 needs X25519 and libsodium's box constructions; 02 §7 lists no crate for them. Stamps may name DNSCrypt, DoH, DoT, or DoQ servers.

Decision

  • New dependency crypto_box 0.9 (RustCrypto; its SalsaBox and ChaChaBox are libsodium's crypto_box and crypto_box_curve25519xchacha20poly1305, on the curve25519-dalek already in the tree) and ed25519-dalek (already in the tree) for certificates. The IETF XChaCha20-Poly1305 AEAD is a different construction and wouldn't interoperate. The detached API is used and the tag written first (libsodium's _easy layout); verified live against five providers' servers.
  • One X25519 key per upstream for its lifetime (like dnscrypt-proxy's default); a fresh random nonce per query; ISO 7816-4 padding to 64 bytes, at least 256 over UDP; TCP when the answer is truncated.
  • Certificates: TXT records of the provider name fetched in clear, each checked with the stamp's Ed25519 key (verify_strict) and its validity dates; the newest construction, then the highest serial, wins; refreshed after an hour or at expiry.
  • DoH/DoT/DoQ stamps become those transports (host name as the TLS name, an address in the stamp pinned); the stamp's certificate hashes aren't enforced yet.
  • Not supported: anonymized DNSCrypt relays, DNSCrypt stamps' props flags (DNSSEC, no-log, no-filter are informational).

Consequences

  • Users of dnscrypt-proxy's server list can paste stamps directly.
  • About 100 KiB more in the binary.

Requirements UPS-003

ADR-080 Router integrations: read-only polling of UniFi and OPNsenseAcceptedAPIT8.2 (M8 "router integrations"; API-010 lists DHCP leases as a naming source).

Context

T8.2 (M8 "router integrations"; API-010 lists DHCP leases as a naming source). Most homes run DHCP on the router, so TelltaleDNS's own DHCP (T7.19) doesn't see those leases.

Decision

  • [[router]] entries are polled read-only (interval_secs, 300; 15 s after a failure). UniFi: stat/sta (connected clients) with an API key (X-API-KEY), or a local user's login (UniFi OS /api/auth/login and /proxy/network/..., falling back to the classic /api/login); the UniFi alias wins over the host name. OPNsense: the lease search of Dnsmasq, then Kea, then ISC, with the API key and secret (basic auth).
  • The result replaces that router's part of a lease view next to the DHCP server's; device names fall back to configured clients, then DHCP leases, then router leases. GET /api/v1/dhcp/leases lists both with a source.
  • Self-signed consoles: tls_ca (the console's certificate), or tls_insecure_skip_verify with a configuration warning. Secrets come from files.
  • Not done: OPNsense static mappings without a lease, UniFi's offline known clients (rest/user), pfSense, OpenWrt.

Consequences

  • Device names appear on networks where the router runs DHCP, with no change on the router beyond a read-only account or key.

Requirements API-010

ADR-081 mDNS device naming: passive, A records of <name>.localAcceptedAPIT8.3 (M8 "mDNS client naming"; API-010 lists mDNS as a naming source).

Context

T8.3 (M8 "mDNS client naming"; API-010 lists mDNS as a naming source). Many devices announce their host name over mDNS (RFC 6762) whether or not anyone asks.

Decision

  • [clients] mdns (off by default): join 224.0.0.251 on the mDNS port (5353, address and port reuse) and read A records owned by a two-label <name>.local in the answer and additional sections of responses others send. Never send queries or responses (no probing, no load on the network). IPv4 only, like the DHCP and router names.
  • The latest name per address is kept in memory (4096 at most, the oldest tenth dropped when full) and published every 5 s; device names fall back to configured clients, DHCP leases, router leases, then mDNS.
  • Service instance names (Living Room._airplay._tcp.local), link-local and unspecified addresses are ignored.
  • Not done: asking (QU queries to fill the gaps), IPv6 (AAAA owners would need the neighbor table to match clients), persisting across restarts, showing the names in the leases API.

Consequences

  • Devices get readable names without a router integration or DHCP, on networks where TelltaleDNS can hear multicast.

Requirements API-010

ADR-082 vqlog: a pipe language compiled to qlog searches, aggregated in memoryAcceptedAGTCLUT8.4, AGT-012: "a constrained, safe query DSL (vqlog) over the query log and rollups (filter, group by, top-K, percentiles, time bucket), with a cost estimate, exposed as one tool".

Context

T8.4, AGT-012: "a constrained, safe query DSL (vqlog) over the query log and rollups (filter, group by, top-K, percentiles, time bucket), with a cost estimate, exposed as one tool". The spec doesn't fix the syntax, the limits, or the scope.

Decision

  • Syntax: stages separated by | — from, where (conditions joined by and; in (…) lists instead of or), bucket, by, stats, top, sort, limit. A fixed grammar parsed into a plan in telltale-api (syntax errors are 400s with hints); no expressions, functions, or joins. The name kept from the spec (vqlog) although the product was renamed.
  • Execution reads the query log only, not the rollups: rollups can't filter by name or client, and percentiles over them would be approximations of approximations. The conditions the qlog index can use are pushed into the search (time, one name, one client, group, statuses, qtypes, rcodes, one upstream, minimum latency); every condition is also checked on each row. Grouping is exact, in memory; percentiles are exact (nearest rank).
  • Cost: a header-only estimate (segments, blocks, and rows that could match, using the same block skipping as search) before the scan, returned with every answer; dryRun returns only that. Above 100 million estimated rows the query is refused. The scan stops at 2 million matches, 100,000 groups, or 25 seconds and reports truncated with the reason; results then cover the newest queries.
  • Access: querylog:read (it reads per-device, per-name rows); not group-aware, so group-restricted agents can't use it. Scope: this node's log plus the logs shipped to it (CLU-007), so the primary of a shipping cluster sees everything; missingNodes as elsewhere.
  • Not done: or between fields, rollup-backed long windows, per-cluster fan-out with mergeable sketches, a UI editor.

Consequences

  • One MCP tool answers most analytics questions with exact numbers and a visible cost; the narrow tools stay for the common ones.
  • Long windows on busy networks are slow (it scans), and the limits say so instead of degrading DNS (search threads run at background priority).

Requirements AGT-012 CLU-007

Referenced by ADR-089

ADR-083 io_uring UDP: measured, not adoptedAcceptedM8 lists io_uring; 02 §3 calls it "a P2 experiment behind a feature flag".

Context

M8 lists io_uring; 02 §3 calls it "a P2 experiment behind a feature flag". The UDP listener already batches with recvmmsg/sendmmsg on one SO_REUSEPORT socket per worker, so the syscall cost per query is already amortized.

Measured (2026-10-06): a standalone spike, not in the repo: one server thread doing the same per-datagram work both ways (64-slot RecvMsg/SendMsg io_uring ring vs 64-message recvmmsg/sendmmsg), loopback, WSL2 kernel 6.18, 8 cores, alternating 5-second rounds.

  • With 4 client threads: 530.8k vs 526.3k answered per round (io_uring −0.9%).
  • With 7 client threads (server about 90% CPU): 456.8k vs 471.6k (+3%).
  • The spread between rounds was about ±15%, so neither difference is meaningful.

Decision

keep recvmmsg/sendmmsg; don't add an io_uring listener or feature. Reasons:

  • No measured gain.
  • More unsafe in the hot path (buffers that must outlive submissions).
  • A second listener to test.
  • io_uring is often unavailable where TelltaleDNS runs: Docker's default seccomp profile blocks it, and some distributions disable it (kernel.io_uring_disabled).

Revisit if a profile on real hardware shows syscalls dominating at the owner's target rates. Multishot receive with provided buffers would be the variant to try.

Consequences

  • One UDP code path.
  • The M8 item is closed with data rather than code.
ADR-084 WASM upstream plugins: deferred, and wasmi rather than wasmtime if builtAcceptedOPSM8 lists WASM upstream plugins; 04 §3 and ADR-007 name wasmtime behind a feature flag.

Context

M8 lists WASM upstream plugins; 04 §3 and ADR-007 name wasmtime behind a feature flag. T7.16 shipped the out-of-process plugins (unix:// and exec://, DNS wire format), which already let any language implement an upstream.

Measured (2026-10-06): stripped, LTO, panic = "abort" release binaries of a program that compiles one module, against an empty program, on x86_64 Linux:

EngineAdds (raw)Adds (gzip -9)
wasmtime 49 (Cranelift JIT, component model)9.4 MiB3.3 MiB
wasmi 2.0 (interpreter, pure Rust)1.7 MiB0.6 MiB

The release binaries are 19–23 MiB raw today, and the image budget is 15 MiB compressed (OPS-001).

Decision

  • Don't build WASM plugins now. There's no user request; the process plugins cover the use case; and a sandboxed host networking API is a design of its own.
  • If they're built, use wasmi behind an off-by-default wasm feature. It costs about a fifth of wasmtime's size, is pure Rust (no JIT, so it runs on armv7 and under seccomp profiles that forbid executable memory), and is fast enough for a per-query function that mostly waits on the network.
  • Revisit on a concrete request, for example a resolver protocol that people want to ship as one file.

Consequences

  • M8 closes without code here.
  • The plugin story stays the process boundary: crash isolation, and any language.

Requirements OPS-001

Builds on ADR-007

ADR-085 Alert email: a small built-in SMTP submission clientAcceptedOBST9.4 (OBS-010 lists email as a destination; deferred in T7.12).

Context

T9.4 (OBS-010 lists email as a destination; deferred in T7.12). The owner has no relay of their own.

Decision

  • A minimal client in the binary (smtp.rs, written from RFC 5321/3207/4954/5322) over the rustls stack already shipped, instead of a mail library: one message per connection, STARTTLS (smtp://, 587) or implicit TLS (smtps://, 465), AUTH PLAIN or LOGIN, the public roots plus tls_ca, base64 text bodies, RFC 2047 subjects.
  • A password is only sent encrypted: STARTTLS can't fall back to plain, and smtp+insecure:// (for local catchers) refuses a username in the configuration check.
  • Validation: CI runs a scripted SMTP server (STARTTLS with a private CA, implicit TLS, a wrong password reported as 535); Mailpit was checked by hand as an independent implementation; real delivery is the owner's choice of provider (a Gmail app password needs no relay).
  • Not done: HTML mail, DKIM (the provider signs), a queue (a failed alert is logged like any destination).

Consequences

email works with any provider that offers SMTP submission, for about 30 KiB of code and no new dependency.

Requirements OBS-010

ADR-086 Recursion and DNSSEC follow-ups: priming, hedging, EDE reasons, an anchors file; RFC 5011 and 8198 deferredAcceptedT9.8 picks up the deferrals of T6.3 (DNSSEC) and T7.15 (recursion).

Amended by

ADR-092. RFC 8198 (aggressive NSEC) was implemented later; RFC 5011 stays deferred.

Context

T9.8 picks up the deferrals of T6.3 (DNSSEC) and T7.15 (recursion).

Decision

  • Root priming (RFC 8109): on first use, the hints are asked for . NS; the answer's glue (or, without it, lookups of the names) becomes the cached root zone cut, renewed when it expires (TTL capped at a day). A failed attempt waits a minute before the next.
  • Hedged queries: a server that hasn't answered after 1.5 × its smoothed RTT (at least 150 ms; 400 ms when unknown) is joined by the next server; the first usable answer wins. It costs at most one extra query per slow step, inside the 96-query budget.
  • EDE reasons: a bogus answer's RRSIGs give the code: none for the records that answer → 10, all expired → 7, all not yet valid → 8 (serial-number time comparison), else 6. The library's validator doesn't expose its reason, so this reads the signatures it returns.
  • Trust anchors: both current root KSKs (20326, 38696) ship built in (a test pins them). [dnssec] trust_anchors_file replaces them with a file of DNSKEY records; a bad file keeps the built-in ones.
  • Deferred — RFC 5011 automated trust-anchor updates: they need a persistent state machine with a 30-day hold-down for a root KSK change that happens every several years, which builds already carry. The anchors file covers the gap.
  • Deferred — RFC 8198 aggressive NSEC caching: it needs a cache of validated NSEC/NSEC3 ranges next to the library's validator and answer synthesis on the query path; worth doing with measurements, not as a follow-up.

Consequences

  • Recursive lookups ride out a slow server and a moved root server.
  • SERVFAILs say why a signature failed.
  • Validation survives the root key rollover with no action.

Builds on ADR-092

Referenced by ADR-092

ADR-087 Upstream follow-ups: DoH GET, UDP over SOCKS5, DNSCrypt relays, per-client ECSAcceptedT9.9 picks up the deferrals of T7.8 (DoH), T7.16 (proxies), T7.23 (ECS), and T7.24 (DNSCrypt).

Context

T9.9 picks up the deferrals of T7.8 (DoH), T7.16 (proxies), T7.23 (ECS), and T7.24 (DNSCrypt).

Decision

  • DoH GET: doh_method = "get" puts the query in ?dns= (base64url, no padding) with ID 0; POST stays the default (smaller requests, no URL length concerns).
  • UDP through SOCKS5: a udp:// upstream with a socks5:// proxy uses UDP ASSOCIATE (RFC 1928 §7). Up to four associations are kept for reuse; one whose control connection or relay fails is dropped. The TCP retry for a truncated answer goes through the proxy too. HTTP proxies still refuse udp:// in the configuration check.
  • Anonymized DNSCrypt: relay (a relay stamp, type 0x81, or ip:port) on a DNSCrypt upstream prefixes every packet with the anonymized-DNSCrypt header (8 × 0xff, 0x0000, the server's address as IPv6, its port), the certificate query included, over UDP and TCP. One relay per upstream; choosing relays at random from a list is left to configuring several upstreams in a group.
  • Per-client ECS (ecs = "client"): the client's source address cut to /24 or /56 (RFC 7871 §11.1). Gaps resolved conservatively:
    • nothing is sent for non-public sources (private, loopback, link-local, CGNAT, ULA);
    • ECS options sent by clients are still never forwarded or echoed, so a downstream forwarder never caches an answer as if it were scoped;
    • the cache key gains the packed subnet (8 bytes per key, 0 when unused), but only for groups that have a "client" member, which is a flag checked once per miss;
    • entries are keyed by the source prefix, not the upstream's SCOPE: never shared more widely than the subnet sent, at the cost of fewer hits;
    • scoped entries aren't written to the cache file, whose format has no subnet.

Consequences

  • DNSCrypt users get the anonymity dnscrypt-proxy's relays give, and SOCKS5 users keep UDP's latency.
  • ecs = "client" multiplies cache entries by the number of client subnets; it's meant for resolvers with public clients, and does nothing on a home network.
ADR-088 DNS64 exclusions and reverse names; syslog over TLS; a webhook spillAcceptedT9.10 and T9.11 pick up the deferrals of T7.21 (DNS64) and T7.13 (event sinks).

Context

T9.10 and T9.11 pick up the deferrals of T7.21 (DNS64) and T7.13 (event sinks).

Decision

  • DNS64 exclusion set (RFC 6147 §5.1.4): dns64_exclude on the group takes networks of either family. An excluded AAAA counts as missing. With all of a name's AAAA records excluded, AAAA records are made up for it. With some left, the excluded ones are removed, along with signatures, since the set they signed changed; DNS64 already never applies to CD clients. An excluded A record is never made into AAAA. ::ffff:0:0/96 is always excluded, as the RFC recommends.
  • DNS64 reverse names (§5.3.1): a PTR question for an address inside the client group's /96 gets a CNAME to the embedded IPv4 address's in-addr.arpa name, resolved like a safe-search target. A private IPv4 address keeps RFC 6303's local NXDOMAIN, so it never goes upstream. Of the RFC's two options, the CNAME was chosen over making up a PTR record: it needs no state and shows where the answer comes from.
  • Syslog over TLS (RFC 5425): tls://host:port, with octet-counted frames as on TCP, rustls with the public roots plus tls_ca, and the certificate checked against the host in the address. No client certificates (none asked for yet).
  • Webhook spill: off unless spill_max_bytes is set (at least 1 MiB).
    • Refused batches are appended as JSON lines to <data_dir>/sinks/<name>.spill, with the send offset in a .pos file, so a restart doesn't resend what was sent. The file is compacted when it needs room and removed when it's empty.
    • Replay runs up to 20 batches after a delivered batch or on a quiet tick, and stops at the first failure. Events that arrive meanwhile wait in the sink's normal buffer.

Consequences

  • IPv6-only networks get working reverse names and can route around unreachable IPv6 ranges.
  • A SIEM can take events over TLS.
  • A collector outage no longer loses events, up to the cap. The disk writes happen only during an outage, on the sink's own thread.

Referenced by ADR-072

ADR-089 vqlog or and rollup windows; pre-save checksAcceptedT9.12 picks up ADR-082's "not done" (or, rollup-backed long windows) and the pre-save checks deferred from T7.5.

Context

T9.12 picks up ADR-082's "not done" (or, rollup-backed long windows) and the pre-save checks deferred from T7.5.

Decision

  • or: and binds tighter. A parenthesized group is one term of the and list, one level deep. A bare or makes the whole stage one group, and mixing it with parentheses is refused as ambiguous rather than guessed. The plan keeps the plain conditions (still pushed into the index search) and adds any_of groups checked per row. At most 32 alternatives in all; in (…) covers many values of one field.
  • Rollup windows:
    • When: a query whose from is older than retention_days, or any query while the query log is off, is answered from the rollups when it fits them.
    • What fits: count only, no or, buckets of whole hours, and at most one of status, qtype (the named ones), rcode, proto, and group, used as the key, the filter, or both. The rollups keep each breakdown on its own, so two can't be combined.
    • Levels: day buckets use the day rollups; everything else uses the hour rollups.
    • Labels: source says which store answered. A query that doesn't fit runs over the log as before, with a note saying the log only covers retention_days.
    • Not chosen: answering partly from each store. Its numbers would mix two sources, with no line between them.
  • Pre-save checks: POST /api/v1/checks/upstream and /checks/list take the same body as the PUT.
    • The upstream check builds the upstream through the normal router (a configuration holding only it) and asks for . NS within 5 s. The list check downloads with [filter]'s limits, or takes inline rules, and parses as the compiler does.
    • path lists aren't checked: reading server files on request would be a new capability.
    • Both need the operator role and the entry's write scope (they reach out from the node), and both are audited. The paths sit under /checks/ so they can't collide with an upstream or list named test.

Consequences

  • An agent can ask "blocked per day for six months" and get exact counts.
  • Questions that need names or clients stay limited to what the log keeps, and the answer says so.
  • A typo in an upstream URL or a dead list URL shows up before it's saved.

Builds on ADR-082

ADR-090 DoH load from h2load; restart counts in the data directoryAcceptedT9.13 picks up the deferrals of T0.3 (DoT/DoH load, a realistic profile) and T6.14 (restart counts, Grafana panels).

Context

T9.13 picks up the deferrals of T0.3 (DoT/DoH load, a realistic profile) and T6.14 (restart counts, Grafana panels).

Decision

  • Transports bench: dnsperf (-m tcp|dot) measures TCP and DoT.
    • DoH uses h2load, a bench-only tool that isn't shipped. dnsperf's DoH client ran at about 40 ms a query against a listener that answers curl in 175 µs, so it would have measured itself.
    • DoH requests are GETs built from the same corpus, so the names match the other transports.
    • The existing realistic-home corpus stays with the swap test. The new mode uses cache hits, because the Python stub upstream would saturate first and hide the transports' cost.
  • Restart counts: <data_dir>/runs.json holds starts, unclean, and running. A start increments starts, and adds to unclean if running is still set, then sets running. A clean stop clears it, even when startup fails.
    • It's written only by the process holding the data-directory lock.
    • An unreadable file starts over, rather than failing startup.
    • These are counters, so increase() over a range works across restarts. The heartbeat-observed restarts column in the cluster view stays as it is (what peers saw).
  • Grafana: new rows for transports (queries by protocol, TLS/DoH/DoQ/PROXY problems) and the process (uptime, restarts, unclean starts, OOM kills, CPU, temperature, data-disk free). Every expression is checked with promtool check rules in the Prometheus image.

Consequences

  • What DoT and DoH cost compared with UDP is measured, and repeatable on any machine with h2load.
  • A crash loop shows up as a number on the dashboard instead of a short uptime.
ADR-091 No DHCP server: routers hand out addressesAcceptedT7.19 added an optional DHCPv4 server (OPS-008, ADR-078).

Context

T7.19 added an optional DHCPv4 server (OPS-008, ADR-078). The owner (2026-10-06): most people let their router serve DHCP, and DHCP sits outside what a DNS product should own. Keeping it means a privileged port (67), a lease state machine, and a failover question in clusters, all for few users.

Decision

  • The server goes: the code, [dhcp], its end-to-end test, and its documentation. OPS-008 is descoped, and ADR-078 is superseded.
  • Device names keep coming from outside DHCP sources:
    • routers' DHCP clients ([[router]]: UniFi, OPNsense);
    • mDNS announcements;
    • the importers, which still turn Pi-hole's and Technitium's DHCP reservations into named devices.
  • GET /api/v1/dhcp/leases stays: these are the router's DHCP leases. It drops reserved and source = dhcp.
  • A configuration that still has [dhcp] fails to load with the usual unknown-field error rather than being ignored. Neither of the owner's nodes used it.

Consequences

  • About 1,100 lines and a privileged port are gone. The scope stays DNS.
  • Anyone who wants TelltaleDNS to serve DHCP needs their router, or dnsmasq beside it.

Builds on ADR-078

Referenced by ADR-078

ADR-092 Aggressive NSEC (RFC 8198): NSEC ranges per upstream group, NSEC3 laterAcceptedADR-086 deferred RFC 8198.

Context

ADR-086 deferred RFC 8198. The owner asked for the deferred features that make TelltaleDNS a more complete DNS product. Unbound and BIND both do this by default, and on a home network the root zone's NSEC records cover the steady trickle of made-up TLDs (.lan, .home, .localdomain).

Decision

  • What's kept: after the validator proves a negative answer secure, its NSEC records are cached, keyed by owner in canonical order, per upstream group. Each is kept only if:
    • it's proven secure;
    • it's inside the zone of the SOA that came with it;
    • it's signed by that zone. The SOA and all RRSIGs are kept too.
  • How long: the NSEC's TTL, capped by the SOA's negative TTL (RFC 8198 §5.4). At most 20,000 ranges and 2,000 zones per group; past that, the ones expiring first go.
  • Answering on a miss, before going upstream:
    • NXDOMAIN when one range covers the name and one covers the wildcard at the closest encloser (RFC 4035 §5.4).
    • NODATA when an NSEC at the name lacks the type and CNAME, and isn't a delegation (unless the question is DS).
    • Nothing below a delegation's or a DNAME's owner. The child zone, or the alias, decides there.
  • The answer: NXDOMAIN or NODATA with the SOA and NSEC records, plus their signatures for DO clients, and AD as for any secure answer. TTLs are counted down to the earliest expiry. It's counted as synthesized, not as a validation.
  • Switch: [dnssec] aggressive_nsec, on by default, as in Unbound.
  • Deferred — NSEC3: synthesis needs the zone's hash parameters and a hash per lookup, and opt-out spans prove nothing for unsigned delegations. Worth doing with measurements.

Consequences

  • Repeated junk lookups under NSEC-signed zones (the root above all) stop reaching upstreams, and are answered in microseconds instead of a round trip.
  • Answers still carry their proof, so a validating client downstream can check them.
  • A config reload starts the cache empty, like the validator's own cache.

Builds on ADR-086

Referenced by ADR-086

ADR-093 DNS stamps: enforce their certificate hashesAcceptedT7.24 parsed DoH/DoT/DoQ stamps but skipped their hashes.

Context

T7.24 parsed DoH/DoT/DoQ stamps but skipped their hashes. A stamp's hashes are SHA-256 digests of the tbsCertificate of certificates in the server's chain: the publisher's way of saying "only these CAs (or this certificate) may vouch for it". dnscrypt-proxy enforces them.

Decision

  • Hashes are parsed. Each must be 32 bytes, or the stamp is malformed.
  • When a stamp has hashes, the normal verification runs first, then some certificate in the presented chain must match one. With tls_insecure_skip_verify the match is still required, as for SPKI pins.
  • No opt-out switch. Someone who doesn't want the pinning uses the plain URL instead of the stamp, which is clearer than a flag that silently weakens what the stamp says.
  • Failures name the cause.

Consequences

  • Stamps mean what they say.
  • When a provider rotates its intermediate CA before republishing the stamp, that upstream fails until the stamp is updated. A group with another member fails over meanwhile.
  • No built-in preset uses stamps.
ADR-094 Local zones answer SOA and NS at their apex; no zone transfersAcceptedT7.22's zones answered their records, but not SOA or NS at the apex.

Context

T7.22's zones answered their records, but not SOA or NS at the apex. Their negative answers carried a placeholder SOA owned by the question name. Tools (dig SOA, monitoring checks, some resolvers' zone-cut detection) expect real ones.

Decision

  • From the zone file: if it has SOA and NS at its apex, they're served as written. Relative names are completed with the origin.
  • Otherwise, made up as Unbound does for local zones:
    • NS localhost.;
    • SOA localhost. hostmaster.<zone>;
    • serial = load time;
    • refresh 3600, retry 600, expire 86400;
    • MINIMUM = negative_ttl.
  • Negative answers (NXDOMAIN and NODATA) carry that SOA owned by the apex, with TTL negative_ttl (RFC 2308).
  • Not done: zone transfers (AXFR/IXFR). They need TSIG and secondary servers, which home and lab installs rarely have. The zone file, or Git, is the source to copy instead.

Consequences

  • dig SOA/NS and negative caching downstream behave as with any authoritative server.
  • The made-up localhost. name server can't be queried from outside, which is the point: these zones are served by the resolver itself.
ADR-095 $dnsrewrite in lists: AdGuard's precedence, a supported subsetAcceptedT2.3 kept $dnsrewrite rules in the snapshot but didn't apply them (T7.20 put rewrites in the configuration).

Context

T2.3 kept $dnsrewrite rules in the snapshot but didn't apply them (T7.20 put rewrites in the configuration). AdGuard-format lists and personal rule sets use them for local names, safe-search CNAMEs, and blanking names out. Without them those lists lose part of their meaning.

Decision

  • Values supported:
    • an address, either family (A/AAAA);
    • a name (CNAME);
    • NXDOMAIN, REFUSED, or SERVFAIL;
    • the full form for NOERROR with A, AAAA, or CNAME, and any rcode with an empty answer. Other record types (MX, TXT, SRV, HTTPS, …) make the line unsupported (counted, skipped), so a list author's intent is never half-applied.
  • Precedence, as AdGuard does it:
    • a matching rewrite wins over blocking;
    • an exception (@@…$dnsrewrite, with or without a value) turns rewrites off for the names it covers;
    • $client, $dnstype, and $denyallow narrow a rewrite like any rule;
    • schedules and quick rules still come first.
  • Combining: an rcode wins, then the first CNAME, then every address of the asked family. When the name is rewritten only for the other family, the answer is an empty NOERROR.
  • On the query path: a matcher flag skips everything when no list has rewrites. Otherwise the modifier index is walked once more for the name. The answer has TTL 60, is attributed to the list in the event, and a CNAME is resolved through the normal path (cache, upstreams, filtering of the target).

Consequences

  • AdGuard-style lists and personal rules work as their authors meant.
  • Configuration rewrites ([[group.rewrite]]) stay the per-group tool. List rewrites follow the lists a group uses.
ADR-096 A client group's own upstream groupAcceptedUPST9.25 (UPS-007, owner request 2026-10-06).

Context

T9.25 (UPS-007, owner request 2026-10-06). UPS-007 already routes by client group ([[route]] match_group), but that lives in routing, not on the group, and the UI had no way to set it. The spec doesn't say how it ranks against domain routes, which group wins for a device in several, or what removing an upstream group a group uses does.

Decision

  • [[group]] upstreams = "<upstream group>", validated like a route's upstream_group. The router turns it into a route for the group's clients that matches any name, placed after the explicit [[route]] entries. Precedence: a domain route (deeper suffix), then an explicit route of equal depth (a match_group route, which comes first), then the group's upstreams, then default.
  • A device in several groups: the derived routes are ordered by group priority (highest first), then configuration order, so its highest-priority group with upstreams decides.
  • Removing an upstream group that a client group or a route still uses is refused with a hint, rather than clearing the choice. Clearing it would quietly send, say, the kids' questions to default instead of a family resolver. Reverting an API override of the files' entry is allowed (the group stays).

Consequences

  • No hot-path cost: routes are consulted only on a cache miss, and the cache is already keyed by upstream group, so groups never share answers from different resolvers.
  • [[route]] match_group stays for narrower cases (one group's TXT questions, say).

Requirements UPS-007

ADR-097 Releases: one workflow, checked before and verified after; main moves on at onceProposedOPST9.26 (OPS-004, owner request 2026-10-07: cut v0.1.0, and make releases deterministic for future agents).

Context

T9.26 (OPS-004, owner request 2026-10-07: cut v0.1.0, and make releases deterministic for future agents). Tag builds already published signed binaries, versioned images, and a chart (ADR-038, ADR-046), but cutting one was a manual tag push, nothing checked the version against Cargo.toml or the chart, nothing confirmed the result, and edge builds kept the released version's number.

Decision

  • A release workflow (manual, input X.Y.Z) runs deploy/release/preflight.sh: X.Y.Z with no suffix, equal to Cargo.toml's and the chart's version, a new tag, a notes file deploy/release/notes/vX.Y.Z.md, and a passing CI run on main's head. Only then does it tag main's head.
  • A tag pushed with the workflow's token starts no workflows, so the release workflow starts the image workflow on the tag itself (workflow_dispatch). The image workflow's publish jobs run for pushes and dispatches on main or v* tags.
  • The tag's build ends with verify (deploy/release/verify.sh): signatures, the binary's version and channel, "latest release", the four image tags as one multi-arch image, and the chart's version and appVersion. The release is done when it passes.
  • Right after tagging, main moves to the next version (default: next minor) with deploy/release/bump.sh, because semver sorts X.Y.Z-edge.N below X.Y.Z. The workflow then starts CI and edge for that commit.
  • A published tag is never moved or reused; a broken release is followed by a patch release.
  • Release notes are hand-written per version; GitHub's generated changelog is appended.

Consequences

  • Releases are one button with the same checks every time; docs/releasing.md is the procedure for maintainers and agents.
  • The workflow pushes one commit to main per release. If branch protection is added later, the bump step needs an exception or becomes a pull request.

Requirements OPS-004

Builds on ADR-038 ADR-046

ADR-098 DNSSEC: library limits are indeterminate; bogus names are logged while trying validationProposedRunning [dnssec] mode = "permissive" on the live cluster (2026-10-07) counted 14 bogus answers in five minutes.

Context

Running [dnssec] mode = "permissive" on the live cluster (2026-10-07) counted 14 bogus answers in five minutes. hickory 0.26's validator returns an error both for a broken proof and for its own limit ("exceeded max validation depth", logged at ERROR); TelltaleDNS treated every non-network error as bogus, so validate mode would have answered SERVFAIL for such names. Its log named none of them. On prod.ftl.netflix.com (unsigned zone, CNAMEs) hickory repeats NS and DS lookups for the same name until the limit. Raising the limit from 26 to 40 overflowed a runtime thread's stack in a debug build.

Decision

  • The depth limit stays at hickory's default (26). Hitting it is indeterminate: the answer is fetched again unvalidated and served without AD, in both modes. Other validation errors stay bogus.
  • In permissive mode every bogus answer is logged at INFO with its name, type, and EDE code (DNSSEC: bogus answer served (permissive mode)), so a trial shows which names would fail.
  • hickory's hickory_net::dnssec log target is off: TelltaleDNS counts and logs each verdict itself (that module warned on every unsigned NODATA).
  • Unsigned zones proven by TelltaleDNS (T10.9). The trial also showed hickory judging names in unsigned zones bogus when reached through CNAMEs (www.netflix.com, www.amazon.com, www.bbc.co.uk: EDE 10), though it validated the parent's denial of their DS correctly. Before accepting a bogus verdict, an error, or a validation that timed out, TelltaleDNS proves the zone unsigned itself: walking down from the domain under the TLD, the first zone apex (it has its own SOA, from an ordinary query) whose DS the parent validly denies (secure NSEC, or NSEC3 opt-out, which validates as insecure) is unsigned. If every owner in the answer and the question is in such a zone, the answer is insecure. Proven zones are remembered for 15 minutes (at most 4,096), and questions under them aren't handed to hickory again (which also avoids its NS/DS loop).
  • Only the DS denial is trusted: whether a name is an apex comes from an unvalidated query, but a signed zone's DS can't be validly denied, so a spoofed apex answer can't make a signed zone insecure. A name inside a signed zone that isn't a cut (www.isc.org) is skipped because it has no SOA of its own.
  • Correction (2026-10-07, review 03-01): the paragraph above is wrong for every name in a signed zone except its apex. A validated denial of a DS proves only that the name has no DS, and every ordinary name in a signed zone has none; the SOA comes from an unvalidated query, so a forger supplies it. Together they let a spoofed answer under a signed zone pass as insecure (and be remembered for 15 minutes). The denial must also show that the name is a delegation: the NSEC owned by the name (or the NSEC3 whose owner is the name's hash) lists NS and lists neither SOA nor DS (RFC 4035 §5.2, RFC 5155 §8.9; the check hickory applies in its own insecure-delegation case). An NSEC3 opt-out denial is still read as unsigned, as hickory has already checked it. The SOA query stays as a cheap pre-filter, never as evidence. The text above is kept as the record of what was first decided.
  • Deadline and limits (2026-10-07, review 03-05, owner's answer: give up at twice the query budget and serve the conservative verdict). A validation holds one of the pipeline's in-flight permits (spec/02 §8.4), and the proof walk could run for far longer than the client's budget (up to 3 × budget for the lookup, then up to 12 more queries of one budget each). Now the whole validation, the lookup and any unsigned-zone proof, ends within two query budgets; the lookup may use three quarters of that, so a lookup that loops (the Netflix case) is cut off in time for the proof to rescue it, and each step of the proof gets only what the deadline leaves. At most 8 proofs run at once; one that finds no slot is "not proven". A name whose zone could not be proven unsigned is not walked again for one minute. All three can only make the answer more conservative: not proven means the bogus verdict stands, or, if the lookup produced no answer, stale or SERVFAIL as for any upstream failure. The cold-chain allowance for the lookup drops from three budgets to one and a half.

Consequences

  • Switching to validate can't SERVFAIL a name only because hickory gave up or misjudged an unsigned CNAME chain. Checked live (Quad9, 2026-10-07): in validate mode the Netflix, Amazon, BBC, and Apple names resolve (insecure), isc.org and example.com validate with AD, and dnssec-failed.org is refused (SERVFAIL); repeat questions are answered from the cache.
  • A bogus verdict costs a few DS and SOA queries the first time a zone is seen (cached after). The first question under a looping name can still take about 1.5 s.
  • hickory reports a negative answer whose proof isn't secure as an error carrying the proof (DnsError::Nsec); for a DS, Insecure is an NSEC3 opt-out denial, read as unsigned like a secure one.
  • Open (cold starts): right after a start, Quad9 was seen closing DoT connections with queries unanswered (logged as upstream connection ended why="closed by the server"); a validation that loses . DNSKEY that way caches failures and calls signed zones bogus for a while. The pool now retries such a query once on a fresh connection, which fixed most cases in testing but not every one. Until the cause is known, permissive stays the recommended mode.
ADR-099 Connection limits per listener, and per client addressProposedCode review of v0.2.0, pass 01 (owner's answer to question 2, 2026-10-07: a per-address cap of 32, configurable).

Context

Code review of v0.2.0, pass 01 (owner's answer to question 2, 2026-10-07: a per-address cap of 32, configurable). Every stream listener capped connections globally at 1,024 and nothing per client, so one host could take every slot (TCP, DoT, DoH) until a restart; the QUIC listeners had no cap at all (01-02). The spec doesn't say how a client is told apart, or what a balancer does to the count.

Decision

  • [[listen]] max_connections (default 1,024) and max_connections_per_address (default 32, 0 = no limit) on tcp, dot, doh, doh3, and doq listeners. Over a limit, a connection is refused or closed at once and counted (telltale_{tcp,doh,doq}_rejected_total for the overall limit, …_rejected_per_address_total for the per-address one).
  • The address counted is the client's: after the PROXY protocol v2 header when proxy_protocol is on (a balancer's own address is never counted for everyone), and after the QUIC handshake for doq and doh3 (before it, the source address can be spoofed, which would let an attacker spend a victim's budget). An IPv4 client is one address; an IPv6 client is its /64, because a host rotates through its prefix; IPv4-mapped IPv6 addresses count as IPv4.
  • No address is exempt (loopback included): a limit of 32 is far above what a client needs, and operators who proxy many clients through one address set the limit higher or 0.

Consequences

  • Behind a load balancer that hides client addresses (no PROXY protocol, source NAT, Kubernetes externalTrafficPolicy: Cluster), every client shares one count, so the default 32 can starve legitimate TCP/DoT/DoH clients. docs/running.md says so next to the key; the Helm chart's default exposure (externalTrafficPolicy: Local) keeps client addresses.
  • Changing a limit restarts that listener (the existing replace-on-change rule); open connections finish first.
  • The count is one short lock per accepted connection, off the UDP path.
ADR-100 The advertised EDNS payload size is [dns] edns_payloadProposedDNSCode review of v0.2.0, finding 01-12 (owner's answer, 2026-10-07: a configuration key).

Context

Code review of v0.2.0, finding 01-12 (owner's answer, 2026-10-07: a configuration key). DNS-005 says "a configurable advertised UDP payload size (default 1232)", but the value was a constant in the pipeline's settings and no key reached it. The config has no section for answer behaviour, and the spec doesn't name one.

Decision

A new [dns] section with one key for now, edns_payload (default 1232, 512 to 4096: the UDP workers' send buffer is 4,096 bytes, crates/telltale-net/src/sys.rs). It needs a restart (listed with the other restart-only sections); other answer-shaping settings can join the section later.

Consequences

Operators on a jumbo-frame or fragmentation-safe network can raise the size; the default and the DNS Flag Day reasoning are unchanged. Upstream-facing payload sizes (edns_payload per upstream, spec/04) are separate and untouched.

Requirements DNS-005