Guide

Change things safely

A ladder of safety nets, from small to large: see what a change will do before you save it, replay it over yesterday's queries, send it to a few nodes first, and go back to an earlier version if it goes wrong.

Preview a change before it's saved
Try it against yesterday's traffic
Apply file changes without a restart
Roll it out to canary nodes first
Pin the cluster to an earlier version
Take a node out for maintenance
Changes made by AI agents
What to use when

DNS keeps answering through all of it. A check, a simulation, a rollout, or a pin never makes a query wait.

Preview a change before it's saved

Every change is checked first. A change that would leave the configuration invalid is refused with the reason.

Web UI

Make the change in an editor, such as the Manage table at the end of the Upstreams, Lists, or Groups page.
Choose Check. It shows what the change will do and any warnings: an upstream group that would lose its last member, or a list no group uses.
For upstreams and lists, Test it also tries the draft from this node. An upstream is asked for the root's NS records: you see the answer and the time, or the error (a bad URL, a TLS name that doesn't match, a timeout). A list is downloaded and parsed: you see how many rules it has, and the first lines it can't use.
Nothing is saved until you choose Apply.

Operators and admins can make changes. Every change is audited and reaches every node in a cluster. Your config files are never rewritten.

Config file

Check the file before you load it. Every problem is reported at once, with its exact key path. Unknown keys are always errors, so a typo is never silently ignored.

telltale config check telltale.toml               # "config OK" or a list of errors
telltale config check telltale.toml node.toml     # later files override earlier ones
telltale config check telltale.toml --print       # show the effective config (defaults + files + env)

The configuration reference lists every key.

API / CLI

Add ?dryRun=true to a PUT or DELETE, for example on /api/v1/upstreams/{name}, /api/v1/upstream-groups/{name}, /api/v1/lists/{name}, or /api/v1/groups/{name}. The body has the same fields as the TOML section.

Test it is POST /api/v1/checks/upstream and POST /api/v1/checks/list, with the same body as saving. AI agents use the MCP tools check_upstream and check_list.

Try it against yesterday's traffic

A simulation decides the queries already in the query log again, with the change and without it, and reports only the differences. You see four counts (newly blocked, newly allowed, a changed route, a changed answer), the names behind each with the list or rule that decided, and the devices and groups.

Web UI

Look for What would this have done? on the quick-rule form, in the device drawer (Add to group…), and in the checked preview of a list, group, upstream, or schedule. A name in the result links to the query log. Nothing is saved until you save the change.

The settings are under Settings → Query handling → Change simulation.

Config file

[simulate]
enabled = true
default_window = "24h"     # at most 7d
max_secs = 20              # 1 to 300
max_rows = 2000000         # at least 1,000
plans_by_default = false   # true: agents' plans simulate even when they don't ask

The values shown are the defaults. The section is shared across a cluster. See the configuration reference.

API / CLI

Add simulate to any dry run. The value is a window, or true for the default one. The response gains simulation:

curl -X PUT 'https://dns.example.com/api/v1/rules/kids-tiktok?dryRun=true&simulate=24h' \
  -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{"action":"block","domain":"tiktok.com","groups":["kids"]}'

simulateUntil moves the end of the window back (-24h, or an RFC 3339 time).

To test a whole configuration, such as a Git commit before you push it, use POST /api/v1/simulate, or let telltale config simulate read the TOML for you. Node-local sections in the file are ignored.

TELLTALE_TOKEN=... telltale config simulate shared.toml --window 24h --url https://dns.example.com:8053

What it can't tell you. A simulation is about decisions. It never asks an upstream or touches the cache, it can't count names nobody asked, and it doesn't predict latency or what an upstream would answer.

Limits. One simulation runs at a time per node (another gets 409 simulation_busy; retry in a few seconds). Past max_rows or max_secs the answer says partial. The window is at most 7 days. At query-log privacy level 1, a change that depends on names can't be replayed; at level 2 and above, nothing can.

In a cluster every node replays its own log and the counts are added up. Details: Trying a change against yesterday's traffic.

Apply file changes without a restart

Edit the config file, then send SIGHUP:

sudo systemctl reload telltale      # native install
docker kill -s HUP telltale         # Docker
kill -HUP <pid>                     # anything else

The whole file is validated first. If anything is wrong, the errors are logged and the previous configuration keeps serving.
Applied at once, with no dropped queries: upstreams, groups, routes, local records and hosts files, [access], [ratelimit], [special], and [[listen]] (new listeners are bound before removed ones are closed).
Applied at the next restart: [cache], [telemetry], and [node]. A warning names them. The cache is kept across reloads, so warm answers aren't lost.
Upstream health starts fresh after a reload and is learned again within a few queries.

Run telltale config check before you reload, so you see every problem at once.

Roll it out to canary nodes first

In a cluster, a change normally reaches every node within seconds. With canary nodes set, it reaches them (and the primary) first. Every other node gets it only after a bake of 5 minutes, and only if nothing went wrong meanwhile. If something did, the cluster is pinned to the version before the change, and the other nodes never see the bad version.

Off until you set it: with no canaries, changes reach every node at once.

Web UI

Open Settings → Cluster → Staged rollouts and pick the canaries: any of the cluster's nodes, a whole site, or "resolver pods".
While a version bakes, the Cluster page's Rollouts tab shows which nodes run it and which stay on the version before.
Choose Promote now to give it to everyone without waiting, or Abort to pin to the version before it. Both need an admin.

Config file

[cluster.rollout]
canaries = ["ephemeral"]   # node IDs, "site:<name>", or "ephemeral" (the Kubernetes resolver pods)
bake_secs = 300            # at least 5
servfail_pct = 5
require_traffic = false    # true: keep baking until the canaries answered 50 queries, then judge
max_bake_secs = 3600
fail_on_disconnect = true  # a canary gone for a minute after applying fails the change
history = 20               # versions the primary keeps for pinning back to

The section is shared, so every node knows the canaries. A change to it is published at once and never staged itself, so a bad canary list can always be fixed. With the Helm chart, use cluster.rollout (see Helm values).

API / CLI

telltale ctl rollout status     # the version baking, the canaries and what each runs, the time left, the guard
telltale ctl rollout promote
telltale ctl rollout abort

Over the API: GET /api/v1/cluster/rollout, and POST /api/v1/cluster/rollout/promote or /abort (admin; audited). The settings are PUT/DELETE /api/v1/cluster/rollout-settings/default. AI agents read rollout_status.

The guard

Every 5 seconds the primary looks at what each canary reports. A version fails at once if a canary can't apply it, isn't ready for 15 seconds after applying it, fails its listener checks, or (with fail_on_disconnect) is gone for a minute. It also fails when the SERVFAIL share of the canaries and the primary goes over servfail_pct and over the share before the change by more than 2 points, judged once there are 50 answers. Latency isn't judged.

A pass gives the version to every node. A fail pins the cluster to the version before, fires the rollout_failed alert, and records the reason and the readings.

Good to know. A newer change during a bake replaces the one baking, and the bake starts over. Users, tokens, and the member list wait for the bake to end. If no canary is online, a change goes to every node at once ("rollout skipped: no canary online").

The full rules: Rolling a change out in stages.

Pin the cluster to an earlier version

A pin makes every node serve an older version again: its shared configuration and its filter snapshot. The guard pins by itself when a rollout fails; an admin can pin any kept version (the primary keeps the last 20 by default).

Web UI

Open the Cluster page's Rollouts tab. The Versions table lists the kept versions: when, what made each, its outcome, and the guard's readings.
Choose Pin… on a version, give a reason, and look over the preview of what changes on every node.
When you're done, choose Unpin on the pin banner.

API / CLI

telltale ctl versions                                           # newest first; * = what every node runs
telltale ctl pin 3.124 --reason "the new list blocks the bank" --dry-run
telltale ctl pin 3.124 --reason "the new list blocks the bank"
telltale ctl unpin

Over the API: GET /api/v1/cluster/versions, POST /api/v1/cluster/versions/{epoch}.{seq}/pin with {"reason": "…"} (?dryRun=true shows the diff), and DELETE /api/v1/cluster/pin. All need an admin and are audited.

While pinned, changes wait. Changes from the UI, the API, and agents' plans answer 409 cluster_pinned, with a diff of what the pin holds back. A dry run still works.
It's loud: a banner on the Cluster page, the health condition cluster_pinned, and the cluster_pinned alert.
It survives failover. A replica promoted to primary keeps the pin.

A pin doesn't undo the change for you. It doesn't rewrite your files, the entries made in the UI, or Git. Undo the bad change there (the diff says what it was), then unpin. Unpinning publishes the current configuration again, through a staged rollout when canaries are set.

Take a node out for maintenance

Swapping a Pi's SD card, moving a box, or upgrading a node by hand? Put the node in maintenance first. For a bounded time (an hour by default, a day at most) it:

reports not ready (/readyz answers 503), so load balancers and Kubernetes stop sending it new queries;
keeps answering every query that still arrives: no listener closes;
is left out of alerts, the health level, and the Cluster page's checks;
isn't elected primary (it still votes), and stays in maintenance across a restart.

Web UI

On the Cluster page's Nodes tab, choose Maintenance… on the node's row, and say how long and why. End maintenance ends it early.

API / CLI

telltale ctl maintenance start --for 30m --reason "SD card swap"           # this node
telltale ctl maintenance start --for 2h --reason "kernel upgrade" --node pi   # another node
telltale ctl maintenance status
telltale ctl maintenance end --node pi

Over the API: POST /api/v1/nodes/{id}/maintenance with {"forSecs": 1800, "reason": "…"}, and DELETE to end it. id is local, or a node's ID, site, or pod name. Any node passes the request on to the node it's for.

Maintenance on the primary. Under automatic failover with another eligible node online, the primary hands its role over first (configuration changes pause for about 30 seconds). In manual failover, or with no other eligible node online, the primary stays primary: promote another node first if you're taking it offline.

A node that's really failing during its window stays quiet until the window ends, so keep windows short. More: Taking a node down for maintenance.

Changes made by AI agents

An AI agent doesn't change settings directly. A plan_* tool runs the same dry run as ?dryRun=true and keeps the answer as a plan for 10 minutes. The agent gets a preview: before and after, how many recent queries it affects, and any warnings.

Require approval. With [agents] require_approval = true, every plan waits for a person.
Review it. The header shows N agent changes to review. Approve or reject each on the Agent changes page (or POST /api/v1/plans/{id}/approve and /reject). Agent tokens can't approve anything, their own plans included.
It's applied as planned. If the configuration changed since the plan was made, the plan is stale and nothing happens; the agent plans again.

A plan tool takes simulate ("24h", up to "7d") to include a simulation in the preview. To make every plan simulate, set [simulate] plans_by_default = true (Settings → Query handling → Change simulation). No agent tool can turn that on.
The change is audited as the agent, with its owner and the plan's reason; the approval is a separate entry under your name.
Plans live in the memory of the node that made them. In a cluster, point agents at the node whose UI you use (normally the primary).

What to use when

You want to…	Use
Catch a mistake in a form or a file	Check in the UI, `?dryRun=true`, or `telltale config check`
Make sure an upstream or a list works	Test it
Know who a block or allow rule would affect	What would this have done?, or `simulate` on a dry run
Test a Git commit before you push it	`telltale config simulate`
Load an edited config file	A reload (`SIGHUP`); a restart for `[cache]`, `[telemetry]`, `[node]`
Limit the damage of a bad change in a cluster	Staged rollouts with canary nodes
Go back after a bad change	Pin to an earlier version, fix the change, unpin
Work on a node without alerts or traffic	Maintenance
Let an AI agent suggest changes	Plans with `[agents] require_approval = true`

← Encrypted DNS and upstreams Next: Monitoring and alerts →