Guide

Build a cluster

Two or more nodes on one configuration. Every node answers DNS on its own, you change settings in one place, and every node's dashboard shows the whole cluster.

• The roles
• Create it
• What it shares
• When the primary goes down
• Upgrade and maintenance
• Keep an eye on it
• Troubleshooting

The roles

Create it

Run these as the user TelltaleDNS runs as: sudo -u telltale for a native install, or docker compose exec telltale with Compose.

1. Create the cluster on the first node. List every URL other nodes might use to reach it.

   telltale cluster init --name home --advertise https://192.168.3.2:8443 --site home-pi

2. Make a join token on it. --eligible lets the new node be promoted later; leave it out for a node that should only follow. Treat the token like a password; it's reusable until it expires.

   telltale cluster token create --ttl 1h --eligible

3. Join from the second node. Give an eligible node its own --advertise URL, so it can be reached if it's ever promoted.

   telltale cluster join tt_join_... --site office --advertise https://192.168.3.3:8443

4. Restart TelltaleDNS on both nodes, then check: telltale cluster status, or the web UI's Cluster page (both nodes up on Nodes; every check passing on Health).
5. Hand out both nodes in DHCP as the first and second DNS server. Then check that devices really use the second one: a failover drill (Drill… on a node's row) silences the first for a few seconds and lists who switched and who was stuck.

A Pi joining a Kubernetes cluster works the same way, with the token made inside the controller pod: see Run on Kubernetes.

What it shares

• Settings: upstreams, lists, groups, devices, local names, rules, and the rest reach every node within seconds, including what was added in the web UI. Each node keeps its own listeners, API, sign-in, telemetry, and cache settings. What's shared and what isn't.
• Change it anywhere: a change in any node's UI or API goes to the primary, which applies it for everyone. For config files, edit the primary's.
• Blocklists: only the primary downloads and compiles them; the others fetch just the parts that changed.
• Accounts: users and API tokens are the primary's, so the same sign-in works on every node.
• One view: every node's dashboard, top lists, query log, and live view cover the whole cluster, and each query-log row names the node that answered. Showing on the dashboard narrows it to one node or site.

When the primary goes down

DNS keeps answering on every node either way. Only configuration changes wait until there's a primary again.

# On the primary: a token for the witness
telltale cluster token create --witness

# On the witness host (a config with [node] data_dir and [cluster] listen is enough)
telltale cluster join tt_join_... --witness --advertise https://witness.lan:8443 -c witness.toml
telltale cluster witness -c witness.toml        # run it as a service

# On the primary, then restart it
telltale cluster set-failover auto

If the cluster's configuration comes from Git, a node that isn't Git-managed can only take over as an emergency primary: see Git, for the whole cluster.

Upgrade and maintenance

• Upgrade one node at a time: the replicas first, then the primary. Versions one cluster protocol apart work together, so a half-upgraded cluster is fine, and the Cluster page's versions check lists the mix until you're done.
• Taking a node down for longer than a restart? Put it in maintenance first: Maintenance… on its row (Cluster → Nodes), or telltale ctl maintenance start --for 30m --reason "SD card swap". It reports not ready so load balancers move away, its alerts pause, and it keeps answering what still arrives.
• Roll configuration changes out in stages if you like: name canary nodes in Settings → Cluster, and a change reaches them first and everyone else after a bake, pinning back by itself if the canaries start failing. Staged rollouts in the docs.

Keep an eye on it

• The Cluster page: the summary on top, then Nodes (topology and every node's state), Health (checks with what to do when one fails), Rollouts, and Events. A tab with something wrong shows a count.
• The health icon at the bottom of the menu turns amber when a node is unreachable or behind on configuration for a minute.
• Metrics on every node: telltale_cluster_peer_up, telltale_cluster_peer_config_lag, telltale_cluster_behind_seconds, and telltale_cluster_cert_expiry_timestamp_seconds. The Helm chart's alert rules cover a peer down, configuration behind, failing syncs, and expiring certificates.
• Certificates renew themselves: each node's lasts 90 days and is renewed with 30 left. To replace the cluster's certificate authority, run telltale cluster rotate-ca on the primary; DNS keeps answering while it runs.

Troubleshooting