Guide

Monitoring and alerts

Most of what TelltaleDNS can show you works from the first start. This guide covers what to add: Prometheus and Grafana, objectives, alerts that reach you, and exports to the tools you already run. For what each feature does, see Monitoring and alerts.

• What you get without setting anything up
• Scrape metrics and add the Grafana dashboard
• Set service-level objectives
• Get alerts somewhere
• Send events and traces elsewhere
• Keep names and devices out of the log
• Review and acknowledge anomalies
• Check your listeners
• Which tool for which question

What you get without setting anything up

The same health answer is at GET /api/v1/system/health. Details: Health, Query log, Host resources.

Scrape metrics and add the Grafana dashboard

serviceMonitor:
  enabled: true        # Prometheus scrapes the controller and every resolver pod
prometheusRule:
  enabled: true        # alert rules, plus the SLO recording rules and burn-rate alerts
grafanaDashboard:
  enabled: true        # the dashboard as a ConfigMap, labelled grafana_dashboard: "1"

The main metric names are in the Monitoring docs.

Set service-level objectives

Two promises, each with an error budget over a window: answers that aren't SERVFAIL, and answers sent within a time limit. They work out of the box with the defaults below. Change them if your network needs different numbers.

[slo]
enabled = true
availability_target = 99.9   # percent of answers that aren't SERVFAIL
latency_target = 99          # percent of answers sent within latency_ms
latency_ms = 250             # 1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, or 5000
window_days = 30             # 1 to 90

prometheusRule:
  enabled: true
  slo:
    availabilityTarget: 99.9
    latencyTarget: 99
    latencyMs: 250

• When it warns: a fast burn (14.4× over 1 hour and 5 minutes) or a slow burn (6× over 6 hours and 30 minutes) turns the health icon amber and fires slo_burn alert rules. A pair needs at least 50 answers in its short window and 10 bad ones in its long window, so one SERVFAIL on a quiet night stays quiet.
• Where to look: the dashboard's Service level card (Advanced view), the Grafana dashboard's Service level row, GET /api/v1/stats/slo, and the MCP tool slo_status.

What counts and how burn rates work: Service-level objectives.

Get alerts somewhere

Alerts have two parts: destinations (where messages go: email, ntfy, gotify, slack, or webhook) and rules (what to watch, and which destinations to tell).

[alerts]
interval_secs = 30                       # how often rules are checked

[[alerts.destination]]
name = "phone"
type = "ntfy"                            # webhook | ntfy | gotify | slack | email
url = "https://ntfy.sh/my-home-dns"
# token_file = "/run/secrets/ntfy-token" # an access token, kept out of the config

[[alerts.rule]]
name = "Upstream down"
when = "upstream_down"                   # an upstream's circuit breaker is open
for_secs = 60                            # only if it lasts a minute
to = ["phone"]

Rules worth starting with: upstream_down, servfail_rate, list_failing, disk_full, slo_burn, probe_failing, and cert_expiring if you serve encrypted DNS. In a cluster add node_down, and sync_lag with for_secs around 300 so a normal sync stays quiet. Every condition, and what each sends one alert per, is in the Alerts docs.

• A condition must hold for for_secs (60 by default) before it alerts, and a "Resolved" message follows when it clears. Anomalies, updates, new devices, pending plans, and failed rollouts go out once each.
• In a cluster the primary checks the rules for the whole cluster, so you get one message, not one per node. Nodes in maintenance are left out of per-node rules.

Email

Any mail account that allows SMTP sign-in works; you don't need your own mail server.

[[alerts.destination]]
name = "inbox"
type = "email"
url = "smtp://smtp.gmail.com:587"        # STARTTLS; smtps://host:465 for TLS from the start
from = "you@gmail.com"
to = ["you@gmail.com"]                   # one or more
username = "you@gmail.com"
password_file = "/run/secrets/smtp-password"

• Gmail: turn on 2-Step Verification, make an app password (Google Account → Security → App passwords), and put it in password_file. Most other providers work the same way with their SMTP host and an app password.
• Encryption is required whenever a password is sent; there's no fallback to plain. For a private mail server, give its CA in tls_ca.
• The subject starts with [TelltaleDNS], for easy mail filters. In Kubernetes, keep the password in a Secret mounted as a file, never in values or Git.

Send events and traces elsewhere

All of these are config-file settings, read at startup: restart after changing them. Event sinks and traces follow the query log's privacy level (names hashed at 1 and above, no client addresses at 2 and above).

[telemetry.otlp]
endpoint = "http://otel-collector:4318"   # OTLP/HTTP
interval_secs = 60
traces_sample_every = 1000                # 1 query in 1,000 (0: none, the default)
traces_slow_ms = 250                      # and every query that took at least 250 ms (0: none)

[[telemetry.sink]]
name = "siem"
type = "syslog"                           # RFC 5424
address = "udp://192.168.1.20:514"        # or tcp://, or tls://host:6514
statuses = ["blocked"]                    # only blocks; empty means every event

A destination that's slow or down drops events and counts them (telltale_sink_dropped_total, telltale_dnstap_dropped_total, telltale_traces_total{result="dropped"}). Webhook sinks can keep refused batches on disk with spill_max_bytes. Every option: OpenTelemetry, Event sinks, dnstap.

Keep names and devices out of the log

Phones checking their connection, time servers, and a monitoring probe asking every few seconds drown out what matters. Excluded queries are still answered and still counted in Prometheus, but left out of the query log, the live view, the dashboard and top lists, device pages, anomalies, event sinks, OTLP traces, and exemplars.

[exclusions]
enabled = true                                           # off keeps the lists, logs everything again
names = ["connectivitycheck.gstatic.com", "ntp.org"]     # each with its subdomains
clients = ["192.168.1.10", "10.20.0.0/24"]               # addresses or networks

• It applies from the next query on, on every node of a cluster ([exclusions] is shared). Queries logged before stay.
• The dashboard's totals leave excluded queries out, so they'll be lower than Prometheus's. telltale_queries_excluded_total counts them.

Review and acknowledge anomalies

1. Open the Anomalies page. Each finding shows what was seen, the usual value, the threshold, and the window. In a cluster, Found by names the nodes.
2. Look at the device's queries, and move it to a stricter group if needed. Findings never block anything themselves.
3. Press Acknowledge on the card (or Acknowledge all). It stops counting as new: the sidebar badge and anomaly alerts skip it. Show acknowledged shows it again with who acknowledged it, when, and the note; Undo acknowledge takes it back.

It needs the operator role and is recorded in the audit log. API: POST /api/v1/analytics/anomalies/acknowledge with the finding's id; MCP: acknowledge_anomalies. An acknowledgement on any node reaches every node within seconds.

Too noisy, or too quiet? Tune detection, or ignore domains you expect:

[telemetry.anomaly]
enabled = true
learning_days = 7
sensitivity = "normal"         # low | normal | high
ignore_domains = ["apple.com"] # never reported (e.g. expected connectivity checks)

To hear about findings, add an alert rule with when = "anomaly". Details: Device anomalies.

Check your listeners

A listener can be bound and still not answer: a stuck TLS stack, a firewall rule, a load balancer in front. So every node asks each of its own listeners a question over that listener's protocol, every 30 seconds, and reads each certificate's expiry.

• See the results in Settings → System → Listener checks, at GET /api/v1/system/probes (every node), or with the MCP tool probe_status.
• Add your load balancer's address (or another IP) as an extra target:

[probe]
cert_warn_days = 14                     # a certificate expiring sooner degrades the health level
targets = ["udp://192.168.5.112:53"]     # extra: by IP, e.g. a load balancer's address

• Targets take udp://, tcp://, tls://, https://…/dns-query, h3://, and quic:// with an IP address. A listener with proxy_protocol = true is skipped.
• Two failures in a row turn the health icon amber; an expired certificate turns it red. Alert on it with probe_failing and cert_expiring rules.
• Probe queries stay out of the query log, top lists, device analytics, and event sinks.

Details: Listener checks.

Which tool for which question