Observability: alerting rules¶

mesh-DNS (`mesh-dns-alerts.yml`)¶

Covers the aether-mesh-dns DaemonSet, which is on the critical path for every managed pod's DNS (the CNI DNATs all UDP+TCP :53 to HOST_IP:18054).

Since #578 the resolver runs in its own process and answers from a snapshot file the node agent writes. That cross-process dependency is the reason most of these rules exist — and why the external prober alone is not sufficient.

Alert	Severity	Catches
`MeshDNSSnapshotStale`	critical	agent stopped writing → daemon serves frozen records
`MeshDNSNoRecords`	critical	empty snapshot → every mesh name NXDOMAINs
`MeshDNSNoUpstreams`	critical	no forward upstream → all non-mesh DNS fails
`MeshDNSWatcherInactive`	warning	fsnotify watcher died → updates silently stop
`MeshDNSReloadFailing`	warning	corrupt/unreadable snapshot
`MeshDNSResolutionFailing`	critical	external prober can't resolve (per path)
`MeshDNSMetricsAbsent`	critical	daemons down fleet-wide, or the OTLP path is broken

Why staleness is the important one¶

If the agent dies, no fsnotify event ever fires, so the daemon keeps serving its last known table indefinitely. Because ready is true, misses are answered as authoritative NXDOMAINs, which clients negatively cache. New services never resolve; re-IP'd services resolve to dead ClusterIPs.

The external prober stays 100% green through all of it — its long-lived target keeps resolving from the stale snapshot. MeshDNSSnapshotStale is the only signal for a failure that is silently wrong rather than loudly broken.

The agent re-stamps the snapshot every 60s (capture.MeshDNSHeartbeat) even when records are unchanged, precisely so that age is meaningful on a quiet cluster. snapshot_generation advances only on a real content change, so the two are distinguishable.

Probe targets cover different paths¶

The mesh_dns prober tier carries two targets, separable by the target label:

*.aether.internal → answered authoritatively from the snapshot
*.svc.cluster.local → forwarded to kube-dns

Both matter: the forward path carries the majority of a real workload's lookups but almost no organic traffic here, so without the second target a kube-dns or upstream-failover regression would be invisible.

Node agent CNI conflist chaining (`agent-cni-alerts.yml`)¶

Covers the node agent's presence in the node's active CNI conflist. Aether is a chained plugin inside another CNI's conflist, so any competing writer that rewrites that file from its own template silently unchains us — on Talos, kube-flannel's init container cp -fs its ConfigMap template over 10-flannel.conflist on every flannel pod recreation, which a bootstrap-manifest re-sync triggers. That was the 2026-08-29 fleet-wide ~2h outage (#645).

Alert	Severity	Catches
`AetherCNIConflistUnchained`	critical	per node: entry gone and unrepairable, or the agent reporting nothing at all
`AetherCNIConflistReasserting`	warning	strips the agent self-healed — invisible to the gauge, visible only in the counter
`AetherCNIConflistMetricsAbsent`	critical	metric renamed, OTLP path broken, or the re-assert loop disabled fleet-wide

Why `absent()` is not the per-node answer¶

The gauge is pushed by the agent, so a down agent produces no series rather than a 0. absent() over the bare selector is fleet-wide: it drops the node label and fires only when every agent is silent, so one node's agent going away produces neither the == 0 arm nor the absent() arm — vacuous for exactly the outage it would be written for.

Per-node absence needs an inventory joined with unless. The inventory is aether_mesh_dns_ready, from the separate mesh-dns DaemonSet (#583) carrying the same node label — which is why it survives the agent outage the rule exists to catch. Residual gap, stated plainly: a node where both the agent and mesh-dns are silent falls out of the inventory and is covered only by the fleet-wide meta-rule.

Why the counter is a separate rule¶

The re-assert loop repairs in ~2.5s and the OTLP PeriodicReader exports every 60s, so a healed strip is exported as 1 and never observed as 0. The gauge under-reports by construction; aether_agent_cni_conflist_reasserts_total is the only evidence the strip happened at all.

These expressions have promtool unit tests in the GitOps repo (clusters/talos-main/prometheus/rules_test.yaml, run by CI), including a node whose chained series is omitted entirely — the case absent() misses.

Installing¶

There is no Prometheus operator on talos-main (no PrometheusRule CRD) and the Grafana install has no alerting sidecar — only dashboard and datasource sidecars. So these rules are delivered through the Prometheus helm values:

# prometheus helm values
serverFiles:
  alerting_rules.yml:
    groups:
      # contents of mesh-dns-alerts.yml
      # contents of agent-cni-alerts.yml

prometheus.yml's rule_files already lists /etc/config/alerting_rules.yml — the wiring exists, the file was just empty — so populating that key is the only change needed to make the rules evaluate.

Do not helm upgrade by hand: those values are reconciled by Flux (see below).

Alert delivery (Alertmanager -> Slack + GitHub issue)¶

This file is the source of truth for the rules; it is not where they are deployed from. talos-main is GitOps-managed by Flux, so nothing here is applied by hand — the rules and the delivery path both live in bpalermo/k8s-talos-main:

what	where
alert rule groups	`clusters/talos-main/prometheus/values.yaml` → `serverFiles.alerting_rules.yml`
Alertmanager routing + `github-slack` receiver	same file → `alertmanager.config`
Slack webhook URL	SOPS Secret `alertmanager-slack` (ns `prometheus`), mounted as a file via `extraSecretMounts` → `global.slack_api_url_file` (Alertmanager cannot interpolate env vars, and `alertmanager.config` renders into a plaintext ConfigMap)
GitHub receiver Deployment	`clusters/talos-main/alertmanager-github-receiver/`
GitHub PAT	SOPS-encrypted `secret.sops.yaml` in that dir (AWS KMS + PGP)

One receiver, two legs. Slack (#alerts) is the pager: GitHub issues created with your own PAT are self-authored, and GitHub does not notify you about your own actions — issues alone reach nobody. The GitHub issue is the durable record: a firing alert opens an issue on this repo labelled alert, and closes it on resolve. Issues are keyed on GroupKey, and group_by: [alertname] makes that stable — so one issue per condition listing every firing node, and a flapping alert reopens its issue rather than opening a new one.

Things that are easy to get wrong, already handled there:

There is no Watchdog rule, on purpose. The stock always-firing dead-man's switch was removed together with its → "null" route: nothing outside the cluster consumed the heartbeat, so it proved nothing and would file an immortal issue if it ever leaked to a real receiver. (If you re-introduce an always-firing rule, remove rule before route on teardown — the reverse ordering once briefly filed one.) A real dead-man's switch needs an external sink and is still an open item.
measurementlab/alertmanager-github-receiver cannot run on talos-main. It is the receiver everyone cites, but it is published amd64-only (neither latest nor v0.11 is a multi-arch index) while every talos-main node is arm64. We use ghcr.io/pfnet-research/alertmanager-to-github, which ships a genuine multi-arch index, pinned by digest.

Verify what is actually loaded:

kubectl get cm prometheus-server -n prometheus \
  -o go-template='{{index .data "alerting_rules.yml"}}' | head

Rules appear under Alerts in the Prometheus UI, and ALERTS{alertstate="firing"} becomes queryable via the Grafana Prometheus datasource.

Do the collector bump BEFORE enabling these¶

Prometheus here is OTLP-receive-only (all scrape configs disabled, web.enable-otlp-receiver), so series arrive by push. Under memory pressure the otel-collector sheds gauge exports -- during the 2026-07-25/26 soak one node vanished from aether_mesh_dns_records and snapshot_age_seconds for a stretch while its resolver was provably healthy.

Four of the seven rules key on exactly those gauges (MeshDNSNoRecords, MeshDNSNoUpstreams, MeshDNSWatcherInactive, MeshDNSMetricsAbsent). With a shedding collector a healthy node whose export was dropped is indistinguishable from a broken one, and absent() fires on the telemetry gap rather than a resolver failure. Enabling these before the collector has headroom trains everyone to ignore them within a week.

MeshDNSSnapshotStale and MeshDNSResolutionFailing are safer -- the latter is counter-based, and counters self-heal across a refused export.

Prove each rule fires before trusting it¶

Break something deliberately and confirm the expected rule -- and only that rule -- fires. MeshDNSResolutionFailing most of all: its exclusion of http_error and inclusion of generic timeout is reasoned (a hung resolver surfaces as a context deadline, while http_error is the cross-node backend path) but has never been observed firing.