Developer Runbook

A practical guide for building, testing, and running Aether from a clone — the day-to-day developer loop and the local multi-cluster end-to-end harness.

For installing and operating Aether on a real cluster (workload onboarding, routing, observability), see getting-started.md. For the chart values and CLI/annotation reference, see configuration.md.

---

1. Prerequisites

Tool	Why	Notes
Bazel via Bazelisk	build system	The pinned version (9.2.0) is read from .bazelversion; just run bazel … and Bazelisk fetches it.
Go 1.27.1	language toolchain	Managed by rules_go; you rarely invoke go directly (use bazel run @rules_go//go …).
Docker (or Colima on macOS)	container images + integration tests	Integration tests spin up a real etcd via testcontainers-go.
kubectl, Helm 3 (OCI)	deploy / e2e
kind	local e2e	Only needed for the e2e/*.sh harnesses (e2e/README.md).

macOS + Colima one-time setup

If you use Colima for Docker, generate the Bazel Docker-socket config once:

./bazel/configure_colima.sh

This writes .bazelrc.colima (gitignored); it is auto-enabled on macOS via --config=colima, so sandboxed integration tests can reach the Docker socket.

---

2. Build

All build/test entry points are in the Makefile (thin wrappers over Bazel).

make build              # bazel build //...  (everything)

make build-agent        # //agent/cmd/agent/...        (node agent + edge)
make build-mesh-dns     # //agent/cmd/mesh-dns/...     (slim standalone mesh-DNS daemon)
make build-proxy-supervisor  # //agent/cmd/proxy-supervisor/... (Envoy hot-restart supervisor)
make build-registrar    # //registrar/cmd/registrar/...
make build-cni-install  # //cni/cmd/cni-install/...

There is no make build-controller; build it directly with bazel build //controller/cmd/controller/....

The aether-proxy (custom Envoy) is NOT built here. It lives in a separate sibling Bazel workspace under proxy/ (its own .bazelversion = 8.7.0) and compiles Envoy from source (multi-hour; use a warm cache / CI). Build/load it with make load-proxy-image only when you need a fresh proxy image. See proxy/README.md and proposal 010.

Bumping the Envoy pin

The proxy's Envoy is ONE pin written in two files: the envoy/envoy_api versions (plus the envoy single_version_override that carries the patches, and the .envoy-suffixed sibling deps) in proxy/MODULE.bazel, and the envoyproxy/bazel-registry commit on proxy/.bazelrc's --registry= line. The registry drops old snapshot directories as it moves on, so the version resolves only at a registry commit that still carries it (proxy/README.md, "Envoy version bumps", has the background and the traps).

Two scripts own it:

• scripts/check-envoy-pin.sh is the gate. Offline it asserts envoy == envoy_api == the override's version, that proxy/.bazelrc has exactly one registry line and it names a full 40-hex commit (never a branch), and that no other snapshot version is spelled out in either file. --online (CI's envoy-api-parity job, which runs on every PR including proxy-only ones) also asserts the pinned registry commit serves modules/<m>/<v>/MODULE.bazel for envoy, envoy_api and every .envoy sibling, and that the snapshot's source.json archives the Envoy commit whose short sha is in the version (the workspace records only the short sha; the full one lives in that file). --self-test is its offline harness (CI's shell job).
• scripts/bump-envoy-pin.sh <envoy-commit | latest> is the maintainer tool. It is never run in CI and never builds Envoy.

The procedure:

1. scripts/bump-envoy-pin.sh latest (or an Envoy commit a snapshot was cut from). Dry run: it resolves the registry commit (the newest one that still carries that snapshot), prints the rewrite of both files as one diff, flags any .envoy module the new envoy/envoy_api request that the registry does not serve at that commit (then it is not self-consistent there: wait, or pick the commit by hand with the README's loop), and prints two reports:
2. the upstream .bazelrc diff between the old and new Envoy commits, restricted to the scopes proxy/.bazelrc mirrors (unconditional, :linux, :clang*, :libc++). ~ changed, - removed, + added; [mirrored] marks a flag proxy/.bazelrc sets. Every [mirrored] line and every + line needs a decision in proxy/.bazelrc's "UPSTREAM: module wiring" section.
3. the carried-patch report: each patch in proxy/bazel/patches/, in MODULE.bazel order, applied to a sparse shallow checkout of the new Envoy commit (and of the current one as a baseline, where all of them must say applies). applies = still needed; UPSTREAM = reverse-applies cleanly, the fix is in this Envoy, drop the patch (issue #980); CONFLICT = refresh it against the new tree. A CONFLICT is not applied, so a later patch stacked on it can report CONFLICT only because of it.
4. Re-run with --write, then rewrite the pin's explanatory comment in proxy/.bazelrc ("why this bump") by hand; the script moves the commit and version it mentions but not the reasoning.
5. Drop every UPSTREAM patch (the file, its single_version_override entry, its comment block in MODULE.bazel and its proxy/README.md row) and refresh every CONFLICT one.
6. Refresh the lock and check resolution with the proxy workspace's own Bazel (module resolution only, no Envoy build): cd proxy && bazel mod graph --depth=1 --lockfile_mode=update, then bazel mod explain @quiche @protobuf @abseil-cpp.
7. cd proxy && bazel test //bazel/patches:carried_patch_tests.
8. scripts/check-envoy-pin.sh --online && scripts/check-envoy-api-parity.sh (a new Envoy minor can require bumping go-control-plane/envoy; the parity script prints the command).
9. One PR. Any change under proxy/** runs the hours-long proxy build on the PR and proxy-release on merge, so keep unrelated edits out of it.

---

3. Test

make test               # bazel test //...              (unit + integration; needs Docker)
make test-unit          # --test_tag_filters=-integration  (no Docker)
make test-integration   # --test_tag_filters=integration   (needs Docker)
make test-race          # all tests with the Go race detector

Run a single target directly:

bazel test //agent/internal/xds/cache:cache_test

Integration tests are tagged integration and sized medium; many are also guarded by testing.Short(), so you can force unit-only behavior on a specific target with:

bazel test //... --test_arg=-test.short

envoy --mode validate: the released proxy or the one a PR builds

//agent/test/envoy_validate runs envoy --mode validate over aether-generated configs, and it has two modes. Default (bazel test //agent/test/envoy_validate/..., what the ci workflow runs): the binary is @pinned_envoy_linux_<arch>, lifted by //bazel/proxy_pin from the aether-proxy image charts/aether/values.yaml pins — the proxy the mesh deploys today (#709). Built proxy (agent/test/envoy_validate/validate-built-proxy.sh <envoy-binary>): the script points --override_repository for the host arch's pinned repo at a local repo whose envoy symlinks to the given binary, so the same tests validate against an Envoy built elsewhere — typically the //proxy workspace's //:envoy. Bazel digests the binary behind the symlink, so a rebuilt proxy reruns the gate rather than hitting a cached pass. .github/workflows/proxy.yml runs this mode on every PR that touches proxy/, on both arch legs, against the binary that leg just built: without it a proxy PR that compiles an extension out or changes a carried patch was validated against the previous release. Either way the log names the binary (envoy under validation: <path> (version: ...)), and the test fails on anything whose --version is not Envoy's, so a stand-in like /bin/true cannot pass the positive cases.

The race gate

.bazelrc defines test:race --@rules_go//go/config:race, so --config=race is the supported spelling and make test-race is the whole-tree run of it:

bazel test --config=race //agent/internal/meshdns:all //agent/storage:all
bazel test --config=race --runs_per_test=3 --nocache_test_results //common/grpcserver:all

Prefer the scoped spelling while developing: a bare //... race run currently reports known test-only races that are being fixed separately (#772 phase A2), so its failures are not necessarily yours. --runs_per_test=3 --nocache_test_results is what actually shakes out a scheduling-dependent race. And a clean run is not evidence of absence: the detector only reports interleavings a test actually produced, so a race between two goroutines no test runs concurrently stays invisible no matter how often you run it.

What CI's race job runs how often. Impacted tests run once under the detector, and the tests of the concurrency-heavy packages (the list is in .github/workflows/ci.yaml) run three times. One target of such a package is the exception and runs once: //agent/internal/xds/cache:cache_walk_test, the fifteen seeded random walks of proposal 043. They are single-goroutine and deterministic for a seed, so a second and a third run repeat the first step for step; they are nine tenths of that package's test time, and at three runs they had brought the job to 54 of its 60 minutes. Three runs was the figure until then, and is still the figure for everything else in the package (cache_test, the interleaving tests among them). The split is one macro call in the package's BUILD file (cache_go_test, defs.bzl): a new walk is added to the list there, and a --test_filter that names a walk goes to cache_walk_test.

The external-harness contract

A harness outside this repository (one that rolls the mesh's workloads, reads its metrics and logs and grades a run) depends on names this repository chooses. They are written down in test/harnesscontract/external-harness.yaml: the agent's pin metrics with their labels and the closed reason set, the prober's aether_probe_requests_total with its tier and result sets, the fields of the AETHER_PROBE_FAIL lines, the annotations, labels and ports a workload manifest names, and the chart objects a harness addresses (workload names and namespaces, pod labels, container names, rolling-update strategies).

bazel test //test/harnesscontract:checks

Each entry is compared with the code that produces the thing, by a test in the package that owns it: the real instruments' collected series, lines the real fail log wrote, a listener the real generator built, a helm template of the packaged chart. A rename fails with a message that names the contract file.

The rule. Change the code and the contract together, in one pull request, and say so in its description. Removing an entry or changing what it means (a rename, a value removed from or added to a closed set, another roll strategy, a metric that keeps its name and counts something else) bumps the file's version; adding an entry does not. A harness pins the version it was written against; the README's "Versions" table says what each bump changed (version 2: what aether_agent_xds_acked_tls_clusters counts and when it is absent, #1508). For what the file's keys say, the bump is a test: external-harness.lock.yaml beside the contract holds one line per promise of the current version, and //test/harnesscontract:harnesscontract_test fails when a promise left or changed and version did not (it prints the lock for the new version once it is bumped, and the lines to add for a new promise). On a pull request scripts/check-harness-contract-bump.sh also compares the lock with the one at the base, so an entry removed together with its lock lines needs the bump too. A change of meaning that leaves every key as it was, like version 2's, is not seen by either and is bumped by hand. test/harnesscontract/README.md has the table of what each test compares, what is kept by review alone, and how to add or remove an entry.

Which contract a deployed mesh satisfies. Since chart 2.4.28 the agent DaemonSet carries the contract's version in an annotation on its own metadata, so a harness can ask the mesh before it reads anything else:

kubectl -n aether-system get daemonset aether-agent \
  -o jsonpath='{.metadata.annotations.aether\.io/harness-contract-version}'

(aether-system and aether-agent are the documented release's namespace and name.) The answer is the version of external-harness.yaml at the commit the chart was packaged from, as a string. Three things it is not:

• It identifies the contract, not the build. Charts of different versions answer the same for as long as the contract's version stays, and the annotation says nothing about which images run. For the build, read the chart version (helm.sh/chart on the same object) and the image digests.
• An empty answer is "not known", never a version: the chart is older than 2.4.28, or it was rendered from the source tree (helm template charts/aether) instead of from the published package, where the file the number comes from does not exist.
• It is on the DaemonSet object, not on its pods. A contract bump changes that one annotation on upgrade and replaces no pod; //charts/aether:aether_harness_contract_version_rolls_no_pod_test renders the package with another number and without one and requires every pod template to be byte-identical.

No value sets it and the chart holds no copy of the number: a build step of //charts/aether reads version from the contract file into the package (files/harness-contract-version), and //test/harnesscontract:aether_chart_test fails when the annotation either packaging renders and the contract's version disagree.

Code coverage

make coverage                                   # the whole unit suite
make coverage COVERAGE_FLAGS="--jobs=6"         # extra `bazel coverage` flags

scripts/coverage.sh writes coverage-report/ (git-ignored): coverage.lcov, coverage.xml (Cobertura), summary.md (the per-component table) and unrecorded.txt. .github/workflows/coverage.yaml runs the same script on every pull request and on every push to main, puts the table in the job summary and keeps the files as the coverage-report artifact (90 days). On a pull request its gate job then compares the total with main's and fails the coverage check when it dropped by more than one percentage point: "The gate" below. The comparison is the repository's own, because GitHub's native code coverage is not available to it ("The native upload is parked").

What the number is. Line coverage of every go_test not tagged integration, requires-root or manual (97 of 101 when this was written), over all first-party Go. Integration, e2e, kind, netns and soak runs do not count yet, so code only they exercise reads as uncovered.

• The whole suite, every time. ci tests bazel-diff's impacted subset; coverage cannot, because a percentage over a different set of tests per pull request is not comparable with main's. Bazel caches each test's coverage result, so locally an unchanged test is a cache hit (a second run: 1 s, 0 of 97 tests executed). In CI only the compiles are; see "CI cost" below.
• Components are derived, not listed. Every top-level directory with a go_library or go_binary, minus api (generated proto Go), test and e2e (harnesses) and bazel (build tooling) — the EXCLUDED list in the script. A new top-level component is measured from its first Go target with no edit. cmd/ mains are in: they are code.
• One subtree of a component is out too: agent/test (EXCLUDED_SUBTREES). It holds the Envoy-driven harnesses (envoy_validate, its generator, mtlspool, envoyargs, envoybin), which import agent/internal/... and therefore have to sit under agent/ (#1311); before that they were in test/ and excluded with it. Their two non-test files (the config builders and the generator's main) are fixtures, not product code, so they stay out of the denominator and the move changed no number. Their tests run like any other unit test, and the agent/internal/... lines those tests execute count. A subtree entry that no longer exists fails the run.
• Untested code counts as zero. By default bazel coverage instruments only packages that have a test in the run. The script passes an --instrumentation_filter over every component and makes every component go_library a target, so rules_go's baseline action (go tool cover over the sources the compile action would compile) gives each file no test links a zero-hit record with the line set a measured run would have. Files excluded by build constraints on linux/amd64 are absent, not uncovered; unrecorded.txt lists them (three today). Anything else in that list is a file no go_library has in its srcs: run make gazelle.
• _test.go files are never instrumented.

Baseline at introduction (2026-10-06, main at 56a330fc): 77.82 % (21 993 of 28 262 lines). The same run reads 77.96 % without the zero-hit records, and 78.91 % with Bazel's default instrumentation (27 853 lines): the honest denominator is 409 lines larger than the default one.

Run-to-run variation. Four uncached runs covered 21 993, 21 994, 21 991 and 21 989 lines: 77.80–77.82 %, a spread of 0.018 percentage points. Seventeen lines in six files were timing-dependent in those runs (cni/internal/util/watcher.go, common/signals/signals.go, agent/internal/proxy/hotrestart/supervisor.go, agent/internal/meshdns/lameduck.go, agent/internal/xds/server/refresh.go, agent/internal/xds/cache/identitybinding.go), so the widest gap two runs could show is 0.06 points. The first two CI runs read 77.84 % and 77.81 % (21 998 and 21 991 lines). Six of the runner's extra lines were watcher.go's closed-Errors exit (lines 119–121 and 135–137): Close leaves the watch goroutine with three ready select arms and the scheduler picks, which it did on the runner in every run, in about a third of plain go test runs on the workstation and never under Bazel there — a scheduling difference, not a privilege or kernel one (#1315). watcher_loop_test.go now takes each exit of that loop deterministically, so watcher.go is out of the timing-dependent set. Because CI re-runs every test (next paragraph), a pull request and main are two independent samples of that noise: a COVERAGE_MAX_DROP below roughly 0.1 would fail pull requests that changed nothing. The default of 1.0 is sixteen times the widest gap.

CI cost. The report job took 13 min 5 s cold (every instrumented compile executed on RBE) and 5 min 6 s warm (3 642 remote cache hits, 97 tests executed on the runner). It never gets faster than the warm figure, because test results are not cached in CI: Bazel reports "--remote_upload_local_results is set, but … the current account is not authorized to write local results to the remote cache", and test execution is local by design (--config=remote, --strategy=TestRunner=local). The same holds for every leg of ci; it is only visible here because this job runs the whole suite.

Since #1314 the jobs that run on main write those results and pull requests read them. Two BuildBuddy keys share one secret name, BUILDBUDDY_ORG_API_KEY: the repository-level secret is a CAS-only key (it may upload inputs and outputs, never an action result: the right type for a pull request, which could otherwise store a forged "test passed" that main would later trust), and the main environment holds a Read+Write key that shadows it. main.yaml's test job and this workflow's report job on main declare environment: main; the environment admits the main branch only, so a pull request's run cannot obtain the key. (release holds a Read+Write key too, for the publishing builds, next to the registry credentials; test jobs stay out of it.) What to expect: on main the warning above is gone; on a pull request it remains (that run still cannot write) and Bazel reports the unchanged tests as (cached) / remote cache hit. Two caveats: a coverage result is reused only by another coverage run, and race runs nowhere on main, so a pull request's race leg still executes every impacted test.

The gate. On every pull request the gate job of the coverage workflow compares the pull request's total line coverage with a baseline from main and fails when it is more than COVERAGE_MAX_DROP percentage points lower (default 1.0; a drop of exactly the threshold passes). The aggregate coverage job fails with it, so coverage is the one check to require. Only the total gates. The job summary also shows, for the reviewer and without gating: the two totals and their delta, a per-component table with each component's delta, and the coverage of every changed file (git diff <baseline>...<head>) that either report names, new and deleted files included. When the total dropped it adds the largest per-file drops: the ten files (--top-drops) whose covered-line count fell most, whatever the diff touched, and the files that left the report (removed) — open when the gate fails, collapsed when the drop is within the threshold, absent when nothing dropped. Patch coverage is deliberately not a gate: it punishes the pull request that touches old untested code.

• Which baseline. scripts/coverage-baseline.sh takes the coverage-report artifact of a successful run of the coverage workflow on main (a push or a workflow_dispatch run, never a pull request's or a fork's):
• the run for the pull request's base commit (github.event.pull_request.base.sha). The ruleset keeps a branch up to date with main, so this is normally main's head and the delta is the pull request's own. If that run is still going (a pull request pushed right after a merge) the job waits up to ten minutes for it;
• otherwise main's most recent successful run whose artifact still exists. The summary and the notice then say so and name the commit (main at 1a2b3c4d (main's latest report, not the base commit …)). The delta then also contains whatever main gained between the two commits.
• none: the job fails with Coverage gate: no baseline. It never passes with nothing to compare. Fix: run the workflow on main (Actions > coverage > Run workflow, or gh workflow run coverage.yaml --ref main), wait for it, re-run the failed job. Artifacts are kept 90 days, so this only happens after a quarter with no push to main, or if every recent run on main failed.
• Stacked pull requests. A gh stack member's base is the upgrade/** branch beneath it, a commit main never had, so it always takes path 2: it is compared with main's latest, cumulatively with the members beneath it. That is the number that matters, because the stack lands on main: a member cannot hide a drop behind the one below it, and conversely a stack whose first member drops 0.8 and second 0.4 fails at the second, where the total crosses the line. Add the tests there or split differently.
• The threshold. One constant, DEFAULT_MAX_DROP in scripts/coverage-compare.sh, overridden by the repository variable COVERAGE_MAX_DROP (Settings > Secrets and variables > Actions > Variables; gh variable set COVERAGE_MAX_DROP --body 0.5). A plain number of percentage points between 0 and 100; unset or empty means the default, and anything else (1%, -1, one) fails the job naming the value rather than being ignored. COVERAGE_MIN is the same for an absolute floor on the total, in percent; unset (today) means no floor. A relative gate alone lets the number erode by up to a point per pull request, and the floor is what stops that if it ever matters.
• Locally. The same comparison, against any baseline you have: gh run download <run id> --name coverage-report --dir /tmp/base, make coverage, then scripts/coverage-compare.sh --baseline /tmp/base/coverage.lcov --head coverage-report/coverage.lcov.

The gate failed. The ::error on the run gives both totals, the line counts and the delta; the component and changed-file tables say where it came from. When no measured file changed (the changed-files table reads None of the changed files is in either report), read Largest per-file drops: it names the files that lost covered lines although the diff did not touch them.

• Code lost its tests, or new code has none. Add the tests. A new file no test links counts at zero, which is the point.
• A test stopped running. A go_test that gained manual, integration or requires-root, or a deleted test target, drops out of the measured set while its library's lines stay in the denominator. No Go file changes, so only the per-file drops table shows which files that test was covering (#1319).
• The drop is legitimate. Deleting a large, well-tested package lowers the percentage although nothing got worse: 1 000 fully covered lines removed from today's tree is about −0.8 points. The lines columns of the tables show that shape (lines and covered fall together). The escape hatch is the threshold itself, and only the owner holds it: set COVERAGE_MAX_DROP high enough for that pull request (gh variable set COVERAGE_MAX_DROP --body 2.5), re-run the failed gate job, merge, and restore it (gh variable delete COVERAGE_MAX_DROP). While raised it applies to every open pull request, so keep the window short. The next pull request is compared with main's new, lower number; nothing has to be reset. There is deliberately no label or commit-message bypass: those are things a pull request's author controls, and a gate its subject can switch off is not one.
• Coverage gate: no baseline. See path 3 above.
• Coverage gate: malformed threshold. Fix or delete the variable it names.

//scripts:coverage_gate_test holds the comparison to a golden report and to verdicts worked out by hand, and the baseline selection to a fake gh.

The one remaining owner step. Nothing requires the workflow until coverage is added to the main ruleset's required status checks (next to ci, proxy, codeql and CodeQL). Do not add the ruleset's "Restrict code coverage" rule: it reads GitHub's native coverage, which has no data here.

The native upload is parked. GitHub's native code coverage (GitHub Code Quality) is not available to this repository: as of 2026-10-06 the upload API answers HTTP 404 on pull requests and HTTP 500 on main even with code-quality: write granted, and a repository owned by a personal account has no Code quality page under Settings to enable anything on (actions/upload-code-coverage#16). The upload job is kept but runs only when the repository variable CODE_COVERAGE_UPLOAD is true; by default it is skipped, with no warning, and a skipped upload does not fail coverage. If GitHub enables the feature, set the variable. With it on, a rejected upload is red (there is no continue-on-error): Coverage upload failed (HTTP 404): Not Found means the endpoint still does not exist for the repository; HTTP 403 with "not authorized" is a missing code-quality: write on the job; a processing failure names what GitHub could not parse. Convert locally and look at the document: scripts/lcov-to-cobertura.sh coverage-report/coverage.lcov. The three things read from it are the root line-rate, each <class filename=…> (repo-relative; the converter refuses anything else) and each <line number=… hits=…>. A fork's pull request is never uploaded (read-only token). //scripts:lcov_to_cobertura_test holds the converter to a golden file and //scripts:coverage_test holds the script's target selection.

CI: external repository fetches (#1001)

BuildBuddy caches actions, not repository fetches. Every external repository a CI job needs — each Go module behind gazelle++go_deps+… (fetch_repo → proxy.golang.org), every release asset from github.com — comes from Bazel's repository cache on the runner or from the network. Two mechanisms keep a flaky upstream from turning a job red:

• A persisted repository cache. bazel-contrib/setup-bazel restores ~/.cache/bazel-repo (Bazel 9 also keeps its repo contents cache under it, contents/, which is what makes a go_repository a local hit instead of a fetch_repo run). The key hashes MODULE.bazel, MODULE.bazel.lock, go.mod and go.sum, so a Go dependency bump gets a new key and the newest older entry is restored as the fallback. The root and //proxy workspaces are in separate namespaces (cache-version: root-1 / proxy-1). Exactly one job per workflow saves — diff in ci.yaml and main.yaml (the main entry is the one every PR falls back to), the test/build-push matrix per arch for the proxy — because its warm-up fetches for all of //...; every other job restores only. All of it lives in one composite action, .github/actions/setup-bazel: a job picks workspace: proxy or the root default and, if it is the saver, cache-save: true (the release and signing workflows keep explicit copies of the same values). The first job to finish used to save, so a job that fetched a handful of repositories (or, before the namespaces, a proxy job) could write the entry every later job restored as an exact hit and never re-saved. That is how the 2026-09-27 netns failure fetched googleapis/api from the network with a 1.9 GB cache freshly restored.
• A retried warm-up. Each Bazel job's first Bazel step is scripts/ci-bazel-warmup.sh <flags> <targets>: bazel build --nobuild (loading + analysis, which is where the fetches happen), retried up to 3 times with a 20 s / 40 s backoff only when the output shows a repository-fetch error. A broken BUILD file fails on the first attempt. The warm-up clears the remote executor, cache and BES backend, so it needs no BuildBuddy key and never shows up as an invocation; the real steps keep --config=ci / --config=remote unchanged. scripts/check-ci-bazel-warmup.sh (in the shell job) pins that contract with a stub bazel.

Reading a red job:

The log shows	What it means	Do
::warning::bazel warm-up attempt 1/3 hit a repository-fetch error, then success	the retry absorbed a blip	nothing
::error::bazel warm-up: repository fetch failed on all 3 attempts	the upstream was down for the whole ~1 min of backoff	re-run the job once the upstream answers (gh run rerun <run id> --failed: the job concluded failure, so --failed takes it); a re-run restores the same cache, so it only re-fetches what failed
a fetch_repo / Error downloading failure in a step after the warm-up	that step needed a repository the warm-up's targets did not cover (e.g. a bazel run of a tool, bazel-diff at the base revision, which falls back to a full run on failure)	re-run the failed job (gh run rerun <run id> --failed); if it repeats, add the target to that job's warm-up
bazel warm-up failed (exit N) with no repository-fetch error	a real analysis failure	fix the change; a re-run will not help

A plain re-run is still the answer for anything the warm-up does not cover: the Go tool's own downloads in deps-audit (go list -m all talks to proxy.golang.org directly, outside Bazel), docker/kind image pulls in the e2e jobs (CI: Docker Hub pulls below says which of them still reach Docker Hub), and a cold namespace (the first run after a cache-version bump, or after GitHub evicted the entry: the repository has a 10 GB cache budget and each entry is ~2 GB).

To check what a job restored, open its Setup Bazel step: Cache hit for: setup-bazel-root-1-linux-x64-repository-<hash> is an exact hit; Successfully restored cache from … with a different hash is the fallback.

CI: Docker Hub pulls (#1602)

The repository holds no Docker Hub credential, so a pull from Docker Hub on a runner is anonymous: it shares the runner address's rate limit and needs auth.docker.io/token to answer. Both have failed required jobs in which no test had failed. The signature, in a race, integration or e2e log:

Get "https://auth.docker.io/token?…": net/http: request canceled … (Client.Timeout exceeded while awaiting headers)
toomanyrequests: You have reached your unauthenticated pull rate limit

Two things keep the required jobs off it:

• The testcontainers reaper is off under --config=ci (test:ci --test_env=TESTCONTAINERS_RYUK_DISABLED=true in .bazelrc). testcontainers/ryuk was the only Docker Hub image of the race and integration jobs (the etcd they start comes from gcr.io), pulled by tag with no credential. It cleans up after a test that crashed; a runner is discarded with everything on it, and each package's TestMain terminates its own containers. A local run keeps the reaper.
• The kind node image comes through a mirror. .github/actions/setup-kind, the one way a job gets kind, runs scripts/docker-hub-mirror.sh before any cluster exists. It adds https://mirror.gcr.io (Google's public pull-through cache of Docker Hub, no credential) to the runner's /etc/docker/daemon.json, reloads the daemon (a signal, not a restart: registry-mirrors is re-read live), and pulls KIND_NODE_IMAGE by its unchanged reference. The daemon asks the mirror first and Docker Hub only for what the mirror does not answer, and the reference keeps its @sha256 digest, so the content is the pinned one whichever served it. kind then finds the image present.

Reading the Pull the node image through the Docker Hub mirror step:

The log shows	What it means	Do
https://mirror.gcr.io serves kindest/node@sha256:…; the daemon asks it first	the mirror has the pinned digest and the daemon uses it	nothing
::warning::… does not serve kindest/node@sha256:… (HTTP 404)	the mirror does not have that digest; the pull went to Docker Hub anonymously	nothing if the job passed. If it repeats on every run after a node image bump, the mirror never cached the new image: say so on the bump's pull request
::warning::… the daemon was not given the mirror (after a warning about daemon.json or the reload)	the runner image changed how its Docker daemon is configured; every pull is back on Docker Hub	read the warning before it; fix scripts/docker-hub-mirror.sh
::error::… could not pull … in 3 attempts	neither the mirror nor Docker Hub answered	re-run the job; no test ran

What still pulls from Docker Hub anonymously: the workloads the nightly e2e/*.sh harnesses start inside their kind clusters (curlimages/curl, hashicorp/http-echo, openpolicyagent/opa). Those are pulled by the node's own containerd, which the runner's daemon configuration does not reach. No required job runs them; the per-PR e2e job's only other image is echo-basic, from gcr.io. A red nightly suite with the signature above is a re-run.

//scripts:docker_hub_mirror_test holds the script, the .bazelrc line and the workflow steps that depend on it; //e2e:kind_pin_test holds the step's place in setup-kind.

CI: what the ci check tests, and what makes it pass (#1459, #1460)

One tree. On a pull request every job of ci.yaml builds the run's commit, which is the merge of the pull request's head into its base. The diff job lists the impacted targets on that same commit: scripts/ci-merge-range.sh reads the range from the merge (its first parent, the base, to the merge itself) and scripts/ci-impacted-targets.sh refuses any other HEAD_SHA than the checkout. The artifact carries the commit in impacted_commit.txt; a job that reads the lists gets them through .github/actions/impacted-lists, which fails when a list is missing, when the lists are for another commit than the job's checkout, or when a has_* output disagrees with its list. The test job of main.yaml (the post-merge run) gets its lists through the same action, under the name its own diff uploads them (artifact-name, #1488).

The required check. ci passes when scripts/ci-gate.sh accepts the results and outputs of every other job. A skipped job passes only where a job said so: changes wrote control_plane=false, or diff wrote false to the output the leg depends on. "Nothing impacted" is diff writing false four times, and the log of the ci job says which case it was:

The ci job's log shows	What it means	Do
decision: impacted; had to run: test race …	those legs ran and succeeded	nothing
decision: nothing impacted (diff wrote false to has_any, …)	bazel-diff found no target the change reaches; the legs are skipped by design	nothing
decision: the control plane is untouched (changes wrote control_plane=false)	a proxy-only change; the proxy workflow is its gate	nothing
job <x> ended as failure / cancelled	that job is the one to read	fix it, or re-run it if no step ran: gh run rerun <run id> --failed when it ended failure, the whole run (gh run rerun <run id>) when it was cancelled (which command)
job diff succeeded and its output has_… is not set	diff wrote no decision, so the legs skipped on an empty value	read the Compute impacted targets step; this is a defect in the workflow or the script, not in the change
job <x> was skipped, and diff says it had to run	a leg did not run although its output is true (usually a job it needs failed, reported on its own line)	read the failed job first
job <x> is not in the needs of the ci job / has no rule for it	a job was added to ci.yaml and not to the needs of ci or to scripts/ci-gate.sh	add both; //scripts:ci_gate_test holds them to the workflow

The proxy check is decided by the same script with the table for proxy.yml (scripts/ci-gate.sh proxy, #1489). Its shell and test jobs skip on a pull request that does not touch the proxy workspace, and that passes only because changes wrote proxy=false:

The proxy job's log shows	What it means	Do
decision: the proxy workspace changed (changes wrote proxy=true); had to run: shell test	both ran and succeeded	nothing
decision: the proxy workspace is untouched (changes wrote proxy=false)	nothing under the path filter changed; ci is the gate of this change	nothing
job changes succeeded and its output proxy is not set	the path filter wrote no decision, so shell and test skipped on an empty value	read the changes job; this is a defect in the workflow, not in the change
job <x> was skipped, and changes says proxy=true	a job did not run although the workspace changed	read that job; if no step ran, re-run the WHOLE run, gh run rerun <run id>: a skipped job is not one --failed selects (which command)

The post-merge run (main-post-merge, main.yaml) is decided by the same script with a third table (scripts/ci-gate.sh main, #1501), in its main job. No ruleset requires it, since it runs on a push: a failure blocks nothing and shows as a red run of that commit on main, which the rolling issue below reports. Its test job skips when bazel-diff finds nothing the merge reaches, and that passes only because diff wrote false:

The main job's log shows	What it means	Do
decision: impacted; had to run: test	test ran and succeeded	nothing
decision: nothing impacted (diff wrote false to has_any, has_unit and has_integration)	bazel-diff found no target the merge reaches; test is skipped by design	nothing
job diff succeeded and its output has_… is not set	diff wrote no decision, so test skipped on an empty value and the merge was not tested	read the Compute impacted targets step. If it failed on something transient, re-run the WHOLE run, gh run rerun <run id>: diff concluded success, so --failed would re-run the main job alone and it would read the same empty output (which command). If the workflow or the script is at fault, a re-run repeats it (a re-run executes that commit's own workflow): fix it in a new commit, and test the missed commit yourself (make test on a checkout of it), or the merge stays untested
job test was skipped, and diff says it had to run	test did not run although has_any is true	read the test job; if no step ran, re-run the WHOLE run, gh run rerun <run id>: a skipped job is not one --failed selects (which command)

A post-merge run that did not pass files an issue (#1506): "main-post-merge: a commit on main failed its post-merge run", labelled bug and ci. One issue, reused: main-post-merge-watch (main-watch.yaml) runs when a post-merge run completes and records the commit, the run, the jobs that did not pass, whether the run's gate passed and the command that re-runs it, in the issue body for the first failure and as a comment for each later one. The logic is scripts/main-post-merge-watch.sh.

What the issue means: the commit it names is on main and its post-merge run is red. Nothing was blocked. Each entry says which of two things that is, from the run's gate, the main job:

• The main job did not succeed (it failed, was cancelled or never ran): the commit is not validated. If a target is broken, then until it is dealt with a pull request whose impacted targets include it fails ci for a reason that is not its own.
• The main job succeeded, and another job did not (refresh-pin-prs, which runs whatever the gate concluded): the commit was validated, and no target on main is broken by it. Do not look for a regression. The run is red all the same and the entry stays until that job passes: read that job, and re-run it. What it missed is its own work (the open pin-bump pull requests were not brought up to date for this push).

A run that was cancelled with the gate green and no failed job is not filed at all (the table below); one that ended failure with the gate green is, as the second case, and so is a cancelled one in which another job had already failed.

After gh run rerun --failed on the second case, the new attempt re-runs that job alone and its jobs may not include main. The watcher then takes the gate's conclusion from the latest earlier attempt that has it, and the entry says so ("succeeded in attempt 1 and was not re-run in this attempt"). If that attempt cannot be read, the entry says that whether the commit was validated is not known: read the main job on the run page.

Re-runs and the order of things. A re-run is a new attempt of the same run, and the watcher of an attempt reads the run as it is when the watcher runs. If a re-run was started quickly, the watcher of the failed attempt finds the run in progress and judges nothing. So each watcher also looks at the attempts before the latest one (unless the latest passed), back to the first or to one that succeeded, and records what did not pass and is not on the issue yet; such an entry says "Recorded late". The rule is: an attempt that did not pass is recorded once, unless a later attempt of the same run succeeded. For two attempts of one run the issue ends up with:

Attempt 1	Attempt 2	On the issue
did not pass	none, or still running	attempt 1
did not pass	success	nothing failing: attempt 1 is cleared if it was recorded, and never recorded otherwise
did not pass	did not pass	both attempts
did not pass	cancelled, validated (not filed by itself)	attempt 1: only a green attempt clears it
cancelled, validated	did not pass	attempt 2
cancelled, validated	anything else	nothing

That holds whether the watcher of attempt 1 ran before or after attempt 2 began. If an earlier attempt cannot be read, the watcher records what it can and its run fails: re-run it, and it adds the rest without repeating anything.

What to do, for each commit the issue names:

1. Open the run it links and read the first failed step.
2. No step ran, a registry or GitHub answered 5xx, a job never got a runner or hit its time limit: re-run it, with the command its entry gives. Which one depends on the run's jobs:

The jobs of the run	Command
every job that did not pass concluded failure	gh run rerun <run id> --failed: the failed jobs and the jobs that depend on them (the main job among them)
no job started (startup_failure, a run skipped as a whole), a job was cancelled or timed out, or the jobs could not be listed	gh run rerun <run id>: the whole run

--failed asks GitHub to re-run "the failed jobs and their dependent jobs". A run with no job that concluded failure gives it nothing to select, so no new attempt is made and the issue never clears; whether it takes a cancelled job is not documented, so the watcher does not rely on it. The whole run is always enough. When the re-run passes, the watcher comments that the commit passes now, and closes the issue once no commit named on it is still failing. 3. A real failure: fix it in a new pull request and write Closes #<the issue> in it. Re-running the old run cannot pass, since it tests the old commit. The same holds for a startup_failure whose cause is the workflow file of that commit: a re-run executes the commit's own workflow.

A green run of a later commit never closes the issue. Each post-merge run tests only what its own merge reaches, and a merge that reaches nothing passes without running a test: the next documentation-only merge would close the issue with main still broken. So a commit is cleared only by its own run, and a failure fixed by a later commit is closed from that pull request or by hand. For the same reason an older commit's failure is filed even when it arrives after a newer commit's green run; the entry then says that main has moved on and names its head.

The run ended	Filed?	Why
failure, timed_out	yes	the run is red: the commit failed it, or (failure with the main job green) another job did, and the entry says which
startup_failure	yes	the workflow file on main is not valid: no job ran, and every later push fails the same way
skipped, action_required, anything unknown	yes	nothing validated the commit
cancelled, the main job succeeded, and another job failed	yes	the red run with a green gate, cut short: refresh-pin-prs had failed when the run was cancelled. The entry says the commit was validated
cancelled, and the main job did not succeed	yes	main.yaml has no concurrency group, so no newer push cancels a run: a job hit its time limit or never got a runner, or someone cancelled it, and the commit is not validated
cancelled, the main job succeeded, and every other job succeeded, was skipped or was cancelled	no	diff and test did their work; what was cancelled is another job (refresh-pin-prs waiting for a runner)
success	no	clears that commit if the issue names it

A closed issue is not written to: the next failure opens a new one. The one exception is a close that raced with a failure. Watchers of different runs are not serialised, so a green run's watcher can close the issue in the moment another records a failure on it; whichever of the two notices reopens it, and the log of that main-post-merge-watch run says reopened #<n> and why. An issue left open with every commit on it passing is the harmless side of the same race: close it by hand. Only the issue that github-actions[bot] opened, and that account's comments, are read, so an issue or a comment someone else writes with the same title or marker changes nothing.

The watcher is a workflow_run workflow, so it runs from main only and its own failures are red runs of main-post-merge-watch (an API error while reading or writing the issue). Re-run a failed one: it writes nothing twice. To see what it would do with a given run without writing anything:

gh workflow run main-watch.yaml -f run_id=<id of a main-post-merge run>   # dry_run defaults to true

The log of that run prints the verdict and the text it would file. A dry run has a concurrency group of its own, so it never takes the place of a pending watcher of the same run. With -f dry_run=false it files for real; use that only on a run whose commit you then mean to clear or whose issue you will close by hand.

A cancelled run of coverage or codeql (both cancel the previous run of the same pull request when a new commit arrives) shows its summary job as failed on the commit that was superseded. That is deliberate: a summary job that was skipped would satisfy the ruleset, so it runs and fails. The checks of the newest commit are the ones that count. ci has no concurrency group and is not cancelled this way.

Re-running a run: which command (#1553)

"Re-run it" is one of two commands, and the wrong one does nothing:

What the run looks like	Command	Why
ALL of: at least one job did not pass; EVERY job that did not pass concluded failure (a step failed, a registry or GitHub answered 5xx); and the gate job (ci, proxy, main) is not the only one that failed	gh run rerun <run id> --failed	re-runs "the failed jobs and their dependent jobs", and keeps the jobs that passed
anything else: no job started (startup_failure); a job was cancelled (it never got a runner, the run was cancelled) or timed out, alone or next to a failed one; a job was skipped that should have run; or the gate failed and nothing else did (the job at fault concluded success)	gh run rerun <run id>	--failed has nothing to select, is not documented to select that job, or selects only the gate, which reads the same inputs again
you do not know	gh run rerun <run id>	the whole run is always enough; it costs the jobs that had passed

The rows do not overlap: the first needs all three of its conditions. Whether --failed also takes a cancelled job is not documented and not relied on here (#1552).

<run id> is the number in the run's URL (…/actions/runs/<run id>); a job's URL holds it too, before /job/. One job alone is gh run rerun --job <job id>, the number after /job/: it re-runs that job and the jobs that depend on it.

The issues the workflows open (#1532, #1568)

Nine titles are opened by workflows, never by a person: the five below that stand alone, and the four of the expected-red control. Each is ONE rolling issue: reused while the condition lasts, and (where the table says so) closed by the workflow when it ends.

Title	Opened by	Labels	Closed by
main-post-merge: a commit on main failed its post-merge run	main-watch.yaml	bug, ci	itself, when no commit on it is failing
CI: workflow runs stuck before they started	stuck-runs.yaml	bug, ci	itself, when nothing is stuck
publish-verify: the expected-red control … (four titles)	publish-verify.yaml	bug, ci	itself, by the next run whose control goes red
publish: artifacts missing for a commit on main	publish-verify.yaml	bug, ci	itself, by a green run for main's head or a green sweep
Nightly e2e failing	the report-failure job of e2e.yaml	bug, ci	a person
CI: third-party image pins are behind their tags	third-party-images.yaml	enhancement, ci	itself, when every pin is current

Three rules hold for the issues of stuck-runs.yaml, publish-verify.yaml and the report-failure job, which share scripts/rolling-issue-lib.sh. The other two differ, as said after them.

• Only the workflow's own issue is its rolling issue. It is the one opened by github-actions[bot] under exactly the title, and only what that account wrote on it is read back (the hidden markers that hold a set of runs, or the uncancellable runs). The repository is public: an issue you open under one of these titles is yours. No workflow comments on it or closes it, and it does not stop the workflow opening its own. A marker pasted into a comment does nothing.
• It is found by listing, not by search. The search index lags a create by seconds, which is how two failures close together opened two issues. If two runs do open one in the same moment, the newer is closed as a duplicate and its report becomes a comment on the older.
• It is opened with a kind and an area label. If a label no longer exists the issue is still filed, without it, and the run that filed it fails with was opened without the label(s) …: create the label again (gh label create ci), or change the script that names it.

The two that do not use the library:

• The post-merge watcher (scripts/main-post-merge-watch.sh) follows the first two rules with its own code. When a label is gone it files the issue without labels and warns; its run does not fail.
• The pin report (scripts/third-party-images-report.sh) takes only an issue github-actions opened, but finds it by search, so two runs close together can still open two. When a label is gone its create fails, no issue is filed, and the run fails: create the label, then run it again (gh workflow run third-party-images.yaml -f dry_run=false).

Stuck workflow runs

publish.yaml, proxy-release.yml and pages.yaml serialise on a concurrency group with cancel-in-progress: false, and GitHub keeps at most one pending run per group. A run whose job never gets a runner therefore blocks every later run of that workflow, silently: the later ones show pending with no jobs, and each newer one replaces the previous as cancelled. On 2026-10-05 a publish run sat queued for five hours this way, and a pages run queued since 2026-10-02 had held up every website deploy for three days. A job timeout-minutes does not help: it only counts once the job has started.

.github/workflows/stuck-runs.yaml checks every 30 minutes (scripts/stuck-runs.sh). A run counts as stuck once it has been pending / waiting / requested, or has had a job sitting queued, for more than 60 minutes. Every stuck run goes on one rolling issue, CI: workflow runs stuck before they started. The issue gets a new comment only when the set of stuck runs changes, and it is closed once nothing is stuck.

The watchdog cancels a run only when the run is superseded. All of these must hold:

• it is a push or pull_request run, on its first attempt, and not from a fork;
• either its branch has been deleted, or both:
• its commit is no longer the head of its branch, and
• a newer run of the same workflow on that branch is queued, running, or has succeeded.

The newer-run condition is there for path-filtered workflows like pages: a newer commit on main does not mean a newer deploy is coming. The watchdog never cancels the run for the head of its branch, a re-run, or a scheduled or dispatched run. Those need a person:

gh run cancel <id>                       # or, if it will not cancel:
gh api -X POST repos/bpalermo/aether/actions/runs/<id>/force-cancel
gh run rerun <id>                        # the head's own run: run it again

GitHub can strand a run it then refuses to cancel. Run 34723047990, a proxy run queued since 2026-09-12 on a branch that was later deleted, answers both cancel and force-cancel with HTTP 409 ("Cannot cancel a workflow run that is not in progress"). Nothing in this repository can clear a run like that (#1301). When a cancel and the force-cancel after it both return 409, the watchdog reports the run once as uncancellable — needs GitHub support. It records the run ID in a hidden <!-- stuck-runs-uncancellable: … --> marker on the issue and stops counting the run as stuck. Each check reads the marker back from its own issue with that title (the open one; with none open, the newest closed one that is not a folded duplicate), so the issue can close and new stuck runs are still reported. Only a marker the watchdog itself wrote is read (#1532): pasting one into a comment does not hide a run. The ID drops out of the marker once GitHub stops listing the run. To have such a run removed, open a GitHub support ticket.

To see what the watchdog would do without it cancelling or writing anything, dispatch it as a dry run: gh workflow run stuck-runs.yaml -f dry_run=true. A dispatch is a dry run unless you pass dry_run=false.

---

4. Format & lint

make format             # bazel run //:format        — gofumpt, buildifier (+ -lint=fix), shfmt, buf (in place)
make format-check       # bazel run //:format.check  — CI-friendly, fails on drift AND on buildifier lint warnings
make lint               # bazel build --config=lint //...  — buf, shellcheck, gocognit aspects

After changing Go imports or adding/removing Go files, regenerate BUILD files:

make gazelle            # bazel run //:gazelle

To add a Go dependency (never edit go.mod by hand, and never run go mod tidy directly):

bazel run @rules_go//go get <package>
make gazelle
make tidy               # bazel mod tidy

MODULE.bazel.lock in CI (#1284)

CI runs with --lockfile_mode=error (common:ci in .bazelrc), so a change to MODULE.bazel (or a bump that moves a transitive module) without the matching MODULE.bazel.lock fails the job with Bazel's own message naming the stale entry. Fix it locally and commit the lock:

bazel mod deps --lockfile_mode=update

The one non-reproducible extension in the graph was rules_rust's crate, used by protobuf for its Rust bindings (nothing here builds Rust): crate.from_specs() without a lockfile re-resolved against live crates.io on every evaluation, so any bazel mod command or a stale digest rewrote the lock with newer crates. A patch on protobuf (single_version_override in MODULE.bazel) points it at bazel/protobuf_crates/{Cargo.lock,Cargo.Bazel.lock}; with both set the extension is reproducible and is not recorded in MODULE.bazel.lock at all. When a protobuf or rules_rust bump changes the specs or the cargo-bazel digest, any command that evaluates that extension (a bazel mod subcommand; a build of //... does not, it never loads @crates) fails with "The current lockfile is out of date for 'crates'"; repin with CARGO_BAZEL_REPIN=1 bazel mod deps and commit both files.

Go dependency hygiene

make deps-audit         # scripts/go-deps-audit.sh — also a required CI job

go mod tidy is forbidden here, and go mod tidy -e is worse than useless. The generated proto packages aethermesh.dev/api/aether/{agent,cni,config,registrar,registry}/v1 exist only as Bazel outputs — there are no .go files for them in the tree — so the Go tool cannot resolve a single import of them and fails with "no matching versions for query latest". The -e escape hatch does not fix that: it just keeps going and strips every module that only generated code or a BUILD file needs, which then breaks the go_deps.gazelle_overrides and leaves bazel mod tidy failing with "Some gazelle_overrides did not target a Go module with a matching path". Nothing that go mod tidy -e prints about this module is trustworthy.

So make deps-audit does the job instead, from the module graph (go list -m all, which works from go.mod alone) and the Bazel graph (@repo// references in BUILD/.bzl files), and CI runs it on every PR. It reports:

1. stale go.sum modules — modules no longer anywhere in the module graph. Nothing prunes these automatically (bazel mod tidy does not touch go.sum), so they accumulate after every removal. Fix: delete their lines, then bazel build //... //test/e2e:e2e_test — gazelle's go_deps verifies every module it fetches against go.sum, so an over-eager deletion fails loudly and bazel run @rules_go//go -- mod download <module> puts it back.
2. unused direct requires — no Go import, no BUILD/.bzl reference, no tool directive.
3. direct requires with no Go import that are only reachable from Bazel and are not annotated.

Two rules follow from that, and both matter:

• Reclassify, don't drop. Most unimported requires (golang.org/x/crypto, github.com/go-openapi/swag, github.com/moby/go-archive, …) exist to force a CVE-clean version through MVS. Move them into the // indirect block — the pin, and therefore the selected version, survives; only the directness changes. Deleting the line silently lets the vulnerable version back in. Note that go get marks whatever it fetched as direct, so a floor bump needs its // indirect marker put back by hand.
• Annotate the Bazel-only tools. A module that is genuinely only named by a BUILD/.bzl file (buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go, from the gazelle:resolve directive in //BUILD.bazel; github.com/uudashr/gocognit, the lint aspect's binary) stays direct and carries a // bazel-only: comment, so the next reader knows why it looks unused. //bazel/protodoc is a real Go package, so protoc-gen-doc and protokit are ordinary imports, not Bazel-only.

To actually remove a module, use go mod edit -droprequire, not go get <mod>@none: @none downgrades everything that requires the module, which is how removing the unused go.etcd.io/etcd/server/v3 once dragged controller-runtime from 0.25.0 back to 0.9.7 (k8s.io/apiextensions-apiserver requires it). Delete any matching go_deps.gazelle_override in the same change, then make tidy and make gazelle.

CodeQL code scanning

Code scanning is the advanced-setup workflow .github/workflows/codeql.yaml (push to main, every pull request, weekly), not GitHub's default setup. The two are mutually exclusive: with default setup enabled in the repository's security settings GitHub rejects this workflow's uploads, so default setup must stay off. Its last job, codeql, aggregates every language's job and is the one name to require in the main ruleset (next to ci and proxy); it also runs on pull requests whose base is upgrade/**, so a gh stack member can report it. A status check only says the scan ran: blocking on findings is the ruleset's separate "Require code scanning results" rule. A fork's pull request is analysed but not uploaded (read-only token).

It exists because default setup cannot run a step before the scan. The proto packages under api/aether/ are Bazel outputs with no .go file in the tree, so default setup's Go autobuilder reported "6 packages could not be found" and analysed the 85 files that import them, the gRPC and API surface, without their types. It also ran go mod tidy -e (forbidden, see above) and make.

What the Analyze (go) job does, in order:

1. scripts/materialize-generated-go.sh asks Bazel for every go_proto_library, builds their go_generated_srcs output group and copies the .pb.go / _grpc.pb.go files to api/aether/<pkg>/v1/, where their import paths point. The copies are git-ignored (/api/aether/**/*.pb.go) and listed in .materialized-generated-go. Bazel runs before CodeQL is initialised so the tracer never sees rules_go's own go processes.
2. scripts/go-build-plain.sh builds the module with the plain go command under CodeQL's tracer: one go build ./... over the whole module, pinned Go from go.mod, GOFLAGS=-mod=readonly, GOTOOLCHAIN=local, GOOS=linux, CGO_ENABLED=0. go.mod and go.sum must come out untouched. The tracer extracts each package and then lets the real build run, so this step is also the proof that the module compiles outside Bazel.
3. CodeQL analyses with upload: never.
4. scripts/codeql-go-diagnostics.sh reads the SARIF and fails the job if the extraction was incomplete (below).
5. Only then are the results uploaded, under the same categories default setup used (/language:go, /language:c-cpp, …), so existing alerts and dismissals keep matching. Every job also keeps its SARIF as a codeql-sarif-<language> workflow artifact.

The other four languages (actions, c-cpp, javascript-typescript, python) are analysed from source with no build. Query suite default, threat model remote (.github/codeql/codeql-config.yml). Go test files are not scanned.

Locally:

make materialize-go        # copy the generated Go into the tree (git-ignored)
make go-build-plain        # materialize, then go build + go vet with the plain
                           # go command and the Bazel-pinned SDK
make materialize-go-clean  # remove the copies

The copies are inert for Bazel and Gazelle (no target lists them, and Gazelle skips a .pb.go that sits next to its .proto), but they go stale when a .proto changes: re-run make materialize-go, or clean them.

Nothing is excluded from the plain build. scripts/go-build-plain.sh runs go build ./... (and, with --vet, go vet ./..., which also type-checks every _test.go) over the whole module: no package list, no exception. //scripts:go_build_plain_test holds it to exactly those two invocations.

Bazel does not enforce Go's internal-package rule (visibility is its own, looser mechanism), so a package outside agent/ can import agent/internal/... and stay green in Bazel while the go command, gopls and CodeQL's extractor refuse it ("use of internal package ... not allowed"). That was the case for test/envoy_validate, its generator and test/mtlspool until #1311 moved them to agent/test/, and it is the only way such a failure is fixed: move the importer under the tree whose internal/ it uses (or give it a non-internal package to import). Do not exclude it from the build and do not allow its extraction error.

What ./... leaves out is decided by build constraints in the files themselves, never by a list. Three non-test files are never part of a Linux build and so are not in the database: agent/internal/udscsi/mounter_other.go, common/file/dirsync_windows.go, common/file/fadvise_unspecified.go. And test/conformance holds one _test.go behind //go:build conformance: it is compiled inside the upstream gateway-api module, not this one (its README), so the directory is not a package here for the go command or for Bazel.

"The Go extraction is complete" failed. CodeQL does not fail a scan it could only half do, and a partial result uploads cleanly and then closes every alert in the code it did not understand as fixed. The step reads the SARIF's runs[].invocations[].toolExecutionNotifications[] and fails on any go/diagnostics/extraction-errors entry (an unresolved package or type, an internal-package violation): there is no allow-list, and //scripts:codeql_go_diagnostics_test fails if one comes back. It also fails on any go/autobuilder/* warning (the "packages could not be found" family), on an extracted _test.go, and when a materialized file is missing from the database. Nothing was uploaded; the SARIF artifact has the full list. The usual cause is a generated package the materialize step did not cover.

Adding a generated Go package. A new go_proto_library anywhere in the root workspace is discovered by bazel query; nothing lists the targets by hand. If its import path is outside api/aether/, the script refuses to write it until .gitignore covers the destination: add a pattern scoped to *.pb.go in that directory, never a pattern that could match a hand-written file. Generated Go that is not a proto (a genrule output in a go_library's srcs, a generated embedsrcs file) makes the script fail by name: teach it where the file belongs first. scripts/materialize-generated-go.sh --check-untracked (the shell CI job) fails if a generated file is ever committed.

---

5. Container images

In-repo images (agent, mesh-dns, cni-install, registrar) build with rules_img:

make load-all           # load agent + mesh-dns + cni-install + registrar into local Docker
make load-agent-image   # a single image (…-mesh-dns-image, …-registrar-image, …-cni-install-image likewise)
make push-all           # push all to the registry

mesh-dns is its own image (#583): the aether-mesh-dns DaemonSet must NOT ship the full agent, so it gets the slim /mesh-dns binary alone (~7Mi vs the agent's ~20Mi) and the chart pins it separately via agent.meshDnsDaemon.image.

The controller image is not in load-all; load it with bazel run //controller/cmd/controller:image_load.

Which build is this binary (--version, #1378, #1429)

The main binary of each component image answers --version and exits without starting anything: agent, registrar, controller, prober, proxy-supervisor, mesh-dns, uds-csi and cni-install. The other binaries the images ship do not, on purpose:

• the probe binaries (proxy-ready, agent-ready, mesh-dns-ready, identity-ready) are exec'd every few seconds or gate a pod's start, and each has a test that holds it to the smallest possible link set;
• the CNI plugin (cni) is not run by hand: the container runtime execs it with its command in the environment, and the CNI specification defines its own VERSION command for that;
• the e2e fixtures (l4echo, udsecho) are test workloads.

readelf -n reads the build ID of any of them. make check-build-id lists every binary in an image the release pushes, with or without the flag: the component binaries, the probe binaries, the CNI plugin, and udsecho, whose image is pushed with its chart. l4echo is not pushed and is not listed.

$ kubectl -n aether-system exec ds/aether-agent -c agent -- /agent --version
agent build-id bbf3ba6860da40c812de4d2317532ec7d5f17ec0
package aethermesh.dev/agent/cmd/agent
go1.27.1 linux/amd64

The build ID is the binary's GNU build-ID note, which //bazel/buildid derives from the binary's own bytes. It is the same value:

• readelf -n <binary> prints as Build ID;
• the component reports as OTel service.version (and uds-csi as its CSI vendor_version, visible in its starting log line);
• the continuous profiler files that binary's symbols under;
• make check-build-id prints for a checkout.

It is not a commit. No Go binary of this workspace has the commit linked in any more (the separately built Envoy in the proxy image still does): a version taken from the commit made every binary, and so every image, different from one commit to the next even when its code had not changed. Two consequences:

• Two pods reporting the same service.version run byte-identical binaries, whatever commits their images were published from; a service.version that changes across a deploy means that component's code really changed.
• To go from a build ID to source, find the commit whose build produced it: make check-build-id at a candidate commit, or the image's provenance (see verifying-releases.md). A build ID belongs to one architecture's binary: the amd64 and arm64 builds of a component have different IDs, and a pod reports its node's. make check-build-id prints both from any machine, one line per binary and platform, linux/amd64 first (it builds //bazel/buildid:release_build_ids, which reads the binaries under each image index, built for both platforms). The chart's appVersion still names the commit the chart was packaged from.

There is no module line because a Bazel build records the main package and every dependency's version, but no version for the main module itself (go version -m <binary> shows the same). A binary run outside an image (bazel run //agent/cmd/agent) reports the build ID the Go linker wrote, which under rules_go is one constant shared by every binary; only the binaries inside an image have a content-derived one. A binary with no readable note reports dev.

---

6. Local multi-cluster end-to-end (proposal 026)

e2e/multicluster_config.sh stands up two kind clusters (a = exporter, b = importer) that share one etcd (a Docker container on kind's network) and drives the cross-cluster GAMMA config loop. It runs every night (the multicluster-config job of e2e.yaml):

cluster a: HTTPRoute + ServiceExport ──(registrar config-export)──▶ shared etcd
                                                                       │
cluster b: agent --import-config  ◀──(registrar ListAllConfig reads same etcd)─┘

Commands

e2e/multicluster_config.sh up      # build+load images, create clusters + etcd, install aether on both, deploy
e2e/multicluster_config.sh verify  # the assertions below (re-runnable; `test` is the same)
e2e/multicluster_config.sh down    # delete both clusters + the shared etcd
e2e/multicluster_config.sh         # up + verify (full run)

up builds the images with Bazel, installs the Gateway API (experimental channel) + MCS-API CRDs, then installs both aether charts per cluster with spire.enabled=false, registrar.registryBackend=etcd, registrar.enableMCS=true, a clusterName of its own (cluster-a, cluster-b: it is each cluster's partition of the shared store) and agent.importConfig=true on cluster b. Cluster a gets a Service echo and an HTTPRoute (/v2 to echo-v2, everything else to echo-v1) and runs no workload; cluster b gets echo, echo-v1, echo-v2 and a client, and no HTTPRoute.

verify asserts, with one probe from the client in b:

1. nothing exported: the store holds no projection for demo/echo, and echo itself answers /v2 and /;
2. a ServiceExport for echo in a: the projection appears under cluster-a's partition of the store and names both backends;
3. with nothing applied to b: /v2 is answered by echo-v2 and / by echo-v1, 20 of 20;
4. the ServiceExport deleted: the projection is pruned and echo answers again.

To see step 3 fail, bring b up without the import: CONFIG_IMPORT=off e2e/multicluster_config.sh up.

The fs.inotify gotcha

Two kind clusters each run an inotify-heavy agent DaemonSet; the host's default limits are easily exhausted, leaving cluster b's agent stuck in Init:Error. up raises the limits when it can (passwordless sudo), and says so when it cannot:

sudo sysctl -w fs.inotify.max_user_instances=8192 fs.inotify.max_user_watches=524288

Without it up fails, naming the agent DaemonSet that never became Ready: the suite asserts the import on b's data path, so b's agent is not optional. e2e/kind-cluster.yaml assigns non-overlapping pod/service CIDRs per cluster, so cross-cluster endpoints in the shared registry never collide.

The other multi-cluster harnesses

Two sibling harnesses reuse the same two-kind pattern (same up/test/ verify/down verbs, same inotify gotcha); both run SPIRE on each cluster under a shared trust domain (shared upstream CA in e2e/certs/):

• e2e/multicluster_waypoint.sh — the 019 east/west waypoint data path: two clusters + ONE shared etcd, agent.eastWestWaypoint=true; asserts client(a) → echo(b) returns 200 over the node tunnel with mTLS end-to-end (nightly CI: e2e.yaml waypoint job).
• e2e/multicluster_replicator.sh — the 006 two-region replicator failover: one etcd PER cluster, registrar.peerEtcd cross-wired; asserts mirror visibility, the data path over the mirror, lease-lapse failover when a region's registrar dies, and recovery (nightly CI: e2e.yaml replicator job).

Bumping the e2e Kubernetes version

Every kind e2e surface runs the same Kubernetes: the nightly conformance jobs, the per-PR //test/e2e, the script-driven nightly suites and a local run of any e2e/*.sh harness (#1251). e2e/kind-version.sh is the one place it is set: KIND_VERSION (the kind release), KIND_NODE_IMAGE (a kindest/node image from that release's notes, with its @sha256 digest — node images are built per kind release) and KUBECTL_VERSION (the same Kubernetes patch). To bump, change those three, copy the node image into test/e2e/testdata/kind-config.yaml (the e2e framework reads a kind config, not a shell file), and run bazel test //e2e:kind_pin_test, which fails on any copy that disagrees — a kind config, a workflow's helm/kind-action value, or a harness that creates a cluster without --image "$KIND_NODE_IMAGE". Locally the harnesses refuse a kind binary older than KIND_VERSION (go install sigs.k8s.io/kind@<KIND_VERSION>, or KIND_ALLOW_SKEW=1 to try anyway) and only warn about a newer one; KIND_NODE_IMAGE=<image> overrides the node image for one run.

CI pulls the node image through a Docker Hub mirror (CI: Docker Hub pulls,

1602). After a bump, read the **Pull the node image through the Docker Hub

mirror** step of the pull request's e2e job: it says whether the mirror serves the new digest, and warns when the pull fell back to Docker Hub.

Bumping Go

The Go toolchain is pinned once: go_sdk.download(version = ...) in MODULE.bazel, the SDK rules_go builds and tests everything with. The few CI steps that run a bare go outside Bazel (the nightly conformance suites' go test, the cloud-provider-kind install) get the same version from actions/setup-go with go-version-file: go.mod, so go.mod's go line (or a toolchain line, which setup-go prefers) must name the same full X.Y.Z (#1283). Bump both together and run bazel test //e2e:go_pin_test. It fails on a go.mod that disagrees with the SDK pin, a setup-go step with a literal go-version or check-latest, and any workflow job or composite action that runs go without setting it up first, which would leave it on the runner image's Go.

Bumping Helm

No workflow job runs the runner image's Helm (it moves, majors included, when GitHub updates the image) or azure/setup-helm's default, which is the newest release (#1580). e2e/helm-version.sh pins one Helm 3 release, one Helm 4 release and HELM_DEFAULT_MAJOR, the major a job gets when it does not ask; .github/actions/setup-helm installs from that file and is the only installer. The default is Helm 3. Helm 4 renders every chart in the Bazel template tests (the rules_helm toolchain), and the nightly first-install job runs e2e/first-install.sh under both majors, one matrix leg each (#1543).

To bump a release, change its line. HELM_V4_VERSION is held equal to the rules_helm toolchain's Helm, so a rules_helm bump that moves its Helm moves that line in the same pull request. To move the default major, change HELM_DEFAULT_MAJOR and prove it with a workflow_dispatch of e2e.yaml on the branch: no pull request check runs the kind harnesses. bazel test //e2e:helm_pin_test fails on a job or composite action that runs helm or an e2e/*.sh harness without setup-helm before it, on azure/setup-helm used anywhere else, on a caller that names a version instead of a major, and on a helm list -a: Helm 4 dropped that flag, so anything that lists releases takes its flags from helm_list_all_flags in the same file (#1581).

Bumping Gateway API

e2e/gateway-api-version.sh is the one Gateway API release every e2e surface installs: each harness sources it for the CRD bundle it applies, and the nightly conformance jobs install the same release's bundle and run its suite (#1583). Bump it together with go.mod's sigs.k8s.io/gateway-api (the release the code is built against) and GATEWAY_API_VERSION in .github/workflows/e2e.yaml (a workflow cannot source a shell file, so it carries the one copy). bazel test //e2e:gateway_api_pin_test fails until the three agree, and on a harness that assigns GWAPI_VERSION itself or a download URL that names a release. GWAPI_VERSION=<release> overrides it for one local run.

Refreshing third-party image pins

An image this repository does not build (curl, the echo servers, OPA, etcd, the OpenTelemetry collector, the kind node) is named in chart values, in the e2e harnesses and their manifests, and in test fixtures. Every such reference carries the digest of the image's multi-arch index, and scripts/third-party-images.txt is the one list of them, one pin <name> <tag> <digest> line per image (#1400, #1401). A reference is written <name>:<tag>@sha256:…: the digest is what a node or docker run pulls, the tag is there for the reader. A few older references are <name>@sha256:…, with the tag recorded only in the list.

The base images of what this repository builds are on the same list (#1477): gcr.io/distroless/static-debian13 under every Go image, pulled by MODULE.bazel, and gcr.io/distroless/cc-debian12 under the proxy image, pulled by proxy/MODULE.bazel. Starlark does not write them as one reference, so check reads exactly two constructs in those two files, the rules_img pull(registry =, repository =, tag =, digest =) and the rules_oci oci.pull(image =, digest =), and puts each together as a reference. It is a line reader for the shape buildifier writes and it fails closed: a pull written on one line, an attribute that is not a plain string, or a pull rule bound under a spelling it does not know is a finding. A module file reached by include("//<package>:<file>.MODULE.bazel") is read like the file that includes it (#1571); an include() in any other shape, and any other rule of a ruleset that pulls images (rules_img, rules_oci, rules_docker, rules_apko), are findings until the reader is taught them. A pull by a hand-written repository rule is still not seen.

A YAML file is read with a YAML parser, not line by line (#1525): an alias is followed to its anchor, and a flow mapping broken across lines, a value on the line after its key and a reference under a child key (image: / ref: x) are all read. A file named *.yaml that no parser can read is a finding, unless it is a Helm template (it holds {{), which is read line by line like a shell script. This is the one part of check that needs more than bash, awk and git: python3 with PyYAML (python3 -c 'import yaml'). Without it check stops with exit 2 and has checked nothing. The test runs the reader that Bazel builds with a pinned PyYAML (//scripts:third_party_images_yaml).

scripts/third-party-images.sh has five commands:

Command	Network	What it does
check	no	The gate (CI's shell job). Fails on an image named by tag only or with no tag, on a digest the list does not have for that name, on a tag that disagrees with the list, on a pin or an exception nothing uses any more, on a YAML file no parser can read, and on a base-image pull, an include() or an image rule in a Bazel module file that it cannot read
list	no	Every pin, with the files and lines that use it
outdated [--newer-tags] [<name>...]	yes	Asks each pin's registry what its tag points at now: current, MOVED (the tag was re-pushed; the new digest is printed), NOT-MULTI-ARCH (the pinned index itself lacks linux/amd64 or linux/arm64; counted as behind) or ERROR (no answer; never reported as current). --newer-tags adds the registry's tags that have the pinned tag's shape and sort after it. Exit 0 all current, 1 a pin is behind, 2 a pin could not be checked
newer [<name>...]	yes	Reads each pin's tag list and prints NEWER <name>:<tag> <n> newer: … for a pin that has tags of the pinned tag's shape sorting after it (the newest eight), and ERROR for a list it could not read. Exit 0 every list was read, 2 one was not
resolve <name>:<tag>...	yes	Prints the pin line for a tag, and says so when its index lacks linux/amd64 or linux/arm64

outdated, newer and resolve read public registries anonymously: no credential is read from the machine or sent (the anonymous pull token a registry hands out is the only Authorization header there is). A private or rate-limited registry answers ERROR, not current.

Nothing moves a pin by itself. There is no bot: Dependabot opens no pull requests here, because they run without the repository's secrets and could never pass CI. What there is, is a daily look (#1478): .github/workflows/third-party-images.yaml runs outdated at 05:43 UTC through scripts/third-party-images-report.sh and keeps one rolling issue, CI: third-party image pins are behind their tags.

outdated said	What the run does
a pin is MOVED or NOT-MULTI-ARCH	Opens the issue, or rewrites the open one so it says where the tags point now. It adds a comment only when the set of pins behind changed
every pin current	Comments on the open issue and closes it
a pin is ERROR (the registry did not answer)	A warning on the run. That run does not know the whole set, so it neither closes an open issue nor rewrites what it says is behind; with none open, the pins it did find behind open one
the same pin is ERROR in three runs in a row	The pin is reported (#1570): the issue is opened for it, or the open one gains a section and a comment. It stays open until a run checks every pin

The run is green in all four cases: a pin that is behind still pulls the digest it names, and the workflow is not a required check. It fails only when the check itself is broken (outdated died, or gh did).

A lookup that keeps failing. One ERROR is a registry having a bad minute. A pin whose lookup fails on every run (the repository went private, the tag was deleted, a rate limit that never lifts) would otherwise stay a warning on a green run for good. So the rolling issue also holds, in a hidden line of its body (<!-- third-party-images-state: errors@… <name>:<tag>=<runs> -->), in how many runs in a row each pin's lookup has failed; a pin that answers again is forgotten at once. The count lives in the open issue when there is one, and otherwise in the newest closed one, which is edited without being reopened, so a streak shorter than three notifies nobody. When there has never been a rolling issue, the first failed lookup opens one and closes it in the same run to hold the count: expect that once. Do not delete that hidden line by hand; a line that is not a pin and a count is ignored, never run.

Newer versions. The report above is about the tag each pin names; it says nothing when 8.23.0 exists next to a pinned 8.22.0. The same run asks scripts/third-party-images.sh newer, and its answer is a section of the issue (#1569) and of the run's summary. That section is for the reader only: it never opens the issue, never keeps it open, never closes it and never causes a comment, and when a tag list cannot be read it says so and the run goes on. With no issue open, read it in the run's summary, or run newer in the same pass that bumps the Bazel, Go and Actions pins.

The issue is opened with the enhancement and ci labels; if one of them is deleted the run fails instead of filing an unlabelled issue. Only an issue the workflow itself opened counts as the rolling one: an issue someone else files under the same title is never rewritten or closed, and the report opens its own next to it.

To see what it would write without writing it, dispatch a dry run: gh workflow run third-party-images.yaml. A dispatch is a dry run unless you pass -f dry_run=false on main. A pull request that touches the scripts or the list runs the same dry run, in a job without issues: write.

To move a pin (or add an image):

scripts/third-party-images.sh resolve curlimages/curl:8.23.0   # prints: pin curlimages/curl 8.23.0 sha256:…
scripts/third-party-images.sh list                             # where the old pin is used
# Replace the pin line in scripts/third-party-images.txt, then the old
# <tag>@<digest> in every file `list` printed.
scripts/third-party-images.sh check
bazel test //scripts:third_party_images_test //charts/...

What a moved pin obliges:

• A pin used under charts/<chart>/ is a change to that chart: bump its Chart.yaml and say what rolls. charts/aether's proxy.authzSidecar.opa.image is a container of the proxy pod, so moving it rolls the proxy DaemonSet on every node where the OPA preset is on; charts/udsecho's client.image and charts/prober's authzCanary.image each roll one single-replica Deployment.
• The two distroless base pins move in the module file and the list together: digest in the pull of MODULE.bazel, or in the oci.pull of proxy/MODULE.bazel, and the pin line. distroless rebuilds nonroot in place, so the issue lists them whenever the base was patched. A new base digest changes the digest of every image built on it: the MODULE.bazel one is a release of every Go image, and the proxy/MODULE.bazel one is a proxy build (proxy.yml) and a proxy release.
• The two etcd pins (quay.io/coreos/etcd for the kind harnesses, gcr.io/etcd-development/etcd for the testcontainers tests) move together, and stay on the minor line of the Go client in go.mod (#1223).
• kindest/node moves only with KIND_VERSION (Bumping the e2e Kubernetes version above); //e2e:kind_pin_test holds its copies together.

An exception is an allow <path> <reference> or skip <path prefix> line in the list, with its reason beside it. An allow or skip that matches nothing any more fails check. There is none of either kind today.

check does not read comments, whether a comment line or the comment that ends a line of code: a pin that only a comment still names is reported as unused. In a YAML file it fails closed on flow mappings: an image: that follows {, [ or , outside quotes is read as a key, so a line of prose written that way must be quoted.

What check cannot see: an image no pin names yet, written where no image key, --image flag or *_IMAGE variable introduces it (a positional docker run <image>). Give such a reference a *_IMAGE variable, as e2e/etcd-image.sh does. Nor does it see an unquoted image: value in the middle of a line of a multi-line flow mapping when that YAML is embedded in a shell here-document or a Go string (the header of the script has the list). It also reads only charts/, e2e/, test/ and registry/etcdtest/; the images the Bazel image rules pull (MODULE.bazel) are pinned there by digest and are not in the list.

---

7. Installing on a real cluster

Aether ships two charts, and install order matters:

charts/crds     — the CRDs: MeshConfig + HTTPFilter + EdgeConfig + EndpointPolicy   ← install FIRST
charts/aether    — the whole system (agent DaemonSet + proxy + mesh-dns + registrar + controller)

Install the CRDs before the system chart. The agent crashes if it starts before the HTTPFilter CRD exists (it watches that type at startup); installing the crds chart first avoids the race (see #453).

# The commit you intend to deploy (the `publish` run that built it), plus each
# chart's X.Y.Z from its charts/<name>/Chart.yaml at that commit.
COMMIT=<full 40-char git sha>
CRDS_VERSION=<X.Y.Z>-$COMMIT
AETHER_VERSION=<X.Y.Z>-$COMMIT

# 1) CRDs first
helm upgrade --install aether-crds oci://quay.io/aethermesh/chart-crds \
  --version "$CRDS_VERSION"

# 2) then the system. Use this commit-pinned chart tag, not the bare
#    `--version <X.Y.Z>`: the bare tag is mutable and re-pushed by every publish,
#    the commit tag never is (#692). Anything that PINS a chart (a lock file, a
#    GitOps source, a `tag@sha256:` reference) names the commit tag or the
#    digest alone, never the bare tag with a digest: see "Pinning a released
#    chart" below (#1588).
#    The chart does not create the namespace (namespace.create=false, the
#    default since 2.4.21, #1403): --create-namespace does. On a FIRST install
#    into a cluster that enforces Pod Security admission, label the namespace
#    before this command, or the agent, proxy, mesh-dns and uds-csi pods are
#    refused:
#      kubectl create namespace aether-system
#      kubectl label namespace aether-system \
#        pod-security.kubernetes.io/enforce=privileged \
#        pod-security.kubernetes.io/audit=privileged \
#        pod-security.kubernetes.io/warn=privileged
#    (see getting-started.md, "Install", and "Recovering from a failed first
#    install" below).
helm upgrade --install aether oci://quay.io/aethermesh/chart-aether \
  --version "$AETHER_VERSION" -n aether-system --create-namespace \
  --set clusterName=my-cluster --set meshDomain=aether.internal

# 3) ALWAYS assert what actually landed. The chart's appVersion is the commit
#    that built it, so this is the only check that catches a chart tag serving
#    a different build than you asked for — which happened on 2026-09-05, when
#    two racing publishes wrote the same bare tag and the older build won,
#    silently deploying a release without #689 in it (#692).
got="$(helm get metadata -n aether-system aether -o json | jq -r .appVersion)"
case "$got" in
  *"$COMMIT"*) echo "deployed $got — OK" ;;
  *) echo "DEPLOY MISMATCH: appVersion=$got, expected $COMMIT" >&2; exit 1 ;;
esac

appVersion is the chart's {STABLE_GIT_VERSION} — git describe --tags --always --long --dirty --abbrev=40 — so the full sha is embedded in it but may carry a v1.2.3-N-g prefix or a -dirty suffix; hence the substring match rather than string equality. If it fails, the chart tag you pulled was built from a different commit: re-run that commit's publish workflow and upgrade again.

Pinning a released chart (#1588). The aether chart is published under two tags, and only one of them can be pinned:

Tag	Example	Written	Its digest
commit-suffixed	chart-aether:2.4.26-<full sha>	once, by that commit's publish	never changes
bare version	chart-aether:2.4.26	again by every commit on main that still carries that Chart.yaml version	changes with each of those commits, while the version does not

Most commits do not touch the chart, so most commits republish the bare tag: it is "the newest build of this chart version", not a release. A reference that pins a digest under the bare tag (chart-aether:2.4.26@sha256:…, or a tool's lock entry of tag plus digest) is valid until the next commit publishes, and then fails; Helm reports chart reference digest mismatch (seen within an hour of writing such a reference). The crds, prober and udsecho charts have no bare tag: their Chart.yaml version carries the commit.

So a pin is one of these two, and nothing else:

# the commit-suffixed tag (what this runbook uses everywhere)
helm pull oci://quay.io/aethermesh/chart-aether --version "<X.Y.Z>-$COMMIT"
# or the digest alone, with no tag for it to disagree with
helm pull oci://quay.io/aethermesh/chart-aether@sha256:<digest of that commit's chart>

helm pull of the commit tag prints that chart's digest (its Digest: line). A digest read from the bare tag is the digest of whichever commit published last.

After a proxy release, normally there is nothing to do. proxy-release publishes the image, pins charts/aether/values.yaml (tag: and digest:), pushes chore/proxy-image-bump-<short sha>, opens the pin PR with the RELEASE_PAT fine-grained token and arms --auto --squash (#727). GitHub merges it the moment ci + proxy are green — the main ruleset is still the only gate, and it has no bypass actors. Any superseded chore/proxy-image-bump-* PR is closed and its branch deleted by the same run; main-post-merge updates the pin PR's branch whenever another merge leaves it BEHIND (auto-merge does not do that by itself). Watch the rolling proxy releases pending chart pin issue (#724): every release comments there with the PR, the digest, the chart version, the run link and the token's remaining life. The deploy stays manual — after the pin merges, take that commit's publish run and use the commit-suffixed chart tag (<X.Y.Z>-<full sha>, above) with the appVersion assert.

Manual fallback. When #724 says the token is missing, expired or rejected, the run degrades to #703's hand-off instead of failing: the branch is pushed with GITHUB_TOKEN and both the run summary and the #724 comment carry the exact gh pr create one-liner (its | release token | row says which of the three it was). Run it from a checkout, let ci + proxy go green, squash-merge, close any superseded chore/proxy-image-bump-* PR. A PR opened by GITHUB_TOKEN itself is never an option: GitHub's recursion guard means it fires no pull_request events, so the required checks would never report.

Rotating RELEASE_PAT. The workflow warns 30 days ahead — a ::warning:: on the release run and the | release token | row on #724 — and, once expired, falls back to the manual hand-off. To rotate: Settings → Developer settings → Fine-grained tokens → new token, resource owner bpalermo, only the bpalermo/aether repository, repository permissions Contents: Read and write + Pull requests: Read and write and nothing else (no Workflows, no Administration, no Actions; Metadata: Read is implied), maximum expiration. Then Settings → Environments → release → replace the environment secret RELEASE_PAT with it. It must stay an environment secret of release, whose deployment-branch policy names main only: that is what keeps a pull_request run, or a run of a workflow edited on a branch, from ever reading it. The next proxy-release run reports the new expiry on #724.

From a checkout, the Bazel install targets do the same in order:

bazel run //charts/crds:crds.install
bazel run //charts/aether:aether.install

Always pass the full values on every helm upgrade of the aether chart — do not use --reuse-values (it keeps the stale digest-pinned image). Bump the chart's version: on any change to its templates/values (CI enforces this).

Pre-flight: did that commit actually publish? (#880)

Before pinning a deploy to a commit, check that the commit has artifacts:

make check-published COMMIT=<any commit-ish>   # one commit
make check-published                           # the last day of main

Read-only, no credentials needed (the packages are public), and it cannot push anything. It prints every coordinate it checked — four commit-tagged charts, nine images (eight for a commit before uds-csi), a cosign signature for each image's index and for every child manifest it lists (40 coordinates today), plus the aether-proxy digest the commit's chart pins (see "Verifying the aether-proxy signature") — and exits non-zero naming each one that is missing. It checks that a signature exists; to check that it verifies, see "Verifying image signatures" below.

Do not use gh run list for this. A commit whose publish was superseded has a run whose conclusion is cancelled, not failure: green-ish in the Actions UI, nothing pushed. .github/workflows/publish.yaml serialises on one concurrency group (#692), and GitHub keeps at most one pending run per group, so a merge landing while a publish runs is cancelled by the next merge before it starts a job. f332061 (#875) was lost that way on 2026-09-20. Two more traps: gh run list --commit=<sha> matches only the full 40 characters and answers an abbreviation with an empty list — which reads exactly like "no publish ever ran" — and a run that did exit 0 still tells you nothing about what reached the registry. Only the registry answers that question.

Only push heads are published (#975). publish runs once per push to main, for the push's head commit. An atomic stack merge lands several squash commits in one push, and only the last is ever built: the others have no artifacts by construction and cannot be deployed — pin the stack's head instead. make check-published COMMIT=<intermediate> correctly reports them MISSING; the --recent sweep skips them, printing skip <sha> (not a push head …). It learns the heads from GitHub's activity log for refs/heads/main, so it needs gh authenticated (or PUSH_HEADS_FILE=<file of full shas>); a push whose publish run never started is still a head and still fails.

Superseded push heads are skipped (#1282). A push head whose publish run did not succeed (cancelled by the next merge, or failure) on a main that has since moved past it was superseded: the newer push's publish built a tree that contains it. Both the workflow_run path (#1277) and the --recent sweep skip it with a ::notice:: naming the commit — skip <sha> (superseded push head …) in the log — instead of reporting its commit-addressed tags missing. main's own head, a head whose publish succeeded and a head with no publish run on record are always checked. The rule is scripts/publish-verify-superseded.sh decide; the sweep reads the publish runs with gh (PUBLISH_RUNS_FILE=<file of "<sha> <conclusion>"> overrides it) and main's head with git ls-remote (MAIN_HEAD overrides it). Do not deploy a superseded commit: pin the head that superseded it.

publish-verify runs the same check automatically after every publish run reaches a conclusion and every two hours over the push heads of main that the last green scheduled sweep did not already verify — at most the last day, the whole day when no sweep has been green within it (#1281; RECENT_SINCE_LAST_GREEN=1, sweep_since in scripts/push-heads-lib.sh) — and files (or comments on) the rolling publish: artifacts missing for a commit on main issue. A sweep that times out or is cancelled files on the same issue, saying the check did not finish: before #1281 three timed-out sweeps in a row concluded cancelled and reported nothing. If you see that issue: re-run the cancelled publish run — gh run rerun <id>, which re-runs at that same commit — or, if the commit is not the one you need, deploy a later commit that did publish. Never push images or charts by hand: the release workflow is the only publisher.

The issue closes itself (#1316). A green run for main's head, or a green scheduled sweep, closes it with one comment naming the run (scripts/publish-verify-close-issue.sh). A green run for a commit main has moved past, a superseded skip and a manual run leave it open. So an open issue means no run has verified the head since the failure; a closed one names the run that did.

A registry hiccup is retried, not reported (#1316). Run 37522284996 died in the signature pass on one 502 from quay.io's token endpoint, with a Python traceback, and filed the issue as UNVERIFIED for artifacts that were fine. The pull-token and child-manifest lookups now retry a missing answer, a 408, 429 or 5xx and a 200 that is not JSON (four attempts, 2 s doubling; REGISTRY_FETCH_ATTEMPTS, REGISTRY_FETCH_INTERVAL), and a failed cosign verify is re-run (three attempts; VERIFY_ATTEMPTS, at most VERIFY_RETRY_BUDGET re-runs per invocation) unless cosign fetched a signature whose identity is another's. A lookup that still fails prints one registry-lib: GET <url> answered HTTP <status>; giving up after 4 attempt(s) line (or how the body starts) in place of the traceback; a 401, 403 or 404 is the registry's answer and is not retried. The tag listing (registry_all_tags) retries each page the same way (#1337) and prints nothing if a page cannot be read. publish.yaml's sign step lists through registry_commit_tag, whose 13 listings 5 s apart are already a retry, so there each page is asked once (REGISTRY_COMMIT_TAG_FETCH_ATTEMPTS): 60 s of waiting at worst per repository, not the 242 s of the two retries nested.

It asks for each tag by name (#985). Every coordinate is checked with HEAD /v2/<repo>/manifests/<tag>. No tag list is read. The old scan paged through every tag, and a sweep that overlapped a publish reported a present signature MISSING (d526bf2, 2026-09-27); a re-run minutes later passed. 200 means present, and 404 means MISSING, but only beside a witness: a tag that the same repository lists answered 200 to the same lookup. Any other answer, or a repository with no witness, is exit 2 (inconclusive), never MISSING. A MISSING line from this check is a real absence, so don't re-run it hoping it goes away.

Every run proves the gate can fail first (#930). (The control covers the per-commit coordinates only: it runs the verifier with PROXY_PIN_CHECK=0, because the constructed commit pins main's signed aether-proxy digest, which is legitimately present; the proxy pin's own red is case 7 of scripts/check-publish-verify-control.sh.) Before the gate step, publish-verify runs an expected-red control: scripts/publish-verify-control.sh builds a commit with git commit-tree on origin/main's tree (fixed identity and dates, so the same base always gives the same sha). No ref points at that commit and it is never pushed, so no publish can ever have produced it. The control then asserts that the verifier goes red for the right reason: exit exactly 1, 20 MISSING lines that all name that sha, no ok line, and every chart and image absence backed by a witness (witness <tag>: 200: a tag the same repository lists, answering 200 to the same lookup). A constructed commit, rather than a pinned one like ef44437 or 1e31e3a, because a pinned red input ages out of the window it is looked up through. That is how the gate lost its only demonstrated red state (#929). The two steps read Expected-red control and Gate in the log. A control's MISSING lines are expected and never reach verify.log or the job summary. If the control fails, the gate still runs, and the run files (or comments on) a rolling issue whose title says which failure it was (#1340, scripts/publish-verify-control-issue.sh): publish-verify: the expected-red control went GREEN (the verifier passed a never-published commit: the gate is vacuous, a real defect), … went red for the wrong reason (part of the gate cannot fail), or … was inconclusive (exit 2: the registry could not be read; no verdict on the gate either way). The body quotes what the control said. An inconclusive verifier is run again first, three runs in all, 10 s then 20 s apart (CONTROL_ATTEMPTS, CONTROL_RETRY_INTERVAL), so one bad answer from the registry files nothing; a verifier that exited 0 or 1 is never asked twice. Any later run whose control goes red as expected closes every open control issue with a comment naming the run (scripts/publish-verify-close-issue.sh). While a GREEN or wrong-reason issue is open, a green gate proves nothing. An inconclusive control fails the run too. Reproduce with scripts/publish-verify-control.sh [<base>]. The offline check, scripts/check-publish-verify-control.sh, runs in ci's shell job. It drives the real verifier against a fake registry and shows the control rejects a verifier that stopped counting MISSING, an absence with no witness, a MISSING line naming another commit, a stray ok, and a red reported against the wrong registry (every MISSING line must name the registry the control's tree names — proposal 040). Its fake registry, like scripts/check-registry-lookup.sh's, accepts only the bearer token it issued and answers any other with a 401, so a verifier that sends the wrong value as the token goes red offline (#999).

Where images and charts are published: one setting (proposal 040)

Every published coordinate derives from bazel/registry/registry.bzl — registry host, namespace, per-component name overrides and the chart-repository prefix — and from nowhere else:

reader	how
image pushes, chart pushes, chart template tests, e2e go_test defaults	load("@aether_registry//:registry.bzl", ...) — image_repository(), image_reference(), chart_registry_url(<chart>)
the //proxy workspace (oci_push)	the same file, load("@aether_registry//:registry.bzl", ...): bazel/registry/ is a local module each workspace reaches with local_path_override (../bazel/registry from proxy/), since //proxy cannot load from the root module
//bazel/proxy_pin (the Envoy the validate gate runs)	image_reference("proxy") + registry_token_url()
workflows	scripts/image-registry.sh >> "$GITHUB_ENV" after checkout → IMAGE_REGISTRY_HOST, IMAGE_NAMESPACE, IMAGE_REGISTRY (host/namespace), PROXY_IMAGE, IMAGE_SIGNATURE_LAYOUT
verifiers, e2e scripts	scripts/image-registry.sh {prefix,host,repo <c>,ref <c>,chart-repo <c>,chart-ref <c>,signature-layout}; scripts/registry-lib.sh resolves its repository lists through it

It says quay.io / aethermesh since the phase-2 cut-over (proposal 040): images quay.io/aethermesh/<component> (the proxy is plain proxy), charts quay.io/aethermesh/chart-<chart>, signatures as OCI 1.1 referrers (SIGNATURE_LAYOUT = "referrer"). Before it: ghcr.io/bpalermo/aether/<component>, the proxy …/aether-proxy, charts …/charts/<chart>, signatures as cosign tags. The cut-over commit is the phase-2 PR's merge commit — the first whose registry.bzl says quay.io.

Charts are pushed with oras, not helm push. helm push appends the chart's name: to its base and cannot name a flat chart-<name> repository; Quay has no nested repositories (and bare prober / udsecho would collide with those images). chart_push (//bazel/helm:defs.bzl) writes exactly the artifact helm push writes — the packaged .tgz as the application/vnd.cncf.helm.chart.content.v1.tar+gzip layer, Chart.yaml as the application/vnd.cncf.helm.config.v1+json config (//bazel/chartconfig) — with the pinned oras (//bazel/oras, v1.3.4, version_test) to chart_registry_url(<chart>), tagged with the packaged version. publish.yaml runs <chart>.push_images, then <chart>.chart_push (and aether_commit.chart_push for the commit-suffixed tag). helm pull oci://quay.io/aethermesh/chart-aether --version <v> reads it like any helm-pushed chart (proved against a local registry: same layer digest, same config, helm template renders).

Credentials. Every job that pushes or signs (publish.yaml's publish, proxy-release.yml's build-push, manifest, sign) declares environment: release and logs in to $IMAGE_REGISTRY_HOST with the Quay robot (QUAY_USERNAME / QUAY_TOKEN, secrets of the release environment only; deployment-branch policy main). GITHUB_TOKEN is no longer a registry credential; it stays for GitHub API calls (PRs, issues). The repositories are pre-created public in the org — the robot cannot create repositories (the quay-smoke gate), so a new component needs its repository created (public, robot write) before its first publish.

The sweep across the cut-over (the split rule). verify-published-artifacts.sh reads the registry setting as of each push head it checks (git show <sha>:bazel/registry/registry.bzl, or bazel/img/registry.bzl for a head from before the setting became its own module — REGISTRY_SETTING_PATHS in scripts/registry-lib.sh), exactly as it reads each chart's version and the release-tag prefix: heads at or after the cut-over are checked on quay.io (chart-<name>, flat names, a signature REFERRER — and nothing else: a fallback tag there, or a referrer plus a tag, is MISSING). Heads from before the cut-over — a registry.bzl without SIGNATURE_LAYOUT, or none at all (before

998) — are exit 2 and never read: phase 4 decommissioned the sweep's ghcr.io

branch (the first-version fallback, the tag default and GHCR_TOKEN are gone), so naming such a commit by hand gets "decommissioned (proposal 040 phase 4)", not a verdict. A post-cut-over head whose artifacts exist only on ghcr.io is MISSING, never borrowed from the old registry. Each commit block of the output starts with a registry <host>/<namespace>, signatures as <layout> line saying which setting it used. The aether-proxy pin is looked up where the pin says (it moves with the next proxy release, not with the flip), in the layout of the newest registry.bzl under which that reference was image_reference("proxy").

The aether-proxy pin during the cut-over. The flip could not move the pin (it is data only proxy-release.yml's bump-chart job writes), so right after the cut-over charts/aether/values.yaml still named the ghcr.io image, and every pin reader briefly accepted it too (PROXY_PIN_LEGACY_REFERENCES). The phase-2 merge triggered a proxy release whose bump-chart PR (#1027) re-pinned to quay.io/aethermesh/proxy. Phase 4 removed the allowance: every pin reader (//bazel/proxy_pin, the sweep, ci.yaml's proxy-pin job, proxy_pin_rewrite in scripts/proxy-pin-lib.sh) accepts exactly image_reference("proxy"), and a pin on any other reference is a hard failure — never a lookup on the old registry.

Migrating a chart consumer. Every chart took a major bump (aether 1.0.0, and crds / prober / udsecho to 1.0.0) because the default image repositories moved. Upgrade from the new coordinates with the values you run today — helm get values <release> -n <ns> -o yaml > values.yaml, then helm upgrade <release> oci://quay.io/aethermesh/chart-<name> --version <v> -f values.yaml; never --reuse-values, which pins the old chart's defaults. Anyone mirroring images and overriding repository by prefix must now mirror from quay.io/aethermesh/<component> and override each image's repository individually. Releases published before the cut-over went to ghcr.io and were not copied to quay.io.

ghcr decommissioned (proposal 040 phase 4, #1167). Nothing in the repository reads, verifies or publishes to ghcr.io any more: the ghcr-lib.sh shim and the ghcr_* aliases are gone, PROXY_PIN_LEGACY_REFERENCES is gone (the proxy pin must name image_reference("proxy")), and the sweep refuses pre-cut-over heads (above) with no GHCR_TOKEN. The ghcr.io packages themselves were not touched: whether to delete them or keep them read-only as an archive is a separate maintainer decision (deleting them breaks every historical pin).

scripts/check-registry-config.sh (in ci's and proxy's shell jobs) keeps it one setting: the file parses, the proxy copy is identical, the aether-proxy pin in charts/aether/values.yaml names exactly image_reference("proxy"), no file outside its written-down allow-list (docs, the website, the two READMEs, the setting itself, the pin) spells the current registry out, and no file outside its legacy allow-list (the setting's history note, proposals, this runbook's records, observability notes) spells a pre-cut-over coordinate out. //bazel/img:registry_test pins image-registry.sh against the Starlark helpers.

Registry library. scripts/registry-lib.sh (was ghcr-lib.sh; the shim and the ghcr_* aliases were removed in proposal 040 phase 4) is registry-neutral: it speaks the OCI distribution API against REGISTRY_HOST (default: the setting's host), fetches the anonymous pull token from the registry's own endpoint (ghcr.io /token, quay.io /v2/auth), and for private repositories takes REGISTRY_USERNAME + REGISTRY_PASSWORD (a Quay robot account, sent only to the host it was issued for). registry_referrers reads the OCI 1.1 Referrers API (GET /v2/<repo>/referrers/<digest>): quay.io serves it, ghcr.io answers 404 — which the library reads as "no referrers API", never as "no signatures".

Quay smoke (the phase-2 gate). Proves push + keyless sign + verify on quay.io with one throwaway image. It gated the flip (green: run 36345331611 — the robot pushes to a pre-created repository but cannot create one; cosign v3.1.2 writes the signature as a referrer, no tag, on the index and every child; verify_image_signatures passes), and stays dispatchable for re-checking quay after a cosign or robot change:

gh workflow run quay-smoke.yaml --ref main    # main only: the `release` environment holds the robot secrets
gh run watch "$(gh run list --workflow quay-smoke.yaml --limit 1 --json databaseId -q '.[0].databaseId')"

.github/workflows/quay-smoke.yaml (dispatch-only, every step is scripts/quay-smoke.sh <subcommand>) pushes //e2e/l4echo:smoke_push — the l4echo test image's rules_img amd64+arm64 index, built and pushed by the same rules_img path as the released images — to quay.io/aethermesh/smoke:<run id>-<attempt> with the robot account, signs it cosign sign --recursive by digest as …/quay-smoke.yaml@refs/heads/main, verifies the index and every child with //bazel/cosign:verify_image_signatures, then deletes the tag (and any cosign fallback tags) and leaves the repository. The run summary is a table:

row	meaning
repository existed before / created by this push	robot tags/list before the push. no then a successful push = the robot can create repositories in the org (open question "Repository auto-creation")
push	denied fails the step: pre-create the repository public and give the robot write, or give the robot the org creator role
public (anonymous pull)	Quay API is_public (anonymous) AND an anonymous pull of the tag. Anything but yes fails the job: make it public (repository settings), or set the org's default visibility before the robot creates repositories
digest	the index digest the registry holds under the tag (cross-checked against what Bazel pushed)
signed before this run	the digest is deterministic per commit, so a re-run at the same commit finds the earlier run's signature; the layout rows then include it
signature layout (index) / (index + children)	referrer, bundle (sha256-<hex> fallback tag), legacy (.sig), or both (a referrer AND a tag — the sweep's double-write defect), from registry_signature_layout_direct. mixed in the second row = the children disagree with the index. This answers where cosign v3.1.2 puts a signature on quay and decides phase 2's verify path
referrer artifactTypes	what the referrers API lists for the index (application/vnd.dev.sigstore.bundle.v0.3+json expected)
verify (index + every child)	verify_image_signatures with identity ^https://github\.com/<repo>/\.github/workflows/quay-smoke\.yaml@refs/heads/main$
certificate SAN	the X.509 SAN URI of the signing certificate — the workflow ref, NOT the OIDC sub — read from the sigstore bundle cosign download signature returns (verificationMaterial.certificate.rawBytes, or the chain's leaf), decoded with openssl. cosign v3's verify JSON has no optional block, so there is nothing to read there. A SAN that cannot be read, or that does not match the verify identity, fails the summary
OIDC token	the Actions token's sub (with environment: release: repo:<owner>/<repo>:environment:release) and job_workflow_ref; the token itself is masked and never printed
cleanup	tags deleted with a robot pull,push token (DELETE /v2/<repo>/manifests/<tag>); a failure is a warning, not a red

Verifying image signatures (cosign v3, #925)

publish.yaml signs every published image keyless with cosign sign --recursive — the multi-arch index and each per-architecture child manifest.

Each digest is signed and attested once (#1378). A digest can be published by more than one run: a re-run of the workflow, or a later commit that did not change that image. scripts/publish-sign.sh asks first whether the index and every child already verify under this workflow's identity (the registry's referrers index, then the same verify_image_signatures the verify step runs) and signs only what does not; a question that goes unanswered is answered by signing. scripts/publish-provenance.sh subjects asks GitHub's attestation store whether the digest has provenance that verifies as this workflow and hands actions/attest only the digests that have none; there an unanswered lookup fails the step, and when nothing is new the attest step is skipped with a ::notice::. Nothing is skipped in the checks: every reference, signed or attested now or before, goes through both verify steps. The run summary lists every artifact with its digest, whether the digest is new, the commit its provenance names and the tag: that is the list of what a deploy of that commit rolls. //scripts:publish_sign_test and //scripts:publish_provenance_test hold the logic and the workflow's wiring.

The user-facing commands (signature and provenance, images and charts) are in Verifying a release; this section is how it is built.

Charts are signed as well, and everything gets build provenance. After the images, publish.yaml signs each chart manifest (cosign sign, no --recursive: a chart is one manifest). The references come from scripts/published-chart-refs.sh <sha>: every chart of REGISTRY_CHARTS under its X.Y.Z-<sha> tag, plus aether's bare X.Y.Z tag, each resolved to a digest right after the push. The run verifies them with verify_image_signatures -- --single --file signed-charts.txt (--single: no child walk, and a ref that turns out to be an index is refused). Then actions/attest writes ONE SLSA v1 provenance statement whose subjects are the image indexes and chart manifests that had no provenance yet (every chart, and each image whose digest is new), stored in the repository's attestation store on GitHub (not in the registry: push-to-registry is off), and the last step runs gh attestation verify oci://<ref> for every reference with the workflow pinned: with --source-digest <commit> as well for a digest this run attested, without it for one attested by an earlier commit (its provenance names that commit, and the step prints it). A failure in any of these fails the publish run after the artifacts are pushed, like a failed image signature does: re-run the workflow, which signs and attests only what the failed run left unsigned or unattested. The post-publish sweep (publish-verify) does not look at chart signatures or attestations yet. The proxy image (proxy-release.yml) is signed but not attested yet.

Getting cosign: bazel run //bazel/cosign. There is one cosign for this repository, in CI and on a workstation: the official release binary that the rules_img_signer_cosign bazel_dep (MODULE.bazel) downloads for the current platform, sha256-pinned in that module's cli/cosign_cli.lock.json — cosign v3.1.2 at module 0.0.1. publish.yaml, proxy-release.yml and publish-verify.yaml all sign and verify through //bazel/cosign, run locally on the runner (never on RBE), so the three cannot drift apart.

bazel run //bazel/cosign -- version            # GitVersion: v3.1.2
bazel run //bazel/cosign -- verify …           # any cosign subcommand

The version is the module's lock, so bumping cosign is bumping the bazel_dep; //bazel/cosign:version_test (network-free) fails until its EXPECTED moves too, so a new cosign never lands unread. Until 2026-09-27 this was sigstore/cosign-installer pinned by SHA (v4.1.2 → cosign v3.0.6).

Why only the module's CLI, not its signer plugin. rules_img_signer_cosign also ships sign-oci-artifact, a signer plugin for rules_img's signing_config / img deploy. Signing stays an explicit cosign sign --recursive step after the pushes, by digest, with the same cosign every verifier runs: that path is the one the quay-smoke gate measured on quay.io (referrer, no tag), and the plugin's layout there is unmeasured.

Layout. quay.io serves the OCI 1.1 Referrers API, so cosign 3 attaches the sigstore bundle as a real referrer of the signed manifest (artifactType application/vnd.dev.sigstore.bundle.v0.3+json, annotation dev.sigstore.bundle.predicateType: https://sigstore.dev/cosign/sign/v1) and writes no tag — measured by the quay-smoke gate on our own index and children, and observed on quay.io/argoproj/argocd and quay.io/cilium/cilium. bazel/registry/registry.bzl records that as SIGNATURE_LAYOUT = "referrer", and the sweep holds every post-cut-over commit to it: a fallback tag on quay.io, or a referrer plus a tag, is MISSING. Before the cut-over, on ghcr.io (no Referrers API), the signature landed in a tag in the image's own repository: cosign 3's OCI 1.1 fallback index sha256-<hex> (the bundle format — keyless v3 requires --new-bundle-format; =false is rejected), and before 2026-09-24 cosign 2's sha256-<hex>.sig; a registry.bzl without SIGNATURE_LAYOUT promises that tag layout. Exactly one of the three layouts may exist per digest (scripts/registry-lib.sh, registry_signature_layout; a bundle referrer with any other predicateType is an attestation, not a signature). cosign verify finds all three itself. Do not pass --new-bundle-format to cosign verify expecting it to assert the layout: =true still accepts a legacy .sig.

Children are not verified unless you walk them. cosign verify has no --recursive (not v2.4.1, v3.0.6 or v3.1.2). Verifying the index digest says nothing about the per-arch manifests a node actually pulls. Use the script, which walks the index's .manifests[] from the registry and verifies every child with the same identity and issuer:

# Read-only; no credentials. The target sets COSIGN to the pinned cosign.
bazel run //bazel/cosign:verify_image_signatures -- quay.io/aethermesh/agent@sha256:<index digest>
#   verified index quay.io/aethermesh/agent@sha256:…
#   verified child quay.io/aethermesh/agent@sha256:…   (linux/amd64)
#   verified child quay.io/aethermesh/agent@sha256:…   (linux/arm64)

Identity is ^https://github\.com/bpalermo/aether/\.github/workflows/publish\.yaml@, issuer https://token.actions.githubusercontent.com. Exit 1 names every index or child that did not verify; exit 2 means it could not check (e.g. the digest is not an index, or lists no children — a walk over nothing is never a pass). The same target runs in publish.yaml's verify step and in publish-verify. --file <refs> takes one ref per line; relative paths resolve against your working directory. Standalone, COSIGN=/path/to/cosign scripts/verify-image-signatures.sh … still works with any cosign — the version is then yours to vouch for.

cosign v3.1.2 verifies every layout: a pre-2026-09-24 legacy .sig (agent@sha256:14949387…, commit 5b0c199) and a v3 fallback-tag bundle (agent@sha256:066267b6…, commit 7f8f825) on ghcr.io both pass, index and children, and so does a referrer on quay.io (the quay-smoke gate). Images published before the cut-over are verified at their ghcr.io coordinates.

Not signed by this pipeline: any agent-family image published before f332061 (#875, 2026-09-20) — those verify as no signatures found. The proxy is signed by its own workflow; see the next section.

Verifying the aether-proxy signature (#984)

The proxy image (quay.io/aethermesh/proxy since the cut-over; before it, and in the chart until the first proxy release after it re-pins, ghcr.io/bpalermo/aether/aether-proxy) is built by proxy-release.yml, not publish.yaml, and is versioned by the commit that changed proxy/, not by the aether commit — the chart carries its digest in charts/aether/values.yaml (proxy.image.digest). Since #984 that workflow's sign job runs cosign sign --recursive on the index right after the manifest job publishes it (same Bazel-pinned //bazel/cosign, keyless, v3 bundle layout), verifies the index and every child, and only then lets bump-chart open the pin PR. A proxy index that does not verify is never pinned.

The identity is different. Fulcio binds the certificate to the workflow that signed, so proxy signatures carry

https://github.com/bpalermo/aether/.github/workflows/proxy-release.yml@refs/heads/main

(issuer https://token.actions.githubusercontent.com). The publish identity rejects them, and vice versa. By hand, for whatever digest a chart pins:

# The pin names its own registry: read repository AND digest from the chart.
pinned="$(bash -c '. scripts/proxy-pin-lib.sh && proxy_pinned_ref' < charts/aether/values.yaml)"   # "<repository> <digest>"
CERT_IDENTITY_REGEXP='^https://github\.com/bpalermo/aether/\.github/workflows/proxy-release\.yml@refs/heads/main$' \
  bazel run //bazel/cosign:verify_image_signatures -- "${pinned% *}@${pinned#* }"

Pins older than signing are unsigned, permanently. Every proxy image published before #984 has no signature (0 signature tags in the repository), and that includes the digest pinned when #984 merged (sha256:938c5a57…, children bb53bed6… amd64 / 4a90a7fb… arm64). The hand check above fails on it with no signatures found ×3 — that is the expected answer, not a regression. The first signed digest is whatever the proxy-release run triggered by #984's merge pins; the first one verified by hand (2026-09-27, cosign v3.1.2 under the identity above) is sha256:574d5211…, index plus both children (4d7b967b…, 1505e36b…).

The sweep. make check-published / publish-verify check, for every commit, the proxy digest that commit's values.yaml pins: it must exist, and the index and every child must carry a signature (the cosign pass then verifies them under the proxy identity above). Every pin is checked, so an unsigned proxy pin goes red, and that includes a revert to an old unsigned digest. There used to be a PROXY_SIGNING_CUTOVER that skipped the unsigned pre-signing pins as history.

1191 removed it as unreachable: every one of those pins named the pre-cut-over

registry, which the pin reader refuses, and the sweep refuses any commit that predates the Quay cut-over (proposal 040 phase 4). Each digest ever pinned under image_reference("proxy") was introduced after the signing cut-over.

Recovering from a failed first install (#1406)

Every command below was run on kind (Kubernetes 1.35, Helm 3.18.4) with the API server enforcing Pod Security baseline by default. Start by looking at what is there:

helm list -n aether-system -a                      # the release and its status
kubectl get namespace aether-system --show-labels  # exists? labelled privileged?
kubectl -n aether-system get ds,deploy
kubectl -n aether-system get events --field-selector reason=FailedCreate

namespaces "aether-system" already exists — a chart older than 2.4.21 installed with --create-namespace (the chart rendered the namespace Helm had just created). What it leaves: a release in status failed, a namespace without the pod-security labels, and everything else the chart renders. The Deployments run; where Pod Security is enforced the four DaemonSets have no pods (FailedCreate: violates PodSecurity). Start again from nothing, with chart 2.4.21 or newer:

helm uninstall aether -n aether-system     # prints: kept ... [MeshConfig] default
kubectl delete namespace aether-system     # takes the kept MeshConfig with it
# then the install from getting-started.md, "Install": create and label the
# namespace, and `helm upgrade --install ... --create-namespace`

helm install over the failed release answers cannot re-use a name that is still in use, which is why the uninstall comes first. Nothing cluster-scoped is left behind by the uninstall.

To keep the namespace instead (say, something else already lives in it), do not simply run the new chart over the failed release: that release's manifest lists the namespace, the new chart's does not, and Helm deletes an object that leaves the manifest. Chart 2.4.21 refuses that upgrade and names the fix, which is to mark the namespace first:

kubectl annotate namespace aether-system helm.sh/resource-policy=keep
kubectl label namespace aether-system --overwrite \
  pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/audit=privileged \
  pod-security.kubernetes.io/warn=privileged
# then the same `helm upgrade --install ... --create-namespace` again

namespaces "aether-system" not found or invalid ownership metadata — a chart older than 2.4.21 installed without --create-namespace, or into a namespace made with a plain kubectl create namespace. Nothing was created and no release was stored. Run the documented install with chart 2.4.21 or newer; an existing namespace can stay.

The release is deployed but the DaemonSets have no pods — chart 2.4.21 or newer on a cluster that enforces Pod Security admission, namespace not labelled. The install notes print a NAMESPACE NOTE, and the events say violates PodSecurity "baseline:latest". Label the namespace; nothing has to be reinstalled (--overwrite because the namespace may already carry another level):

kubectl label namespace aether-system --overwrite \
  pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/audit=privileged \
  pod-security.kubernetes.io/warn=privileged
kubectl -n aether-system rollout restart daemonset
kubectl -n aether-system get ds   # CURRENT reaches DESIRED

The rollout restart is there because the DaemonSet controller retries a refused pod at a growing interval and a label on the namespace does not wake it. On kind the pods appeared within 5 s of the label when it came half a minute after the install; when it came four minutes after, they had not appeared 20 s later and did at once after the restart.

namespace.create=true cannot create "aether-system" — a render error; nothing was created, not even the namespace --create-namespace would have made. Drop namespace.create=true (and label the namespace yourself), or store the release in another namespace (helm -n <other> ... --set namespace.name=aether-system).

--wait succeeding is not the check that the DaemonSets run (#1473). Helm counts a DaemonSet as ready once numberReady >= desiredNumberScheduled - maxUnavailable, which for a DaemonSet that may have one pod unavailable (the agent at its default strategy, uds-csi, the prober chart's DaemonSet) is zero pods on a one-node cluster: helm upgrade --install --wait reports deployed with none running. Follow an install with kubectl -n aether-system rollout status daemonset/<name> for each DaemonSet (it waits until every scheduled pod is updated and available), or read the pod list.

A first install that failed for another reason (a --wait that timed out) is retried with the same command: helm upgrade --install upgrades the failed release in place. If it failed very early, before the agent ServiceAccount was written (one of that name already existed, say), the retry may ask once for kubectl annotate namespace aether-system helm.sh/resource-policy=keep: without the mark that ServiceAccount carries, the chart cannot see that the failed revision rendered no namespace, and it refuses rather than risk a delete. The command is safe to run; nothing is deleted either way.

Uninstalling: the proxy pods take close to their whole grace period (#1472)

After helm uninstall (or anything else that deletes the aether-proxy DaemonSet) each proxy pod stays Terminating for most of proxy.terminationGracePeriodSeconds, 180 s by default, and a namespace deleted afterwards is only gone once they are. That is the supervisor's successor_wait branch of "What the proxy supervisor does on SIGTERM" below, not a hang:

• On SIGTERM with its Envoy still serving, the supervisor waits for a surge replacement to hot-restart it. It has no Kubernetes client (by design: see //agent/cmd/proxy-supervisor:deps_test), so it cannot tell "my DaemonSet was deleted" from "I am being rolled", and a pod delete, a node drain and an uninstall look the same to it.
• The wait is bounded by the pod's own deadline: terminationGracePeriodSeconds - (hotRestart.drainTime + 5 s + 10 s), which is 155 s at the defaults. Then it drains Envoy over hotRestart.drainTime (10 s) and stops it. From the code that is about 165 to 170 s per pod at the defaults; #1472 observed the full 180 s on kind. Neither figure was measured for this note.

The log lines of the "no successor can come" case in that section are the ones to expect, and aether_supervisor_shutdown_branch_total reports drain_fallback.

To remove the mesh faster, tell the supervisors beforehand that no replacement will come: one upgrade with proxy.shutdownDrainImmediately=true (values read back with helm get values <release> -n <ns> -o yaml and passed with -f, never --reuse-values), let that roll finish, then uninstall. Each proxy then drains on SIGTERM without the wait. That upgrade is itself a proxy roll, and with the flag set a later pod delete or node drain is no longer hitless, so this is for a mesh that is being removed, not for one that stays.

helm uninstall does not delete the namespace, and keeps the seeded default MeshConfig (it prints kept); kubectl delete namespace removes both.

Which workloads a chart upgrade rolls (#1363)

Since chart 2.4.15 the chart itself no longer puts the release into a pod template: a helm upgrade rolls a workload only when that workload's pod template changed (an image digest, a flag, a resource, a volume).

And an image changes only when what is in it changes (#1378). Until #1378 every image this repository builds got a new digest with every commit, whether or not its content changed: the image carried the commit as its org.opencontainers.image.revision label and annotations (#837), and seven binaries had the commit linked in as their version. On a test cluster all seven in-repo image digests changed in each of four consecutive deploys, including one whose commits touched only the registrar, tests and scripts; and since every workload runs at least one of those images, a deploy of any new commit rolled everything, the node proxy included.

Neither is true any more for the Go images this workspace builds (agent, mesh-dns, proxy-supervisor, uds-csi, cni-install, registrar, controller, prober): none of them and none of their binaries carries the commit, so the same inputs build the same digest at any commit, the chart pins the same digest, and the pod template does not change. CI holds the build to it, for those images: the test job fails if any action under one of their image indexes takes a workspace-status file, then builds every index twice, as two made-up commits, and fails on a digest that differs (scripts/check-image-digest-stability.sh, or make check-image-digests locally; it takes seconds once the images are built).

The same check holds the images to one more thing: a digest does not depend on the name of the directory Bazel built the image in (#1500). Bazel names bazel-out/<configuration>/ from the flags, so a flag added to .bazelrc, a transition a rule gains or a Bazel upgrade can rename it with no change to any source. Generated Go source (every .pb.go) is compiled from under that directory, and the Go compiler records the path of each file in the binary, so the six binaries that link generated protobuf code (agent, identity-ready, uds-csi, cni, controller, registrar) used to change with the name, GNU build ID and image digest with them. build --experimental_output_paths=strip in .bazelrc is what stops it: rules_go's compile actions are run with every bazel-out/<configuration>/ rewritten to bazel-out/cfg/. A stack trace through generated code reads bazel-out/cfg/bin/api/aether/cni/v1/cniv1_go_proto_/aethermesh.dev/api/aether/cni/v1/service_grpc.pb.go:121 whichever configuration built the binary. The check builds every index a third time under --platform_suffix, which renames the directories and nothing else, and fails on a digest that differs; it also fails if the binaries of the images did not move to other directories, because then it compared a build with itself.

Two consequences for a build. A path-mapped action runs only in a sandbox or on a remote executor; with --spawn_strategy=local (or standalone) Bazel stops with "requires sandboxing due to path mapping" instead of building something else. And to see whether a binary still holds a configuration's name: every line strings <binary> | grep bazel-out/ prints must start with bazel-out/cfg/. A binary that links no generated source (mesh-dns, for one) prints none, and that is fine too.

The aether-proxy image is the exception, and stays one. It is built in the separate proxy/ workspace by its own release workflow, and it still carries the aether commit twice: as org.opencontainers.image.revision (label and annotation, proxy/BUILD.bazel, the image_metadata genrule) and inside the Envoy binary, through Envoy's own version linkstamp (proxy/README.md, "Where the Envoy revision actually is"). That costs no roll: the chart pins the proxy image statically (proxy.image in values.yaml), so it changes only when the pin is bumped. What a deploy of a new commit rolls:

The commits changed	Rolls	Does not roll
docs, tests, CI, scripts only	nothing	everything
the agent only (agent/ code outside the other binaries)	the agent DaemonSet, the edge Deployment (it runs the agent image) and the controller (its identity-gate init container image defaults to the agent image, which is a flag in the controller's pod template)	the node proxy: no Envoy hot restart; mesh-dns, uds-csi, the registrar
one other component only (the registrar, mesh-dns, uds-csi, the controller, the proxy supervisor, cni-install)	the workloads that run that image: the proxy DaemonSet for the supervisor, the agent DaemonSet for cni-install (its init container)	the rest
the chart only (templates, values)	only the workloads whose pod template changed	the rest
a package under common/ (or api/, or a dependency or toolchain pin)	every workload whose image links it, which for a widely used package is all of them	workloads whose images do not link it

Two things follow from "a function of what is in it":

• "Did this deploy change component X" has an exact answer before the deploy: compare the digest the new chart pins for X with the one that is running. bazel query 'rdeps(//..., //common/foo)' restricted to the image targets says which images a source change reaches.
• A proxy roll now means the supervisor image (or the proxy image, a static pin in the chart) really changed. The headroom pre-flight below is for those deploys, not for every deploy.

The first deploy from a commit that has #1378 rolls everything one last time: removing the label, the annotations and the linked version changes every digest once. Plan it like any full roll. The same holds for a rollback across that commit.

Those Go images no longer say which commit built them; three things outside the digest do (the signature's certificate, the provenance attestation and the dev-<sha> tag). See verifying-releases.md, "Which commit built this digest". A Go component's own --version and service.version are its binary's build ID, not a commit (section 5). The proxy image still names its commit in its labels, and Envoy in its version.

What chart 2.4.15 removed is the chart's own share of this: an upgrade that changes the chart version but not the images (a re-cut chart, a values-only change) rolls nothing it did not change.

Until 2.4.15 every pod template carried helm.sh/chart: aether-<version> and app.kubernetes.io/version: <appVersion>. The chart version changes with every release, and a deploy from the commit tag (<X.Y.Z>-<full sha>, above) changes both labels with every commit, so every upgrade rolled the agent, the proxy (a hot restart on every node), mesh-dns, uds-csi, the registrar, the controller and the edge. Both labels are still on the DaemonSet / Deployment objects; they are no longer on the pods.

The upgrade that crosses 2.4.15 rolls everything one last time, because removing the two labels is itself a pod-template change. Plan it like any full roll (the headroom pre-flight below). The same is true of a rollback to a chart older than 2.4.15, which puts the labels back.

To see what an upgrade rolled, compare the pods' template hashes before and after it. A pod that was not rolled keeps its name, its age and its hash:

for ns in aether-system aether-ingress; do
  kubectl -n "$ns" get pods -l app.kubernetes.io/part-of=aether \
    -L controller-revision-hash,pod-template-hash \
    --sort-by=.metadata.creationTimestamp
done
# The release each workload object belongs to (the labels the pods lost):
kubectl get ds,deploy -A -l app.kubernetes.io/part-of=aether \
  -L helm.sh/chart,app.kubernetes.io/version

To know before an upgrade, render both charts with the values you deploy with and compare; a workload whose spec.template is unchanged is not rolled:

helm get values aether -n aether-system -o yaml > values.yaml   # never --reuse-values
helm get manifest aether -n aether-system > before.yaml
helm template aether oci://quay.io/aethermesh/chart-aether --version "$AETHER_VERSION" \
  -n aether-system -f values.yaml > after.yaml
diff before.yaml after.yaml

//charts/aether:aether_pod_template_version_test keeps it that way: it renders the chart at two versions and fails if any pod template differs. A new pod label, annotation, env var or argument that carries the chart version or the appVersion fails it.

A config checksum annotation is a different thing: it rolls a pod when the configuration it reads at start changes, which is the point. The version labels used to do that by accident, so 2.4.15 added the one that is needed:

Pod	Annotation	Changes when	Why a roll is needed
edge	checksum/edge-config	the edge's Envoy bootstrap (aether-edge-config, data only)	plain envoy -c, nothing watches the file

Two ConfigMaps have no checksum on purpose, because their reader watches the mounted file. Each node picks the change up when its own kubelet delivers it (tens of seconds observed, no guaranteed bound), independently of the others, not one node at a time like a DaemonSet roll:

• The node proxy's own bootstrap (aether-proxy-config): the supervisor watches it (--watch-config=true) and hot-restarts Envoy in place, without replacing the pod. The watch is in place before the supervisor starts its first Envoy, so a change delivered while the pod is starting is either read by that Envoy or followed by a hot restart; until #1470 one that landed just after the first start could go unnoticed until the next change.
• The OPA preset's policy (aether-opa-policy), since chart 2.4.19 (#1383): the sidecar runs opa run --watch and reloads it with no restart of anything. Chart 2.4.15–2.4.18 carried checksum/opa-policy and rolled the proxy DaemonSet for every policy change. The upgrade that crosses 2.4.19 with the OPA preset on rolls the proxy DaemonSet one last time (the annotation leaves the pod template and the sidecar's arguments gain --watch). See Changing the OPA policy under the ext_authz sidecar.

An agent image change also rolls the controller (#1428)

The controller's pod template names the agent image: the egress identity gate's init container runs /identity-ready, which ships as a layer of the agent image, and the controller is told which image to inject with --identity-gate-image (default: agent.image). A new agent image digest is therefore a new controller argument, and the controller Deployment rolls with the agent DaemonSet even when the controller's own image did not change. This is accepted, not a defect: the node's agent pod runs the same image, so the init container normally finds it in the node's image cache, and the alternative (an image of its own for identity-ready) would give that up to save a controller roll.

"Normally", not always. During an upgrade the controller and the agent DaemonSet roll independently: a controller replica that already has the new digest can admit a pod onto a node whose agent has not been replaced yet, and that pod's init container then pulls the new agent image itself (pullPolicy: IfNotPresent) before it can start. The same holds on a node that has just joined. The cost is one image pull on that pod's start, once per node.

Three things follow:

• A controller roll replaces two replicas behind a Service one at a time (surge 1, none unavailable) and does not touch running mesh pods. A pod keeps the init container image it was admitted with; only pods admitted afterwards get the new one.
• controller.webhook.identityGate.image.* pointed at a fixed mirror tag removes the coupling, at the price of keeping that mirror in step with the agent image yourself.
• The argument is not passed at all when the gate is off: spire.enabled=false, controller.webhook.identityGate.enabled=false, or no mutating webhook (namespaceInjection and injectPodNdots both off, chart >= 2.4.20, #1432). In those releases an agent image change does not roll the controller.

To see the coupling on a cluster (the two digests are the same):

kubectl -n aether-system get deploy aether-controller \
  -o jsonpath='{range .spec.template.spec.containers[0].args[*]}{@}{"\n"}{end}' | grep identity-gate-image=
kubectl -n aether-system get ds aether-agent -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Chart 2.4.20: the registrar, the controller and the edge roll once (#1434)

The upgrade that crosses 2.4.20 changes the pod template of the three Deployments that run two replicas, so each rolls once, on Kubernetes >= 1.27:

Deployment	What changes in its pod template
registrar	its spread constraint gains matchLabelKeys: [pod-template-hash]
edge (when enabled)	the same
controller	its default affinity (two preferred anti-affinity terms: its own replicas, the registrars' node) is gone; a spread constraint with matchLabelKeys keeps its replicas apart. A controller.affinity you set is rendered as before
controller, only with namespaceInjection and injectPodNdots both off	additionally --identity-gate=true and the gate's flags become --identity-gate=false (#1432)

No DaemonSet changes. Below Kubernetes 1.27 the registrar and the edge render exactly as before and only the controller rolls.

What it is for: the registrar and the edge already had a soft spread constraint, and both registrar replicas were still found on one node of a five-worker cluster. Two things defeated it, measured on kind (Kubernetes 1.35, a tainted control plane plus workers, stand-in pods carrying the chart's scheduling fields, the three Deployments rolled together twenty times; rollouts that ended with both replicas of a Deployment on one node, as controller / edge / registrar):

Pod templates	2 workers	3 workers
chart 2.4.18	8 / 5 / 10	1 / 0 / 5
2.4.18 without the controller's "off the registrars' node" term	0 / 6 / 2	0 / 0 / 1
matchLabelKeys added, that term kept	9 / 0 / 11	1 / 0 / 3
chart 2.4.20: matchLabelKeys added, that term gone	0 / 0 / 0	0 / 0 / 0

• The controller's default affinity preferred nodes without a registrar. The scheduler applies a preferred anti-affinity term in both directions, so every new registrar was also pushed off the nodes that held a controller, with the same weight as its own spread constraint pulled it onto an empty one. The two cancelled, and the tie fell to the scheduler's other scores.
• Without matchLabelKeys the constraint counted the pods of the ReplicaSet being rolled away beside the new ones.

The default stays a preference (<component>.nodeSpread: soft); required makes it a rule, and none removes it (docs/configuration.md). If you set controller.affinity to keep the controller off the registrars' node, expect the third row (matchLabelKeys added, that term kept): the constraint still has matchLabelKeys, and the term cancels it as before.

Nothing moves a running pod, before or after the upgrade: a spread constraint acts when a pod is scheduled. The roll this upgrade causes is what re-places the replicas. To see where they are, before and after:

kubectl get pods -A \
  -l 'app.kubernetes.io/part-of=aether,app.kubernetes.io/component in (registrar,controller,edge)' \
  -o 'custom-columns=COMPONENT:.metadata.labels.app\.kubernetes\.io/component,POD:.metadata.name,NODE:.spec.nodeName' \
  --sort-by='.metadata.labels.app\.kubernetes\.io/component'
# The constraint each Deployment carries:
kubectl get deploy -A -l app.kubernetes.io/part-of=aether \
  -o custom-columns=NAME:.metadata.name,SPREAD:.spec.template.spec.topologySpreadConstraints

Two replicas of one component on the same node after the roll means the preference lost to the scheduler's other scores (or only one node had room). kubectl -n aether-system rollout restart deploy/aether-registrar schedules them again; registrar.nodeSpread=required stops it happening.

Chart 2.4.21: the chart no longer creates the release's namespace by default (#1403)

Before upgrading a release that an older chart installed with namespace.create=false, run this once (it is what every install that passed --set namespace.create=false, or whose values file says so, is):

kubectl annotate namespace aether-system helm.sh/resource-policy=keep
# the prober chart, before its first upgrade to 1.0.6 (its default was
# already namespace.create=false, so this is every prober release):
kubectl annotate namespace <prober namespace> helm.sh/resource-policy=keep

Without it the first upgrade is refused with a message naming that command; nothing is changed and nothing is deleted. Under a GitOps controller that drives Helm the symptom is a release that fails to upgrade, with that message, until the namespace is annotated. A release installed with the old default (namespace.create=true) needs nothing.

namespace.create defaults to false. No workload rolls: no pod template changed. What the first upgrade to 2.4.21 does depends on what the chart can see on the live namespace, and in no case does it let Helm delete it:

The release was installed with	The live namespace	First upgrade to 2.4.21 (any namespace.create)
the old default, or namespace.create=true	carries Helm's ownership annotations for the release, or at least the chart's labels (app.kubernetes.io/managed-by: Helm, app.kubernetes.io/instance: <release>)	goes through; the Namespace stays in the manifest and gains helm.sh/resource-policy: keep
namespace.create=false (+ --create-namespace, or a namespace you made)	carries neither	refused once, with the command below; goes through after it
the old default, but the namespace lost the annotations and the chart's labels	carries neither	the same refusal

The one extra step, for a release installed with namespace.create=false (and for the rare namespace that lost every sign of its owner). Run it before the upgrade, or when the upgrade's error names it:

kubectl annotate namespace aether-system helm.sh/resource-policy=keep

Why: Helm deletes an object that was in the previous manifest and is not in the new one, and a chart cannot read the previous manifest. A release written by an older chart looks the same from the cluster whether its manifest holds the Namespace or not, so the chart leaves the Namespace out only when the live object says keep, or when the previous revision was written by 2.4.21 or newer and recorded that it rendered none (the aether.io/release-namespace-rendered: "false" annotation on the agent ServiceAccount). The annotation is harmless on a namespace the release never rendered, the command is needed once, and you may remove the annotation again after that upgrade. With a GitOps controller that drives Helm, the refused upgrade shows as a failed release with this message; annotate, then let it retry. The prober chart 1.0.6 asks the same of its namespace on the first upgrade from an older chart (kubectl annotate namespace <prober namespace> helm.sh/resource-policy=keep).

The check, before and after (the UID must not change):

# Owned by the release? Both annotations name it when it is.
kubectl get namespace aether-system \
  -o jsonpath='{.metadata.uid}{"\n"}{.metadata.annotations}{"\n"}'
# Is the Namespace part of the release? 1 = yes, 0 = no.
helm get manifest aether -n aether-system | grep -c '^kind: Namespace'
# What the upgrade would send, rendered against the cluster (never
# --reuse-values: read the values back and pass them with -f):
helm get values aether -n aether-system -o yaml > values.yaml
helm upgrade aether oci://quay.io/aethermesh/chart-aether --version "$AETHER_VERSION" \
  -n aether-system -f values.yaml --dry-run=server | grep -A6 '^kind: Namespace'

For a namespace in the first row the last command must print the Namespace with helm.sh/resource-policy: keep; for the other rows it fails with the kubectl annotate command until that has been run. helm template cannot answer this question: it renders without a cluster, sees no owner, and prints no Namespace. For the same reason, a pipeline that renders with helm template and prunes what is no longer rendered (kubectl apply --prune, a GitOps tool in that mode) would remove a namespace the old chart rendered: exclude the namespace from pruning in that tool, or keep it in your own manifests, before upgrading. Helm itself (the CLI, or a controller that drives the Helm SDK against the cluster) is not affected.

Consequences to know:

• helm uninstall no longer deletes a namespace the chart rendered; delete it yourself.
• namespace.create=true with the release stored in the namespace it names is refused at render time on a first install (it could never be installed); an existing release that owns its namespace is not refused.
• A release whose only revision is a failed first install of an older chart gets the same refusal and the same command; see "Recovering from a failed first install".
• Moving a release to another namespace (namespace.name, or edge.namespace, changed on an upgrade) is refused while a namespace the release owns would be left behind unprotected: Helm would delete it with everything in it. The message names it; annotate it with helm.sh/resource-policy=keep, upgrade, and delete the old namespace yourself if it should go. The prober chart checks this when namespace.name or namespace.create is set.

Chart 2.4.22: with the OPA preset on, the proxy DaemonSet rolls once (#1401)

proxy.authzSidecar.opa.image was openpolicyagent/opa:1.21.1-envoy-static, a tag, and is now the same tag with the digest of its multi-arch index (…-envoy-static@sha256:b4a8bbe8…344b): a tag can be re-pushed, and this container sits next to the proxy on every node. The OPA version does not change.

The sidecar is a container of the proxy pod, so the upgrade that crosses 2.4.22 rolls the proxy DaemonSet once on a cluster where proxy.authzSidecar.opa.enabled is true and opa.image is the chart's default. With the preset off (the default), or with opa.image set to your own reference (a mirror), nothing renders differently and nothing rolls. A mirror reference is used as written: pin it by digest yourself.

Before and after the upgrade:

# Is the preset on, and which image does the release set? (never --reuse-values)
helm get values aether -n aether-system -a -o yaml | grep -A12 'authzSidecar:'
# The image each proxy pod's sidecar really runs: the digest the node resolved.
kubectl -n aether-system get pods -l app.kubernetes.io/component=proxy \
  -o jsonpath='{range .items[*]}{.spec.nodeName}{"  "}{.spec.initContainers[?(@.name=="authz")].image}{"  "}{.status.initContainerStatuses[?(@.name=="authz")].imageID}{"\n"}{end}'
# Whether the DaemonSet rolled: its pods' revision hash and age.
kubectl -n aether-system get pods -l app.kubernetes.io/component=proxy \
  -L controller-revision-hash --sort-by=.metadata.creationTimestamp

Chart 2.4.24: the seeded MeshConfig is a hook; rolling back to a release's first revision (#1471)

The chart seeds the default MeshConfig once and never renders it again, so that it is yours to edit. Until 2.4.24 the seed was an object of the release: in the manifest of the one revision that created it, in no later manifest, and still live. That revision is normally the first; it is a later one when an upgrade found the MeshConfig absent and seeded it then (meshConfig.createDefault or edge.enabled turned on later, a deleted object). Helm cannot roll back to such a revision once a later one exists:

$ helm rollback aether 1 -n aether-system
Error: no MeshConfig with the name "default" found

For an object that is live and in the target manifest, Helm takes the copy in the current manifest as its starting point, and there is none. The rollback is recorded as a new revision, and that revision is failed; the objects Helm reached before the MeshConfig are already rolled back. No revision is deployed afterwards (#1582): the revision that was current before the rollback is marked superseded. helm history aether -n aether-system shows the new revision as failed and every revision below it as superseded, and helm status aether -n aether-system prints STATUS: failed at the new revision number.

A failed upgrade reads differently (the seed-hook case further down): there the previous revision stays deployed beside the failed one. So after a failed rollback, do not look for the deployed revision to go back to: there is none. The revision to return to is the one just below the failed one. A rollback to any revision whose stored manifest holds no MeshConfig works, that one included.

Since 2.4.24 the seed is a Helm hook (helm.sh/hook: pre-install,pre-upgrade), which is in no manifest. Upgrading to 2.4.24 rolls no workload and does not touch the MeshConfig: it is live, so it is not rendered, as before (Helm still patches the helm.sh/chart label on the objects it renders, as on every chart upgrade). What changes:

• A release first installed with 2.4.24 or later can be rolled back to any revision, the first included. The MeshConfig is not touched by a rollback: it keeps its UID and your edits.
• A release that an older chart seeded keeps that revision as it was stored, so it still cannot be a rollback target: any revision whose stored manifest holds a MeshConfig, usually revision 1 and no other. Every other revision can be rolled back to, across the upgrade to 2.4.24 in both directions. To see which revisions are affected (the count is not 0 for a revision that cannot be rolled back to):

for rev in $(helm history aether -n aether-system -o json | jq -r '.[].revision'); do
  echo "revision $rev: $(helm get manifest aether -n aether-system --revision "$rev" | grep -c '^kind: MeshConfig')"
done

To get what such a revision ran without a rollback, upgrade to that revision's chart with that revision's values (never --reuse-values). The MeshConfig is live, so an upgrade does not render it and nothing stands in the way. With REV the affected revision:

REV=1
helm history aether -n aether-system            # the CHART column of revision $REV
helm get values aether -n aether-system --revision "$REV" -o yaml > "revision-$REV-values.yaml"
helm upgrade aether oci://quay.io/aethermesh/chart-aether --version <that revision's chart version> \
  --namespace aether-system -f "revision-$REV-values.yaml"

The same command recovers a release that a rollback to such a revision left failed; so does helm rollback aether <the revision that was current before the failed rollback>. helm history shows that revision as superseded, not deployed: it is the one just below the failed one. - One case still seeds the MeshConfig as a release object, because a pre-install hook runs before any object of the release exists and so cannot create a MeshConfig in a namespace the same revision creates: the control plane's with namespace.create=true (the release stored in another namespace). Under Helm 3 that one revision cannot be rolled back to once a later one exists; the check and the way round above apply to it. The edge's MeshConfig on the revision that creates the edge namespace (edge.enabled=true with edge.namespaceCreate=true, its default) was the second such case in 2.4.24 and is not one since 2.4.25: see "Chart 2.4.25" below, also for an edge namespace that 2.4.24 or an older chart created. - For a MeshConfig seeded by the hook, helm get manifest does not list it and helm get hooks does, on the revision that seeded it; the object carries no meta.helm.sh/release-* annotations. One that an older chart seeded keeps the annotations it has. helm uninstall leaves either behind. - The seed hook never deletes a MeshConfig. The chart decides to render the seed when it finds no MeshConfig, and Helm creates it later in the same upgrade, as a pre-upgrade hook. Helm's default for a hook is to delete the live object of that name first; a MeshConfig created in between (by you, by a GitOps tool, by a second helm upgrade) would be deleted and replaced by the seed, with the upgrade reporting success. The hook therefore carries helm.sh/hook-delete-policy: never. never is not one of Helm's policies: naming any value turns the default off, and Helm acts only on the three it knows, so nothing is deleted (hook-failed, the documented value, can still delete your object under Helm 4's server-side apply). What you see instead, when a MeshConfig appears during an upgrade that is seeding one:

Error: UPGRADE FAILED: pre-upgrade hooks failed: warning: Hook pre-upgrade aether/templates/controller-meshconfig.yaml failed: 1 error occurred:
  * meshconfigs.config.aether.io "default" already exists

The MeshConfig is the one that was created, untouched. The release is failed at a new revision and the previous revision is still deployed; pre-upgrade hooks run before Helm updates anything, so no workload changed. Run the same helm upgrade again: the MeshConfig is live now, the seed is not rendered, and the upgrade goes through. With Helm 4 and a release it applies server side, the first upgrade goes through instead: the MeshConfig is not deleted (same UID) and keeps every field it set, but the seed is applied to it: it gains the chart's labels, the hook annotations and any meshConfig.proxy field of your values that the object does not set itself (measured with Helm 4.2.0: a seed with tracingEnabled: true over an object that set only accessLogsEnabled: true left both set). With the default, empty meshConfig.proxy the spec does not change. If the seed and the object set the same field to different values, the upgrade fails on Apply failed with 1 conflict and the object is untouched (run it again). So with Helm 4, a non-empty meshConfig.proxy and a MeshConfig that something else may create during an upgrade, compare the spec as well as the UID before and after, or set meshConfig.createDefault=false and own the object yourself:

kubectl -n aether-system get meshconfig default -o jsonpath='{.metadata.uid} {.spec}{"\n"}'

- helm install --no-hooks seeds no MeshConfig, and the agent pods then stay in ContainerCreating (they mount the ConfigMap the controller projects from it). Run a helm upgrade without --no-hooks, or apply a MeshConfig yourself. - A tool that renders with helm template and applies the result has no cluster to look the MeshConfig up in: it renders the seed on every sync (as it did before), now with the hook annotation on it. Set meshConfig.createDefault=false there and keep the MeshConfig with your other manifests. A controller that drives Helm itself (Flux's helm-controller) is not affected.

After an upgrade to 2.4.24, check that the MeshConfig was left alone (the same UID and generation as before the upgrade) and that nothing rolled:

kubectl -n aether-system get meshconfig default \
  -o jsonpath='{.metadata.uid} generation={.metadata.generation}{"\n"}'
kubectl -n aether-system get ds,deploy \
  -o custom-columns=KIND:.kind,NAME:.metadata.name,GENERATION:.metadata.generation

Chart 2.4.25: the edge's MeshConfig is in every revision; a kept Namespace and helm rollback (#1514, #1515)

Both items are about Helm 3. For an object that is live and in the manifest it rolls back to, Helm 3 starts from the copy in the current manifest and fails when there is none (no <Kind> with the name "<name>" found; measured with 3.18.4). Helm 4 starts from the live object instead (resource exists on cluster but not in original release, using cluster state as baseline) and rolls back; measured with 4.2.0 on both cases below.

The edge's MeshConfig (#1514). With edge.enabled=true and edge.namespaceCreate=true (its default) the chart creates the edge namespace and the default MeshConfig in it on the same revision. A hook cannot do that (it runs before the namespace exists), so that MeshConfig is an object of the release. Up to 2.4.24 it was seeded once: in that revision's manifest, in no later one, and still live, so under Helm 3 helm rollback to that revision failed like #1471:

$ helm rollback aether 1 -n aether-system
Error: no MeshConfig with the name "default" found

Since 2.4.25 the chart renders it on every revision, for a release whose edge namespace it created: the namespace and the MeshConfig both carry aether.io/edge-meshconfig-in-manifest: "true", and either one is enough afterwards. The chart sets no field of that MeshConfig's spec (spec: {} whatever the values), so rendering it again changes nothing in it: your edits survive every upgrade and every rollback (measured with Helm 3.18.4 and Helm 4.2.0: the same UID and the edited field after an upgrade and a rollback in each direction). helm get manifest lists it on every revision.

• Upgrading from 2.4.24 to 2.4.25 rolls no workload and changes no MeshConfig (measured: every DaemonSet and Deployment keeps its generation, both MeshConfigs keep their UID, generation and spec, and the edge namespace gets no new annotation). What Helm does patch, as on every chart upgrade, is the helm.sh/chart label on the objects it renders; pod templates do not carry it. An edge namespace that 2.4.24 or an older chart created is not marked, and the chart treats it as before: the MeshConfig is live, so it is not rendered. The revision that created that namespace keeps the MeshConfig in its stored manifest and still cannot be a Helm 3 rollback target; the check and the way round of "Chart 2.4.24" apply (an upgrade to that revision's chart with that revision's values).
• Do not add the annotation to such a namespace or MeshConfig by hand. The next upgrade would bring the live MeshConfig back into the manifest, and that revision could then not be rolled back to from an older one: the same failure, one revision later.
• Back through 2.4.24 (an upgrade to the 2.4.24 chart, or a rollback to one of its revisions, on a release whose edge namespace 2.4.25 created). Chart 2.4.24 renders the edge Namespace without the annotation, so Helm takes it off the namespace, and it does not render the live MeshConfig, which leaves the manifest and stays in the cluster with its annotations. The next upgrade to 2.4.25 or later reads the marker from the MeshConfig, renders it again and marks the namespace again, and rollbacks to the 2.4.25 revisions work from then on (measured with Helm 3.18.4 and Helm 4.2.0; e2e leg viii-b). What does not work under Helm 3 is a helm rollback straight from a 2.4.24 revision to a 2.4.25 one: the 2.4.24 manifest holds no MeshConfig, and nothing in a chart can change a stored manifest. Use helm upgrade to go forward from 2.4.24. The same holds from a rollback copy of a 2.4.24 revision.
• Which form a release has:

kubectl get namespace aether-ingress \
  -o jsonpath='{.metadata.annotations.aether\.io/edge-meshconfig-in-manifest}{"\n"}'
kubectl -n aether-ingress get meshconfig default \
  -o jsonpath='{.metadata.annotations.aether\.io/edge-meshconfig-in-manifest}{"\n"}'
# "true" on either: the edge's MeshConfig is rendered on every revision of
# this chart. Both empty: seeded once (the namespace is older than 2.4.25,
# or was created with meshConfig.createDefault=false).

- With meshConfig.createDefault=false the chart renders no MeshConfig and the namespace is created unmarked. If a marked release turns the seed off, the MeshConfig leaves the manifest and stays in the cluster (helm.sh/resource-policy: keep); under Helm 3 the revisions before that one can then not be rolled back to, as in the next item. - helm uninstall, and an upgrade with edge.enabled=false, delete the edge namespace as before, and the MeshConfig with it. - The control plane's MeshConfig with namespace.create=true is not changed: its seed carries meshConfig.proxy, and a spec that is rendered again is patched over yours. It stays the one exception of "Chart 2.4.24".

After an upgrade to 2.4.25 with the edge on, check that the edge's MeshConfig was left alone and that the edge did not roll:

kubectl -n aether-ingress get meshconfig default \
  -o jsonpath='{.metadata.uid} generation={.metadata.generation}{"\n"}'
kubectl -n aether-ingress get deploy aether-edge \
  -o custom-columns=NAME:.metadata.name,GENERATION:.metadata.generation

A Namespace that left the manifest (#1515). The chart keeps rendering a Namespace the release owns, so that Helm never deletes it (#1403). It stops when the release no longer owns it: the ownership annotations (meta.helm.sh/release-name, meta.helm.sh/release-namespace) were removed from the live Namespace, or a first install failed and was upgraded after the keep command of "Chart 2.4.21". The Namespace stays (helm.sh/resource-policy: keep), and from then on a revision whose stored manifest holds it cannot be a Helm 3 rollback target. Measured on kind, Helm 3.18.4, revision 2 holding the Namespace and revision 3 not:

$ helm rollback aether 2 -n aether-system
Error: no Namespace with the name "aether-system" found

The namespace is untouched (same UID, Active), and the release is failed at a new revision, with the revision that was current now superseded and none deployed (see "Chart 2.4.24"). A rollback to a revision without the Namespace works and puts the release back to deployed, and so does Helm 4.2.0 on the same steps. Nothing in a chart can change this: the manifests are already stored, and Helm refuses a Namespace the release does not own if the chart renders it again. The check and the way round are those of "Chart 2.4.24", with Namespace in place of MeshConfig:

for rev in $(helm history aether -n aether-system -o json | jq -r '.[].revision'); do
  echo "revision $rev: $(helm get manifest aether -n aether-system --revision "$rev" | grep -c '^kind: Namespace')"
done
# To get what an affected revision ran (also after a failed rollback):
REV=2
helm get values aether -n aether-system --revision "$REV" -o yaml > "revision-$REV-values.yaml"
helm upgrade aether oci://quay.io/aethermesh/chart-aether --version <that revision's chart version> \
  --namespace aether-system -f "revision-$REV-values.yaml"

Chart 2.4.26: the node proxy's Envoy stats name their service (#1561)

The node proxy's Envoy stats (envoy_*, aether_requests_total) are exported with the resource attribute service.name=aether-proxy. They had no service.name. Where the backend maps service.name to job, their series gain job="aether-proxy": Prometheus's OTLP ingestion does that by itself, another backend may need the mapping made (observability/metric-labels.md, "What you must do"). Everything below about job holds for such a backend; where nothing maps the attribute, the series keep the labels they had.

What was measured and what was not. Measured: the renders of 2.4.25 and 2.4.26; the pinned Envoy exporting that service.name with the chart's bootstrap block (a test); the queries below against synthetic series, with promtool. Not measured: the upgrade itself on a cluster. What the supervisor does with a changed bootstrap is taken from "Which workloads a chart upgrade rolls" above.

What the upgrade that crosses 2.4.26 changes in the render:

Object	What changes	Effect
aether-proxy-config ConfigMap, only when otel.endpoint is set	its envoy.yaml: the Envoy stats sink gains a second resource detector that sets service.name: aether-proxy	no pod is replaced; the supervisor watches the mounted file and hot-restarts Envoy in place, on every node. See below
every Deployment and DaemonSet	nothing in a pod template	not rolled by the chart

With otel.endpoint empty the ConfigMap's envoy.yaml is what 2.4.25 rendered (the object's helm.sh/chart label moves with the version, as on every object, and is not mounted), so no Envoy restarts for it.

"Not rolled by the chart" is about the chart: a release built from a newer commit can carry a new image digest for a workload, and that rolls it, as on any upgrade.

The hot restart. A changed envoy.yaml is picked up as "Which workloads a chart upgrade rolls" describes for this ConfigMap: each node's supervisor starts a new Envoy epoch when that node's kubelet delivers the file, on every node independently, not one node at a time, so the nodes can overlap, unlike a DaemonSet roll. For the length of each restart two Envoy processes run on the node: check the headroom of "Sizing nodes for a proxy hot restart" before the upgrade, and watch the upgrade as you would a proxy roll. A rollback to 2.4.25 changes envoy.yaml back, so expect the same again.

What changes in the stored metrics, where service.name becomes job:

• job="aether-proxy" appears on the node proxy's Envoy series, from each node's restart on. A label that appears makes each of them a new series; the old one receives no more samples.
• For five minutes the old and the new series are both read. A pushed series has no staleness marker, so the last sample of an old (job-less) series stays in the 5-minute lookback beside the new one. An aggregation that does not keep job counts both. With promtool: sum by (node, aether_cluster) (envoy_cluster_membership_healthy) over 4 hosts read 8 two minutes after the restart, and 4 again once the five minutes had passed. The same arithmetic applies to a sum over any gauge these proxies export. An alert of your own with a threshold on such a sum can fire in that window.
• Selectors, with promtool:
• {job="aether-proxy"} reads the new series only and does not double.
• {job!="aether-edge-proxy"} selects the node proxy's series before and after; inside the five minutes it reads the old and the new one.
• {job=""}, a selector that relied on the label being absent, matches nothing once the old series have left the lookback. Change it to {job="aether-proxy"}, or to {job=~"|aether-proxy"} while both chart versions are stored.
• Counters. Not measured. A rate() or increase() whose window spans the restart is computed per series, over one that ends and one that starts, so a sum(rate()) can read low around the restart.
• The supervisor's name. The proxy container's environment is as it was (its pod template renders unchanged), and that environment is all the supervisor's resource is built from, so aether_supervisor_* should stay job="aether-proxy-supervisor". The last query below checks it.
• No rule in docs/observability/ selects on the node proxy's Envoy series.

The label changes once, so this happens at the upgrade (and at a rollback across it), not at each later proxy roll.

Before and after (never --reuse-values; read the values back and pass them with -f):

# Which pods were replaced. The chart replaces none; a pod with a new name and
# age was rolled for another reason (a new image digest).
kubectl -n aether-system get pods -L controller-revision-hash,pod-template-hash \
  --sort-by=.metadata.creationTimestamp
# The bootstrap the chart rendered: the static detector is there with
# otel.endpoint set, and absent without it.
kubectl -n aether-system get configmap aether-proxy-config \
  -o jsonpath='{.data.envoy\.yaml}' | grep -A4 'resource_detectors.static_config'

# With service.name mapped to job: one result per proxy kind (aether-proxy,
# and aether-edge-proxy with the edge on). A result with no job is a node
# whose Envoy has not restarted on the new bootstrap yet, or, for five minutes
# after it did, the last sample of its old series.
count by (job) (envoy_server_live)
# Each node's Envoy epoch. Compare with the value before the upgrade: a node
# whose epoch did not move has not picked the bootstrap up.
max by (node) (envoy_server_hot_restart_epoch{job="aether-proxy"})
# The supervisor's series: the same job as before the upgrade.
count by (job) ({__name__=~"aether_supervisor_.+"})

Chart 2.4.27: one release per cluster, the SPIRE-served webhook's identity, narrower webhook RBAC (#1540, #1457, #1456)

What an upgrade changes. No object is renamed. No pod template changes, so this chart change rolls no workload by itself (a release built from another commit carries other image digests, and those roll the workloads as always: "Which workloads a chart upgrade rolls" above). The objects that change are the controller's ClusterRole, and only with controller.webhook.spire=true, and the helm.sh/chart label on each object's own metadata.

Two renders that were accepted and are now refused:

Refused when	Message starts with	Do
controller.webhook.spire=true, controller.webhook.clusterSpiffeID.create=true (the default), controller.webhook.clusterSpiffeID.className empty, and the cluster serves the spire.spiffe.io/v1alpha1 ClusterSPIFFEID API	controller.webhook.spire=true with controller.webhook.clusterSpiffeID.create=true, but controller.webhook.clusterSpiffeID.className is empty	Set className to your spire-controller-manager class; or, if the controller's identity is registered by a ClusterSPIFFEID or an entry of your own, set controller.webhook.clusterSpiffeID.create=false. Nothing else changes with either.
The aether-proxy DaemonSet in the release's namespace, or the csi.aether.io CSIDriver, exists and names another Helm release	... already exists and belongs to another aether release	Upgrade that release instead of installing a second one, or uninstall it first (docs/configuration.md, "One release per cluster"). An upgrade of the release that owns them is not affected. A first install is also refused when one of them names a release of the same name by its app.kubernetes.io/instance label alone (no meta.helm.sh/release-namespace): if it is a leftover of a release that is gone, delete it and install again.

The first reads what the cluster serves (.Capabilities.APIVersions), the second reads the two live objects (lookup). A render without a cluster (helm template, a client-side dry run, a GitOps tool that renders with helm template) is refused by neither, unless it passes --api-versions spire.spiffe.io/v1alpha1/ClusterSPIFFEID, which arms the first. In the first state on a cluster that does not serve the API, the install notes carry a WEBHOOK IDENTITY NOTE.

To see before upgrading whether the first applies to a release:

helm get values <release> -n <ns> -a -o json |
  jq '.controller.webhook | {spire, create: .clusterSpiffeID.create, className: .clusterSpiffeID.className}'
kubectl api-resources --api-group=spire.spiffe.io -o name   # clusterspiffeids...: the API is served

The controller's webhook-configuration grant is get and update (it was get, list, watch, update, patch), with controller.webhook.spire=true only. The caBundle injector has read each configuration by name, uncached, and written it with an update since #1453, which every chart from 2.4.19 ships; its test runs it with list, watch and patch refused. A controller image older than that keeps an informer on both kinds and stops injecting when list and watch are refused (#1431): during the upgrade from a chart older than 2.4.19 that is the old controller pods, until the Deployment has rolled; do not pin an older controller image under this chart. After the upgrade, with the SPIRE-served webhook:

NS=aether-system
SA=$(kubectl -n "$NS" get deploy -l app.kubernetes.io/name=aether-controller \
  -o jsonpath='{.items[0].spec.template.spec.serviceAccountName}')
for kind in validatingwebhookconfigurations mutatingwebhookconfigurations; do
  for verb in get update list watch patch; do   # expect: yes yes no no no
    printf '%s %s: ' "$kind" "$verb"
    kubectl auth can-i "$verb" "$kind.admissionregistration.k8s.io" \
      --as="system:serviceaccount:$NS:$SA"
  done
done

and the injector keeps working: no webhook caBundle injection failed line from the controller's leader, and this stays 0 (the caBundle length check of "The pod-mutating webhook's caBundle is empty" shows the result):

sum by (kind, reason) (increase(aether_controller_webhook_cabundle_injection_failures_total[30m]))

None of this was run on a cluster for this change: the grant is tested as a render and the injector against a fake API server (TestCABundleInjectorNeedsOnlyGetAndUpdate).

Chart 2.4.28: the agent DaemonSet says which external-harness contract the chart satisfies (#1544)

What an upgrade changes. One annotation appears on the agent DaemonSet's own metadata, aether.io/harness-contract-version, holding the version of test/harnesscontract/external-harness.yaml the chart was packaged with ("The external-harness contract" above has what it means and what it does not). No object is renamed and no pod template changes, so this chart change rolls no workload by itself (a release built from another commit carries other image digests, and those roll the workloads as always: "Which workloads a chart upgrade rolls" above). Besides the annotation, the helm.sh/chart label on each object's own metadata changes, as with every chart version.

There is nothing to set and nothing to do before the upgrade. Afterwards:

NS=aether-system
# The contract the chart satisfies. Empty: a chart older than 2.4.28, or one
# rendered from the source tree.
kubectl -n "$NS" get daemonset aether-agent \
  -o jsonpath='{.metadata.annotations.aether\.io/harness-contract-version}{"\n"}'
# The annotation did not reach the pods, and adding it replaced none: the
# DaemonSet's pod template has no annotation of that name...
kubectl -n "$NS" get daemonset aether-agent \
  -o jsonpath='{.spec.template.metadata.annotations.aether\.io/harness-contract-version}{"\n"}'
# ...and the agent pods' controller-revision-hash is what it was before the
# upgrade, when the upgrade changed nothing else of the agent (the same image
# digests and values). Read it before and after.
kubectl -n "$NS" get pods -l app.kubernetes.io/component=agent \
  -L controller-revision-hash

A later bump of the contract's version changes that one annotation at the next chart upgrade and nothing else of the render.

None of this was run on a cluster for this change. What was measured is the render: the packages built from this change and from the commit before it have byte-identical pod templates for every workload, the render differs by the annotation and the helm.sh/chart labels only, and the two jsonpath expressions above, run over the rendered DaemonSet with kubectl's own printer and no cluster, print the version and an empty line.

Chart 2.5.0: per-pod listener, cluster and stat names carry the pod's namespace (#1584)

What changed. The node agent named everything it builds for one pod after the pod's name alone. Two mesh pods of one name in two namespaces on one node (two StatefulSets called web, each with a web-0) then shared every one of those names. The control plane library keeps one resource per name, and which pod's it kept was decided by map order on every snapshot build, separately for listeners and for clusters: one pod had no listeners, and the other's inbound listener could route to an app cluster that dials the first pod's loopback. Every per-pod name is now built from <namespace>_<pod> (an underscore is legal in neither part, so it cannot be ambiguous):

Before	Since 2.5.0
inbound_<pod>, inbound_<pod>_h3	inbound_<namespace>_<pod>, inbound_<namespace>_<pod>_h3
outbound_http_<pod>	outbound_http_<namespace>_<pod>
capture_<pod>, capture_udp_<pod>	capture_<namespace>_<pod>, capture_udp_<namespace>_<pod>
app_<pod>_<port>	app_<namespace>_<pod>_<port>
health_<pod>, inboundready_<pod>	health_<namespace>_<pod>, inboundready_<namespace>_<pod>
/healthz/health_<pod>, /healthz/inboundready_<pod> (health gateway)	/healthz/health_<namespace>_<pod>, /healthz/inboundready_<namespace>_<pod>
stat prefixes out_http_<pod>, in_tcp_<pod>[_<port>]; filter chains in_<pod>…, in_tcp_<pod>…, in_h3_<pod>…	the same with <namespace>_<pod>

There is no flag and no way to keep the old names.

Metric labels. The proxy bootstrap extracts two tags from the listener stat prefixes where it extracted one:

• envoy_listener_inbound_* and envoy_listener_out_http_* gain the label aether_namespace. aether_pod keeps its value, the pod name (<pod>_h3 for the HTTP/3 inbound listener, as before). A query or an alert that matches on aether_pod alone now covers a same-named pod of every namespace; add aether_namespace where that matters.
• aether_cluster of the per-pod clusters is the new cluster name (health_<namespace>_<pod>, inboundready_<namespace>_<pod>). A selector such as aether_cluster=~"inboundready_.*" is unaffected; one that spells a pod (aether_cluster="health_web-0") has to be rewritten.
• The TCP floor's stat prefix is part of the metric NAME, and that name now holds the namespace: envoy_tcp_in_tcp_<namespace>_<pod>_…. The queries under "Cross-pod L4 landings" below are spelled for it. Anything outside this repository that excludes a workload by envoy_tcp_in_tcp_<pod>_ matches nothing after the upgrade.
• The stats exclusions of the bootstrap are unchanged and still apply to the renamed probe clusters; the membership_* gauges the health gateway reads stay allocated. //agent/test/envoy_validate (TestChartStatsConfigOnNamespacedPodNames) runs the pinned proxy with the chart's stats_config and checks both.

While a node is between versions the labels are these (the agent and the proxy roll separately): a 2.5.0 proxy under an older agent exports aether_pod="<pod>" with no aether_namespace (and, for the HTTP/3 inbound listener only, aether_namespace="<pod>", aether_pod="h3"); an older proxy under a 2.5.0 agent exports aether_pod="<namespace>_<pod>". Neither puts a pod into a metric name. The first of the two was measured on the pinned proxy, the second follows from the old extractor.

The first of those two exists only because the namespace part of the aether.pod extractor is optional. That tolerance is for this minor: chart 2.6.0 removes it (the extractor then requires the namespace). An upgrade to 2.6.0 or later therefore starts from a 2.5.x chart whose agents have all rolled; from an older chart, upgrade to 2.5.x first. Skipping that step breaks no traffic. It leaves one metric family per pod in the metrics store's name index for as long as the store keeps names, which is what the tolerance prevents. The compatibility window is defined in charts/README.md, "Compatibility window".

What an upgrade does on a node. When the node's agent is replaced, the proxy's stream reconnects and the first snapshot removes every per-pod listener and cluster under its old name and adds it under the new one, at the same address in the same network namespace. Measured on the pinned proxy (//agent/test/mtlspool, TestListenerRenamedAtTheSameAddress; a pod is a loopback port there, not a network namespace):

• The proxy accepts the update. Nothing is rejected, and it ends with one listener per name at the same address.
• New connections are not refused. Of about 1,000 connections opened across the swap, none was refused; in some runs one request got a 503 (it met the old inbound listener after its app cluster was gone and before the listener was replaced).
• Connections established before the swap are closed by the proxy at the end of its drain time (proxy.hotRestart.drainTime, 10 s by default; the test used 3 s and saw the close at 3.1 s). Until then:
• on the outbound and capture listeners they are served as before (the mesh clusters they route to keep their names). An application's keep-alive connections to the mesh, and every captured TCP connection, end at that point; a request in flight at that moment is cut. Not measured: what an in-flight request sees.
• on the inbound listener every request gets 503: the listener that accepted the connection is draining and routes to app_<namespace>_<pod>_<port>, which no longer exists. A calling proxy retries that reply on another endpoint (two retries, another host each time; since #1641 because the destination reports the application cluster gone in x-aether-outcome, not because the status is 503), so a service with a replica on a node that is not at the same point of the roll is covered; a service whose only endpoints are on nodes whose agents are replaced at the same moment answers 503 to callers with warm connections for up to the drain time.

To avoid the second case, replace the agents node by node (the DaemonSet's default) and give single-replica workloads a second replica for the upgrade, or drain a node before its agent is replaced. A rollback to an older chart renames everything back and costs the same once more.

After the upgrade.

NS=aether-system
# Every per-pod listener carries a namespace: no name without two parts after
# the prefix. (Run in one aether-proxy pod per node.)
kubectl -n "$NS" exec <proxy-pod> -c aether-proxy -- \
  curl -s 'http://127.0.0.1:9901/listeners' | grep -E '^(inbound|outbound_http|capture)_'

# The agent found two different resources under one name in a snapshot it
# built. Zero where nothing shares a name; the one known input is below.
sum by (node) (increase(aether_agent_snapshot_duplicate_resource_names_total[1h])) > 0

That counter is the check that says so if a name is ever shared again. The agent logs, at ERROR and once per change of the set, more than one resource of a type carries the same name: the proxy is sent only one of them, and which one can change from build to build with the xDS type and the names, and counts one per name on every build the condition lasts. It reports and changes nothing: the snapshot is published as before. Two resources of one name that are equal are not reported. The check is about names of any kind, and one input is known to produce it today:

• two sandboxes of the same pod (same namespace and name) stored at once, which is a replacement whose predecessor's CNI DEL was missed while the old network namespace still exists ("A pod is stuck Terminating, or a node has stale netns entries" below is the way out). The line names listeners and clusters.

A line that names the endpoint type and a service (demo/db) was a second input until #1635, and is not expected from an agent that carries that fix. A service's HTTP cluster, its TCP floor and its UDP floor share one load assignment, named after the service, and exactly one of the service's entries publishes it: the one built from a listing that still has endpoints, HTTP before TCP before UDP. When one listing goes while another stays (a workload re-annotated from http to tcp, say), the cluster of the listing that went is retained for the service retention grace (90 s by default) and resolves to the endpoints of the listing that stayed. Before the fix the retained entry published an empty load assignment under the same name: the service's published endpoints changed between none and the live ones from build to build, with an EDS push each time, and its TCP floor and UDP clusters had no endpoints at all for those 90 s. A service listed under TCP and UDP at once, with different pods under each, did the same for as long as it was listed under both.

None of this was run on a cluster for this change. What was measured is named above; the labels, the swap and the two-pod case were each run against the pinned proxy binary, and the agent side against its own snapshot cache.

Chart 2.5.2: a pod whose name contains a dot (#1636)

Who this is about. A pod name may contain dots (it is an RFC 1123 subdomain). No workload controller generates one; a pod created directly, or by an operator that names its pods, can have one. List them:

kubectl get pods -A -o json \
  | jq -r '.items[] | select(.metadata.name | contains(".")) | "\(.metadata.namespace)/\(.metadata.name)"'

If that prints nothing, this release changes nothing on the cluster's proxies: the listeners and clusters of a pod without a dot in its name are what they were (//agent/internal/xds/proxy, TestPodWithoutADotKeepsItsStatNames).

What was wrong. A dot is the separator of an Envoy stat name, and the agent put the pod name into its per-pod stat names as it is. For a pod web.external-7d9f in team-a:

• cluster.health_team-a_web.external-7d9f.membership_healthy reads, to the bootstrap's stats exclusions, as cluster health_team-a_web and stat external-7d9f.membership_healthy, which the exclusion of the probe clusters' dead external.* subtree matches. The gauge was never allocated, the health gateway reads it, and the gateway answered 503 for the pod for as long as it lived: its endpoint was never promoted. The same for a pod whose second name label starts with http1, http2, retry or assignment (the exclusion's other alternatives contain an underscore, which no pod name does). Measured on the pinned proxy, the gateway included (//agent/test/envoy_validate, TestHealthGatewayAcrossTheStatNameChange keeps the old agent's cluster for web.external-0 and reads 503 from it).
• for every dotted pod, the part of the name after the first dot stayed in the metric NAME (envoy_listener_inbound_external_7d9f_downstream_cx_total, envoy_cluster_external_7d9f_membership_healthy): one set of metric families per pod, with aether_pod="web" and aether_cluster="health_team-a_web".

What changed. In a STAT name the agent now writes ~ for each dot of the pod name. ~ is legal in neither a namespace nor a pod name, so two pods never share a stat name and the pod name can be read back.

	Before	Since 2.5.2
listener stat prefixes	inbound_team-a_web.external-7d9f (and …_h3, out_http_…, capture_…, capture_udp_…)	inbound_team-a_web~external-7d9f (and the same four)
TCP floor stat prefix	in_tcp_team-a_web.external-7d9f[_<port>]	in_tcp_team-a_web~external-7d9f[_<port>]
probe clusters' stats	keyed by the cluster name	keyed by alt_stat_name: health_team-a_web~external-7d9f, inboundready_team-a_web~external-7d9f

No resource is renamed. Listener, cluster and filter chain names, and the health gateway paths (/healthz/health_team-a_web.external-7d9f), carry the pod's own name, dots included, as they have since 2.5.0. The bootstrap is byte-identical to 2.5.1's: its tag extractors and exclusions were already right for a stat name without a dot in the pod part, and //charts/aether:aether_proxy_bootstrap_probe_cluster_exclusions_test now pins the two exclusion patterns.

Metric labels, for a dotted pod only.

• aether_pod is the pod name with ~ for each dot: web~external-7d9f (web~external-7d9f_h3 for the HTTP/3 inbound listener). It was web. A regex tag extractor takes a piece of the stat name and cannot rewrite it, so the label cannot be the pod's own name; to join it with a label that is (pod from kube-state-metrics, destination_pod of aether_requests_total):

label_replace(envoy_listener_inbound_downstream_cx_active, "pod", "$1.$2", "aether_pod", "([^~]*)~([^~_]*)(?:_h3)?")

(one label_replace per dot; a name with two dots needs a second). - aether_cluster of its probe clusters is health_team-a_web~external-7d9f and inboundready_team-a_web~external-7d9f. It was health_team-a_web. - The metric families that carried the rest of its name stop receiving samples. The metrics store keeps their names until its retention drops them. - The TCP floor's stat prefix is part of the metric name, where both . and ~ become _: envoy_tcp_in_tcp_team_a_web_external_7d9f_… before and after.

What an upgrade does on a node. The proxy DaemonSet does not roll and the bootstrap ConfigMap does not change, so no proxy hot-restarts. The agent DaemonSet rolls (its image changed). When a node's agent is replaced:

• for a pod without a dot in its name, the first snapshot of the new agent holds the same listeners and clusters, and the proxy changes nothing;
• for a dotted pod, its five listeners differ in stat_prefix and its two probe clusters in alt_stat_name, so the proxy replaces them under the same names.
• The probe clusters, measured on the pinned proxy (TestHealthGatewayAcrossTheStatNameChange: the old agent's clusters replaced by this agent's, the gateway requested every millisecond or so throughout): the path of a healthy pod answered 200 at every request. The proxy warms the replacing cluster, its first health check included, before it takes the old one's place.
• The listeners, not measured for this change. From the 2.5.0 measurement above, which replaced listeners at the same address: new connections are not refused, and connections established before are closed at the end of the proxy's drain time (proxy.hotRestart.drainTime, 10 s). Unlike 2.5.0, the app clusters keep their names, so a draining inbound listener keeps routing until then.
• a dotted pod that was never promoted because of the excluded gauges is promoted once its node's agent is replaced (the same test: 503 from the old agent's cluster, 200 from this agent's, the application up throughout).

While a node's agent is older than this release, its dotted pods are as before: the bootstrap cannot tell where a dotted pod name ends in a stat name. A rollback of the agent puts the dots back.

After the upgrade.

# A pod name in a metric NAME. Each must return nothing once every agent is
# replaced and the old series have gone stale (5 minutes).
count by (__name__) ({__name__=~"envoy_cluster_.+_membership_(healthy|total)"})
count by (__name__) ({__name__=~"envoy_listener_(inbound|out_http)_.+_downstream_cx_total", __name__!~"envoy_listener_(inbound|out_http)_worker_[0-9]+_downstream_cx_total"})

# The dotted pods, as the proxies export them: one row per pod, and 1 from
# the membership gauge of a pod whose application is up.
count by (aether_namespace, aether_pod) (envoy_listener_inbound_downstream_cx_total{aether_pod=~".*~.*"})
envoy_cluster_membership_healthy{aether_cluster=~"health_.*~.*"}

# On one node: the gauges the health gateway reads exist for every pod
# (the count is the number of mesh pods on the node, dotted or not).
kubectl -n aether-system exec <proxy-pod> -c aether-proxy -- \
  curl -s 'http://127.0.0.1:9901/stats?filter=membership_healthy' \
  | grep -c '^cluster\.health_[^.]*\.membership_healthy: '

None of this was run on a cluster. What was measured, on the pinned proxy binary with the chart's stats_config: that ~ is kept as it is in a stat name, a tag value and a label value (: is rewritten to _ by Envoy, and @ already marks the HTTP/3 twin clusters in aether_cluster); the labels and the absence of any per-pod metric family for eighteen pod names chosen against the bootstrap's patterns; the health gateway answering 200 for a dotted pod whose application is up, 503 for one whose application is down, and following a pod whose application comes up; and the gateway across the replacement of the probe clusters.

Chart 2.5.3: the registrar, the controller and the edge control plane report per pod (#1560)

What changed. The registrar, the controller and the agent container of the edge Deployment now set service.instance.id=$(POD_NAME) in OTEL_RESOURCE_ATTRIBUTES (the edge's proxy container already did). Prometheus's OTLP ingestion stores that attribute as the label instance, so each replica of those three is its own series with no pipeline configuration. Before, a pipeline that promoted nothing per replica stored all replicas of a component as ONE series, and AetherRegistrarSnapshotDiverged, which compares the replicas' hashes, could never fire.

The rule files of this release changed with it: every selector in docs/observability/agent-pin-alerts.yml and registrar-alerts.yml is wrapped in last_over_time(...[150s]).

What rolls. The variable is rendered whether or not otel.endpoint is set, so on every install:

Workload	Rolls on the upgrade across 2.5.3
registrar Deployment	yes, once
controller Deployment	yes, once
edge Deployment (edge.enabled)	yes, once: both containers restart, the edge proxy with them
agent, mesh-dns, proxy and uds-csi DaemonSets	no: their pod templates render as in 2.5.2 (compared, with the edge and telemetry on)

There is no value to turn it off. The node proxy's bootstrap is unchanged, so no Envoy restarts on the nodes.

Order of work. Update the rules first. The rule files of this release are correct against a chart that does not set the attribute (they read each series' newest sample of the last 150 s, and these components export every 60 s). Rule files of an earlier release are NOT correct against this chart: with the pod name in a label, a replaced pod's last sample is read for five minutes beside its replacement's, and (measured with promtool test rules, an old pod's series ending at minute 9 and its replacement's starting at minute 10):

• AetherMeshClusterUnpinned fired at minutes 12 and 13 from ONE last sample of a replaced edge pod, a critical alert for a pod that no longer exists;
• AetherRegistrarSnapshotDiverged fired at minutes 12 and 13 when a replaced registrar's last export fell between an agent RPC and its sync, and a replaced registrar's last non-zero aether_registrar_writebehind_queue_depth kept a real divergence quiet until minute 13.

The chart cannot check which rule files you run. So:

1. Install the rule files of this release (docs/observability/README.md, "Installing"). Nothing changes for a chart at 2.5.2: an alert that is firing stays firing, and one whose pod or agent stopped reporting now ends 2.5 minutes after its last export instead of 5.
2. Upgrade the chart, reading your values back and passing them with -f (never --reuse-values).
3. Run the checks below.

If your pipeline already made a per-pod label for these components (the pod label earlier versions of metric-labels.md recommended), the false alerts above were possible after every roll already; the new rule files end that. The pod promotion is no longer needed for the rules.

What changes in the stored metrics, on the registrar's, the controller's and the edge control plane's series only:

• instance="<pod name>" appears, from each pod's first export on. A label that appears makes a new series: the series without instance receive no more samples and are returned by a bare selector for five more minutes.
• A panel that does not aggregate shows one line per pod where it showed one line per component (or per node, for the edge), and at every later roll the old pods' lines end and new ones start. For one line per component, aggregate the label away: max without (instance) (...) for a gauge that is the same on every replica, sum without (instance) (rate(...[5m])) for a counter.
• A panel or rule of your own that aggregates a gauge over these series (sum, count, max, a comparison between replicas) counts a replaced pod for five minutes after a roll unless its selector is wrapped the way the shipped rules' are: last_over_time(<selector>[150s]).
• Counters. rate() and increase() are computed per series. Across this upgrade the series without instance end and the ones with it begin, so a sum(rate()) can read low for the length of its window around the roll. Not measured.
• Selectors. A selector that names instance on these components did not match before. One that relies on its absence ({instance=""}) stops matching.
• Cardinality. One set of series per pod instead of one per component, and a new set at every roll: with the default two replicas each, a handful of series sets. The per-node DaemonSets are unchanged.
• Do not drop the label. A Collector processor or relabelling rule that removes service.instance.id or instance from these series puts the replicas back into one series.

A rollback to 2.5.2 rolls the same three Deployments again and removes the label from new samples; the new rule files stay correct.

Checks, after the three Deployments have rolled. Nothing here was run on a cluster for this change; the rule behaviour above is from promtool test rules (bazel test //:observability_rules_test), and the rendered attribute from the chart's template tests.

# Newest last: the registrar, controller and edge pods are new and Ready, and
# no agent, proxy, mesh-dns or uds-csi pod was replaced by the upgrade.
kubectl get pods -A --sort-by=.metadata.creationTimestamp | grep aether

# One row per registrar pod, named after it (kubectl get pods). One row with no
# `instance` means the label is dropped on the way to the store.
count by (instance) (last_over_time(aether_registrar_snapshot_content_hash[150s]))

# The value has to equal the number of registrar replicas, also in the five
# minutes after a roll (a bare count() reads the replaced pods as well there).
count(last_over_time(aether_registrar_snapshot_content_hash[150s]))

# With the edge on: one row per edge pod, two rows for a node that runs two.
count by (node, instance) (last_over_time(aether_agent_snapshot_tls_clusters{job="aether-edge", pin="pinned"}[150s]))

# The divergence rule's expression can now see two replicas. MUST be empty.
count by (job, revision) (
  count_values by (job, revision) (
    "hash",
    last_over_time(aether_registrar_snapshot_content_hash[150s])
    * ignoring (revision) group_left (revision)
    count_values without () ("revision", last_over_time(aether_registrar_snapshot_revision[150s]))
  )
) > 1

# No alert of the two rule files fired because of the roll: empty over the
# half hour that follows it.
max_over_time(ALERTS{alertstate="firing", alertname=~"AetherMeshCluster.*|AetherProxyHoldsUnpinnedClusters|AetherProxyPinStateUnknown|AetherRegistrarSnapshotDiverged"}[30m])

Chart 2.5.7: the compatibility window is one chart minor; mesh-DNS reads only the snapshot envelope

The rule, now written down. "Supported for one release" was said in several places and defined in none. It means one minor version of the aether chart: something deprecated in a 2.5.x chart is still there in every 2.5.x and may be removed in 2.6.0, never earlier. charts/README.md, "Compatibility window", has the operator's side (which chart and image skew is supported, and where a deprecation is announced); AGENTS.md, "Rules for agents", has the contributor's. Nothing changes on a cluster because of the rule itself.

What is deprecated today, and the first chart that may be without it:

Deprecated	Since chart	May be removed in	Use instead
The optional namespace group of the proxy bootstrap's aether.pod stats extractor (the tolerance for an agent older than #1584)	2.5.0	2.6.0	nothing to do: see "Chart 2.5.0"
aether_registrar_snapshot_content{content_hash}	2.4.12	any chart from 2.5.0 on; removed in chart 2.5.9 (see "Chart 2.5.9")	aether_registrar_snapshot_content_hash ("Stored vs in place vs applied")
cni-install --otlp-endpoint, --otlp-pin-endpoint (ignored)	2.4.0	any chart from 2.5.0 on; still accepted in this one	nothing: the chart has not passed them since 2.4.0

What was removed: the pre-envelope mesh-DNS snapshot. The node agent writes the mesh-DNS record table to --mesh-dns-snapshot-path, and the resolver (the aether-mesh-dns daemon) reads it, at start and on every change. Since chart 0.87.0 (#586, July 2026) the file is an envelope, {"writtenAt":…,"generation":…,"records":{…}}, and every agent since has written only that. Until this chart the reader also accepted what agents wrote before it: the bare map {"<ns>/<svc>":"<ip>"}. That reader is gone.

A node that still carries a snapshot file written before chart 0.87.0 is not supported. An agent from 0.87.0 on replaces the file with an envelope on its first projection and re-writes it on every heartbeat, so such a file is left only where no such agent has run with mesh-DNS on. What a resolver from this chart does with one:

• it does not serve it, and does not serve an empty table either;
• at start it stays cold: mesh names answer SERVFAIL (clients retry) until the agent writes a snapshot, which it does on its first projection and then on every heartbeat;
• on a reload it keeps the table it is serving;
• each attempt logs failed to read mesh-DNS snapshot; starting cold or failed to reload mesh-DNS snapshot; keeping current records with no records object, and counts aether_mesh_dns_snapshot_reloads_total{result="parse_error"}, which the shipped docs/observability/mesh-dns-alerts.yml rule on result!="success" reads.

A file that is valid JSON without a records object (an empty {} included) is treated the same way. Before, such a file was read as an empty table, and a reload of it answered NXDOMAIN for every mesh name.

Check, after the upgrade.

# 0 on every node. Non-zero: read that node's mesh-dns log for the reason.
sum by (node) (increase(aether_mesh_dns_snapshot_reloads_total{result="parse_error"}[15m]))

What the upgrade rolls. The agent and mesh-dns images changed (both link the snapshot package), so those two DaemonSets roll, and so does the controller, whose --identity-gate-image flag names the agent image. The node proxy does not: its bootstrap ConfigMap is the same but for the helm.sh/chart label (compared by rendering the 2.5.6 and 2.5.7 charts offline).

Measured: unit tests of the reader and of a resolver started on, and reloaded onto, a bare-map file. Not run on a cluster.

Chart 2.5.8: cni-install no longer accepts the OTLP flags

The last row of the table under "Chart 2.5.7" is closed: cni-install --otlp-endpoint and --otlp-pin-endpoint, deprecated no-ops since chart 2.4.0 (#1166) and removable from 2.5.0 on, are gone from the binary.

A chart older than 2.4.0 cannot run a cni-install image from chart 2.5.8 on. Charts before 2.4.0 passed both flags to the agent pod's cni-install init container whenever an OTLP endpoint was configured (otel.endpoint or cniInstall.otlpEndpoint), and neither when none was. Any chart from 2.4.0 on passes neither, whatever otel.endpoint is (//charts/aether:aether_cni_no_otlp_test).

What it looks like: the init container exits 1 with Error: unknown flag: --otlp-endpoint, the agent pod sits in Init:CrashLoopBackOff, and the node's agent never starts.

Who can hit it: only someone who overrides cniInstall.image on a chart older than 2.4.0 with an image built from this release or later, and has an OTLP endpoint configured. That is outside the one-chart-minor window. A chart pins its own images by digest, so installing, upgrading to, or rolling back to any released chart is unaffected, in either direction: an old chart runs its own old image, which still has the flags. If you are in that combination, upgrade the chart rather than the image.

What the upgrade rolls. The cni-install image changed, so the agent DaemonSet rolls (it is that pod's init container). No template changed.

Measured: a unit test that the binary refuses each flag by name before the installer runs. Not run on a cluster.

Chart 2.5.9: the registrar no longer exports aether_registrar_snapshot_content

The second row of the table under "Chart 2.5.7" is closed. The registrar's labelled info gauge aether_registrar_snapshot_content{content_hash} (OTel aether.registrar.snapshot.content: always 1, the hash in a label), deprecated in chart 2.4.12 (#1329) and removable from 2.5.0 on, is gone from the binary. That table said "2.6.0 at the earliest" for it: that was one minor too cautious (deprecated in 2.4.x means removable in 2.5.0), and this chart is the one that removes it.

A rule, a dashboard or a recording rule that still reads the metric returns no data, and says nothing. A selector on a metric nobody exports is an empty result, not an error: an alert on it stops being able to fire and stays green, and a panel shows "No data". Prometheus keeps answering from the old pods' last samples for its five-minute lookback after the registrar rolls, so the change shows a few minutes after the upgrade, not at it.

Use the value gauge every registrar has exported since chart 2.4.12: aether_registrar_snapshot_content_hash. Its value is the hash (the first 13 hex digits, as an integer), on one series per replica with no label that changes. The divergence expression, as the shipped rule AetherRegistrarSnapshotDiverged has it (docs/observability/registrar-alerts.yml; "Stored vs in place vs applied" explains each part):

count by (job, revision) (
  count_values by (job, revision) (
    "hash",
    last_over_time(aether_registrar_snapshot_content_hash[150s])
    * ignoring (revision) group_left (revision)
    count_values without () ("revision", last_over_time(aether_registrar_snapshot_revision[150s]))
  )
) > 1
unless on (job) (max by (job) (last_over_time(aether_registrar_writebehind_queue_depth[150s])) > 0)

A rule that carried the transitional second arm on the old metric (the or this runbook recommended while registrars with and without the value gauge could run side by side) keeps working: or with an arm that returns nothing is the first arm. Delete the arm anyway, it can no longer match.

Before the upgrade, find what still reads it. This must print nothing:

# Rule files and dashboards, wherever yours are kept. The pattern does not match
# the replacement, whose name goes on with `_hash`.
grep -rnE 'aether_registrar_snapshot_content([^_]|$)' <rules-and-dashboards-dir>

After it, with the registrar rolled and five minutes gone:

# Empty: no registrar exports the old metric.
aether_registrar_snapshot_content

# The value equals the number of registrar replicas: every one exports the
# replacement.
count(last_over_time(aether_registrar_snapshot_content_hash[150s]))

Skew, both ways. A registrar of any chart from 2.4.12 on exports the value gauge, so during the roll (and after a rollback to any 2.5.x or to 2.4.12 and later) the shipped rule sees every replica. A rollback brings the old metric back with the old image; nothing reads it.

What the upgrade rolls. The registrar image changed, so the registrar Deployment rolls. No template changed, and no other workload's image did.

Measured: unit tests that the registrar's exported metric set does not contain the name and that the value gauge is unchanged; promtool test rules on the shipped rule (bazel test //:observability_rules_test), which never read the old metric. Not run on a cluster.

Chart 2.6.0: clusterName is required; the aether.pod stats extractor requires the namespace

Two changes, and both need a look before the upgrade.

clusterName has no default any more

Breaking. Until 2.5.x values.yaml carried a default cluster name, so a release that never set clusterName ran under that default without anyone having chosen it. The default is gone and nothing replaces it: the chart refuses to render without the value.

What fails, and how. helm install, helm upgrade and helm template stop at the render, with (as helm template prints it; an upgrade or an install says UPGRADE FAILED: or INSTALLATION FAILED: first)

Error: execution error at (aether/templates/registrar-deployment.yaml:44:33): clusterName is required and has no default: pass --set clusterName=<name>, or set clusterName in your values. It is this cluster's name in the mesh (--cluster-name of the agent, the registrar and the edge control plane) and a part of the registry's keys, so choose it once and keep it. …

Helm renders the whole chart before it sends anything, so the release is left exactly as it was: no object is changed, no pod is replaced, and no new revision is recorded. --set clusterName= (the empty string) is refused the same way. helm lint only warns. A GitOps controller reports the same message on its release object and keeps the revision that is running. (The render failing is tested, //charts/aether:aether_cluster_name_required_test and its neighbours; that Helm leaves the release alone follows from the order in which it works and was not run on a cluster.)

Who has to do something. Every release whose values do not set clusterName. A release that sets it already is not affected by this half.

Pass the name the release runs with. Not a new one. Read it from the running agent before the upgrade:

NS=aether-system
kubectl -n "$NS" get daemonset -l app.kubernetes.io/component=agent \
  -o jsonpath='{range .items[*].spec.template.spec.containers[?(@.name=="agent")].args[*]}{@}{"\n"}{end}' \
  | grep -- '^--cluster-name='
# The same answer from the release (-a: computed values, the chart's defaults
# included):
helm get values aether -n "$NS" -a -o yaml | grep '^clusterName:'

Then pass that name, and put it where the release's values live (a clusterName: line in the values file, or the values of the GitOps release) so every later upgrade carries it. Helm stores a --set value with the release, so the read-back below returns it from then on:

helm get values aether -n "$NS" -o yaml > values.yaml   # never --reuse-values
helm upgrade aether oci://quay.io/aethermesh/chart-aether --version "$AETHER_VERSION" \
  -n "$NS" -f values.yaml --set 'clusterName=<the name read above>'

With the same name the render is the 2.5.9 render in everything that follows from this value: compared offline (helm template of 2.5.9 at its defaults against this chart with the old default passed explicitly), the only lines that differ for this half are the helm.sh/chart labels.

What a different name does. The name is not a label. It is --cluster-name of the agent, the registrar and the edge control plane, and:

• Every endpoint carries it, stamped by the agent that registers the pod (and, with the kubernetes backend, by the registrar that derives the endpoint from the pod).
• An agent removes only endpoints that carry its own name. Its sweep for endpoints whose pod is gone skips every endpoint of another cluster, because node names are not unique across the clusters that share a registry.
• With the etcd backend it is a segment of every key the registrar writes: /aether/v1/regions/<region>/clusters/<clusterName>/… (endpoints, the MCS export marks and the cross-cluster config projections). A registrar writes and deletes under its own name only, and reads every cluster's keys.
• With agent.eastWestWaypoint, an endpoint whose cluster name is not the agent's own is a remote endpoint, dialed at its node's address and the tunnel port instead of at the pod.
• With controlCluster set, the registrar exports configuration only when its own name equals it, and an agent imports only what originates there.

So an upgrade that passes another name is a rename of the cluster, and this is what it does:

Backend	What happens
etcd	Everything registered so far stays under the old name's keys, and nothing removes it any more: endpoint keys have no TTL, a deregistration deletes the key under the new name, and the agents' sweep skips endpoints of the old name. The registry is read across all names, so every proxy keeps those endpoints. Pods that are still running are fine at first (the address is right), and are listed a second time once their agent registers them under the new name. Each of them that later goes away leaves its old endpoint behind for good: callers are balanced onto an address that no longer answers, or that another pod has been given since. The export marks and config projections written under the old name stay as well, and now read as another cluster's: with agent.importConfig, this cluster's own agents import its old route configuration as a peer's (local routes win where both exist).
etcd, controlCluster set to the old name	Additionally the hub's registrar stops exporting configuration (its name no longer equals controlCluster), and what it exported before stays in the registry, unmaintained, as the configuration every spoke trusts.
kubernetes	Nothing is stored under the name, so the rename settles once the registrar and every agent have rolled. Until then the two names coexist.
either, with agent.eastWestWaypoint	While the two names coexist (and, with etcd, for every endpoint left under the old name) an agent treats endpoints of its own cluster as remote and sends that traffic to the node's tunnel port.

Undoing a rename made by mistake: upgrade again with the name the cluster had, let the registrar and the agents roll, and with the etcd backend remove what was written under the wrong name (etcdctl del --prefix /aether/v1/regions/<region>/clusters/<wrong name>/). There is no in-place rename procedure in this runbook; the table is why.

The consequences above are read from the code (registry/internal/etcd, the agent's ghost sweep in agent/internal/cni/server, proxy.WaypointRewrite, the registrar's export gate). None of them was reproduced on a cluster for this change.

After the upgrade.

NS=aether-system
# One name, the one read before the upgrade, on every component (the edge
# control plane runs in its own namespace when it is enabled). The label's
# value is the release's name.
kubectl get pods -A -l app.kubernetes.io/instance=aether -o yaml \
  | grep -o -- '--cluster-name=[^"]*' | sort | uniq -c

# etcd backend: the cluster names that own keys. Only names you know.
etcdctl get --prefix --keys-only /aether/v1/regions/ \
  | sed -n 's|.*/clusters/\([^/]*\)/.*|\1|p' | sort | uniq -c

The aether.pod extractor requires the namespace

The first row of the table under "Chart 2.5.7" is closed. Chart 2.5.0 made the namespace part of the proxy bootstrap's aether.pod stats extractor optional, so that a 2.5.x proxy under an agent older than 2.5.0 (which names a pod's listeners inbound_<pod>, without the namespace) still exported one metric family per stat. That tolerance was announced for one chart minor ("Chart 2.5.0") and is removed:

	2.5.x	2.6.0
aether.pod	^listener\.(?:inbound\|out_http)(?:_[^_.]+)?(_([^.]+))\.	^listener\.(?:inbound\|out_http)_[^_.]+(_([^.]+))\.

For the names every agent since 2.5.0 writes (inbound_<namespace>_<pod>, inbound_<namespace>_<pod>_h3, out_http_<namespace>_<pod>) nothing changes: the same two labels, aether_namespace and aether_pod, on the same families. //agent/test/envoy_validate (TestChartStatsConfigOnNamespacedPodNames) runs the pinned proxy with the chart's stats_config and reads them.

A 2.6.0 proxy under an agent older than chart 2.5.0 is unsupported. That is an upgrade that skipped 2.5.x, or one started before every agent of the 2.5.x roll was replaced. Go through 2.5.x and let it finish first (charts/README.md, "Compatibility window"). What an operator who skipped it sees, measured on the pinned proxy by the same test: no traffic is affected, and for as long as a node's agent is the old one, that node's proxy exports each pod's listener stats under a name of its own (envoy_listener_inbound_<pod>_downstream_cx_total, one metric family per pod and stat, no aether_pod label). The families stop being written when the agent is replaced, and stay in the metrics store's name index for as long as the store keeps names.

# Empty when no proxy is being sent names without a namespace. (Envoy's own
# per-worker listener stats are excluded.)
count by (__name__) ({__name__=~"envoy_listener_(inbound|out_http)_.+_downstream_cx_total", __name__!~".+_worker_[0-9]+_.+"})

What the upgrade restarts and rolls

Object	What changes	Effect
aether-proxy-config ConfigMap	its envoy.yaml: the aether.pod regex above, and comments (the chart's templates no longer name the environment the project is developed on; the comments of the bootstrap are part of the file)	no pod is replaced; the supervisor hot-restarts Envoy in place once, on every node
agent DaemonSet	the agent image (the description of one metric, aether.agent.liveness.inbound_gate_pods, was reworded; its name, labels and values are as they were)	rolls
edge Deployment (edge.enabled)	it runs the agent image	rolls
controller Deployment	its --identity-gate-image flag names the agent image	rolls
every other workload	no pod template change: the comments that left the agent, proxy and registrar templates were YAML comments, which are in helm template output and in no object	not rolled by the chart

The hot restart is the one described under "Chart 2.4.26": each node's supervisor starts a new Envoy epoch when that node's kubelet delivers the file, on every node independently and not one node at a time, and two Envoy processes run on the node for the length of it. Check the headroom of "Sizing nodes for a proxy hot restart" before the upgrade and watch it as you would a proxy roll. A rollback to 2.5.x changes envoy.yaml back and costs the same again. It needs nothing for the cluster name: helm rollback applies the stored manifest of the older revision, which has the name in it.

# Each node's Envoy epoch. Compare with the value before the upgrade: a node
# whose epoch did not move has not picked the bootstrap up.
max by (node) (envoy_server_hot_restart_epoch{job="aether-proxy"})

The prober chart is 1.0.10 with a comment reworded in one template: no object of it changes.

Measured: the renders (offline, against 2.5.9) and the extractor on the pinned proxy. Not run on a cluster.

Chart 2.6.2: a proxy termination grace period the supervisor cannot use is refused

What changed. The chart refuses to render when proxy.terminationGracePeriodSeconds is not greater than proxy.hotRestart.drainTime + 15 s. At the default drainTime: 10s the smallest value that renders is 26. The default, 180, is not affected, and a release that never set either value has nothing to do.

Why. The supervisor is told the grace period (--termination-grace) and works out from it the last moment at which it can still drain Envoy itself before the kubelet's SIGKILL: grace − (drainTime + 5 s for Envoy to exit) − 10 s of margin. When that is zero or less there is no such moment, and it applies no bound at all:

• a pod terminated where no successor comes (a node shutdown, a scale-down, an uninstall) waits for the SIGKILL with its connections open (#771);
• a pod terminated while it replaces a predecessor it has already asked to drain (a roll that changes proxy.concurrency and is reverted, "A reverted roll that changes the Envoy worker count" below) holds its termination until its own Envoy is LIVE, for 56 s at most at the default drain time instead of for what the grace period allows. The kubelet can kill it first, with the predecessor's listeners closed and its own Envoy not started, and the node then has no listeners until the kept pod restarts its Envoy in place.

Such a grace period never worked. It used to render.

What fails, and how. helm install, helm upgrade and helm template stop at the render (an upgrade says UPGRADE FAILED: first):

Error: execution error at (aether/templates/agent-proxy-daemonset.yaml:73:40): proxy.terminationGracePeriodSeconds (20) must be greater than proxy.hotRestart.drainTime (10s) + 15s: set it to at least 26. The proxy supervisor needs the drain time, 5s for Envoy to exit and a 10s margin before the kubelet's SIGKILL; …

Helm renders the whole chart before it sends anything, so the release is left as it was: no object changes and no pod is replaced. A GitOps controller reports the same message on its release object and keeps the running revision. (The render failing is tested; that Helm leaves the release alone follows from the order in which it works and was not run on a cluster.)

Who has to do something. A release whose values set proxy.terminationGracePeriodSeconds to drainTime + 15 s or less. Read what the running DaemonSet has, before the upgrade:

NS=aether-system
# The grace period of the proxy pods...
kubectl -n "$NS" get daemonset aether-proxy \
  -o jsonpath='{.spec.template.spec.terminationGracePeriodSeconds}{"\n"}'
# ...and the drain time their supervisor runs with. The first must be greater
# than the second plus 15 s.
kubectl -n "$NS" get daemonset aether-proxy \
  -o jsonpath='{range .spec.template.spec.containers[*].command[*]}{@}{"\n"}{end}' \
  | grep -- '^--drain-time='

What to set. A grace period above the minimum the message names, in the release's values, or a shorter proxy.hotRestart.drainTime. The minimum is where the supervisor's bound starts to exist, not a good value: at the minimum a deleted pod waits one second for its successor before it drains Envoy itself, which no surge replacement meets, so no roll or pod delete is hitless. Unless there is a reason for a short grace period, remove the override and take the default.

helm get values aether -n "$NS" -o yaml > values.yaml   # never --reuse-values
# edit proxy.terminationGracePeriodSeconds in values.yaml (or delete the line)
helm upgrade aether oci://quay.io/aethermesh/chart-aether --version "$AETHER_VERSION" \
  -n "$NS" -f values.yaml

Changing the value changes the proxy pod template, so that upgrade is a proxy roll.

proxy.shutdownDrainImmediately: true does not lower the minimum. It skips the wait for a successor, not the drain, and a termination during a worker-count replacement is held the same way with it.

proxy.hotRestart.drainTime is read as the supervisor reads it, a Go duration (10s, 1m30s, 1.5s, 500ms, 0). A value the supervisor's flag parser refuses, such as a number with no unit, is refused at render too, instead of at every proxy start.

What the upgrade rolls. Nothing, by this change: with values that render, the output is the 2.6.1 output except for the helm.sh/chart labels, which are on no pod template (compared offline at the defaults).

After the upgrade, the same two commands show the values in force. On a termination, the supervisor's log line waiting for a successor before draining carries successorWaitBudget: it is a positive duration, never 0s.

Measured: the renders and the template tests (bazel test //charts/aether:all), and a unit test that holds the chart's 15 s to the supervisor's two constants. Not run on a cluster.

Chart 3.0.0: a port is typed by its own declaration, and the dialled port decides (proposal 043; #1672, #1682)

A breaking release, and the reason for the major version. A service is reached on the ports its pods declare, by their numbers, and on nothing else. The two HTTP spellings that meant "the port the pods declare" are gone (<name>:18081 and a name without a port), a raw-TCP service no longer answers on ports it does not declare (<name>:18082 is kept and still reaches its declared raw-TCP port), a name without a namespace is the caller's namespace's, a pod cannot declare the mesh's own two ports, and a registry row must say the protocol of each of its ports. Each has a section below headed BREAKING, with what to look for on 2.6.x before upgrading.

No chart value, flag or CRD changes. One RBAC rule changes: the agent's ClusterRole gains list, get and watch on endpointslices (discovery.k8s.io), which a cluster that pins or audits the chart's roles will see (see "a Service port whose targetPort is a name" below, and the rollback note). 18081 stays, permanently, as the mesh's HTTP proxy port (curl --proxy http://127.0.0.1:18081), where the Host header says which service and which port.

• Charts 2.7.0 and 2.7.1 are not releases. They exist in the registry because every merge to main publishes a chart and the first pull requests of this work were merged ahead of the rest. Each holds a part of the change described here, neither was deployed or validated as a whole, and neither is supported: do not install them. 3.0.0 follows 2.6.x.
• The upgrade is not hitless across versions, by decision. See "During the roll, and rolling back" before scheduling it.
• Deploy prober chart 1.1.1 first (its section follows this one).

What the major version changes for the compatibility window. "One release" stays one minor of the aether chart (charts/README.md, "Compatibility window"), and nothing here was deprecated in a published 2.x chart first: the removals are made in a major version instead, which is what a major version is for, and this section is their announcement. From 3.0.0 the window is counted in 3.x minors as before: something deprecated in 3.0.x may go in 3.1.0. The skew the project designs and tests for is still two adjacent minors; 2.6 with 3.0 in one cluster is NOT such a pair, and what it does is measured below instead of promised. The harness contract's version (4) and the crds chart are unchanged by the number; prober is 1.1.1 and udsecho 2.0.6, for template tests only (their renders are those of 1.1.0 and 2.0.5).

What is fixed

• A service with a raw-TCP port and an HTTP port works both ways round, on both registry backends. A pod that declares endpoint.aether.io/port: "9000", protocol: tcp, ports: "9000,8080=h1" had no raw-TCP path at all (#1672), and on the kubernetes backend the mirror shape (ports: "8080,9000=tcp") had a capture chain for :9000 with no cluster behind it. Each port is now typed by its own declaration, read from the registry's endpoints. Nothing reads a protocol for the service as a whole.
• A Kubernetes name with a port reaches that port (#1682). http://<svc>.<ns>.svc.cluster.local:9090/ and http://<svc>.<ns>:9090/ were answered by the port the pods declare in endpoint.aether.io/port, whatever port was dialled, with a 200.
• A port-less Host on a dialled port means that port. curl -H 'Host: <name>' http://<name>:9090/ was answered by the declared port too.
• HTTP on a port number that another service declares as raw TCP stays in the mesh. One service declaring 9000=tcp sent every other service's HTTP on port 9000 through the passthrough, outside the mesh.
• A route applies to every port of its parent Service, and a backendRef port is read. An HTTPRoute whose parentRef names no port applied to the port-less name only, and its backendRef.port was ignored (every backend was the backend's declared port).

The four refusals

An HTTP request that names a mesh service and no HTTP port of it is answered by the calling pod's proxy, at once, with 421 Misdirected Request. The response header x-aether-error says why, and the body lists the service's ports when the node knows them:

x-aether-error	Means	Before
port-mismatch	the Host header names a port other than the one the client connected to	routed by Host: answered by a port the client did not dial
port-not-declared	the service declares no such port	a 503 after the 2 s on-demand timeout (mesh name), or the declared port's answer (Kubernetes name)
port-not-http	the port is declared raw TCP or UDP	a 502 or 503 from an h2 cluster in front of a port that does not speak HTTP
port-required	the proxy port (127.0.0.1:18081, or a dial to <name>:18081) with a Host that has no port, for a service that does not serve HTTP on port 80	the port the pods declare answered (or 502 when that port was raw TCP)

no-http-port (a service with no HTTP port at all) is unchanged.

To find clients that now see one, look in the proxy's HTTP access log for responses with status 421. The case to look for is a proxy or a sidecar inside a pod that forwards requests with their original Host to a different port.

BREAKING: a route's ports are real ports, and what else a client can notice

• A name without a namespace (http://<svc>/, Host: <svc>:8080) is the service of that name in the caller's own namespace, and nothing else. See "BREAKING: a name without a namespace is the caller's namespace's" below.
• A port-less URL to a service that declares port 80 (http://<svc>.<ns>/ from a captured pod, which is a dial to port 80) is answered by port 80, like a dial to any declared port. It used to be answered by the port in endpoint.aether.io/port. A service that does not declare 80 is unchanged. Only a service whose pods declare both 80 and another declared port can see a different answer; address the other port by its number.
• A route's ports are real ports (BREAKING for a route that names one that is not). A parentRef.port and a backendRef.port name a port the Service really has: for a generated mesh Service the ports its pods declare, for your own Service its spec.ports. 18081, and 80 on a service that does not declare it, are not ports of a service; through chart 2.6 a route that named one was read as "the port the pods declare", and any other wrong backendRef port fell back to that port too.
• A backendRef to a port the Service does not have, or to one that is not HTTP, is invalid: its share of the rule's traffic (by weight) is answered 500, as the Gateway API specifies, and the route's status says ResolvedRefs=False with BackendNotFound or UnsupportedProtocol. The two come from the same facts, also for a Service of yours. There the Service says only that a port exists and which pod port is behind it; whether the pods serve that port as HTTP is what they declare (endpoint.aether.io/ports). The status does not read the registry for it: the agent that writes it takes the class of the pod port from its own node's route tables, the very value they were built from, so that node's status and that node's proxy cannot differ. A backendRef to a port of your Service whose target port the registered pods declare raw TCP or UDP is UnsupportedProtocol; one whose target port no registered pod declares is BackendNotFound; both are answered 500. Where pods disagree about the port, both go by what most pods declare. A parentRef to such a port is judged the same way and is Accepted=False. The same holds for a service whose Service says nothing of its ports yet: it has no Service object (pods can register before the registrar has generated it), or a generated one that carries no real port. A reference to it is judged by the ports its pods declare, in the tables and in the status alike.
• Every node's agent writes the one status of a route, and the nodes do not all hold the pods of a Service of yours. Each condition is computed again on every reconcile, from what the agent reads itself: the route, the ReferenceGrants, the Services and their ports. Nothing the status said before is kept, so a False goes the moment its cause does (a grant is created, a Service gains the port). The one thing a node may not be able to judge is what the pods behind a port of your Service serve; only a node whose proxy holds the service can, and such a node says what its own tables do. Its verdict is written in the route's status, in a third condition beside Accepted and ResolvedRefs on the same parent, of type gateway.aether.io/PodPortsJudged, whose message lists <ns>/<svc>:<service port>><pod port>=<http|tcp|udp|undeclared>. A node that holds no pod of the Service takes the verdict from there, for that port, that target port and that generation of the route, a True like a False, and writes the same conditions the judging node did; it never writes "Not checked" over a verdict, and never keeps one about anything else. A port nobody has judged is "Not checked" (and answered by no 500 anywhere). The condition is always True; it is a record, not a verdict on the route, and it is absent when the route names no such port. Nodes that hold the same rows of the Service compute the same status, and so does a node that holds none: such a set settles after at most a write or two per node. A verdict whose pods have all gone stays until a node holds a pod of the Service again or the route's spec changes.
• Known limit: two writers that do not agree. Two nodes that hold different rows of one service (a registry update one of them has not received) each judge by their own tables, and neither takes the other's verdict. So does an agent of chart 2.6 during the roll, which writes Accepted=True always and ResolvedRefs from the ReferenceGrants alone (see "During the roll"). They cannot settle while they differ, and every write of one is an event for the other. An agent of this chart that finds its own entry overwritten with something else writes it back once at once, then no sooner than 5 s, 10 s, 20 s, 40 s, 80 s, 160 s and from there 5 minutes after its previous write: at most 5 writes in the first minute of a disagreement and at most one every 5 minutes from about ten minutes on.
  • The bound is per route and per NODE. Every agent keeps its own count, so N agents of this chart against one writer that disagrees write N times as often, and that writer answers each: with 10 agents, 50 writes of one route by this chart in the first minute and 50 by the other (held by TestStatus_TheBoundIsPerNode).
  • While it lasts, the OTHER writer's text is what you read most of the time. It answers at once; an agent of this chart waits. The status shows this chart's verdict for the moment between its write and the answer, and the other's for the rest of every wait.
  • When the other writer stops (the rows arrive, the roll completes), the next write of an agent that holds the current rows stands.
  • The wait is only for writing the SAME status again. A status the agent has not written before is written at once, always: another generation of the route, or another verdict of its own (a fault that appears, clears, or comes back), also while it is being held back for the old one. The count also starts again after ten quiet minutes, and a route deleted and created again under the same name is a new route.
• A status write that fails is retried. A timeout or an error of the API server fails the agent's reconcile, which the controller runs again with a back-off capped at 30 s; it logs failed to write route status. (It used to be logged and dropped: the status then stayed as it was, True while the proxy answered 500, until some other event came by.) A conflict is not retried that way: the route changed, and that change is itself an event.
• A condition's message is at most 32768 characters, or the API server refuses the whole status. A route can name some 290 ports of Services with long names in a long namespace, and both the "Not checked" message and the record can pass that. A message is cut and ends in ... (truncated). The record is never cut: one that would not fit is written with a 32-digit digest (128 bits) in place of every key (#1a2b3c4d5e6f...=http, ...), which every agent reads as it reads the long form. (For a few builds it left verdicts out instead, and a node that held no pod of a left-out Service wrote "Not checked", True, over the False of the node that did.)
• The status is judged again when what a node's tables are built from changes: the agent's cache tells its route reconciler whenever a registry refresh changed the ports of any service (first held, dropped, or declared differently). That includes the refresh a route itself causes, by making its Services dependencies of the node. (Through the first builds of this chart it was a 30 s timer, and a new route to a port its pods declare raw TCP said "Not checked" for up to that long while the proxy already answered 500.) For a generated Service the status follows the Service's own ports, which the registrar writes from the registry; between a pod registering and that write the status is one registrar update behind the proxy.
• A parentRef to such a port attaches the route to nothing: the service answers as if the route did not exist, and the status says Accepted=False with NoMatchingParent.

• BREAKING: a Service port whose targetPort is a name is refused until the name resolves (owner's ruling, 2026-10-13). The registry carries no port names; the name is resolved through the Service's EndpointSlices, whose ports carry the Service port's name and the number it has on the pods. While it does not resolve, the port is not a port the mesh can route:

  • a backendRef to it is invalid: its share is answered 500, on the loopback listener, the proxy port and every captured dial, and the route's status says ResolvedRefs=False / BackendNotFound: backendRef <ns>/<svc>: port 80 of the Service has the named targetPort "http", which no EndpointSlice of the Service resolves yet; requests for this backendRef are answered 500 until it resolves;
  • a parentRef to it attaches the route to nothing, Accepted=False / NoMatchingParent, with the same reason; a request to that port of the Service is 421 port-not-declared;
  • on the proxy port and the loopback listener the mesh name with that port is 421 port-not-declared too. Every one of these 421s says why in its body, on every table: Its Service's port 8000 has a targetPort that is a name the Service's EndpointSlices do not resolve yet: it is refused until they do. That holds for every refusal the name gets while one of its Service's ports is unresolved, whatever the refusal is for: port-required for the name without a port, port-not-http, port-mismatch, and no-http-port. The last one says, for a name the node knows no port of at all (a filter or a route and the port map alone), <name> has no port it can be addressed on yet and not "serves no HTTP port. Reach it over TCP instead", which it said until this was fixed.

  Whose names are resolved. Every Service of yours the node's proxy can reach a mesh service through: one a route names, one a service-wide filter targets, and one that fronts a mesh service the node holds endpoints of, with or without a route. (For a few builds only a Service a route named was resolved. A mesh service behind an unrouted Service with targetPort: <name>, which is the common way to write a Service, then had no port at all on the proxy port: 421, "Its ports: none", on every spelling and for as long as the Service existed, with nothing logged.) A Service that fronts nothing of the mesh on the node is not read.

  What it costs. The agent keeps, per Service, what the Service's EndpointSlices said, and lists them again only for the Service a slice event was seen for: one list per slice event, of that Service, not one per concerned Service. (Read anew on every reconcile it was the latter: with 200 such Services, 200 lists on every node for each pod of any of them starting or stopping.) Everything is read once when the agent starts. A kept answer is dropped, and the Service read again, when the Service's named ports change, when the Service object is replaced, when the Service has not been one of the concerned ones in between, and when the answer is ten minutes old: that last is the bound on a slice change the agent was never told of, and the agent wakes itself for it. The bound is ten minutes plus however long the API server cannot be read at that moment (below): a ten-minute re-read that fails keeps the answer it came to check.

  Through chart 2.6, and the first builds of this one, such a reference was served by the port the pods declare in endpoint.aether.io/port: a route that worked that way needs the name to resolve now, or a numeric targetPort, which has none of this. The three reasons a name does not resolve, each named in the status and in the agent's WARN (a Service port has a named targetPort that does not resolve to a pod port, logged once per Service, port and reason, not on every reconcile; for a Service no route names it is the only trace): no EndpointSlice of the Service names the port yet; the slices resolve it to different ports (... resolve to different ports (8080, 9090): pods of one Service whose containers give the name different numbers; it stays refused while they disagree); the slices could not be read, which also fails the agent's reconcile so that it is retried.

  The window after a Service is created before its pods. A Service with targetPort: http has no EndpointSlice that names the port until a pod behind it exists and is selected. A route that names the port answers 500 (or is attached to nothing) for exactly that long. The agent watches the EndpointSlices of the Services whose names it resolves, their metadata only, and judges again on the slice's event: the 500 ends when the slice appears, not on a timer. Every event of such a slice is one reconcile, also one that changes only its endpoints (a pod of that Service starting or going) and not its ports; slices of other Services cause none. A slice that goes again (the Service scaled to nothing) un-resolves the name, and the port is refused again. The agent's ClusterRole has list, get and watch on endpointslices for this; the slices themselves are read from the API for those Services only and are not cached.

  When the API server cannot be read. A reconcile that has to list a Service's EndpointSlices and cannot fails, and what happens to the Service depends on why it was being read:

  • Something says its answer is no longer current (a slice event of it was seen, its named ports changed, the Service object was replaced): THAT Service's named ports are refused, not served from the older answer. The other Services keep their answers and stay routed.
  • Nothing happened to it, and its answer was only due for the ten-minute re-read: the answer stays in use, exactly as it was. (In the first build of this the re-read dropped it: an API server that was away when the agent's own timer fired had every such Service of the node refused, with no evidence that any of them had changed.) A slice change the agent was never told of is therefore taken up ten minutes after it at the latest when the API can be read, and otherwise by the first read that succeeds.

  Either way the controller runs the reconcile again with a back-off that starts at 5 ms and is capped at 30 s (controller-runtime's default grows to 1000 s), and the retry reads only what failed or is still due. So a Service whose slices changed during an API outage, with the event seen, is refused until the API answers, and routed again within 30 s of that; a Service nothing happened to is not touched by the outage. The same cap holds for a status write that fails.

  A refusal's body is at most 4096 bytes. Every 421 of a name whose Service has unresolved ports lists them, and a body lists a service's ports. Envoy rejects a route table in which a direct response's body is longer than that, and every route of the table with it, so a body that would be longer is cut and ends in ... (truncated). The route's status and the agent's WARNs name every port. - A backendRef of another kind than a Service, or to a Service in another namespace with no ReferenceGrant, is invalid in the same way: its share is answered 500 (the status already said InvalidKind or RefNotPermitted). Through chart 2.6 it was dropped silently: its share went to the rule's other backends, and a rule left with none was routed to its parent Service. - A Service is a port map whether or not a route names it: port: 80, targetPort: 8080 on a Service that fronts a mesh service makes port 80 a port of it, on the loopback listener too (Host: <name> is then the pods' 8080). The agent reads every Service's ports for this from the watch it already keeps; no new watch, no new memory. - The port in an authority is the Service's port, on both paths. For a name a Service of yours owns, port: 8000, targetPort: 8080:

  Request	Captured dial	Proxy port (127.0.0.1:18081, or a dial to <name>:18081)
  <name>:8000	the pods' 8080	the pods' 8080
  <name>:8080 (a pod port the Service has no port for)	421 port-not-declared	421 port-not-declared

  Through the first builds of this chart the proxy port took the POD port's number instead: Host: <name>:8080 was delivered and Host: <name>:8000 refused, the opposite of a dial. A client that put a pod port in Host for such a name must use the Service's port.

Before upgrading, list the routes that name 18081 or 80 (review each 80 against what the Service declares):

kubectl get httproutes,grpcroutes -A -o json | jq -r '
  .items[] | . as $r
  | ([.spec.parentRefs[]? | select((.kind // "Gateway") == "Service") | {where: "parentRef", name, port}]
     + [.spec.rules[]?.backendRefs[]? | {where: "backendRef", name, port}])[]
  | select(.port == 18081 or .port == 80)
  | "\($r.kind) \($r.metadata.namespace)/\($r.metadata.name): \(.where) \(.name):\(.port)"'

After upgrading, the status names every route the mesh refused a port of:

kubectl get httproutes,grpcroutes -A -o json | jq -r '
  .items[] | . as $r | .status.parents[]?
  | select(.controllerName == "gateway.aether.io/mesh")
  | .conditions[] | select(.status == "False")
  | "\($r.kind) \($r.metadata.namespace)/\($r.metadata.name): \(.type)=\(.reason): \(.message)"'

Fixed: requests that skipped a service-wide HTTP filter

A service-wide (CHAIN-scope) HTTPFilter is enabled by the destination service's own virtual host in the source proxy's route table. ext_authz is one such filter. Through chart 2.6, two kinds of request were delivered to the service without it:

• A captured request that spelled a real port of the service (http://<svc>.<ns>.<mesh-domain>:8080/ from a redirect-all pod). A mesh service's virtual host claimed its port-less and :18081 authorities; an authority with another port fell to the catch-all, which routes by name and enables no service's filter.
• The first request to a service from a node that held no endpoint of it yet, on either path. The catch-all parked it while the cluster was fetched, and it resumed on the catch-all's route.

Both are closed: every table gives a mesh service a virtual host of its own for every authority the mesh claims for it, with the filter, before and after the node holds its cluster. (The Kubernetes names of a mesh service behind a Service of yours that no route names are not the mesh's: a dial to them reaches the pods through your Service, with no filter. See the known limit under "While a node holds no endpoint of a service".) If a service relied on such a filter for authorization on 2.6 or earlier, requests of these two shapes were not authorized by it; its own access log (not the filter's) shows them.

The same held for a Gateway API route's own rules (an HTTPRoute parented to the service): that first, parked request on the loopback path was delivered to the service's own cluster with the route's matches, filters and backends skipped. It is closed the same way: the service's virtual host that stands in before its cluster arrives carries its routes' rules.

BREAKING: a name without a namespace is the caller's namespace's

A Host with no namespace (echo, echo:8080) means the service of that name in the namespace of the pod that sent the request, as the cluster's DNS search path does. Never a service in another namespace, whether or not the name is unique in the cluster.

Request	Through chart 2.6	Now
Host: echo:8080, the caller's namespace has a mesh service echo	echo if exactly one namespace had one	the caller's namespace's echo
the same, and another namespace gains an echo	every caller of the short name lost it	nothing changes for the existing callers
Host: echo:8080, the caller's namespace has no MESH service echo, another has	that other namespace's echo	not a mesh name for this caller: handled like any destination that is not the mesh's (below). Never the other namespace's echo

• <svc>.<ns> and every longer form are unchanged.
• A short name that is no mesh service in the caller's namespace is not a mesh name for that caller, whether or not another namespace has a mesh service called the same (ruling of 2026-10-12). It gets what any Host that is not the mesh's gets on the same path:

Path	What happens
a captured dial, redirect-all	the passthrough to the address that was dialled. A plain Service echo in the caller's own namespace is reached as if the mesh were not there
a captured dial, scoped capture	404
the proxy port (127.0.0.1:18081)	404: nothing was dialled, so there is nowhere to pass it to

The passthrough is not a way to another namespace's mesh service: it goes to the address the client dialled, and the address of a mesh service is one that only the mesh serves (measured on kind: a dial to another namespace's mesh address with the short name in Host is answered 503, by neither service). Through the first builds of this chart the case was refused 421 name-not-in-namespace, which broke a plain Service in the caller's namespace as soon as any other namespace had a mesh service of its name. - A short name is refused in one case only: 421, x-aether-error: caller-namespace-unknown, when the connection does not carry the caller's namespace. Every pod's listeners store it, so this is a fault of the node proxy's configuration and not something a request can cause; without it the two cases above cannot be told apart. - The per-port rules apply to the short name exactly as to the full name: a name without a port is port 80 only if the service declares it; an undeclared port, a port that is not HTTP and a Host that disagrees with the dialled port are refused with the same reasons. - It holds on both paths: a captured dial, and the proxy port (loopback, and a pod that is not on redirect-all). - A service-wide HTTPFilter and the rules of an HTTPRoute parented to the service apply to a short-name request exactly as to the full name. That includes a rule whose own ExtensionRef is the same kind of filter as the service-wide one: the proxy then has two levels of that filter's configuration, and the short name carries what the two amount to, per filter. headerMutation applies both (the service's first); extAuthz merges them (the union of the context extensions, the route's entry winning a key; a route that disables the check disables it); rbac and headerToMetadata take the route's alone. Measured on the pinned proxy for the first three. - What the caller's namespace is cannot be chosen by the client: it is the namespace of the pod the connection came from, set by the node proxy on that pod's listeners, not a request header.

Who is affected: a workload that calls a service in ANOTHER namespace by its short name alone. It worked while that name existed in one namespace only. Address it as <svc>.<ns>. To find such calls on 2.6.x, look in the HTTP access log for an authority with no dot whose destination namespace differs from the source's.

While a node holds no endpoint of a service

A request is delivered only where it would be once the node holds an endpoint of the service, and refused as it would be then. The node does not wait for an endpoint to know a service's ports: it reads them from the generated mesh Service (p<port>-tcp, appProtocol), which the registrar derives from the registry and the agent watches.

What the node knows of the service	A request for it
an endpoint (the registry)	routed or refused by the ports its pods declare
no endpoint, the generated Service's ports	the same routes and the same 421 reasons, at once; a declared port's request is parked while its cluster is fetched
no endpoint, a user Service that owns the name (its port map)	the Service's ports route to their pod port's cluster, fetched on demand, with the rules of that port; every other port is refused 421 port-not-declared at once, a pod port's own number included. This holds for a name that carries a route or a service-wide filter. A mesh service behind a Service of yours that has neither is, to a node that holds no endpoint of it, a name it knows nothing of (next row): the agent has the port map of every Service in the cluster, and a port map alone does not make a name the mesh's. Whether a port is HTTP is not known yet, so a raw-TCP port is not refused as port-not-http until an endpoint is held
nothing but its name (a generated Service without ports, or only a route or a filter names it)	with a port: parked while a cluster of that name and port is fetched, and answered 503 after 2 s if none arrives. Without a port: refused at once (421 port-required when a filter or a route gives it a virtual host, 404 otherwise; on a captured dial, passed through to the address dialled)

• A name without a port never asks for a cluster by the bare service name: it is port 80, or it is refused.
• The parked request runs the service's service-wide filter and its routes' rules, as any later request does.
• A rule imported from a peer cluster carries no port. Warm, it applies on the port the pods declare; while the node does not know which that is, it applies on every HTTP port the generated Service lists. For a service with one HTTP port that is the same place. For one with several, a cold request to another port runs the imported rule where a warm one does not: the rule and its filter are never skipped, at the price of that difference, until the projection carries the route's parent port.
• A short name of a service known by name only. Host: <svc>:<port> from the service's own namespace is parked and delivered like the full name when the node has a virtual host for that short name and port (another namespace declares the port, or a route of the service names it). On any other port it is refused at once, 421, x-aether-error: short-name-cold: a request can only wait for a cluster named by its authority, and a short authority names none. The full name works; the short one does from the moment the node knows the service's ports. A same-named service in ANOTHER namespace changes which ports that is, not where the request goes: when that service has a virtual host for the port, the short name on that port is parked and delivered to the caller's OWN namespace's service (the route names its cluster); it is never the other namespace's.

• One limit is left, in the name-only state alone (a generated Service with no real port on it, which is what a chart 2.6 registrar writes): a route of that service with a backendRef answers whatever port the authority spells from the rule's backends, where a port the service does not have is refused once the node holds an endpoint. Nothing on the node says what the ports are, and refusing would leave the service unreachable from that node for as long as the old registrar runs. With a registrar of this chart the generated Service carries its ports and the state does not occur.

• Known limit, by decision (2026-10-12): a mesh service fronted by a Service of yours is not a captured mesh name. The registrar generates a Service for a registered service only when no other Service owns the name. When one of yours does, a dial to it is a dial to your Service's ClusterIP, which the node does not capture as a mesh address: the connection leaves through the passthrough to kube-proxy and reaches the pods from outside the mesh (no mTLS between proxies on that hop, no route rule, no service-wide filter). The mesh name (<svc>.<ns>.<mesh-domain>) and the proxy port still reach it through the mesh, by your Service's port map (the table above). To have the Kubernetes name captured, let the registrar own the Service: delete yours. A service-wide HTTP filter that targets such a Service changes none of this: it is in force on the mesh name and on the proxy port, and the Service's Kubernetes names, the name without a namespace included, still pass through to the Service unfiltered. An HTTPRoute that names the Service does make its Kubernetes names the mesh's, by the Service's port map.

A captured dial with the mesh name in Host (curl -H 'Host: <svc>.<ns>.<mesh-domain>' http://<some Service of yours>:<port>/, to your Service or to any other Service that selects the same pods) is never refused for such a service, with or without a filter: attaching a filter does not turn a call that worked into a 421. It is one of two things, decided by the port that was dialled:

Dialled port	Host	What happens
a port number the service's own pods serve as HTTP	the mesh name with that port, or with no port	delivered through the mesh, with the service's filters (before a filter existed the proxy fetched the cluster on demand; a filter that denies the caller answers its own 403)
any other port number, the ports of your Service that owns the name included	any spelling of the mesh name	leaves to the address that was dialled, as it did before the filter
any	the mesh name with a port other than the one dialled	leaves to the address that was dialled

Only the service's own pods' ports decide which row applies: not the port map of your Service, and not what any other service declares. (For a few builds a port of your Service was a delivery too, but only where another mesh service happened to declare the same port number.) It is never delivered through the mesh without the filter.

One case changes with the filter, for a service the node holds no row of (nothing is registered under the name the filter targets, or the registry has not answered yet). The node cannot tell which port its pods serve, so on every port number some mesh service serves as HTTP the mesh name is parked for that port's cluster, with the filter, until the rows arrive: with the port in Host as before the filter, and also with no port in Host, which before the filter left to the address that was dialled. It ends in a filtered delivery or, if the service never has that port, in the on-demand timeout (503); it is never an unfiltered delivery.

(Through the first builds of this chart the filter made the node hold the service, and the last two rows, and the first on the pods' own port, became 421 port-not-declared or 421 port-mismatch.) The proxy port and the loopback listener are not such a dial: there a port the service cannot be addressed on is refused as for any service, and the body lists the ports of your Service ("Its ports: 8000 (http)"), the ones an authority can name, not the pods'.

• An authority that names 18081 or 18082 is never offered on demand, for a name the node knows nothing of either: 421 port-not-declared, at once. No cluster is named for those ports (the <fqdn>:18081 alias cluster of chart 2.6 is gone), so nothing that enters the node's scope while a request waits can answer it.

BREAKING: a pod cannot declare 18081 or 18082

18081 (the HTTP proxy port) and 18082 (the L4 port, TCP and UDP) are the mesh's own and are a port of no service. A pod that names either in endpoint.aether.io/port or endpoint.aether.io/ports, with any protocol, is refused:

Where	What happens	What the operator sees
admission (the controller's pod webhook, on by default)	the pod is denied	the kubectl apply / ReplicaSet event: the pod declares port 18081 as an application port (...); 18081 and 18082 are the mesh's own ports ...
the registrar (a pod that runs anyway: admitted before the upgrade, or the webhook is off)	the port is not written onto the generated Service; the Service keeps mesh and mesh-tcp	WARN a mesh service's ports cannot all be written as its endpoints declare them, ports="port 18081: reserved for the mesh, not a port of the service (N declarations)", once per service and state
the node agent (the same pod)	the port has no cluster, chain or virtual host and is not the service's default port; the pod's other ports are served	WARN a mesh service's ports cannot all be served as its endpoints declare them, ports="18081: reserved for the mesh, not served (N endpoints declare it)", once per service and state

A service whose pods declare nothing but a reserved port is known to the node and serves nothing: its names are refused at once (421 no-http-port without a port, 421 port-not-declared with one), and its raw-TCP spelling <name>:18082 has no cluster.

Through chart 2.6 such a pod was served: 18082 declared raw TCP was an ordinary raw-TCP port, and a declared 18081 replaced the proxy port on the service's Service. Before upgrading, find them:

kubectl get pods -A -l aether.io/managed=true -o json | jq -r '.items[]
  | select(((.metadata.annotations["endpoint.aether.io/port"] // "") + "," + (.metadata.annotations["endpoint.aether.io/ports"] // ""))
           | test("(^|,)\\s*1808[12](=|,|$)"))
  | .metadata.namespace + "/" + .metadata.name'

Move them to another port number first. The container may keep listening on the old number; what is refused is declaring it to the mesh.

Nothing inside the mesh dials a cluster named for those ports. 18081 is a listener port of the node proxy (loopback, and the capture chain a dial to <name>:18081 lands on); 18082 is the destination port the capture listener matches for the raw-TCP and UDP floors, whose clusters are named tcp:<fqdn> / udp:<fqdn> and are selected by the address that was dialled, never by an authority.

BREAKING: :18081 and a name without a port no longer mean "the port the pods declare"

Through chart 2.6 three spellings reached the port in endpoint.aether.io/port, whatever its number: http://<name>:18081/, the port-less http://<name>/ (a dial to port 80), and a Host without a port, or with :18081, sent to 127.0.0.1:18081. None of them does any more. A service is addressed on a port it declares.

What the client sends	Chart 2.6	Chart 3.0.0
http://<name>:<declared port>/	that port	that port (unchanged)
http://<name>/ (dials 80), the service declares 80	the port in endpoint.aether.io/port	port 80
http://<name>/ (dials 80), the service does not declare 80	the port in endpoint.aether.io/port	421 port-not-declared
http://<name>:18081/ (so Host: <name>:18081)	the port in endpoint.aether.io/port	421 port-not-declared
127.0.0.1:18081 or <name>:18081, Host: <name>:<declared port>	that port on loopback; 421 port-mismatch on <name>:18081	that port, on both
127.0.0.1:18081, Host: <name>	the port in endpoint.aether.io/port	port 80 if the service declares it, else 421 port-required
127.0.0.1:18081, Host: <name>:18081	the port in endpoint.aether.io/port	421 port-not-declared

18081 is the mesh's HTTP proxy port, not a port of any service: on the loopback listener and on a dial to <name>:18081 alike, the port in Host (or in an absolute request line) is the port. See "Using the proxy port" in the getting-started guide. A 421 for a service whose ports the node knows lists them in its body ("Its ports: 8080 (http), 9000 (tcp)"). One body says "Its ports: none": port-not-declared for a name the node knows by a Service's port map only (a Service of yours with a route or a filter, no endpoint held), since a port map says which Service ports exist and not what the pods serve. These bodies have no "Its ports:" line at all, because the node has no port to name:

• a Host that names 18081 or 18082 (port-not-declared: the port is the mesh's);
• short-name-cold and caller-namespace-unknown;
• port-required for a service the node knows by name only;
• no-http-port for a service that declares nothing but UDP, or only ports reserved for the mesh: there is no raw-TCP address to offer instead.

One answer in this area is not a 421. 500 with x-aether-error: filter-levels: a route reached through a name without a namespace carries a service-wide HTTP filter and a route-level filter of the same kind whose two configs could not be combined into one. The request is not delivered, because delivering it would run it without one of the two. It cannot happen for a filter the mesh allows with configs that decode. The full name (<svc>.<ns>) is unaffected: there the proxy has both levels and combines them itself.

A pod that is not on redirect-all (capture.aether.io/redirect-all: "false"): only the mesh ports are diverted for it, so a dial to a service's own port is not the mesh's, and curl http://<name>:18081/ alone is now refused. Such a pod reaches an HTTP service through the proxy port: HTTP_PROXY=http://127.0.0.1:18081 (curl reads the lower-case http_proxy), or Host: <name>:<port> sent to 127.0.0.1:18081 or to <name>:18081. The divert of :18081 and :18082 for such a pod is unchanged. :18082 (raw TCP and UDP) is unchanged for every pod.

Before upgrading, on 2.6.x, find the clients. The proxy's HTTP access log (log_name=aether_access_logs) has the authority of every request; the ones to move are answered today and name :18081 or no port:

log_name:aether_access_logs response_code:200 (authority:~":18081$" OR NOT authority:~":[0-9]+$")

(Field names are the ones in test/harnesscontract/external-harness.yaml, proxy.access_log.http; adapt the query to your store.) Group by source workload and authority, and move each client to <name>:<declared port>. The prober, the udsecho chart and this repository's own suites were moved first (prober chart 1.1.1).

What a first request gets from a node that holds no endpoint of the service yet is in "While a node holds no endpoint of a service", above: it is refused like any later one.

BREAKING: a raw-TCP service is reached on its declared ports only

Until this chart, a service whose declared port is raw TCP and that has no HTTP port answered on any port of its address: a connection to a port nothing declared was delivered to the declared port (the any-port chain, cap_tcp_anyport_<cluster>). That chain is gone, without a deprecation release. The service is reached on the ports it declares and on :18082. A connection to any other port of its address is not the service's: it leaves through the passthrough to the address that was dialled, which has no such port, and is not answered.

Before upgrading, on 2.6.x, find the clients that depend on it. The proxy counts every connection the chain took:

sum by (__name__) (increase({__name__=~"envoy_tcp_cap_tcp_anyport_.*_downstream_cx_total"}[7d])) > 0

A non-zero series names the service (the cluster is part of the metric name, every character outside [a-zA-Z0-9_] flattened to _); its L4 access log lines (filter_chain_name starting cap_tcp_anyport_) name the source pod and the port it dialled. Move each client to a declared port or to :18082. The counter undercounts in one case: it does not move for a port number another service declares as raw TCP. After the upgrade the series does not exist; an absent series no longer means "no client used an undeclared port".

The harness contract (test/harnesscontract/external-harness.yaml) is version 4: its envoy_stats list is empty, the entry for this counter is removed.

BREAKING: a registry row must say the protocol of each of its ports

port_protocols on a registry endpoint is required. A row is served only when it names every port it lists (port, ports), each with a protocol. A row that does not is not served by anything: no endpoint, no port, no class, on the HTTP path, the raw-TCP and UDP floors, the QUIC twins, the edge and the waypoint alike. Through the previous chart such a row took the protocol of the registry key it was stored under, for every port; nothing reads a protocol off a key or a listing any more.

Who writes such a row. No node agent since chart 0.93.2 (the release that added the field): the agent and the kubernetes backend build a row from one parser, which names the declared port and every port of endpoint.aether.io/ports. A row without the field was written by an agent older than that and never rewritten, or by something other than aether writing to the registrar API or to etcd.

What happens to one, and where it shows:

Where	What happens	How it shows
A write (RegisterEndpoint, every backend)	refused, INVALID_ARGUMENT; nothing stored, nothing sent to agents	registrar WARN refused a registration: the row does not say the protocol of each of its ports with service, cluster, ip, reason; counter aether_registrar_rows_refused_total{reason}
Already in etcd (before the upgrade, or written there by a peer cluster's older registrar)	filtered by the registrar: it stays in etcd and is in no listing the agents watch or the Service generator reads	registrar WARN a stored registry row is not served once per row (not once per sync), INFO no stored registry row is without port_protocols any more when the last is gone; gauge aether_registrar_rows_untyped{reason} (0 = none)
Handed to an agent anyway (a registrar of the previous chart during the roll)	dropped by the agent before anything is built from it	agent WARN a registry row is not served once per row; counter aether_agent_registry_rows_refused_total{reason}

reason is no_port_protocols (the row carries none), port_not_typed (it lists a port its map does not name, or names with no protocol or with one this build does not know) or port_not_listed (its map names a port that neither port nor ports lists).

The kubernetes backend stores nothing and derives every row from a pod's annotations at each listing, so it cannot hold such a row. A pod whose annotations do not parse is in no listing there, as before (WARN pod has an invalid protocol or ports annotation).

Before upgrading, count them.

• kubernetes backend: nothing to count (see above).
• etcd backend: the values are binary protobuf, so the check decodes each one and looks for field 12 (port_protocols). With etcdctl pointed at the registry's etcd and protoc on the path:

PREFIX=/aether/v1/regions        # the registry's root in etcd
etcdctl get --prefix "$PREFIX" --keys-only | grep '/endpoints/' |
  while read -r key; do
    etcdctl get "$key" --print-value-only </dev/null | head -c -1 |
      protoc --decode_raw | grep -q '^12 {' || echo "$key"
  done | tee untyped-rows.txt | wc -l

It prints the number of rows that carry no port_protocols and leaves their keys in untyped-rows.txt (.../clusters/<cluster>/ns/<ns>/services/<svc>/protocols/<PROTOCOL>/endpoints/<ip>: the cluster, the service and the pod IP are in the key). The prefix covers every region and cluster that shares the etcd, so rows mirrored from a peer are counted too. Expect 0. (Checked against an etcd holding two rows with the field and two without: it printed 2 and the two keys. It finds the no_port_protocols rows; a row whose map and lists disagree (port_not_typed, port_not_listed) is not found by it, and no aether writer has produced one.)

For each key it lists: if the pod exists, its node runs an agent older than 0.93.2, or the pod was registered by one. Upgrade that node's agent first or restart the pod, and the row is rewritten. If no pod has that IP, the row is a leftover; it was never going to be served after the upgrade, and deleting the key is safe. After the upgrade, aether_registrar_rows_untyped is the same count, kept current.

Several clusters on one etcd, or a mirror. A registrar of this chart filters what a peer wrote, whatever chart the peer runs; the peer is not told. A peer on a chart from 0.93.2 on writes rows with the field and sees no difference. A peer older than that keeps writing rows that this cluster does not serve: its services are missing here (the gauge above counts them) while they still work in the peer's own cluster and in clusters that have not upgraded. This cluster's rows always carry the field, so nothing changes for a peer that reads them.

BREAKING: a port that stops being HTTP is refused at once; a service that vanishes keeps its grace

When every HTTP endpoint of a service leaves the registry, the node keeps the service's cluster and virtual host, empty, for 90 s: a request is a retriable 503, not a refusal, while a single-replica pod restarts (#1635). That is kept, for the case it exists for: nothing says what the service is now.

It is no longer kept against evidence. If the node holds endpoints of the service that do not declare the port as HTTP (the service now serves another port, or that number as raw TCP or UDP), or holds none and its generated Service declares ports that do not include it as HTTP, the cluster and virtual host of that port go in the same refresh, and a request to it is answered 421 with x-aether-error: port-not-declared or port-not-http, like any port the service does not serve. The ports the evidence declares are routed in the same refresh. When the evidence is the generated Service (the node holds no endpoint of the service) and it declares other ports than the node last saw, a port gained included, the service becomes exactly what a node that starts at that moment knows of it: nothing the node remembered is kept, of any port or protocol, and every declared port is routed, with the service's filter and routes, to a cluster fetched on the first request. Before, such a request was sent for up to 90 s to the pods the service still had, on a port they do not serve as HTTP.

What an operator can see change:

• A service whose pods are not all alike (one pod serves HTTP on 8080, another only raw TCP on 9000): while the HTTP pod restarts, :8080 answers 421 port-not-declared instead of 503. The registry says the service has no HTTP port at that moment, and the node believes it. Clients that retry 503 and not 421 see errors for the length of the restart. Give such a service two replicas of each shape, or split it into two services.
• A rollout that moves a port number (8080 to 9090): the old number is refused as soon as the last pod declaring it is gone, not 90 s later.
• The agent logs HTTP cluster/vhost not retained: the service no longer declares its port as HTTP with service, port and evidence (listed endpoints or generated Service).

One state cannot be told apart and is treated as a replacement: the registry returns no HTTP row for a service while it still returns rows of the same service under another protocol. If that were a fault of the registry and not the truth, the HTTP port is refused until the next listing has the rows back. A listing that fails outright changes nothing (the refresh is abandoned).

BREAKING: the generated Service loses its placeholder ports and two annotations

kubectl get svc on a generated mesh Service shows what the service serves: one port per declared port, named p<port>-<tcp|udp>, with appProtocol http or tcp, and the mesh's two ports (mesh 18081, the HTTP proxy port, and mesh-tcp 18082). The http 80 and https 443 placeholders are gone: a port the pods do not declare is not a port of the service.

BREAKING for anything outside the mesh that read them: aether.io/app-protocol and aether.io/port are no longer written on the Service, and the registrar removes them from every generated Service on its first reconcile. They carried a protocol and a port for the service as a whole, which no longer exists: each port is typed by its own appProtocol. A chart 2.6 agent reads aether.io/app-protocol to choose a service's capture chain; behind a 3.0 registrar it finds none and treats every service as HTTP. That is one of the reasons this upgrade is not hitless across versions (see "During the roll").

BREAKING for dashboards: the capture chain is renamed, and the proxy port gets a chain and a table of its own

(Chart 3.0.2 folds the proxy port's chain and table back into cap_http: see "Chart 3.0.2" below.)

The capture listener of every pod has two HTTP filter chains, cap_http and cap_http_18081 (the mesh's HTTP proxy port), where it had one (capture_<pod>); the node has a route table of each name, where it had cap_http alone. Which chain a connection takes is decided by the listener's filter_chain_matcher, Envoy's generic matching tree, and no chain carries a filter_chain_match any more. cap_http records the port the client dialled (filter state aether.dialled_port) and its one route table selects a port's routes with it (RouteMatch.filter_state): a request is answered by the rules of the port it was dialled on, whatever port number the mesh declares elsewhere. The table has one virtual host per mesh name, with a plain route list.

Two refusals changed with it. A 421's body no longer names the dialled port (the x-aether-error reason is the contract; the body is a diagnostic), and on a port a name is not served on, the answer is port-not-declared whatever port the Host names: a dial to <address>:9000 with Host: <name>:8080, for a name not served on 9000, answered port-mismatch in the first 3.0.0 builds. port-mismatch stays for a port the name is served on, with a Host that names another.

A service-wide HTTPFilter and an HTTPRoute rule's ExtensionRef of the same kind now combine on a short name (and on cap_http) as they do under the service's own virtual host: a header_mutation that both levels write leaves the service's value (the route's mutations apply first), and an ext_authz rule does not inherit the service-wide filter's authorization service override. Earlier builds put the route's write last on a short name, and sent its check to the service-wide override.

On a full name the two levels are the proxy's own: the service-wide filter stays on the service's virtual host in cap_http, exactly as on the proxy port, and the 421 refusals of that virtual host disable it per route, so a refusal runs no service-wide filter (no rbac 403, no ext_authz check, no header mutation). On a short name, whose virtual host several services share, the two levels are folded onto each route; a pair that no single config applies in the proxy's order is answered 500 with x-aether-error: filter-levels on the short name, while the full name is served: today that is a rule whose header_mutation mutates query parameters under a service-wide header_mutation that mutates request headers or query parameters. Address the service by its full name, or move the query mutation to the service-wide filter.

Two counters watch the compile. aether_agent_capture_table_compile_failures_total counts the times the agent could not compile cap_http from the per-port tables; it then publishes a table that refuses every mesh name with 421 port-not-declared (no mesh name is delivered by a wrong port's rules) and logs the error. It should be 0; anything else is a bug to report. aether_agent_capture_duplicate_service_address_total counts two services claiming one address on a capture listener; the agent keeps one, logs a warning naming both, and the other is not reachable through capture there.

A port number entering or leaving the mesh changes cap_http's table and nothing else: no listener is rebuilt for it. For 200 services on three HTTP port numbers, 20 raw-TCP ports and 50 pods on a node: 23 chains per pod listener (the passthrough default included) where there were 22, 2 HTTP connection managers per pod where there was 1, 3 route tables per node where there were 2. A service whose only ports are raw TCP costs two chains per pod listener (its declared port and :18082), where it had three. A TLSRoute costs one chain per rule (there is no copy of it per port).

(The first builds of chart 3.0.0 had one HTTP chain and one route table per HTTP port number of the mesh, cap_http_<port>, plus cap_http_other. On the reference cluster that measured as 1.6× the capture listeners' LDS size, an RDS apply 8–10× slower, a slower successor start at a proxy hot restart and +34% proxy memory, and it was replaced by the shape above. Proposal 043, "Amendment: one HTTP chain, selected by a matching tree", has the design and the gates. Rolling from one of those builds to this one is node-local: the capture listener and its tables are a node's own, and what one proxy sends another is unchanged. On each node the listener is updated in place; the per-port chains it loses drain the connections they hold, and the answer to every request is the same before and after.)

A dashboard, an alert or a query that names the filter chain capture_<pod> (the access log's filter_chain_name) needs the new names. The route table of captured HTTP is still cap_http, and the HTTP stat prefix (capture_http) is unchanged; the RDS stats of the proxy port's table are under cap_http_18081 (until chart 3.0.2, which compiles that table into cap_http). A TLSRoute's chain no longer has per-port copies (cap_tls_<cluster>_<rule>_p<port>): its L4 access log and tcp_proxy stats are under cap_tls_<cluster>_<rule> on every port.

During the roll, and rolling back

This upgrade is not hitless, by decision: no compatibility with chart 2.6 components is kept. The helm upgrade replaces the registrar at once and rolls the agents node by node, so for the length of the agent roll the cluster is mixed. What each mix does, measured on kind with chart 2.6.2 images on one side and this release on the other, both registry backends (e2e/README.md, "Version skew with multiprotocol.sh"; the runs are in the pull request that removed the registrar's 2.6 annotations, #1693). Measured: the two mixes of agents and registrar below, on one node. Not measured: a 3.0 source proxy with a 2.6 destination proxy or the reverse (two nodes on different versions); a rollback to 2.6.x; a mixed cluster holding a registry row without port_protocols; the mixes after the later pull requests of the stack (the short-name rule, the reserved ports, the retention rules), which change what a 3.0 node answers and not what the two versions exchange. Each of those is marked below where it comes up.

• 2.6 agents behind the 3.0 registrar (every node not yet rolled; the window of a normal helm upgrade). The generated Services have lost aether.io/app-protocol, which a 2.6 agent reads to choose a service's capture chain: it takes every service for HTTP.
• A service whose declared port is raw TCP gets no reply from clients on those nodes, on its declared port and on :18082 (measured on a service with only a raw-TCP port, both backends). HTTP sent to such a service answers 502 or 503 where it was refused by name.
• HTTP services answer exactly as on 2.6, old spellings included: the addressing is the source agent's, and it is still 2.6.
• 3.0 agents behind a 2.6 registrar (only if the registrar is held back).
• On etcd every data-plane check passes; only the generated Service still has its 2.6 shape (placeholders and annotations, no real ports).
• On kubernetes the services whose pods serve both HTTP and raw TCP are half-served until the registrar is 3.0: its listing puts such a pod under one protocol only, so the other class's ports are refused (421 no-http-port or port-not-declared) or get no reply.
• A route's status alternates while both versions run. Every node's agent writes the status of every Service-parented route. A 2.6 agent writes Accepted=True always and ResolvedRefs from the ReferenceGrants alone; a 3.0 agent also says False for a port that is not an HTTP port of its Service, and adds the gateway.aether.io/PodPortsJudged condition. For a route the two judge alike nothing happens. For a route 3.0 finds a fault with, each overwrites the other. The 2.6 agent answers every write at once; the 3.0 agent writes back at a bounded rate (once at once, then no sooner than 5 s, 10 s, 20 s ... up to 5 minutes after its previous write: at most 5 writes of one route by one node in the first minute, then fewer), so the status of such a route shows the old agents' text most of the time and the new agents' briefly, until the last 2.6 agent is gone; the next write of a 3.0 agent then stands. The bound is per node: with N agents already on 3.0 the route is written N times as often, and the remaining 2.6 agents answer each write (10 agents: 50 writes each side in the first minute). A roll that stalls leaves it so for as long as it is stalled: read the status of a route from a 3.0 agent's point of view only once the roll has completed. Measured on kind before the bound existed: 450 rewrites of one route in 20 s. With it: the unit tests TestStatus_AContestedStatusIsWrittenAtABoundedRate and TestStatus_TheBoundIsPerNode hold the schedule; the mixed pair was not re-run on kind.
• The PodPortsJudged condition outlives a rollback. A 2.6 agent writes its two conditions into aether's entry and leaves every other one there, so a gateway.aether.io/PodPortsJudged condition a 3.0 agent wrote stays on the route after every agent is back on 2.6. It is inert: 2.6 does not read it, and a 3.0 agent reads it only for the generation it was written for. To remove it from one route:

kubectl -n <ns> get httproute <name> -o json \
  | jq 'del(.status.parents[].conditions[] | select(.type == "gateway.aether.io/PodPortsJudged"))' \
  | kubectl replace --subresource=status -f -

- A client on a 3.0 node is held to the new addressing from the moment its node's agent is 3.0, whatever version the destination's node runs: the refusals are decided by the source proxy. - Registry rows across the mix. Every 2.x agent writes rows with port_protocols, so a not-yet-rolled agent's registrations are accepted by the new registrar, and a rolled agent behind a held-back registrar serves them. Only a row without the field is affected, in either mix: the new registrar refuses or filters it, and a rolled agent drops one that an old registrar still hands it (see "a registry row must say the protocol of each of its ports"). Read from the code and unit-tested per component; no mixed cluster was built with such a row in it. - A 3.0 source and a 2.6 destination, or the reverse: the hop between proxies is unchanged (not measured separately: both cells above mix versions only between agents and registrar, on one node). - Rollback to 2.6.x: the 2.6 registrar stamps the annotations and the placeholder ports back on its next reconcile, and the old spellings work again node by node as the agents roll back (read from the code, not run). - The ClusterRole goes back at once, the agents node by node. helm rollback replaces the agent's ClusterRole in one step, and the 2.6 role has no endpointslices rule at all. A 3.0 agent that is already running keeps running. A 3.0 agent that restarts under the 2.6 role (its node not yet rolled back) cannot start its EndpointSlice informer: it never becomes Ready, logs a cache-sync timeout about every two and a half minutes and is restarted, until the DaemonSet roll replaces it with a 2.6 agent. The exposure is bounded by the length of the roll and limited to the nodes whose agent restarts during it. Watch kubectl -n <aether namespace> rollout status ds/<agent> and the agents' restart counts; do not pause a rollback half way. - Under a role that has list and get on endpointslices but no watch (a role edited by hand, or an older copy of this chart's) the agent IS Ready and resolves names, and sees a slice change only when something else makes it reconcile: tens of seconds, not at once. The chart's own role has the three verbs.

Release order. The change was built as a stack of pull requests. Its first twelve were merged to main ahead of the last two, and each merge to main publishes images and a chart: charts 2.7.0 and 2.7.1 are those intermediates, with part of the addressing rule each. They are not releases (see the top of this section). The release is 3.0.0, which is the first chart that holds all of it.

Do it in a maintenance window, clients moved first (see the two BREAKING sections above and "Prober chart 1.1.1"), and expect raw-TCP services to be unreachable from not-yet-rolled nodes for the duration of the agent roll.

Chart 3.0.2: one HTTP chain per capture listener; the proxy port is a branch of cap_http

The capture listener of every pod has one HTTP filter chain, cap_http, where chart 3.0.0 gave it two (cap_http and cap_http_18081), and the node publishes one capture route table, cap_http, where it published two. A captured dial to the mesh's HTTP proxy port (18081) is one more dialled-port branch of cap_http's table: cap_http records the dialled port for every connection (filter state aether.dialled_port), and the routes of the 18081 branch are compiled from the same table cap_http_18081 held (the loopback table's virtual hosts, where the port in Host decides). Port 18081 itself is unchanged and stays, permanently, the mesh's HTTP proxy port (proposal 043, the ruling of 2026-10-12). Proposal 043, "Folding the proxy port into cap_http", has the design and the gates.

What a request gets is unchanged. Every request dialled to 18081 gets the answer cap_http_18081 gave it, and every request dialled to any other port the answer it got before, for every Host spelling, caller namespace, path and header a route matches; the agent's tests hold the compiled table to it (cache/onehcm_equivalence_043_test.go), and the live probes drive the pinned proxy through it (agent/test/mtlspool/onehcm_probe_test.go, cases X1-X9).

For dashboards. A connection to :18081 on the captured path is logged and counted under the chain cap_http (the access log's filter_chain_name), not cap_http_18081. The RDS stats of cap_http_18081 (http.capture_http.rds.cap_http_18081.*) are gone; those of cap_http cover the whole table. The HTTP stat prefix (capture_http) is unchanged: both chains already shared it. Nothing else is renamed.

During the roll, and rolling back. Only the agent changes; the proxy image and its configuration outside the capture listener are the same. Each node's agent publishes, in one snapshot, the capture listener without the chain and cap_http with the 18081 branch, and no cap_http_18081. The listener is updated in place (Envoy compares listeners without their filter chains and their filter_chain_matcher), so the cap_http chain and every connection on it are untouched; connections that cap_http_18081 holds are drained over the proxy's drain time (--drain-time-s), as on any change of a listener's chains, and their next connection takes cap_http with the same answer. Envoy ignores the removal of a route table it still uses (a delta RDS removal is logged and dropped), so the draining chain keeps its table until it is gone, in whichever order the listener and the route table updates arrive. A rollback to 3.0.1 is the same in reverse: the chain comes back with its table in one snapshot.

Chart 3.0.3: the HTTP proxy port answers Kubernetes names

The mesh's HTTP proxy port (127.0.0.1:18081, http_proxy, or a captured dial to <name>:18081) answers a service's Kubernetes names (<svc>.<ns>.svc.cluster.local, <svc>.<ns>.svc, <svc>.<ns>) exactly as its mesh name <svc>.<ns>.<meshDomain>, by the port in Host. Through chart 3.0.2 it answered the mesh name only: Host: <svc>.<ns>.svc.cluster.local:8080 got the floor (the passthrough on a captured dial to 18081, 404 on the loopback listener). Proposal 043, "Kubernetes names on the proxy port", has the design, the gates and the sizes.

• A Service the mesh generated: its ports are the mesh ports. A user's Service that a route names: the port in Host is the Service's port, through its port map, as on a dial.
• Not answered, as on a dial: the Kubernetes names of a mesh service behind a user's Service that no route names. Also not answered: those of a service this node holds no endpoint of and knows no ports of, until it does. Its mesh name is answered on demand in the meantime.
• Rolling: each node's agent changes out_http and cap_http in one snapshot. A client that sends a Kubernetes name to 18081 is answered by the service on an upgraded node and gets the floor on a node not yet upgraded. A rollback to 3.0.2 brings the floor back.

Chart 3.0.6: the passthrough counts Service-range connections apart (proposal 044, Phase 0)

Measurement only: no path changes. Once the agent knows the cluster's Service range (chart 3.0.4), each pod's capture listener sends a connection that no mesh chain and not cap_http takes, and whose destination address is inside that range, to a second passthrough chain, cap_passthrough_svc. It is cap_passthrough under its own name: the same passthrough_original_dst cluster, so the connection goes where it went before. Only its tcp_proxy stat prefix differs, so tcp.cap_passthrough_svc.* beside tcp.cap_passthrough.* splits a node's passthrough egress into "a Service of the cluster" and "anything else". Neither chain is access-logged.

• Cleartext HTTP is unchanged: the range is tried only after cap_http, so a mesh service that serves HTTP only (inside the range, with no chain of its own) still reaches cap_http, and so does HTTP to a non-mesh Service.
• No range known (the agent has not read one yet, or ServiceCIDR is not served and --service-cidrs is unset): the listener is the one 3.0.5 builds, byte for byte.
• A change of the range rebuilds every pod's capture listener on the node; a resync with the same range rebuilds nothing. The listener is updated in place, so connections on its other chains are untouched. One range adds about 560 bytes to each pod's capture listener, whatever the Service count.
• A restarted agent's first serve also waits for the Service range (the flag, "API not served", or the first ServiceCIDR reconcile), within the same 10 s bound as the mesh-Service projection. A first serve that went ahead at the bound logs a warning naming the missing input and counts aether.agent.xds.capture_gate.timeouts{input="projection"|"service_range"}. Serving before it would publish the listener without cap_passthrough_svc and close that chain's connections when its drain ends. A real change of the range does the same to the connections on a chain it removes; a range that only grows removes nothing.
• Rolling or rolling back: per node, with the agent. A node not yet upgraded counts all of it under cap_passthrough.

Chart 3.1.0: the edge reads TLS Secrets only in the namespaces it is given

Until 3.1.0, edge.tls.enabled=true added secrets get/list/watch to the edge's ClusterRole: anyone who could run a pod as the edge's ServiceAccount could read every Secret in the cluster. The edge only needs the kubernetes.io/tls Secrets its Gateway listeners name in certificateRefs.

• Changed default. With edge.tls.enabled, the chart renders a Role and RoleBinding named <release>-aether-edge-secrets in each namespace of edge.tls.secretNamespaces (default: the edge namespace, plus edge.gateway.tlsSecretNamespace when the chart creates the Gateway), and passes the same list to the edge as --secret-namespaces. The edge's Secret informer then covers only those namespaces; every other kind stays cluster-wide. The ClusterRole has no Secret rule.
• What breaks. A Gateway listener whose certificateRefs point at a Secret in a namespace outside the list stops resolving: the listener reports ResolvedRefs=False with certificateRef "<name>" did not resolve: ... and the edge logs gateway listener TLS cert unresolved. Before upgrading, list the namespaces your Gateways' certificates live in: kubectl get gateways -A -o jsonpath='{range .items[*]}{range .spec.listeners[*]}{.tls.certificateRefs[*].namespace}{"\n"}{end}{end}' (an empty line is the Gateway's own namespace) and set edge.tls.secretNamespaces to them. Each must exist when the chart is installed.
• Opt-in to the old grant. edge.tls.clusterWideSecrets=true restores the ClusterRole rule and omits --secret-namespaces, for Gateways whose namespaces are not known at install (the conformance workflow sets it: the suite creates its namespaces after the install).
• Skew. The edge runs the agent image of the same release, so a chart and its edge image move together. An edge image older than 3.1.0 under a 3.1.0 chart fails at start on the unknown --secret-namespaces flag, and would not sync its cluster-wide Secret informer under the namespaced Roles anyway; set edge.tls.clusterWideSecrets=true if the edge image must be pinned older. A 3.1.0 edge under an older chart (no flag) watches Secrets cluster-wide, as before. Not measured on a cluster; the template tests and the cache-scoping unit test cover the rendering and the informer scope.

Prober chart 1.1.1: targets name a declared port (proposal 043)

Deploy prober chart 1.1.1 before aether chart 3.0.0. It works against 2.6 (a declared port has always been addressable), and the prober of chart 1.0.x cannot probe a 3.0.0 mesh: it addressed its targets as <name>:18081 and with a port-less Host, which 3.0.0 no longer reads as "the port the pods declare".

Tier	Dials	What a failure isolates
liveness	127.0.0.1:18081/-/-/live, a reserved authority	this node's proxy is serving its loopback listener and has its config. Unchanged.
reachability	127.0.0.1:18081 with Host: <svc>.<ns>.<meshDomain>:<port>	the data path to the target, without DNS and without capture. The port is new; the series keeps its name (the target without the port).
mesh_dns	http://<fqdn>:<port>/, resolved	mesh DNS, the capture divert and the data path. It dialled :18081; it now dials the declared port, which is the path a workload's own request takes.

The mesh_dns tier, the authz canary and the udsecho client dial a service's own port, which is captured only under redirect-all (agent.captureRedirectAllDefault, the chart default, or the pod's capture.aether.io/redirect-all). Through chart 2.6 they dialled :18081, which scoped capture diverts too. On a cluster that runs with redirect-all off, or for a pod annotated out of it, they get no reply: the connection goes to the Service's address, which has no endpoint behind it. Measured:

• e2e/multiprotocol.sh, leg S1, on both registry backends, asserts it for a plain curl: a pod with redirect-all off that dials <name>:8080 is not answered, and the same pod reaches the service through 127.0.0.1:18081.
• The prober binary itself, run on kind (etcd backend) in a pod with capture.aether.io/redirect-all: "false", with a reachability and a mesh_dns target on the same service: in 15 s at one probe a second, the mesh_dns tier logged 14 AETHER_PROBE_FAIL lines, all connection_error, and the liveness and reachability tiers none. The same binary in a pod on redirect-all logged none on any tier.
• The authz canary and the udsecho client were not deployed with redirect-all off. Both are curl loops against http://<name>:8080/, which is what S1 asserts. The liveness and reachability tiers use the proxy port and are unaffected. Run the prober with redirect-all on.

The series of the mesh_dns tier change their target label: it is the target as written, so echo.aether-test.aether.internal:18081 becomes ...:8080. Dashboards and alerts that select a mesh_dns series by target need the new value; the old series stop at the upgrade.

Between the two upgrades the authz canary reads deny=200. Do not page on it. This concerns a cluster that runs the canary (authzCanary.enabled). With the prober chart at 1.1.1 and the mesh still on 2.6.x, the canary's request without the header is answered 200 where 403 is expected, and its log says so on every cycle:

<time> FAIL allow=200 (want 200) deny=200 (want 403)

It starts when the new canary pod runs and ends when the 3.0.0 agent has reached the node that pod runs on (the filter is enabled in the source proxy's route table). Nothing regressed. The canary now dials the service's own port, and on 2.6 a captured request to a service's own port does not pass through the service-wide authorization filter at all: that is the bypass chart 3.0.0 closes ("Fixed: requests that skipped a service-wide HTTP filter" above). Through chart 1.0.x the canary dialled :18081, a path on which 2.6 did apply the filter, so it never showed. The allow side reads 200 throughout.

It was observed at this project's own upgrade and cleared with the 3.0.0 agents, with nothing done. If you alert on the canary's FAIL lines or on deny != 403, silence that for the window between the two upgrades, and keep the window short: the prober chart, then the aether chart, in one maintenance. A deny=200 that is still there once every node runs a 3.0.0 agent is a real finding: the filter is not being applied.

What changes in your values:

• probe.reachabilityTargets: write <svc>.<ns>:<port>. An entry without a port sends a port-less Host, which on 3.0.0 is port 80 and nothing else.
• probe.meshDNSTargets: write <fqdn>:<port>. No port is appended any more (the default was 18081); an entry without one dials 80. The chart's default targets name echo.aether-test on 8080: check that against the port your echo declares.
• The authz-canary (authzCanary.enabled) dials its own echo on 8080, the port it declares.

The upgrade rolls the prober DaemonSet (its arguments change).

The prober chart (#1372, #1373, #1374)

The prober chart has the same rule since chart 1.0.5: its DaemonSet's pod template no longer carries helm.sh/chart or app.kubernetes.io/version (both stay on the DaemonSet object). The chart is deployed from its commit tag, so before 1.0.5 every prober release replaced all prober pods, and each replaced pod starts new probe series (the pod label). As with the aether chart, a release built from a new commit still carries a new prober image digest and still rolls the DaemonSet; what is gone is the chart's own share.

The upgrade that crosses 1.0.5 rolls the prober DaemonSet one last time (removing the labels is a pod-template change), and so does a rollback below it. With authzCanary.enabled it also rolls the one-replica authz-canary Deployment once: its image is now written by digest (curlimages/curl@sha256:…, the index of 8.22.0). The authz-echo Deployment is not rolled: its pod template is unchanged. No selector changed.

Before and after the upgrade (the prober's namespace; aether-test on the reference cluster):

NS=aether-test
# A pod that was not rolled keeps its name, its age and its hash.
kubectl -n "$NS" get pods -l 'app in (authz-canary,authz-echo)' \
  -L pod-template-hash --sort-by=.metadata.creationTimestamp
kubectl -n "$NS" get pods -l app.kubernetes.io/component=prober \
  -L controller-revision-hash
# The release each workload object belongs to (the labels the pods lost):
kubectl -n "$NS" get ds,deploy -l app.kubernetes.io/instance=prober \
  -L helm.sh/chart,app.kubernetes.io/component
# The image the canary really runs: the digest the node resolved.
kubectl -n "$NS" get pods -l app=authz-canary \
  -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"  "}{.status.containerStatuses[0].imageID}{"\n"}{end}'

To know before the upgrade, render with the values you deploy with (never --reuse-values) and compare:

helm get values prober -n "$NS" -o yaml > values.yaml
helm get manifest prober -n "$NS" > before.yaml
helm template prober oci://quay.io/aethermesh/chart-prober --version "$PROBER_VERSION" \
  -n "$NS" -f values.yaml > after.yaml
diff before.yaml after.yaml

Since 1.0.5 the canary's five objects (two ServiceAccounts, two Deployments, the HTTPFilter) carry app.kubernetes.io/component: authz-canary on their own metadata; until then they carried component: prober, so -l app.kubernetes.io/component=prober listed them with the prober. That selector now returns the prober's DaemonSet, ServiceAccount and pods (and the Namespace, when the chart creates it with namespace.create) only; use -l app.kubernetes.io/component=authz-canary for the canary's objects, and -l app=authz-canary / -l app=authz-echo for its pods, whose labels did not change.

Nothing moves the authzCanary.image and authzCanary.echo.image digests by itself. Both are listed in scripts/third-party-images.txt since chart 1.0.7 (#1401; the values themselves did not change, so that upgrade rolls nothing): scripts/third-party-images.sh outdated says when the registry has moved past one. See Refreshing third-party image pins.

The udsecho chart (#1400, #1402)

Two changes in chart 2.0.3, neither of which touches a selector:

• Labels. Until 2.0.3 no object of the chart carried any label. Its seven objects (three ServiceAccounts, three Deployments, the EndpointPolicy) now carry the standard set on their own metadata: helm.sh/chart, app.kubernetes.io/name: udsecho, instance, component (the workload: uds-echo, uds-cr-echo or uds-client; the EndpointPolicy belongs to uds-cr-echo), part-of: aether, managed-by and version. The pod templates are unchanged (app: <name> and aether.io/managed: "true", as before), so the labels roll nothing, and the pods themselves are still found with -l app=<name>, not with the new labels.
• The client image is pinned by digest. client.image was curlimages/curl:8.22.0 and is now that tag with the digest of its multi-arch index (curlimages/curl:8.22.0@sha256:58adaa4e…6777). The upgrade that crosses 2.0.3 rolls the one-replica uds-client Deployment once for it, unless you set client.image yourself. The two echo Deployments are not rolled by the chart (a release built from a new commit still carries a new udsecho image and rolls them for that).

Before and after the upgrade (the chart's namespace, aether-test by default):

NS=aether-test
# Everything the release owns (empty before 2.0.3):
kubectl -n "$NS" get deploy,sa,endpointpolicy -l app.kubernetes.io/name=udsecho \
  -L helm.sh/chart,app.kubernetes.io/component
# A pod that was not rolled keeps its name, its age and its hash; only
# uds-client's changes.
kubectl -n "$NS" get pods -l 'app in (uds-echo,uds-cr-echo,uds-client)' \
  -L pod-template-hash --sort-by=.metadata.creationTimestamp
# The image the client really runs: the digest the node resolved.
kubectl -n "$NS" get pods -l app=uds-client \
  -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"  "}{.status.containerStatuses[0].imageID}{"\n"}{end}'

udsecho chart 2.0.7: the echo pods sleep before SIGTERM

Until 2.0.7 the two echo Deployments (uds-echo, uds-cr-echo) had no preStop hook, and udsecho exits on SIGTERM at once. A deleted echo pod was gone before mesh sources had learned its endpoint was DRAINING, and a caller's retry could land on a refusing socket: in two soaks on the reference cluster the UDS-CSI churn step, which deletes an echo pod while the CSI node plugin is down, failed 4 requests each time. That step exists to test the plugin's window, not this exit race.

From 2.0.7 both echo containers carry lifecycle.preStop.sleep.seconds: 10 (echo.preStopSleepSeconds), the value docs/workload-requirements.md asks of every mesh workload, and both pods state terminationGracePeriodSeconds: 30 (echo.terminationGracePeriodSeconds, the Kubernetes default). The hook is the native sleep action, not an exec of sleep: the image is distroless, and the mesh's drain timing reads a sleep hook (pools close 1 s before SIGTERM, common/drain.PoolCloseDelay) but cannot read an exec one. A grace period below the sleep plus 5 s (udsecho's own graceful shutdown) fails the render. echo.preStopSleepSeconds=0 renders no hook, as before.

The upgrade that crosses 2.0.7 rolls both echo Deployments once, and from then on deleting an echo pod takes about 10 s longer. Before and after:

NS=aether-test
# The hook and the grace period each echo pod runs with (empty before 2.0.7):
kubectl -n "$NS" get pods -l 'app in (uds-echo,uds-cr-echo)' \
  -o jsonpath='{range .items[*]}{.metadata.name}{"  sleep="}{.spec.containers[0].lifecycle.preStop.sleep.seconds}{"  grace="}{.spec.terminationGracePeriodSeconds}{"\n"}{end}'
# The effect: delete one echo pod and count the client's failures across it.
# Expect no FAIL line for the deleted workload (before 2.0.7 a deletion could
# cost a few).
kubectl -n "$NS" delete pod "$(kubectl -n "$NS" get pods -l app=uds-echo -o name | head -1 | cut -d/ -f2)" --wait=false
sleep 40
kubectl -n "$NS" logs deploy/uds-client --since=60s | grep -c ' FAIL '

Rendering the chart reproducibly (#1364)

Two helm template renders of the aether chart with the same values are byte-identical, with one exception: with the default controller.webhook.spire=false the chart generates the controller's self-signed webhook CA and serving certificate, and a render without a cluster generates a new pair every time. The lines that differ are exactly six: the Secret's ca.crt, tls.crt and tls.key, and the caBundle of the three webhooks. Nothing else in the chart is generated at render time (//charts/aether:aether_reproducible_masked_test fails if that stops being true).

To compare two renders (two chart versions, two values files, a change under review), mask those six lines on both sides:

mask() { sed -E 's/^([[:space:]]*(ca\.crt|tls\.crt|tls\.key|caBundle):).*$/\1 <masked>/'; }
helm template aether "$OLD_CHART" -n aether-system -f values.yaml | mask > before.yaml
helm template aether "$NEW_CHART" -n aether-system -f values.yaml | mask > after.yaml
diff before.yaml after.yaml

or render both sides with --set controller.webhook.spire=true, which renders no certificate at all. That is only a way to diff, not a way to run: it also drops the Secret and adds the controller's SPIRE flags, and it renders the controller's ClusterSPIFFEID only when controller.webhook.clusterSpiffeID.className is set (empty by default), so the flag alone does not give the controller a usable SPIRE identity. Use the mask when the change under review touches the webhook.

A real install or upgrade is not affected. helm upgrade looks the Secret up in the cluster and renders the certificate it finds, so an upgrade changes neither the Secret's data nor any caBundle, and the controller keeps serving the same certificate. (The Secret object is still patched by each release, because its helm.sh/chart label changes; its data is not.) What does not look the Secret up, and therefore shows a new certificate every time:

• helm template, a client-side helm upgrade --dry-run, and helm diff unless it is run with --dry-run=server. Use the mask, or the server-side dry run.
• A GitOps tool that renders with helm template and applies the result (Argo CD). It would apply a new CA on every sync: serve the webhook with SPIRE there (controller.webhook.spire=true and a controller.webhook.clusterSpiffeID.className, or a ClusterSPIFFEID of your own that grants the controller the webhook Service's DNS names; see docs/configuration.md). Flux's helm-controller runs a real upgrade and is not affected.

To check on a cluster that an upgrade left the certificate alone, hash it before and after (never print it):

kubectl -n aether-system get secret aether-controller-webhook-cert \
  -o jsonpath='{.data}' | sha256sum
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration \
  -l app.kubernetes.io/name=aether-controller \
  -o jsonpath='{range .items[*].webhooks[*]}{.clientConfig.caBundle}{"\n"}{end}' | sha256sum

The chart generates a new pair in three cases only:

Case	What the upgrade does
No Secret (first install, or you deleted it)	new CA and certificate; caBundle = the new CA
The Secret has an empty or missing ca.crt, tls.crt or tls.key	new CA and certificate. Until chart 2.4.16 only a missing ca.crt did that: with an empty tls.key every upgrade failed with Secret "aether-controller-webhook-cert" is invalid: data[tls.key]: Required value until someone deleted the Secret
controller.webhook.certRotation set to a value the Secret was not generated for	new CA and certificate, the value stamped on the Secret as aether.io/webhook-cert-rotation

Rotating. The pair is valid for ten years and nothing renews it. To rotate, set controller.webhook.certRotation to a new value and upgrade:

helm get values aether -n aether-system -o yaml > values.yaml   # never --reuse-values
helm upgrade aether oci://quay.io/aethermesh/chart-aether --version "$AETHER_VERSION" \
  -n aether-system -f values.yaml --set-string controller.webhook.certRotation="$(date +%F)"

Afterwards leave the value set or remove it: neither rotates again, and the Secret keeps its stamp when the value is removed, so putting the same value back later (a reverted values change) does not rotate either. Only a different, non-empty value does.

After that upgrade the webhooks trust both CAs (caBundle holds the new one and the one being replaced), so the certificate the controller is still serving keeps verifying. The controller reloads the new certificate from its mounted Secret by itself, once the kubelet has synced the volume (a minute or two); kubectl -n aether-system rollout restart deployment/aether-controller does it at once. The next upgrade renders the new CA alone, so do not run it before the controller has reloaded. If it does happen, the webhooks fail open (failurePolicy: Ignore) until the reload: pods admitted in that window are not mutated (no mesh injection, no ndots, no identity gate).

The rotation is not gap-free, so do it in a quiet moment and check the upgrade succeeded. The Secret and the webhook configurations are separate objects and Helm patches the Secret first (its kind order). Normally the webhook configurations follow within the same upgrade, long before the kubelet delivers the new Secret to a running pod. But if the upgrade fails in between, or a controller pod starts in between, that controller serves the new certificate while the webhooks still trust only the old CA, and they fail open until the upgrade completes. If an upgrade with a new certRotation fails, re-run it before anything else. Closing the gap needs the new CA trusted one upgrade before the new certificate is served; the chart does not stage that.

Pre-flight: node headroom before a roll (#812)

Do this before any helm upgrade that rolls the DaemonSets. A roll is a scheduling event, and on a node that is already tight the scheduler — not the code — decides whether it succeeds.

Two distinct exposures, and only one of them is fixed in the chart:

• Edge (Deployment). Fixed by default since #812: with replicaCount >= 2 the strategy is surge-free (maxSurge: 0, maxUnavailable: 1), so no extra 400m pod has to be placed and the scheduler is out of the rollout path. If you override edge.rollingUpdate back to a surging strategy, this pre-flight is mandatory — on rev214 that surge pod was preempted on a node at 84 % CPU requests and wedged the roll.
• The DaemonSets (aether-agent, aether-proxy, aether-mesh-dns). The proxy and mesh-dns always surge, and so does the agent when agent.updateStrategy.surge is on (proposal 041). A surge pod needs its whole request on top of the old pod's. Not fixed and not fixable by a strategy: a DaemonSet pod must land on its node or not at all. The replacement needs the departing pod's request back, and if anything on that node is stuck Terminating it is still holding its CPU request — at which point the scheduler will not even preempt (preemption: not eligible due to a terminating pod on the nominated node). That is #796.

# 1) Rank nodes by CPU requests. Anything at/above ~85 % is where a roll stalls.
#    Sort on the percentage as a NUMBER: a lexical sort puts "(100%)" below
#    "(58%)" and hides the one node you are looking for.
kubectl get nodes -o name | sed 's|node/||' | while read -r n; do
  kubectl describe node "$n" \
    | sed -n '/Allocated resources/,/Events/p' \
    | awk -v n="$n" '/^  cpu /{pct=$3; gsub(/[()%]/,"",pct); printf "%6.1f  %-14s %s %s\n", pct, n, $2, $3}'
done | sort -rn

# 2) On any node above the line, name what is actually holding the requests, so
#    the decision is "move this tenant" rather than "hope it lands elsewhere".
#    QUOTE the -o argument: the unquoted [*] is a glob and zsh fails the
#    command outright ("no matches found") before kubectl ever runs.
kubectl get pods -A --field-selector spec.nodeName=<node> \
  -o 'custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu,PRIO:.spec.priorityClassName' \
  | sort -k3 -hr | head -20
#    A multi-container pod prints its requests comma-joined (`500m,10m`, the
#    proxy), so read that column rather than trusting its sort position.

# 3) Nothing may be stuck Terminating anywhere — one of these makes its whole
#    node unrollable regardless of headroom.
kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded \
  -o wide | grep -i terminating

Verdict. All nodes comfortably under the line and nothing Terminating → roll. A node over it → either move the tenant workload off first (the platform did exactly this on 2026-09-19, spreading the observability stack and taking node A from 88 % to 66 %), or accept that that node's DaemonSet pods may sit Pending and watch them specifically. A pod stuck Terminating → resolve it first via §8 "A pod is stuck Terminating"; do not start a roll on top of it.

Load generators must not be the victim. A soak's priority-0 load generator was picked as the preemption victim twice on a dense node and cost the soak its data; since #811 the load generators run under a PriorityClass of their own (the soak is run by an external soak harness, maintained outside this repository, and the class is that harness's object). Any new load or probe workload needs a PriorityClass for the same reason — an evicted generator looks exactly like a passing test.

Agent surge roll (proposal 041)

agent.updateStrategy.surge (chart >= 2.3.0, default false) rolls the agent DaemonSet with maxSurge: 1, maxUnavailable: 0 instead of delete-then-create. It removes the pod replacement and the new agent's startup from the window in which the node's proxy has no ADS stream (#1123): the gap becomes the proxy's reconnect backoff plus a small storage diff.

Turning it on: two upgrades, never one.

# 1. Upgrade to chart >= 2.3.0 with surge OFF (the default). Delete-then-create:
#    every agent comes up holding the node lock, and the registrar learns the
#    per-agent watch key.
helm upgrade aether <chart> -n aether-system -f values.yaml --set agent.updateStrategy.surge=false
kubectl -n aether-system rollout status ds/aether-agent
kubectl -n aether-system rollout status deploy/aether-registrar
# 2. Only then turn surge on. Changing only the strategy rolls nothing; the
#    next agent roll is the first surge.
helm upgrade aether <chart> -n aether-system -f values.yaml --set agent.updateStrategy.surge=true

Turning surge on in the same upgrade that first ships it is unsafe. That surge would run the new agent beside an old one, and three things go wrong:

• An older registrar ignores the watch instance field. It keys watch streams by cluster/node alone, so the owner's and the standby's WatchEndpoints streams replace each other. Each replacement is a DataLoss close and a full resync, and they ping-pong for the whole overlap. The registrar Deployment rolls in the same upgrade, so this cannot be ordered away.
• The old agent holds no lock. The standby falls back to probing the node sockets. An old agent that restarts between those probes can rebind xds.sock/cni.sock after the standby has taken the node. When its pod is deleted, Go unlinks both paths by name, and the node is left with no sockets while the new agent runs.
• The old agent's last flush can land after the takeover. Without a lock it flushes the observed upstreams on its way out, possibly after the new agent has already written them.

Do the node-headroom pre-flight below first. With surge each node briefly runs two agents, so it needs room for a second agent.resources request (200m CPU, 96Mi since chart 2.4.2; 64Mi before, see The node agent is OOMKilled). The pod is system-node-critical and preempts if it has to. If it cannot be placed, it stays Pending and the roll stalls on that node, visibly. Nothing breaks: the old agent keeps serving.

What the overlap looks like. Per node, in the new agent's log:

another agent owns this node; starting as a standby: building everything, binding nothing until its lock is released
... (the usual startup: SVID, registry connected, client certificates)
node lock acquired: the previous owner is gone; taking the node over   standby=6.2s
takeover step done   step="merge the previous agent's persisted node state"
takeover: applied the previous agent's CNI ADD/DEL from the overlap   added=1 updated=0 removed=1
this agent owns the node; binding its sockets and starting its writers   takeover=1ms

takeover: local storage unchanged during the overlap is the common case. The pod is Ready once its first snapshot is complete (standby readiness check). The DaemonSet controller then deletes the old pod, the old agent exits in 20-100 ms, and the kernel hands the lock (/run/aether/agent.lock) to the standby. Proxy side, control_plane.connected_state drops to 0 for about 0.1 s (on kind: median 0.08 s, p99 0.24 s, measured by the kind reproduction agent-restart-gap.sh, maintained with an external harness, outside this repository):

kubectl -n aether-system exec <proxy-pod> -c aether-proxy -- \
  curl -s 'http://127.0.0.1:9901/stats?filter=^control_plane.connected_state$'

A standby that never takes over is waiting on an old agent stuck Terminating. It stays Ready and binds nothing; the old pod's terminationGracePeriodSeconds (30 s) bounds the wait, then the kubelet kills the process and the kernel releases the lock. If the old pod is stuck past that (a wedged kubelet, #796), the old process may already be gone. Check:

kubectl -n aether-system exec <new-agent-pod> -c agent -- /agent-ready --path=/readyz   # prints the failing check, if any
kubectl -n aether-system logs <new-agent-pod> -c agent | grep -E "standby|node lock|owns the node"

A standby still logging another agent owns this node with no node lock acquired means some process on the node still holds the lock. On the node, fuser /run/aether/agent.lock names it. Do not delete the lock file: an unlinked lock file is a different inode from the one the next agent opens, and the two would both believe they own the node.

kubectl delete pod of an agent (any strategy) is still delete-then-create: the lock is free when the new pod starts, so it owns the node at once and serves exactly as before (no takeover step runs).

Turning it off (surge: false) takes effect on the next roll. The lock stays: it costs nothing when nothing contends for it.

Version-ordering constraint: issue #815 (per-source client certificates)

**You may not upgrade a cluster from a chart/agent older than 0.92.27 (the

819 + #821 "release one" build) straight to 0.92.28 or newer.** Go through a

release-one build first, let every node's proxy pick up the new listeners, and only then upgrade again. Skipping the step is not a hard failure, but for the length of one config transition a pod's egress can present the node identity instead of its own, which inbound RBAC / ext_authz / peer-identity expectations will see.

Why: the client certificate a pod's egress presents is selected by a cluster transport-socket matcher that reads a filter-state key the pod's listener stamps. Release one (0.92.27) started stamping aether.source.spiffe_id alongside the old aether.network.network_namespace; release two (0.92.28) moved the matcher onto it. The agent publishes listeners and clusters in the same snapshot, but LDS and CDS are separate xDS responses with no ordering guarantee, so a release-two cluster can briefly face a pre-release-one listener. That connection matches nothing and takes on_no_match = the node identity.

A third release that dropped aether.network.network_namespace from the listeners was evaluated on 2026-09-19 and closed: on a listener that key carries no per-pod state (its value is a constant format string), so removing it would save about 240 bytes per mesh-originating filter chain and nothing else — while costing another full re-key of every mesh pod's filter chains to deploy. The netns key is still stamped (the access log's source_netns reads it).

While the listeners stamped both keys, rolling back below 0.92.28 was safe: a proxy whose listeners stamp both keys matches correctly against clusters from either side of release two. This issue needed exactly that once — release one was rolled back on the reference cluster on 2026-09-19. That safety ended with #1165 (next section). Verification and the runtime failure signature are under "#815 release two" in §8.

Rollback floor: chart 0.93.0 (issue #1165)

After the chart that ships #1165, rolling back to a chart older than 0.93.0 is no longer hitless. 0.93.0 (2026-09-20) is the chart that shipped #842, the per-connection certificate selector; use 0.93.1 or newer as the floor in practice, since 0.93.0 itself carried the #842 SDS-dedup stall that 0.93.1 (#865) fixed. Rolling back to any chart >= 0.93.0 is unaffected.

Why: from 0.92.27 the source pod's SPIFFE ID was stamped under aether.source.spiffe_id, and every cluster before 0.93.0 selects its client certificate with a transport_socket_matcher that reads that key. #842 moved selection to the filter_state_override certificate mapper, which reads envoy.tls.certificate_mappers.on_demand_secret, and kept stamping the old key for one release purely so a rollback stayed hitless. #1165 stopped stamping it. A node proxy running post-#1165 listeners that is handed pre-0.93.0 clusters during a rollback matches nothing and takes on_no_match — the agent's own SVID (spiffe://<td>/ns/aether-system/sa/aether-agent, #825) — so every pod's egress presents the wrong identity until the old agent's listeners land too (LDS and CDS are separate responses with no ordering guarantee). ext_authz / RBAC / peer-identity checks at the destination will see it.

If you must go below 0.93.0: step through a 0.93.x / 0.94.x chart first (its listeners stamp both keys), let every node's proxy take the new listeners, then roll back again.

Nothing else in the observable surface changed. The access log's source_spiffe_id attribute (HTTP aether_access_logs and L4 aether_l4_access_logs streams) keeps its name and value; only the filter-state key it reads moved to the mapper key, so VictoriaLogs queries and dashboards that filter or extract on source_spiffe_id need no change. The proposal 038 QUIC selection matcher reads the mapper key too, so the arm a request takes and the client certificate it presents now come from one value. Rolling forward re-keys every mesh-originating filter chain (one set_filter_state entry fewer) and every QUIC-selecting route, so those chains drain once on --drain-time-s.

There are also two standalone charts, installed independently: prober (charts/prober) — the external mesh-availability prober (proposal 013; its flags, metrics and deployment shape are in configuration.md § prober) — and udsecho (charts/udsecho) — the UDS validation workloads (proposal 034) that exercise both socket-delivery paths (annotation and EndpointPolicy) under continuous mesh traffic, on the csi.aether.io carrier (udsecho 2.x needs aether 2.x; see Upgrading to chart 2.0.0 below).

See charts/README.md for chart layout, image mirroring, and the --stamp versioning scheme, and getting-started.md for the full install + onboarding walkthrough.

Moving or renaming a binary inside an image — the Bazel image rules, a tars_layer entry (/proxy-ready, /mesh-dns-ready, /opt/cni/bin/aether-cni), or proxy/ — breaks profile symbolisation, silently: flame graphs decay to hex addresses with no error and no failing test. Adding a binary to an image is the same trap; it profiles as hex until it is listed. See observability/profiling-symbols.md for the path table that has to be updated alongside it.

Upgrading to chart 2.0.0: the UDS carrier cut-over (proposal 039 Phase 2)

Breaking, no compatibility window. Chart 2.0.0 makes the csi.aether.io CSI volume the only carrier for a UDS-delivered workload socket (proposal 034). The emptyDir carrier, the proxy's /var/lib/kubelet/pods mount, the agent's --kubelet-pods-dir and the proxy.udsWorkloads value are gone; udsCsi.enabled (the plugin DaemonSet + CSIDriver, the agent's --uds-csi-root, the proxy's read-only mount of udsCsi.root) is now on by default. TCP-served workloads are untouched.

Before upgrading the chart, every UDS workload must switch its socket volume (or ship the switch in the same rollout — the old carrier stops resolving the moment the agent rolls):

spec:
  securityContext:
    fsGroup: 65532                  # NEW, required: the socket dir is root:<fsGroup> 2770
  volumes:
    - name: s                       # the <volume> the annotation / EndpointPolicy names
      csi: {driver: csi.aether.io}  # was: emptyDir: {} (or {medium: Memory})

Find them first — every pod carrying the annotation, and every EndpointPolicy:

kubectl get pods -A -o json | jq -r '.items[]
  | select(.metadata.annotations["endpoint.aether.io/uds-socket"] != null)
  | "\(.metadata.namespace)/\(.metadata.name) \(.metadata.annotations["endpoint.aether.io/uds-socket"])"'
kubectl get endpointpolicies -A

A workload that misses the switch is not deleted, but it stops being delivered:

• new pods (a roll, a reschedule) carrying the annotation are denied by the controller's pod webhook, with a message naming the fix (… cannot be delivered (not_csi): … change the volume's source to csi: {driver: csi.aether.io} and set the pod's securityContext.fsGroup);
• running pods — and policy-declared pods, which admission cannot see — fall back to TCP; a UDS-only app has nothing listening there, so its endpoint stays unpromoted, counted as aether_agent_uds_resolve_failures_total{reason="not_csi"}.

Verify after the upgrade (every reason is exported at zero, so an absent series means the agent is not exporting, not "no failures"):

# expect 0 for every reason; non-zero names what is wrong (see §8)
kubectl get --raw "/api/v1/namespaces/aether-system/pods/<agent-pod>:8080/proxy/metrics" \
  | grep '^aether_agent_uds_resolve_failures_total'
kubectl -n aether-system rollout status ds/aether-uds-csi
kubectl get csinode -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.spec.drivers[*].name}{"\n"}{end}'   # lists csi.aether.io

Talos needs nothing extra (the default udsCsi.kubeletRoot, /var/lib/kubelet, is right). On k0s / microk8s set udsCsi.kubeletRoot. Rolling back to 1.x requires reverting the workloads to emptyDir too.

East-west QUIC (proposal 038 Phase 4): prerequisites, verifying, escape hatch

East-west QUIC is unconditional. Every mesh pod has an HTTP/3 inbound on UDP:18008 beside the TCP one (#953: same SVID, same client-certificate requirement, same SAN pin), and every node proxy dials every service in its dependency set over HTTP/3 from every local ServiceAccount that calls it (#956; the per-source clusters are built on first use, #1020). There is no value or flag for it: the per-destination allow-list (a chart value + agent flag) was a proving gate for QUIC on a real cluster and was removed by #979 (33ff5e9, chart 1.0.12; decision 2026-09-26: no opt-in for QUIC). The first proving soak (2026-09-27) did not pass — k6 saw 1,060 × 503 NC at first use (#1008, closed) — and #979 merged on 2026-09-29; the re-soak of the merged build: 2026-09-30, chart 1.0.13-84fb413 (T0 10:57:18Z): every QUIC gate passed; the run failed only on liveness, 9 of 719,984, with a cause outside QUIC. - The two defects that failed the 09-29 run did not recur. - No agent logged a pruned persisted east-west QUIC pairs line, and all 30 fresh xDS streams re-stated their held twins as served. - There was no source-side 503 NC in the window. - There was no Envoy crash across 6 hot restarts per node. - The new-ServiceAccount steps, wedge gates, stateless-reset gate and L4 gates were all at zero. h3 costs 1.18× h2 per request. - The 9 liveness errors are a new race (#1085). - In the TRIPLE, node E's agent was down while its proxy forked a successor. - The successor's CDS and LDS initial fetches timed out (15 s each), and Envoy started workers with no listeners. - The parent then drained, so 127.0.0.1:18081 refused new connections for 1.6 s. - A successor that logs initial fetch timed out for …Listener next to starting workers is this case. - k6 had 147 failures in 9.18 M requests. - 141 were 503 NC first-use timeouts at loader start, before T0 (#1086). - 6 were 504 UT to a terminating pod of one of the soak's services in the TRIPLE (#1087). The one exception is a service with any endpoint behind the east/west waypoint (019): it stays h2, because the waypoint tunnel has no QUIC leg.

Mesh-wide SPIRE prerequisite. Every workload SVID must carry two DNS SANs — <sa>.<ns>.<meshDomain> and *.<sa>.<ns>.<meshDomain> — until envoyproxy/envoy#47740 is in a plain proxy pin. Envoy's QUIC client verifies the SNI (<port>.<sa>.<ns>.<meshDomain>) against the leaf's DNS SANs after the SPIFFE pin succeeds (aether#957); without the SANs every HTTP/3 handshake fails closed (503 at the source, Leaf certificate doesn't match hostname / QUIC_TLS_CERTIFICATE_UNKNOWN in the proxy log at quic:info). The proxy pin since chart 0.95.3 carries #47740 as a patch (#972), which defers that check to the SAN pin, so today a missing SAN does not fail — but the carry is transitional and a pin bump can drop it, so the SANs stay required until upstream ships it. On the spiffe/spire chart:

spire-server:
  controllerManager:
    identities:
      clusterSPIFFEIDs:
        default:
          dnsNameTemplates:
            - "{{ .PodSpec.ServiceAccountName }}.{{ .PodMeta.Namespace }}.<meshDomain>"
            - "*.{{ .PodSpec.ServiceAccountName }}.{{ .PodMeta.Namespace }}.<meshDomain>"

Changing the entry re-issues every workload SVID on the agents' next fetch; no pod roll. Order of operations on a live cluster upgrading from a chart that still had the allow-list: SPIRE first, wait for the agents' envoy_sds_*_version to move on every node, then the chart — the upgrade turns QUIC on for every destination at once.

Network prerequisite. UDP:18008 must be open wherever TCP:18008 is: from every node (the source proxy is host-network) to every mesh pod IP. A NetworkPolicy or host firewall that allows only TCP:18008 breaks every mesh request, not just a few.

What to expect. Every service in a node's dependency set is QUIC-eligible (default entry only, identity-ready, never a service with an endpoint behind the east/west waypoint), but twins are demand-scoped (#1020): a node builds a quic:<svc>.<ns>.<domain>@<ns>/<sa> cluster only for a (source ServiceAccount, destination) pair that has actually dialled. The route to an eligible destination carries one selection arm per local ServiceAccount (a matcher cluster specifier keyed on the connection's envoy.tls.certificate_mappers.on_demand_secret filter state — the same source-identity stamp the certificate mapper reads; aether.source.spiffe_id before #1165), each naming that source's twin, built or not. A source's first request resolves to a twin the proxy does not have yet; the HCM's on_demand filter asks the agent for it by name over ODCDS; the agent checks the name (destination in the dependency set, source a ServiceAccount with a pod on the node), records the pair and publishes the twin with its own load assignment; the paused request resumes on it, over HTTP/3. That costs one local round trip. It was 17–24 ms in //agent/test/mtlspool's TestOnDemandQUICTwinPerPair, and every later request from the pair skips it. A connection with no identity stamp (stamped before the trust domain was known) takes on_no_match, the h2 cluster. A GAMMA (HTTPRoute) rule whose single backendRef is the parent rides QUIC too; a weighted split stays h2 (#961) and never fetches a twin.

The agent logs east-west QUIC fan-out quic_clusters=N observed_pairs=P local_identities=I awaiting_client_cert=W whenever N or W changes, and for each new pair observed east-west QUIC pair (ODCDS); building its twin cluster=…. N is the number of observed pairs whose source is still local, so it is at most I × S, where S is the node's eligible dependency-set services (the line no longer carries a service count since #979), and on a real fleet much less (rev242, with an allow-list of 2: 118 possible, 2 used; without the allow-list the ceiling on the reference cluster is 8–14 SAs × ~19 services ≈ 150–270 per node, the pairs actually dialled stay the same).

What a healthy node reads. observed_pairs ≈ the (source ServiceAccount, destination) pairs that carry traffic on that node: under a soak, its load generators and dialers plus the prober, so single digits per node, and quic_clusters equal to it. observed_pairs that is a whole multiple of local_identities on every node (every local SA × the same destinations) is the #1033 red reading (rev245, 2026-09-28: 24/24/12, 18/18/9, 28/28/14, 20/20/10, 20/20/10), not a busy fleet.

Reading twin count against pairs. On one node, the quic: clusters in the proxy admin (/clusters) must be exactly the pairs with traffic. Pairs are persisted beside the observed upstreams (state/observed-upstreams.json, quic_pairs), so they survive an agent or agent+proxy roll. There is no idle expiry: nothing in xDS tells the agent that a twin stopped carrying traffic. A pair is pruned when its source ServiceAccount has no pod left on the node, or when its destination leaves the dependency set, or by the post-start prune below. In the first two cases, a pair whose twin the proxy fetched on demand loses its twin but is kept dormant and republished when it is valid again (#1036, "Stranded twin" below). A pair that did receive traffic therefore keeps its twin while both ends stay put, even if it goes quiet. envoy_cluster_manager_active_clusters rises by one when a pair first dials, falls when a pair is pruned, and is otherwise flat. A step that tracks pod churn rather than new callers is the pre-#1020 fan-out and a regression.

Agent restarts (#1033). A restarted agent opens a fresh xDS stream, and the running proxy re-states its twins in the stream's first CDS request. Only the persisted pairs carry over; the re-statement is read two ways:

• A twin the proxy merely holds (initial_resource_versions, delivered by the wildcard, e.g. built up front by an older agent) admits nothing. If its pair is not persisted it is answered absent (removed_resources) and the proxy drops it; the next request that routes to it opens an on-demand fetch, one ~20 ms round trip, no 503. The #1032 agent re-admitted these, which is how rev245 persisted the whole SAs × destinations fan-out on every node.
• A twin the proxy re-subscribes by name holds a live on-demand subscription: a request routed to it at some point in this proxy's life. Its pair is admitted. This is not optional. Envoy's ODCDS manager keeps one subscription per name for the life of the process and answers every later request for the name "already subscribed, skipping", so a subscribed twin answered absent is stranded: every request from that pair 503s NC at the 2 s on_demand timeout until the proxy restarts. //agent/test/mtlspool's TestOnDemandQUICSubscribedTwinsServedAfterAgentRestart/strand_control shows it.

The agent logs one line per fresh stream: fresh xDS stream re-stated QUIC twins: held-only twins admit nothing, live on-demand subscriptions are served resubscribed=R resumed_pairs=P held_only=H held_served=S answered_absent=A.

Post-start prune of pairs with no evidence of use (--east-west-quic-pair-fetch-window, default 1h). A persisted pair that no agent process has ever seen evidence of use for is dropped with its twin once the window has elapsed (checked on the one-minute prune tick). 0 disables the prune. Evidence of use is any of:

• an on-demand fetch of the twin (first use);
• an on-demand subscription for it, held by a live stream or re-subscribed on a fresh one;
• the twin re-stated as held (initial_resource_versions) on a fresh stream while the agent serves the pair (#1073).

The evidence is persisted with the pair (demand_confirmed in observed-upstreams.json), so it survives agent restarts and node reboots. A pair with evidence is never pruned by the window; it goes only on removal evidence (source left the node, destination left the dependency set).

What is left for the window is the fan-out rev245 persisted, and any pair persisted before #1073 that the proxy neither holds nor fetches. The agent has no traffic signal of its own: it makes no admin calls, and a twin the proxy already holds is never fetched again. So a held twin counts as used.

#1073, why "fetched" was the wrong signal. After a proxy restart, every twin reaches the new generation through the wildcard, with no fetch and no subscription, and carries its traffic that way. On an agent-only roll the fresh stream re-states those twins as held-only. The pre-#1073 rule ("no on-demand fetch since agent start") then pruned every one of them exactly one window later, while they carried traffic:

• the #979 proving soak saw about 200 × 503 NC per burst fleet-wide;
• Envoy crashed on one node (#1074, SIGBUS removing a QUIC cluster with a live connection; fixed by the carried patch #1077, chart 1.0.13 — see "Envoy SIGBUS/SEGV in QuicConnection after an h3 cluster removal (#1074)" below).

A held twin whose pair the agent does not serve still admits nothing and is answered absent (#1033). The fresh-stream line's held_served=S counts the held twins that were confirmed. The first time a pair is confirmed that way, the agent logs:

confirmed east-west QUIC pairs the proxy holds without an on-demand subscription: never pruned by the fetch window count=N held_served=S

When the window prunes anything, the agent logs one line per node:

pruned persisted east-west QUIC pairs with no evidence of use: never fetched, subscribed or held by the proxy count=N remaining=M window=1h0m0s pairs=[…]

It is followed by a east-west QUIC fan-out line with the new, smaller quic_clusters. Builds before #1073 logged … with no on-demand fetch since agent start instead.

If that line shows up after an agent roll on a node whose proxy kept running, with 503 NC on QUIC destinations right after it, that is #1073 again: the prune removed twins in use. File it with the fresh-stream line from the same agent generation.

Stranded twin: 503 NC at 2 s for a source that came back (#1036). Envoy's ODCDS manager keeps one subscription per cluster name for the life of the process. When the agent removes a twin the proxy fetched on demand, Envoy drops the cluster and keeps the subscription. Every later on-demand request for that name is skipped inside the proxy and never reaches the agent. The pair is then stranded until the proxy restarts. The failure signature:

• Access log. reporter:source, response_code:503, response_flags:NC, duration_ms ≈ 2000 (the on_demand timeout) on every request from one source ServiceAccount to one QUIC destination, starting when a pod of that ServiceAccount came back to the node. upstream_cluster is - on an NC line, so scope by authority and the source pod.
• Proxy debug log. ODCDS-manager: resource quic:<svc>.<ns>.<domain>@<ns>/<sa> is already subscribed to, skipping for each of those requests.
• Agent. Nothing for that name: no observed east-west QUIC pair (ODCDS) line, no refusal, and aether_agent_quic_twin_refused_total flat, because no request arrives. envoy_cluster_manager_odcds_init_fetch_timeout_total does not move either. The timeout is the HCM on_demand filter's, not the subscription's.

Until #1036 this was reachable on every Deployment roll. A pair was forgotten when its source ServiceAccount's last pod left the node, or when its destination left the dependency set (or, before #979, the allow-list). The next pod of that ServiceAccount on the node then routed to a name Envoy would never ask for again. //agent/test/mtlspool TestOnDemandQUICDormantTwinRepublishedWhenSourceReturns/forget_control reproduces it: status=503 … in 2.000099268s, and no CDS request reaches the control plane.

DC 200s on a QUIC destination at a source-proxy roll (#1009) are benign when the line is DC + downstream_remote_disconnect + 200 + the clean-line bytes_sent: the HTTP/1.1 client closed after a complete body before the h3 FIN was decoded. upstream_rx_ms and downstream_tx_end_ms read - on such a line. The soak (run by an external soak harness, maintained outside this repository) grades its DC lines by this rule; any other DC is a real failure.

Why the agent never forgets a subscribed pair. The agent tracks which twins the proxy holds an ODCDS subscription for: those it asked for by name, and those it re-subscribes on a fresh stream. When such a pair loses its source or its destination, it goes dormant:

• Leaving. The twin still leaves the snapshot, exactly as before. The pair moves to dormant_quic_pairs in state/observed-upstreams.json, and the prune line reads pruned east-west QUIC pairs … (source_left_node, dormant).
• Returning. The pair is valid again when a pod of the ServiceAccount is back on the node, or the destination is back in the dependency set. The same snapshot then republishes the twin and its load assignment, with no request. The subscription never closed, so its delta watch is still on the stream and Envoy takes the pushed cluster. The agent logs republished dormant east-west QUIC pairs: the proxy still holds their on-demand subscriptions, so the twin is pushed with no request count=N pairs=[…]. The returning source's first request is 200 over HTTP/3 in milliseconds (6.7 ms in the gate above).
• Pruning. A dormant pair is dropped only when no live proxy generation holds the subscription. Subscriptions are tracked per xDS stream, because one Envoy process speaks one ADS stream and the node proxy's Node does not tell hot-restart generations apart (#1052). A pair is held by the stream that re-subscribed it in its first CDS request, or that asked for it since. It is pruned when a fresh stream's re-statement leaves no live stream holding it, or when the stream that held it ends while another stream is live (the draining parent of a hot restart exiting). The last live stream ending prunes nothing, since the proxy may be reconnecting. The agent logs pruned dormant east-west QUIC pairs: no live proxy stream holds an on-demand subscription for their twins stream=S count=N dormant=M pairs=[…] or … the proxy generation that held their on-demand subscriptions ended its stream while a newer one is live …. After that the name is free, and the next request that routes to it is an ordinary first use. Before #1052 an agent restart that landed mid hot restart could leave dormant pairs no live generation subscribed to: the child's fresh stream (naming nothing) pruned them, and the parent's stream seconds later (naming them) parked them again, past the parent's exit.
• Refusals. A first-use request the agent has to refuse opens a subscription too: a source_not_local race, say, with a pod whose records are not loaded yet. So the pair is parked dormant as well, and only the refused request 503s. A re-subscribed pair that is not servable on a fresh stream logs kept east-west QUIC pairs dormant … count=N.
• The fetch-window prune. It never touches a dormant pair, because a dormant pair is not a served pair. It also never touches a served pair that a live stream holds a subscription for. And it never touches a pair with persisted evidence of use: fetched, subscribed, or re-stated as held (#1073).

A dormant pair costs a few bytes in the state file and nothing in the proxy. It is bounded by the names the proxy has subscribed to since its last restart.

If you see the signature anyway: restart the node's proxy (a hot restart, kubectl -n aether-system delete pod <aether-proxy pod>). The new generation holds no subscription, so the next request fetches the twin (~20 ms). Then file it. The agent's republished dormant / pruned dormant lines around the source's return say which half failed.

# twins per node, then the pairs among them that carried traffic in the last hour.
# Under steady load the two are equal; the difference is pairs that went quiet
# (kept by design, see above), never pairs that did not dial.
count by (node) (envoy_cluster_upstream_rq_total{aether_cluster=~".+@.+"})
count by (node) (increase(envoy_cluster_upstream_rq_total{aether_cluster=~".+@.+"}[1h]) > 0)
envoy_cluster_manager_active_clusters

The failure mode of the first request. A twin the agent refuses, or one that takes longer than the on_demand timeout (2 s, onDemandClusterTimeout, the same bound the mesh catch-all's cold path uses), fails that request with 503 NC. Envoy's API has no per-route fallback from an on-demand miss to the h2 cluster: the matcher's action names one cluster and nothing else. Refusals are counted by the agent as aether_agent_quic_twin_refused_total{reason}, and each one is a request that 503'd. source_not_local and destination_not_in_dependency_set are races with a pod leaving or the destination's idle TTL expiring (destination_not_quic_enabled is gone with the allow-list, #979). malformed_name means a route and the agent disagree on the naming. On the proxy, watch the ODCDS subscription:

# on-demand CDS fetches that timed out (a 503 NC per paused request): MUST stay 0
sum(increase(envoy_cluster_manager_odcds_init_fetch_timeout_total[1h]))
# agent refusals, by reason
sum by (reason) (increase(aether_agent_quic_twin_refused_total[1h]))

The one gap that remains by design is an agent that is down while a new pair dials. That is the same exposure the capture catch-all already has (#682): the paused request 503s at the timeout, and the next request retries the fetch.

The fast path, and why a twin waits for its source's certificate (#1049). The agent side of a first use is not the slow part. It publishes the twin 2-10 ms after the ODCDS request (a dormant pair returning is republished by the CNI ADD's own snapshot), with no debounce and no wait on a registry reload. What used to lose to the 2 s timeout was Envoy warming the twin. A twin names its source ServiceAccount's SVID statically in its transport socket, so a twin published before that secret is in the snapshot warms on SDS until SPIRE delivers it. On rev248 (2026-09-28 15:44Z, the soak load generators' first pods on every node right after an agent roll) that took 6.9-7.4 s from the CNI ADD, and 428 requests failed 503 NC. Two rules close it:

• An identity whose certificate is not in the snapshot yet gets no selection arm and no twin. Its requests ride the warm h2 cluster, whose certificate is fetched per connection. The snapshot that carries the certificate carries the arm, the twin and the twin's load assignment together. The fan-out line shows the wait: east-west QUIC fan-out … awaiting_client_cert=N. N above 0 for more than a few seconds is a stuck SVID (see the SPIRE sections), not a QUIC fault. Before SPIRE serves any secret at all, nothing is held back.
• A named subscribe for a twin is always answered. Envoy's ODCDS manager subscribes to a twin by name whenever a request routes to it while it is not active, including when the wildcard already delivered it. go-control-plane stays silent on an unchanged resource. The per-name subscription then waited out its 15 s initial-fetch timeout and Envoy logged cm odcds: cluster quic:… not found during on-demand discovery, which fails any request still waiting on the name. That was node C at 15:45:04.7, 15 s after its loader's first request. The agent now re-sends a subscribed twin it already sent (cache.SnapshotCache.CreateDeltaWatch).

The invariant, with #1035/#1036: for a well-formed twin name of an eligible destination, the agent answers with the twin or holds the subscription open. It never answers such a name absent while the pair is servable. A twin leaves the snapshot only in two cases. Its source or destination has gone, and then the pair is dormant and republished on return (#1036). Or the proxy holds the twin through the wildcard alone, with no subscription (#1035).

Verifying. Per twin, the admin /clusters host rows (<cluster>::<ip:port>:: rq_total::N) are the ground truth. In Prometheus the twins carry their own stats key <ns>/<svc>@<ns>/<sa> (#960):

# HTTP/3 requests per (destination, source ServiceAccount)
sum by (aether_cluster) (rate(envoy_cluster_upstream_rq_total{aether_cluster=~".*@.*"}[5m]))
# h2 + h3 together for one destination
sum(rate(envoy_cluster_upstream_rq_total{aether_cluster=~"aether-test/svc-1(@.*)?"}[5m]))
# a twin that cannot connect: the #957 DNS-SAN shape, or UDP:18008 blocked between nodes
increase(envoy_cluster_upstream_cx_connect_fail{aether_cluster=~".*@.*"}[5m])
# a twin whose endpoints never arrived (#1008): MUST be 0, fleet-wide, always
sum(envoy_cluster_init_fetch_timeout_total{aether_cluster=~".*@.*"})

Each twin subscribes to its own EDS resource, named after the twin cluster (quic:<svc>.<ns>.<domain>@<ns>/<sa>), and the agent publishes the h2 cluster's load assignment under that name too. In /config_dump a twin's eds_cluster_config.service_name equals its own cluster name, never the bare <ns>/<svc> the h2 cluster uses; if it ever does again, see "QUIC twin never leaves warming" below.

On the destination, listener.inbound_<namespace>_<pod>_h3.http.inbound.downstream_rq_2xx (admin /stats) is the per-pod count of requests that arrived over HTTP/3. The kind harness e2e/eastwest-quic.sh asserts all of this end to end (E0–E5), including E4c: the node's quic: cluster count equals the (source, destination) pairs the suite drove. Its QUIC_DNS_SANS=off negative control reproduces the missing-SAN failure.

HTTP/3 per-request cost and connection counts (#1021). A same-build A/B on 2026-09-28 (rev247, 300 rps, matched 80-min no-roll windows, Pyroscope fleet envoy cores) measured an HTTP/3 mesh request at 1.18× the proxy CPU of an h2 one (10.5 ms vs 8.9 ms across both proxies) — under the ≤ 1.5× gate the allow-list drop (#979, merged as 33ff5e9) was held to (#1021, "Same-revision measurement, 2026-09-28"). The earlier ~3.3× (~11 ms vs ~3.3 ms, rev242 QUIC vs rev239 h2) compared two builds and is superseded. Grade it only with the matched-window method of the soak's QUIC per-request cost gate (the soak is run by an external soak harness, maintained outside this repository): envoy-only Pyroscope cores over the T0+6h05m→T0+7h25m no-roll window of a QUIC run and of an h2 reference run with matched per-destination rps, loaded minus idle, per request. Fleet CPU alone says nothing, because the QUIC share of the load changes between runs.

Expected upstream QUIC connections to one destination are one per (source node, source ServiceAccount that dialled it, Envoy worker that SA's app connections landed on, destination endpoint). They are NOT one per app connection. A twin does not pool per downstream connection (QUICClusterFrom pins connection_pool_per_downstream_connection off), so a load generator's 60 keep-alive connections share its node's per-worker pools. Pods of one ServiceAccount share a twin's connections, because the SA is the identity. The upper bound per destination is Σ_nodes (dialling SAs × workers × endpoints):

# live QUIC connections per twin, and the node's worker count (the per-endpoint multiplier's ceiling)
sum by (node, aether_cluster) (envoy_cluster_upstream_cx_active{aether_cluster=~".*@.*"})
max by (node) (envoy_server_concurrency)
# endpoints per destination
max by (aether_cluster) (envoy_cluster_membership_total{aether_cluster=~".*@.*"})
# density: QUIC rps per live QUIC connection (#1006: ~0.8, i.e. every request is its own flight)
sum(rate(envoy_cluster_upstream_rq_total{aether_cluster=~".*@.*"}[5m]))
  / sum(envoy_cluster_upstream_cx_active{aether_cluster=~".*@.*"})

A connection count that grows with app connections (k6 VUs, a client's pool size) rather than with SAs × workers × endpoints means a twin started pooling per downstream connection again. //agent/test/mtlspool TestQUICTwinUpstreamConnections reproduces both shapes: 12 app connections open 1 QUIC connection (one worker), 4 (four workers), or 12 (option forced on). The same counts hold on the h2 path, whose pool key has carried the source identity instead of the downstream connection since

842.

The inbound QUIC listener reads with UDP GRO (prefer_gro: true, off by Envoy default for listeners). It writes with Envoy's automatic GSO batch writer, which is used when the kernel has UDP_SEGMENT (Linux ≥ 4.18; Talos 6.18). A kernel without UDP GRO logs GRO requested but not supported by the OS once per listener and reads without it. That warning costs performance, never correctness.

If UDP:18008 is blocked, or HTTP/3 fails mesh-wide. Symptom: twins' cx_connect_fail climbing, 503s from every caller, h2 clusters idle. There is no per-destination or per-node off switch any more, by design, and none should be added. In order:

1. Fix the path — open UDP:18008 (NetworkPolicy, host firewall, cloud security group) or, for handshake failures, re-issue the SVIDs with the DNS SANs above. Both converge without a pod or proxy roll.
2. Roll the chart back to the last release that still had the allow-list (1.0.11, the release before #979): its default (an empty list) is h2 for every destination, and the twins and selection disappear on the next push. Take the values from helm get values -o yaml and pass them with -f; never --reuse-values. This is the escape hatch; there is no other.

It is not proxy-neutral. 1.0.11 pins proxy.image at a1bcf05… (#1070); main pins 71be75f… (#1078, chart 1.0.13). A plain rollback therefore rolls every node's proxy (and the edge, which runs the same image) and loses:

• the #1074 carried patch (#1077). Without it Envoy can SIGBUS/SEGV when a CDS removal drops an h3 cluster with streams in flight — and the rollback's first push removes every twin at once;
• the #1073 fix (#1076): the agent goes back to the build whose fetch-window prune removes held twins. Moot while every destination is h2, but it returns with QUIC.

To keep the proxy where it is, roll back the agent only: add the current proxy.image and proxy.supervisor.image to the -f file. The 1.0.11 chart reads the same keys, its aether.image helper uses the digest when one is set, and its proxy DaemonSet template is unchanged through 1.0.13, so with both images pinned the proxy DaemonSet does not roll at all. (Pinning only proxy.image keeps the patched Envoy but still rolls the proxy pods, because the supervisor image is in the same pod template.) The proxy half, for chart 1.0.13:

proxy:
  image:
    repository: quay.io/aethermesh/proxy
    tag: 71be75f9244c3eaddb1f05e0b9a0f08f0afcb6fa
    digest: "sha256:7cb11392fc028946e7dc5bb4a42a4882932bb214c15535a07240d3fa1dc1d167"

Copy proxy.supervisor.image (and, for a later chart, proxy.image) from helm get values <release> -n <ns> --all -o yaml.

spire.enabled=false also removes QUIC (no TLS, no QUIC) but turns off mTLS mesh-wide — it is not an escape hatch.

8. Troubleshooting

QUIC twin never leaves warming / 503 NC on a new ServiceAccount (#1008)

Symptom. The first pod of a ServiceAccount that is new on a node gets 503 with response flag NC (no cluster) for ~15 s on every request to every QUIC destination, then recovers on its own. Other callers on the node are unaffected; h2 destinations are unaffected. On the reference cluster (rev242) it was 1,060 client-visible 503/NC in 11 s when the soak's load generators started.

Since #1020 every twin is a late twin, because it is built on the pair's first request. A regression of this fix would therefore hit every new (source, destination) pair, not only a new ServiceAccount. A 503 NC that ends after ~2 s is a different failure: the on-demand fetch itself timed out or was refused. See "The failure mode of the first request" above.

Read.

# the twin sat in warming (1) from its CDS add until the timeout
envoy_cluster_warming_state{aether_cluster=~".*@.*"}
envoy_cluster_manager_warming_clusters
# and gave up waiting for its endpoints: 1 per affected twin per proxy
envoy_cluster_init_fetch_timeout_total{aether_cluster=~".*@.*"}

In the proxy log: cds: response indicates N added/updated cluster(s), then ~15 s later gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment, one per late twin. The agent is idle through the gap: nothing on the control-plane side is pending.

Cause. A twin that shares its h2 base's EDS resource name. Envoy's delta-ADS WatchMap deduplicates subscription interest per (type_url, resource name): when the twin arrives after the base is subscribed, its watch adds nothing to resource_names_subscribe, no request is sent, the control plane (correctly) sends nothing because the resource did not change, and the twin waits out its 15 s initial_fetch_timeout. At agent start base and twins arrive in one CDS response, so only a late twin — a new local ServiceAccount — is hit. Fixed in

1008: proxy.QUICClusterFrom points the twin at its own EDS name and the cache

publishes the base's ClusterLoadAssignment under it (proxy.LoadAssignmentAlias). Seeing this again means that pairing broke; //agent/test/mtlspool's TestLateQUICTwin* pair reproduces it against the pinned proxy (the negative control times out at ~15 s by design).

Invariant. A delta-ADS subscriber must never share a resource name with an already-subscribed sibling. It has bitten twice and was found latent a third time:

• SDS (#842): the on-demand certificate selector behind every static SVID reference. Fixed in #865 with its own SotW stream.
• EDS, QUIC twins (#1008): a twin behind its h2 base.
• EDS, port aliases and TCP floors (#1013): the <fqdn>:<port> alias clusters (the ODCDS cold-path authorities, e.g. :18081) and the TCP floor tcp:<fqdn> with its primary-port alias tcp:<fqdn>:<port> all subscribed to the bare <ns>/<svc> name that the default cluster holds. They usually arrived in the same CDS response as the default cluster, so nothing showed. A later one would warm for 15 s: a Service gaining a port after the node depends on it, or a service joining the capture TCP set (a TCPRoute attached) after it is in the dependency set. For a floor that means every captured TCP connection to the service is closed, because tcp_proxy has no cold path.

The fix is the same for every EDS case. The cluster's eds_cluster_config.service_name is its own cluster name, and the agent republishes the bare ClusterLoadAssignment under that name in the same snapshot pass that emits the cluster (proxy.LoadAssignmentAlias, the one helper for twins, aliases and floors). The TCP per-port clusters tcp:<fqdn>:<port> keep their own port-filtered membership, now named after the cluster rather than the HTTP spelling <fqdn>:<port>. In /config_dump, only a service's default cluster <svc>.<ns>.<domain> may carry service_name: <ns>/<svc>. Every other EDS cluster's service_name equals its own name.

Three gates keep it that way:

• //agent/internal/xds/cache TestNoNonDefaultClusterSharesTheBareServiceEDSName and TestLate{PortAlias,TCPFloor}SubscribesToItsOwnEDSResource.
• //agent/test/envoy_validate ClustersSharingServiceEDSName, over every fixture.
• //agent/test/mtlspool TestLate{PortAlias,TCPFloor,QUICTwin}*, against the pinned proxy, each with a shared-name negative control that must time out at ~15 s.

Any new cluster, secret or config that is a clone or second consumer of an existing resource needs either its own resource name or its own api_config_source.

Fleet gate. HTTP port aliases and per-port HTTP clusters keep the default cluster's alt_stat_name (the bare <ns>/<svc>), so their stats land in the default cluster's cluster.<ns>/<svc>.* tree. L4 clusters no longer do (#1023): the TCP floor reports as tcp_<ns>/<svc>, a TCP per-port or primary-port alias cluster as tcp_<ns>/<svc>_<port>, a UDP floor as udp_<ns>/<svc> (see "L4 stat keys and the L4 access log"). A twin has its own key (<ns>/<svc>@<ns>/<sa>,

960). A mesh EDS cluster that follows the invariant never times out, so the gate

is zero on the whole family, twins included, across a soak:

# #1013 (HTTP aliases, per-port and default clusters as <ns>/<svc>; L4 clusters as
# tcp_<ns>/<svc>[_<port>] -- the selector matches both, as it did before #1023)
sum(increase(envoy_cluster_init_fetch_timeout_total{aether_cluster=~"[^@]+/[^@]+"}[8h]))   # MUST be 0
# #1008 (QUIC twins)
sum(increase(envoy_cluster_init_fetch_timeout_total{aether_cluster=~".+@.+"}[8h]))         # MUST be 0

A non-zero first line says which service and, since #1023, whether it was an L4 cluster (and which port) or the HTTP family. Tell the HTTP clusters apart with /config_dump, as above, and the proxy log line initial fetch timed out for …ClusterLoadAssignment.

Attributing a prober failure: its labels and the AETHER_PROBE_FAIL line (#1040, #1041)

Labels. aether_probe_requests_total and aether_probe_request_duration_seconds carry:

label	value	set by
node	the Kubernetes node the prober pod runs on (node-c): the source of the probe, never where it went	the collector, from the resource's k8s.node.name (chart env OTEL_RESOURCE_ATTRIBUTES, downward API spec.nodeName)
pod	the prober pod (prober-h2mzs)	the prober, as a datapoint attribute
tier	liveness, reachability, mesh_dns	the prober
target	the probed name (egress, echo.aether-test.aether.internal:8080, …; ...:18081 on series written by prober charts before 1.1.1): what was asked for, not the endpoint that would have answered	the prober
result	success, http_error, connection_error, timeout, saturated, dns_error, dns_nxdomain, dns_timeout	the prober (classifyFailure: the error, plus the request phase the deadline interrupted)

There is no destination label (#1391). A burst grouped by (node) is placed at its source. Where each failed probe went is not in the metric and cannot be: a label for it would be a series per endpoint. It is found from the failure line, through the access logs: see "Where a failed probe went" below.

timeout vs dns_timeout (#1252). A probe whose deadline fires while its name lookup is still in flight is dns_timeout. Before #1252 it was timeout: Go's transport dials on a context detached from the request and, when the request deadline fires first, returns that deadline (context deadline exceeded), never a DNS error, so the prober could not tell a resolution stall from a connect or upstream stall. Every probe now carries an httptrace trace, and the deadline is classified by the phase it interrupted. A deadline in any later phase (connect, TLS, write, waiting for the first response byte) is still timeout; which one it was is the phase field of the failure line below, not a label. In series from before #1252, part of the mesh_dns timeout count was resolution stalls; from #1252 on, those count as dns_timeout.

Until #1041, node held the POD name (node="prober-h2mzs"). The prober's resource carried host.name, and for a pod that is not hostNetwork that is the pod name. The reference-cluster collector's transform/promote set node from host.name before it looked at k8s.node.name. A pod that had since been rolled away could not be placed on a node (#1040). Two changes fix this: the prober no longer sets host.name, and the collector now prefers k8s.node.name (the platform's GitOps repository). Series from before the fix still show a pod name in node. For any window that spans the change, group by pod, which exists only on series from after the fix, or translate the old values with kubectl -n aether-test get pods -o wide while those pods still exist.

Group per node with by (node), and per prober generation with by (node, pod). A DaemonSet roll starts a new pod series on the same node, so only a by (node) aggregate spans the roll:

# per node, across prober generations
sum by (node, tier, result) (increase(aether_probe_requests_total{result!="success"}[10m]))
# per prober pod: a new generation starts a new series
sum by (node, pod, tier, result) (increase(aether_probe_requests_total{result!="success"}[10m]))

Any rule that guarded against dead prober generations with and on (node) max by (node) (present_over_time(...[3m])) (#47) relied on node being the pod. Now that node is the node, a new pod on the same node satisfies that guard for a dead pod's frozen burst. Guard on (node, pod) instead.

The failure line. Every probe that does not succeed prints one line to the prober's stdout: a fixed marker, then one JSON object:

AETHER_PROBE_FAIL {"t":"2026-09-28T04:37:52.114Z","tier":"mesh_dns","target":"echo.aether-test.aether.internal:8080","result":"timeout","err":"Get \"http://echo.aether-test.aether.internal:8080/\": context deadline exceeded","elapsed_ms":2000.4,"phase":"first_byte","reused":false,"conn_ms":412.6,"dns_ms":0.9,"connect_ms":411.5,"tls_ms":-1,"write_ms":0.1,"ttfb_ms":1587.6,"dial":"10.96.0.7:8080","remote":"10.96.0.7:8080","local":"10.0.3.114:49292","trace_id":"9fa32b3befe77ea2253e8831d3472fa8","pod":"prober-h2mzs","node":"node-a","n":1,"truncated":false}

• t is the client-side timestamp. Line it up against the proxy's hot-restart parent-exit time and the mesh-dns handoff on node.
• elapsed_ms separates a probe that used its whole budget (timeout at about 2000 ms, meaning the request went out and nothing came back) from a fast connection_error (a refusal or reset in a few ms, meaning nothing was listening).
• phase (#1252) is the step of the request the probe was in when it ended: the one the deadline interrupted for a timeout/dns_timeout, the one that failed for a connection_error. One of dns, connect, tls, write (connection up, request not fully written), first_byte (request written, nothing back yet), headers (first byte back, headers incomplete), conn_wait (waiting for a connection with no dial step running), response (an http_error: the answer arrived, the status is the failure), or empty when the probe never reached the transport (saturated). The example above connected after a slow 411 ms and then waited 1.6 s for the proxy to answer: the time went to the proxy, not to DNS. (Its phase values are illustrative; lines written before #1252 carry no phase fields.)
• dns_ms, connect_ms, tls_ms, write_ms, ttfb_ms are each phase's duration in ms (ttfb_ms is from the request written to the first response byte). conn_ms is everything it took to get a connection (pool wait, resolution, connect, TLS). The phase the probe ended in is measured up to the failure. -1 means the phase never started: dns_ms is always -1 on the liveness and reachability tiers (they dial an IP), and dns_ms/connect_ms are -1 on a reused connection.
• reused is true when the probe ran on a pooled keep-alive connection (liveness, reachability). The mesh_dns tier never reuses, so it is always false there.
• pod and node are the prober's own: the source.
• dial, remote, local and trace_id (#1391) are what the client knows of where the probe went. dial is the address of the last connect attempt, which for the mesh_dns tier is what the name resolved to (empty on a reused connection and when the lookup failed). remote and local are the two ends of the connection the request was sent on (both empty when the probe never had one: a refused or blackholed connect, a lookup that failed). trace_id is the trace id of the traceparent header the probe sent (empty for saturated, which sent nothing). Lines written before #1391 have none of the four.
• Every key is present on every detail line, so a query can filter on any of them (AND "\"phase\":\"dns\"").
• err is the Go error string. For http_error it is HTTP <status>. For saturated the probe was never sent because --max-concurrent probes were already in flight.

Where a failed probe went (#1391). The client cannot say. On the mesh its connection ends at the node proxy of its own node, so remote is the address it dialled: 127.0.0.1:18081 for the liveness and reachability tiers, the mesh Service's address for mesh_dns (the capture diverts the connection and keeps the original destination). It is never the pod that would have answered, and the proxy adds no response header that names its upstream. What the line has instead is the key to the proxies' own records of that one request: trace_id, the trace id of the traceparent header the probe sent. The HTTP access log has a traceparent field, and every row of the request carries that trace id in it:

log_name:aether_access_logs AND traceparent:"00-9fa32b3befe77ea2253e8831d3472fa8-"*

The trace id, not the whole header. A proxy with tracing on writes its own span id into the header at every hop, also for a request that is marked not-sampled, as every probe is. The rows of one probe therefore share the 32 hex digits of the trace id and differ in the 16 that follow, and a search for the header exactly as the prober sent it finds nothing. Two rows of one mesh_dns probe, read from a test cluster on 2026-10-08:

reporter=source       node_name=node-a  traceparent=00-7562d29beca08c9375101ae165a94196-5fe29c53fe07c4da-00  downstream_remote_address=10.0.3.114:47098  upstream_host=10.0.4.106:18008
reporter=destination  node_name=node-b  traceparent=00-7562d29beca08c9375101ae165a94196-64629e6a312a85b9-00  upstream_host=127.0.0.1:8080

• The reporter:source row is the prober's own node proxy. Its upstream_host is the endpoint it chose (<pod ip>:18008), its upstream_cluster the service, and its response_flags what the proxy thinks happened (DC: the prober gave up first). Its downstream_remote_address is the line's local, and its downstream_local_address the line's remote. Translate the pod IP to a node with kubectl get pods -A -o wide while the pod exists, or from that pod's own access-log rows (node_name).
• A reporter:destination row with the same trace id, written by another node's proxy, means the request arrived there: the destination is that row's node_name. No such row means it did not arrive, or the destination proxy has not logged it yet (a row is written when the stream ends).
• No row at all means the request never became an HTTP request at the source proxy: phase is dns or connect, and dial is all there is. A liveness probe has no row either way: its route is a local reply, excluded from the access log.

What was measured and what was not: the two rows above, and that a prober row's downstream_remote_address is the prober pod's ip:port and its downstream_local_address the mesh Service address, were read from a test cluster's access logs for probes that succeeded. A failed probe's line has not yet been joined to its rows on a cluster: the prober that prints trace_id had not been deployed when this was written.

It is bounded: at most 20 detail lines per (tier, result) per minute (n counts them, and truncated:true marks the 20th). Later failures in that minute are only counted, and when the minute closes they produce ONE summary line under the same marker:

AETHER_PROBE_FAIL {"t":"…","tier":"mesh_dns","result":"timeout","suppressed":122,"window_s":60,"window_start":"…","pod":"prober-h2mzs","node":"node-a"}

A summary's t is when it was written, and the failures it counts lie between window_start (the minute's first failure) and t. A prober built before the fix for

1463 has no window_start, and dates the summary it writes as it stops 60 s in the

future: a last line dated after the pod was gone is that.

A 30 s burst of about 142 timeouts therefore prints 20 lines plus one summary, not 142 lines. The budget renews every minute, so the next burst is still attributable.

Pull the lines from VictoriaLogs (the logs are not in Loki):

_stream:{k8s.namespace.name="aether-test"} AND "k8s.container.name":prober AND "AETHER_PROBE_FAIL"

To see one pod or one class, add AND "prober-h2mzs" or AND "\"result\":\"timeout\"". To read them straight from a live pod: kubectl -n aether-test logs <prober-pod> | grep AETHER_PROBE_FAIL.

Forwarded DNS keeps failing after a kube-dns roll

The mesh-DNS forward path keeps a small pool of connected UDP sockets per upstream (issue #674) rather than dialling one per query. The upstream is a ClusterIP, so connecting pins a conntrack entry to one kube-dns backend pod; when that pod rolls the entry survives pointing at a corpse and datagrams are black-holed with no ICMP — the socket only ever sees a read timeout.

This self-heals: any exchange error retires the socket, and every socket also expires on its own budget (a jittered ~30s, or 1000 queries). Symptoms are therefore a burst of aether_mesh_dns_forward_conn_recycles_total{reason="error"}, not a sustained outage.

Forward timeouts (#1254). A pod's resolver gives up on a datagram after 1 s and sends it again (timeout:1 attempts:3), so the forwarder retries inside that window. For each upstream, in order:

step	bound	what happens
first try	600 ms (forwardTryTimeout)	the query goes out on a pooled socket (or a fresh one when every slot is busy or pooling is off)
one re-send	until 2 s from the start (forwardTimeout, the per-upstream budget)	only if the first try timed out: the query is sent once more on a freshly dialled socket (a new source port, so a new conntrack entry and possibly another kube-dns backend). The first socket keeps listening; the first reply from either socket is delivered and the other is read and dropped
next upstream / SERVFAIL		neither answered within 2 s

• A lost datagram costs about 0.6 s plus one round trip, inside the client's 1 s window. Before, it cost 2 s and then a cold dial.
• A black-holed upstream fails at 2 s with exactly one re-send. Before, a pooled socket waited 2 s and its cold-dial fallback waited another 2 s, so 4 s per upstream. The budget is per upstream: N dead upstreams still cost N x 2 s.
• A reply that arrives within 2 s is accepted, as before. The change only makes answers come sooner.
• A first try that fails fast (ECONNREFUSED from an ICMP port-unreachable, a write error) does not re-send. It works as before: the pooled socket is retired and the query gets one cold dial (forward_conn_dials_total{reason="fallback"}).
• A pooled socket whose reply came back after the re-send went out is healthy and is kept. One that got nothing by 2 s is retired (reason="error"), because that is what a stale conntrack entry looks like.
• TCP queries and the TC=1 re-fetch over TCP do not re-send: TCP retransmits on its own.

Both bounds are compile-time constants in agent/internal/meshdns, like the 2 s timeout before them. They follow the clients' resolv.conf timeout, so there is no flag and no chart value.

How often the re-send fires, and why:

# Re-sends per forwarded query, by which reply was delivered:
#   resend = the re-send answered (datagrams to the upstream are being lost)
#   late   = the first try answered after 600 ms (the upstream is slow, not lossy)
#   failed = nothing within 2 s (the upstream is down or black-holed)
sum by (result) (rate(aether_mesh_dns_forward_retries_total[5m]))
  / scalar(sum(rate(aether_mesh_dns_queries_total{result=~"forwarded|forward_error"}[5m])))

A steady late share means kube-dns answers slower than 600 ms. That is a kube-dns capacity problem, and every such query also pays an extra datagram and a dial (forward_conn_dials_total{reason="retry"}).

If it does NOT settle:

# Dials per forwarded query -- should be well under 0.01, and is 1.0 when pooling is off.
sum(rate(aether_mesh_dns_forward_conn_dials_total[5m]))
  / sum(rate(aether_mesh_dns_queries_total{result="forwarded"}[5m]))

# Open pooled sockets -- flat at pool size x upstreams at steady state.
aether_mesh_dns_forward_conn_pool_open

To rule the pool out entirely, disable it — --forward-pool-size=0 restores the exact pre-#674 dial-per-query behaviour:

helm upgrade ... --set agent.meshDnsDaemon.forwardPoolSize=0

Grading the mesh-DNS lame-duck handoff across a roll

On SIGTERM the resolver keeps serving until it has PROVEN a successor is answering on the shared SO_REUSEPORT address, and only then closes its sockets (issue #729). Every window ends with exactly one exit, counted by reason:

reason	meaning
successor	a DIFFERENT instance answered on our listen address — the handoff drained. This is what a healthy DaemonSet roll must show on every node.
deadline	--lame-duck-max elapsed with no successor. Expected on a scale-down or node drain; on a roll it means the successor never came up, and the queued datagrams were dropped as they were before #729.
signal	a second termination signal cut the window short.

The exit is a once-per-process event, so the counter carries an instance attribute — this process's 64-bit lame-duck identity stamp, exactly one value per mesh-DNS generation (issue #736). Without it every successor restarts the same {node,reason} series at 1, Prometheus never sees a reset, and increase() over a window spanning several rolls reads ~0 (the #729 validation had to count 50 exits from raw per-roll snapshots). With it each generation is its own series and a genuine 0 → 1 rise:

# Exits per node over the last 30m. After two rolls this is 2 per node -- it counts
# every generation, which the pre-#736 series could not.
sum by (node) (increase(aether_mesh_dns_lame_duck_exits_total[30m]))

# The only breakdown that matters: everything must be reason="successor".
sum by (node, reason) (increase(aether_mesh_dns_lame_duck_exits_total[30m]))

# How long each generation held its sockets open, p95 (10s is --lame-duck-max,
# i.e. a window that ran to its deadline).
histogram_quantile(0.95,
  sum by (le) (rate(aether_mesh_dns_lame_duck_duration_seconds_bucket[30m])))

Sample count. A generation records its exit and then dies, so its series is exported by the shutdown flush and usually carries a single sample. rate() and increase() need two samples in the range to produce anything for a series, so if a window you know had rolls still comes back empty, count the series instead — each one tops out at exactly 1, so this is exact regardless of how many samples landed:

# Rolls per node, sample-count independent.
sum by (node) (max_over_time(aether_mesh_dns_lame_duck_exits_total[30m]))
sum by (node, reason) (max_over_time(aether_mesh_dns_lame_duck_exits_total[30m]))

max_over_time is also what survives the series ageing out with its generation — the same trap that produced two premature "zero" readings on #638.

instance is also the join key back to the logs: the same stamp appears as instance= on that generation's lame duck started and successor observed lines (and as successor= on the predecessor's line, naming the generation that replaced it), so a suspicious series resolves to one pod's window.

_stream:{service.name="aether-mesh-dns"} AND instance:"<stamp>"

Cardinality is one series per generation per node — bounded by how often the DaemonSet rolls, and the series age out with the generation. That ageing-out is why an instant query is never the right read here: minutes after a roll the generation's series is gone, and the instant read returns nothing at all rather than the exit it recorded.

mesh-dns CPU throttling

mesh-dns is every managed pod's resolver, so a CFS-throttled period on it is added to the DNS latency of every lookup in flight on that node. Until chart 2.4.8 it ran with a 25m request and a 100m limit and was throttled in 8.7 % of 100 ms periods at steady state (node E of the reference cluster, #1253) while averaging ~24.5m: its CPU comes in bursts, and a burst that spends the period's 10 ms of quota parks the daemon until the next period. Since #1253 it has a 50m request and no CPU limit by default, with GOMAXPROCS=2 pinned in the chart (agent.meshDnsDaemon.goMaxProcs).

Read the throttling straight from the container's cgroup. The image is distroless (no shell), so go through a node debug pod; the find works under both the systemd and cgroupfs cgroup drivers:

NODE=<node>
POD=$(kubectl get pod -n aether-system -l app.kubernetes.io/component=mesh-dns \
  --field-selector spec.nodeName="$NODE" -o jsonpath='{.items[0].metadata.name}')
CID=$(kubectl get pod -n aether-system "$POD" \
  -o jsonpath='{.status.containerStatuses[?(@.name=="mesh-dns")].containerID}' | sed 's|.*://||')
kubectl debug node/"$NODE" -it --profile=sysadmin --image=busybox:1.37 -- sh -c \
  "d=\$(find /host/sys/fs/cgroup -type d -name \"*$CID*\" | head -1); echo \$d; cat \$d/cpu.max \$d/cpu.stat \$d/cpu.pressure"

• cpu.max is max 100000 with no limit, 10000 100000 at 100m.
• cpu.stat: nr_throttled / nr_periods is the fraction of periods throttled and throttled_usec the total time parked. With no limit both stay at 0 — nr_periods only counts while a quota is set. usage_usec over the pod's age is its average CPU.
• cpu.pressure: the PSI some avg10/avg60 is the share of time the daemon had a runnable task waiting for CPU for any reason — quota throttling or node contention. With no limit, this is the number to watch: non-zero means the 50m request (its CFS weight) is too low for the node it shares.

If the kubelet's cAdvisor is scraped, the same counters are container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total:

# Fraction of CFS periods throttled, per mesh-dns pod (0 once the limit is gone)
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="aether-system", container="mesh-dns"}[5m]))
  / sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="aether-system", container="mesh-dns"}[5m]))

# CPU actually used, millicores
1000 * sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="aether-system", container="mesh-dns"}[5m]))

What the throttling costs shows in the daemon's own latency histogram, split by path (answered = an authoritative mesh hit from memory, forwarded = a kube-dns round trip). Throttling lifts the tail, not the median:

histogram_quantile(0.99,
  sum by (le, result) (rate(aether_mesh_dns_query_duration_seconds_bucket{result=~"answered|forwarded"}[30m])))

To put a cap back, set agent.meshDnsDaemon.resources.limits.cpu (an empty value renders no limit) and consider agent.meshDnsDaemon.goMaxProcs=0, which hands GOMAXPROCS back to the Go runtime to derive from that limit.

uds-csi CPU throttling (and the controller's and registrar's CPU limits)

The csi.aether.io plugin serves NodePublishVolume / NodeUnpublishVolume, so a CFS-throttled period on it is added to the start (or the termination) of a UDS pod on that node. Until chart 2.4.10 it ran with a 5m request and a 100m limit, and the 2026-10-06 8 h soak — the first with cAdvisor scraped — showed it throttled in 57.6 % of the periods it ran in (20,248 of 35,169, 15 pods, #1321) while using 11–23 CPU-seconds per pod over the whole run. Same mechanism as mesh-dns above: an average far under the limit, delivered in bursts (a publish; and every 30 s the liveness probe, a second Go process inside the same cgroup) that spend a period's 10 ms of quota and park the container until the next one. Since #1321 it keeps the 5m request and has no CPU limit by default, with GOMAXPROCS=2 pinned in the chart (udsCsi.goMaxProcs).

The same soak measured the controller at 1.0 % and the registrar at 1.5 %. Their 100m limits were kept: neither is on a request path (a throttled period is under 100 ms on an admission call with a 5 s timeout, or on an endpoint update on its way to the agents), the registrar's requests == limits is what makes it QoS Guaranteed, and both derive GOMAXPROCS from limits.cpu. The reasoning is next to controller.resources and registrar.resources in charts/aether/values.yaml.

Fraction of CFS periods throttled, per container, over a window (cAdvisor; set the range to the run you are grading):

sum by (container) (increase(container_cpu_cfs_throttled_periods_total{namespace="aether-system", container!=""}[8h]))
  / sum by (container) (increase(container_cpu_cfs_periods_total{namespace="aether-system", container!=""}[8h]))

Per uds-csi pod, as a rate, with the time parked and the CPU actually used:

# Fraction of CFS periods throttled, per uds-csi pod
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="aether-system", container="uds-csi"}[30m]))
  / sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="aether-system", container="uds-csi"}[30m]))

# Seconds parked per second
sum by (pod) (rate(container_cpu_cfs_throttled_seconds_total{namespace="aether-system", container="uds-csi"}[30m]))

# CPU actually used, millicores
1000 * sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="aether-system", container="uds-csi"}[30m]))

A container with no CPU limit has no container_cpu_cfs_* series at all — cAdvisor only exports them for a cgroup with a quota — so after #1321 the uds-csi queries return no data, not 0, and the first query simply stops listing uds-csi (as it does not list agent, mesh-dns or proxy). Read "no data" as the pass, but prove the scrape is alive with the usage query, which must still return one series per pod, and confirm the limit is really gone:

# Expect: empty. Any series here is a uds-csi container that still has a quota
# (an old pod not yet rolled, or an operator-set limits.cpu).
count by (pod) (container_cpu_cfs_periods_total{namespace="aether-system", container="uds-csi"})

kubectl get ds -n aether-system aether-uds-csi \
  -o jsonpath='{.spec.template.spec.containers[0].resources}{"\n"}'   # no limits.cpu

With no quota, what can still make the plugin wait is node contention, where its 5m request is its CFS weight. The cgroup's cpu.pressure shows that; read it with the node-debug recipe in the mesh-dns section above, with app.kubernetes.io/component=uds-csi and container name uds-csi (the image is distroless too). A some avg60 that stays above zero is the signal to raise udsCsi.resources.requests.cpu.

To put a cap back, set udsCsi.resources.limits.cpu (an empty value renders no limit) and consider udsCsi.goMaxProcs=0, which hands GOMAXPROCS back to the Go runtime to derive from that limit.

cni-install CPU limit (an init container cAdvisor never sees)

cni-install is the agent pod's init container: it runs ahead of the agent on every agent pod start (a roll, a node boot, a crash restart). Until chart 2.4.12 it had request = limit = 100m CPU. Since #1335 it keeps the 100m request and has no CPU limit.

The throttling could not be read from metrics, and it will not be for any sub-second init container. On the reference cluster (2026-10-07, cAdvisor scraped every ~60 s for the aether-.* namespaces) there is no container_cpu_cfs_* and no container_cpu_usage_seconds_total series for container="cni-install" at all — nor for the proxy pod's install-supervisor — in the 45 h cAdvisor had been scraped, eight agent rolls included: the container's cgroup lives for under a second and never meets a scrape. kube-state-metrics exports only kube_pod_init_container_status_restarts_total there (no ..._state_started), and kubectl shows startedAt/finishedAt at one-second resolution and no CPU time. So the evidence is the installer's own log timestamps (nanosecond, in VictoriaLogs), and the CPU figure below is an inference, not a counter.

What it does (cni/internal/install): Go start-up, then for the plugin binary a byte-for-byte compare of the copy already on the host against the one in the image (#1123: an unchanged plugin is not rewritten), then a few small conflist reads. On that path it writes nothing, and the timings below say the wait is the quota rather than the reads: disk latency does not come in 100 ms units. All 40 agent starts of 2026-10-05..07 (8 rolls × 5 nodes) took the unchanged path:

First log line → last, 40 starts
min / median / max	496 ms / 598 ms / 802 ms
"running CNI installer" → "already installed and unchanged" (the compare)	400–700 ms
every other gap between consecutive lines	~0–10 ms or ~90–100 ms

Every total is a multiple of ~100 ms (500, 600, 700, 800) and every gap is either nothing or a whole CFS period: the process spends its 10 ms of quota, is parked until the next 100 ms period, and repeats. Five to eight periods at 10 ms each bounds the work at 50–80 ms of CPU, so roughly half a second of each agent start was the quota (estimated; Go start-up before the first log line is not visible and is throttled the same way). A start that actually rewrites the plugin (a new CNI build: copy, fsync, directory sync) was not in the sample and can only be longer.

To repeat the measurement on any chart version (LogsQL, VictoriaLogs; group the lines by k8s.pod.name and subtract the first timestamp from the last):

_time:48h "cni-install" ("installing CNI binaries" OR "running CNI installer" OR "not rewriting it" OR "CNI binaries installed")
  | sort by (_time) | fields _time, _msg, k8s.pod.name, k8s.node.name

After 2.4.12 the totals should fall to tens of milliseconds with no ~100 ms steps (not yet measured on a cluster when this was written). If the steps are still there, check that the limit is really gone:

kubectl get ds -n aether-system aether-agent \
  -o jsonpath='{.spec.template.spec.initContainers[0].resources}{"\n"}'   # no limits.cpu

The other init containers, checked in the same pass: the proxy pod's install-supervisor rendered no resources at all until chart 2.4.13 (no limit, so nothing to throttle, but also no request: it ran at the minimum CFS weight and was invisible to request accounting; ≤ 1 s wall on the reference cluster). Since #1344 it has a 100m / 32Mi request, a 64Mi memory limit and no CPU limit (proxy.supervisor.resources); the self-copy it does costs about 50 ms of CPU and peaks at 14 MB RSS (a workstation build of the binary, /usr/bin/time -v). The authz native sidecar has a 10m request and no CPU limit, and the webhook-injected aether-identity-ready has a 5m request and no CPU limit (controller.webhook.identityGate.resources, limits.cpu: ""). The edge pod's agent container has no CPU limit either, so its GOMAXPROCS is pinned in the chart (edge.goMaxProcs, default 2, #1335) like the node agent's.

kubectl get ds -n aether-system aether-proxy \
  -o jsonpath='{.spec.template.spec.initContainers[0].resources}{"\n"}'   # requests, limits.memory, no limits.cpu

Edge Envoy worker count

Until chart 2.4.13 the edge envoy container was started without --concurrency, so it ran one worker per node core (4 on the reference cluster) for about 12m of CPU on average. Since #1344 the chart passes --concurrency from edge.concurrency (default 2, the node proxies' count on the reference cluster; 0 omits the flag). Unlike proxy.concurrency, changing it is an ordinary rolling update of the edge Deployment: there is no hot restart and nothing shared between the old and the new replica.

To read the count a running edge Envoy actually has, before and after the upgrade (the first is the pod spec, the second is Envoy's own gauge in the OTLP stats; expect one series per edge replica, each equal to edge.concurrency, or, when it is 0, to Envoy's default: the node's core count unless the container has a CPU limit or a restricted CPU mask, which lower it):

kubectl get deploy -n aether-ingress aether-edge \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="envoy")].command}{"\n"}'

envoy_server_concurrency{job="aether-edge-proxy"}

A worker count that is too low shows up as the edge Envoy's CPU approaching edge.concurrency whole cores (sum by (pod) (rate(container_cpu_usage_seconds_total{container="envoy", namespace="aether-ingress"}[5m]))); at 12m it is over two orders of magnitude away.

Labels and annotations on a Gateway's LoadBalancer Service

A Gateway's spec.infrastructure.labels and .annotations are copied to its per-Gateway LoadBalancer Service in the edge namespace (docs/configuration.md, "Labelling and annotating a Gateway's LoadBalancer Service"). To see what the edge copied, and what it refused:

# the Service of Gateway <ns>/<name>, its labels and annotations
kubectl get svc -n aether-ingress -l aether.io/edge-gateway=<ns>.<name> \
  -o jsonpath='{.items[0].metadata.labels}{"\n"}{.items[0].metadata.annotations}{"\n"}'
# refused keys are named in the Gateway's Accepted condition (which stays True)
kubectl get gateway -n <ns> <name> \
  -o jsonpath='{.status.conditions[?(@.type=="Accepted")].message}{"\n"}'

• The annotation aether.io/infrastructure-keys on the Service is the edge's record of the keys it copied. Do not edit it. If it is deleted or damaged, the keys the Gateway still sets are re-applied and recorded again on the next reconcile; a key that was removed from the Gateway in the meantime stays on the Service until someone deletes it by hand, because the edge no longer knows it put it there.
• A label or annotation that is on the Service but not in that record was put there by something else. The edge never removes it. If the Gateway later sets the same key, the Gateway's value replaces it and the key is the Gateway's from then on (removing it from the Gateway removes it from the Service).
• The aether.io/edge-gateway label value is capped at 63 bytes, so for a long <ns>.<name> select by the Service name instead; the edge logs it as created per-Gateway LoadBalancer Service.
• The edge logs spec.infrastructure keys ignored on the per-Gateway Service once per change of the refused set, on each replica.
• With --edge-service-name unset, or when a per-Gateway port cannot be allocated, there is no per-Gateway Service and nothing is copied.

What the proxy supervisor does on SIGTERM (kubectl delete pod, drain, eviction)

The aether-proxy pod is hostNetwork with maxSurge: 1, so the node's Envoy is a node-scoped resource that happens to live in a pod. Terminating that pod is only free if the node's Envoy is handed to a successor rather than killed, and SIGTERM is not a drain — Envoy's handler exits the server outright (0.33–0.74 s measured), taking every connection on the node with it. So the supervisor resolves a SIGTERM onto exactly one of four branches, chosen from one authoritative /server_info probe, and records which one it took (issue #795):

branch	when	what happens	data-plane cost
handoff	admin answers at a newer epoch — a successor already holds our listen sockets	wait, without signalling Envoy, for the successor's parent-shutdown protocol to terminate it	none
successor_wait	admin answers LIVE at our own epoch — no handoff has begun	keep serving and wait (bounded by successorWaitBudget, 155 s at the chart default) for the DaemonSet's surge replacement, which is created within ~1 s and hot-restarts us	none
drain_fallback	that wait expires, or --shutdown-drain-immediately is set	POST /drain_listeners?graceful to our own Envoy only (see below), wait --drain-time, then SIGTERM and reap	in-flight requests finish; new connections go to whatever else serves the node
child_dead	the admin does not answer at all	SIGTERM and reap; there is nothing to drain	already gone

The drain only ever reaches this pod's own Envoy (#1127). 127.0.0.1:9901 is one address for the whole node, so whoever answers it may be another proxy pod's Envoy: a successor that took the admin over the hot-restart protocol, or a fresh Envoy the new pod started after its successor crashed. Before #1127 the old pod drained whatever answered. On 2026-10-01 that was the new pod's fresh epoch-0 Envoy, which then added no listeners for pods created on the node until the next handoff (about 12 min on node A and node C). Now every supervisor starts its Envoys with --admin-address-path <ready-marker dir>/envoy-admin-address.<pod>.<nonce>, a per-supervisor nonce that /server_info echoes as command_line_options.admin_address_path. The drain is sent only when /server_info returns that nonce and an epoch whose child this supervisor still tracks. The check and the drain go over one TCP connection, so the drain reaches the process that passed the check, or nothing. If the check fails, or our Envoy has already exited, the supervisor sends nothing, logs one WARN, and goes on to SIGTERM its own Envoy:

not sending envoy admin request: the shared admin address is answered by ANOTHER envoy, not this supervisor's …
    adminOwner=foreign answeredBy=envoy-admin-address.aether-proxy-<new>.<nonce> answeredEpoch=0 ourEpoch=1
not sending envoy admin request: this supervisor's envoy has exited, …   adminOwner=own_envoy_gone

listeners drained; stopping envoy then logs drainAccepted=false. The decision is counted in aether_supervisor_admin_mutations_total{aether_supervisor_admin_request="drain_listeners", aether_supervisor_admin_owner=own|foreign|unreachable|own_envoy_gone}, seeded at zero. Only own sends the request. foreign during a roll means the old pod's Envoy had already handed its admin to another Envoy, so it could not be drained gracefully. The supervisor reserves --admin-address-path and refuses to start if an --envoy-arg sets it (as it does for every other Envoy flag it passes itself; the list is in docs/configuration.md).

kubectl delete pod aether-proxy-<x> takes the successor_wait branch. Expect ≈20–25 s of termination (that is the successor initializing, not a hang) and zero prober errors on that node. A 1–2 s termination is the symptom to be alarmed by: it means the supervisor won the race against its own replacement and left the node with no Envoy — which cost 7.95 s / 8.11 s of blackout and 130 / 126 prober connection_errors per delete on rev214, before this branch existed.

The supervisor's lines are in VictoriaLogs. Since #772 the supervisor is PID 1 of the pod's proxy container, so its stdout (and Envoy's, which it inherits) is that container's log stream: service.name aether-proxy, k8s.container.name proxy. They outlive the pod, so this is where to read a delete after the fact. The whole #1050 timeline (live predecessor confirmed, successor ready gate anchored, liveness watchdog fired, drain deadline elapsed) came from this stream (#1059):

_stream:{service.name="aether-proxy"} AND "k8s.container.name":proxy AND "liveness watchdog fired"

Swap the phrase for any line below. Add AND "<pod-name>" for one pod. Keep the container filter: service.name="aether-proxy" also matches the access logs (by log_name). To follow a delete live instead:

kubectl -n aether-system logs -f <proxy-pod> -c proxy | tee /tmp/sigterm.log &
kubectl -n aether-system delete pod <proxy-pod>

Lines to look for, in order, on a healthy delete:

waiting for a successor before draining        successorWaitBudget=2m35s drainTime=10s
envoy epoch terminated by successor            epoch=N
successor took over during shutdown wait       epoch=N midHandoff=false

and on a termination where no successor can come (node shutdown, scale-down, kubectl delete daemonset, a replacement stuck Pending/unschedulable):

no successor terminated our envoy within the termination-grace budget; …   (WARN)
no successor within budget; draining listeners  successorWaitBudget=2m35s drainTime=10s
listeners drained; stopping envoy               drainAccepted=true elapsed=10.0s
reaped envoy epoch                              epoch=N

Both are also gradeable without logs, which matters because the pod is gone by the time you look. The branch counter is seeded at zero for all four values (#717), so an empty result means the metric never arrived, not that nothing happened:

# Which branch each node's last terminating supervisor took. drain_fallback or
# child_dead during a rolling upgrade means the surge replacement never arrived;
# handoff during a rolling upgrade means the old pod was deleted mid-handoff
# (#991, see the next section) -- expected ~0 per roll.
sum by (node, aether_supervisor_shutdown_branch) (
  increase(aether_supervisor_shutdown_branch_total[30m]))

# How long a fallback drain actually took (the graceful window included).
histogram_quantile(0.95,
  sum by (le) (rate(aether_supervisor_drain_duration_seconds_bucket[30m])))

A dying process exports once. The branch is recorded immediately before Run returns, and supervisorcmd's deferred telemetry flush is what pushes it — a supervisor SIGKILLed at the grace period flushes nothing, so a missing branch sample on a node is itself the finding. Use increase()/max_over_time, never an instant read: like the mesh-DNS lame-duck series, these age out with the generation that wrote them.

How a proxy roll hands the node over: the successor's ready gate and the parent's hold (#991)

A rolling upgrade (maxSurge: 1, maxUnavailable: 0) is safe only if the old pod stays until the new pod's Envoy has taken over — the #132/#795 invariant. The DaemonSet deletes the old pod about 1 s after the new one turns Ready, and at once if the old one turns NotReady while a surge pod exists. Two supervisor rules decide both moments.

1. The successor's ready gate is anchored on its first LIVE, not only on its fork. Envoy arms --parent-shutdown-time-s in startWorkers(), the step that also turns its admin LIVE. The predecessor Envoy therefore exits at workers + ParentShutdownTime (measured 14.93–15.00 s after starting workers, then 0.4–0.9 s to exit), not at fork + ParentShutdownTime. The successor's supervisor goes Ready at

max(fork + ParentShutdownTime + 3s,  firstLiveObserved + ParentShutdownTime + 2s)

ParentShutdownTime is the chart's proxy.hotRestart.parentShutdownTime (15 s), the value the supervisor passes to Envoy, not a literal. The fork anchor used to be the only one. Under soak load on rev242 the successor took 3.2–6.0 s from fork to workers, so the old Envoy was still serving at fork + 18 s, and the old pod was deleted mid-handoff in 13 of 30 rolls. Cost: Ready moves from fork + 18 s to roughly fork + 19.5–23 s under load, 1.5–5 s per node-roll. An unloaded successor that is LIVE within ~1 s keeps the fork gate.

2. The old pod keeps its readiness through a busy successor's admin. While the old supervisor's Envoy is still tracked, it holds its ready marker whenever the admin answers at another epoch (#132). During the handoff that admin port belongs to the successor, whose main thread is busy loading its first listener batch before starting workers, so a /server_info probe can miss its 1 s timeout. The hold used to drop on the first miss, and three failed exec probes (2 s period) later the DaemonSet deleted the pod. That happened in 9 of 30 rolls on rev242, 5.0–6.3 s after pod not ready. The hold now lasts through an unreachable admin for up to the admin watchdog's existing bound (--admin-unresponsive-deadline, 30 s default). A genuinely wedged admin still ends it at that bound, on the same tick the admin watchdog restarts the container. With no Envoy of ours tracked, an unreachable admin never holds.

Lines to grep in the aether-proxy container logs, per roll:

# successor pod: which anchor set its gate, and where the gate landed
live predecessor confirmed; starting cross-pod hot restart  ... readyGateIn=18s readyGateAnchor=fork
successor ready gate anchored   epoch=N anchor=live sinceFork=4.9s readyGateIn=17s readyGateAfterFork=21.9s parentShutdownTime=15s
pod ready: envoy live at newest epoch                       epoch=N

# old pod: the hold carried through the successor's busy init (once per unreachable streak)
holding readiness: serving as hot-restart parent mid-handoff              epoch=N-1
holding readiness through unreachable admin: successor likely initializing  epoch=N-1 unreachableFor=0s holdBound=30s
newest epoch terminated cleanly by successor; awaiting pod deletion        epoch=N-1

anchor=fork on the second line means the successor went LIVE fast enough that the fork gate was already the later one. On a healthy roll the old pod logs no termination requested mid-handoff. Its Envoy is terminated by the successor's parent-shutdown first, the pod turns NotReady because nothing of its own is left running, and the DaemonSet deletes an idle pod. So during a rolling upgrade:

• aether_supervisor_shutdown_branch_total{branch="handoff"} is ≈ 0 per roll. The fleet gate for a soak is the per-node increase() over the roll window. A non-zero count means an old pod was deleted while its Envoy was still the serving parent, which is a finding again, not background. On rev242, before this fix, it was 1 per node on 22 of 30 rolls.
• termination requested mid-handoff lines are 0. One that comes 5–6 s after the same pod's pod not ready is rule 2 failing. One at about fork + 19 s, after the successor's pod ready, is rule 1 failing.

A proxy roll reverted while the surge pod starts

A roll can be taken back before it has finished on a node: helm rollback during an upgrade, or an upgrade followed at once by its revert. The DaemonSet's template is then the old pod's revision again, so the controller deletes the surge pod and keeps the old one. By then the surge pod's Envoy has usually taken the node over (fork to workers is a fraction of a second on a quiet node), and it terminates the old pod's Envoy when its parent-shutdown time is up. The node's only Envoy is now in a pod that is terminating, and the pod the DaemonSet kept has none.

Until the supervisor handled this, that was the end of it. The kept pod logged newest epoch terminated cleanly by successor; awaiting pod deletion and waited for a deletion that never came, NotReady with stat /var/run/aether-proxy/ready: no such file or directory. The deleted pod waited successor_wait (155 s at the defaults) for a replacement the DaemonSet had no reason to create, drained and exited, and the node had no proxy until the next roll or a kubectl delete pod (nightly e2e, 2026-10-11: ds/aether-proxy never became Ready).

Now the kept pod's supervisor takes the node back, in one of two ways:

restart_triggers value	when	what happens	data-plane cost
successor_terminating	the terminating supervisor asks for a successor: while it waits with its Envoy LIVE it refreshes the node epoch heartbeat and writes successor-wanted beside it (in --state-dir, /run/aether/hotrestart by default), every second	the kept pod hot-restarts from that Envoy as a new pod would (epoch N+1, the usual readiness gate), and from no other epoch; the terminating pod then ends on the successor_wait branch	none (with the pinned Envoy on both sides, one request every 20 ms through the take-back: 0 failed, //agent/test/hotrestartrevert)
successor_gone	nothing asks, and nothing is there: no process holds a hot-restart socket of the base id, the epoch heartbeat is stale (8 s after its last refresh) and the admin is silent, on every poll for 3 s. The successor was killed outright, its supervisor is older than this, or its Envoy could not be hot-restarted from	a fresh Envoy at epoch 0	the node has no proxy from the successor's exit until that Envoy is up: about 12 s after a successor killed outright (measured with the pinned Envoy), 3 to 4 s after one that drained and exited

# kept pod
newest epoch terminated cleanly by successor; awaiting pod deletion   epoch=N-1
the successor that terminated our envoy is going away and this pod was not deleted; starting envoy again
    trigger=successor_terminating lastEpoch=N-1 startEpoch=N+1 hotRestartFromEpoch=N      (WARN)
live predecessor confirmed; starting cross-pod hot restart            predecessorEpoch=N startEpoch=N+1
pod ready: envoy live at newest epoch                                 epoch=N+1

# deleted (surge) pod
waiting for a successor before draining                               epoch=N
successor took over during shutdown wait                              epoch=N

aether_supervisor_restart_triggers_total{aether_supervisor_trigger="successor_terminating"|"successor_gone"} counts them. Either is expected only when a roll was reverted or superseded, or a surge pod was deleted by hand; successor_gone outside those is a successor pod that died after taking the node, and is worth reading that pod's last lines for.

What the kept pod will not do.

• Take a node from a healthy successor. It does nothing while the admin answers, the heartbeat is fresh or a process holds the base id, and nobody asks: that is every roll that completes.
• Start an Envoy to find out whether the node has one. A fresh --restart-epoch 0 beside a live Envoy is not a harmless probe. Envoy binds its two hot-restart domain sockets before it touches the shared memory, and their names carry the restart epoch modulo 3, so the new one fails on the bind (unable to bind domain socket … errno=98) only when the live epoch is a multiple of 3. Otherwise it binds, unlinks the base id's shared memory and creates a new one (epoch 0 means "first"), sets the "initializing" flag in it, and exits on the admin address the live Envoy holds. The live Envoy keeps serving, and every later hot restart from it exits with previous envoy process is still initializing until that Envoy is gone: the next roll's pod can never attach. //agent/test/hotrestartrevert:hotrestartrevert_test pins that behaviour of the pinned Envoy. So the supervisor asks the kernel instead: it connects to the abstract sockets @envoy_domain_socket_{parent,child}_<base id × 10 + {0,1,2}>, which exist exactly as long as an Envoy of that base id does, stopped or wedged included, and are the same for every pod on the node (hostNetwork). While one is held it starts nothing, and logs not starting envoy again; still awaiting pod deletion reason="an envoy holds …" if it had got as far as deciding.
• Keep trying, or give up on the node. An Envoy it started that exits within 5 s without going LIVE (a lost race for the next epoch against another pod, a crash) sends it back to waiting, after a 3 s pause, with the envoy started again after the successor went away exited at once … trigger=… attempt=n budget=5. Five in a row end differently for the two recoveries, and neither ends the watch for an empty node:
• answers to one asker (successor_terminating): it logs one ERROR, every hot restart from the terminating successor's envoy exited at once; no longer answering its requests, and starting a fresh envoy once it is gone, ignores that epoch's requests, and takes the successor_gone path when the asker has drained and exited. That Envoy cannot be hot-restarted from (see the limits), and a fresh start, which replaces the shared memory, is what works.
• fresh starts on an empty node (successor_gone): the supervisor exits 1 (… fresh envoys in a row, started on a node with no envoy, exited at once: exiting for a container restart). An Envoy that cannot start on an empty node is a pod crash like any other: the container restarts, and the kubelet's backoff paces it.

What the terminating pod does about it. It asks for the first successorWaitBudget − (8 s + --parent-shutdown-time) of its wait, 132 s of 155 s at the defaults, then withdraws the request and stops refreshing the heartbeat (no longer asking for a successor …). By the time it drains, the heartbeat is stale, so a pod that starts during the drain does not hot-restart from an Envoy about to be SIGTERMed under it. A terminationGracePeriodSeconds so short that this window is zero (below 48 s with the default drain and parent-shutdown times) means it never asks, and a reverted roll is recovered by successor_gone only.

Limits.

• Both pods need a supervisor that knows this. In a roll from an older supervisor image that is then reverted, the kept pod is the old one and waits as before: delete it (kubectl -n aether-system delete pod aether-proxy-<x>), and its replacement starts fresh.
• A reverted roll that also changes the Envoy worker count takes the #1136 path (the surge pod drains the kept pod's Envoy instead of hot-restarting from it): see "A reverted roll that changes the Envoy worker count" below.
• The shared-memory poisoning above was also reachable by the supervisor's ordinary start, a new pod that found the heartbeat stale while the predecessor's Envoy was alive: see "A new proxy pod beside an Envoy that has no heartbeat" below.
• A surge pod that gets its SIGTERM in the few tens of milliseconds between its fork and its Envoy's admin coming up takes the child_dead branch and stops an Envoy that is mid-attach.

A surge pod that gets its SIGTERM before its first fork (deleted while its image was still being pulled, started and stopped in the same moment) starts no Envoy at all: termination requested before the first envoy was started; starting none.

A reverted roll that changes the Envoy worker count (#1679)

A roll that changes proxy.concurrency (or a CPU limit that changes Envoy's default worker count) is not a hot restart: the surge pod's supervisor asks the old pod's Envoy to drain its listeners (POST /drain_listeners?graceful), waits proxy.hotRestart.drainTime, stops it, and starts a fresh Envoy at epoch 0 (#1136; see proxy.concurrency under the chart values). Envoy cannot take a drain back, and an Envoy that has drained its listeners looks healthy: measured on the pinned Envoy with --drain-time-s 3, the listener served for 3 s after the request and then refused every connection, while /server_info and /ready went on answering LIVE. The one thing that changed on the admin was listener_manager.listener_stopped, from 0 to 1.

So when such a roll was reverted while the surge pod started, the surge pod got its SIGTERM during the drain-time wait, stopped where it was and exited; the kept pod's Envoy closed its listeners a moment later, stayed LIVE, and the kept pod stayed Ready with a node that served nothing, until it was deleted by hand. Two things now prevent that, one on each side.

The surge pod finishes what it started. A supervisor that has asked a predecessor to drain owns the node's listeners until its own Envoy has them. A termination requested in between is held back until that Envoy is LIVE, and then proceeds as for any pod terminated while it serves: it asks for a successor, and the kept pod takes the node back (successor_terminating, above; across a worker-count difference that take-back is again a drain and a fresh start, so the node has two short gaps with no listeners, each the fresh Envoy's start). A termination requested before the drain is sent is obeyed at once and nothing is drained.

# surge pod
envoy worker count changes across this handoff; NOT hot-restarting: draining the predecessor, …
termination requested while replacing a predecessor that was asked to drain its listeners …; finishing the replacement first   budget=…   (WARN)
predecessor drained for a worker-count change; starting a fresh envoy lineage
replacement finished: this pod's envoy is LIVE; proceeding with the termination   heldFor=…
waiting for a successor before draining

The hold is bounded: twice the drain time plus 36 s (56 s at the defaults), and never more than the pod's successor_wait budget; the time it took comes off that budget. (A termination grace period with no such budget would leave the hold at its full length, longer than the grace period itself; the chart refuses to render one, "Chart 2.6.2".) If it runs out (the replacement of the drained predecessor did not finish in its budget; terminating anyway, ERROR), the kept pod's side below is what recovers the node.

The kept pod does not report Ready with its listeners closed, and restores them. Every supervisor now reads listener_manager.listener_stopped from its own Envoy (every parentShutdownTime / 3, 5 s at the defaults; on a fresh connection, after /server_info on that connection has shown the Envoy to be its own at its newest epoch). An increase while the Envoy is the pod's only one means its listeners were closed and no hot-restart child serves in its place:

what the kept pod sees	what it does
the counter rose	drops Ready at once: this pod's envoy is LIVE on its admin but has stopped its listeners … and pod not ready: envoy is live at newest epoch but has stopped its listeners (both WARN)
the Envoy is stopped by the supervisor that drained it (the normal #1136 handoff)	nothing more: newest epoch terminated cleanly by successor; awaiting pod deletion
5 s later the Envoy is still there and still answers as its own	hot-restarts it in place: nobody replaced the envoy whose listeners were stopped; hot-restarting it in place (WARN), aether_supervisor_restart_triggers_total{aether_supervisor_trigger="listeners_stopped"}. The next epoch binds the listeners again (measured: serving within 0.5 s of the fork) and the pod goes Ready

This is the recovery for a surge pod that was killed outright after it had sent the drain, and for one whose supervisor is older than the fix above (a roll to an older image that is reverted). The node has no listeners from the end of the drain until that restart: up to 10 s plus the Envoy's start at the defaults. listeners_stopped is expected only then; anywhere else, something asked a proxy's Envoy to drain and did not replace it.

A pod is also no longer made Ready by another pod's Envoy answering at its epoch number: a fresh lineage starts at epoch 0 again, so the kept pod (epoch 0, no Envoy) used to read the surge pod's LIVE at epoch 0 as its own. An answer on the shared admin address that carries another supervisor's identity (admin_address_path, #1127) is never counted as the pod's own.

A new proxy pod beside an Envoy that has no heartbeat (#1680)

A new proxy pod chooses its restart epoch from the node epoch heartbeat (epoch in --state-dir): fresh and confirmed LIVE on the admin, it hot-restarts from that epoch; stale or absent, it used to start at epoch 0. But the heartbeat is written by the predecessor's supervisor, once a second while it sees its Envoy LIVE. A supervisor that is stalled, or whose admin probes time out on a starved node, stops writing it while its Envoy goes on serving, and an epoch 0 started beside that Envoy is the start described under "What the kept pod will not do" above: unless the live epoch is a multiple of 3 it replaces the shared memory and leaves it marked "initializing", and every later hot restart from the healthy Envoy exits with previous envoy process is still initializing. The next roll's pod then retried for minutes and crash-looped until the old pod was deleted by hand.

Now a stale heartbeat is not enough. While any process holds a hot-restart socket of the base id, the new pod forks nothing:

the node epoch heartbeat is stale or absent, but an envoy holds this base id: not starting at epoch 0 beside it …
    heldSocketIDs=[1] adminLive=true adminLiveEpoch=1 waitBound=4m30s        (WARN)

and waits for one of two things: the heartbeat becomes fresh again (live predecessor confirmed; starting cross-pod hot restart), or the sockets are released (the envoy that held this base id without a heartbeat has exited, then no live predecessor; starting fresh at epoch 0). It does not attach on the admin's word alone, although the admin gives the epoch: a stale heartbeat is also how a terminating pod says "do not attach to my Envoy" before it drains, and an Envoy attached to then loses its parent mid-handoff. The pod is NotReady while it waits. After 4.5 minutes (the time the bind-collision retries used to take) the supervisor exits with an envoy has held hot-restart socket id(s) … and no supervisor reports it LIVE (no fresh heartbeat): not starting an envoy beside it; the container restarts and waits again. That state is an Envoy whose supervisor never reports it: find the pod it belongs to (heldSocketIDs is its restart epoch modulo 3; curl -s 127.0.0.1:9901/server_info names the pod in command_line_options.admin_address_path) and delete it.

The same check is the last thing before every fork at epoch 0, whatever chose it (refusing to start envoy at epoch 0: a process holds …), and the worker-count path no longer starts its fresh Envoy while the predecessor it drained still holds the base id: it chooses the epoch again (an envoy still holds the base id after the predecessor was drained; …).

A node that is already in that state repairs itself. The flag is also left behind by a supervisor older than this, and by a hot-restart child that is killed while it initializes (the OOM killer, the handoff watchdog). Before it forks an Envoy at an epoch above 0, the supervisor reads the flag from the shared memory (/dev/shm/envoy_shared_memory_<base id × 10>). When it is set, the node's admin reports LIVE at the parent epoch, and the parent's is the only hot-restart socket id held, no Envoy of the base id is initializing: the flag is stale, and the supervisor clears it:

cleared a stale initializing flag in the hot-restart shared memory: an envoy of this base id died while it initialized …   (WARN)

counted as aether_supervisor_restart_triggers_total{aether_supervisor_trigger="stale_initializing_flag_cleared"}. The healthy Envoy serves throughout. The write is held to the pinned Envoy's layout (the segment's size field equals the file's size, its version field is the pinned HOT_RESTART_VERSION, the flags are its last 8 bytes; //agent/test/hotrestartrevert fails on a pin bump that changes it). A segment it does not recognise is left alone, with cannot read the hot-restart shared memory's initializing flag … (WARN); a flag that may be true (another Envoy of the base id exists) is left alone too, and the fork is retried as before. If a surge pod still loops on suspected base-id bind collision with a live predecessor; retrying epoch detection in-process with sharedMemoryInitializing=true, the manual remedy is unchanged: delete the old proxy pod (kubectl -n aether-system delete pod aether-proxy-<x>); the new one then starts fresh at epoch 0, which replaces the shared memory.

The ext_authz sidecar across a proxy roll (#1275)

Symptom (chart <= 2.4.8). ext_authz errors on a node right after its proxy pod was replaced: envoy_http_<stat_prefix>_ext_authz_error_total goes up by a few, and with failureMode: DENY those requests got a 403. The prober's AetherAuthzCanaryErrors fires on it. It is worst on the first roll onto a new sidecar image. On the reference cluster on 2026-10-05 (OPA 1.21.1, chart 2.4.8) the authz container started 7–13 s after proxy in every new pod, because the kubelet starts regular containers in order without waiting and the new image had to be pulled (10.4 s on worker-01). The new Envoy hot-restarted and took the node's listeners before anything listened on /run/aether/authz/authz.sock.

Since chart 2.4.9 authz is a native sidecar: an init container with restartPolicy: Always, after install-supervisor and before proxy. Its startupProbe execs the staged proxy-ready --unix-socket=/run/aether/authz/authz.sock in the sidecar's container, and passes once a connect(2) to the socket succeeds.

What the ordering guarantees:

• Startup. The kubelet does not start proxy until the probe passes. A new pod's Envoy therefore cannot fork, hot-restart, or take a listener before its own authz accepts. The image pull and OPA's start now delay the new pod instead: it turns Ready later, and with maxUnavailable: 0 the old pod serves meanwhile. If the sidecar never comes up, the new pod stays in Init, the roll stops, and the old pod keeps serving.
• Shutdown. On deletion the kubelet sends SIGTERM to proxy and stops authz only after proxy has exited. The old Envoy is a child of the supervisor in proxy, so it has its pod's authz for as long as it runs. That covers successor_wait after kubectl delete pod (about 20–25 s of serving, see What the proxy supervisor does on SIGTERM), the drain_fallback drain, and a parent that is still draining when its pod is deleted. Before 2.4.9 both containers got SIGTERM at once, and OPA closed its socket while the old Envoy was still the node's only Envoy. In a plain DaemonSet roll this matters less: the successor turns Ready only after the parent Envoy has exited (see How a proxy roll hands the node over), so the old pod is usually deleted after its Envoy is gone.
• Both Envoys of a handoff have an authz. Each Envoy reaches the socket in its own pod's authz-socket emptyDir. The old one uses the old pod's sidecar and the new one uses the new pod's sidecar.

What it does not guarantee:

• A sidecar that dies mid-life. The kubelet restarts it in place (restartPolicy: Always) while proxy keeps serving. Checks fail per failureMode until it is back, as before.
• "Accepting" is not "has the latest policy". For the OPA preset the gRPC listener opens after /policy/policy.rego has loaded, so a pass means a policy is in place (and a policy that does not compile means the probe never passes; see Changing the OPA policy). A bring-your-own sidecar that loads its policy after it binds the socket needs startupProbe.override with its own readiness test.
• The grace period still applies. The sidecar's shutdown shares the pod's terminationGracePeriodSeconds (proxy.terminationGracePeriodSeconds, 180 s). If the supervisor uses all of it, the kubelet kills both.
• With startupProbe.enabled: false the kubelet starts proxy as soon as the sidecar's process is running, not when it serves.

Check the order on a live pod. authz is listed under initContainerStatuses, and its started turns true only when the probe passes. The proxy container's startedAt must be later than that:

kubectl -n aether-system get pod <proxy-pod> -o jsonpath='{range .status.initContainerStatuses[?(@.name=="authz")]}authz started={.started} at {.state.running.startedAt}{"\n"}{end}{range .status.containerStatuses[?(@.name=="proxy")]}proxy at {.state.running.startedAt}{"\n"}{end}'

Which ext_authz counters exist (#1325). In Prometheus the filter's counters are envoy_http_<stat_prefix>_ext_authz_<name>_total (capture_http is the capture listener's prefix). A missing series means the event never happened on that proxy. It does not mean the stat is filtered:

• Envoy's OTLP stats sink exports a counter only after it was first incremented. So ok, denied and error exist only for nodes whose proxy has run a check with that result. On the reference cluster that is the node carrying the canary. The first sample appears at its value, and increase() reads 0 for that first jump. An alert on these needs the "present now, absent 10 minutes ago" arm next to increase(): sum(m unless m offset 10m) > 0.
• failure_mode_allowed is incremented only when the filter runs fail-open (proxy.authzSidecar.failureMode: ALLOW, --authz-sidecar-failure-mode-allow). With the default DENY the series never exists.
• There is no timeout counter. Envoy's HTTP ext_authz filter does not define one (checked at the pinned Envoy, ALL_EXT_AUTHZ_FILTER_STATS in source/extensions/filters/http/ext_authz/ext_authz.h). A check that exceeds proxy.authzSidecar.timeout is counted in error. Do not gate on …_ext_authz_timeout_total: it reads as a clean zero forever.
• Neither the proxy's stats_matcher (an exclusion list, charts/aether/templates/agent-proxy-configmap.yaml) nor the collector drops any ext_authz stat.

The chart needs Kubernetes >= 1.29 for this and refuses to render with the sidecar enabled on an older cluster. Kind e2e: e2e/authz.sh (nightly job authz), steps a to c. It evicts the OPA image from the node and rolls the proxy under load. Against the 2.4.8 layout that gave 495 × 403 and 990 ext_authz.error in 4 s; against 2.4.9 it gave 0 and 0.

Changing the OPA policy (#1383)

Since chart 2.4.19 the OPA preset's sidecar watches its policy (opa run --server --watch … /policy/policy.rego). Changing proxy.authzSidecar.opa.policy and running helm upgrade changes the ConfigMap aether-opa-policy and nothing else: the proxy pods are not replaced and Envoy is not hot-restarted.

What was observed (kind, Kubernetes 1.35.8, OPA 1.21.1, one node; these are observations, not bounds):

	Observed
helm upgrade returning to the new policy deciding	55 s, 46 s, 64 s; the kubelet's delivery over eight changes was 27–64 s (the reload itself takes about 1 ms)
Proxy pod	same UID, authz and proxy restart counts unchanged, DaemonSet generation unchanged, Envoy restart_epoch unchanged
Requests through a reload	8,607 of 8,607 answered 200, ext_authz.error +0

It is unstaged and asynchronous. A ConfigMap volume is refreshed by each node's kubelet on its own schedule: the kubelet's sync period plus the delay of its ConfigMap cache (watch, or a TTL, depending on the kubelet's configuration). Nodes therefore get the new policy at different moments, in no order, and for a while the fleet decides with two policies. Nothing bounds that window and nothing reports its end: do not assume every node has converged after a minute; confirm it per proxy pod (below). A DaemonSet roll used to apply a policy one node at a time and stop at the first pod that failed to become Ready; that brake is gone. The chart validates nothing, so validate the policy before you update it, with the image the chart runs:

OPA_IMAGE="$(helm get values aether -n aether-system -a -o json | jq -r .proxy.authzSidecar.opa.image)"
# parses and compiles
docker run --rm -v "$PWD/policy.rego:/policy/policy.rego:ro" "$OPA_IMAGE" \
  check /policy/policy.rego
# the decision the sidecar queries is defined for ONE request you write down
# (request.json, below); prints it, exit 1 when undefined for that request
docker run --rm -v "$PWD/policy.rego:/policy/policy.rego:ro" \
  -v "$PWD/request.json:/policy/request.json:ro" "$OPA_IMAGE" \
  eval --fail -f pretty -d /policy/policy.rego --input /policy/request.json \
  'data.envoy.authz.allow'
# your own tests (*_test.rego next to the policy)
docker run --rm -v "$PWD:/policy:ro" "$OPA_IMAGE" test /policy

request.json is one request in the shape the sidecar receives from Envoy (input.attributes.request.http, header names in lower case; a sidecar's decision log shows real ones under "input"). Use a request the policy should allow:

cat > request.json <<'EOF'
{
  "attributes": {
    "request": {
      "http": {
        "method": "GET",
        "path": "/",
        "host": "my-service.my-namespace.aether.internal:8080",
        "scheme": "http",
        "protocol": "HTTP/1.1",
        "headers": {
          ":authority": "my-service.my-namespace.aether.internal:8080",
          ":method": "GET",
          ":path": "/",
          "x-authz": "letmein"
        }
      }
    }
  },
  "parsed_path": [""],
  "parsed_query": {}
}
EOF

What each command proves, checked with OPA 1.21.1:

• opa check exits 1 for a policy that does not parse or compile, and 0 for anything else, including a policy that defines no envoy.authz.allow.
• opa eval --fail --input request.json says whether the decision is defined for that one request and prints it. The e2e policy (default allow := false) prints true for the request above and false with another header value, exit 0 both times. A policy with no default prints true for a request that matches a rule and exits 1 (undefined) for one that matches none. A policy that does not define the decision exits 1 for every request. Without --input the policy is evaluated against no request, which rejects a correct policy that has no default and proves nothing about real requests.
• Only your own opa test cases cover behaviour: which requests are allowed and which are not. Neither command above does.

A policy that does not parse or compile (both measured: a rego_parse_error and a rego_type_error):

• A running sidecar does not load it. It keeps deciding with the last good policy, does not exit, is not restarted, and /health stays 200. The mesh looks healthy.
• A sidecar that starts with that file on the node exits 1 (error: load error: … rego_parse_error), because the last good policy lived only in the process that is gone. What follows depends on what restarted:
• The authz container alone (an OOM kill, a crash): the kubelet restarts it into CrashLoopBackOff, the pod shows Init:CrashLoopBackOff and 1/2 Ready, and proxy keeps running with nothing behind the authz socket. Every check on that node is an ext_authz error: a 403 with failureMode: DENY, allowed unchecked with ALLOW.
• A new proxy pod (a pod deleted, a node rebooted or added, a DaemonSet roll): authz never passes its startup probe, so proxy is never started and the pod stays in Init:CrashLoopBackOff. A surge roll stalls there with the old pod still serving. A deleted pod's Envoy waits for its successor for successor_wait (155 s at the defaults), then drains and exits, and the node has no proxy: mesh requests from that node fail to connect.

So a bad policy does nothing visible when it is applied and takes a node out later. Check for it after every policy change (below).

Confirming every node has the new policy, and seeing a failed reload. OPA exports no metric for a reload (its /metrics has only Go runtime and HTTP request series), and it logs a failed one at level info. There are two ways to see what each sidecar did; both are per proxy pod, because each node converges on its own.

# 1. The latest watch events of every sidecar, successes included.
for p in $(kubectl -n aether-system get pods -l app.kubernetes.io/component=proxy -o name); do
  echo "== $p"
  kubectl -n aether-system logs "$p" -c authz --since=15m \
    | grep 'Processed file watch event' | tail -n 3
done

Read each pod's lines by their time and err:

• {"duration":1779985,"err":null,"level":"info","msg":"Processed file watch event.","time":"2026-10-08T13:32:47Z"} with a time after your update: this sidecar loaded the new policy.
• {"duration":197736,"err":"1 error occurred during loading: /policy/policy.rego:7: rego_parse_error: unexpected eof token …","level":"info","msg":"Processed file watch event.","time":"2026-10-08T13:33:58Z"}: this sidecar rejected it and is still deciding with the last good policy.
• No line, or only lines older than your update: the kubelet has not delivered the update to that node yet (or the lines are older than --since). That is not a success; repeat until every pod shows an event newer than the update. A sidecar that restarted after the update has no event either: it read the file when it started, and a running authz container means it loaded.

One ConfigMap update produces about five lines per pod, with the same err (one per file-system event of the volume's symlink swap).

# 2. The policy a sidecar is deciding with, compared with the ConfigMap. Run it
#    on the node (the diagnostic API is a Unix socket in the pod's emptyDir; the
#    full path is longer than a socket address may be, hence the cd).
UID_=$(kubectl -n aether-system get pod <proxy-pod> -o jsonpath='{.metadata.uid}')
cd "/var/lib/kubelet/pods/$UID_/volumes/kubernetes.io~empty-dir/authz-socket" &&
  curl -s --unix-socket opa-api.sock http://opa/v1/policies | jq -r '.result[].raw'
kubectl -n aether-system get configmap aether-opa-policy -o jsonpath='{.data.policy\.rego}'

If they are the same, that node has converged. If they differ, either the update has not reached that node yet or the sidecar rejected it; the watch events above say which.

Recovering. Apply a policy that compiles; nothing needs a restart by hand.

• A running sidecar loads it when its node's kubelet delivers it (46 s in one measurement).
• A crash-looping authz container starts with it at the kubelet's next restart attempt. The back-off doubles up to 5 minutes, so recovery can take that long after the file is fixed: 8 s after helm upgrade in one measurement (the back-off happened to expire then), 125 s in another (a new pod on its sixth restart).

Why the file and not the directory. The sidecar is given /policy/policy.rego. A ConfigMap volume holds each file three times (policy.rego, ..data/policy.rego and the timestamped directory both point at), and OPA given /policy loads all three and refuses to start (rego_type_error: multiple default rules). OPA watches the file's directory in either case, which is how it notices the kubelet's ..data swap. A subPath mount would never be updated, so the mount is the whole volume.

Kind e2e: e2e/authz.sh, step d (nightly job authz): a policy change, a policy that does not parse, and the first policy again, with the pod UID, the restart counts and the Envoy epoch required to stay the same. The sidecar-restart and new-pod cases above were measured by hand and are not in CI.

A roll wedges both epochs after starting workers (#1050)

Symptom, per roll: the successor's last Envoy line is all dependencies initialized. starting workers, the draining parent's last is closing and draining listeners, the admin on 127.0.0.1:9901 stops answering, and about 30 s later the supervisor logs liveness watchdog fired; terminating for container restart for both epochs and, 15 s after that, drain deadline elapsed, killing envoy epoch for both. Existing connections keep serving (worker threads are fine); every new connection to or from the node fails until the fresh epoch 0 comes up, and source nodes log 503 UC …QUIC_TOO_MANY_RTOS / Network_blackhole_detected toward the node's QUIC twins.

The cause is a deadlock between the two Envoys' main threads over the hot-restart domain sockets (analysis). Once the child asks the parent to drain, the parent forwards every QUIC/UDP datagram that no parent session owns to the child, from its main thread, with a blocking sendmsg with no timeout; the child's UDP listeners stay paused until the parent exits, so for the whole parentShutdownTime every packet of every new QUIC connection takes that path. Meanwhile the child asks the parent for its stats on every 5 s stats flush with a blocking recvmsg, also with no timeout. When about 24 forwarded datagrams fill the child's queue as it enters that recvmsg, the parent is parked in sendmsg and never reads the stats request. Signals are handled on the main dispatcher, so neither process honours SIGTERM, which is why the supervisor ends up killing both.

The fix is the carried Envoy patch in the aether-proxy image (#1060, proxy/bazel/patches/envoy-aether1050-hotrestart-nonblocking-forward.patch): the parent forwards UDP to the child with a non-blocking sendmsg behind a bounded queue, and every child wait for a parent reply (stats, listen-socket hand-off, admin shutdown) keeps draining the forwarded datagrams and is bounded, so neither main thread can park on the other. It also covers the duplicateParentListenSocket (LDS add inside the window) case the stats-only workaround could not. On kind the forced fault wedged 0 of 8 restarts with it; the reference cluster's rev252 ran the e2e with the workaround off.

The workaround, proxy.hotRestart.skipParentStats, is now off by default (it was on from #1056 until the patch landed). It passes Envoy --skip-hot-restart-parent-stats, so the child never makes the stats call. Keep it as an emergency switch: if a roll wedges again with the signature above, set it true and roll. The cost is that the parent's gauges and its last ≤5 s of counter deltas are not merged into the child. Counters are already per generation (#708), so dashboards built on increase()/rate() do not change.

What the supervisor does about it (#1058). Two changes, so that a recurrence is both shorter and visible sooner:

• Early signal. A hot-restart child (epoch > 0) whose admin stops answering for 10 s after its first observed LIVE is logged once per epoch and counted. LIVE is the step that logs starting workers, and it is the anchor the #991 ready gate uses. The check covers streaks that begin before the parent is gone (first LIVE + parentShutdownTime + 2 s). It is not anchored on the fork, because a successor's xDS-gated init legitimately misses admin probes (#991). Silence before LIVE is still the handoff watchdog's job. On the #1050 timeline this fires about 20 s before the watchdog.

hot-restart child silent   epoch=54 silentSeconds=10 sinceLiveSeconds=17.8 threshold=10s

# Expected 0 per roll. Seeded at zero, so an empty result means the metric never arrived.
sum by (node) (increase(aether_supervisor_child_silent_total[30m]))

• No dead grace. When the liveness watchdog fires and both epochs of the hot-restart pair have been silent on the admin for the watchdog's bound, the supervisor SIGKILLs straight away. It no longer sends a SIGTERM that a blocked main thread cannot take and then waits drainTime + 5 s (15 s) for it. The pair is every epoch the supervisor tracks, plus the cross-pod predecessor until the pod has gone Ready, plus a newer epoch the admin last answered as. So the old pod and the successor pod each kill their own Envoy at once. Answers are attributed per epoch, so a handoff watchdog on a child that never went LIVE while the parent still answers is one silent epoch, and keeps SIGTERM then grace, as does a single wedged epoch with no handoff in flight:

liveness watchdog fired; terminating for container restart   error="admin watchdog: …" exitCode=1
both hot-restart epochs silent; SIGKILLing now instead of SIGTERM + drain grace (a blocked main thread cannot take SIGTERM)   epochs=53,54 silentFor="53=41.8s 54=31.0s" bound=30s graceSkipped=15s
killing wedged envoy epoch   epoch=54

drain deadline elapsed, killing envoy epoch after a watchdog line now means the SIGTERM path was taken, with one epoch still answering.

The container restarts in place. A watchdog exit is Run returning an error, which proxy-supervisor turns into os.Exit(1). The kubelet restarts the proxy container in the same pod (DaemonSet pods are restartPolicy: Always), so its restartCount goes up by one and its lastState.terminated.exitCode is 1. Nothing re-execs in place, and the DaemonSet does not replace the pod. The kind reproduction shows restartCount 1. Only a pod that is already terminating (it has a deletionTimestamp) does not get its container restarted. The reference-cluster reading of restartCount 0 after #1050 is therefore not what this code does. Read the count by container name, not by index. Before chart 2.4.9 containerStatuses held authz and proxy sorted by name, so with the ext_authz sidecar enabled [0] was authz. Since #1275 authz is a native sidecar and is listed under initContainerStatuses.

kubectl -n aether-system get pod <proxy-pod> -o jsonpath='{range .status.containerStatuses[?(@.name=="proxy")]}{.restartCount} {.lastState.terminated.exitCode} {.lastState.terminated.finishedAt}{"\n"}{end}'

The kind reproduction is hotrestart-wedge.sh: WEDGE_SKIP_PARENT_STATS=false WEDGE_FREEZE_S=6 wedged 4 of 4 restarts on 2026-09-28 against the unpatched image, and 0 against the patched one (see the header of the script). It and the per-roll soak gates (the hot-restart wedge gates, #1050) are maintained and documented with an external soak harness, outside this repository.

A node stalls for seconds during a handoff (#1093)

Symptom: on some handoffs, for about 2–7 s, requests from every node to pods on the rolling node take 1–3 s, the node's own 127.0.0.1:18081/-/-/live (a direct_response, no upstream) misses the prober's 2 s budget, and the mesh_dns tier times out on that node. It is not #1085 (listeners are present, no initial fetch timed out) and not #1050 (the admin keeps answering, child_silent stays 0).

What the 2026-10-01 soak established:

• It is node-local and not specific to the handoff. The same shape recurs at 0.5–1 s on node D and node E, dozens of times in 8 h, including in the no-roll window. The shape is destination-side h3 requests whose upstream_service_time to the local app is about 1 s on every pod at once. It is almost never seen on nodes A–C. Envoy's worker watchdog (envoy_server_worker_<n>_watchdog_miss_total, ≥ 200 ms without a loop iteration) moved only on node D and node E. A handoff is the tail of that distribution.
• The successor's workers are the ones that stall. At node D's 02:16 roll, workers 0 and 3 of the new epoch missed the watchdog between 02:16:45 and :50, which is the stall. A request a node D source received at 46.618 reached its node B destination at 49.099, 2.5 s later, and was then served in 6 ms. The new epoch's h3 connects took 250 ms–2.5 s for 40% of them, against ≤ 100 ms for most of the parent's.
• It is not CPU the profiler can see. Pyroscope (on-CPU only) shows both Envoys at 0.6–1.1 cores and the node at about 2.7 of 4 cores, with no unusual frame.

Whether the threads were starved of CPU, blocked, or busy is the open question. Since

1093 the supervisor answers it. Every 100 ms (--stall-sample-interval, 0 disables it)

it reads /proc/<envoy>/task/<tid>/{schedstat,stat,wchan} for the main and wrk:* threads of every epoch it runs. When a thread spends --stall-threshold (default 200 ms, Envoy's own watchdog miss bound) of a one-second window in one of the states below, it logs one line per epoch for that window:

class	meaning
starved	Runnable but waiting for a CPU (the schedstat runqueue delay). This is CPU contention or a node-level scheduling stall.
blocked	Asleep anywhere other than the idle ep_poll/do_epoll_wait. wchan= names where, e.g. __futex_wait (a lock) or unix_wait_for_peer (a full hot-restart socket).
busy	On a CPU for ≥ 90% of the window. The event loop is saturated.

Example line:

envoy thread stall   epoch=119 pid=15 windowMs=1000 thresholdMs=200
  threads=["wrk:worker_3[starved] cpu=140ms runq=620ms blocked=0ms"]
  nodeBusyPct=71.2 nodeIrqPct=3.1 nodeSoftirqPct=9.8 nodeStealPct=0
  nodeHottestCPUIrqSoftirqPct=38.5 nodePSICPUSomeMs=640 nodePSIIRQFullMs=55
  trackedEpochs=1 handoffPeer=118

The kernel charges a runqueue wait when it ends, so runq can exceed windowMs: a 5 s starvation is reported whole in the second it ended. The supervisor shares the proxy container's cgroup, so it is starved along with Envoy and reports the stall afterwards. On kind, a 6 s cpu.max throttle of the proxy container logged every worker as [starved] cpu=0ms runq=~5500ms with nodeBusyPct=7.2: Envoy-only starvation on a quiet node.

The node* fields come from /proc/stat and /proc/pressure, which are not namespaced, so they describe the node for the same second. Read them like this:

• A stall that every process on the node shares shows high PSI cpu-some and a hot CPU.
• A stall only Envoy has shows blocked with a wchan, or busy, on a quiet node.

Who had the CPU (#1392). A line with a [starved] thread also names the top CPU consumers on the node over the stall. [blocked] and [busy] lines do not carry them: a blocked thread is not waiting for a CPU, and a busy one is the consumer.

envoy thread stall   epoch=210 pid=15 windowMs=1000 thresholdMs=200
  threads=["wrk:worker_0[starved] cpu=310ms runq=611ms blocked=0ms"]
  nodeBusyPct=99.1 … nodePSICPUSomeMs=1010 nodePSIIRQFullMs=12
  topCgroups="/podruntime/kubelet=1240ms /kubepods/burstable/pod0f3c1a2b-…=820ms /kubepods/besteffort/pod9d8c7b6a-…=610ms /podruntime/runtime=180ms /=95ms"
  topCgroupsOverMs=1000
  topProcs="unavailable: /proc shows only this pod's PID namespace (no hostPID)"
  trackedEpochs=1 handoffPeer=-1

(An illustration of the format, not a line from a cluster: the fields have not run on a cluster yet. The cgroup names are shortened here; the line carries each pod's whole UID, and a path longer than 64 bytes keeps its tail behind ....)

field	meaning
topCgroups	The cgroups that used the most CPU time, largest first, as <cgroup path>=<n>ms. At most --stall-top-consumers entries (default 5, at most 20; 0 turns the consumer sampling off; the chart does not pass the flag, so the default applies). CPU time, not wall time: on a 4-core node one second holds up to 4000 ms.
topCgroupsOverMs	The interval the figures cover, ending when the line is logged. It is the stall, not the window, and it starts where the wait BEGAN: the kernel charges a wait when it ends, so a thread that waited 5 s is reported in the second its wait ended, and a 500 ms wait seen 100 ms into a window began 400 ms before that window. The supervisor works the start out for each starved thread (the tick it saw the wait on, minus the wait) and takes the consumers from the last one-second sample at or before the earliest such start, so the field is usually a whole number of seconds and can be longer than both windowMs and runq= (that 500 ms wait is reported with 2000). When the supervisor's own tick came late (it was waiting for a CPU too), the start is moved back by how late the tick was, because the wait may have ended that long ago; the interval then errs on the long side. Never less than the window.
topCgroupsTruncated	On the line, as true, only when the stall began before the oldest sample kept: the supervisor keeps 32 cgroup samples (about 32 s; fewer right after it starts) and 4 process scans. The figures then cover only the END of the stall, the last topCgroupsOverMs of it. Absent: the whole stall is covered.
topProcs, topProcsOverMs, topProcsTruncated	The same per process, as <comm>(<pid>)=<n>ms. Only when the proxy pod shares the host PID namespace, which the chart does not ask for; otherwise the field says so.
unavailable: <reason>	The value of topCgroups or topProcs when that source could not be read (no cgroup v2, only the container's own cgroup visible, a scan over its 250 ms budget, no earlier sample yet). The stall line is logged regardless.

No figure is larger than the interval on every CPU of the node. That count is the node's online CPUs (/sys/devices/system/cpu/online), not the CPUs the proxy container may run on: under a cpuset those differ, and the smaller one would cut real usage down. The envoy thread-stall sampler running line at startup carries it as nodeCPUs; when the file cannot be read it says unknown (...) instead and the figures are not capped by a CPU count at all (a cgroup is still held to what its parent used).

How to read topCgroups:

• The entries do not overlap. Each cgroup's figure excludes the cgroups below it that have their own entry, so /kubepods/burstable is what ran there outside every pod and / is what was charged to the root cgroup itself once every listed cgroup below it is taken out. That is commonly kernel threads, but a userspace task attached directly to the root is counted there too (every task is in some cgroup), so / alone does not name the consumer. A pod is one entry: its containers are not listed separately.
• On Talos the kubelet is /podruntime/kubelet, containerd /podruntime/runtime, the Talos services /system/<name>, and a pod /kubepods/[burstable/|besteffort/]pod<uid>. Name a pod from its UID with kubectl get pods -A -o jsonpath='{range .items[?(@.metadata.uid=="<uid>")]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'. The proxy's own pod is in the list like any other.
• A pod created in the last 10 s may not have its own entry yet: the supervisor lists the cgroup directories every 10 s and reads only the known ones in between, so the new pod's time is in its parent's entry (/kubepods/burstable) until then. The 10 s hold while the node is starved too: every tenth starved second the line is made from a fresh listing, so a pod created during a long incident appears on it. On that one line the new pod's figure is an upper bound (its usage since it was created, held to what its parent used in the interval); it is exact once the interval starts at or after that listing, which for a one-second stall is the next line. If that listing runs past the 250 ms budget, that one line says unavailable and the next reads the known cgroups again.
• cpu.stat is kept by the kernel, so the figures include processes that lived and died inside the interval (an exec probe, a container start), which a process list misses.

Grep it like the rest of the line, by message text:

service.name:aether-proxy "envoy thread stall" "[starved]" "topCgroups=" k8s.node.name:<node>

service.name:aether-proxy "envoy thread stall" "[starved]" "/podruntime/kubelet=" | stats by (k8s.node.name) count() lines

The second query counts the starved seconds in which the kubelet was one of the top five consumers on each node. That is the question #1389 had to leave as an inference: on one worker node the hot seconds sat on a 30 s phase that followed the kubelet and cAdvisor scrapes, and nothing recorded CPU per consumer at one-second resolution to say the kubelet was what burned it. With these fields each of those seconds names its consumers: /podruntime/kubelet at the top with several hundred ms on the :12/:42 seconds confirms the scrape, and a pod UID at the top instead names the second load that issue could not identify.

What it costs, measured on a 20-thread x86 workstation (not on a Talos node): the cgroup sample is one cpu.stat per cgroup down to pod level, read once a second whether or not anything is starved (a delta needs a "before", and a stall is only known when it is over). 134 cgroups took 2.6 ms and 402 system calls (open, read, close per file), so about 0.3% of one core for a node with ~110 pods. Listing the directories is dearer than reading the files (126 directories: about 7 ms) and is done every 10 s, starved or not, together with one read of the node's online-CPU list. The process scan is one /proc/<pid>/stat per process, about 22 µs each: 300 processes are about 300 files, 907 system calls and 6.6 ms. It runs every 10 s for a baseline and once more for each starved window, and not at all without the host PID namespace. A scan that runs past 250 ms is abandoned and the field says so.

What the proxy pod can read. It is privileged, and the runtime gives a privileged container the host's cgroup namespace, so /sys/fs/cgroup there is the node's whole hierarchy with no extra mount. It does not set hostPID, so /proc lists only the container's own processes; /proc/stat and /proc/pressure are not namespaced, which is why the node* fields work. Per-process names would need hostPID: true on the proxy DaemonSet.

The counter aether_supervisor_envoy_thread_stalls_total{aether_supervisor_stall_class} is seeded per class. Grade it per roll and per node:

sum by (node, aether_supervisor_stall_class) (increase(aether_supervisor_envoy_thread_stalls_total[30m]))

service.name:aether-proxy "envoy thread stall"

How to tell which class you have, and where to go from there:

what the lines show	what it is	where to look
[starved] with a large runq=, nodeBusyPct in the high 90s and a high nodePSICPUSomeMs	The node is out of CPU. Every runnable thread waits, Envoy's included.	"Sizing nodes for a proxy hot restart" below.
[blocked] with a wchan=	A kernel path: a lock, a full socket, I/O.	The function wchan= names.
[busy]	One callback keeps an event loop on its CPU.	The profiler (Pyroscope) for that Envoy, over the same seconds.

Sizing nodes for a proxy hot restart

The measurements in this section come from the reference cluster (5 workers, 4 cores each, arm64) on 2026-10-01 and 2026-10-02. They are readings from that cluster under the soak load, not constants. Take the method, then measure your own nodes.

What a handoff costs. For the length of a hot restart two Envoy processes run on the node: the successor's init (about 3–15 s, gated on xDS) plus proxy.hotRestart.parentShutdownTime (15 s by default), during which the parent drains. On the reference cluster a handoff cost about 10–13 CPU-seconds of Envoy CPU on top of steady state, and up to about 1.9 cores in the 5 s after the successor started its workers, on a proxy that used 0.8–0.9 core in steady state. Rule of thumb: keep about one core of headroom on each node for the roughly 20 s of a handoff.

What happens without it. On the nodes that ran 81–85% busy in steady state, handoffs pushed them to 99–100%. The stall lines classed every Envoy thread starved (0 blocked), parent and child alike. The prober's 2 s liveness probe timed out only in the most starved handoffs. Of the 30 handoffs in one 8 h soak, the two with the most starvation failed 4 probes of 719,979. Starvation there was 54.8k and 85.9k ms of runq, summed over all threads in the bracket [parent drain − 5 s, parent exit + 5 s]. No handoff at or below 41.8k ms failed one. Client traffic was not affected (k6: 0 failures in 9,179,946 requests) because the failed attempts were retried. This is a threshold effect of CPU oversubscription, not a functional bug: below the threshold a handoff costs latency, above it a probe misses its budget.

How to see it. Count the stall windows per node and class over the roll, then read the lines for the node that stands out (the fields are explained in the section above):

sum by (node, aether_supervisor_stall_class) (increase(aether_supervisor_envoy_thread_stalls_total[30m]))

service.name:aether-proxy "envoy thread stall" | stats by (k8s.node.name) count() stalls

service.name:aether-proxy "envoy thread stall" "[starved]" k8s.node.name:<node>

The supervisor's records are not field-parsed: the whole line sits in _msg, so there is no threads field to filter on, and a filter has to match the message text. Select a class with its bracketed phrase, "[starved]", "[blocked]" or "[busy]". The bare word blocked matches every line, because each thread entry carries blocked=<n>ms.

A node whose steady-state count is several times its peers' is the one a handoff will tip over. A starved line whose nodeBusyPct is near 100 and whose nodePSICPUSomeMs is in the hundreds is this section. Rank the nodes by CPU requests and name what holds them with the commands in §7 "Pre-flight: node headroom before a roll (#812)".

What to size.

• Workload CPU requests. The scheduler places pods by request, not by use. Pods with token requests (the reference-cluster test services request 10m) are packed onto whichever node shows room, and on the reference cluster one node ended up with about 4× the steady-state starvation of its peers. Give workloads requests that reflect what they actually use, so the scheduler spreads them.
• proxy.resources.requests.cpu (chart default 500m). Raising it to 800m on the reference cluster cut steady-state starvation on the two hot nodes by 26–35%. It buys cgroup weight (the proxy wins more of a contended CPU), not capacity: the stall during a handoff was unchanged within noise. The proxy has no CPU limit on purpose, and should not get one. Remember that during a DaemonSet roll the surge pod's request and the departing pod's are both on the node, so the node needs room for two proxy requests at once.
• Co-tenants. Latency-insensitive pods on a hot node (profilers, collectors, batch jobs) use exactly the headroom a handoff needs. Move them to a quieter node, or keep them off the nodes that carry the mesh's busiest services.
• proxy.hotRestart.drainTime (default 10s). Do not stretch it to spread the cost. At 30s on the reference cluster the peak was no lower, there were more starved seconds, Envoy spent about 40% more CPU per handoff, and rolls took 70% longer. Keep the default.
• proxy.concurrency (default 0: Envoy's own default, one worker per core unless the container has a CPU limit or a restricted CPU mask, see below). In an A/B, 2 workers instead of 4 cut handoff starvation per thread by about 27% and steady-state starvation by 57–81%. Do not change it on a live mesh until both prerequisites are deployed. On the reference cluster a 4→2 change crashed successors (Mismatched worker index in HotRestartingChild::onForwardedUdpPacket, #1126), and the old pods' self-drain drained the new pods' Envoys (#1127). Together they left two nodes not accepting new pods' connections for about 12 minutes. The crash is fixed by the carried Envoy patch envoy-aether1126-forwarded-udp-worker-index.patch, in proxy images built from it or later. The #1127 supervisor fix is the other prerequisite: the old pod's drain now reaches only its own Envoy (see "What the proxy supervisor does on SIGTERM"). A crashed successor still costs the old pod its graceful drain, because its Envoy has already handed the admin over, so it is SIGTERMed instead.

A count change is a drain + fresh start, not a hot restart (#1136). Even without the crash, a hot restart between different counts is not hitless: the child attaches its QUIC connection-ID steering program (CID % concurrency) to the sockets it inherits, the program applies to the whole reuse-port group, and about half of the parent's live HTTP/3 connections are steered to a worker that does not own them (Mismatched worker index. expected 3, actual 1, then a stateless reset). On the 2026-10-02 4→2 rollout that was 34 mismatched batches and 12 stateless resets fleet-wide; on kind, 14 mismatched lines and 5–10 503s per roll in either direction. So the new pod's supervisor reads the live predecessor's command_line_options.concurrency from /server_info (trusted only from a LIVE aether supervisor's Envoy at the heartbeat's epoch; anything else hot-restarts as before) and, when it differs from its own, logs envoy worker count changes across this handoff; NOT hot-restarting with both counts, drains the predecessor gracefully over proxy.hotRestart.drainTime, stops it, and starts a fresh Envoy at epoch 0. The pod goes Ready only when its own fresh Envoy is LIVE. The cost per node is the drain window plus a gap with no listeners while the fresh Envoy initializes: about 2 s on kind (requests waited up to 2.1 s on SYN retransmits; 0 failed), longer on a busy node, where a handoff's init takes 3–15 s, so expect some failed requests and prober misses on each node in turn. Count it with aether_supervisor_handoff_mode_total{mode="fresh_after_drain"} (expected once per node on the rollout that changes the count, 0 otherwise). proxy.hotRestart.hotRestartOnConcurrencyChange: true forces the old hot restart. A surge pod deleted in the middle of this (the roll was reverted) finishes the replacement before it terminates, and an Envoy left drained by a supervisor that died is hot-restarted in place by its own: see "A reverted roll that changes the Envoy worker count" (#1679). The kind reproduction, proxy-concurrency-change.sh, is maintained with an external harness, outside this repository.

What "its own" count is when proxy.concurrency is 0 (#1442). The supervisor then passes no --concurrency, and Envoy does not run one worker per core: it runs max(1, min(online CPUs, CPUs in its affinity mask, its cgroup CPU limit)), where the limit is the quota of the container's own cgroup (cgroup v2 cpu.max, or cgroup v1 cpu.cfs_quota_us / cpu.cfs_period_us) divided by the period and rounded down: a 1500m limit is one worker, 2500m two. No ancestor cgroup is read, so a limit set on the pod as a whole and not on the proxy container does not lower the count (measured: a 1.5-CPU quota one level up left 20 workers), and ENVOY_CGROUP_CPU_DETECTION=false in the container's environment turns the cgroup term off. Measured on the pinned proxy on a 20-CPU machine: 20 workers unrestricted; 1, 2 and 3 under a mask of 1, 2 and 3 CPUs; 1 under a quota of 0.5, 1, 1.5 and 1.99 CPUs, 2 under 2 and 2.5, 3 under 3; an explicit --concurrency 4 stayed 4 under a 2-CPU mask and under a 1.5-CPU quota. The supervisor computes the same number from the same sources (it forks Envoy, so both are in one container and see the same mask and cgroup), and the WARN of a count change says where its number came from (successorConcurrencyFrom, for example envoy default: min(online CPUs 4, CPU affinity 4, cgroup CPU limit 2)). Before #1442 it assumed one worker per online CPU, so a proxy with a CPU limit or a restricted mask and no explicit count compared, say, 4 with a predecessor's 2 and took the drain + fresh start on every roll: fresh_after_drain counting on every restart, not only on the rollout that changes the count, is the signature. Two consequences remain by design. Changing proxy.resources.limits.cpu across a whole-CPU boundary with proxy.concurrency: 0 is a real count change and is handled as one. And when the online CPUs cannot be read the count is unknown and the supervisor hot-restarts, as for every other inconclusive check. //agent/test/envoyargs starts the pinned Envoy under an affinity mask (and, where the machine lets an unprivileged test make a cgroup with a quota, under a CPU limit) and fails when the supervisor's number is not Envoy's, so a proxy pin bump that changes the rule is caught there. - The node agent. It has no CPU limit since #1119 (agent.resources.requests.cpu 200m, GOMAXPROCS=2), because a CFS quota parked snapshot builds while they held the snapshot-cache mutex. Do not add one back to save headroom.

Timeouts under starvation (#1128). The destination's connect timeout to its local app is 1 s (meshconst.AppConnectTimeout, #1122), so the 503 UF for a pod in teardown leaves the node while the pod's veth still exists and the source can retry it elsewhere. A live app answers a loopback connect in microseconds, but on a starved node the same 1 s timeout fires for live pods too: the two starved handoffs above produced 111 destination 503 UF with connection_timeout. The source retried each one, at a cost of at least 1 s of latency. That is the intended trade, not a fault, and the fix is CPU headroom, not a longer timeout. Note also that 1 s is the kernel's initial SYN retransmit timeout, so a single dropped SYN cannot be recovered inside it (tracked on

1093).

Reading a hot-restart overlap (#1333)

During a proxy roll two Envoy generations serve the node at once: the parent drains while the child takes new connections. The stats cannot tell them apart; the access log can.

What the stats do during the overlap (pinned Envoy; the first three were also seen in worker-04's raw samples at the 2026-10-06 20:39Z roll):

• The parent stops exporting when the child starts. The child's start makes the parent shut its admin down, and that step cancels the parent's stat-flush timer. The parent does not flush at exit either. From then on every exported sample is the child's. Expect one missing flush interval at the switch.
• The child's first counter sample contains the parent's delta. The child pulls the parent's counter deltas and adds them to its own, so a counter keeps rising across the roll and nothing in it says which generation counted what.
• envoy_server_uptime roughly doubles while the parent lives. It is an accumulated gauge: the child's value plus the parent's. Every gauge that accumulates does the same (active connections, active requests), then drops back when the parent exits. A step down at parent exit is the merge ending, not a loss.
• Connections are the one thing the gauges split. envoy_server_parent_connections is what the parent still holds, and envoy_server_total_connections is both generations, so the child's connections are envoy_server_total_connections - envoy_server_parent_connections.
• Histogram samples the parent records during the overlap are not exported. This one is inferred from the Envoy source (histograms are not part of the parent-to-child transfer), not measured.

So a metric label naming the pod or the epoch separates nothing: there is only ever one exporter, and its numbers already include the other generation's. It would also create about 4,150 new series per node per roll.

What the access log says instead. Every HTTP access-log row (log_name:aether_access_logs, both reporters) carries:

field	is	reads - when
proxy_epoch	the hot-restart epoch of the Envoy process that wrote the row; the same number as envoy_server_hot_restart_epoch. Within one overlap the lower number is the parent	the proxy's supervisor predates the field (it exports AETHER_RESTART_EPOCH to each Envoy child)
connection_id	Envoy's id for the downstream connection the request rode. Unique within one Envoy process only: pair it with node_name and proxy_epoch. A small counter for a TCP connection, a 64-bit number for a QUIC one	never (0 = no id)
ds_cx_age_ms	how old that connection was when the request started: from Envoy accepting it to the request's first byte. Small = the request paid for a new connection, large = reuse	never
ds_hs_ms	from accepting the connection to the end of its TLS handshake. A property of the connection, repeated on every row that rode it	the downstream is not TLS: every reporter:source row (the application speaks cleartext to its proxy) and a cleartext inbound
us_tx_beg_ms	from the request's first byte to the first byte sent upstream: routing, plus waiting for an upstream connection (a new upstream handshake shows up here)	nothing was sent upstream (local reply, no healthy upstream, connect failure)

The L4 stream (log_name:aether_l4_access_logs) carries proxy_epoch only. An L4 record is written when the connection closes, so there it names the generation that held the connection.

# Which generation served, or failed, the requests of a roll window on one node
log_name:aether_access_logs AND reporter:destination AND node_name:"node-d" AND _time:[<start>, <end>]
  | stats by (proxy_epoch, response_code, response_flags) count()

# Requests that paid for a new inbound connection and a slow handshake
log_name:aether_access_logs AND reporter:destination AND ds_cx_age_ms:<50 AND ds_hs_ms:>=100
  | stats by (node_name, proxy_epoch, protocol) count(), max(ds_hs_ms)

# Failures per connection, not per request: N rows sharing one connection are one event
log_name:aether_access_logs AND reporter:destination AND response_flags:!"-"
  | stats by (node_name, proxy_epoch, connection_id) count()

What these fields cannot tell you. Time spent before Envoy read the packet. Envoy takes no kernel receive timestamp, so ds_cx_age_ms and ds_hs_ms start when a worker accepted the connection (for QUIC, when it created the session from the first datagram it read), not when the peer sent its SYN or first datagram. A request that waited in a socket queue on an established connection while the workers were starved shows normal values on the destination row and a long duration_ms on the source row. For that gap, compare the two sides' start_time on one x_request_id, and read node CPU and run-queue data (see "A node stalls for seconds during a handoff (#1093)" above).

The time-point names behind the three durations are checked against the pinned Envoy by a live test (//agent/test/mtlspool, TestAccessLogConnectionTiming), on an mTLS connection and on an HTTP/3 one. envoy --mode validate cannot do it: Envoy accepts a time-point name it does not know and renders -.

Envoy's UDP dropped-datagram counter after a hot restart (#1353)

Do not gate or alert on it across a proxy roll. The stat listener.<prefix>.udp.downstream_rx_datagram_dropped, and the warning Kernel dropped N datagram(s) in the proxy log, over-report after every hot restart that follows a real drop.

Why. Envoy derives the count from SO_RXQ_OVFL, which the kernel keeps for the lifetime of the socket. Envoy's own running total starts at 0 in each process. A hot-restarted child inherits the parent's UDP sockets, so at its first read it reports the socket's whole history as new drops. That first read is when the parent exits.

Measured (kind, the pinned proxy image, during the #1332 experiment):

• A roll with 743 real kernel drops: the stat read 1486, and the child logged 731 + 12 at parent exit.
• The next roll, with 0 real drops: the child still logged Kernel dropped 273 and 21, and the stat read 294.

So a non-zero value after a roll says only that this socket dropped a datagram at some point since it was bound. It does not say the roll dropped anything.

What to read instead. The kernel's own counters in the pod's network namespace, taken before and after the window you care about:

# On the node. PID is any process of the pod (its pause container will do).
PID=12345
# Per-socket drops (last column) for the QUIC inbound port:
nsenter --net="/proc/${PID}/ns/net" cat /proc/net/udp | awk 'NR==1 || $2 ~ /:4658$/'   # 0x4658 = 18008
# Namespace-wide receive-buffer overflows:
nsenter --net="/proc/${PID}/ns/net" nstat -az UdpRcvbufErrors

Both are cumulative as well, but they are read by you at two instants, so the difference is the window's drops.

Nothing in this repository or in the reference-cluster alert rules uses the Envoy stat today. A larger receive buffer is not the fix for drops during a starved handoff: see #1332 (it removes the drops and changes neither the slow requests nor the failures) and "Sizing nodes for a proxy hot restart" above.

Source h3 requests die on a stateless reset at a destination's roll (#1054)

Symptom, per roll of a node's proxy: source proxies on other nodes log a few 503 UC lines toward the rolling node's pods with response_code_details containing QUIC_PUBLIC_RESET|FROM_PEER|Received_stateless_reset, clustered at the moment the draining parent exits (about parentShutdownTime after the successor started).

The cause (mechanism): a source's h3 connection to the draining parent was still open when the parent exited. Its next packet reached the child, which does not own the connection ID and answers with a QUIC stateless reset; the source accepts it because the token is derived from the connection ID alone, so parent and child mint the same one. The request in flight fails, and the source's retry policy does not cover it (the request was already sent). Two things kept such a connection alive past the parent: under Envoy's default gradual drain strategy the GOAWAY is a coin flip per response, so a busy connection can dodge it for the whole drain, and an idle connection gets no GOAWAY at all while the h3 pool's 30 s idle timeout outlasts the 15 s parent-shutdown window.

The fix is two chart values:

• proxy.hotRestart.drainStrategy (default gradual; immediate is an opt-in) passes Envoy --drain-strategy. immediate puts a GOAWAY on every response from the start of the drain, but on the reference cluster (2026-09-28) it made the #1054 resets worse (4/8/0 per roll vs 1–3 per run under gradual): more parent connections close inside the drain window, and those closes are what sources then see reset. It is also server-wide (h2 and h3 reconnections bunch at drain start; LDS and pod-termination drains switch too). Three Envoy changes are the fix under gradual: #1064 (envoy-aether1054-hotrestart-terminate-wait-h3-goaway.patch: the child unpauses its UDP listeners only after the parent exits), #1066 (the draining parent answers its own time-wait connections; carried as a patch until the 1.40.0.envoy pin, which contains it upstream as Envoy PR 48206, behind the same envoy.reloadable_features.quic_hot_restart_time_wait_before_forward) and #1069 (envoy-aether1054c-paused-udp-listener-no-read.patch, chart 1.0.11), the actual root cause: a paused child UDP listener still read the inherited socket when the parent forwarded it a CHLO, because the QUIC listener injects a read event to process the buffered handshake, and every packet it dequeued that way belonged to a parent connection and drew a stateless reset.
• agent.eastWestQuicIdleTimeout: 8s (the agent's --east-west-quic-idle-timeout) is the idle timeout on the quic: twins only; h1/h2 keep 30 s. It closes the connections that were idle when the drain started, before the parent exits. The chart refuses to render unless eastWestQuicIdleTimeout + 5s < proxy.hotRestart.parentShutdownTime, so lowering the parent-shutdown time needs the idle timeout lowered with it. The cost is one extra QUIC handshake for a (source, destination) pair that sits idle between 8 s and 30 s. Since #1087 the QUIC transport's own idle_network_timeout (8 s, negotiated to the minimum on both ends) also closes a twin connection with no open stream after 8 s, so raising this flag above 8 s no longer lengthens reuse.

A request in flight at the parent's exit still dies, as it does on h2. The soak gate (the h3 stateless-reset gate, #1054) is documented with an external soak harness, maintained outside this repository; so is the kind leg, eastwest-quic-hotrestart.sh with HR_MODE=sparse (two nodes).

504 UT after exactly 15 s over a quic: twin to a terminating pod (#1087)

Symptom (before the fix): a few source-reporter 504 UT response_timeout lines per churn-heavy roll, each duration_ms ≈ 15000, upstream_cluster a quic: twin, upstream_host an endpoint whose pod was terminating. The destination-reporter lines for the same x_request_id show the request arrived: 503 UF after 5 s, or code 0 after 3–5 s.

Cause: the destination pod's network went away (CNI DEL deletes the veth) after the request was delivered and ACKed, so nothing the destination proxy wrote back could leave the pod netns. QUIC has no RST, and the source had nothing in flight, so no PTO or blackhole detector ran. The only timers left were the QUIC idle timeout (300 s from the inbound listener's default) and QUICHE's 15 s keep-alive, and the 15 s route timeout won. h2 had the same exposure for an ACKed request (no TCP keepalive or HTTP/2 PING was configured); #1104 closed it, see the next section.

Since #1087 every twin carries quic_protocol_options with connection_keepalive.max_interval: 1s and idle_network_timeout: 8s. While a request stream is open the source PINGs after 1 s of silence; a live destination ACKs it, however slow its application is, and a dead one is closed 8 s after the first unanswered PING. The bound is about 10 s after the destination's last packet (9 s measured on kind). It is 8 s and not shorter because of the #1093 node-local worker stalls (5–7 s episodes): an idle deadline inside a stall would fail live requests, and the soak's failure count is the same either way (those requests were delivered, so they are not retryable), so only the hang gets shorter. Tighten it once #1093 is fixed. What you see now is a 503 UC with response_code_details starting upstream_reset_before_response_started{connection_termination and mentioning QUIC_NETWORK_IDLE_TIMEOUT, well under 15 s.

The request is not retried, GET or POST. It was sent, so it may have reached the application, and the mesh retry policy retries only connect-failure, refused-stream, reset-before-request and a 503 response. New requests to the dead endpoint fail the QUIC handshake within the 2 s connect timeout and are retried on another endpoint. The kind proof is eastwest-quic-deadpeer.sh, maintained with an external harness, outside this repository.

A source keeps picking an endpoint seconds after its drain mark (#1103)

Symptom: a request is sent to an endpoint whose destination agent logged pod terminating: endpoint marked draining ahead of shutdown seconds earlier, and fails (#1087's 504 UT/503 UC). Join the source access-log line to the destination agent's drain mark by upstream_host / pod IP.

The drain mark itself is fast. The registrar applies it to its snapshot and broadcasts it the moment the RPC lands; the other replica picks it up through its etcd watch (200 ms debounce). On 2026-10-01 the peer replica had it 0.15 s after the mark. A source whose ADS stream to its node agent is up rebuilds and pushes EDS within the snapshot cost (#1105). Envoy excludes an EDS DRAINING host from new selections (upstream_impl.cc setEdsHealthFlag → EDS_STATUS_DRAINING; excludeBasedOnHealthFlag puts it in the excluded set) and panic routing is off (healthy_panic_threshold: 0), so nothing selects it once the update is applied.

On the kubernetes registry backend (the chart default) the mark itself never leaves the replica that received it (#1124). That backend derives endpoints from Pods and ignores writes, so the agent's DRAINING mark lives only in the receiving replica's snapshot. Before #1124 the other replica listed the pod HEALTHY for its whole preStop (the kubelet keeps a terminating pod Ready until its containers stop) and only polled every 5 s, so every source agent attached to it kept sending new requests into the dying pod. On kind with a 15 s preStop: 0.22–0.26 s mark → last request when the source and destination agents were on the same replica, 14.9 s (the full preStop) when they were not. Since #1124 the backend lists a pod whose deletion was requested as DRAINING while it is still Ready (UNHEALTHY once it is not), and the registrar syncs from a managed-pod informer instead of only the poll, so every replica hears the drain from the pod's deletionTimestamp within the 200 ms debounce (kind: cross-replica 0.46–0.60 s, same-replica 0.17–0.25 s). The registrar logs kubernetes registry initialized ... podWatch=true and registry supports change notifications. The agent's phase-2 UNHEALTHY (pool close ~1 s before SIGTERM) is derived from the Pod the same way since #1144. Every replica lists a terminating, still-Ready pod UNHEALTHY from drain.PoolCloseAt, which is the deletion request (deletionTimestamp − deletionGracePeriodSeconds) plus the agent's own drain.PoolCloseDelay (sleep preStop − 1 s, floor 2 s). A per-pod timer in the backend wakes the sync at that moment. Because deletionTimestamp has one-second resolution, this can land up to 1 s before the agent's own mark, never after it. Before #1144 the other replicas saw UNHEALTHY only when the Ready condition dropped, after SIGTERM. Since #1145 the receiving replica holds an agent's write only until its next sync (the first listing that started after the RPC; the informer or the 5 s poll), then serves the Pod-derived endpoint like every other replica. Before, it kept the agent's version for the pod's lifetime whenever the two differed in any field (the CNI path's health-check mode, for one). Find which replica an agent is on with conntrack -L -p tcp --orig-src <agent IP> --orig-dst <registrar Service IP> on its node (the kind reproduction drain-propagation.sh, maintained with an external harness, outside this repository, does this per run).

What was slow is a source proxy that had no ADS stream: its own node agent was restarting. While the stream is down the proxy routes on its last config, so it cannot hear a drain mark however fast the registrar is. All five 2026-10-01 TRIPLE failures were of this kind:

source	source agent	drain mark (destination)	requests	proxy reconnected
Node B (proxy also rolling, parent epoch serving)	down 05:41:08, serving xDS 05:41:20.51	05:41:20.14 (node E)	05:41:23.92, 24.04	child CDS 05:41:22.51
Node D	down 05:41:24.45, new pod 05:41:33.16, serving xDS 05:41:39.93	05:41:36.42 (node B)	05:41:39.78–39.97	05:41:48.34, 8.4 s after the agent was serving

The 8.4 s is Envoy's xDS reconnect backoff: fully jittered exponential, 500 ms base, 30 s cap (xds_manager_impl.cc SubscriptionFactory::RetryInitialDelayMs / RetryMaxDelayMs; backoff_strategy.cc returns random() % interval and doubles the interval). The longer the agent was away, the longer the proxy waits after it is back. Since #1103 the bootstrap's ads_config carries retry_policy.retry_back_off {base_interval: 0.1s, max_interval: 1s}, so the proxy reconnects within 1 s of the agent serving. Checked by //agent/test/envoy_validate:envoy_validate_test (TestNodeProxyADSReconnectBackoffIsBounded) and //charts/aether:aether_proxy_bootstrap_ads_reconnect_backoff_test; the kind proof is drain-propagation.sh, in that external harness.

The fast reconnect needs the agent's half of #1103. A restarted agent holds its own SVID before the SPIRE bridge has re-delivered the pods' certificates, and until then the #1049 gate keeps those identities' quic: twins out of the snapshot. A proxy that connects to that snapshot is told to remove the twins it holds (cds: response indicates 0 added/updated cluster(s), 1 removed cluster(s) right after the reconnect) and the requests its routes still send to a twin fail until the certificate's snapshot re-adds it. On kind, with the 1 s cap and main's agent, that happened on 8 of 18 restarts, for 1–9.6 s each, and once cost 56 × 503. The 30 s default mostly hid it by reconnecting late. The agent now opens its xDS socket only once the snapshot carries every local certificate (bounded at 5 s; local workloads' client certificates delivered; serving the complete snapshot, or a WARN with awaiting_client_cert on timeout). On the reference cluster the certificates already arrive before the registry load finishes (all five TRIPLE agents on 2026-10-01: 0.3–2.6 s before), so the wait costs nothing there.

What it does not cover: the agent's own outage. A drain mark that lands while the source agent is down is heard only once the new agent serves xDS: on node D that was 15.5 s after the old agent stopped (8.7 s pod replacement, then 6.8 s to identity, the registrar watch and the first registry load). The two-phase drain gives the destination only its preStop window, so a source agent restart that overlaps a destination pod's deletion can still send it requests.

To see whether a proxy is blind right now: control_plane.connected_state on its admin (0 = no ADS stream). On reconnect the proxy logs cds: response indicates ..., and the agent logs fresh xDS stream re-stated QUIC twins when the proxy holds twins.

Stored vs in place vs applied: registrar snapshot lag and divergence (#1193)

An endpoint change passes through three places: stored (the external registry: etcd), in place (a registrar replica's snapshot, rebuilt by its sync loop), and applied (each agent's watch cache). Since #1193 the registrar's snapshot version carries the etcd revision the snapshot was listed at, which makes the three comparable across processes and replicas, and always the content hash, which alone names the contents:

Version	Meaning
<rev>.<hash>	exactly the etcd listing at revision rev
<rev>+<hash>	the listing at rev plus something not yet in it: a write-behind intent still overlaid, or an agent RPC applied since the last sync
hash:<hash>	the backend has no revision (kubernetes). The version is content-addressed

<hash> is 16 hex digits of a sha256 over the snapshot's endpoints. The revision is never trusted for equality: a rebuilt etcd restarts its revisions at 1, and two binaries can decode one stored value differently.

# in place: how far each registrar replica's snapshot trails the store
aether_registrar_store_revision - aether_registrar_snapshot_revision

# applied: how far the fleet's agents trail the store (worst agent)
max(aether_registrar_store_revision) - min(aether_agent_registry_last_version)

# divergence: more than one endpoint set at ONE revision, with no write-behind
# intent pending. MUST be empty. See "Divergence rule" below. Each selector
# reads a replica only while its newest sample is at most 150 s old, so a
# registrar pod that was replaced is not compared (#1560).
count by (job, revision) (
  count_values by (job, revision) (
    "hash",
    last_over_time(aether_registrar_snapshot_content_hash[150s])
    * ignoring (revision) group_left (revision)
    count_values without () ("revision", last_over_time(aether_registrar_snapshot_revision[150s]))
  )
) > 1
unless on (job) (max by (job) (aether_registrar_writebehind_queue_depth) > 0)

The replica-lag line settles at 0 between changes. A short climb during churn is the watch-to-sync debounce (200 ms). A replica that stays behind has a stalled sync loop: check aether_registrar_sync_errors_total.

The agent line settles to 0 between changes (#1241). An agent learns a revision from an event that reaches it: the last event of a batch it is sent, the SNAPSHOT_COMPLETE of a (re)connect, or a version marker. Two kinds of store write move the revision without sending an agent any endpoint event:

• a write that changes no contents: a write-behind flush landing in etcd after the RPC already applied it, a re-Put of an identical endpoint;
• a change to a service outside a demand-scoped agent's watch filter.

At the end of every sync cycle the registrar sends each watch stream that is not already at the snapshot's version a version-only SNAPSHOT_COMPLETE carrying it (Broadcaster.MarkVersion). The cycle is the throttle: a burst of store writes is one debounced sync (200 ms), and a cycle that moved nothing sends nothing. Markers are counted in aether_registrar_broadcast_events_total{aether_event_type="EVENT_TYPE_SNAPSHOT_COMPLETE"}; the sync span carries aether.sync.version_markers. A marker is handed out under the same rule as a batch's version (#1203): the registrar holds every publication off while it sends them, so every batch the version includes is already ahead of the marker on the stream.

So a gap that persists past a sync cycle (the poll interval at worst) is lag: check aether_agent_registry_reconnects_total and watch_errors_total on that agent (and watch_token_drops_total, the reconnects it chose to make full resends, #1269, and watch_resends_abandoned_total, the resends cut before their marker, #1334), and aether_registrar_broadcast_dropped_events_total (a full stream skips its marker and is retried next cycle; a dropped endpoint event force-resyncs it).

Skew: a registrar older than #1241 sends no markers, and against it the line is only an upper bound (on kind, both agents sat one revision behind for 11 s with identical contents). An agent older than #1241 handles a marker the way it handles any SNAPSHOT_COMPLETE after its first: it adopts the version and re-derives its xDS from an unchanged cache (one debounced refresh per sync cycle that moved the version, until the agent DaemonSet has rolled). Alert on this line only once both are on #1241; content-hash agreement (below) is alertable regardless.

Divergence rule. Two replicas reporting the same aether_registrar_snapshot_revision, with no write-behind intent pending (aether_registrar_writebehind_queue_depth == 0 on both), must report the same value of aether_registrar_snapshot_content_hash. The same revision with a different hash is a bug: the listing is a pure function of the revision, so two replicas that disagree are not serving what the store holds.

aether_registrar_snapshot_content_hash (#1329) carries the hash as the gauge's value: the first 13 hex digits of the <hash> in the snapshot version, as an integer. That is 52 bits. A float64 holds every integer up to 2^53 exactly, so 53 would fit; 52 is a whole number of hex digits, which keeps the value readable against a version string:

printf '%013x\n' 20015998343868   # -> 0123456789abc, the start of <hash>

The 12 bits it drops do not matter for this use. The gauge answers one question, "do the replicas at this revision agree", between a handful of replicas; two replicas serving different contents report the same value with probability 2^-52 (about 2.2e-16). It is not an identifier to look contents up by, and the resume token still compares all 16 digits. Each replica has one series, with no label that changes, in every version form (<rev>.<hash>, <rev>+<hash>, hash:<hash>).

How the expression above works:

• count_values without () ("revision", aether_registrar_snapshot_revision) turns each replica's revision into a revision label. without () keeps all of the series' labels, so the expression does not depend on the name of the replica label (node carries the pod name on the reference cluster).
• Multiplying the hash gauge by that (ignoring (revision) group_left (revision)) gives one series per replica: value = hash, labelled with the replica's revision. A replica with no revision has no match and drops out.
• count_values by (job, revision) ("hash", ...) is one series per distinct hash at one revision, and the outer count is how many there are. A replica that lags is at another revision and is not compared.
• unless ... > 0 is quiet only when some replica reports a pending intent. aether_registrar_writebehind_queue_depth is recorded only when an intent is queued or flushed, so on a registrar that never queued one the series is missing; unless on a missing series changes nothing, where and on() (... == 0) would have made the expression unable to fire.

Timestamps do not matter to it. The registrar observes the revision and the hash in one callback from one read of the snapshot, so the pair in an export always belongs to the same contents. The OTel SDK stamps each instrument's data point separately (timestamp() of two registrar gauges differed by at most 4 ms over 24 h on the reference cluster), both travel in one OTLP request, the collector's batch processor does not split a request, and Prometheus's OTLP receiver commits a request as one append. The expression matches the two metrics on labels, so it needs none of that to be exact. A pipeline that did deliver them separately could pair a new revision with the previous hash for one evaluation, on one replica; the alert's for: 3m (kept from #1328 for the RPC-between-syncs window) covers it.

To read the pairs by hand:

# one row per replica: value = the hash, revision label = where it serves it
last_over_time(aether_registrar_snapshot_content_hash[150s])
  * ignoring (revision) group_left (revision)
  count_values without () ("revision", last_over_time(aether_registrar_snapshot_revision[150s]))

The alert is AetherRegistrarSnapshotDiverged (docs/observability/registrar-alerts.yml, for: 3m). Its promtool tests are docs/observability/registrar-alerts_test.yml (bazel test //:observability_rules_test), and more in the GitOps repo (clusters/<cluster>/prometheus/rules_test.yaml).

A replica is its pod, and a pod that is gone must not be compared (#1560). Since chart 2.5.3 the registrar's series carry instance, its pod name, so the replicas are separate series in any backend that maps service.instance.id (Prometheus does by itself). Every registrar roll therefore ends one set of series and starts another, and nothing marks the ended ones stale: a bare selector returns a replaced pod's last sample for five minutes. That is what last_over_time(...[150s]) is for, here and in the rule: a replica is read while its newest sample is at most 150 s old (two and a half 60 s exports). Without it a replaced registrar whose last export fell between an agent RPC and its sync fired the alert against the live replicas, and one whose last write-behind depth was above zero kept a real divergence quiet for five minutes; both are cases of the rule's test file.

Before chart 2.5.9: the hash as a label. Until #1329 the hash was the content_hash label of an info gauge, aether_registrar_snapshot_content (always 1). A content change ended one series and started another, OTLP carries no staleness marker for a series that is no longer exported, and Prometheus kept returning the superseded hash for its 5-minute lookback, so each replica showed two hashes after every change (#1322). Counting hashes over the bare selector read that as a divergence: on an 8 h soak with 66 revisions, none diverged, it was true at 108 of 1,920 15-second steps. #1328 worked around it by keeping, per replica, the series whose timestamp() is the newest. The value gauge has no label to go stale and needs no such filter. It replaced the labelled gauge in chart 2.4.12; the labelled gauge was exported beside it through chart 2.5.8 and is gone since chart 2.5.9. A rule that still reads it returns nothing: "Chart 2.5.9" has what to check.

Notes:

• kubernetes backend: the version is content-addressed, and the lag metrics are etcd-only. A kubernetes listing is not a function of the Pod list's resourceVersion: endpoint health derives from the clock (the drain pool-close deadline), locality comes from a separately cached node list, and the list RV moves on every write anywhere in the cluster. So that backend does not implement registry.RevisionedLister, and store_revision, snapshot_revision and the agent's last_version are not reported. The content-hash gauge is still exported there, but the divergence expression above returns nothing there: it compares replicas at one revision, and there is no revision. Two kubernetes-backend replicas can also differ for a moment legitimately, because health derives from each replica's clock. Read each replica's current hash straight from aether_registrar_snapshot_content_hash (it is reported on this backend too: it names the contents, with or without a revision) and treat only a disagreement that persists across several sync cycles as a finding. It is not alerted on.
• aether_registrar_snapshot_version keeps its pre-#1193 series name but is now the snapshot generation: a per-process count of content changes. It moves only when the served endpoint set changes, and it is not comparable across replicas.
• Reconnect outcomes, counted by aether_registrar_watch_starts_total{resume}:
• current: the agent's last_version is the current version. It gets only SNAPSHOT_COMPLETE.
• renamed: the hash matches but the name differs, for example because the revision moved with no content change. It gets the service catalog plus SNAPSHOT_COMPLETE carrying the current version. No endpoint events are sent.
• extended: a dependency-set growth (#1239), below. It gets only the added services' endpoints, plus the catalog when its token was renamed.
• resent: anything else. It gets the full snapshot.
• Dependency-set changes (#1239). An agent's watch is filtered to its dependency set, and a server-streaming RPC cannot change its filter, so every change of the set (a TCP service appearing or disappearing anywhere changes it on every node; service left dependency set in the agent log) re-opens the stream. Before #1239 that re-open always dropped the resume token and every agent was resent its whole filtered snapshot. Now the agent keeps the token for the services its cache still holds:
• The set shrank (or is unchanged): it sends last_version as on any reconnect, and gets current or renamed. The version names the registrar's whole snapshot, so a token that is current is current for any subset of the services it was earned on.
• The set grew: the added services were never delivered at that version, so last_version would wrongly be answered current. The agent leaves it empty and sends partial_resume {version, services it holds} instead. A registrar that knows the field answers extended when the version names its current contents, and resent otherwise. One that predates it sees no token and resends: a one-release skew in either direction is safe, it only costs the resend.
• A service that leaves and re-enters the set while one stream is open, including while its initial exchange is still arriving, is not held (its endpoints were purged when it left, and the stream keeps delivering only what came after), so it is re-requested.
• A resend of a filter with no endpoints carries no FULL_SNAPSHOT to clear the cache with. The agent clears it at the marker when the marker's content hash differs from the token it presented (renamed always has an equal hash), or when it presented none. resent therefore still counts every agent and registrar restart and every stale token (a change landed while the stream was being re-opened), and on a registrar older than #1239 every growth. A resent step on every node at each dependency-set change, with agents on #1239, is a bug.
• The version is sent only where the receiver holds everything it names: on SNAPSHOT_COMPLETE, and on the last event of each broadcast batch per watcher (#1203). The agent also drops its token when a resend starts clearing its cache, so a stream cut mid-snapshot resends in full. Agents older than #1204 keep the old token across a cut resend, so until the agent DaemonSet has rolled they are exposed to an empty cache on a reconnect that matches it. The window is milliseconds per reconnect.
• A stream cut mid-batch (#1269) leaves the batch's unversioned prefix applied while the token still names the version before the batch. Presenting that token is safe only while the registrar has moved on (its hash differs, so it resends). If its contents return to the token's hash, a change and its exact reversal or a reconnect to a peer replica that never got the change (a lost write-behind write), it answers current, renamed or extended and the prefix stays in the cache for good. So the agent drops its token, and the services it holds, whenever a stream ends after a live event that carried no version (a batch's versioned last event, a version marker or the initial SNAPSHOT_COMPLETE ends that state). The next stream is resent, logged as watch stream ended inside a batch; requesting a full snapshot on reconnect and counted in aether_agent_registry_watch_token_drops_total{reason="midbatch"}. That costs one resend per mid-batch cut, whatever ended the stream: a failure, a server drain, or a dependency-set change whose cancellation landed inside a batch. Read the counter after a soak, against aether_registrar_watch_starts_total{resume="resent"}: the suspected cost is a batch whose own event changes the dependency set (a TCP service's SERVICE_ADDED/SERVICE_REMOVED wakes the xDS cache, which re-asserts the filter and cancels the stream before the rest of the batch arrives), which would turn #1239's current/extended re-opens back into resends exactly at dependency-set changes. A midbatch step on every node at each such change is that case. A registrar older than #1203 versions every event, so its streams never end inside a batch. Agents older than #1269 keep the token and stay exposed to this case until the DaemonSet has rolled.
• A stream that ends before its first SNAPSHOT_COMPLETE with no resume token (#1334) abandons the full snapshot the registrar was sending it, and the next stream is resent in full again: two resent for one cause. It is not a token drop (there was no token to drop, so watch_token_drops stays flat) and it is correct (the abandoned half is never announced as complete; the second resend clears and rebuilds the cache). It is only wasted work, and it was invisible: on the 2026-10-06 soak it showed as 6 resent at an agent roll where 5 were expected (#1324). The agent now logs it at INFO as watch stream ended before its first SNAPSHOT_COMPLETE with no resume token; the snapshot it was being sent is abandoned and the next stream is resent in full and counts it in aether_agent_registry_watch_resends_abandoned_total{reason}, where reason is what ended the stream:
• filter_change: the dependency set changed while the snapshot was arriving and the agent re-opened the stream to assert the new filter. This is the #1324 case. At start-up the first stream already carries the local pods' dependency set (the xDS PreListen asserts it ahead of the identity hold), but the set keeps growing for a few seconds as the agent's informers sync (the mesh-Service projection's TCP services, GAMMA and L4 route backends, chain filters, imported config) and, on a surge standby, at the takeover (the previous agent's persisted observed upstreams, the overlap's CNI ADD/DEL, then the clusters the reconnecting proxy reports it holds). A change that lands after the marker costs an extended or current re-open; one that lands inside the first exchange costs this.
• server_drain, eof, forced_resync, error: the registrar drained, closed, force-resynced or failed the stream mid-snapshot. The same line carries eventsReceived and openFor (how far the stream got), filterServices and nextFilterServices (the abandoned stream's filter size and the one that superseded it) and tokenPresented (true when the stream did present a token, the registrar resent past it, and the cut landed inside that resend: #1203 drops the token at the first FULL_SNAPSHOT). A stream that ends before its marker with its token intact is not counted: nothing was being resent, and the next stream resumes.

Read it against the registrar's resends over the same window:

# Abandoned resends per node, by what ended the stream. max_over_time, not
# increase(): the typical one happens in an agent's first second, so its
# series is born at 1 and increase() reads 0 for it.
sum by (node, reason) (max_over_time(aether_agent_registry_watch_resends_abandoned_total[30m]))

# Full resends the registrar served in the window.
sum(increase(aether_registrar_watch_starts_total{resume="resent"}[30m]))

Each abandoned resend accounts for at most one resent beyond the expected ones (one per agent start, one per agent per registrar it reconnects to with a stale token, one per watch_token_drops); "at most" because a stream cancelled before the registrar began serving it was never counted there. A non-zero filter_change at agent rolls is the known start-up race and costs one filtered snapshot send per count. filter_change on every node at every roll, or outside rolls, means dependency-set changes are landing inside resends as a rule (check eventsReceived/openFor: a registrar that takes seconds to finish an initial exchange widens the window). Any other reason that keeps counting is a registrar that cannot complete an initial exchange: read it with aether_agent_registry_watch_errors_total and the registrar's own log. - Publications (an RPC's or a sync's snapshot change plus its broadcast) are serialized: every watcher receives batches in the order they changed the snapshot, each one contiguous. Before the #1239 review they ran concurrently, and two on one endpoint (a register and an unregister, or two agents registering it during a surge roll) could reach a watcher in reverse, leaving it holding the endpoint under an older version (healed only by a resend on its next reconnect).

504 UT after exactly 15 s over an h2 mesh cluster to a terminating pod (#1104)

The h2 sibling of #1087. Since QUIC went unconditional, h2 carries the GAMMA weighted split (by design, #961), waypointed (cross-cluster) traffic and the edge -> mesh hop; all of them are NewServiceCluster. A request the destination proxy had received (TCP-ACKed) whose pod veth was then deleted got no FIN and no RST (the node-shared destination proxy keeps its sockets in the dead netns), the source had no unacked bytes, and nothing was configured to probe, so it waited for the 15 s route timeout.

Since #1104 every h2 mesh cluster carries http2_protocol_options.connection_keepalive {interval: 1s, interval_jitter: 15%, timeout: 8s}. Envoy PINGs every connection (open streams or not) about once a second; the destination proxy's codec answers, never the application, so a slow application is not touched (kind: a 10 s handler on the live backend answers 200). An unanswered PING closes the connection 8 s later: at most ~9.2 s after the peer's last frame (8.1-8.95 s after the cut on kind). It is 8 s, not less, for the same #1093 reason as the twins: a destination worker stalled for 8 s or more cannot answer the PING. What you see now is 503 UC (upstream connection termination) and cluster.<svc>.http2.keepalive_timeout incrementing. The request is not retried, GET or POST: the close is a ConnectionTermination reset of a request already sent, which reset-before-request excludes. The PING is NOT on the app hop, authz, xDS or collector clusters (their h2 peer is not an aether proxy; a gRPC server answers frequent PINGs with GOAWAY too_many_pings). The kind proof is eastwest-quic-deadpeer.sh verify-h2, in the external harness named under #1087 above.

An HTTP/1.1 client gets a reset on a reused keep-alive connection (#1350)

The proxy's HTTP connection managers have two downstream idle timeouts, chosen by who the downstream is (Go constants in agent/internal/xds/proxy/networkfilter.go; not a chart value):

Listener	Downstream	Idle timeout
mesh inbound (18008, TCP and QUIC; stat prefix inbound)	a peer proxy	5 min (peerFacingIdleTimeout)
per-pod capture chain and per-pod outbound (18001 / 18081; capture_http, outbound_http)	the pod's application	1 h (appFacingIdleTimeout)
edge gateway listeners	an external client	1 h (edgeDefaultIdleTimeout; per-Gateway listeners take EdgeConfig idleTimeout)

The rule on the app-facing side: the server's idle timeout must exceed the client's, because HTTP/1.1 has no GOAWAY. A server that closes an idle kept-alive connection cannot tell the client first, so a request the client writes onto that connection at the same instant is lost. A client that retries on a reused-connection failure (Go's net/http, so k6; browsers) hides it; one that does not (an Envoy-based client, many plain HTTP/1 pools) reports a reset. With 1 h on the proxy, an ordinary client (idle timeout 30-120 s) always closes first. HTTP/2 and HTTP/3 are not affected: the idle timeout drains them with a GOAWAY.

The inbound side is the other way round on purpose. Its client is a peer proxy that idles its pools out at 30 s (config.UpstreamIdleTimeout), so 5 min already exceeds it, and it is short because the 1 h default once let a peer's leaked connections pin ~7k inbound connections on one listener.

Before #1350 both sides had 5 min, and the signature was: one client-side connection_termination / reset before headers in a few hundred thousand HTTP/1.1 requests, no access-log row on either side (the request never reached the HTTP layer), and envoy_http_capture_http_downstream_cx_idle_timeout_total ticking on the source node within seconds of it. If you see that again:

• On a proxy with the fix, …_capture_http_downstream_cx_idle_timeout_total and …_outbound_http_downstream_cx_idle_timeout_total should barely move. If they move, either a client really does hold a connection idle for an hour, or the node proxy is under memory pressure: its overload manager (proxy.overload, on by default) scales this very timer down toward 2 s between 85 % and 95 % of the heap budget and disables keep-alive at 90 %. Check Envoy's overload.envoy.overload_actions.reduce_timeouts.scale_percent and overload.envoy.overload_actions.disable_http_keepalive.active first.
• A load driver or client you control should set its own idle timeout below the proxy's, and retry an idempotent request on a reused-connection failure.

The cost of 1 h: a connection an application leaks (opens and never closes or uses) stays on the proxy for up to an hour instead of 5 minutes. Nothing caps downstream connections per listener; the overload manager above is what reclaims them. Watch envoy_http_capture_http_downstream_cx_active per node if a workload is suspected of leaking.

A client gets a 503 the mesh did not retry: the request had begun (#1641)

A destination proxy answers 503 both when it never reached its application and when it had already sent the request to the application and the application's connection then closed or reset before any response. The second kind must not be replayed for a method that is not idempotent, because the application may have run it. Until #1641 the caller retried "a 503" on another endpoint, whatever the method.

Now the destination says what happened and the caller retries on that alone. Every response of a mesh inbound listener carries the response header x-aether-outcome:

<code>;<flags>;<method>;<details>

code is the application's HTTP status, or 0 for a reply the proxy generated itself; flags and details are Envoy's response flags and response code details; method is the request's. The grammar and the rule live in one place, agent/internal/xds/proxy/outcome.go. The caller's retry policy is connect-failure,refused-stream,reset-before-request,retriable-headers with one matcher on that header. There is no status-code condition any more. A retry goes to another host (previous_hosts) and, first, to another priority (previous_priorities, update_frequency: 1): the endpoints are banded by locality (same zone 0, same region 1, elsewhere 2), so without it a source whose priority 0 is one dying endpoint retried onto that endpoint every time and returned the final 503. With one priority, or a request pinned to one endpoint, it changes nothing.

x-aether-outcome (destination)	Client sees, if nothing is retried	Caller
200;-;POST;via_upstream (any application status but 503)	that status	does not retry
503;-;POST;via_upstream (the application answered 503)	503	retries, any method
0;UF;POST;… connect refused or timed out, 0;UH;… no healthy host, 0;UO;… overflow, 0;NC;… app cluster gone	503	retries, any method
0;-;POST;overload or 503;-;POST;overload (the destination's proxy is overloaded: stop_accepting_requests)	503	retries, any method
0;UC;GET;…{connection_termination}, 0;UR;GET;…{remote_reset}, 0;LR;… with GET, HEAD, OPTIONS, TRACE, PUT, DELETE	503	retries
the same with any other method (POST, PATCH, …)	503	does not retry
0;UR;POST;…{remote_refused_stream_reset} (an HTTP/2 application refused the stream: not processed, by the protocol)	503	retries, any method
anything else (0;UT;… timeout, 0;UPE;…, a flag the rule does not know)	504, 502, …	does not retry
no header	whatever it is	does not retry

A gRPC request follows the same table (#1646): the destination's own failures go out as HTTP 200 with grpc-status: 14, the header is the same (0;UF;POST;…, 0;UC;POST;…), so a call the destination could not deliver is now retried on another endpoint and one the application had received is not.

The caller removes the header before the response reaches the client application, and the edge before it leaves the mesh, so it is in no access-log field on the caller's side. To find a request that was not retried:

• Destination row (reporter=destination): response_code=503, response_flags one of UC / UR / LR, response_code_details starting upstream_reset_before_response_started{, us_tx_beg_ms a number (the request was written to the application; - means it never was), and a method outside the idempotent six. Those four fields are the header.
• Source row with the same x_request_id: response_code=503, response_flags -, response_code_details=via_upstream, and upstream_host the pod of the destination row.

What causes a UC at the destination: the application exited or was killed with the request in flight, or it closed an idle kept-alive connection at the instant the proxy reused it (HTTP/1.1 has no GOAWAY; the rule in the #1350 section above, seen from the other side: here the proxy is the client and the application is the server). The node proxy keeps an idle HTTP/1.1 connection to the application for up to 1 h (Envoy's default; the app cluster sets none) and an idle HTTP/2 one for 30 s, so an application with a shorter keep-alive timeout always closes first. For idempotent methods the caller's retry hides that, as it always did. A POST that loses the race is now a 503 at the client where it used to be replayed, although in that case the application had not read it; the proxy cannot tell the two apart, which is the reason it does not replay. Measured with a harness aimed at the race (requests paced at the application's keep-alive timeout): 2 client-visible 503s in 3,600 POSTs. If a workload sees these outside its own restarts, raise the application's keep-alive (idle) timeout above an hour, or make the endpoint safe to retry and retry it in the client.

Other things to know:

• A cleartext Kubernetes Service behind the edge has no destination proxy and so no header. A route whose backends are all of that kind keeps the status-code policy (the edge is the last proxy before the application there, so a 503 response is the application's own). A rule that splits between a mesh service and a cleartext Service takes the mesh policy: the cleartext share of it loses the retry of an application's 503.
• An application cannot write its own outcome: the destination overwrites the header. A request header of that name is removed where a request enters the mesh's hands: on every edge route (so an external client's goes no further), on the caller's routes to mesh services, and at the destination before the application. The one place a response header of that name is not overwritten is the cleartext share of an edge rule that also has a mesh backend (Envoy has one retry policy per route, not per weighted backend, so that share carries the mesh policy): a cleartext backend that sets x-aether-outcome on its own response decides its own retry there, as it always could by answering 503.
• In a mesh run without SPIRE (cleartext, a testing posture) the pod's single inbound chain also receives connections from a source's TCP floor (tcp_proxy, no HTTP awareness on the caller's side). HTTP carried that way gets x-aether-outcome on every response and nothing removes it, so the client application sees it. With mTLS the TCP floor lands on the inbound tcp_proxy chain and no header exists.
• x-envoy-ratelimited is Envoy's own: a response that carries it is never retried by Envoy, whatever the policy. The mesh neither sets nor removes it, so an application that answers 503 with that header is not retried and its client sees both.

Replies the destination's connection manager writes before any route (#1641)

A virtual host's headers apply only when a route was chosen. For the replies below the connection manager's local-reply mapper stamps x-aether-outcome itself when it is absent, with the reply's HTTP status as the code. Only the first is ever retried.

Reply	Status	x-aether-outcome	Retried before #1641	Retried now	Right?
Overload, stop_accepting_requests ("envoy overloaded")	503	0;-;<method>;overload over the mesh hop, 503;-;<method>;overload for a plain client (both measured)	yes (a 503)	yes, any method (measured, GET and POST)	yes: no application saw it, and the chart's overload ladder relies on it
The same for a gRPC request	200 + grpc-status: 14	0;-;POST;overload when the route stamped it; 200;-;POST;overload otherwise	no	yes in the first shape, no in the second (not measured)	the first is; the second is the pre-#1646 behaviour
Malformed request (codec error)	400	400;-;;http1.codec_error (measured)	no	no	yes: the request is bad everywhere
Request headers too large	431	431;…	no	no	yes
Request-header / request / max-stream-duration timeouts	408 / 504	not configured on the inbound (Envoy defaults: none)	no	no	n/a
Stream idle timeout (5 min default)	408 / 504, flag SI	no rule matches SI	no	no	yes: the request may be in the application
Path normalisation rejects	400	not configured on the inbound	no	no	n/a
Listener drain (hot restart, listener update)	none: GOAWAY / connection: close	responses in flight keep their normal value	n/a	n/a	n/a

The overload manager's other rungs write no reply: shrink_heap frees memory; disable_http_keepalive closes connections after their response; reduce_timeouts shortens idle timers; stop_accepting_connections refuses the caller's connect, which the caller retries as connect-failure, before and now. (reset_high_memory_stream, not in the chart's ladder, resets a stream in flight: the caller sees a reset after it sent the request, which it did not retry before and does not now.)

Upgrading across #1641: the agent roll is a window with fewer retries

The header is a contract between a caller's proxy and a destination's proxy, and each node's proxy gets its half when its own agent is upgraded. There is no dual mode. While the agent DaemonSet rolls, and again during a rollback:

Caller's node	Destination's node	What happens
upgraded	not yet	The destination sends no header, so nothing it answers is retried: not a 503 for a connect failure behind it, not an application's 503. Measured: a POST and a GET the application closed on, and an application 503, all reach the client as 503. The caller's own conditions still work (it could not connect to the destination pod, the stream was refused, the reset came before the request was sent).
not yet	upgraded	The old behaviour, replay included: the old caller retries the 503 by its status. It also forwards x-aether-outcome to its client application on every response.
upgraded	upgraded	As the table above.

What an operator sees in the first row: 503s at clients for requests whose destination pod is being rolled, restarted or is failing readiness at that moment, on source rows with response_code_details=via_upstream; on the destination rows response_flags UF or UH. Outlier detection still ejects a failing endpoint after 5 consecutive 5xx, and EDS still withdraws a draining pod, so this is about individual requests in the sub-second windows the retry used to cover.

The window is the agent rollout: from the first agent pod replaced to the last one Ready (for a roll that replaces one node's agent at a time, about the number of nodes times the time one agent takes to become Ready). To keep it harmless, do not roll workloads while the agents roll: with every application pod up and Ready there is nothing for the missing retry to cover. Clusters that exchange traffic (east-west waypoints, imported services) are in the same position: upgrade them together, back to back, and expect the first row for cross-cluster requests between an upgraded and a not-yet-upgraded cluster until both are done.

Envoy SIGBUS/SEGV in QuicConnection after an h3 cluster removal (#1074)

Symptom: the proxy container restarts with a SIGBUS or SIGSEGV whose backtrace ends in quic::QuicConnection::OnCanWrite / CanWrite (or a QUIC alarm), shortly after a CDS push removed a quic: twin while requests were still in flight on it. Seen on node A of the reference cluster when the pre-#1073 fetch-window prune removed held twins.

Cause: the per-cluster PersistentQuicInfoImpl (connection helper and clock, alarm factory, QUIC config) was owned by the worker's thread-local ClusterEntry, while the HTTP/3 pools held it by reference and their connections kept raw pointers into it. A CDS removal destroys the entry after only draining its pools, so a pool with live streams outlived the info and its next write or alarm read freed memory.

Fix: the carried patch envoy-aether1074-quic-persistent-info-lifetime.patch (#1077; upstream as envoyproxy/envoy#47893): the info is shared, and every HTTP/3 pool and connectivity grid holds a reference. It is in the proxy pinned by chart 1.0.13 (71be75f…, #1078). A proxy older than that is exposed on any h3 cluster removal with streams in flight, which includes rolling the chart back to 1.0.11 without pinning proxy.image (see "If UDP:18008 is blocked" above).

The node agent is OOMKilled (exit 137)

Symptom: agent pods restart with exit code 137 and the reason is OOMKilled, typically at churn peaks (many pod ADD/DEL, snapshot rebuilds):

kubectl get pod -n aether-system -l app.kubernetes.io/component=agent \
  -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[0].restartCount,LAST:.status.containerStatuses[0].lastState.terminated.reason,EXIT:.status.containerStatuses[0].lastState.terminated.exitCode'

The check: compare the Go runtime's memory with the container limit and with GOMEMLIMIT.

# the limit and the GOMEMLIMIT the chart rendered (bytes)
kubectl get ds -n aether-system aether-agent -o jsonpath='{.spec.template.spec.containers[?(@.name=="agent")].resources.limits.memory}{"\n"}{.spec.template.spec.containers[?(@.name=="agent")].env[?(@.name=="GOMEMLIMIT")].value}{"\n"}'

# peak Go runtime memory per agent over the incident window, MiB
max by (node) (max_over_time(go_memory_used_bytes{job="aether-agent"}[1h])) / 1048576

Read it as follows:

• GOMEMLIMIT must be below limits.memory. Since chart 2.4.2 the chart renders it at 90 % of the limit (aether.goMemLimit in _helpers.tpl). Before that it was a resourceFieldRef on limits.memory, which is the whole limit: the heap could fill the cgroup before the GC tightened. If it is a valueFrom again, or absent while a limit is set, that is the bug.
• The cgroup holds more than the Go runtime. The exec probes (proposal 041: /agent-ready, readiness every 2 s and liveness every 10 s) are processes in the agent container's cgroup, ~3 MiB RSS each (measured), plus runc's exec helper, plus kernel socket buffers and page cache. Count on ~10 MiB beyond go_memory_used_bytes.
• Peaks near GOMEMLIMIT with steady growth between churn waves point at a leak: capture a heap profile (Pyroscope) before raising anything. Flat peaks (the 2026-10-04 soak: 44-61 MiB per agent, the same as the passing 10-03 soak) mean the limit is simply too small for the load.

The knobs: agent.resources.limits.memory (default 128Mi since 2.4.2; was 64Mi, which the 2026-10-04 soak OOMKilled 3 times on 2 of 5 nodes within ~1 h of churn) and agent.resources.requests.memory (96Mi). GOMEMLIMIT follows the limit at 90 % with no separate setting. Raising the request also raises what a surge roll needs per node (see Agent surge roll (proposal 041)).

The agent reports an unrepairable conflist

Symptom: AetherCNIConflistUnchained fires for a node, the agent there is NotReady and the startup taint is held (proposal 033, #667), and the agent logs

aether is not chained in the active CNI conflist and no known-good entry could be
recovered; cni-install must run

Aether is a chained plugin in another CNI's conflist, so this node issues no CNI ADD to the agent at all: every pod that starts on it comes up unmeshed. The re-assert loop re-appends the entry it last observed, and this message means it has none — the strip landed before its first check (the priming window, #680).

Check the durable entry cni-install leaves beside the conflist; the loop primes from it and repairs within a check (~2.5s) when it is present and valid:

# On the node (talosctl, or a debug pod with /etc/cni/net.d mounted):
ls -la /etc/cni/net.d/                     # .aether-cni-entry must be there
cat /etc/cni/net.d/.aether-cni-entry       # one JSON object, "type": "aether-cni"
grep -c aether-cni /etc/cni/net.d/*.conflist

• Present and valid, agent still refusing — look for primed known-good entry from durable file in the agent log. Its absence with the file present means the agent is reading a different directory: compare its --mounted-cni-net-dir (log line CNI conflist re-assert loop started dir=…) with where cni-install wrote.
• Missing — the node predates #680, or cni-install failed to write it (search the init container's log for failed to write the durable aether CNI entry). Recreate the agent pod on that node (kubectl -n aether-system delete pod aether-agent-…): only the init container renders the entry, so restarting the container is not enough.
• Present but garbage — the agent logs the durable aether entry is unusable and ignores it, by design; recreate the agent pod to have cni-install rewrite it.

Never hand-write either file: cni-install is the durable entry's only writer, and the conflist belongs to the primary CNI plus the re-assert loop.

The agent is stuck waiting for SPIRE

Symptom: the agent logs, once per retry,

waiting for the SPIRE Workload API to issue this workload's SVID socket=/run/secrets/workload-spiffe-uds/socket attempt=7 elapsed=41.2s warnAfter=2m0s error=...

That line is the fix working, not the failure (issue #740). Startup no longer dies when the Workload API is not serving yet: the agent programs everything that does not need identity, keeps /healthz and /readyz answering, and folds the SVID in when it arrives. The signature of a healthy wait is the waiting lines plus 0 restarts and no failed to create SPIRE Workload API source anywhere. Before #740 that error was followed by exit 1, which is how a cold boot produced 2-4 restarts per node.

What to check, in order:

# 1. Is it still waiting, or did it resolve? (arrival is one INFO line)
kubectl -n aether-system logs ds/aether-agent | grep -E "waiting for the SPIRE Workload API|obtained this workload's SVID|resolved workload trust domain"

# 2. Restarts must be zero — a restarting agent is a DIFFERENT problem
kubectl -n aether-system get pods -l app.kubernetes.io/name=aether-agent

# 3. The readiness gate and its dwell
# (chart >= 2.3.0: health is a pod-local socket; the probe prints the verbose
# body, naming the failing check, and exits 1 when not ready)
kubectl -n aether-system exec ds/aether-agent -c agent -- /agent-ready --path=/readyz

The readiness semantics are NOT the same on every component

This is the first thing to get straight, because the same spire-svid check deliberately answers differently on the agent and on the three Deployments:

Component	Workload	Dwell before NotReady	What NotReady does
aether-agent	DaemonSet	2 m (the Go constant spire.NotReadyDwell, not a chart value)	The controller's node-taint guard re-arms aether.io/agent-not-ready:NoSchedule on this node
aether-registrar	Deployment	0 (the Go constant spire.ServiceNotReadyDwell)	This replica leaves the registrar Service's endpoints
aether-controller	Deployment	0	This replica leaves the webhook Service's endpoints
aether-edge	Deployment	0	This replica leaves the LoadBalancer's endpoints

The agent's dwell is not a general tolerance for slow SPIRE. It exists solely because NotReady on the agent DaemonSet is the signal the node-taint guard consumes, and a taint is a fleet-level action a 40s SPIRE hiccup must not trigger. Nothing behind a Service has that coupling: there, NotReady removes one replica from one endpoint set, which is precisely the right handling of a pod that cannot complete an mTLS handshake, so those three go NotReady the instant they are known to lack an identity.

That asymmetry is a fix, not an inconsistency. On the rev210 upgrade roll (2026-09-07 20:03:45Z) the registrar carried the agent's 2 m dwell, so a registrar Pod was Ready — and in its Service's endpoints — before it had an SVID, and an agent on node-a that dialled it logged transport: authentication handshake failed: x509svid: could not get X509 bundle.

The --spire-wait-warn-after flag is a logging threshold only; since #740 PR 4 it no longer drives readiness on any component. On the agent the two values coincide at 2 m.

readyz?verbose reads spire-svid ok for the first 2 minutes of an agent's wait (the dwell above) and then fails. It prints spire-svid failed: reason withheld — controller-runtime redacts checker errors, so the endpoint never shows the reason. The reason is in the log, once per transition:

kubectl -n aether-system logs ds/aether-agent | grep -E "readiness (failing|passing)"
# spire-svid readiness failing   reason="no SPIRE SVID after 2m10s (socket /run/secrets/…)"
# spire-svid readiness passing

The dwell is deliberate: the controller's node-taint guard re-arms aether.io/agent-not-ready:NoSchedule after 30s of NotReady, so a gate that failed immediately would turn a brief SPIRE hiccup into a fleet-wide taint. Past the dwell the NotReady is the point: stop scheduling pods onto a node that cannot give them an identity. Since #740 the agent's own taint remover holds that taint while any readiness gate is failing, instead of clearing it 50ms after the guard arms it — so spire-server, spire-agent and the SPIFFE CSI driver must tolerate the taint (spire.waitWarnAfter's note in values.yaml; applied on the reference cluster in the platform's GitOps repository). Without those tolerations the outage fences out its own cure.

The node keeps serving. While the agent has no SVID it does NOT open the xDS socket — it logs holding xDS until this agent has an SVID; Envoy keeps its current configuration every 15s with a rising elapsed, and Envoy goes on serving the last config it was given. This matters because the alternative is worse than the crash loop

740 replaced: a live agent that cannot reach the registrar publishes a local-only

snapshot, which REPLACES a complete one and strips every cross-node endpoint. On 2026-09-07 that failed ~95% of one node's mesh probes for a whole 6m41s outage. If you see registry unavailable for initial snapshot; starting with local-only config while SPIRE is down, that is the bug — the hold is missing.

Recovery is announced only when both halves of the identity exist (the SVID and the bundle for its trust domain), and the registrar client is kicked out of its gRPC backoff the moment they do:

kubectl -n aether-system logs ds/aether-agent | grep -E "identity acquired"
# identity acquired; generating the initial snapshot                      held=6m41.2s
# identity acquired; reconnecting the registrar client immediately …

Acquiring the SVID is not the same as being able to use it. Every handshake made before the SVID landed failed, so the registrar client's gRPC connection is still carrying that failure for the second or so it takes to redial — and a snapshot built in that window has no cross-node endpoints, which is the same local-only outage as above by another route. Since #740 PR 5 the initial snapshot therefore waits for a registry that actually answers, not merely for its readiness latch, and says so:

kubectl -n aether-system logs ds/aether-agent | grep -E "registry connected|not yet re-established|local-only"
# registry connected; generating the initial snapshot                     waited=1.106s
# registrar connection not yet re-established after identity; retrying    reconnecting=true …

registry connected is the healthy end of every agent start; waited is normally a few milliseconds and rises to about a second when the start raced identity. The not yet re-established INFO is that race being named correctly — it is our own connection catching up, not the registrar, so do not go reading registrar logs for it. Reserve that for registrar has no identity yet; retrying, which is only ever logged once this agent's identity has been settled for more than a few seconds. On the rev211 deploy roll (2026-09-07 20:47Z) both lines read registrar has no identity yet, and node-b published an endpoint-less snapshot and logged 316 prober http_error in ~30s while both registrar replicas had been Ready for 43 seconds. The wait is bounded by the same 15s budget as before, so a registrar that is genuinely unreachable still ends at registry unavailable for initial snapshot; starting with local-only config rather than stalling the node.

The upstream cause is almost always spire-server or the node's spire-agent, not aether:

kubectl -n spire-server get pods                  # spire-server-0 Running and 1/1?
kubectl -n spire-system get pods -o wide          # this node's spire-agent Running?
kubectl -n spire-server logs spire-server-0 | tail -50

Metrics for the same question, fleet-wide:

# nodes without an SVID right now (0 = waiting)
min by (node) (aether_agent_spire_svid_ready)
# how long the wait took on the last boot
histogram_quantile(0.99, sum by (le) (rate(aether_agent_spire_wait_seconds_bucket[15m])))
# must stay flat at 0: a source that connected but could not serve an SVID
sum(increase(aether_agent_spire_source_restarts_total[1h]))

While the wait is on, the registrar watch stream logs watch stream deferred until this agent has an SVID at INFO and counts no watch_errors — the agent cannot handshake without a certificate, and that is not a registrar fault. Envoy keeps serving on the certificates and the configuration it already holds. A new pod on that node is still registered and still gets its CNI redirect, but nothing is pushed to Envoy until the SVID lands (and it could not have meshed without an identity anyway) — which is why the node reports NotReady and stays tainted for the duration.

A pod has no SDS certificate: the SPIFFE Broker Endpoint

Distinct from the wait above, and the failure mode is per-pod rather than per-node. Since proposal 036 the agent does not mint pod SVIDs from a selector set it builds itself; it presents a Kubernetes object reference (pods/core, namespace + name + UID) over the SPIRE agent's SPIFFE Broker Endpoint (--spire-broker-socket, mounted from /run/spire/agent/sockets/csi.spiffe.io/broker), and SPIRE resolves and attests the pod itself. A reference resolves at request time, which is a class of failure the delegated path did not have.

# references that did not resolve — a handful per pod creation is the expected
# CNI-ADD-beats-the-kubelet-list race; a sustained rate is not
sum by (node) (rate(aether_agent_spire_broker_reference_not_found_total[5m]))
# the provider refused us: ALWAYS a policy/config problem, never transient churn
sum by (node) (rate(aether_agent_spire_broker_permission_denied_total[5m]))

Both counters are seeded at zero, so "no series" means the agent is too old, not that the value is zero (#717).

Symptom	Cause	Fix
has not resolved this pod yet at INFO, converging in seconds	The CNI ADD beat the pod into the kubelet's pod list. Expected.	Nothing. The subscription retries on the jittered backoff.
The same line escalating to WARN (>30s on one reference)	The pod is gone, the UID does not match what SPIRE resolved, or podReferenceScope is wrong for this topology.	Check the pod still exists and its UID matches; check spire-agent's experimental.broker block lists this agent.
denied this agent the pod's identity + permission_denied climbing	SPIRE's access_policy is enforced without the impersonate-via-spire RBAC grant, or the agent's SPIFFE ID is not in the k8s attestor's broker list (broker "…" is not configured).	Set workloadAttestors.k8s.brokerAPI.accessPolicy=permissive, or grant the RBAC. accessPolicy: auto is not safe here — it resolves to enforced for our reference type.
rejected the request as malformed; not retrying	A bug in the agent (missing security header, bad reference). Never self-heals.	File it; the subscription is dead until the agent restarts.
Every subscribe fails Unavailable on a healthy SPIRE agent	The agent's SPIFFE ID is not in spire-agent.brokerAPI.brokers.* — an unauthorised broker is rejected at the TLS layer, which gRPC reports as Unavailable, not PermissionDenied.	Fix idTemplate to the agent's real ServiceAccount ID.

Is rotation happening? The counters above only count failures; the healthy path has its own pair, also seeded at zero per attribute set:

# pod SVIDs rotated per node — expect about one per managed pod per SVID
# half-life (2h at the default 4h TTL). `initial` is every pod after an AETHER
# agent restart (a new process has served nothing yet). `unchanged` is the same
# certificate redelivered: a stream that dropped and re-subscribed while the
# SPIRE agent stayed up.
sum by (node) (increase(aether_agent_spire_svid_updates_total{aether_spire_identity="pod", aether_spire_update="rotated"}[3h]))
# the agent's own SVID (identity="node"), and the trust-bundle inputs:
# bundle="own", update="rotated" is a trust-ROOT change — SPIRE's 24h signing-CA
# rotation does not move it, because the bundle is the upstream root.
sum by (node, aether_spire_bundle, aether_spire_update) (increase(aether_agent_spire_bundle_updates_total[24h]))

rotated is "the certificate changed", not "the TTL ran down". A restarted SPIRE agent re-attests and mints fresh SVIDs for every pod on its node, so a spire-agent restart moves rotated by that node's pod count (+1 for the node SVID) and leaves unchanged at zero — measured on rev219: one spire-agent pod delete, pod/rotated 0 → 7, node/rotated 0 → 1. That is correct, and it means a rotation-cadence reading has to exclude the minutes in which that node's spire-agent restarted (kube_pod_start_time{namespace="spire-system"}), exactly as proxy-roll minutes had to be excluded from the Envoy gauges this replaced.

A pod whose rotated count stays at zero past its SVID half-life while its neighbours rotate is holding a certificate that will expire; the agent also logs pod SVID rotated / node SVID rotated at INFO. Before these existed a rotation could only be inferred from Envoy's envoy_sds_*_version gauges, with every proxy-roll minute excluded by hand (a new Envoy changes every gauge once).

Trust bundles do not come from the broker (its bundle RPC also needs a workload reference, which a node with no managed pods does not have). They are built from the agent's own Workload API bundle plus the union of the federated_bundles each pod stream carries, so a node with zero pods still serves validation contexts — if it does not, the agent has no identity yet and the section above is the one you want.

The controller, registrar or edge is stuck waiting for SPIRE

All four binaries wait the same way, log the same two lines, and register the same spire-svid readiness gate, so the commands above work verbatim against deploy/aether-controller, deploy/aether-registrar and deploy/aether-edge.

Two things differ, and both matter. The dwell is 0 on all three Deployments (see the table above): they go NotReady the moment they are known to have no identity, and leave their Service's endpoints, rather than sitting Ready for 2 minutes absorbing dials they cannot handshake. And the metric namespace differs: aether_controller_spire_*, aether_registrar_spire_*, aether_edge_spire_* (not aether_agent_spire_* -- a false zero if you query the wrong one).

What each component does while it waits:

• Controller -- the admission webhooks cannot complete a TLS handshake, so the apiserver fails open: both webhook configurations are failurePolicy: Ignore, so MeshConfig/HTTPFilter/EdgeConfig/EndpointPolicy/HTTPRoute creations are admitted unvalidated and pods are not mutated (no mesh-domain ndots injection) for the duration. The caBundle injector logs webhook caBundle injection deferred until this workload has an SVID at INFO and injects on the first SVID. Symptom of the wait having ended: one injected SPIRE trust bundle into webhook caBundle. The replica is NotReady for the whole wait and leaves the webhook Service's endpoints, which is what makes the fail-open immediate instead of a 10s webhook timeout per request.
• Registrar -- agents cannot handshake, so their watch streams retry (they log watch stream deferred until this agent has an SVID, not errors). While the SVID is pending the registrar authorizes nothing; the trust domain it authorizes against is read from its own SVID at handshake time and announced once, as resolved workload trust domain from SPIRE. The replica is NotReady for the whole wait and leaves the registrar Service's endpoints, so agents dial one that can serve instead of one that will reject the handshake. An agent that dials it anyway (a single-replica registrar, or the endpoint update racing the dial) logs registrar has no identity yet; retrying at INFO and counts no watch_errors; the cluster-refresh path logs cluster refresh deferred: the registrar has no identity yet, keeping the current snapshot at WARN and counts no refresh_errors. Both are transients of the server's startup, and both used to be ERRORs (#740 PR 4).
• Edge -- ingress keeps serving on the certificates its Envoy already holds. The control plane starts with seeds (mesh domain, empty SPIFFE ID), so newly loaded clusters carry no upstream mTLS until the SVID lands; the arrival logs resolved edge identity from SPIRE and pushes a new snapshot, so no restart is needed. The replica is NotReady for the whole wait and leaves the LoadBalancer's endpoints; a second replica goes on serving ingress.

for d in aether-controller aether-registrar aether-edge; do
  echo "== $d"
  kubectl -n aether-system logs deploy/$d | grep -E "waiting for the SPIRE Workload API|obtained this workload's SVID"
done

Before #740 each of these exited with failed to create SPIRE Workload API source (the controller: failed to open SPIRE Workload API source) and crash-looped. Seeing that error at all now means an OLD image.

Expected side effects while an agent holds xDS (#743). The hold keeps the xDS socket closed so Envoy keeps its last configuration; from Prometheus that is envoy_control_plane_connected_state == 0 with envoy_server_live == 1 on that node, and EnvoyControlPlaneDisconnected fires for the duration. It resolves by itself when the identity lands (identity acquired; generating the initial snapshot held=…). A firing EnvoyControlPlaneDisconnected with aether_agent_spire_svid_ready == 0 on the same node is this case, not a broken agent. Attestation-to-Ready is bounded by the wait loop's 30 s backoff cap; the node taint is armed ~30 s after NotReady and stays until readiness returns.

Reproducing the SPIRE-outage path on purpose. Scaling spire-server to 0 alone does not reproduce it (the node's spire-agent serves from cache; measured: 0 prober delta). Delete that node's spire-agent pod and then its aether-agent pod. For a fleet test, do not use kubectl rollout restart ds/aether-agent: the DaemonSet is maxUnavailable: 1, so the rollout stalls as soon as the first replaced agent goes NotReady — delete the agent pods instead. Keep each window under 10 minutes (SVID TTL 4 h) and recover with kubectl -n spire-server scale statefulset spire-server --replicas=1.

The pod-mutating webhook's caBundle is empty (SPIRE-served webhook, #1411)

With controller.webhook.spire=true both webhook configurations render with an empty caBundle and the controller's leader fills them in. For the pod-mutating one it needs three things that the chart now derives from one condition (controller.namespaceInjection or controller.injectPodNdots): the configuration itself, --mutating-webhook-config-name on the controller, and the mutatingwebhookconfigurations rule in the controller's ClusterRole.

Charts up to 2.4.17 rendered the rule only with injectPodNdots=true. A release with controller.webhook.spire=true, namespaceInjection=true and injectPodNdots=false therefore told the controller to patch an object it could not read. The default self-signed webhook, and any release with injectPodNdots on (the default), was never affected. To check a release:

NS=aether-system
SA=$(kubectl -n "$NS" get deploy -l app.kubernetes.io/name=aether-controller \
  -o jsonpath='{.items[0].spec.template.spec.serviceAccountName}')
# Expect "yes" for get and update when the webhook is SPIRE-served and the
# MutatingWebhookConfiguration exists. list, watch and patch: "yes" up to chart
# 2.4.26, "no" from 2.4.27 (#1456), unless another binding grants them.
for verb in get update list watch patch; do
  printf '%s: ' "$verb"
  kubectl auth can-i "$verb" mutatingwebhookconfigurations.admissionregistration.k8s.io \
    --as="system:serviceaccount:$NS:$SA"
done
# Every entry of every aether webhook configuration must show a non-zero length.
kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration \
  -l app.kubernetes.io/name=aether-controller \
  -o jsonpath='{range .items[*]}{range .webhooks[*]}{.name}{" caBundle bytes: "}{.clientConfig.caBundle}{"\n"}{end}{end}' |
  awk '{ printf "%s %s %s %d\n", $1, $2, $3, length($4) }'

An entry at 0 is a webhook the apiserver cannot call; with failurePolicy: Ignore it is skipped silently, so pods in an aether.io/managed namespace come up outside the mesh and without the identity gate.

What the controller's leader does when it cannot read or write a webhook configuration (since #1431; reproduced in the controller's tests against a real informer cache, not on a cluster):

• It logs webhook caBundle injection failed at ERROR, once per attempt, with the object in kind and webhookConfig, the step in reason (get, update, or trust_bundle when SPIRE gave an SVID but no bundle), the apiserver's error, and retryIn.
• It counts the attempt in aether.controller.webhook.cabundle_injection_failures (Prometheus: aether_controller_webhook_cabundle_injection_failures_total), by kind (validating, mutating) and reason. Expected to stay 0.
• It retries with a backoff that doubles from 1 s to 1 min, so a permission that appears is picked up within a minute, with no restart and no trust-bundle rotation needed.
• The two configurations are independent: the one that works keeps following SPIRE trust-bundle rotations while the other fails.
• The leader stays Ready. Leaving the webhook Service would not help the webhook whose caBundle is missing (the apiserver cannot verify any replica for it) and would take the working webhook's endpoint away. The log line and the counter are the signal, not readiness.

The injector reads the two objects from the apiserver directly, by name, and needs only get and update on them; the list and watch the chart also grants are no longer used by it.

A controller built before #1431 behaved differently in the same state: it read through the manager's cache, whose informer could never list the configuration, so the read waited instead of failing. That leader logs client-go's Failed to watch ... mutatingwebhookconfigurations is forbidden lines and no webhook caBundle injection failed line, stops refreshing the validating webhook's caBundle on a trust-bundle rotation too, and is NotReady on its cache-sync check. Adding the rule lets its informer sync on the next retry; if the bundle does not appear within a few minutes, restart the controller (a decision for the cluster's owner; the webhooks fail open while no replica answers):

kubectl -n "$NS" rollout restart deploy -l app.kubernetes.io/name=aether-controller

Grepping the outbound identity bindings during a soak (issue #638)

ssl_fail_verify_san bursts a few tens of seconds into a fresh proxy generation point at the node's netns → SPIFFE ID index (localWorkloads), which is the whole of the (source pod → outbound cluster → SDS client-cert secret) binding: every mTLS-injected outbound cluster carries one transport-socket match per local identity, named by the SPIFFE ID whose SDS secret it fetches, plus one matcher mapping each source pod's netns to one of those names. The agent logs that binding at INFO, outbound identity binding, only when a (source pod, cluster) pair re-binds or is seen for the first time — so steady state is silent and a startup re-bind is exactly the handful of lines you want. A binding whose bound secret is not the owning pod's own SPIFFE ID additionally logs WARN outbound cluster bound to a foreign identity and increments aether_agent_identity_outbound_binding_mismatch_total; an identity mapping no pod owns any more (a missed CNI DEL) logs WARN outbound identity mapping has no owning pod. Above 200 changed pairs in one snapshot a single outbound identity bindings changed summary replaces the per-pair lines, keeping the distinct source→identity transitions.

What snapshot_version on a binding line means (#1621). The outbound and the inbound lines are made from what the snapshot of that version was built from: its own listeners, its own secrets, and the clusters its build read. A pod is named only when the listener its entry held is one of that snapshot's listeners. So a line never names a pod, a cluster or a served secret that the snapshot of its version does not carry. (Agents before this read the agent's maps after the snapshot was set, so a change that landed during a build was named under the version before it, and the build that published it said nothing.) Two things are not exact. A pod whose listeners were written in the few statements between the build's two reads of the listener map is in the snapshot and is first named by the next build; one that was rebuilt in that gap is named again by the next build although nothing about it changed. And the netns → identity index is no part of a snapshot: it is read during the build. A pod whose network namespace is gone is not named while the agent leaves its listeners out of the snapshot.

# Every re-bind on one node, around the roll.
kubectl -n aether-system logs ds/aether-agent --since=10m \
  | grep -E 'outbound identity (binding|bindings changed|mapping)|foreign identity'

# The alarm, in VictoriaLogs (field syntax; never `| stats`, it false-zeroes).
_stream:{service.name="aether-agent"} AND "outbound cluster bound to a foreign identity"

Note: the stream labels on these logs are the dotted OTel resource fields (`service.name`,
`k8s.pod.name`, `k8s.namespace.name`). A selector on `k8s_container_name` matches nothing and
returns a FALSE ZERO — control-test any negative by dropping the selector.

# Same, as a counter: zero at steady state, any increase is the #638 defect.
# increase(), never a raw read — the raw value is per-process (an agent restart
# resets it) and an instant query lands between samples and false-zeroes.
sum by (node) (increase(aether_agent_identity_outbound_binding_mismatch_total[1h]))

Join the snapshot_version on the WARN with the first envoy_cluster_ssl_fail_verify_san_total sample of the new proxy generation: a WARN at (or just before) that timestamp proves the source-side binding was wrong; the absence of one over a window that did produce SAN failures rules the agent-side index out and moves the search to the destination proxy's inbound chain selection.

Attributing an ssl_fail_verify_san event to the TERMINATING node (issue #638, inbound side)

The inversion. Envoy's default_validator.cc:332 message verify cert failed: SAN matcher, certificate SANs are [...] prints the SANs of the certificate being validated — the one the peer presented. So envoy_cluster_ssl_fail_verify_san_total is a client-side counter about the server's certificate, and every #638 ledger join that read it as "the restarting proxy presented X as its client identity" had it backwards. The wrong identity belongs to a server certificate: the inbound filter chain → SDS server-secret binding of whichever proxy terminated the connection. For same-node traffic that is the restarting proxy itself, which is the observed "node-wide constant per time slice" shape. The outbound discriminator above watches the client side and structurally cannot fire for this.

The inbound discriminator. On every snapshot the agent reads each local pod's inbound listener back out of the snapshot it just handed Envoy, extracts each filter chain's DownstreamTlsContext.tls_certificate_sds_secret_configs[0].name, and compares it with the SPIFFE ID of the pod that listener entry belongs to:

• INFO inbound identity binding — chain (<listener>/<chain>), pod, pod_spiffe_id, secret (the server certificate the chain will present), previous_secret (empty on a first bind), secret_served, snapshot_version. Emitted only on a first bind or a change; steady state is silent.
• WARN inbound chain bound to a foreign identity + counter aether_agent_identity_inbound_binding_mismatch_total — the chain would terminate mesh mTLS for its pod while presenting another workload's SVID.
• WARN inbound chain references a secret absent from the snapshot — the chain reached Envoy before its own SVID did (the other candidate mechanism). Reported, not counted.
• Above 200 changed chains in one snapshot a single inbound identity bindings changed summary replaces the per-chain lines.

The binding holds by construction inside proxy.NewInboundListener (the chain's secret name and the chain's own name both come from the same CNIPod). That is the point: if the counter stays 0 through a #638 event while the INFO lines show the chains re-binding correctly, the mis-binding is not in the agent's snapshot — it is in Envoy's SDS/secret lifecycle across the hot restart, and the agent-side line of enquiry closes. A non-zero counter names the pod, both identities and the snapshot version.

kubectl -n aether-system logs ds/aether-agent --since=10m \
  | grep -E 'inbound identity (binding|bindings changed)|inbound chain '

# VictoriaLogs (field syntax; never `| stats`, it false-zeroes).
_stream:{service.name="aether-agent"} AND "inbound chain bound to a foreign identity"

# Zero at steady state; any increase is an agent-side inbound mis-binding.
# increase(), never a raw read: the value is per-process and an instant query
# lands between samples and false-zeroes.
sum by (node) (increase(aether_agent_identity_inbound_binding_mismatch_total[1h]))

The unpinned-cluster signal (#832)

The two discriminators above ask "is the identity we present the right one". This one asks "are we checking the identity we are handed at all". A mesh cluster whose upstream validation context carries no match_typed_subject_alt_names authenticates any workload in the trust domain, so a foreign endpoint in its load assignment produces a clean handshake and a delivered request instead of the ssl_fail_verify_san rejection that caught #829. It is the fail-open direction.

There are four signals, and one vocabulary (reason) joins them. The alert rules built on them are in docs/observability/agent-pin-alerts.yml:

Signal	Says	Shape
WARN mesh clusters published with no server-identity SAN pin	which clusters, and why each	one line per reason per snapshot, first 20 names per line
counter aether_agent_identity_cluster_unpinned_total{reason}	that a snapshot went out with unpinned clusters	adds the unpinned count on every snapshot; seeded at zero per reason
gauge aether_agent_snapshot_tls_clusters{pin,reason}	how many clusters are pinned and unpinned now, per reason	written on every snapshot, zeros included
gauge aether_agent_xds_acked_tls_clusters{pin,reason}	the same, for the clusters the proxy has accepted, cluster by cluster	written when the proxy answers a cluster response: the ACK of one that added or removed a cluster, and the answer (ACK or NACK) to the first one of a stream, and by a snapshot build that changes how a cluster the proxy has accepted is counted; absent before the first answer; not written while the pin state of a cluster the proxy holds is unknown
gauge aether_agent_xds_acked_tls_clusters_unknown	how many clusters the proxy holds whose pin state the agent cannot determine (#1509)	one series, no label of its own; zero while the gauge above is written, the number of such clusters while it is not written for that reason; absent before the first answer

None of the metrics carries a cluster name: the names are in the log line only.

The line (#1424). Each cluster is named under the reason that emptied its pin, so a snapshot logs at most one line per reason (four), each with at most 20 names followed by .... count is that reason's exact number, unpinned and pinned are the snapshot's totals, trust_domain is the trust domain in force when the snapshot was set. Agents before #1424 logged one line with one reason for every cluster it named, chosen from the trust domain at the time of the report (trust domain not yet known or service endpoints carry no namespace metadata); a cluster listed there may have had the other cause.

reason	What happened	Transient?	What to do
trust_domain_unknown	The pin was rendered while the agent did not know its trust domain, so there was no identity to name. The deliberate lesser evil over spiffe:///ns/…, which no certificate can satisfy and cost rev222 four endpoints (#815/#819). Every mesh cluster on the node has this reason at once.	Yes: it ends with the first snapshot after the agent learns its trust domain.	Nothing if it lasts a snapshot or two after an agent start. If it persists it is the bug: check that the agent has its own SVID (the agent's identity readiness, #740) and whether trust_domain on the line is still empty. A non-empty trust_domain under this reason means the trust domain was learned and the pins have not been re-rendered yet; the next snapshot does it. While it lasts the node publishes these clusters with no TLS at all (a peer's mesh inbound refuses them), not with an unpinned TLS context.
tls_not_published	None of the service's endpoints carries a Kubernetes namespace in the registry, and the node agent has no SVID of its own yet, so it publishes no TLS for the cluster at all: an HTTP cluster goes out bare and the TCP floor cluster is withheld (#1482). No handshake is made without a pin, because none is made. The entry is named because the pin is what will be missing: the snapshot that first publishes TLS for it reports it as no_namespace_metadata. The same reason is given to a TCP floor entry (tcp:<svc> on the line) with no namespace metadata whose floor cluster is not in the snapshot, because the service is not in the capture TCP set or, on the edge, no TCPRoute or TLSRoute references it: no TLS is published for it either.	Yes for the first case: it ends when the node SVID is served. On a five-node test cluster every running agent had its SVID within 0.5 s of asking; an agent still without one after 2 minutes reports NotReady.	Nothing if it lasts a snapshot or two after an agent start. If it persists, the agent has no identity: check its identity readiness and its SPIRE Workload API socket (#740). A mesh run without SPIRE never leaves this state, by design. A tcp: name under this reason on an agent that has its identity is the unpublished-floor case: it lasts until the floor is published, and nothing is wrong with the agent. Either way, the endpoints behind clusters need the fix in the next row before TLS is published for them.
no_namespace_metadata	None of the service's endpoints carries a Kubernetes namespace in the registry, so the expected SPIFFE ID (spiffe://<td>/ns/<ns>/sa/<svc>) cannot be built, and the cluster is published with TLS and without a pin. Since #1482 this reason means exactly that: a snapshot that publishes no TLS reports the same entries as tls_not_published instead, and a snapshot that is able to publish TLS never does. (Agents before #1482 used this reason for both states; there the gauge's pin="pinned" series, zero on a node that publishes no TLS yet, was the only way to tell.)	No. It lasts as long as the registry serves those endpoints.	This is the mTLS validation gap. Find who registered the endpoints (clusters on the line names the service) and why their kubernetes_metadata.namespace is empty; a destination that is not a Kubernetes workload with a SPIFFE identity of that shape cannot be pinned.
pin_not_rendered	The entry reached a snapshot without its pin ever having been rendered.	No code path does this.	An agent defect: file it with the line.

The UDP floor's udp:<svc> clusters are never named and never counted, in any of the four signals. They carry no transport socket at all (UDP rides the mesh in plaintext, proposal 038), so they have no pin to lose: plaintext by design, not unpinned. Agents up to chart 2.4.16 did name them, once per snapshot on every node with a UDP service in scope, so on those versions the counter is never at rest and a line whose clusters is only udp:… is not an authentication event (#1393).

The gauges (#1425). The counter cannot answer "how many clusters are unpinned now": it grows by the unpinned count on every snapshot, so its rate is clusters times snapshots. aether_agent_snapshot_tls_clusters is the count itself. It has five series per agent whatever the size of the mesh: pin="pinned", and pin="unpinned" once per reason. A reason that no longer applies reads 0; a cluster that gains its pin moves from its unpinned series to pinned; a cluster that is removed leaves both. It counts cluster entries: an HTTP service is three (its default cluster and two port aliases), a QUIC twin is not counted apart from the cluster it is derived from, and an entry whose pin is rendered is in neither series while the node cannot publish TLS for it (no node SVID yet): it is not a pinned TLS cluster until one is published.

"Every snapshot" includes one whose set returned an error (#1549). The control plane library installs a snapshot before it answers the watches that are open, and only that second step can fail, so the snapshot is the one the agent serves from then on. The agent reports it like any other (this gauge, the counter and the line above, the identity-binding lines) and logs snapshot installed, but an open watch was not answered from it at WARN with the error. Agents before #1549 skipped the report for such a snapshot, and the gauge kept the previous snapshot's values until the next build.

That line is not expected from a connected proxy (#1619, #1620). The agent hands each open watch its response through a channel the proxy's stream owns, and waits at most 5 s for room in it. With the control plane library the agent pins there is always room, also for a proxy that has stopped reading its stream: the channel of a delta stream holds twenty responses and at most seven can be waiting in it (one per resource type the agent serves, and one more), and a secret stream makes a channel for every request. The unit tests hold that against the library's own stream handlers. The wait no longer ends with the caller's context either: before #1620 a snapshot built for a caller whose context had already ended (a CNI ADD whose RPC was abandoned) could stop at any open watch, by a random choice made once per watch, and leave that one and the ones after it unanswered: the proxy was not sent the change until the next build, and the caller was told the build had failed. If the line does appear, the build that logged it held up every other build on the node for those 5 s, and the proxy is sent the change by a later build; the first thing to check is whether the control plane library was upgraded, since how a stream buffers is the library's.

A pod change whose build logged it is not a failed change (#1620). The agent's CNI server is told the snapshot is installed and treats the change as made: a CNI ADD is answered with success (before, it was answered Internal, and the runtime tore down and retried a sandbox whose listeners were published), and a CNI DEL, the ghost sweep and the surge takeover no longer log the change as failed. Each such change logs pod's snapshot change is installed, but an open watch was not answered from it at WARN with pod, namespace and caller, and adds one to aether.agent.cni.snapshot_watch_unanswered{aether.snapshot.caller} (Prometheus aether_agent_cni_snapshot_watch_unanswered_total{aether_snapshot_caller}; caller is cni_add, cni_del, takeover or ghost_sweep, each exported at zero). Any other error of a build is still a failure: the ADD is answered Internal and retried.

# Pod changes on this node that were installed while a watch went unanswered.
sum by (job, node, aether_snapshot_caller) (increase(aether_agent_cni_snapshot_watch_unanswered_total[1h])) > 0

# N clusters unpinned for reason R on this node, now.
sum by (job, node, reason) (aether_agent_snapshot_tls_clusters{pin="unpinned"}) > 0

# The validation gap alone: TLS published without a pin. This is the alert
# (docs/observability/agent-pin-alerts.yml, AetherMeshClusterUnpinned; the two
# reasons under which no TLS is published are a separate, slower warning).
sum by (job, node, reason) (aether_agent_snapshot_tls_clusters{pin="unpinned", reason=~"no_namespace_metadata|pin_not_rendered"}) > 0

# The positive answer: TLS clusters were published, and all of them pinned.
sum by (node) (aether_agent_snapshot_tls_clusters{pin="pinned"}) > 0
  and sum by (node) (aether_agent_snapshot_tls_clusters{pin="unpinned"}) == 0

# Did it happen at all in a window, however briefly (a gauge is sampled; the counter
# is not). Seeded at zero, so a live zero is a real series (not an absent one).
sum by (node, reason) (increase(aether_agent_identity_cluster_unpinned_total[1h]))

The gauge queries here and below are written with bare selectors, which return a series' last sample for five minutes after it stopped. For a node agent that is one series per node and reason, and the next sample replaces the last. The edge control plane's series carry its pod name (instance, chart 2.5.3), so for five minutes after an edge roll these sums count the replaced pod as well. The alert rules do not: they wrap each selector in last_over_time(...[150s]) (docs/observability/metric-labels.md, "A label that holds the pod name"). Do the same in a query whose answer you act on after a roll.

Published is not held. The gauge above is what the agent published. aether_agent_xds_acked_tls_clusters has the same series and counts the clusters the proxy has accepted. The agent keeps, for every mesh cluster entry, the version the proxy last accepted, and counts that version the way it last counted it when it published it. It is the closest the agent gets to what the proxy holds without asking the proxy's admin interface. It learns it from two things the proxy says:

• An ACK is about the clusters its response carried, and no other (#1508). Each of them is then held at the version it was sent at; each one the response removed is gone. The control plane sends a response only for what changed, and it treats a cluster as delivered once it has sent it, accepted or not, so after a rejected update the next response, and its ACK, are about other clusters.
• The first cluster request of a stream states every cluster the proxy holds, by version (a version is a hash of the cluster's content, so it means the same to any agent process). The agent reads that statement when the proxy answers the first cluster response of the stream: the proxy holds what it stated, with the response over it if it acknowledged it, and nothing of the response if it rejected it. That replaces what the agent knew, cluster by cluster.

What follows from that:

• The two gauges agree at rest, for every cluster the snapshot publishes. An entry with no cluster in the snapshot (a tcp: floor that is not captured) is in the published gauge, and in this one only while the proxy still holds that cluster from an earlier snapshot: a floor that was captured, is no longer, and whose removal the proxy rejected stays counted at the version the proxy accepted. One whose cluster was never published, or whose removal the proxy accepted, is not in this gauge. So with a proxy that has accepted everything it was sent, removals included, acknowledged sits below published by exactly those entries. Acknowledged above published, in any series, is the other case: the proxy holds a cluster the agent no longer publishes in that state (a rejected update or a rejected removal, below).
• They differ for the moment an update is in flight.
• They stay different while the proxy rejects a cluster: the rejected cluster stays at the version the proxy last accepted, through every later ACK of other clusters, until a response that carries it is acknowledged (the NACK counter moves, see below, and the agent logs envoy NACKed delta response). A cluster the proxy never accepted in any version is in no series. Published unpinned == 0 with acknowledged unpinned > 0 reads: the agent has pinned it, the proxy still holds the unpinned cluster. AetherProxyHoldsUnpinnedClusters in docs/observability/agent-pin-alerts.yml is that reading as a rule. (Agents before #1508 took the whole gauge from the snapshot of the last cluster ACK. There the ACK of any other cluster after the rejected one made the gauge read as if the rejected cluster had been accepted, and the rule resolved with the proxy unchanged.)
• A rejected cluster is not sent again on that stream unless it changes (#1510). The state above therefore lasts until the cluster changes and is accepted, or until the proxy's stream is re-established, when the proxy states its old version and is sent the current one again (which it can reject again).
• Removals. A cluster the agent no longer publishes stays counted, at the state it was accepted in, until the proxy acknowledges the response that removes it or opens a stream without it. A proxy that rejected the removal still holds the cluster.
• An agent restart does not lose it (#1483). A proxy that reconnects states the clusters it holds; when they are exactly the clusters of the agent's snapshot the agent has nothing to send, answers with an empty response, and the proxy acknowledges it: the gauge has its sample as soon as the proxy has reconnected and answered, normally within a second or so. When they are not, the proxy is sent the difference, and each cluster moves when the proxy acknowledges it. (Agents before #1483 dropped the empty exchange, so after a restart on a quiet node the gauge had no series until a cluster next changed.) Only the first cluster response of a stream is read with the statement. A later empty one carries no cluster and says nothing about any.
• Absent is "not known", not zero. Nothing is written until a proxy answers this agent process for the first time: the node's proxy has not connected since the agent started, or this agent is a surge-rolled standby that does not serve xDS yet.
• A sample is not a live connection. Once the gauge has samples, a proxy stream that drops, or a proxy that goes down, does not withdraw them: the agent keeps what the proxy last accepted (Envoy keeps its clusters across a reconnect) and goes on exporting it until the proxy answers again. The only thing that withdraws the gauge is an unknown held version, below.
• A slow proxy's answer is still read. The agent keeps what it sent, version by version, until the proxy answers it or the stream ends, however many times the cluster is rebuilt in between. An ACK that arrives late is read against the version that was sent, with the pin class that version was last published with. (What is kept is taken when the response is written. A response written after its cluster was rebuilt three times, or after the record of a cluster that was removed has been dropped, inside the agent process, is the exception: its version is unknown, or its ACK is not counted until the proxy next opens a stream. The record of a removed cluster is kept until the third snapshot build without the cluster, and for one minute from the first of those builds however many run (#1551); a build that finds more than 1,024 records of clusters its snapshot does not have keeps none of them for the minute, and an answer that releases one while there are more than 1,024 of them, that one counted, does not keep it for the minute either. With exactly 1,024 the minute applies.)
• Not written while a held cluster's state is unknown. The agent can count a version only if it has that version's pin class on record: this agent process published it, and recently enough (the last three versions of a cluster, plus any in flight). An agent that restarts while its proxy is rejecting a cluster update is told the version the proxy held before that update (a proxy never states a version it rejected), which the new process never built; it sends the update again and the proxy rejects it again. A count that left that cluster out would say the proxy holds fewer unpinned clusters than it may, so the gauge is not written: after a restart it stays absent (and a gauge that had samples stops having them: it is withdrawn, not left on its last values), and the agent logs, once, WARN proxy holds mesh clusters whose pin state this agent cannot determine; the acknowledged pin gauge is not written with the number of clusters (INFO the pin state of every mesh cluster this agent has on record as held by the proxy can be determined again; the acknowledged pin gauge is written when it ends). In that state AetherProxyHoldsUnpinnedClusters is silent although the proxy may hold unpinned clusters. The state has a sample of its own (#1509): aether_agent_xds_acked_tls_clusters_unknown is the number of clusters on that WARN line, and zero whenever the acknowledged gauge is written. AetherProxyPinStateUnknown fires when it has been above zero for 15 minutes. It says the agent cannot classify what the proxy holds, and that AetherProxyHoldsUnpinnedClusters is blind for the node. It does not say the proxy holds an unpinned cluster, and it does not say the proxy is rejecting updates: the state also arises with a proxy that accepts, when the version it accepted (or states on a new stream) is one whose pin class the agent had already dropped from its record (the bounds in the previous point). What tells the two apart is aether_agent_xds_nacks_total for the Cluster type, which moves each time the proxy rejects a response, and the agent's envoy NACKed delta response line, which carries the proxy's reason. With no NACK, the state ends when the proxy next accepts a version of that cluster the agent has on record. The ACK of some other cluster does not end it; the proxy accepting that cluster does. The same holds for a cluster the proxy states it holds when it opens a stream that the agent no longer publishes and has no record of (the agent restarted, or the stream ended before the agent read the proxy's answer and the cluster was removed since), when the proxy then rejects the response that removes it: it is counted as unknown until the proxy accepts the removal, stops stating it, or the agent publishes that version again. A cluster of a family the pin gauges do not count is never treated that way, whatever its TLS: per-pod clusters (application, health probe, inbound readiness), QUIC twins, UDP floors, east/west waypoint ingress clusters, the edge's cleartext backend clusters, the passthrough and the blackhole.
• Accepted is not applied. Envoy applies the valid clusters of a response and then rejects the response as a whole when one cluster in it is invalid; it goes on stating the versions it had before. So after a rejected update the proxy can run a newer cluster than the gauge (and the proxy's own statement at a reconnect) says. The sharpest case: a proxy that starts with nothing and rejects its first cluster response has accepted nothing, the gauge counts nothing for it, and it runs every valid cluster of that response; each comes back into the gauge when it next changes and is acknowledged. The gauge follows acknowledgements, on purpose: a NACK is the event to chase.
• A cluster of one of those families is in no series of either gauge, whether or not the agent still publishes it. Any other cluster the proxy states it holds that this agent process neither publishes nor has a record of (left over from before an agent restart, with its removal rejected) is the unknown state above, not a cluster left out of the count.
• During a proxy hot restart two generations are connected and there is one record per cluster, not one per generation: the new generation's first answer replaces the set with its own, and after that each cluster shows the last answer from either. A late ACK of an older version from the generation that is leaving moves that one cluster back until the next answer about it.
• The gauge is written on each of those answers. A snapshot build moves it too, in two cases: it publishes again a version the proxy holds that the agent had no class for (the unknown state ends), or it counts the very cluster the proxy holds under another reason (an HTTP cluster published without TLS is tls_not_published, and no_namespace_metadata from the snapshot that sees the node's identity on, #1482). Nothing is sent for that, so the build itself rewrites the gauge and the two keep agreeing.
• The agent logs the acknowledged state only when it changes: WARN proxy acknowledged mesh clusters with no server-identity SAN pin (with unpinned, pinned, one count per reason and snapshot_version, the snapshot of the response or of the build that changed it), and INFO proxy acknowledged mesh clusters, all with a server-identity SAN pin when it returns to none.

# What the proxy has accepted, where it differs from what is published. A node
# with no acknowledged series (the "not known" states above) is not in the
# result: a comparison returns nothing for a side that is absent. An entry with
# no cluster in the snapshot that the proxy does not hold (never published, or
# its removal accepted) shows here as published without acknowledged
# (tls_not_published, a `tcp:` floor that is not captured).
sum by (job, node, reason) (aether_agent_xds_acked_tls_clusters{pin="unpinned"})
  != sum by (job, node, reason) (aether_agent_snapshot_tls_clusters{pin="unpinned"})

# Per series, how many more cluster entries the agent publishes than the proxy
# has accepted in that state: not accepted in any version, accepted in another
# state, or entries with no cluster in the snapshot that the proxy does not
# hold. Clamped, because the same
# rejected update shows as a surplus on the acknowledged side of another series
# (the next query). Lasting above the unpublished entries, it is a rejected
# cluster.
clamp_min(
  sum by (job, node, pin, reason) (aether_agent_snapshot_tls_clusters)
  - sum by (job, node, pin, reason) (aether_agent_xds_acked_tls_clusters), 0) > 0

# The other side: clusters the proxy holds in a state the agent no longer
# publishes (an old version it kept, or a cluster whose removal it rejected).
clamp_min(
  sum by (job, node, pin, reason) (aether_agent_xds_acked_tls_clusters)
  - sum by (job, node, pin, reason) (aether_agent_snapshot_tls_clusters), 0) > 0

# Did the proxy reject a cluster update. Zero is a real answer since #1480.
sum by (job, node) (increase(aether_agent_xds_nacks_total{aether_xds_type_url="type.googleapis.com/envoy.config.cluster.v3.Cluster"}[1h]))

Grading a soak on these series (#1423, #1491). A soak is run and graded by an external soak harness, maintained outside this repository. What such a harness reads is written down in the external-harness contract: the counter (agent.identity_cluster_unpinned), the published and the acknowledged gauge (agent.snapshot_tls_clusters, agent.xds_acked_tls_clusters), the closed reason set, and the split of the reasons the alert rules use (agent.unpinned_reason_classes: gap and no_tls). The grading procedure is the harness's own: its thresholds, its flags, its verdicts and the lines it prints are documented with it, and nothing in this repository defines or tests them. What holds for any grader, because it is how the product behaves:

• The two gap reasons (no_namespace_metadata, pin_not_rendered) are the validation gap: a TLS cluster that checks no server identity. Any movement of the counter, and any non-zero sample of the published gauge, is that state.
• The two no_tls reasons (trust_domain_unknown, tls_not_published) are expected at an agent start, and a soak rolls the agents on purpose. What tells a start from a standing state is how long the published gauge shows it, per node, job and reason over every series of them (the sum by and the for: 5m of AetherMeshClusterPinPending). The counter cannot make that call: it grows by entries times snapshots, not by time.
• A standing tls_not_published can be a cluster's design (no SPIRE, or an unpublished TCP floor). Whether that is expected is a statement about the cluster, so a grader has to be told. The series cannot say.
• A series of the counter with no reason label is from an agent before #1424, where the two kinds cannot be told apart. That holds for the counter only (agent.identity_cluster_unpinned), whose every series carries reason. On the two gauges reason is on the pin="unpinned" series alone (the contract's when), so a pin="pinned" series without it is the healthy shape, from any agent.
• A gauge that does not reach the store is not a zero. A replaced pod is not covered by the one before it, so an agent that has the label and whose own gauge is absent has proven nothing (see Absent is "not known", not zero above).

The NACK counters (#1480). aether_agent_xds_nacks_total counts the delta responses a proxy rejected, by aether_xds_type_url; aether_agent_xds_ack_wait_failures_total counts the ACK waits of a pod's listener that failed, by aether_xds_wait (present, absent) and aether_xds_reason (nack, timeout, no_proxy). Both are seeded at zero when the agent starts, so "no NACK" is a series at 0 and not an absent one. The first has seven series: one per resource type the agent serves (…listener.v3.Listener, …cluster.v3.Cluster, …endpoint.v3.ClusterLoadAssignment, …route.v3.RouteConfiguration, …tls.v3.Secret, …core.v3.TypedExtensionConfig, each as its full type.googleapis.com/… URL) and other, which nothing uses today. The second has five since #1624: present by nack, timeout and no_proxy, and absent by timeout and no_proxy. There is no absent/nack any more (it had four, with that one, from #1480 to #1624): no rejection fails a removal wait. Agents before #1480 created each series on its first increment, so on those an absent series is the healthy state and cannot be told from an agent that reports nothing.

What the two waits mean (#1624). The agent keeps what the proxy holds per xDS stream. A stream is one connection of one proxy process: a hot restart has two for a while, and a proxy that reconnects opens a new one. What a proxy holds is what it stated in its first Listener request on the stream (initial_resource_versions), with every response it answered since. Everything goes with the stream when it closes. A proxy counts for a wait from its Listener request on, not from the moment its stream opens: a new generation that is still loading its clusters holds no listener. Read aether_agent_xds_ack_wait_failures_total with that in mind:

• present is the wait of a pod ADD, and it is for a version, not for a name. A pod's listeners are named after the pod (outbound_http_<namespace>_<pod>, inbound_<namespace>_<pod>), so a pod recreated under the same name (a StatefulSet replica) publishes other content under a name the proxy already holds. The wait is answered when one proxy holds the version the agent publishes at that moment and has nothing unanswered about the name. A proxy that states that version when it reconnects answers it, so a pod ADD retried or a pod re-added after an agent restart no longer counts a failure for a listener that was there all along (#1511). present/nack is a connected proxy rejecting the published version; a rejection of an older version, or by a proxy whose stream has closed, fails nothing. During a hot restart one generation holding the listener answers the wait and either generation rejecting it fails it: the proxy's envoy NACKed delta response line in the agent log says which, by time. That an answered wait means a serving proxy has the listener is not promised in that overlap: a parent that has stopped its listeners acknowledges a listener it does not add, and the agent sees only the ACK (//agent/test/mtlspool, TestADrainingProxyAcknowledgesAListenerItDoesNotAdd). The CNI plugin's in-netns probe is the proof that a listener serves.
• absent is the wait of a pod DEL, and it is for every proxy that has asked for its listeners to be known not to hold the listener. Each generation of a hot restart has its own sockets in the pod's network namespace. A proxy is not known before the agent has answered its first Listener request. Before #1624 the agent read "I know of no such listener" as "the proxy has none" and such a DEL returned at once (#1572).
• timeout and no_proxy are the two ways a wait runs out, and they are different findings. timeout: there was a proxy to ask, and it did not give the answer within the wait (2 s). For absent that is a proxy that did not acknowledge the removal of a pod's listener before the pod's network namespace was torn down, which is the one to look into. no_proxy: no connected proxy had asked for its listeners, so there was nobody to ask. That is every managed pod ADD and DEL that arrives after an agent restart until the proxy has reconnected, or while the proxy is down; each takes the full wait and then proceeds. A burst of no_proxy at an agent roll is expected, and its length is how long the proxy took to reconnect. absent/no_proxy is new with #1624: those DELs used to return at once and count nothing.
• A rejected response is read for what the proxy does with it, measured on the pinned proxy (//agent/test/mtlspool). It does every removal of the response before any add, so a listener the response removed is gone and its DEL's wait returns (TestRemovalCarriedByARejectedResponseIsDone). It keeps the listeners of the response that it could build, so a listener the response added is not read as absent: its DEL waits for the removal, which the agent still sends (TestRejectedListenerResponseIsAppliedInPart). One limit remains and is pinned (TestListenerOfARejectedResponseIsNotStatedAtReconnect): a listener kept that way is not in the proxy's statement when it reconnects, so after an agent restart the agent has no word of it.

The label names are the OTLP attribute keys (aether.xds.type_url, aether.xds.wait, aether.xds.reason) with their dots turned into underscores by the OTLP ingest, like every other metric here. Read back from a test cluster's Prometheus on 2026-10-08: aether_agent_xds_ack_wait_failures_total is stored with exactly aether_xds_wait, aether_xds_reason, job and node. aether_agent_xds_nacks_total had no series there (no NACK in 30 days, which is the gap this seeding closes), so its name and its aether_xds_type_url label are the same translation applied to its sibling, not a reading.

# VictoriaLogs (field syntax; never `| stats`, it false-zeroes).
_stream:{service.name="aether-agent"} AND "published with no server-identity SAN pin" AND reason:no_namespace_metadata

The config-shape half is a build-time gate: //agent/test/envoy_validate asserts every upstream TLS context in a generated bootstrap carries a non-empty match_typed_subject_alt_names. envoy --mode validate accepts an unpinned context, so validation passing says nothing about it.

The ledger join, with the terminating-node column

The join that has been missing: each failing request must be attributed to the node whose proxy served it, then compared with the node whose proxy was restarting.

1. Pull the events. Field syntax only — a bare phrase search or | stats returns a documented FALSE ZERO. Control-test every negative by dropping the reason field (normal 200s must come back).

log_name:aether_access_logs
  AND upstream_transport_failure_reason:"CERTIFICATE_VERIFY_FAILED"

Do not prefix it with _stream:{service.name="aether-proxy"}. The access-log records carry an empty _stream and no service.name, so that prefix matches nothing (0 of 20,289,652 records over 2026-10-01 00:40:30–08:40:30Z).

The identity in certificate SANs are [spiffe://…] on these lines is the server's. Keep upstream_host, upstream_cluster, pod_name/pod_namespace (the local pod this hop serves — the client side here), response_flags and the timestamp.

1. upstream_host → pod → node. upstream_host is <pod IP>:18008. Resolve the IP to its pod and that pod's node:

kubectl get pods -A -o wide --field-selector status.phase=Running \
  | awk '$7=="<upstream-ip>" {print $1, $2, $8}'   # ns name node

For an IP that is already gone, use the agent-side record on each node (aether_agent_storage_pods is the per-node count; the entries themselves are the agent's protojson store) or the pod-IP index in the run record. That node is the terminating node — its proxy holds the inbound listener whose chain presented the certificate the client rejected.

1. Compare with the restarting node. The proxy generation boundary per node:

max_over_time(envoy_server_hot_restart_epoch[8h])          # step, per node
max_over_time(envoy_cluster_ssl_fail_verify_san_total[8h]) # never an instant query

max_over_time is mandatory — the series ages out with the proxy generation and an instant query at grade time returns no series at all (that trap has produced two premature "zero" readings on this issue).

1. Read the verdict.
2. Terminating node == the restarting node → same-node termination by the restarting proxy. Grep that node's agent for the WARNs above in the same window; a hit localises the defect to the agent's snapshot, a miss to Envoy's SDS lifecycle.
3. Terminating node != the restarting node → the server was a remote proxy that itself shows no counter (the counter is client-side). Grep that node's agent log.
4. Terminating nodes scattered across many nodes for one presented identity → the identity was not bound per-server, and upstream_host is not the TLS-terminating peer; record it and re-open the transport path.

L4 hops: the source / destination / SAN join from the L4 access log (#1023)

The ledger above reads the HTTP access log, which has no record of an L4 hop: until

1023 an L4 ssl_fail_verify_san tick had no source, no destination and no SAN on

record, and the #1007 event had to be inferred from per-pod connection counters and the chain config. Every capture TCP/TLS chain now writes one record per connection on the aether_l4_access_logs stream (see "L4 stat keys and the L4 access log" below), and a SAN rejection is a connection with a response flag, so it is always logged, never sampled away:

log_name:aether_l4_access_logs
  AND upstream_transport_failure_reason:~"CERTIFICATE_VERIFY_FAILED"

Control-test the negative the same way: drop the reason field and the clean connections of the same chains must come back. Each line is the whole join:

field	answers
node_name (resource attribute), pod_name, pod_namespace, source_netns, source_spiffe_id	source: which pod dialled, from which netns, presenting which identity
filter_chain_name	which capture chain took it: cap_tcp_* (floor, per-port), cap_tls_* (TLSRoute SNI), cap_tcp_blackhole
downstream_local_address	what the client dialled (the restored VIP:port)
upstream_cluster	which L4 cluster was chosen, as its stat key (tcp_<ns>/<svc>[_<port>], the aether_cluster the tick landed on): Envoy's %UPSTREAM_CLUSTER% renders the cluster's alt_stat_name, not its config name tcp:<fqdn>[:<port>]
upstream_host	the intended destination endpoint, <pod IP>:18008 → pod → node as in step 2 above
requested_server_name	SNI (- on the floor; the port on a per-port cluster; the hostname on a TLSRoute chain)
upstream_transport_failure_reason	the rejection, with the presented SAN where the pinned proxy prints it (certificate SANs are [...])
upstream_peer_uri_san	the verified server identity on a successful connection (- on a rejection: the handshake never completed)

Verdict, per line:

• Presented SAN is the identity of the pod at upstream_host → the connection reached the endpoint it dialled and the pin is wrong (check sanURIs, cache/mtls.go).
• Presented SAN is another workload's, and that workload has a pod on the source's node → a local pod's inbound terminated the connection, not upstream_host: the cross-pod landing below (#1007/#1022). The node's newest mesh pod at that minute is the usual suspect; its in_tcp_<namespace>_<pod> counter confirms it.
• Presented SAN is another workload's on a different node → upstream_host was not the terminating peer; record it and re-open the transport path, as for HTTP.

L4 stat keys and the L4 access log (#1023)

Stat keys. Every L4 cluster reports under its own alt_stat_name, which the proxy bootstrap's aether.cluster stats tag turns into the aether_cluster label:

cluster (config_dump name)	aether_cluster
HTTP default <svc>.<ns>.<domain>, its port aliases and per-port clusters	<ns>/<svc> (unchanged)
QUIC twin quic:<fqdn>@<ns>/<sa>	<ns>/<svc>@<ns>/<sa> (unchanged, #960)
TCP floor tcp:<fqdn>	tcp_<ns>/<svc>
TCP per-port or primary-port alias tcp:<fqdn>:<port>	tcp_<ns>/<svc>_<port>
UDP floor udp:<fqdn>	udp_<ns>/<svc>

Before #1023 every L4 row reported as <ns>/<svc>, so aether_cluster="aether-test/mixed-svc" mixed the HTTP cluster with the :9000 TCP cluster, and a tick could not be assigned to a cluster kind (#1007).

• _, not :. Envoy sanitizes every stat name and tag value (Stats::Utility::sanitizeStatsName rewrites : to _), so a tcp:…:9000 key would be exported as tcp_…_9000 anyway, and a selector written as tcp:.* would match nothing, forever. The key aether writes is the label you query. Namespaces and service names are DNS labels, so _ is unambiguous.
• No tls_ key. A TLSRoute SNI chain routes to its backends' per-port tcp:<fqdn>:<port> clusters (the primary-port alias when the backendRef names the primary port), never the floor: every backendRef is resolved by its port, and Gateway API requires a port on a Service backendRef. So its connections count under tcp_<ns>/<svc>_<port>, and a selector anchored on the floor key (^tcp_<ns>/<svc>$) reads nothing for a TLSRoute backend (#1044; on kind, upstream_cluster=tcp_aether-test/l4tls-a_9443). The chain itself is told apart in the L4 access log (filter_chain_name = cap_tls_*).
• Cardinality is one key per cluster (services × raw-TCP ports), never per endpoint or per source.
• HTTP queries are unaffected: an exact <ns>/<svc> or <ns>/<svc>(@.*)? selector cannot match an L4 key. //agent/test/envoy_validate runs the chart's own tag regex over the keys, after Envoy's sanitization, and pins that.

# L4 only, every kind
sum by (node, aether_cluster) (rate(envoy_cluster_upstream_cx_total{aether_cluster=~"tcp_.*|udp_.*"}[5m]))
# One service's L4 clusters: the floor and every port
sum by (aether_cluster) (rate(envoy_cluster_upstream_cx_total{aether_cluster=~"tcp_aether-test/mixed-svc(_[0-9]+)?"}[5m]))
# A TLSRoute backend: always port-qualified (the _<port> suffix is required)
sum by (aether_cluster) (rate(envoy_cluster_upstream_cx_total{aether_cluster=~"tcp_aether-test/l4tls-a_[0-9]+"}[5m]))
# Client-side SAN rejections on L4 clusters (the soak gate; never an instant query)
max by (node, aether_cluster) (max_over_time(envoy_cluster_ssl_fail_verify_san_total{aether_cluster=~"tcp_.*"}[8h]))

A dashboard or alert that selected an L4 service by its bare key (aether_cluster="aether-test/tcp-echo") reads no series from a #1023 proxy onward. Move it to the tcp_ key.

The L4 access log. Every tcp_proxy chain on a capture listener carries an OTel access logger on the stream log_name=aether_l4_access_logs, next to the HTTP stream aether_access_logs. That covers the TCP floor and its port spellings, per-port chains, TCPRoute-weighted chains, TLSRoute SNI chains and the scoped-mode cap_tcp_blackhole.

• Same switch, same sink. It is on exactly when the HTTP log is (MeshConfig accessLogsEnabled; no new flag) and ships to the same otel_collector cluster.
• Connection-level. One record when the downstream connection closes (no access_log_options flush interval). Logged: every connection with a response flag (UF, UH, UO, NR, …, which is where a SAN rejection or a blackholed flow lands) plus the HTTP log's success sample (aether.access_log.sample, the same runtime key).
• Not logged: the redirect-all passthrough DefaultFilterChain (all non-mesh egress), the UDP capture listener (no per-datagram log), and the inbound side.
• A separate stream because the shapes differ: no method, path, authority, status or request id. It deliberately carries no reporter attribute. The collector's identity counters (aether_access_log_*, the otel-collector values in the platform's GitOps repository) select on reporter, because log_name is a resource attribute their transform cannot see, and an L4 record must not enter the HTTP request counters.
• Fields: pod_name, pod_namespace, source_netns, source_spiffe_id (the source), filter_chain_name, downstream_local_address (the dialled VIP:port), downstream_remote_address, upstream_cluster, upstream_host, upstream_local_address, requested_server_name (SNI), upstream_peer_uri_san, response_flags, upstream_transport_failure_reason, connection_termination_details, start_time, duration_ms, bytes_received, bytes_sent, proxy_epoch (the proxy generation that held the connection; see "Reading a hot-restart overlap"). The node is Envoy's resource attribute node_name.

# Every failed L4 connection
log_name:aether_l4_access_logs AND response_flags:!"-"
# One service's L4 traffic
# (upstream_cluster is the stat key, tcp_<ns>/<svc>[_<port>], never the tcp:<fqdn> config name)
log_name:aether_l4_access_logs AND upstream_cluster:~"^tcp_aether-test/tcp-echo(_[0-9]+)?$"
# TLSRoute chains only
log_name:aether_l4_access_logs AND filter_chain_name:~"^cap_tls_"
# One TLSRoute backend (per-port key: tcp_<ns>/<svc>_<port>, never the bare floor key)
log_name:aether_l4_access_logs AND filter_chain_name:~"^cap_tls_" AND upstream_cluster:~"^tcp_aether-test/l4tls-a_[0-9]+$"

Cross-pod L4 landings (#1007/#1022)

Symptom. A node proxy's outbound L4 connection to tcp-echo or mixed-svc is rejected with ssl_fail_verify_san: it reached the inbound :18008 listener of an unrelated pod on the same node (always the node's newest mesh pod), which presented its own SVID. A client that dials several L4 ports in one pass sees it as one failure on every L4 leg at once (the multi-protocol dialer of the soak did; the soak is run by an external soak harness, maintained outside this repository, and the workloads named in this section are that harness's).

Two defects, one proof order.

• (b) #1022, Envoy. Network::Utility::execInNetworkNamespace recorded the namespace to return to from /proc/self/ns/net, which is the main thread's namespace. setns() is per thread, so a worker calling it while the main thread was briefly inside a pod netns (health-check connects every 5 s per pod, listener socket creation) "restored" itself into that pod's netns and stayed there, creating its later upstream sockets inside the pod where redirect-all capture diverted them. The proxy carried envoy-aether1022-exec-in-netns-thread-self.patch (/proc/thread-self/ns/net, fallback /proc/self/task/<tid>/ns/net) until #1643. The pinned Envoy now contains the upstream fix (envoyproxy/envoy#47776, which reads /proc/thread-self/ns/net and has no fallback), so the patch is no longer carried (proxy/README.md).
• (a) #1007, aether. The capture listener's use_original_dst: true hands a diverted connection to "the listener bound to its original address", looked up by the address string only (0.0.0.0:18008, no netns), so the most recently added pod's inbound listener wins.

Fixing (a) alone hides (b): the leaked connection would then leave through the right endpoint's ORIGINAL_DST from a pod IP and succeed silently, and this counter would go quiet for the wrong reason. So (b) is proven on the reference cluster first, with (a) still in place.

The proof signal: TCP-floor connections on pods that serve no raw-TCP port. The inbound listener's DEFAULT chain is the TCP floor (in_tcp_<namespace>_<pod>, stat prefix inboundTCPFloorStatPrefix in agent/internal/xds/proxy/ingress.go). Only a pod whose primary port is raw TCP can legitimately receive a connection there; per-port raw-TCP chains are in_tcp_<namespace>_<pod>_<port> and are excluded. The pod's namespace and name are part of the METRIC NAME, so select by __name__ pattern and read the RAW counters (a series is born on the first stray connection). The namespace is in the name since chart 2.5.0 (#1584); an agent older than that writes in_tcp_<pod>, so the exclusion of the TCP-primary workload below is spelled for one or the other (tcp_echo_ before, aether_test_tcp_echo_ since), and the pod label the second query builds holds <namespace>_<pod> with every - turned into _:

# Stray landings: default floor chain of every pod except tcp-echo (TCP-primary);
# the per-port chains (…_<port>_downstream_cx_total) are legitimate and excluded.
sum by (__name__) ({__name__=~"envoy_tcp_in_tcp_.*_downstream_cx_total",
                    __name__!~"envoy_tcp_in_tcp_aether_test_tcp_echo_.*|envoy_tcp_in_tcp_.*_[0-9]+_downstream_cx_total"})

# The same over a window, with the pod lifted into a label (the soak gate, #1023).
# max_over_time drops __name__, so the label is taken first, inside a subquery.
max by (node, pod) (max_over_time((label_replace(
  {__name__=~"envoy_tcp_in_tcp_.+_downstream_cx_total",
   __name__!~"envoy_tcp_in_tcp_(aether_test_tcp_echo_.+|.+_[0-9]+)_downstream_cx_total"},
  "pod", "$1", "__name__", "envoy_tcp_in_tcp_(.+)_downstream_cx_total"))[8h:1m]))

# The client side: SAN rejections on the L4 clusters, per node and per L4 cluster
# (never an instant query). From #1023 on, the tcp_ keys hold ONLY L4 clusters, and
# aether_cluster names the floor or the port the misdirected connection was dialled on.
max by (node, aether_cluster) (max_over_time(envoy_cluster_ssl_fail_verify_san_total{aether_cluster=~"tcp_.*"}[8h]))
# ...on a pre-#1023 proxy the same ticks read under the bare keys, mixed with HTTP:
max_over_time(envoy_cluster_ssl_fail_verify_san_total{aether_cluster=~"aether-test/(tcp-echo|mixed-svc)"}[8h])

# The landing pod's inbound sees the client abort after its SAN check
max_over_time(envoy_listener_inbound_ssl_connection_error_total[8h]) > 0   # by aether_pod, node

The exclusion list is the soak's: tcp-echo is its only TCP-primary workload. The stat keys cannot derive it (an HTTP-primary service with a raw-TCP port, like mixed-svc, has a tcp_ floor key too), so a cluster with another TCP-primary workload adds it to the __name__!~ alternation. A pod whose 5-character hash suffix happens to be all digits is excluded along with the per-port chains; that is rare (about 0.1% of pods) and errs toward a missed landing, not a false one.

Attribute a tick by joining on node and minute. Since #1023 the L4 access log carries the whole join in one record (see "L4 hops" above): the source pod, the dialled VIP:port, the chosen tcp: cluster, the intended upstream_host and the rejection. Before it, the join was three counters: the verify_san +1 on node N, a new or incremented in_tcp_<namespace>_<pod> series for a pod on N, and that pod's inbound_ssl_connection_error climbing in the same minute (the 2026-09-27 16:37Z node D event in #1007 is the worked example).

Reading it.

build	expected
rev242 and earlier (no thread-self patch) — the negative control	non-zero on the prober and on every workload of the soak (its HTTP services, load generators, dialers and echo servers); ~1 burst per node per hour; verify_san ticks on tcp-echo/mixed-svc (23 over the rev242 soak)
rev243 (unpatched, 1h47m generation, 2026-09-27 22:53Z–09-28 00:39Z)	6 stray floor connections, all on prober pods (node E 2, node C 3, node D 1), and 1 verify_san on node C tcp-echo
first proxy with the #1022 patch, #1007 still unfixed	no new series and no increments after every node's proxy has rolled onto it (series from older generations age out with them)

A landing that persists on the patched proxy refutes #1022 as the (only) cause: something else moves node-proxy sockets into pod netns — keep #1007 unmerged and re-open the attribution. Only once the patched proxy reads zero for a full soak does the #1007 capture fix merge; after it, this counter no longer discriminates (b).

Known-unexercised code paths

Recovery branches that have never run in production. This is the system working, not a backlog — it is recorded so nobody mistakes "no data" for "untested logic", and so nobody re-attempts a trigger that is known not to be forceable. Retired from #813, which was an umbrella that could not close.

path	why it has never fired	how it is covered
snapshot stale-netns guard, aether_agent_snapshot_stale_netns_skipped_total (#798)	a replacement agent returns in ~5 s, inside the 60 s NetnsUnpinDelay	would need an agent held down > 60 s on a node with a terminated app pod there
CNI DEL agent unreachable WARN, the <pin>.delfail give-up path, the pin unlink (#796)	same — the agent comes back too fast	unit-tested
supervisor drain_fallback (#797)	the normal drain path always wins	aether_supervisor_shutdown_branch_total{branch="drain_fallback"} has never been non-zero anywhere
edge xDS "registry unreachable … while this workload waits for its first SVID" (#807)	both edge pods reach the registrar in ~3 s	unit-tested only
svid_updates{update="unchanged"} (#806)	a SPIRE agent restart re-mints, so it counts rotated	would need a Broker stream that drops while the SPIRE agent stays up
orphan prune → SVID unsubscribe (#804)	not forceable from a harness — the runtime re-issues CNI DEL at sandbox removal and a restarted agent serves it before the first sweep pass (measured 2026-09-19)	proven once in production (2026-09-18); the deterministic check is the in-process test from #805

Two cautions when tempted to "test" one of these:

• Contriving the precondition proves the branch compiles, not that it behaves. Forcing drain_fallback or the #807 WARN by hand exercises the code under conditions you invented, which is the same trap as a test you author both sides of.
• #804 in particular should not be re-attempted from a harness. That was measured and ruled out; the in-process test is the coverage.

Genuine test debt is tracked separately — see #868 for L4 route e2e coverage, which is a shipped default-on feature with no end-to-end test at all.

The proxy rejected a Listener update (AetherProxyRejectedListenerUpdate, #1633)

What it means. The agent sent its proxy a Listener (LDS) response and the proxy answered with a NACK: at least one listener in it could not be built. The agent is the only writer of the proxy's configuration and the proxy is the pinned one, so a rejected update is always an agent defect, whatever led to it (a listener bound into a network namespace that is gone, a filter configuration the proxy refuses). Report it, with the log line below.

The agent counts it, per resource type, in aether_agent_xds_nacks_total; the series for listeners is aether_xds_type_url="type.googleapis.com/envoy.config.listener.v3.Listener", seeded at zero when the agent starts (#1480). AetherProxyRejectedListenerUpdate (docs/observability/agent-xds-alerts.yml) fires on one rejection, per job and node:

sum by (job, node) (increase(aether_agent_xds_nacks_total{aether_xds_type_url="type.googleapis.com/envoy.config.listener.v3.Listener"}[1h])) > 0

The rule has a second arm for a series whose first sample is already above zero (a rejection before the agent's first export, on a node nothing reported from before), where increase() is 0. The alert lasts an hour from the last rejection and then ends whatever was done: the counter cannot tell that a proxy was replaced. Read it as an event to act on. An agent restart puts the counter back to zero, and does not undo what is described next.

What the proxy did with the update. A NACK is not "nothing was applied". Measured on the pinned proxy by //agent/test/mtlspool (lds_reconnect_test.go, added with #1632):

• It does every removal of the response before any add, so a listener the response removed is gone (TestRemovalCarriedByARejectedResponseIsDone).
• It adds the listeners of the response that it could build, and runs them. Only the listener it refused is missing (TestRejectedListenerResponseIsAppliedInPart, which also shows the agent removing such a listener later on the same stream).
• A listener it kept that way is not in what it states when it reconnects (TestListenerOfARejectedResponseIsNotStatedAtReconnect). A reconnecting proxy tells the control plane the resources it holds (initial_resource_versions), and the agent removes what it no longer publishes from that statement. The listener kept from a rejected response is not listed.

So while the xDS stream that carried the update lives, the agent knows of the listener and removes it when its pod goes. After an agent restart (a roll, a crash, an upgrade), or after the proxy's xDS stream reset, the agent has no word of it. If the pod is still there, the agent publishes the listener again and nothing is lost. If the pod went while the agent or the stream was down, nothing removes the listener from the proxy: it stays bound in a network namespace that is dead or has been handed to another pod, a pod deletion is not held for it, and no metric of the agent shows it. The agent does not look for such listeners, by design: the precondition is an agent defect, and it is made loud instead (#1633).

The pod whose listener was refused is a different matter: the proxy does not run that listener. A pod ADD that was waiting for its outbound listener when the response carrying it was rejected counts aether_agent_xds_ack_wait_failures_total{aether_xds_wait="present",aether_xds_reason="nack"}. That wait is best-effort and does not fail the ADD.

Finding the rejected listener. The agent logs one INFO line per rejected response, on the node the alert names:

envoy NACKed delta response   typeURL=type.googleapis.com/envoy.config.listener.v3.Listener added=[<listener> …] removed=[<listener> …] error="<the proxy's reason>"

added and removed are the listener names the response carried. error is the proxy's error_detail, which names the listener it refused and why. Every other name in added is a listener the proxy now runs from a rejected response: those are the ones the agent loses at its next restart.

# VictoriaLogs
_stream:{service.name="aether-agent"} AND "envoy NACKed delta response" AND "envoy.config.listener.v3.Listener"

kubectl -n aether-system logs <aether-agent pod on the node> -c agent | grep 'envoy NACKed delta response'

The proxy's own side is listener_manager.lds.update_rejected and listener_manager.listener_create_failure; read them with max_over_time(...[1h]), never as an instant query.

Recovery. In this order:

1. Fix the cause first, so that the agent no longer publishes the listener the proxy refuses (the error field says which and why; added no longer carries it, and the counter stops moving). Do not replace the proxy before that: a hot-restart successor is a new Envoy that receives every listener of the node in its first response, and one that rejects that response can come up with no listeners at all (see "A pod is stuck Terminating, or a node has stale netns entries" below, "Why the snapshot-time skip matters").
2. Replace the proxy pod on that node:

kubectl -n aether-system get pod -o wide --field-selector spec.nodeName=<node> | grep aether-proxy
kubectl -n aether-system delete pod <aether-proxy pod on the node>

The new pod's Envoy is a new process: it holds only the listeners the agent sends it, and the listener nobody knew of ends with the old process. With the hot-restart supervisor this is a handoff and not an outage: the delete takes the successor_wait branch ("What the proxy supervisor does on SIGTERM" above), the surge replacement starts within about a second and hot-restarts the node's Envoy, the old one drains and exits. Expect 20 to 25 s of pod termination and no prober error on the node. Connections still open on the old Envoy when its drain ends are closed, as in any proxy roll. (That a hot restart ends such a listener follows from what a hot restart is; it was not measured for this case.) 3. Report the defect, with the agent's NACK line and its version. The alert ends by itself an hour after the last rejection.

If the alert names the edge control plane's job, the same reading applies to the edge proxy it serves; there is no per-pod listener to lose there, and replacing the edge proxy pod after the fix is the same recovery.

When the alert repeats with no new NACK line. During a surge roll of the agent two agents export for one node. A metrics pipeline that keeps agents apart by node alone stores both in one series, and an old count interleaved with the successor's zero reads as an increase. It can only happen on a node whose outgoing agent had counted a rejection. Check the agent log for the line above before acting on it.

A pod is stuck Terminating, or a node has stale netns entries (#245, #796)

Two related symptoms, one mechanism: the pod's CNI DEL never completed.

Symptom A — the sandbox will not tear down. kubectl describe pod shows repeated KillPodSandbox failures, the pod sits Terminating for minutes, and — the part that hurts — it keeps its CPU request the whole time. On a node that is already tight this starves the replacements for that node's own DaemonSets (0/6 nodes are available: 1 Insufficient cpu), so the agent, proxy and mesh-dns cannot come back and the node goes unmanaged. Priority does not help: the scheduler refuses to preempt on a node that has a terminating pod (preemption: not eligible due to a terminating pod on the nominated node). That is the #796 outage — 12m47s on worker-03.

Since #796 the plugin does not enter that loop when the agent is simply gone. CNI DEL classifies the failure:

• Nothing answered (the agent's socket is missing, or the RPC came back Unavailable/DeadlineExceeded/Canceled) → the DEL logs WARN agent unreachable at CNI DEL; unpinning on the normal delay and letting the ghost sweep reconcile, skips the DEL readiness probe, schedules the usual detached unpin, and returns success. The sandbox tears down, the CPU request is released, and the node keeps its agent.
• A live agent answered with an error → unchanged: the DEL fails, containerd retries, the netns stays pinned. That loop is now bounded by netns_del_give_up_after_seconds (default 5m, tracked in a <pin>.delfail marker beside the netns pin, since the plugin process lives for exactly one CNI call). Past the bound it degrades to the first case and logs WARN CNI DEL has been failing past the give-up bound.

So a pod still stuck Terminating means either a live agent that keeps refusing the removal (grep that node's agent for the RemovePod error) or a failure outside aether (the primary CNI, the runtime). Confirm which before force-deleting:

# Is the node's agent even there? (the unreachable case should no longer stick)
kubectl -n aether-system get pods -o wide --field-selector spec.nodeName=<node>
# The plugin's own verdict, on the node:
journalctl -u containerd | grep -E 'agent unreachable at CNI DEL|give-up bound'
# Node pressure, which is what turns one stuck DEL into an outage:
kubectl describe node <node> | sed -n '/Allocated resources/,/Events/p'

Force-deleting the pod (--grace-period=0 --force) clears it in seconds and is the right escape hatch, but it leaves exactly the state of symptom B.

Symptom B — stale storage entries for pods that no longer exist. The agent's host registry (/var/lib/aether/registry/<containerID>.json) still lists a pod whose netns is gone. This is expected after the unreachable path above: the plugin completed the DEL without the agent, so nothing deregistered the pod. It self-heals — do not hand-edit the registry.

• At startup, LoadListenersFromStorage skips any pod whose netns is missing, so the stale entry never reaches the snapshot (skipping pod with missing network namespace).
• On every snapshot generation the same check runs again (#717): the pod's listeners and its per-pod app/health clusters are left out, logged once per pod as skipping pod with missing network namespace in snapshot generation, and counted in aether_agent_snapshot_stale_netns_skipped_total.
• The ghost sweep (60s) prunes the entry, drops its listeners and per-pod clusters, and deregisters the endpoint. It counts the prune in aether_agent_ghost_sweep_stale_pruned_total (OTel instrument aether.agent.ghost_sweep.stale_pruned).
• CNI GC unpins orphan netns pins and removes orphan .delfail markers as a backstop.

# Prunes per node over the last hour. A burst right after an agent comes back on a node
# is the expected #796 reconciliation; a series that never stops is a defect.
sum by (node) (increase(aether_agent_ghost_sweep_stale_pruned_total[1h]))
# Stale entries the snapshot generator had to step over. Non-zero is normal for a minute
# after a DEL the agent missed; still climbing an hour later means the sweep is not
# pruning. Both counters are seeded, so a flat 0 is a real reading.
sum by (node) (increase(aether_agent_snapshot_stale_netns_skipped_total[1h]))
# Entries the agent is tracking per node -- must settle at the node's managed pod count.
aether_agent_storage_pods

The metric families are aether_agent_ghost_sweep_* and aether_agent_snapshot_*. Querying aether_ghost_sweep_* or the dotted OTel spelling returns a false zero.

Why the snapshot-time skip matters more than the crash ever did. A stale netns on the listener side does not crash the pinned proxy — the listener is rejected (listener_manager.listener_create_failure, listener_manager.lds.update_rejected, the log line failed to open netns file <path>: No such file or directory, and lds.version_text frozen at the last good version) and everything else keeps serving. But a hot-restart successor asked to create that listener NACKs the whole LDS response and comes up with zero listeners — every listener on the node, not just the stale one — because Envoy jumps into the netns before it asks the parent for the socket, so the inheritance that would have worked never happens. The port then dies when the parent finishes draining. That is why the agent filters stale pods out of every generation rather than relying on the sweep's ≤60s window being quiet. An in-place LDS modify of such a listener is silently accepted (the socket factory is cloned and the netns is never reopened), so a dangling path can hide across arbitrarily many updates and only surface at the next roll — do not read "LDS is being accepted" as "no stale netns".

# After a proxy roll on a node that had a stale entry: this must not go to zero.
envoy_listener_manager_total_listeners_active
max_over_time(envoy_listener_manager_lds_update_rejected[1h])   # never an instant query
max_over_time(envoy_listener_manager_listener_create_failure[1h])

Why the stale entry is no longer dangerous. It used to be: Envoy 1.38 dereferenced a nullptr when a dial (or a cold-start health checker) opened a netns that had vanished, so one stale entry could make the node proxy unbootable — the whole reason CNI DEL blocked on the agent's ACK. The pinned proxy snapshot (1.40.0-dev.20260926.726d7ac, and every pin since 20260904.13144fb) carries envoyproxy/envoy#45975 (the pool dial returns a clean LocalConnectionFailure, a UF for that request) and #46503 (the active TCP/HTTP/gRPC health checkers record a NETWORK failure instead of crashing). A stale per-pod cluster now costs at most one failed request plus an unhealthy host until the sweep prunes it. envoyproxy/envoy#45976 (opt-in netns validation at config load) is in the snapshot too and stays off — it would turn a stale pod into an LDS/CDS NACK, which is worse. The netns pin and its 60s unpin delay stay: a hot-restart successor re-creating the pod's listeners and dials deferred 10-13s past removal still need a live netns to succeed rather than merely fail cleanly.

One node agent looks several times hotter than its peers in Pyroscope (#1131)

Symptom. In Pyroscope (process_cpu, service_name="aether-agent", grouped by node) one aether-agent pod shows 2–5× the CPU of the others while doing the same work. Which agent is hot changes when agents restart and is fixed for the life of the process.

Resolution (2026-10-03). It is a sampling artifact, not a hot agent. The kernel's own per-process counter, aether_agent_sched_cpu_time_seconds_total (schedstat sum_exec_runtime, exported by the agent since chart 2.2.4), put the five agents within 1.16–1.24× of each other over the same hours in which Pyroscope showed 2.5× — and named a different node as the hot one. The two sources agree on the fleet total (the profiler's ~10 % surplus is softirq time it attributes to the process); only the split between pods is wrong. The profiler sampled at 20 Hz, exactly 50 ms, which divides the periods of everything a mostly idle agent does (kubelet readiness 2 s and liveness 10 s, controller-runtime's 500 ms and 10 s queue timers, Go's round-millisecond tickers): the sample instants line up with one process's timer bursts and miss another's, so identical work is charged many times more to one pod, and the phase only changes on restart. The flame-graph "excess" was always strictly periodic work (the :8082 health handler, the queue tickers) with identical request rates on every node, and other low-activity timer-driven pods (uds-csi, the MetalLB speaker) show the same spread.

Do not compare mostly idle processes by Pyroscope per-pod process_cpu. Use the kernel counters, which need no profile:

# Hot-agent ratio: max / min agent CPU across nodes, 30 m windows. ~1.2× is normal.
max(rate(aether_agent_sched_cpu_time_seconds_total[30m]))
  / min(rate(aether_agent_sched_cpu_time_seconds_total[30m]))

If this ratio is genuinely high, the agent's scheduler counters tell "more wakeups" from "costlier wakeups". They are read from /proc/self/task/*/{schedstat,status,sched}, summed over every thread, and monotonic even when threads exit; they are observable instruments, so reading them adds no timer.

Metric (Prometheus name)	Source	What it tells you
aether_agent_sched_timeslices_total	schedstat field 3	Times a thread was put on a CPU. Its rate is the wakeup rate.
aether_agent_sched_run_delay_seconds_total	schedstat field 2	Time spent runnable but waiting for a CPU. Divided by timeslices, it is the wait per wakeup.
aether_agent_sched_cpu_time_seconds_total	schedstat field 1	CPU time on a CPU. Divided by timeslices, it is the CPU per wakeup.
aether_agent_sched_context_switches_total{aether_sched_switch="voluntary"\|"involuntary"}	/proc/…/status	Voluntary switches (blocked or slept) follow the wakeup rate. Involuntary ones (preempted) point at contention.
aether_agent_sched_migrations_total	se.nr_migrations in /proc/…/sched	CPU migrations. Absent on a kernel without CONFIG_SCHED_DEBUG.
aether_agent_sched_threads	task count	Live OS threads. A hot agent with many more threads is a different problem.
go_schedule_duration_seconds_bucket	Go runtime/metrics /sched/latencies	How long runnable goroutines waited for a P. This is the Go-level twin of run_delay.

Every series carries job="aether-agent" and the node, both from the resource attributes. No series is seeded: an observable counter reports its cumulative value on every collection, so a missing series means the agent is not exporting it, not a value of zero.

# wakeups per second, by node
sum by (node) (rate(aether_agent_sched_timeslices_total[30m]))
# mean run-queue wait per wakeup (seconds), by node
sum by (node) (rate(aether_agent_sched_run_delay_seconds_total[30m]))
  / sum by (node) (rate(aether_agent_sched_timeslices_total[30m]))
# CPU cost per wakeup and migrations per wakeup
sum by (node) (rate(aether_agent_sched_cpu_time_seconds_total[30m]))
  / sum by (node) (rate(aether_agent_sched_timeslices_total[30m]))
sum by (node) (rate(aether_agent_sched_migrations_total[30m]))
  / sum by (node) (rate(aether_agent_sched_timeslices_total[30m]))
# Go scheduler p99 latency, by node
histogram_quantile(0.99, sum by (node, le) (rate(go_schedule_duration_seconds_bucket{job="aether-agent"}[30m])))

More timeslices per second at the same CPU per timeslice means a remaining ticker; the same timeslice rate with more CPU, run delay or migrations per timeslice means CPU placement or idle states, not the agent's code.

What #1131 changed anyway (#1146). The agent's controllers use a queue with no metrics (common/ctrlqueue), so each one no longer runs a 500 ms updateUnfinishedWorkLoop ticker (nothing scraped the agent's workqueue_* series), and --debug no longer enables controller-runtime's V(2)+ logging: its verbosity is capped at V(1) (manager.ControllerRuntimeMaxVerbosity), so the V(5) workqueue_items … items=[] dump every 10 s per queue is gone. aether's own debug and trace lines are unchanged. These stand as cleanups; they did not, and could not, move the Pyroscope ratio.

Profiler side. The eBPF profiler's sample rate should not divide the common timer periods: 19 Hz costs the same as 20 and does not; 97/99 Hz is the conventional choice if more overhead is acceptable. After the change the agents' Pyroscope max/min ratio should collapse towards the schedstat ~1.2×.

A UDS workload is not delivered: aether_agent_uds_resolve_failures_total{reason}

A pod that asks for UDS delivery (annotation or EndpointPolicy) but whose socket the agent cannot resolve falls back to TCP loopback; a UDS-only app then stays unpromoted (callers see no endpoint, never a blackhole). The agent counts it once per pod per reason and logs one ERROR line naming the pod (failed to resolve the pod's UDS socket; …, fields reason, pod, namespace, socket, source=annotation|endpointpolicy).

reason	Meaning	Fix
not_csi	The named volume exists but is not csi: {driver: csi.aether.io} — almost always an emptyDir left over from before chart 2.0.0	Switch the volume source and add securityContext.fsGroup (see §7 Upgrading to chart 2.0.0)
volume_not_declared	The pod declares no volume of that name (typically an EndpointPolicy drifted from its Deployment)	Fix the policy's udsSocket or the pod spec
bad_file	The file part is not a single clean path segment	Fix the value
path_too_long	<uds-csi-root>/<uid>/<file> over 107 bytes (file over 54 bytes at the default root)	Shorten the socket file name
multiple_csi_volumes	The pod declares two csi.aether.io volumes	Keep one
no_uid	The stored pod record has no UID (written by a pre-034 agent)	Restart the pod
bad_request	The value is not <volume>/<file>	Fix the value
disabled	The chart runs with udsCsi.enabled: false (--uds-csi-root=)	Enable it

A pod stuck in ContainerCreating with FailedMount: driver name csi.aether.io not found in the list of registered CSI drivers is on a node where the aether-uds-csi DaemonSet is not (yet) running; one with FailedMount … requires the pod to set securityContext.fsGroup needs an fsGroup. A deleted UDS pod stays Terminating while the plugin is down on its node (the kubelet cannot unpublish); it finishes once the plugin is back.

A pod started uncaptured, or a pod start is slow in the CNI plugin (#1166)

The CNI plugin binary exports no telemetry of its own. Until #1166 it linked the OTel SDK and the OTLP exporters (18 of its 37 modules, 3.2 MB of its 18.7 MB) and flushed them before every exit, which cost every pod ADD/DEL ~6 ms with a reachable collector and 2 s with an unreachable one — the #950 state the reference cluster was in for weeks. Nothing queried its spans. What only the plugin can see now reaches the collector through the agent:

• Capture divert (the nftables table + policy routing it installs AFTER AddPod answers): the plugin sends it on ReportAddResult, and the agent counts aether_cni_operations_total{aether_cni_operation="capture_divert",aether_cni_result=…} (same name and labels the plugin exported) and logs CNI plugin failed to install the transparent-capture divert; POD IS RUNNING UNCAPTURED at WARN with the pod. A non-zero error series is a pod the mesh silently does nothing for. The success series is the existence proof that reporting works.
• Timings, as attributes on the agent's RPC spans (--trace-export): aether.cni.plugin.pre_call_seconds on AddPod/RemovePod (netconf parse, PID/CRI lookup, netns pin — before the agent's own span starts), and on ReportAddResult aether.cni.plugin.{readiness_probe,capture_divert,total}_seconds plus …readiness_probe_error / …capture_divert_error.
• ADD/DEL counts and latency as the agent served them: otelgrpc's rpc.server.* metrics for aether.cni.v1.CNIService/AddPod and /RemovePod.

Still node-local only: an ADD that never reached the agent (agent down). Its trace is the kubelet's FailedCreatePodSandBox event and the plugin log, /var/log/aether-cni/plugin.log on the node (talosctl -n <node> read …).

A pod never becomes routable: what HEALTHY means since #815

An endpoint is advertised to the mesh only after the node agent's liveness loop promotes it to HEALTH_HEALTHY. Since issue #815 the agent probes two independent facts about each local pod, on two separate gateway paths:

Gateway path	Probe cluster	What it proves	Transport
/healthz/health_<namespace>_<pod>	health_<namespace>_<pod>	the application answers (HTTP GET on the readiness path, or a raw TCP connect for a protocol: tcp pod)	cleartext, in the pod's netns
/healthz/inboundready_<namespace>_<pod>	inboundready_<namespace>_<pod>	the pod's mesh inbound listener is listening, has loaded the pod's own SVID, and it verifies as that pod	mTLS, node identity → 127.0.0.1:18008 in the pod's netns, SAN-pinned to the pod's SPIFFE ID

Each path reflects its own cluster only. /healthz/health_<namespace>_<pod> means exactly what it meant before #815. A 404 on either path means "not programmed" — for the inbound path that is the normal, expected answer for an ungated pod, and it is never read as unhealthy.

inboundready_<namespace>_<pod> exists because health_<namespace>_<pod> has no SDS dependency at all, while the inbound listener does not listen() until its SVID arrives — so an endpoint used to be advertised HEALTHY while its mesh port still returned ECONNREFUSED (measured: promotion p50 5.8 s against an SVID at 6.1–8.4 s). It is not emitted when SPIRE is disabled (the inbound listener is cleartext, so there is nothing to prove), before the node SVID has been served (no client certificate to present), in edge mode, or for a pod with no netns.

Why the two paths are separate, and not ANDed. The first attempt (#819) put both clusters behind the single /healthz/health_<namespace>_<pod> path. The agent could then only see their conjunction, so "the application is fine but the mesh inbound never came up" was indistinguishable from "the application died". On 2026-09-19 node C published inbound listeners with an EMPTY trust domain, the conjunction went 503, and four already-serving endpoints were demoted and never re-promoted. A signal meant to gate a first promotion had become a permanent trap.

The promotion rules

application probe	inbound-readiness probe	endpoint already serving?	agent's decision
fails	any	any	UNHEALTHY (warm-up grace and the demote streak still apply)
passes	404 (ungated)	any	HEALTHY — no gate is programmed for this pod
passes	passing	any	HEALTHY — application up and mesh inbound proven
passes	never passed this epoch	yes	HEALTHY — can't-tell: an already-serving endpoint is never pulled on the TLS probe alone. WARN + inbound_gate_held_pods
passes	never passed this epoch	no	held un-promoted — this is the hole the gate closes. WARN after 60 s
passes	passed earlier, failing now	any	UNHEALTHY after the demote streak — expired SVID, or the listener lost its secret

"Already serving" means the endpoint is or has been advertised HEALTHY in the registry, not merely that the application probe passed once. Active-mode endpoints register HEALTHY at CNI ADD and so qualify immediately; EDS-mode endpoints register UNHEALTHY and qualify only after their first promotion. This survives an Envoy epoch change on purpose — after a proxy roll the endpoint is still advertised, which is exactly why the unproven probe must be can't-tell.

Metrics

Metric	Meaning
aether_agent_liveness_inbound_gate_pods{aether_gate_state="gated"\|"ungated"}	local pods with / without the gate programmed, recorded every liveness pass (5 s)
aether_agent_liveness_inbound_gate_held_pods	pods currently held un-promoted, or serving only on the can't-tell fallback
aether_agent_liveness_health_transitions_total{aether_health_from,aether_health_to}	endpoint health transitions; seeded at 0 for HEALTHY→UNHEALTHY and UNHEALTHY→HEALTHY so "zero demotions" is gradeable

# Is the gate on at all on every node? A standing ungated count with SPIRE
# enabled means the gate is missing, not that the pods are fine.
max_over_time(aether_agent_liveness_inbound_gate_pods{aether_gate_state="ungated"}[5m]) > 0

# Anything held by the TLS probe right now.
max_over_time(aether_agent_liveness_inbound_gate_held_pods[5m]) > 0

# Demotions, per node. This series now exists even when it is zero.
increase(aether_agent_liveness_health_transitions_total{aether_health_from="HEALTH_HEALTHY",aether_health_to="HEALTH_UNHEALTHY"}[10m])

The agent also logs the gate state once per change (and once at startup):

inbound-readiness promotion gate ACTIVE: HEALTHY requires an mTLS handshake with the pod's own inbound listener  gated_pods=5
inbound-readiness promotion gate NOT active for every local pod; those pods keep the application probe alone  gated_pods=0 ungated_pods=5 missing="no node SVID yet (…)"

and one rate-limited WARN per held pod (once per 5 min, after a 60 s grace):

liveness: pod held by the inbound-readiness probe  pod=svc-1-… namespace=team-a probe_cluster=inboundready_team-a_svc-1-… gateway_path=/healthz/inboundready_team-a_svc-1-… why="held un-promoted: the inbound mTLS probe has never passed (inbound listener has no certificate, or its SDS secret name is not served)"

Named failure: envoy_sds_spiffe_ns_* — listeners built with an EMPTY trust domain

Signature. Envoy subscribes to SDS secrets named spiffe:///ns/<ns>/sa/<sa> — note the missing trust domain between the second and third slash. The agent never serves that name, so the secret never resolves, the inbound listener never gets a certificate, and the pod is unreachable on the mesh. The healthy form is spiffe://aether.internal/ns/….

# Any proxy subscribing to a trust-domain-less secret. Should be EMPTY, always.
{__name__=~"envoy_sds_spiffe_ns_.*"}

# The specific shape seen on node C, 2026-09-19:
envoy_sds_spiffe_ns_aether_test_sa_.*_init_fetch_timeout_total == 1
# with update_attempt == 1 and NO update_success.

The agent says the same thing directly — grep its log for:

inbound chain bound to a foreign identity   bound_spiffe_id=spiffe:///ns/aether-test/sa/svc-1
                                            pod_spiffe_id=spiffe://aether.internal/ns/aether-test/sa/svc-1
                                            secret_served=false

Cause. The node agent's trust domain is late-bound (#740). #819 had the listener-regeneration paths sample the cache's copy before blocking on listenerMu, while LoadListenersFromStorage published the listener map before recording the trust domain. A reconciler that started inside that window (~700 ms on node C) read "", waited out the load on the lock, and then rewrote every per-pod listener with the malformed name.

It cannot recur by construction, and every layer is asserted by a test: proxy.SpiffeIDFromPod returns "" rather than spiffe:///; builders that need a real identity refuse with proxy.ErrNoTrustDomain; the cache's trust domain is a single atomic value read at the point of use, inside the same listenerMu section that writes the listeners; and it is recorded before any listener is published.

If you see it anyway: the pods are not repairable in place — the malformed config is already in Envoy. Roll the node's agent (which republishes every listener from the now-known trust domain); if that does not clear it, roll the workloads. Then reopen #815 with the agent's first 2 s of log, which contains the ordering.

Reading a pod that is stuck un-promoted

Both probe clusters keep their per-pod stats (they must — the health_check filter answers by READING their membership gauges; never exclude those, see the 2026-06-11 outage note in charts/aether/templates/agent-proxy-configmap.yaml). The cluster name is lifted into the aether.cluster tag by the proxy's stats_tags, so:

# Which local pods' inbound listeners are not serving their own certificate?
# 5s scrape at a coarser step reads as gaps — always max_over_time.
max_over_time(envoy_cluster_membership_healthy{aether_cluster=~"inboundready_.*"}[5m]) == 0

# The app probe, for comparison. App down => health_ is 0 too; inbound-only
# failure => health_ is 1 and inboundready_ is 0.
max_over_time(envoy_cluster_membership_healthy{aether_cluster=~"health_.*"}[5m])

If inboundready_<namespace>_<pod> alone is 0, the handshake is the problem, and the probe cluster's own SSL counters say which half (these are deliberately NOT excluded from the stats matcher, precisely for this):

increase(envoy_cluster_ssl_fail_verify_san{aether_cluster=~"inboundready_.*"}[5m])   # wrong pod answered on :18008
increase(envoy_cluster_ssl_connection_error{aether_cluster=~"inboundready_.*"}[5m])  # handshake failed / no certificate yet

Ranked causes, most to least common:

1. The pod's SVID has not arrived. Normal for the first 6–8 s after CNI ADD; a never-promoted pod simply waits, and an already-serving one keeps serving. Persistent means SPIRE — check the agent for SubscribeToX509SVID retries and see "The agent is stuck waiting for SPIRE".
2. The listeners were built with an empty trust domain. See the named failure above. envoy_sds_spiffe_ns_* distinguishes it from (1) in one query: in (1) the secret name is correct and merely unserved; here it is malformed.
3. The node SVID has not arrived. Then the probe cluster is not emitted at all, inbound_gate_pods{aether_gate_state="ungated"} is nonzero, the agent logs the "NOT active" line naming this precondition, and the node's outbound mTLS is degraded anyway. Same SPIRE check.
4. fail_verify_san on the probe. Something other than the expected pod is answering :18008 in that netns — the #638 stale-endpoint / recycled-pod-IP shape. See "Attributing an ssl_fail_verify_san event".
5. A stale netns entry. The pod is gone but its entry survives until the ghost sweep; the probe fails cleanly (a NETWORK health-check failure, not a crash, on the pinned snapshot). See the stale-netns section.

The gate is silently absent

inbound_gate_pods{aether_gate_state="ungated"} standing above zero with SPIRE enabled means those pods have no /healthz/inboundready_<namespace>_<pod> path and are being judged on the application probe alone. That is a safe degradation, not an outage — but it means the premature-promotion hole is open again, so it is worth an alert. Node E ran a whole agent lifetime like that on 2026-09-19 with no signal whatsoever; the agent now says which precondition is missing at startup and on every change, and the probes are reconciled on every snapshot rather than off one-shot triggers, so a missed event can no longer strand them.

After a proxy roll

A new Envoy epoch starts every host failed and must re-fetch each pod's SDS secret before the inbound probe can pass. Every pod on the node is then in the can't-tell row (application up, TLS unproven, already serving), so no endpoint is demoted — that is the rule, not a timing budget.

Two guards remain and still earn their place:

• livenessDemoteStreak (3 observations, 15 s). The proxy supervisor's two-epoch overlap keeps the gateway socket answering across a hot restart, so the agent may never observe an unreachable tick and never re-arm. The new epoch's application probe also starts FAILED for up to two ticks. The streak covers that, and absorbs a single transient failure of a TLS probe that has already passed this epoch.
• The gateway re-arm. A proxy container restart does take the socket away; the first tick that reaches the gateway again re-arms the warm-up grace and clears this epoch's tlsPassed marks for every pod.

A node-wide demotion wave aligned with a proxy roll is therefore a bug, not a tuning problem — capture aether_agent_liveness_health_transitions_total and the agent log and reopen #815.

Pod held in Init by aether-identity-ready (#1053)

The section above is the inbound half of "no traffic before identity": an endpoint is not advertised until an mTLS handshake with the pod's own inbound listener proves its SVID is loaded. Since #1053 the egress half is symmetric: the controller's pod-mutating webhook injects the aether-identity-ready init container, first in line, into every mesh pod it admits (controller.webhook.identityGate.enabled, default on; never with spire.enabled=false). It asks the SPIRE Workload API — a csi.spiffe.io volume mounted into the init container only — for the pod's X.509 SVID and exits 0 once SPIRE has issued it. Until then the app containers do not start, so the app cannot send before the node proxy has a client certificate for it.

Without it, a pod that sends in its first seconds gets 503 UF upstream_reset_before_response_started{connection_timeout} (and UO/URX once the pending queue overflows): the Broker API subscription is open, but SPIRE delivers the initial SVID only after its registration entry is created and synced to the node's spire-agent — processed SVID update svids=0 twice, then svids=1 update=initial 7.46 s after the subscribe on the reference cluster (2026-09-28, #1053). Why not block the CNI ADD instead: that makes SPIRE a hard dependency of sandbox creation, and the ADD races SPIRE's own pod-list attestation. Gating the app container keeps the sandbox SPIRE-independent.

Symptom of a gate that does not release: kubectl get pod shows Init:0/N (N counts the gate) and the pod never starts.

# What it is waiting for — one WARN line every 10s, with the socket and the last error
kubectl -n <ns> logs <pod> -c aether-identity-ready
#   level=WARN msg="still waiting for SPIRE to issue this pod's SVID; ..." socket=/run/secrets/workload-spiffe-uds/socket elapsed=40s attempts=78 last_error="rpc error: code = PermissionDenied desc = no identity issued"

# On release (normal: a few seconds after pod creation)
#   level=INFO msg="identity ready: SPIRE issued this pod's SVID; releasing the pod's containers" spiffe_id=spiffe://aether.internal/ns/<ns>/sa/<sa> elapsed=6.9s

Read last_error:

last_error	Meaning	Fix
PermissionDenied … no identity issued	spire-agent is up and attested the pod, but no registration entry matches it	Check the ClusterSPIFFEID podSelector/namespaceSelector covers the pod (kubectl get clusterspiffeid -o yaml); an entry keyed on k8s:container-name/k8s:container-image never matches the init container. Nothing matching also means the node agent's Broker API subscription gets nothing: the pod could not have spoken mTLS anyway
Unavailable … connect: no such file or directory / connection refused	No Workload API on the node	spire-agent not running on this node, or the SPIFFE CSI driver is not (kubectl get pods -A -o wide | grep -E "spire-agent|spiffe-csi"); a FailedMount … csi.spiffe.io event on the pod means the driver is missing entirely
DeadlineExceeded	The spire-agent accepted the call but did not answer within 15 s	spire-agent overloaded or wedged; check its log

It fails closed on purpose (no default timeout): a pod whose identity never comes cannot talk to the mesh, and Init with a reason is a better failure than Running with 503s. Escape hatches, narrowest first: annotate the pod aether.io/identity-gate: "false" (the webhook skips it; its first requests may fail until the SVID lands), set controller.webhook.identityGate.timeout (the init container exits 1 after that long and the kubelet retries it with backoff — or fails the pod if its restartPolicy is Never), or disable the gate chart-wide. The webhook is failurePolicy: Ignore, so pods admitted while no controller replica answers get no gate (and no mesh label) at all.

The Workload API answers as soon as the node's spire-agent holds the pod's entry; the node agent's Broker API stream receives the same entry from the same spire-agent cache. The proxy fetches the client certificate on demand at connect time (#842/#843), so a connect in the sub-second gap before the node agent publishes the SVID waits for it (within connect_timeout) rather than failing: the gate has no settle delay.

#815 release two: every pod event used to re-warm every cluster on the node

⚠ SUPERSEDED BY #842 — see "per-connection certificate selection" below. The transport_socket_matcher this section is about no longer exists, and with it the envoy_cluster_match_count_total queries here measure nothing. The history is still the right context for the re-warm behaviour and for the failure vocabulary, so it is kept; the queries and the stat names are stale. Do not build an alert from this section.

What changed. Every mesh service cluster carries a per-source upstream-mTLS transport_socket_matcher. Its exact_match_map used to be keyed by the source pod's netns path, which is unique per pod — so one CNI ADD or DEL rewrote a field of every mesh cluster on that node. Under delta-xDS each rewritten EDS cluster then re-warmed for the full 15 s EDS initial_fetch_timeout (delta sends no EDS request for a name the old cluster already watches, so no CLA arrives), and the warming→active swap destroyed the old ClusterEntry on every worker, which drainConnPools()-es every upstream pool on the node. The next request to every endpoint then paid a fresh TCP + mTLS handshake. Measured on the reference cluster (rev220 and again on rev224): one new pod → 20–24 clusters warming for 15 s; two new pods on one node → 30 clusters for 30 s; nodes with no new pod → 0. That handshake storm is the latency excursion after a service roll, and it cost an 8 h soak its largest error episode.

The map is now keyed by the source SPIFFE ID (aether.source.spiffe_id filter state; since #842 the matcher is gone and the certificate mapper reads the same ID from envoy.tls.certificate_mappers.on_demand_secret, and #1165 stopped stamping aether.source.spiffe_id), which is per ServiceAccount. The thing the matcher selects was always per-ServiceAccount — the transport_socket_matches entry name is the SPIFFE ID — so nothing about certificate selection changed; the key simply stopped carrying per-pod state.

What still re-warms, and what is now free

event	before	after
pod ADD/DEL, ServiceAccount already on the node	every cluster, 15 s each	nothing
rolling update, replacement lands on the same node	every cluster, twice	nothing
FIRST pod of a ServiceAccount arriving on a node	every cluster	every cluster, once
LAST pod of a ServiceAccount leaving a node	every cluster	every cluster, once
node SVID rotation (same SPIFFE ID)	nothing	nothing
registry change to the cluster itself	that cluster	that cluster

A scale-to-zero-and-back (the soak's SHRINK step) still costs one re-warm per direction per node that held a pod — not one per pod. Keeping a departed ServiceAccount's entry alive for a grace period was considered and rejected: the retained entry would name an SDS secret the agent stops serving when the pod's SVID subscription ends, and while Envoy's delta SDS keeps the last known secret on a removal (sds_api.cc ACKs and ignores it), a new Envoy epoch after a hot restart would not — every service cluster on the node would then warm for its SDS initial_fetch_timeout on every proxy roll and come up with upstream_context_secrets_not_ready, which is the same defect on a worse path. A retained entry is also unreachable by construction (no listener stamps a departed pod's identity), so it would buy byte-stability and nothing else.

Verifying the fix

The re-warm is invisible at a 60 s step — the proxy scrape is 5 s and the whole event lasts 15 s. Always max_over_time at a 5 s step:

# THE measurement. Add a pod to a ServiceAccount that already has one on the
# node: this must stay 0 on every node. Before the fix it read 20-33 for 15 s.
max_over_time(envoy_cluster_manager_warming_clusters[1m])

# The same event from the other side: cluster rebuilds per node.
increase(envoy_cluster_manager_cluster_modified[20m])

cluster_modified running at tens per node per 20 minutes against a handful of pod events is the pre-fix signature.

The failure signature: a connection took the on_no_match path

on_no_match presents the node identity. That is deliberate (some upstream connections legitimately have no source pod), so it is not an error — but a workload's traffic arriving as the node is the thing that says the filter-state key is not reaching the matcher.

Envoy emits one counter per transport-socket match, named by the match:

cluster.<cluster>.<match_name>.total_match_count
cluster.<cluster>.default.total_match_count

Every match on a mesh service cluster is named by a SPIFFE ID, so the chart lifts that into an attribute (stats_tags → aether.transport_socket_match, added with this release). Without it the SPIFFE ID lands in the exported metric name, one family per ServiceAccount, permanently in Prometheus' name index. Because the node identity is only reachable through on_no_match, the node identity's counter on a service cluster is the no-match counter:

# Connections that selected the NODE identity on a mesh service cluster.
#
# TWO traps, both hit during the rev225 validation (2026-09-19):
#  - the exported family is envoy_cluster_match_count_total — Prometheus' OTLP
#    ingest moves "total" to the end; envoy_cluster_total_match_count does not exist;
#  - on a node agent the "node identity" is the AGENT'S OWN SVID,
#    spiffe://<td>/ns/<agent-namespace>/sa/<agent-serviceaccount>
#    (spiffe://aether.internal/ns/aether-system/sa/aether-agent on the reference cluster) —
#    NOT spiffe://<td>/node/<node>. A `/node/` selector matches nothing and reads
#    as a clean zero. Take the value from the agent log line `served node SVID`.
#
# It is 0 over any quiet window and ticks only when a cluster's endpoints change
# (note 2): during a pod ADD the increments sit entirely on THAT service's cluster.
# Growing with traffic across clusters = workloads are presenting the agent's identity.
sum by (node, aether_cluster) (increase(envoy_cluster_match_count_total{
  aether_transport_socket_match="spiffe://aether.internal/ns/aether-system/sa/aether-agent"}[5m]))

# The healthy case, for contrast: per-ServiceAccount selection actually happening
# (every identity EXCEPT the agent's).
sum by (node, aether_transport_socket_match) (increase(envoy_cluster_match_count_total{
  aether_transport_socket_match!="spiffe://aether.internal/ns/aether-system/sa/aether-agent"}[5m]))

Three things about that counter, all confirmed against the pinned proxy's sources — get them wrong and the query lies:

1. default.total_match_count stays at zero and proves nothing. Envoy's matcher treats on_no_match as a match, so an on_no_match that names a socket increments that socket's named counter, never default. default is reached only when there is no on_no_match at all, or when the action names a socket absent from transport_socket_matches (which also logs Transport socket '<name>' not found, using default at warn — there is no stat for that misconfiguration).
2. There is a small constant offset. The counter increments once per upstream connection creation and once per HostImpl construction, and the construction call passes no filter state, so it lands on on_no_match. Expect the node-identity counter to tick up by roughly the endpoint count on every EDS change even when everything is correct. Judge it against the connection rate, not against zero.
3. It only counts connections at all because the matcher uses the filter-state input. Envoy re-resolves per connection only when usesFilterState() && !downstreamSharedFilterStateObjects().empty(). A cluster without the matcher (SPIRE off, or before the node SVID) resolves once per host and the counter is meaningless there.

The authoritative cross-check is on the receiving side — which verified identity did the destination actually see — and since #824 the access log carries it. Every other identity field in a line is something the control plane baked into the emitting pod's own config, so a line can say who it thinks it is; these two are the certificate the other end presented and this proxy validated:

field	on	is
downstream_peer_uri_san	reporter="destination" (the per-pod inbound listener)	the CALLER's verified SPIFFE ID
upstream_peer_uri_san	reporter="source" (outbound/capture)	the SERVER certificate the destination proxy presented

# Mesh traffic arriving as the node agent instead of a workload — the signature of
# the cluster matcher falling to on_no_match (#815). Expect ZERO rows.
log_name:"aether_access_logs" AND reporter:"destination"
  AND downstream_peer_uri_san:"spiffe://aether.internal/ns/aether-system/sa/aether-agent"

# The #638 cross-wiring from the client side: which server identity did we accept?
log_name:"aether_access_logs" AND reporter:"source" AND upstream_peer_uri_san:*

Both render - where the hop is not mTLS (SPIRE disabled, cleartext inbound, the loopback hop to the local application), so a - is not a finding on its own — check reporter and whether that hop is meant to be mTLS. For routes behind ext_authz the OPA decision log's metadataContext.filterMetadata["aether.source"].spiffeId remains available, but it is the source's claim about itself and a weaker control than the fields above. The failure signature is a workload's traffic arriving with the agent's identity. The agent's own #638 discriminator (agent/internal/xds/cache/identitybinding.go) still names the netns→identity index behind it and WARNs on outbound cluster bound to a foreign identity.

Why a wrong cert cannot leak across sources through pooling — now demonstrated, not derived (issue #831). Envoy folds a downstream filter-state object into the upstream connection-pool hash key only if the object implements Hashable; set_filter_state's envoy.string factory builds a Router::StringAccessorImpl, which does not, and a pool freezes the transport-socket options of whichever connection allocated it. The source SPIFFE ID therefore contributes zero bytes to the pool key. Node service clusters set connection_pool_per_downstream_connection: true (proxy.NewServiceCluster, NewTCPServiceCluster), which mixes the downstream connection id into the hash instead — one pool per downstream connection, so there is nothing to share.

//agent/test/mtlspool runs the real pinned proxy with two source ServiceAccounts on one node and measures the identity the destination verifies. With the flag on, each source is verified as itself on its own upstream connection. With it off — the only change — every request from the second source is verified as the first, multiplexed onto one HTTP/2 connection.

So this is security configuration, not a throughput knob. Do not turn it off while the matcher reads a non-hashable filter-state key: the result is a silent workload-to-workload authorization failure (both identities are valid mesh workloads, and the destination's validation context pins no client SAN, so it cannot object). The edge proxy sets it false only because it has exactly one identity. TestNodeProxyPerSourceClustersPoolPerDownstreamConnection (agent/internal/xds/cache/) fails the build if any per-source cluster loses the flag.

Since #831 the source access log also carries source_spiffe_id — the identity the control plane stamped for that hop. Joined to the destination line's downstream_peer_uri_san on x_request_id the two must be equal; a disagreement is either a matcher miss (the node/agent SVID appears at the destination) or a pooling leak. Both were previously invisible.

# The claimed source identity, per hop. "-" means the chain stamped none at
# all, i.e. the trust domain was still unknown when it was generated (#819).
log_name:"aether_access_logs" AND reporter:"source" AND source_spiffe_id:*

Long-lived connections across the upgrade

A connection accepted before its listener started stamping the key carries only the old key for its whole life, and its new upstream connections would take on_no_match. Release one changed the filters list on every mesh-originating filter chain, and Envoy's filter-chain reuse index hashes the whole FilterChain proto — so those chains were replaced and their connections drained. The drain is the server-wide --drain-time-s, which the supervisor sets from proxy.hotRestart.drainTime (10 s, not Envoy's 600 s default), after which the remaining connections are force-closed with filter_chain_is_being_removed. A proxy roll settles it outright. So on a cluster that has been through release one, no pre-release-one connection can still exist.

SPIRE off

No matcher is injected at all: the node SVID is only ever set by the SPIRE bridge, and without it every service cluster is emitted with a plain transport socket. Release two is a literal no-op there (TestServiceClusterBytesUnaffectedByPodChurnWithSpireOff).

The edge proxy and the east/west waypoint tunnel are unaffected — the edge has a single identity and takes the non-matcher branch, and the waypoint's two-level matcher only changed which key its inner sub-trees read.

#842: per-connection certificate selection (and the alert it breaks)

Read the #815 section above first — this replaces the mechanism it describes, and the two share a failure vocabulary.

What changed. A mesh service cluster no longer carries transport_socket_matches or a transport_socket_matcher. It carries one transport socket whose custom_tls_certificate_selector resolves the client certificate per connection:

envoy.tls.certificate_selectors.on_demand_secret
  config_source:      its OWN gRPC SDS stream to agent_xds  <-- NOT `ads: {}`
  certificate_mapper: envoy.tls.upstream_certificate_mappers.filter_state_override
                        default_value: <the node SVID>
  prefetch_secret_names: [<the node SVID>]

⚠ THE SELECTOR MUST NOT FETCH OVER ads: {}. THIS IS WHAT TOOK THE MESH DOWN. It shipped that way as rev228 on 2026-09-20 and every mesh upstream mTLS connection on the fleet stopped completing within seconds of the new proxies taking the config. Full account below under "The rev228 outage". If you are reading this because you are about to "tidy up" the one SDS reference on this proxy that is not ads: {} — don't. Read that section.

The mapper reads the filter-state object named — exactly, and not configurably — envoy.tls.certificate_mappers.on_demand_secret off TransportSocketOptions::downstreamSharedFilterStateObjects() and returns its string as the SDS secret name. Every mesh-originating listener chain stamps the source pod's SPIFFE ID there, and aether's SDS secret names are SPIFFE IDs, so the two join with no lookup table in between.

Two things follow, and they are the point of the change:

• A mesh cluster's bytes no longer depend on the node's workloads at all. #815 release two made them invariant under pod churn within a ServiceAccount; the first pod of a new ServiceAccount arriving, and the last one leaving, still rewrote every mesh cluster on the node and cost one 15 s EDS re-warm each. Those are free now. No pod event changes a service cluster.
• connection_pool_per_downstream_connection is gone from every mesh cluster. The object is stamped with the envoy.hashable_string factory, so the source identity reaches CommonUpstreamTransportSocketFactory::hashKey and upstream pools partition per (host, source identity) instead of per downstream connection. Pods sharing a ServiceAccount now share an upstream HTTP/2 connection; pods that do not, cannot.

⚠ THE FACTORY IS SECURITY CONFIGURATION. envoy.string builds a Router::StringAccessorImpl, which is not Envoy::Hashable; hashKey folds a shared filter-state object into the pool key only behind a dynamic_cast to Hashable. Reverting the factory while the flag stays off re-opens the

831 leak — a pooled connection carries another workload's client

certificate, the handshake succeeds, and the destination stamps the wrong identity into XFCC. It fails open. //agent/test/mtlspool reproduces it as a negative control and would go green-for-the-wrong-reason if the pair were broken; read its TestSharedPoolLeaksSourceIdentity before touching either.

⚠ max_session_keys must stay 0. A client context supports a custom certificate selector only with session resumption off (tls.proto): a cached session is keyed without reference to which on-demand certificate produced it, so a resumed one would carry a certificate the selector did not choose. #834 already set it to 0 for #829; it is now load-bearing for two reasons. Note the upstream factory does not enforce this the way the downstream one does — Envoy will accept a non-zero value here without complaint.

The rev228 outage: why the selector gets its own SDS stream

Symptom. Every mesh upstream mTLS connection stops completing, at the instant the proxies take the config. Inbound is fine, the local data path is fine, same-node upstreams fail too. Requests pause; they do not fail:

what you would normally check	what it said
envoy_cluster_ssl_connection_error_total	no new series
envoy_cluster_ssl_fail_verify_san_total	0 series
access log upstream_transport_failure_reason != "-"	0 records
access log, source reporter	upstream_host SELECTED, upstream_peer_uri_san="-", response_flags=DC, duration_ms≈1980

A green "no handshake errors" dashboard is not evidence of health here. The only signals that move are end-to-end ones, and the two that name the mechanism:

• envoy_cluster_on_demand_secret_cert_requested_total climbing while envoy_cluster_on_demand_secret_cert_updated_total stays absent. Absent means zero: Envoy does not export a counter that never incremented. cert_updated is bumped only by the completion callback of the selector's setContext, which is reachable only from a secret actually arriving. Zero therefore proves no SDS payload ever reached the selector.
• envoy_sds_spiffe_<mangled name>_init_fetch_timeout_total climbing, for the node SVID and for workload SVIDs. That is the 15 s initial_fetch_timeout on a subscription that never got an answer.

cert_requested == cert_active is NOT a health signal. cert_active counts live secret subscriptions, not applied certificates, and it is incremented on the same line as cert_requested. Equality just means nothing has been removed. On the fleet it read equal on four of five nodes while the mesh was entirely down. cert_updated is the only success signal the selector has.

Mechanism. Envoy keys a secret provider on hash(ConfigSource) + "." + name + warm (secret_manager_impl.h). The on-demand selector always asks with warm=false; every statically named SDS reference asks with warm=true. So one secret name gets two independent SdsApi objects with two independent watches — and on a node proxy that is the normal case, not an edge case:

• every local pod's SVID is already named statically by that pod's own inbound listener (DownstreamTransportSocket), and
• the node SVID — the selector's default_value and its prefetch_secret_names — is already named statically by every inboundready_<namespace>_<pod> probe cluster.

The selector is therefore always the second subscriber. On the delta-ADS mux, Envoy's WatchMap deduplicates subscription interest per (type_url, resource name) across watches, so the second watch contributes nothing to resource_names_subscribe. No request goes out. A delta control plane — correctly — sends only what changed, which is nothing. The second watch never receives the resource.

And nothing recovers it: SdsApi::onConfigUpdateFailed only calls init_target_.ready(). It does not notify the parked certificate-selection callback. So doSelectTlsContext's Pending is never resolved and the handshake stays suspended until the cluster's connect_timeout — or, sooner, until the downstream client gives up (DC).

Fix. The selector gets its own api_config_source to the agent_xds cluster, which means its own mux and its own watch map, so its subscription can no longer be deduplicated against a static one. It is state-of-the-world rather than delta on purpose: a SotW response carries every requested resource and GrpcMuxImpl::addWatch queues a request unconditionally, so the failure is structurally impossible on that stream even if a static reference ever lands on it. The agent already serves SecretDiscoveryService on the same socket as ADS (common/xds/xds.go), so there is no new bootstrap cluster.

What it costs. Envoy builds a mux per non-ADS subscription, so this is one gRPC stream per secret name the selector resolves — per identity actually originating traffic on the node, appearing lazily on first use — each carrying exactly one resource. max_concurrent_streams: 10 on the chart's agent_xds cluster does not cap them: on an upstream cluster that field is what Envoy advertises for peer-initiated streams. The governing limit is the agent's grpc.MaxConcurrentStreams(1000).

⚠ Perturbing the ConfigSource hash is NOT a fix. Adding an initial_fetch_timeout to ads: {} gives you a different provider key while leaving the mux — and therefore the watch map that does the deduplication — shared. The stream has to be separate.

⚠ Never point a statically named secret at the selector's stream. That puts a warm=true and a warm=false provider for one name back on one mux and resumes the starvation, silently. TestMeshCertSelectorSDSSourceIsNotSharedWithAnyStaticSecret (agent/internal/xds/proxy) fails the build if any builder does.

The gate. //agent/test/mtlspool's TestOnDemandCertificateResolvesWhenAlreadyStaticallyReferenced runs the pinned proxy against a real delta-ADS control plane with the source identity referenced both ways, and asserts the request completes within a bound. It was red against the shipped configuration for exactly this reason. The earlier harness could not have caught it: it rewrote every SDS config source to a bespoke api_config_source because it had no control plane, which accidentally was the fix.

Diagnosing it again, in order: cert_updated flat while cert_requested climbs → sds.<name>.init_fetch_timeout → /config_dump?resource=dynamic_warming_secrets (the starved names sit there with version_info: "uninitialized").

⚠ cert_requested and cert_active cannot be compared across a hot restart. Two live Envoy epochs both export; when the parent's series goes away the summed GAUGE (cert_active) drops to the child's alone while the COUNTER (cert_requested) keeps the merged total, because StatMerger transfers gauges absolute and counters as deltas. On rev228 that produced a node E reading of requested=55, active=28 that looked like 27 secret removals and was not. Check envoy_server_live and envoy_server_hot_restart_epoch before reading anything into the pair.

Rotation. An SVID rotation re-resolves a secret the selector already holds, and it must arrive on the selector's own stream. New upstream connections then present the new certificate; connections already established keep the one they handshook with, which is correct. SPIRE rotates on a ~4 h TTL, so a 60–75 minute deploy validation crosses no rotation while an 8 h soak crosses about two — i.e. a rotation defect would first appear as a soak going quiet several hours in, with no error anywhere. //agent/test/mtlspool's TestRotatedSVIDIsPickedUpOverTheSelectorStream is the build-time gate for it: it republishes every SVID under a new snapshot version and requires the destination to verify a different certificate SERIAL for the same SPIFFE ID within a bound, with cert_updated moving and every request completing throughout.

THE ALERT THIS BREAKS — cross-repo

AetherClusterIdentityNoMatch (the platform's GitOps repository) queries

sum by (node, aether_cluster) (increase(envoy_cluster_match_count_total{
  aether_transport_socket_match="spiffe://aether.internal/ns/aether-system/sa/aether-agent"}[5m]))

That series no longer exists. cluster.<c>.<match>.total_match_count is emitted per transport_socket_matches entry, and mesh clusters now have none. The rule does not error — it matches nothing and evaluates to a clean zero, so it looks permanently healthy. This is a vacuous gate and must be removed or repointed in the same window this ships, or the fleet loses its "workload traffic is presenting the agent identity" signal without anyone noticing.

The replacement, in order of strength:

1. The destination-side access log — already the authoritative check. It measures what the other end actually verified, not what this proxy intended:

# Mesh traffic arriving as the node agent instead of a workload. Expect ZERO.
log_name:"aether_access_logs" AND reporter:"destination"
  AND downstream_peer_uri_san:"spiffe://aether.internal/ns/aether-system/sa/aether-agent"

The runbook already called this "the authoritative cross-check" while the counter existed; it is now the primary.

1. The source-side access log's source_spiffe_id. Absent (-) is precisely the condition that makes the mapper fall back to default_value, so it is the direct successor to the no-match counter — and it is per request, not per connection. Since #1165 the attribute reads the mapper's own key rather than the aether.source.spiffe_id copy stamped beside it, so "absent here" and "the mapper missed" are the same lookup, not two that happened to be stamped together:

log_name:"aether_access_logs" AND reporter:"source" AND source_spiffe_id:"-"

1. cluster.<c>.on_demand_secret.cert_requested / .cert_updated / .cert_active (the selector's own stats, from ALL_CERT_SELECTION_STATS). These say the on-demand path is working — a cert_active gauge of 1 on a node with several ServiceAccounts sending traffic means only the default is ever being fetched. They do not identify which identity, so they are a liveness signal for the mechanism, not a replacement for (1) or (2).

The agent's own discriminator (agent/internal/xds/cache/identitybinding.go) is unchanged and still WARNs outbound cluster bound to a foreign identity on the control-plane side.

First-connection handshake pause

On-demand SDS pauses the handshake on the first connection per (secret name, worker thread, cluster) while the SDS response lands. The fetch is from the node-local agent over its ADS UDS with the secret already in the snapshot, so it is a local round trip.

The cluster itself does not warm on any of it, and that is a real change in the other direction. A statically referenced SDS certificate is fetched through a warming provider, whose init target fires only once the secret arrives — so a client certificate the agent failed to serve used to hold its cluster in warming for initial_fetch_timeout and then bring it up with upstream_context_secrets_not_ready. The on-demand path creates its providers with warm=false (SecretManagerImpl::DynamicSecretProviders::findOrCreate: "the warming target only fires after a secret is fetched, while non-warming one pre-fetches"), so no mesh cluster blocks on a client-certificate secret any more — including the prefetched node SVID. The cost moves from cluster warming, which is node-wide, to one paused handshake, which is not.

It is bounded and amortised: prefetch_secret_names carries the node SVID, so the no-filter-state path (health checks, anything the proxy originates itself) never pauses at all; a workload identity pays it once per cluster per worker, and the per-identity pool then keeps that connection alive. Set against what it replaces — connection_pool_per_downstream_connection forced a full mTLS handshake on every downstream connection, measured at ~25 ms of a 29 ms p50 hop (#735) — this is strictly cheaper.

Workload identities are deliberately NOT prefetched. Listing them would put the node's identity set back into every cluster's bytes, which is exactly the churn #815 and this change removed.

If a secret name is never served, the handshake stays paused rather than failing fast; the request then ends on its route timeout. That cannot happen for an identity the agent stamped (it stamps only identities it serves), but it is the shape to look for if on_demand_secret.cert_requested climbs without cert_updated following.

What is unchanged

• The server-identity SAN pin (match_typed_subject_alt_names) and the SDS-rotated trust bundle. They were never per-source.
• The inboundready_<namespace>_<pod> probe cluster. It has no selector: it keeps its own statically named certificate (the node SVID) and its own MaxSessionKeys: 0 (#836/#840). A health checker carries no filter state, so had it used the selector it would have taken default_value on every probe — the same node SVID, by a longer route. Leaving it alone also preserves the #836 boundary: nothing the probe does may be observable by application traffic.
• The edge proxy. One identity, no source filter state, and its SDS comes straight from the SPIRE agent rather than the agent's ADS stream. A selector would add a handshake pause and buy nothing, so EdgeUpstreamTransportSocket keeps its statically named certificate.
• The waypoint split (proposal 019, off by default). SNI is a property of the transport socket, not of the certificate, so a waypoint-enabled cluster still carries two sockets — but they are now a fixed two (local, waypoint) selected by endpoint metadata, independent of the node's identity set, and their match names are bounded constants rather than SPIFFE IDs.
• SPIRE off. No node SVID means no default_value, so no selector and no transport socket at all — the cluster is emitted bare exactly as before.

Rollback

For one release the listeners stamped both identity keys: aether.source.spiffe_id (what a pre-#842 cluster's matcher reads) and the certificate-mapper key, so a rollback to release-two clusters stayed hitless — the same dual-key overlap #815 used. #1165 retired aether.source.spiffe_id: the access log's source_spiffe_id attribute now reads the mapper key (the attribute NAME is unchanged), and a rollback below chart 0.93.0 is no longer hitless — see "Rollback floor: chart 0.93.0 (issue #1165)" in §7. The netns copy was kept (it still feeds source_netns).

Rolling forward to #842 replaced every mesh-originating filter chain (the filters list gained an entry), so those chains drained once on --drain-time-s (10 s here) — the same one-time cost release one paid; #1165 (one entry fewer) costs the same once more.