Getting Started with Aether¶

A practical guide to installing Aether and onboarding your first workload.

What is Aether? Aether is a Kubernetes service-mesh data plane written in Go. A per-node agent (DaemonSet) runs an Envoy xDS control plane and a chained CNI plugin; a per-node Envoy proxy (DaemonSet) carries the traffic; an in-cluster registrar tracks service endpoints; and a controller serves the admission webhooks. Every pod-to-pod hop is mTLS with SPIFFE identities — including same-node hops. Interception is transparent by default: managed pods' outbound traffic (TCP and UDP) is diverted into the mesh by the CNI, mesh names resolve via a per-node mesh DNS, and apps simply dial http://<svc>.<ns>.<meshDomain>:<port> (or the ordinary Kubernetes Service name), on a port the service declares. The mesh's HTTP proxy port (127.0.0.1:18081, with a Host header or as an HTTP proxy) remains available for clients that prefer zero interception assumptions.

1. How it works (the 2-minute model)¶

flowchart TB
    backend[("Registry backend<br/>(kubernetes / etcd)")]
    registrar["registrar (Deployment ×2)<br/>watches the backend,<br/>fans endpoint changes to agents"]
    backend <--> registrar

    subgraph node["per node"]
        agent["agent (DaemonSet)<br/>CNI plugin + xDS control plane (Unix socket)<br/>builds the Envoy snapshot<br/>(listeners / clusters / endpoints / routes)"]
        proxy["Envoy proxy (DaemonSet)<br/>capture + outbound listeners<br/>inside EACH managed pod's netns"]
        app["app pod"]
        agent -->|xDS snapshot| proxy
        app -->|"HTTP → my-svc.ns.aether.internal:8080<br/>(transparently captured; mesh DNS)"| proxy
    end

    agent -->|"register / deregister<br/>at CNI ADD / DEL"| registrar
    registrar -->|endpoint stream| agent
    proxy -->|"mTLS — SPIFFE identity per workload<br/>HTTP/3 over QUIC (HTTP/2 where QUIC does not apply)"| dest["destination pod<br/>(this or another node)"]

Identity = ServiceAccount, scoped by namespace. A pod's mesh service is its ServiceAccount name in its namespace (<ns>/<sa>); its SPIFFE ID is spiffe://<trust-domain>/ns/<namespace>/sa/<serviceaccount>. Pods sharing a ServiceAccount (in one namespace) are endpoints of the same service.
Transparent addressing (default). The CNI diverts managed pods' outbound TCP and UDP into the node proxy's per-pod capture listeners with a TPROXY-style mark-and-divert that leaves the IP header intact, so the original destination survives (proposal 038, #947): TCP lands on the capture listener, UDP on a per-VIP listener on 18082. DNS is still a plain DNAT: :53 goes to a per-node mesh DNS. Apps dial http://<svc>.<ns>.<meshDomain>:<port> — or the generated Kubernetes Service name <svc>.<ns>.svc.cluster.local:<port> — on a port the service declares, with no client changes. The proxy port (127.0.0.1:18081 + Host: <svc>.<ns>.<meshDomain>:<port>) works everywhere.
HTTP/3 between proxies. HTTP requests cross the mesh over HTTP/3 (mTLS over QUIC, UDP 18008) with nothing to enable; it needs two DNS SANs on every workload SVID (see HTTP/3 between proxies).
Demand-scoped. Each node only receives the config for the services its local pods actually call (declared up front, or fetched on first use).

2. Prerequisites¶

Requirement	Why	Notes
Kubernetes cluster	host	v1.30+ recommended (native `preStop.sleep` is used for hitless rolls).
A primary CNI (Calico, Cilium, flannel, …)	pod IP + connectivity	Aether installs a chained CNI plugin (`aether.conflist`) that appends itself to your existing CNI config. It does not replace your CNI.
SPIRE (SPIFFE runtime)	mTLS identity	Required when `spire.enabled=true` (the default). SPIRE >= 1.15.2 (`spiffe/spire` chart >= 0.30) — see the broker values below. The agent, registrar and controller consume the Workload API socket (via the `csi.spiffe.io` CSI driver), and the agent additionally brokers each pod's SVID over the SPIRE agent's SPIFFE Broker Endpoint socket. SPIRE pods must run in an ignored namespace (see §5) so they never depend on the mesh.
A registry backend	endpoint storage	Either `kubernetes` (default, no external dependency) or `etcd`.
Helm 3 with OCI support	install	Charts are published as OCI artifacts.
Privileged pod-security in `aether-system`	the agent, proxy, mesh-dns and uds-csi pods need `hostNetwork`, `hostPath` volumes and `NET_ADMIN`	On a cluster that enforces Pod Security admission, label the namespace before installing (step 2 of Install); otherwise those pods are refused.

You can run without mTLS for a quick kick-the-tires install by setting spire.enabled=false, which skips the SPIRE dependency entirely. Don't do that in production — it disables workload mTLS.

2a. SPIRE: enable the Broker Endpoint¶

BREAKING (proposal 036). Aether used to mint per-pod SVIDs through SPIRE's proprietary Delegated Identity API on the agent's admin socket, authorised by authorizedDelegates. It now uses the standard SPIFFE Broker API instead, and the delegated path is gone — there is no flag to put it back. Upgrading aether therefore requires the three SPIRE values below first; without them every pod's SDS certificate is missing and the node proxy cannot complete a single mTLS handshake. The change is additive on the SPIRE side, so apply it while the old aether is still running, then upgrade aether.

With the spiffe/spire chart (>= 0.30, SPIRE >= 1.15.2), three blocks under spire-agent:

spire-agent:
  # 1. Serve the Broker Endpoint on a UDS and expose it on the node, so the
  #    aether agent's DaemonSet can bind-mount it.
  sockets:
    broker:
      enabled: true
      mountOnHost: true

  # 2. Authorise the aether agent's own SPIFFE ID as a broker, limited to
  #    Kubernetes object references over UDS.
  brokerAPI:
    brokers:
      aether-agent:
        enabled: true
        idTemplate: "spiffe://{{ .TrustDomain }}/ns/aether-system/sa/aether-agent"
        allowedReferenceTypes:
          - typeURL: type.googleapis.com/spiffe.broker.KubernetesObjectReference
            allowOverTCP: false

  # 3. Let the Kubernetes workload attestor resolve those references.
  workloadAttestors:
    k8s:
      brokerAPI:
        accessPolicy: permissive
        brokers:
          aether-agent:
            enabled: true

Three things that are easy to get wrong:

Use the same broker key in both blocks (aether-agent above). The attestor block looks its idTemplate up from brokerAPI.brokers.<key>, so a mismatched key fails the chart render; a broker present in only one of them is rejected at runtime with broker "…" is not configured.
Set accessPolicy explicitly to permissive; do not leave it at auto. auto resolves to enforced for any allowed reference type other than WorkloadPIDReference — i.e. for the one aether uses. enforced makes SPIRE run a SubjectAccessReview (verb impersonate-via-spire) for every referenced pod, and the chart only renders the matching RBAC grant when the broker also sets workloadAttestors.k8s.brokerAPI.brokers.<key>.impersonation.clusterWidePodsOnly. auto on its own therefore fails closed: every pod gets PermissionDenied. Hardening to enforced means flipping both at once.
The idTemplate must name where aether actually runs — the agent's ServiceAccount, spiffe://<trust domain>/ns/<release namespace>/sa/<release>-aether-agent under a non-default release name. It is the client certificate the Broker Endpoint authorises, so a mismatch surfaces as Unavailable on every subscribe (the handshake is rejected before gRPC).

Aether's own defaults already point at where that chart puts the socket (/run/spire/agent/sockets/csi.spiffe.io/broker/broker.sock); override spire.brokerSocket.hostPath if your SPIRE install uses a different sockets.hostBasePath.

Recommended: shorten SPIRE's sync intervals. A new pod's SVID reaches the node only after its registration entry has been loaded into the SPIRE server's entry cache (cache_reload_interval) and pulled by the node's spire-agent (sync_interval); both default to 5 s. Every mesh pod waits that long in Init, because the controller injects the aether-identity-ready init container that holds the app containers until the SVID exists (#1053/#1055): the reference cluster measured the initial SVID 7.46 s after the subscribe with the defaults. With the spiffe/spire chart (0.30.2) both knobs sit behind experimental.enabled:

spire-agent:
  experimental:
    enabled: true
    syncInterval: 1s          # agent sync_interval (default 5s)
spire-server:
  experimental:
    enabled: true
    cacheReloadInterval: 1s   # server cache_reload_interval (default 5s)

That bounds the entry-to-node delay to about 2 s instead of about 10 s, at the cost of five times the agent→server sync RPCs and server cache reloads; size the server's datastore for it on a large cluster. experimental.enabled: true also renders the chart's other experimental.* values (requirePQKEM, featureFlags and, on the server, eventsBasedCache), so check they are what you want.

3. Install¶

Aether ships two charts. Install the CRDs first (they are standalone so they can be upgraded independently), then the system chart.

# Pick the published version. Every chart is published under a commit-pinned tag
# `<X.Y.Z>-<full git sha>` — use it, not the `aether` chart's bare `<X.Y.Z>` tag,
# which is mutable and re-pushed by every release (#692): every commit that
# keeps the version rewrites it, so its digest changes and the version does
# not. Anything that pins the chart (a lock file, a GitOps source) names the
# commit-pinned tag, or the digest alone, never the bare tag with a digest
# (#1588; docs/runbook.md, "Pinning a released chart").
COMMIT=<full 40-char git sha>
VERSION=<X.Y.Z>-$COMMIT   # each chart's Chart.yaml version at that commit

# 1) CRDs (MeshConfig, HTTPFilter, EdgeConfig, EndpointPolicy) — install/upgrade first.
helm upgrade --install aether-crds \
  oci://quay.io/aethermesh/chart-crds \
  --version "$VERSION"

# 2) FIRST INSTALL ONLY: the namespace, labelled for Pod Security admission. The
#    agent, proxy, mesh-dns and uds-csi pods need hostNetwork, hostPath volumes
#    and NET_ADMIN, which a cluster enforcing "baseline" or "restricted" refuses
#    in a namespace that does not say otherwise.
kubectl create namespace aether-system
kubectl label namespace aether-system \
  pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/audit=privileged \
  pod-security.kubernetes.io/warn=privileged

# 3) The system: agent + proxy + mesh-dns + registrar + controller. The same
#    command installs and upgrades.
helm upgrade --install aether \
  oci://quay.io/aethermesh/chart-aether \
  --version "$VERSION" \
  --namespace aether-system --create-namespace \
  --set clusterName=my-cluster \
  --set meshDomain=aether.internal

Who creates the namespace. Helm (--create-namespace) or you; the chart does not (namespace.create=false, the default since chart 2.4.21). A chart cannot create the namespace its own release is stored in: Helm writes the release record there before it creates anything the chart renders (#1403). Nothing labels the namespace for you, which is what step 2 is for. On a cluster that does not enforce Pod Security admission, step 3 alone is a complete install. If you skip step 2 on a cluster that does, the release still deploys but the four DaemonSets have no pods; the install notes say so, the refusals are kubectl -n aether-system get events --field-selector reason=FailedCreate, and running the kubectl label command afterwards (with --overwrite if the namespace already carries another level), then kubectl -n aether-system rollout restart daemonset, is enough: nothing has to be reinstalled (runbook.md, "Recovering from a failed first install").

namespace.create=true remains for a release stored in a different namespace (helm -n <other> ... --set namespace.create=true --set namespace.name=aether-system): the chart then renders the Namespace with those three labels. Asking for it in the release's own namespace fails the render with the two ways out. See namespace.create, and runbook.md, "Recovering from a failed first install" if an install went wrong (a retry after a very early failure may ask once for kubectl annotate namespace aether-system helm.sh/resource-policy=keep; it is safe to run).

Upgrading a release installed before chart 2.4.21. If it was installed with the old default (or namespace.create=true): nothing to do, and no value to set. Such a release owns its Namespace, and the chart keeps rendering a namespace its release owns whatever namespace.create says, so Helm never sees it leave the manifest and never deletes it. The upgrade also stamps helm.sh/resource-policy: keep on it; from then on helm uninstall leaves the namespace behind (delete it yourself). If it was installed with namespace.create=false: run kubectl annotate namespace aether-system helm.sh/resource-policy=keep once before the upgrade. The chart cannot tell such a release from one whose namespace lost every sign of its owner, so it refuses the first upgrade, naming that command, rather than risk Helm deleting the namespace. This relies on Helm reading the live namespace at render time (lookup). If you render without a cluster (helm template | kubectl apply --prune, or a GitOps tool that renders with helm template and prunes what left the render) the namespace does leave the render: protect it in your tool before the upgrade (or keep it in your own manifests). runbook.md, "Chart 2.4.21", has the check to run before and after.

Charts and images are published to the aethermesh organisation on quay.io: charts as quay.io/aethermesh/chart-<name>, images as quay.io/aethermesh/<component> (proposal 040).

Upgrading from a 0.x chart (published to ghcr.io). Every chart jumped to 1.0.0 with the move to quay.io, because the default image repositories changed. Point helm upgrade at the oci://quay.io/aethermesh/chart-<name> coordinates above with your existing values (helm get values <release> -n <ns> -o yaml > values.yaml, then -f values.yaml — not --reuse-values, which would carry the old chart's defaults along). If you mirror images and override repository by prefix, mirror from quay.io/aethermesh/<component> and override each image's repository individually. Releases published before the move went to ghcr.io; nothing publishes to or verifies against ghcr.io any more (proposal 040 phase 4).

Resource names derive from the release name — installing as aether yields aether-agent, aether-mesh-dns, aether-registrar, aether-controller. The node proxy is the exception: its DaemonSet is aether-proxy whatever the release is called. Install the chart once per cluster; a second release is refused where the chart can see the first (configuration.md, "One release per cluster").

Upgrade tip: always pass the full values on every helm upgrade of the aether chart — do not use --reuse-values (it keeps the stale digest-pinned image). Bump the chart version per change.

Verify:

# What actually landed — the chart's appVersion is the commit that built it.
# Never skip this: a chart tag can serve a different build than you asked for
# (#692), and this is the only check that catches it.
got="$(helm get metadata -n aether-system aether -o json | jq -r .appVersion)"
case "$got" in
  *"$COMMIT"*) echo "deployed $got — OK" ;;
  *) echo "DEPLOY MISMATCH: appVersion=$got, expected $COMMIT" >&2; exit 1 ;;
esac

kubectl -n aether-system get pods         # agent + proxy + mesh-dns + uds-csi per node, registrar ×2, controller
kubectl -n aether-system get meshconfig   # the seeded "default" MeshConfig CR

# Every DaemonSet pod, on every node. This is the check, not `helm --wait`.
for ds in $(kubectl -n aether-system get daemonset -o name); do
  kubectl -n aether-system rollout status "$ds" --timeout=5m
done

helm upgrade --install --wait returning deployed does not show that the DaemonSet pods run (#1473). Helm counts a DaemonSet as ready once numberReady >= desiredNumberScheduled - maxUnavailable. For a DaemonSet that may have one pod unavailable during a roll (the agent with its default strategy, uds-csi, and the prober chart's DaemonSet) that is zero pods on a one-node cluster, and one pod short on any other. #1473 reports it on kind with Helm 3.18.4: deployed for the prober chart with no pod running (as happens when Pod Security admission refuses the pods). kubectl rollout status waits until every scheduled pod is updated and available.

4. Configure¶

Configuration splits into two layers:

4a. Deploy-time system config (chart values)¶

Set once at the top of the aether chart's values; every component inherits it.

Value	Default	What it does
`clusterName`	none: required	This cluster's name in the mesh: `--cluster-name` of the agent, the registrar and the edge control plane, stamped on every endpoint and part of the registry's keys. The chart refuses to render without it (since chart 2.6.0). Choose it once and keep it; it must be unique among the clusters that share a registry.
`meshDomain`	`aether.internal`	The authority suffix services are addressed under (`<svc>.<ns>.<meshDomain>`). Also defaults the SPIRE trust domain.
`spire.enabled`	`true`	Mesh-wide mTLS switch.
`spire.workloadSocketPath`	`/run/secrets/workload-spiffe-uds/socket`	Workload API socket. Every component resolves its own trust domain from its SVID over this socket — there is no `trustDomain` value to configure (retired: it could silently disagree with what SPIRE issues).
`spire.brokerSocket.*`	see values	SPIRE agent SPIFFE Broker Endpoint socket the agent brokers each pod's SVID over (proposal 036).
`registrar.registryBackend`	`kubernetes`	`kubernetes` \| `etcd`.
`registrar.etcd.endpoints`	`[]`	etcd endpoints when backend is `etcd`.
`otel.enabled` / `otel.endpoint`	`false` / `""`	Turn on OTel and point at an OTLP gRPC collector (`host:port`).
`proxy.enabled`	`true`	Deploy the per-node Envoy. Set false to run only the agent.
`edge.enabled`	`false`	Deploy the north-south ingress gateway (see §11).

4b. Runtime proxy observability (the `MeshConfig` CR)¶

MeshConfig is namespaced: the controller seeds one named default in the control-plane namespace on first install and never overwrites it on upgrade — you own it via kubectl thereafter. Every workload namespace inherits it field-by-field unless it defines its own MeshConfig (at most one per namespace). It only overrides the proxy data plane's observability (access logs, tracing, per-pod stats); everything else stays inherited from the system config.

kubectl -n aether-system edit meshconfig default   # mesh-wide baseline
kubectl -n my-namespace apply -f my-meshconfig.yaml  # per-namespace override

apiVersion: config.aether.io/v1
kind: MeshConfig
spec:
  proxy:
    accessLogsEnabled: true
    accessLogSuccessSampleRate: 100   # % of successful requests logged
    tracingEnabled: true
    traceSampleRate: 0.01             # head sampling, 0.0–1.0
    emitStatsPod: false

Changes are projected into a ConfigMap the agent mounts and applied to the data plane without a redeploy.

5. Which pods are managed (and which are never)¶

Opt in with a label¶

A pod joins the mesh only if it carries:

metadata:
  labels:
    aether.io/managed: "true"

The CNI plugin programs the outbound listener (and mTLS identity) into a pod's network namespace at CNI ADD time only for labelled pods. A pod also needs at least one IP. Everything else on the node is left untouched.

Always-ignored namespaces¶

These namespaces are never intercepted, regardless of labels — the control plane and the mesh's own dependencies must never rely on the mesh to start up (otherwise the mesh and SPIRE deadlock each other):

kube-system
aether-system
spire-mgmt
spire-server
spire-system

(Matching is case-insensitive.) So a aether.io/managed: "true" pod in kube-system is still ignored.

Opt out¶

To exclude a specific pod, simply omit the aether.io/managed label (or set it to anything other than "true"). There is no force-include for ignored namespaces.

6. Onboard a workload¶

A minimal mesh-managed Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-svc
spec:
  replicas: 4
  minReadySeconds: 10                         # see §8
  strategy:
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
  template:
    metadata:
      labels:
        app: my-svc
        aether.io/managed: "true"             # ← opt into the mesh
    spec:
      serviceAccountName: my-svc              # ← SERVICE NAME = ServiceAccount
      containers:
        - name: app
          readinessProbe:                     # gates endpoint promotion
            httpGet: { path: /healthz, port: 8080 }
          lifecycle:
            preStop: { sleep: { seconds: 10 } }   # see §8

Your service is now addressable mesh-wide as my-svc.<namespace>.aether.internal (and via the generated my-svc.<namespace>.svc.cluster.local Service).

Optional endpoint annotations¶

Annotation	Default	Meaning
`endpoint.aether.io/port`	`8080`	Application port the mesh routes to.
`endpoint.aether.io/weight`	`1024`	Load-balancing weight.
`endpoint.aether.io/health-path`	`/`	Path the node-local agent health-checks (delegated liveness).
`endpoint.aether.io/health-check-mode`	`eds`	`eds` = agent vets the endpoint and publishes health over EDS (clients get pre-warmed endpoints); `active` = every client proxy probes the endpoint itself.
`metadata.endpoint.aether.io/<key>`	—	Free-form endpoint metadata, usable as routing subsets (§9).
`endpoint.aether.io/uds-socket`	—	Serve on a Unix socket (`<volume>/<file>`) instead of a TCP port. `<volume>` is the pod's inline `csi: {driver: csi.aether.io}` volume and the pod sets `securityContext.fsGroup` (an `emptyDir` is no longer a carrier since chart 2.0.0) — see workload-requirements.md.
`config.aether.io/upstreams`	—	Comma-separated services this pod calls (§7).

7. Call other services¶

With transparent capture + mesh DNS (both on by default), just dial the destination by name — no client changes, no special listener:

# from inside a managed pod — either form works. 8080 is a port the
# service's pods declare (endpoint.aether.io/port or endpoint.aether.io/ports).
curl http://svc-payments.payments.aether.internal:8080/orders
curl http://svc-payments.payments.svc.cluster.local:8080/orders

Mesh FQDNs are namespace-qualified: <service>.<namespace>.<meshDomain> (proposal 020). The registrar also generates a selectorless Kubernetes Service per mesh service, so the ordinary <service>.<namespace>.svc.cluster.local name is captured too. The generated Service exposes the ports the service's pods declare, each under its own number; a service is addressed on one of them. A port it does not declare is refused with a 421 that names its ports.
With redirect-all (the default, proposal 022) all outbound TCP from a managed pod is captured: mesh destinations are routed by the mesh, and anything else (external APIs, non-mesh Services) passes through untouched. Carve-outs are per-pod annotations (capture.aether.io/*, see the configuration reference).
The callee sees the caller's SPIFFE ID in x-forwarded-client-cert (XFCC).

Using the proxy port¶

127.0.0.1:18081 in every managed pod is the mesh's HTTP proxy port. Clients that prefer zero interception assumptions send it the request with the service's mesh name and port in Host:

curl http://127.0.0.1:18081/orders -H 'Host: svc-payments.payments.aether.internal:8080'

It also works as an ordinary HTTP proxy: the request line's authority decides where the request goes.

curl --proxy http://127.0.0.1:18081 http://svc-payments.payments.aether.internal:8080/orders

# or for a whole process (curl reads the lower-case name only; set both)
export HTTP_PROXY=http://127.0.0.1:18081 http_proxy=http://127.0.0.1:18081
export NO_PROXY=localhost,127.0.0.1,.svc,.cluster.local   # whatever must not go to the mesh
export no_proxy="$NO_PROXY"

It carries HTTP to mesh services only, by their mesh name or their Kubernetes names. A name outside the mesh answers 404, and CONNECT (what a client sends for an https:// URL through an HTTP proxy) answers 404: do not set HTTPS_PROXY to it.
The port in the authority is the service's declared port. 18081 itself is not a port of any service.

The authority names the service and the port: <service>.<namespace>.<meshDomain>:<port>, with a port the pods declare. The port is never stripped, and a name without one is port 80. The Kubernetes names of the service's Service (<service>.<namespace>.svc.cluster.local, <service>.<namespace>.svc, <service>.<namespace>) are answered exactly as the mesh name, with the Service's port (since chart 3.0.3). A name without a namespace is the service of that name in the caller's own namespace. A name that is not a mesh service answers 404 here (on a captured dial it passes through to the address dialled), and a port the service does not serve as HTTP answers 421 with an x-aether-error header that says why.

Declare your upstreams (recommended)¶

The mesh only distributes a service's config to nodes that need it. Declare what a pod calls so those clusters are warm before first use:

metadata:
  annotations:
    config.aether.io/upstreams: "svc-payments,svc-ledger,svc-audit"

Declared upstreams are present the moment the pod lands — declare anything latency- or correctness-critical.
Undeclared upstreams still work via the cold path (ODCDS): the first request pauses ~one node-local xDS round-trip while the cluster is fetched on demand, then stays warm (1h idle TTL). Each miss increments aether.agent.upstreams.miss — the signal to promote it to the annotation.
A pod's own service is always in scope; never declare it.

HTTP/3 between proxies (east-west QUIC)¶

Between the caller's node proxy and the destination pod, HTTP requests ride HTTP/3 over QUIC, still mTLS with the caller's own SVID. UDP itself (a UDPRoute, §10) is carried in plaintext.

East-west QUIC (proposal 038) is unconditional — nothing to enable, one mesh-wide SPIRE requirement. Every service is dialled over HTTP/3 (mTLS over QUIC, UDP 18008) by every caller, so the workloads' ClusterSPIFFEID must issue two DNS SANs per SVID, <sa>.<ns>.<mesh domain> and *.<sa>.<ns>.<mesh domain> (dnsNameTemplates: ["{{ .PodSpec.ServiceAccountName }}.{{ .PodMeta.Namespace }}.<mesh domain>", "*.{{ .PodSpec.ServiceAccountName }}.{{ .PodMeta.Namespace }}.<mesh domain>"]; on the spiffe/spire chart: spire-server.controllerManager.identities.clusterSPIFFEIDs.default.dnsNameTemplates). Envoy's QUIC client checks the SNI against the leaf's DNS SANs after the SPIFFE pin (aether#957); the requirement stands until envoyproxy/envoy#47740 is in a plain proxy pin. Also open UDP:18008 wherever TCP:18008 is allowed. e2e/l4routes.sh shows the SPIRE shape; e2e/eastwest-quic.sh is the end-to-end proof (per-source HTTP/3 with the caller's own identity in XFCC, and a QUIC_DNS_SANS=off negative control), and the runbook's East-west QUIC section has the budget and the escape hatch. Two cases stay on h2 by design: a GAMMA rule with a weighted split (#961; a rule whose single backendRef is the parent Service rides QUIC like the default route), and any service with an endpoint behind the east/west waypoint (019), whose tunnel has no QUIC leg.

Use keep-alive / HTTP-2 connections¶

The mesh pools upstream mTLS connections per downstream connection (so one pod's certificate is never reused for another pod's traffic). A long-lived client connection — HTTP/1.1 keep-alive or an HTTP/2/gRPC channel — reuses its mTLS connection across requests. Connection-per-request clients pay a fresh handshake every time: it works, but it's the expensive shape.

8. Roll without dropping requests¶

The mesh does most of the work (endpoints go DRAINING the instant deletion is requested, new endpoints arrive pre-vetted, connection-level failures retry on another endpoint). Two workload settings close the rest — without them, rolls outrun the mesh and drop requests:

minReadySeconds: 10 — paces the roll so the old endpoint is retired only after the replacement is actually mesh-routable (~5–10s after Ready).
preStop: { sleep: { seconds: 10 } } — delays SIGTERM so in-flight requests finish through the two-phase drain. Measured: sleep 10 → 0 failed requests/roll; sleep 3 (the supported minimum) → ~1 blip per pod.

Also keep maxUnavailable: 0 and a real readinessProbe (endpoint promotion gates on the app actually answering).

Avoid --grace-period=0 force deletes for serving workloads — they skip the draining phase, so brief errors are possible.

What the mesh retries for you¶

On a different endpoint (2 retries, 25–250 ms backoff): connect-failure, refused-stream, reset-before-request, and 503. A request that never reached your app is retried whatever its method. One your app received and did not answer (it closed the connection) is retried only when the method is idempotent (GET, HEAD, OPTIONS, TRACE, PUT, DELETE); a POST gets the 503. Other application 5xx and timeouts are deliberately not retried. See What the mesh retries for you.

9. Advanced routing (pinning, subsets, locality)¶

Pin to one endpoint¶

Header	Meaning
`x-aether-ip: 10.42.1.11`	route to exactly that endpoint IP
`x-aether-pod: my-svc-7f9c4-xv2qp`	route to exactly that pod

Pin-or-fail: if the target is gone, you get a 503 — never a silent fallback.

Provider-defined subsets¶

Endpoints publish routing dimensions; consumers select them:

# on the endpoint (producer)
metadata:
  annotations:
    metadata.endpoint.aether.io/version: "v2"

# on the request (consumer)
curl http://127.0.0.1:18081/ -H 'Host: my-svc.my-namespace.aether.internal:8080' \
     -H 'x-aether-subset-version: v2'

Selection is strict (no fallback): asking for a subset with no endpoints fails rather than spilling onto the rest of the service. Up to 4 subset keys intersect (...-version: v2 + ...-shard: s1 → endpoints matching both). Keys ip/pod/cluster/namespace are reserved.

Locality-aware failover¶

Endpoints carry their node's topology.kubernetes.io/region and zone. Each node's proxy prefers same-zone endpoints (priority 0), spilling to same-region (1) then anywhere (2) only as closer endpoints drain or fail. Nodes without topology labels express no preference.

10. Shape traffic with Gateway API (GAMMA)¶

East-west traffic management uses the standard Gateway API route types parented to a Service (the GAMMA pattern) — no custom routing CRDs. It is on by default (agent.gamma, a kill switch): the agent detects the Gateway API CRDs and simply warns when they are absent. L4 route types are likewise always on, gated only on their CRDs being installed.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-svc-canary
  namespace: payments
spec:
  parentRefs:
    - kind: Service            # ← GAMMA: parented to the SERVICE, not a Gateway
      name: my-svc
  rules:
    - backendRefs:
        - { kind: Service, name: my-svc,    port: 8080, weight: 90 }
        - { kind: Service, name: my-svc-v2, port: 8080, weight: 10 }

What's supported, all applied transparently on the capture path as well as the explicit listener (so callers need no changes):

HTTPRoute — weighted canary, header/path matching + mutation, redirects, URL rewrites, timeouts; Mesh conformance profile (MESH-HTTP Core) green. (Request mirroring is not implemented.)
GRPCRoute — method-based routing (<svc>/<method> matched as a path).
TCPRoute — weighted L4 splits across backends; an explicit weight: 0 drains a backend. (TCPRoute/UDPRoute need the Gateway API experimental channel CRDs.)
TLSRoute — SNI-based passthrough routing.
UDPRoute — a udp_proxy floor to the backend's app UDP port. Clients dial <svc>.<ns>.<domain>:18082/udp — the same L4 spelling raw TCP uses (proposal 038). Note that UDP rides the mesh in plaintext: mTLS is a TCP/TLS construct and DTLS is not implemented. One limit worth knowing before you write one (#873): a split is not expressible — udp_proxy's route action carries a single cluster, so a multi-backend UDPRoute binds the heaviest backend and discards the split (weight: 0 is honoured as a drain). It is logged and counted on aether.agent.l4route.udp_unsupported. A pod may reach any number of UDPRoute-backed services: capture is transparent, so the datagram's original ClusterIP selects the route.
Cross-namespace backends need a standard ReferenceGrant.
The parent must be a registry-backed mesh Service (one with live endpoints) — a selectorless "anchor" Service won't survive reconciliation.

For behavior the route types can't express, the HTTPFilter CRD (proposal 025) attaches a vetted set of raw Envoy HTTP filters — ext_authz (pairs with the node-local authz sidecar + OPA preset, proposal 027), RBAC (peer-SAN allow/deny), header-to-metadata — per route (Gateway API ExtensionRef), per Service (targetRefs), service-wide (SCOPE_CHAIN), or on the destination's inbound (SCOPE_INBOUND). Validation is fail-closed via the controller webhook.

11. North-south ingress (the edge gateway)¶

The optional edge is an unprivileged Deployment (its own aether-ingress namespace by default) running Envoy + an agent edge sidecar. It dials mesh pods directly over mTLS (the node DaemonSet is not in its path) and routes external traffic via the Gateway API (proposal 018): the edge serves the HTTPRoutes attached to Gateways of its GatewayClass (gateway.aether.io/edge controller). The retired EdgeRoute / VirtualHost CRDs are gone — Gateway API is the edge's only routing API. Requires the standard Gateway API CRDs installed in the cluster.

Enable it:

helm upgrade --install aether oci://quay.io/aethermesh/chart-aether \
  --version "$VERSION" -n aether-system \
  --set clusterName=my-cluster \
  --set edge.enabled=true
  # ... plus your other values

The chart renders the GatewayClass and — by default (edge.gateway.create=true) — a Gateway of that class. Expose a service by attaching an HTTPRoute to it (the chart can manage routes declaratively via edge.gateway.httpRoutes, or apply your own):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api
  namespace: aether-ingress          # the namespace the edge watches
spec:
  parentRefs:
    - name: aether-edge              # the chart-managed Gateway
  hostnames: ["api.example.com"]     # external Host/SNI
  rules:
    - backendRefs:
        - name: svc-1                # the mesh service to route to
          port: 8080

The edge routes only by explicit external hostname on the attached routes. The internal mesh FQDN (<svc>.<ns>.<meshDomain>) is deliberately not routable from the edge.
TLS: set edge.tls.enabled=true and reference a kubernetes.io/tls Secret per Gateway listener (spec.listeners[].tls.certificateRefs, or the chart's edge.gateway.tlsSecretName). The edge agent serves the cert to Envoy over SDS (SNI-selected, hot rotation, no pod roll) and 301-redirects HTTP→HTTPS. Aether does not issue certs — bring your own (cert-manager, kubectl, …).
Addressing: per-Gateway addressing is unconditional (proposal 021 Phase 2; the flag and chart value were retired in 031 round 2) — each Gateway gets its own LoadBalancer Service and external IP; pin one with edge.gateway.address. To label or annotate that Service (for example with what your load-balancer controller reads), set spec.infrastructure.labels / .annotations on the Gateway; the edge copies them to the Service and removes them again when they leave the Gateway. Keys under aether.io and gateway.networking.k8s.io, and the address-pin annotation, are refused: see docs/configuration.md, "Labelling and annotating a Gateway's LoadBalancer Service".
HTTP/3: set edge.config.http3.enabled=true (default off) to add a QUIC listener on the HTTPS port and advertise it with alt-svc (proposal 029); it can also be set per Gateway through an EdgeConfig attached with parametersRef.
GeoIP: set edge.geoip.enabled=true with a MaxMind mmdb Secret to emit x-geo-* request headers (proposal 028).
The edge gets its own SVID straight from SPIRE; with spire.enabled the chart can create its ClusterSPIFFEID (edge.spire.clusterSpiffeID).

12. Multi-cluster and multi-region¶

Multi-cluster is layered: each plane is independent and opt-in, and they compose into full cross-cluster routing. All of it requires the etcd registry backend (the cross-cluster plane) and, for any cross-cluster data path, one SPIFFE trust domain shared across clusters (e.g. SPIRE with a shared upstream CA) so identities verify across the cluster line.

Plane	What it does	Enable
Endpoints (MCS)	`ServiceExport` publishes a service to the clusterset; every cluster materializes a `ServiceImport` + clusterset VIP and sees the remote endpoints.	`registrar.enableMCS=true` (+ the MCS-API CRDs)
Config (026)	Exported services' GAMMA routes (`HTTPRoute`/`GRPCRoute`, incl. `HTTPFilter` chain scope) propagate to peer clusters and merge into their proxies (local config wins).	exporter: `registrar.enableMCS=true`; importer: `agent.importConfig=true`. Optionally pin one authoritative exporter with `controlCluster=<name>`
Data path (019)	When pod IPs aren't routable across clusters (the usual case), cross-cluster endpoints are dialed at their node's routable IP on the fixed east/west tunnel port (`18009`); the destination node's proxy SNI-forwards to the local pod. mTLS stays end-to-end pod↔pod. Intra-cluster traffic is unaffected.	`agent.eastWestWaypoint=true` on all clusters (the tunnel port is the fixed constant `18009`)
Multi-region (006)	One etcd per region (`registrar.region`); each region's registrar mirrors its own registry subtree into peer regions' etcds under an origin-heartbeat lease — a dead region's mirror expires everywhere at the lease TTL (~30s), and locality keeps foreign endpoints on the failover path (EDS priority 2).	`registrar.region=<region>` + `registrar.peerEtcd=["<peer>=<endpoints>"]` per direction

Minimal two-cluster recipe (shared regional etcd, cross-cluster service echo living only in cluster b, called from cluster a):

# both clusters: etcd backend + waypoint + a shared trust domain
--set registrar.registryBackend=etcd \
--set "registrar.etcd.endpoints[0]=http://<shared-etcd>:2379" \
--set registrar.enableMCS=true \
--set agent.eastWestWaypoint=true \
--set clusterName=<unique-per-cluster>

Then kubectl apply a ServiceExport for echo in cluster b; a client in a calls http://echo.<ns>.<meshDomain>:<port>/ and the request rides mesh DNS → capture → split-horizon EDS → b's node tunnel → the echo pod, mTLS end-to-end. (The reusable e2e harnesses under e2e/ — multicluster_config.sh, multicluster_waypoint.sh, multicluster_replicator.sh — stand up exactly these topologies with kind.)

13. Observability¶

Turn on OTel once at the system level:

otel:
  enabled: true
  endpoint: "otel-collector.observability.svc:4317"   # OTLP gRPC
  logs: true            # component logs over OTLP (also tee'd to stderr)
  traceExport: true     # spans (needs a collector traces pipeline)
  traceSampleRate: 0.1

The collector endpoint is deploy-time config: the chart passes it to each Go component as a flag and renders it into the Envoy bootstrap. It is not read from a runtime ConfigMap. The CNI plugin takes no endpoint: it exports no telemetry of its own and forwards its timings to the agent, which exports them (#1166/#1185).
Proxy access logs / tracing / per-pod stats are retunable at runtime via the MeshConfig CR (§4b) without a redeploy.
Control-plane metrics, proxy stats, and a per-request source→destination request counter all flow over OTLP. Health-check traffic is excluded from access logs.

14. Troubleshooting¶

Symptom	Likely cause	Check
App can't reach the mesh (connection refused to `127.0.0.1:18081`)	Pod isn't managed, or the proxy hasn't programmed the listener yet	Confirm `aether.io/managed: "true"` and that the pod is not in an ignored namespace (§5); check the agent log for `adding listeners for pod`.
Request 404s immediately	The authority names no mesh service	Use `<service>.<namespace>.<meshDomain>:<port>` (or the k8s Service name). A name without a namespace is looked up in the caller's own namespace only.
Request answers `421`	The authority names a mesh service and a port it does not serve as HTTP	Read the `x-aether-error` response header (`port-not-declared`, `port-not-http`, `port-required`, `port-mismatch`, `no-http-port`); the body lists the service's ports. A name without a port is port 80. See the runbook, "The four refusals".
Request 503s on a pin	The pinned endpoint is gone (drained/ejected/never existed)	Pin-or-fail is by design; drop the `x-aether-ip`/`x-aether-pod` header to load-balance.
First call to a service is slow	Cold path (undeclared upstream) — one xDS round-trip	Declare it in `config.aether.io/upstreams`; watch `aether.agent.upstreams.miss`.
Dropped requests during a roll	Missing `minReadySeconds` / `preStop` / `maxUnavailable: 0`	Apply all three (§8); avoid `--grace-period=0`.
Mesh + SPIRE won't start	SPIRE running in a mesh-managed namespace	SPIRE must live in an ignored namespace (`spire-*`) so it doesn't depend on the mesh.

Useful commands¶

kubectl -n aether-system logs ds/aether-agent           # control plane + CNI
kubectl -n aether-system logs ds/aether-proxy           # the Envoy data plane
kubectl -n aether-system get meshconfig default -o yaml  # effective proxy observability
kubectl get httproutes -A                               # north-south + GAMMA routes