Skip to content

Proposals

Every design decision in aether is written down before it is built, and each record is published as it was written. The status is the proposal's own wording; the dot beside it is that wording sorted into five buckets, nothing more.

# Proposal Status Date
044 Meshed pods reach non-mesh Services without kube-proxy Proposed. Phase 0 is measurement, and every later phase waits on what it shows. The facts below were read from main at bb1d1625. Phase 0 progress: the agent learns the Service range (chart 3.0.4); the capture listener's Service-range branch to cap_passthrough_svc, the L4 split, is built (chart 3.0.6). The cleartext-HTTP split is not. 2026-10-15
043 A mesh service is addressed by its real ports, and nothing else Accepted 2026-10-11 by the owner, with all twelve decisions taken as recommended (see Decisions), and built as one breaking release, chart 3.0.0 (decided 2026-10-12). The four phases this document planned over charts 2.7 to 2.10 were collapsed: see The release for what shipped, what the plan had that did not, and Rulings after acceptance for the decisions taken while it was built, which hold over the older text. The facts were read from main at 2a132053; the measurements were made on the pinned proxy outside any cluster. 2026-10-11
042 sortie as the soak's load driver — k6 retired Implemented. sortie is the soak's load driver and k6 is retired; the harness is now maintained outside this repository, so runs later than the ones below are not recorded here. The harness was built and proven on kind 2026-10-06 (#1339) against sortie b71b37e, and moved to sortie 94cf103 on 2026-10-07, which fixed every defect the first version had to work around ("What sortie 94cf103 retired"). One new sortie defect was found on the way: it waits without limit for an engine whose node has gone silent ("Risks", 5). Phase 1, the short e2e on the reference cluster, ran twice on 2026-10-07 ("Verified on the reference cluster"): the driver works there, and it needs a client queue, which the plan now carries by default ("The client queue"). The harness then moved to sortie 9fcbb81 (2026-10-07, "What sortie 9fcbb81 changed"): that defect is fixed, a request still in flight when a run ends is counted, a stage that was not run is marked as such, the report carries latency and timestamps, and each execution's result is also written as it finishes. With 9fcbb81 the short e2e passed on the reference cluster (run 4, 2026-10-08, 8 of 8) and the first 8-hour soak ran the same day: the sortie gate passed with 9,180,000 of 9,180,000 requests answered 2xx under 35 rollouts ("Verified on the reference cluster"). The harness then moved to sortie f0750ec (2026-10-08, "What sortie f0750ec changed"): a stage refused at the execution cap is marked by a field, and a driver that wakes past a backend's deadline takes the answer that is waiting. Proven on kind only. The harness then moved to sortie 96e6bfb (2026-10-08, "What sortie 96e6bfb changed"): nothing the harness reads or passes changed; the engine has a fix for a data race between executions that start together, and an image no longer gets a new digest from a commit that did not change it. Proven on kind; neither f0750ec nor 96e6bfb has run on the reference cluster yet. 2026-10-06
041 Surge roll for the node agent — a standby agent takes the node over Implemented 2026-10-03, PRs #1155 (lock + standby), #1156 (agent-ready probe), #1157 (chart, agent.updateStrategy.surge), #1158 (kind gate). On kind, with load running and 20 rolls, the proxy-side gap went from median 6.30 s / p99 10.22 s with delete-then-create to median 0.080 s / p99 0.242 s with surge. Of the surge samples, 11 of 40 were shorter than the ~80 ms poll and are reported as their bracketing bound. Open questions resolved below. The cheap half of #1123 shipped earlier in #1135. 2026-10-02
040 Publish to Quay (quay.io/aethermesh) Implemented. Phases 1 (the abstraction, #998), 2 (the cut-over, #1017) and 3 (the reference cluster runs from quay since rev243, 2026-09-27) are done, and phase 4 (decommission ghcr, #1167) is implemented: the ghcr-lib.sh shim (#1179), the 11 ghcr_* aliases (#1181), PROXY_PIN_LEGACY_REFERENCES and every pin reader's ghcr allowance (#1184), and the publish-verify sweep's ghcr branch plus GHCR_TOKEN and the docs that named ghcr.io as a source (#1186). Pre-check on 2026-10-04: no deployed chart or pod on the reference cluster references a ghcr.io/bpalermo/aether/* image. The ghcr.io packages themselves were not deleted; their retention is a separate maintainer decision. The remaining mentions are history and migration notes for pre-1.0.0 releases. 2026-09-27
039 A CSI Driver as the UDS Carrier Implemented — Phases 1–2 shipped; Phase 3 (the reference cluster) complete except the node-reboot validation, tracked in #1106. Phase 1 (#1090, merged 2026-09-30, a07dea6) is the csi.aether.io node plugin, its uds-csi image, the built-in kubelet registration and the chart wiring. Phase 2 (#1092, merged 2026-09-30, 75caf0f) is the breaking cut-over: chart 2.0.0, crds 1.1.0, udsecho 2.0.0 — the CSI volume is the only UDS carrier, and the emptyDir resolver, --kubelet-pods-dir, the proxy's kubelet-pods mount and proxy.udsWorkloads are gone; udsCsi.enabled defaults on. Phase 3 deployed it on the reference cluster as chart 2.0.0 on 2026-09-30 and soaked it on 2026-10-01 (Phase 3 as built). Phase 1/2/3 as built (under Plan) list only where the build departs from the design text. Phase 0 was not run (2026-10-01). The draft gated the whole proposal on a kind reproduction of a symlink confused deputy in the 034 emptyDir design (Phase 0a, #1083): confirmed meant build, refuted meant park. The owner chose to build the structural fix regardless, and #1083 was closed as superseded on 2026-10-01. The draft's gate text is kept below and marked as history; the confused-deputy reasoning stays as the motivation, not reproduced; superseded by the implementation (see The finding that motivated it). The draft's status line read: ~~Draft. Recommends building it, after Phase 0 settles one question, as a breaking cut-over with no carrier coexistence. Phase 0 has to confirm or refute a symlink confused-deputy in the shipped 034 design. If the attack is confirmed, this proposal is the structural fix and goes ahead. If it is refuted, the remaining benefits do not justify a new privileged node component at today's UDS demand, and the proposal is parked.~~ The breaking cut-over with no carrier coexistence (decision 2026-09-27: a release of dual paths is dearer than migrating the two known UDS workloads in the same upgrade) is what shipped. Owner decisions 2026-09-30 (folded in below; superseded text is marked, not deleted): (1) no node-driver-registrar sidecar: the node plugin serves the kubelet plugin-registration API itself; (2) Q2 answered: fsGroupPolicy: File, the per-pod tmpfs is root:<fsGroup> mode 2770, and a pod without fsGroup is rejected; (3) the image is quay.io/aethermesh/uds-csi (proposal 040 naming). → 040 2026-09-26
038 TPROXY Capture for UDP, and East-West QUIC on Top of It Implemented (Phases 0–4c; 4d pending: UDP:18009 on the east-west gateway is not built). Phase 0: settled on a node 2026-09-25 (#940), full-ruleset spike #944. Phases 1, 2 and 5: superseded by the TPROXY-for-both-transports plan and shipped as #945, #946 and #947 (chart 0.94.0, #949). Phase 3: per-VIP selection e2e #948 (no mode flag left to flip). Phase 4a: #953. Phase 4b: #956 (with #958, #959 and #962). Phase 4c: #979. (Accepted 2026-09-23; revised 2026-09-26 to fold east-west QUIC in as a requirement rather than a deferred question. All four premises verified by experiment.) Superseded in part (2026-09-26): Phases 1, 2 and 5 below were replaced by the TPROXY for both transports plan and shipped as one breaking change with no mode flag, no redirect fallback and no REDIRECT kept for a release — #944 (Phase 0b spike: the full ruleset incl. tproxy proven on a node), #945 (constants), #946 (transparent TCP listener, landed before the CNI change on purpose), #947 (CNI mark-and-divert for TCP and UDP, the per-VIP UDP listener on 18082, the root-only kernel gate in CI), #948 (e2e selection assertion), and the chart bump + soak in the PR that carries this note. The decision that made Phase 5's "written analysis" moot: the 0xae7e passthrough mark never matched in the pod netns to begin with (the proxy is hostNetwork), so there is nothing for the divert mark to compose with. Phase 3 (e2e) and Phase 4 (QUIC) stand as written; the shared-port table and D1/D2 are unchanged. Phase 4 shipped (east-west QUIC, 4a–4c): inbound QUIC on UDP:18008 (#953), per-source outbound twins (#956, demand-scoped since #1020), and the allow-list removed — QUIC is unconditional since #979 (merged 2026-09-29, 33ff5e9, chart 1.0.12). Re-soak of the merged build: 2026-09-30 (chart 1.0.13-84fb413): all Phase 4 gates passed, and #1073/#1074 are shown fixed; the run failed on liveness (9 / 719,984), a hot-restart race outside QUIC (agent down during a proxy fork, #1085). 4d (UDP:18009 on the east/west gateway) is not built. 2026-09-23
037 Multi-Protocol Ports on One Mesh Service Implemented (Phases 0–3; Phase 4 pending). Phase 0 shipped (#888, merged 2026-09-21); Phases 1–3 shipped (#889–#902: cache keyed by cluster identity, the TCP mesh port 18082 and per-port protocol, port-qualified L4 routes, the any-port shim and the soak's per-port TCP leg); L4 observability shipped (#1043, per-service tcp_/udp_ floor stat keys and the aether_l4_access_logs stream). Phase 4 (remove the portless floor chain) is pending: the any-port shim (cap_tcp_anyport_<svc>) is still emitted for a TCP-primary service. Phase 4 will not be done as written here: it is superseded in that respect by proposal 043 (accepted 2026-10-11), which removes the shim together with the mesh-port spellings and the per-service primary port that this proposal kept. One Phase 3 item was not built: the GAMMA projector does not reject an HTTPRoute backendRef.port that names a raw-TCP port (see Open questions). 2026-09-20
036 Replace SPIRE's Delegated Identity API with the SPIFFE Broker API Implemented — accepted 2026-09-18. Phase 1, the only phase in this repository (#802), shipped: the Broker API is the node agent's only path to a pod's SVID. Phases 0 and 2 are platform configuration (in the platform's GitOps repository), deployed and validated on the reference cluster, including an 8-hour soak. Phase 3 (enforced access policy with the impersonation grant) is platform configuration too and was merged in the platform's GitOps repository on 2026-09-19; its measurements (SubjectAccessReview load, CNI ADD latency) are not recorded here. The measured results are in the two Outcome sections below. 2026-09-18
035 Vanity Go module path aethermesh.dev Implemented — 2026-08-08 (apex aethermesh.dev, served by the docs site). Both halves shipped: the serving half (go-import meta + build guard) and the module rename (#631–#633). go.mod is module aethermesh.dev and every import in the tree already uses it. 2026-08-08
034 UDS Support for Pods Implemented (Phases 1 + 1b) and production-validated — annotation path (#616/#617) and EndpointPolicy CRD path (#618) shipped, kind e2e in CI (#619), and both paths validated under an 8-hour churn soak on the reference cluster (2026-08-03; follow-ups tracked in #628/#629). Phase 2 (outbound egress socket) remains deferred post-redirect-all. Superseded carrier (2026-09-30): the emptyDir socket carrier this proposal specifies — the kubernetes.io~empty-dir resolver, --kubelet-pods-dir, the proxy's /var/lib/kubelet/pods mount and proxy.udsWorkloads — was removed by proposal 039 Phase 2 (#1092, chart 2.0.0). The carrier is now an inline csi: {driver: csi.aether.io} volume (the pod needs securityContext.fsGroup), resolved to /run/aether/uds/<pod-uid>/<file>; the annotation, EndpointPolicy, the annotation-wins precedence, the failure semantics and the pipe cluster shapes below are unchanged. Read the carrier-specific text below (the volume kind, the kubelet path, the ~16-character budget) as history; the current contract is in docs/workload-requirements.md. Proposal 039 is implemented (Phases 1–2, #1090 / → 039 2026-06-11
033 Node-taint lifecycle — re-arm the agent-not-ready taint across reboots and agent outages Implemented — the taint lifecycle shipped (agent/internal/node): the agent-not-ready taint re-arms across reboots and agent outages. (Accepted 2026-07-19.) Amended 2026-09-04 (#667) — the removal gate now requires CNI conflist chaining as well as the socket; see the Amendment section at the end. (No in-page anchor link here on purpose: the website republishes this Status line verbatim on /proposals/, where a #-relative link would resolve against the index and break the strict build.) 2026-07-19
032 Upgrade Gateway API to v1.6.1 Implemented — Gateway API upgraded to v1.6.1: TCPRoute/UDPRoute watched at v1, plus InvalidParameters (#532). (Accepted 2026-07-18.) 2026-07-18
031 Reduce the agent's feature-flag surface Implemented — the assessed flag retirements landed (e.g. --l4-routes removed; L4 route types now gate only on their Gateway API CRDs being installed). (Assessment 2026-07-12.) 2026-07-12
030 Move the remaining data-plane ports out of Istio's reserved range Implemented — the data-plane ports moved to 18008/18009/18021; nothing remains in Istio's reserved 15xxx range. —
029 EdgeConfig — edge best-practices, per-instance, native Gateway API Implemented — EdgeConfig shipped: edge best-practices defaults, per-Gateway parametersRef overrides, and HTTP/3 (QUIC) support, with its admission webhook in the controller. (Accepted 2026-07-06.) 2026-07-06
028 GeoIP at the edge Implemented — edge GeoIP shipped: x-geo-* request enrichment with inbound spoof-strip and route-cache clearing. (Accepted 2026-07-06.) 2026-07-06
027 external authorization (ext_authz) via a node-proxy sidecar Implemented — the node-local ext_authz sidecar (UDS transport, OPA preset) shipped; deployed enabled-but-inert on the reference cluster. (Accepted 2026-07-05.) Amended 2026-10-05 (#1275) — the sidecar is a Kubernetes native sidecar (an init container with restartPolicy: Always and a startupProbe on the authz socket), so the proxy container starts only once authz accepts and the sidecar outlives the proxy on shutdown; the chart now requires Kubernetes >= 1.29 when the sidecar is enabled. 2026-07-05
026 Multi-cluster config propagation Implemented — Option E (control cluster — centralized config authority) shipped on Option C's export/import channel: the registrar's leader-elected config-export controller writes ServiceConfigProjections to the shared registry, and the agent's --import-config materializer (with --control-cluster restricting trust to one origin; local config wins) imports them — validated by a two-kind-cluster e2e. Option D (producer-waypoint) for class-1 enforcement; Option A (GitOps) for class-2 consumer-local config; Option B rejected. Run the control cluster as a pure-management cluster (no mesh workloads). etcd key-ACL hardening is parked as a follow-up. (Accepted 2026-06-29.) 2026-06-29
025 Proxy-extension escape hatch for Gateway API / GAMMA Implemented — Option C (ExtensionRef → typed HTTPFilter CRD, opaque body, fail-closed in-process proto-validate — no Envoy binary in the webhook) shipped in full and is validated on the reference cluster: the route-rule ExtensionRef form, the Service-targetRef (policy-attachment) form (unblocked once proposal 026 landed), and CHAIN-scoped filters. common/extensionfilter is the single source of truth for the allow-list + validation/rendering. (Accepted 2026-06-29.) 2026-06-29
024 Gateway API conformance in CI Implemented — Gateway API conformance runs in CI and is a hard gate (GATEWAY-HTTP + MESH-HTTP Core). (2026-06-28 design.) 2026-06-28
023 Service-based routing — decoupling the GAMMA route target from ServiceAccount identity Implemented — the route-target catalog + real-ClusterIP:port capture shipped (GAMMA-on-capture); the versioned-backend shape it exists for is exercised by the MESH-HTTP conformance profile, which is green and a hard CI gate (proposal 024). (2026-06-28 design.) 2026-06-28
022 Arbitrary-Service interception (the genuine GAMMA / Mesh data plane) Implemented — arbitrary-service interception (redirect-all) shipped and is the default for managed pods (agent.captureRedirectAllDefault=true), soak-validated hitless on the reference cluster. (2026-06-25 design.) 2026-06-25
021 Per-Gateway addressing for the edge Implemented — per-Gateway addressing (Phase 2) shipped and is unconditional. The edge.perGatewayAddressing chart value that once gated it was deleted by proposal 031; there is nothing to set. (2026-06-25 design.) 2026-06-25
020 Namespace-aware services + opt-in non-mesh endpoint projection Implemented — the namespace-aware <ns>/<serviceAccount> registry-key cutover completed and is the mesh's identity model (registry keys, SVIDs, generated mesh Services). (2026-06-25 design.) 2026-06-25
019 Multi-cluster pod-to-pod via a per-node waypoint (non-routable pod IPs) Implemented — the node proxy acts as the per-node E/W gateway behind --east-west-waypoint (default off; needs shared etcd + shared SPIRE trust domain), e2e-validated. (2026-06-24 design.) 2026-06-24
018 Gateway API for aether — north-south + GAMMA east-west Implemented — all phases shipped and default-on: Phases 1–2 (edge HTTPRoute + GAMMA east-west), Phase 3a (HTTP transparent capture), and Phase 3b (L4 TCPRoute/TLSRoute/UDPRoute, default-on since #493; the agent.l4Routes flag was retired by proposal 031 — each route type now gates only on its Gateway API CRD being installed). (2026-06-22 design.) 2026-06-22
017 VirtualHost CRD — L7 path-based edge routing (supersedes EdgeRoute) Implemented, then superseded — the VirtualHost CRD shipped, but edge L7 routing has since migrated to the Gateway API (HTTPRoute) and the VirtualHost CRD was retired. See proposal 018. (2026-06-21 design.) → 018 2026-06-21
015 aether system config (umbrella globals) + proxy MeshConfig CRD Implemented — the aether system config + MeshConfig CRD shipped (later made namespaced with an aether-system fallback). Live proxy reload remains future work. (2026-06-18 design.) 2026-06-18
014 proxy access logs to VictoriaLogs over OTLP Implemented — proxy access logs over OTLP shipped and validated (2026-06-17 design). 2026-06-17
013 synthetic mesh-availability prober Implemented — the mesh-availability prober shipped (charts/prober). (Originally 2026-06-17; amended: probe a proxy local-reply liveness endpoint, not a real service.) 2026-06-17
012 aether_stats as a native C++ Envoy extension Implemented — aether_stats ships as a native C++ extension in the custom proxy workspace (proxy/source/extensions/filters/http/aether_stats/), default-on in the mesh. (2026-06-15 design.) 2026-06-15
011 Port the aether_stats filter into the proxy build Implemented — superseded in direction by proposal 012: the filter rides the custom proxy build as a native C++ extension rather than a bundled dynamic module (this proposal's envoy --mode validate harness lives on as //test/envoy_validate). (2026-06-15 design.) → 012 2026-06-15
010 Custom Envoy proxy as a separate sibling Bazel workspace Implemented (proposals 010/011/012) — supersedes proposal 009 2026-06-15
009 Envoy in the Bazel workspace — version alignment, validation, customization Superseded by proposal 010 — spike / investigation, recommendation below (2026-06-14) 2026-06-14
008 Rust in the Bazel build (rules_rust + hermetic_cc_toolchain) Implemented, then superseded by proposal 010 — the rules_rust + hermetic_cc toolchain shipped (#177) and carried the proposal-007 Rust dynamic module (#178), but was removed when aether_stats moved to a native C++ extension in the custom proxy workspace (proposals 010–012). No Rust remains in the tree. (2026-06-13 design.) 2026-06-13
007 Source↔Destination Telemetry via a Rust Dynamic Module Implemented — the aether_stats source→destination metrics filter shipped and is default-on (later re-based as a native C++ Envoy extension; see proposals 011/012). Phase-1 module validated end-to-end on stock distroless Envoy 1.38.0 (2026-06-13). 2026-06-13
006 multi-region etcd federation (region-scoped registry ownership) Implemented — the registrar's --peer-etcd replicator (origin-partitioned keys, origin-heartbeat lease) shipped and is exercised by the nightly two-cluster replication e2e gate. (2026-06-20 design.) 2026-06-20
005 Multi-Port Routing per Pod Implemented — multi-port routing shipped (endpoint.aether.io/ports, per-port EDS, SNI demux); design validated by spike (Envoy 1.38.0/BoringSSL, 2026-06-13). 2026-06-13
004 Demand-Scoped Service Distribution Implemented — demand-scoped CDS/watch/health-checking shipped as a breaking change (no fallback flag); cold path = ODCDS, including the capture on-demand catch-all (#288). (2026-06-12 design.) 2026-06-12
003 Edge Proxy (North-South Ingress Gateway) Implemented — PRs #236 (control plane), #237 (EdgeRoute CRD), #238 (chart), #239 (downstream TLS). NOTE: the EdgeRoute CRD described here was later retired; the edge now routes via the Gateway API (Gateway/HTTPRoute) — see proposals 017 and 018. 2026-06-19
002 Agent Concurrency Audit: Config Propagation, Hot Restart, Pod Lifecycle All findings fixed — R1+R3 in #109, R2+R4-R8 in #110, follow-on predecessor re-probe in #111 2026-06-10
001 Hot Restart for the aether-proxy Envoy (Spike) Implemented — the hot-restart supervisor shipped (Strategy B), zero-drop validated on the reference cluster; follow-on audits merged (#109–#111). Carved out of the agent binary into //agent/cmd/proxy-supervisor and its own image by #772. (2026-06-09 spike.) 2026-06-09
000 In-Cluster Registrar Service Implemented 2026-03-22

A number is enough to cite one: /proposals/000/ redirects to the full URL of that proposal, and keeps working when its title changes.