Proposal 043: A mesh service is addressed by its real ports, and nothing else¶
Design record. This proposal is published as written, at its stated status. Later proposals may supersede parts of it, and implementation details drift. It documents the reasoning at a point in time, not the current behaviour of the system — for that, see the docs.
Status: Accepted 2026-10-11 by the owner, with all twelve decisions taken
as recommended (see Decisions), and built as one breaking release, chart
3.0.0 (decided 2026-10-12). The four phases this document planned over
charts 2.7 to 2.10 were collapsed: see The release for what shipped, what the
plan had that did not, and Rulings after acceptance for the decisions taken
while it was built, which hold over the older text. The facts were read from
main at 2a132053; the measurements were made on the pinned proxy outside
any cluster.
Resolves: #1672. The defect reported there is the per-service primary
itself; the owner decided to remove it, not to patch it. The acceptance of
this proposal closes the issue: the phases below are the work, and Phase 1 is
where the reported behaviour is fixed.
Author: Bruno Palermo
Date: 2026-10-11
Related: #1672 (a raw-TCP primary with an HTTP secondary has no TCP path),
1682 (a cluster.local name with a non-primary port reaches the primary¶
port; the same rule broken, fixed in Phase 1), #1677 (the first fix, closed; its branch is kept for Phase 1: it measured both registry backends, and this proposal says which parts of it survive), proposals 037 (per-port protocol; its Phase 4 is subsumed here), 038 (TPROXY capture; why UDP binds 18082), 022 (redirect-all), 018 (Gateway API and GAMMA), 005 (per-port clusters), 026 (config propagation), 030 (the 18xxx ports).
The decision¶
A mesh service is addressed only by its real ports. Port-less addressing is removed for every service, single-port ones included, and with it the per-service "primary" port and protocol. Protocol is a property of a port.
Three client spellings exist today that do not name a real port
(docs/workload-requirements.md:274-281, read):
| Spelling | Means today |
|---|---|
<svc>.<ns>.<domain>:18081 |
"the primary port, over HTTP" |
<svc>.<ns>.<domain>:18082 (TCP and UDP) |
"the primary port, as raw TCP" / "as plaintext UDP" |
http://<svc>.<ns>.<domain>/ (the client dials 80) |
"the primary port, over HTTP" |
Each needs the service to collapse to one port and one protocol. That collapse is the primary. Proposal 037 made protocol a property of a port for every spelling that names one, and kept the primary for the three that do not. #1672 is what the kept part costs: when the primary is raw TCP and a secondary is HTTP, three readers disagree about what "the service's protocol" is, and the answer depends on the registry backend.
After this proposal a client writes <svc>.<ns>.<domain>:<port>, where <port>
is a port of the Service, and the port's own declaration says what speaks there.
http://<svc>.<ns>.<domain>/ still means port 80, as it does everywhere else on
the internet, and works when port 80 is a port of the Service.
Proposal 037 considered "require every caller to name a port" and rejected it
because it breaks every HTTP caller in the mesh
(037_multi-protocol-per-service.md, Alternatives). That cost is real and is
restated under Risks. The decision accepts it; this document is about paying
it in an order that never strands a client without a working spelling.
Prior art¶
Described from general knowledge of Istio's sidecar data plane. Not verified against that project's current source or documentation; cited as design precedent, not as a specification to match.
- There is no mesh-specific client port and no per-service primary. Capture
recovers the original destination, and the client's own
<service>:<port>is the address. - Listeners, routes and clusters are keyed per Service port.
- A port's protocol comes from the Service port (
appProtocol, or a port-name prefix), with protocol sniffing as the fallback when neither says. - A port the Service does not declare is passed through, not routed.
- A route that names no port applies to the ports of the Service.
- Application UDP is not handled by the proxy.
Aether differs in two ways that matter below. Its registry discovers pods by ServiceAccount and reads ports from pod annotations, not from a Service selector, so the only Service the mesh always has is the one the registrar generates. And aether proxies application UDP today.
Facts established¶
Each fact is marked read (the code or document that does it was read at the cited line), measured (run for this proposal or in #1677, and by whom) or inferred (derived from read code, not exercised).
1. Everything that needs "the service" to be one port or one protocol¶
Where the primary comes from (all read):
- The pod annotation
endpoint.aether.io/port, default 8080, becomesServiceEndpoint.port(registry/endpointmeta/endpointmeta.go:55-60,registry/cni.go:42,69,api/aether/registry/v1/endpoint.proto:17). The proto comment calls it "the default/primary port — what the portless FQDN resolves to" (endpoint.proto:19-24). - A service's primary is the first endpoint's:
defaultPort := endpoints[0].GetPort()(agent/internal/xds/cache/cluster.go:518-520), and again atcluster.go:495,:773,:968,:381-386, and in the registrar atregistrar/internal/services/generator.go:119,130. - A service's protocol is whichever listing it is found in first: the generator
iterates
registry.ServedProtocols(HTTP, TCP, UDP) and "the first protocol iterated wins" (generator.go:106-126); the agent reads the result back off the Service (agent/internal/capture/reconciler.go:186-193).
| # | Consumer | What it assumes | Evidence | What replaces it |
|---|---|---|---|---|
| 1 | :18081 on a mesh name |
The service has one HTTP port. cap_http claims <name>:18081 for the default cluster; an alias cluster <fqdn>:18081 exists for the on-demand path |
read: cache/capture.go:872-893, :1158-1161; cache/cluster.go:634-646 |
Removed. <name>:<p> for each declared HTTP port |
| 2 | :18082 on a mesh name, TCP |
The primary is raw TCP: chain cap_tcp_<cluster>_18082 to the floor cluster tcp:<fqdn> |
read: proxy/capture.go:160-162, :623-635 |
Removed. cap_tcp_<cluster>_<p> for each declared TCP port (exists since 037 for non-primary ports, proxy/capture.go:297-318) |
| 3 | :18082 on a mesh name, UDP |
The service has one UDP port, the primary; needs a UDPRoute |
read: proxy/l4route.go:245-331, cache/capture.go:657-674 |
See fact 3 |
| 4 | The bare name (http://<name>/, dials 80) |
Portless authority = the primary, over HTTP | read: cache/cluster.go:528, cache/capture.go:889-892 |
Port 80, if the Service has it |
| 5 | The any-port shim cap_tcp_anyport_<cluster> |
Any port of a TCP-primary VIP is the primary | read: proxy/capture.go:634 |
Removed (037 Phase 4, subsumed) |
| 6 | The generated Service's ports | Four name-only TCP ports: mesh 18081, mesh-tcp 18082, http 80, https 443. No real port, no UDP port, no appProtocol |
read: generator.go:169-176 |
The service's real ports, each with its protocol |
| 7 | aether.io/app-protocol on the Service |
One class per Service | read: written generator.go:255-259; only reader capture/reconciler.go:186-193 |
Per-port protocol from the endpoints; the annotation is frozen, then removed |
| 8 | aether.io/port on the Service |
The primary | read: written generator.go:257; no reader outside the generator's own compare (generator.go:232) |
Removed |
| 9 | Default HTTP cluster <fqdn> |
One port per service; SNI = the primary port; the only subscriber of the bare EDS name | read: cache/cluster.go:524-532, cache/mtls.go:189-205 |
Removed. <fqdn>:<p> per port (005) |
| 10 | Default virtual host | GAMMA rules attach to it only; per-port virtual hosts get the chain filter and no rules | read: cache/cluster.go:528, :1200-1206; cache/capture.go:843-857 |
Rules attach to the virtual host of every port the route's parent selects |
| 11 | HTTPRoute backendRef.port |
Ignored: every backend is the default cluster | read: common/gammaproject/project.go:222-241 |
Resolved to the backend's <fqdn>:<p> |
| 12 | The known-target safety net | <svc>.<ns>.svc.cluster.local:<any port> is pinned to the default cluster |
read: cache/capture.go:953-1041 |
Per-port targets |
| 13 | QUIC twins | Built for the default entry only; the name refuses a port; SNI <primary>.<fqdn> |
read: cache/cluster.go:1278-1304, proxy/egress.go:193-230 |
One twin per (source, destination, port) |
| 14 | On-demand (ODCDS) catch-all | None: :authority is the cluster name verbatim, port included; admission is per service |
read: proxy/route.go:176-295, server/odcds.go:295-315 |
Unchanged. A portless authority stops naming a cluster |
| 15 | TCP mesh hop | Empty SNI means "the primary"; the destination's default chain forwards to the primary | read: cache/capture.go:461-467, proxy/ingress.go:248-258 |
SNI = port for every TCP port; the default chain is removed last |
| 16 | HTTP mesh hop | The destination's no-SNI h2 chain is the primary | read: proxy/ingress.go:188-190 |
A server_names chain per port (already built for every port, ingress.go:199-205) |
| 17 | Delegated liveness | Probes the primary port, typed by the pod-level protocol, not the port's | read: proxy/listener.go:101-117, proxy/cluster.go:365-449 |
The probe port's own protocol (decision 6 names the probe port) |
| 18 | Mesh DNS | Nothing: A records only, name to ClusterIP | read: agent/internal/meshdns/server.go:692-709, capture/reconciler.go:162-167 |
Unchanged |
| 19 | Edge backend resolution | Port 0, or a port equal to the primary, means the default cluster | read: cache/edge.go:663-691 |
The port's cluster, always. Edge TCP publishes only tcp:<fqdn> (cache/capture.go:495-533); a port-qualified backend names a cluster the edge never publishes (inferred) |
| 20 | Cross-cluster config (026) | GammaBackend is {service, cluster, weight} with the default cluster's name; the projection carries no port |
read: api/aether/registry/v1/config.proto:21-37,67-71 |
An additive port on the backend and on the projection's parent |
| 21 | Clusterset VIP and ServiceImport (MCS) |
One port, mesh 18081; an existing clusterset Service is never converged |
read: registrar/internal/mcs/import_generator.go:250-256, :275, :301-306 |
Real ports, converged |
| 22 | East-west waypoint (019) | SNI is already <port>.<fqdn>; "the primary port needs no change" on the destination |
read: cache/mtls.go:197-205, cache/listener.go:84-104 |
A chain per port on the destination |
| 23 | UDS-delivered services | Nothing: every declared port dials the one socket | read: proxy/cluster.go:155,207-209 |
Unchanged |
| 24 | Registry listings | ListEndpoints(svc, protocol); the kubernetes backend filters by the pod-level annotation alone |
read: registry/registry.go:27-36, registry/internal/k8s/kubernetes.go:498-509 |
A pod is listed under every protocol it serves a port of, on both backends |
| 25 | registry.ServedProtocols |
Three callers. The generator uses it as a service class (first wins). The registrar sync and the agent's ghost sweep use it as the key space, with a comment that "a service is registered under exactly one protocol" | read: registry/protocols.go:34-38; generator.go:109; registrar/internal/server/sync.go:36; agent/internal/cni/server/ghostsweep.go:173,423-424 |
Stays as the key space. No consumer reads a class from it |
| 26 | registrytest.RequireProtocolDisjoint |
"a service is HTTP xor TCP" | read: registry/registrytest/contract.go:49-77 |
The per-port contract 037 specified and never built (contract.go:58-62) |
| 27 | Stat names | cap_tcp_<cluster>_18082 and cap_tcp_anyport_<cluster> chain prefixes; the floor key tcp_<ns>/<svc>; udp_<ns>/<svc> |
read: proxy/capture.go:629,634; proxy/egress.go:151-181 |
Per-port keys only (tcp_<ns>/<svc>_<port> exists) |
| 28 | Access logs and metrics | No service-level protocol field; no protocol or port metric attribute |
read: proxy/accesslog.go:100-145,257-277; grep of agent Go |
Unchanged. Documented samples show :18081 authorities (docs/configuration.md:290,294) |
| 29 | Harness contract | port.outbound_http "18081", port.outbound_l4 "18082", the any-port stat prefix, and the sentence "a mesh service is addressed as <service>.<namespace>.<mesh-domain>:<port.outbound_http>" |
read: test/harnesscontract/external-harness.yaml:262-304, version 3 at :39 |
Two contract bumps, see phases |
Who dials the spellings today (read; counts are lines naming 18081 or 18082):
| Client | Spelling | Where |
|---|---|---|
| Prober, liveness and reachability tiers | loopback listener 127.0.0.1:18081, portless Host |
prober/internal/prober/prober.go:86,220-230 |
| Prober, mesh_dns tier | <target>:18081; a bare target gets :18081 appended |
prober.go:46,244,261-265; charts/prober/values.yaml:58-59 |
| Prober authz canary | :18081 |
charts/prober/templates/authz-canary.yaml:174 |
e2e uds, authz, multicluster-config, waypoint, replicator, eastwest-quic |
:18081 |
e2e/uds.sh:70, e2e/authz.sh:74, e2e/multicluster_config.sh:434, e2e/multicluster_waypoint.sh:393, e2e/multicluster_replicator.sh:583, e2e/eastwest-quic.sh:212 |
e2e multiprotocol |
:18081 (M0, M1), bare name (M2), real ports (M3, M4), :18082 (M5, X1) |
e2e/multiprotocol.sh:559-634 |
e2e l4-routes |
TCP and TLS on real ports; UDP on :18082 only |
e2e/l4routes.sh:182,193,207,724,776 |
charts/udsecho client |
:18081 |
charts/udsecho/templates/client.yaml:52 |
| The external soak harness | :18081, by contract |
external-harness.yaml:278-281; 042_sortie-soak-driver.md:106,151 |
| Gateway API conformance | real Service ports already (80, 8080); no mesh port anywhere | test/conformance/mesh/manifests.yaml:51-65; grep of test/conformance |
| Docs | getting-started.md 14 lines, workload-requirements.md 10, runbook.md 9, configuration.md 4, README.md 2 |
grep |
Three things the table turned up that are not about clients:
- Real HTTP ports work today, but bypass the routes.
cap_httphas no virtual host for<name>:<app port>unless a route's parentRef names that port (cache/capture.go:872-893,:1163-1170, read). The request falls to the catch-all, whosecluster_headersends it to the per-port or alias cluster with no GAMMA rule applied (inferred). The nightly leg M3 passes for this reason and asserts only that the echo answers. - Two documents contradict each other on whether the authority's port is
stripped:
getting-started.md:514-516says it is,workload-requirements.md:258-262says it is not. The code does not strip it (proxy/filterchain.go:27-30, read). - The any-port counter has no reader. 037 promised "one WARN per service
per hour naming the count". No such log exists in Go (grep for
anyportfinds onlyproxy/capture.go:558,634).
2. Capture¶
What the two modes are (read, cni/internal/plugin/capture.go:171-210): the pod
netns gets a mark in the output chain and a divert in prerouting. The mark rules
are
udp dport 18082 mark (both modes)
tcp dport {18081, 18082} mark (both modes: this is "scoped")
meta l4proto tcp mark (redirect-all only: every other TCP port)
and the divert is tproxy to :18001 for TCP and a bare accept for UDP.
- Scoped capture is the two mesh ports and nothing else. Remove the mesh ports and the scoped rule set captures no TCP at all.
- Redirect-all is not "all" for UDP. UDP is captured on 18082 only, in both
modes; there is no any-port UDP rule (
capture.go:194, read; restated inworkload-requirements.md:294-295). - Who chooses. Per pod:
capture.aether.io/redirect-all"true"/"false", else the node default (cni/internal/plugin/plugin.go:742-760, read). The node default is the chart valueagent.captureRedirectAllDefault, true (charts/aether/values.yaml:325); the CNI flag's own default is false (cni/internal/cmd/root.go:47). There is no namespace-level switch (grep). - Who uses scoped. One shipped manifest sets the opt-out: the edge
Deployment (
charts/aether/templates/edge-deployment.yaml:51), on a pod that is alsoaether.io/managed: "false"and therefore gets no capture at all (plugin.go:131-140), so the annotation is redundant there (inferred). No e2e suite, test manifest or chart sets either the value or the annotation (grep). Conformance relies on redirect-all (test/conformance/README.md). - The agent already has only one mode. It hardcodes the redirect-all
listener shape for every pod:
snapshotCache.SetCaptureRedirectAll(true)(agent/internal/cmd/root.go:567-568, read; the only non-test caller). The scoped listener shape (cap_tcp_blackhole, the catch-all HCM chain, the 404 fallthrough, theblackholecluster) is built only in unit tests. A pod opted out at the CNI gets scoped rules and a redirect-all listener.
Consequence: removing the mesh ports makes redirect-all the only TCP capture mode. What is lost, exactly:
- A managed pod can no longer ask for "mesh only what I dial on a mesh port,
leave every other connection untouched by the proxy". The replacements are
the per-pod carve-outs that stay (
capture.aether.io/exclude-outbound-ports,capture.aether.io/exclude-outbound-ip-ranges) and the loopback listener (decision 5), which loopback traffic reaches without any capture rule (capture.go:175:127.0.0.0/8is accepted before the marks). capture.aether.io/redirect-all: "false"changes meaning from "scoped" to "this pod's outbound TCP is not captured"."true"becomes a no-op.agent.captureRedirectAllDefaultand--capture-redirect-all-defaultgo.- Nothing changes for unmanaged pods and ignored namespaces: they get no
capture today (
plugin.go:80-85,131-140).
3. UDP¶
How it works today (read): one listener per pod, capture_udp_<pod>, bound
0.0.0.0:18082 in the pod's network namespace with transparent: true, built
only when a UDPRoute arm exists (proxy/l4route.go:245-331). Its udp_proxy
matches on the destination address (one arm per parent ClusterIP) and
forwards plaintext to pod_ip:<port> from the proxy's own network namespace;
there is no inbound listener, no mTLS and no identity on this path
(cache/capture.go:735-738). The port it forwards to is the service's primary,
not a port declared =udp (cache/capture.go:657-674, cache/cluster.go:968;
inferred, and contrary to what 038 says at 038_udp-tproxy-capture.md:167-169).
Endpoints behind a waypoint are dropped (proxy/egress.go, UDPLoadAssignment).
A weighted split and a multi-service route are not representable (#914).
Why 18082: a datagram socket replies from the port it is bound to, so the client must dial the port the listener binds (038, The port is not a free variable).
Measured for this proposal on the pinned proxy (Envoy 1.40.0-dev at proxy
commit 3119155a), Linux 6.8, one unprivileged network namespace, no aether
code, no cluster. A client at 192.0.2.1, a service address 10.96.0.10, two
echo backends tagged A (the matcher's arm) and B (on_no_match). The method is
in the appendix.
| # | Arrangement | Connected client socket | Unconnected socket sees | Reading |
|---|---|---|---|---|
| T1 | Client dials the real port 5353; divert to the listener bound 0.0.0.0:18082 |
no reply | A from 10.96.0.10:18082 |
The reply leaves from the bound port. A connected client drops it |
| T1c | Control: client dials 18082 (today) | A |
A from :18082 |
Today's shape works |
| T1p | As T1, matcher on destination port = 5353 | no reply | B from :18082 |
The matcher's port input is the listener's port, not the datagram's |
| T2 | Listener bound 0.0.0.0:5353 (the real port) |
A |
A from 10.96.0.10:5353 |
Binding the real port works |
| T2m | Two listeners, 5353 and 5354, same address | A, B |
from :5353, :5354 |
One listener per port distinguishes ports |
| C1 | The application already holds 0.0.0.0:5353; then the listener, on 0.0.0.0:5353 or on 10.96.0.10:5353 |
— | — | The proxy fails to bind: Address already in use, also when the application set SO_REUSEADDR |
| C2 | The listener holds 0.0.0.0:5353 or 10.96.0.10:5353; then the application binds 0.0.0.0:5353 |
— | — | The application fails to bind: EADDRINUSE, with and without SO_REUSEADDR. It binds only with SO_REUSEPORT |
| S | Wildcard listener on 5353, no divert; client dials the pod's own address :5353 |
B |
B |
A wildcard listener also takes datagrams addressed to the pod |
| N1 | Conntrack DNAT of the port only (5353 and 5354 to 18082, address kept), one listener on 18082 | A, A |
from :5353, :5354 |
Conntrack repairs the reply's source port. The proxy cannot tell the two ports apart |
| N2 | DNAT 5353 to 18100 and 5354 to 18101, a listener per reserved port | A, A |
from :5353, :5354 |
Works, distinguishes ports, binds no application port |
So, to the question asked: a transparent UDP listener in the pinned proxy cannot reply from the original destination port without being bound to it (T1), and its matcher cannot see that port either (T1p). Only two things put the right source port on the reply: binding the port (T2), or letting conntrack rewrite it (N1, N2).
The options:
- (a) One bound listener per distinct real UDP port, per pod. Works on the
wire (T2, T2m) and fails in the namespace: the listener lives in the pod's
network namespace, where the application binds the same number whenever a
pod is both a server and a client of a UDP protocol (DNS, gossip, NTP peers,
most game and media protocols). Either the proxy's listener is rejected (C1)
or the application cannot start (C2); with
SO_REUSEPORTon the application the two would share the port, and a wildcard listener takes the pod's own inbound datagrams (S). Under different user IDs, which is the case in a pod, theSO_REUSEPORTcase fails too (inferred from the kernel's bind rule; the rig had one user). It also needs the capture rule to follow the port set, which today is written once at CNI ADD and names one constant. Listener count: pods × distinct UDP ports in each pod's scope, each with one socket per worker (inferred forudp_proxy; stated for QUIC atproxy/ingress.go:418-420), and each paused and forwarded across a hot restart by the same generic listener code the carried patches change (proxy/README.md:125-133). No listener count for a real cluster is recorded in the repository, so the number is not established. Rejected: C1 and C2 are not edge cases. - (b) Keep 18082 for UDP only. No work and no risk. It keeps a mesh port, a per-service primary for UDP, and the rule that a UDP service has one port: the three things the decision removes.
- (c) Conntrack port translation to reserved listener ports (N2). Honours
the decision and stays in the proxy. It needs a per-pod kernel map from
(service address, real port) to a reserved listener port, kept current as
services and ports come and go; nothing programs a pod's rules after CNI ADD
today (inferred:
plugin.go:142-172installs once). It needs a listener per distinct real UDP port per pod, and a port allocator. 038 rejected the same shape for this cost (038_udp-tproxy-capture.md:757-763). - (d) Stop proxying application UDP. Give the generated Service its UDP
ports as ordinary Service ports with
EndpointSlices the registrar writes from the registry, and let the cluster's own service routing carry them: no capture, no listener, no port translation of ours, and the reply is repaired by the same machinery that repairs every other Service. This is the prior art's position.
Decided (decision 1): (d), with (b) as the state until it ships. The evidence the recommendation rested on:
- Every proxy-based option needs either a bind the pod can collide with (measured) or a new per-pod dynamic kernel mechanism plus a listener family (N2), to carry traffic the mesh neither encrypts nor authenticates (read).
- What the proxy adds for UDP today is endpoint selection from the registry,
registry health, and
udp_stats with an L4 access-log line. It does not add identity, mTLS, weights, more than one backend, more than one port per service, or a path through a waypoint (all read, above).EndpointSliceconditions carry the first two. - It deletes the per-pod UDP listener, the
udp:clusters, the UDP mark rule and the UDP half of port 18082, and it lets an unmanaged client reach a mesh UDP service, which it cannot today.
What (d) costs: the per-service UDP stats and the UDP lines of
aether_l4_access_logs; UDPRoute served by the proxy (it becomes "the
parent Service's slices carry the chosen backend's endpoints", written by the
registrar, for the single-backend shape only: decision 1); a registrar that
writes
EndpointSlices (new RBAC); endpoint freshness at the speed of the cluster's
service routing instead of EDS. East-west QUIC is not affected: it is mesh
transport on 18008, not application UDP. None of the carried patches goes away:
they serve the QUIC listeners. (d) is not measured. It is stock Kubernetes
behaviour, but no aether cluster has run it. Its work in Phase 2 therefore
starts with a measurement on kind, on both registry backends, before anything
is built on it.
4. The generated Service as an ordinary multi-port Service¶
- Today (read): selectorless,
ClusterIP, named after the ServiceAccount in the service's namespace, four name-only TCP ports, three annotations (generator.go:169-176,:252-266).applyconverges two annotations and the ports by name, port and protocol (generator.go:231-243);targetPortandappProtocolare not compared (:182-197). - Target: one
ServicePortper declared port,port=targetPort= the pod's port,protocolTCP or UDP,appProtocolhttportcp(a UDP port needs none), namedp<port>-<tcp|udp>. - Where the ports come from:
ServiceEndpoint.port_protocols, which both backends already populate (registry/cni.go:61-89,registry/internal/k8s/kubernetes.go:513-538, read). - A rollout that changes ports. The Service carries the union over the
service's endpoints. That is safe for the reason 005 gave: membership of a
port's endpoint set is filtered per port, so a client of
:9090only reaches pods that declare 9090. A port leaves the Service when the last endpoint declaring it leaves the registry. - A rollout that changes a port's protocol. Two endpoint sets exist for the number (037 design (c): membership by port and protocol) but capture can build one chain for it. Rule: the class most endpoints declare, ties to HTTP then TCP, order-independent, with a WARN. A client speaks one protocol, so this cannot be made hitless by any rule; the documented way to retype a port is a new port number.
- Port 80. Today
http://<name>/reaches the primary because the portless authority is the default cluster, and the Service's port 80 is a name-only placeholder. After removal it works only if a pod declares port 80. The default application port is 8080 (common/constants/endpoint.go:5), so for nearly every existing service the bare URL stops working. See Risks. - Collisions. A user Service that owns the name is never touched; the
generator logs and skips, writes no status and no event
(
generator.go:221-224, read). Then there is no mesh VIP and no mesh-DNS record for the name, and the service is reachable only by itscluster.localname under redirect-all. With real ports this case matters more: see fact 5. - One number, two transports.
port_protocolsismap<uint32, PortProtocol>(endpoint.proto:117): a port number has one class. A pod cannot declare 53/TCP and 53/UDP. Not solved here; it is its own change, due before Phase 3 (decision 7). - The two backends. They differ in one way that matters:
etcdlists a pod under every protocol it was registered under (the agent registers once per distinct class,agent/internal/cni/server/dualregistration.go:36-48);kuberneteslists by the pod-level annotation alone (kubernetes.go:498-509). #1677 measured the consequence on kind: on the chart's default backend the nightly-gated shape (HTTP primary,9000=tcp) fails leg M4 (a capture chain for:9000with no cluster behind it), and CI never sees it because every multi-protocol suite runs onetcd. - Multi-cluster. Endpoint keys mirrored between regions keep their
/protocols/<P>/segment and are replayed verbatim (registrar/internal/replicator/replicator.go:251-349, read), so the key layout must not change, and this proposal does not change it. A peer on an older release still lists by protocol (compatible) and still dials:18081on its own nodes, which is that peer's node proxy's business, not ours: the mesh hop between clusters carries a port in the SNI already (cache/mtls.go:197-205). What does cross is config: an exported route names the default cluster (fact 1, row 20).
5. Routes that name no port¶
- The Gateway API rule for a mesh route (GAMMA): a
parentRefto a Service with noportapplies to all ports of the Service; with aport, to that port. A ServicebackendRefcarries a port. - Today (read): the parentRef port only adds
<name>:<port>domains to the service's one virtual host; it never scopes which rules apply (agent/internal/gamma/reconciler.go:292-331,cache/capture.go:1163-1170). Two routes with different parentRef ports merge into one virtual host that matches both (inferred). The backendRef port is not read at all (project.go:222-241). Nothing in aether rejects a missing backendRef port; the CRD does (proxy/egress.go:133-135, comment). - What conformance requires. MESH-HTTP is 7/7 on Gateway API v1.6.3
(
docs/conformance/baseline-2026-10-11-rev22.md). Its fixtures are user Services namedecho,echo-v1,echo-v2withhttp80 →targetPort8080,http-alt8080,https443 → 8443,tcp9090,grpc7070 (test/conformance/mesh/manifests.yaml:51-65), and the client dialshttp://echo/: port 80. It passes today for the reason this proposal removes: the portless authority is the primary, and the primary defaults to 8080. - No code reads a Service's port map. No non-test Go in
agent/orcommon/readsspec.ports[].targetPort(grep). The onlytargetPortin the mesh is the one the generator writes.
So "real ports" has to mean the ports of the Service the client named, not
the ports of the pod. For the generated Service the two are equal by
construction. For a user Service that owns the name, port 80 is real and maps
to pod port 8080, and aether must read that map. Without it MESH-HTTP goes from
7/7 to failing on its first request. HTTPRouteParentRefPort and
HTTPRouteDestinationPortMatching are unsupported and skipped today; the first
becomes supportable as a side effect and is not claimed here.
Design¶
The model¶
- A service has ports. A port has a transport (TCP or UDP) and a class
(HTTP or raw TCP; UDP has no class). The source of truth is the registry's
per-endpoint
port_protocols. No field, annotation, listing or name carries a protocol for a service. - A Service is a port map. The generated Service maps each declared port
to itself. A user Service that owns a mesh service's name maps its own ports
to
targetPorts. A client addresses a Service port; the mesh resolves it to a pod port and uses that pod port's class. A numerictargetPortresolves. A named one does not (the registry carries no port names) and is reported on the route's status and as a WARN. - One cluster family per port. HTTP
<fqdn>:<p>(005), raw TCPtcp:<fqdn>:<p>(037), QUICquic:<fqdn>:<p>@<ns>/<sa>. No default cluster, no floor cluster, no alias cluster. - For captured HTTP, the dialled port is the port;
Hostnames the service. The capture listener has one HCM chain and one route table per distinct HTTP port number in the mesh, selected by the connection's destination port and the sniffed application protocol. This is the prior art's shape, and it is the adopted design because the single-chain layout was measured not to work (The addressing rule, below). - Raw TCP is matched on the destination address and port, as today for
non-primary ports:
/32 + destination_port. The declaration decides; the bytes are not consulted. - Routes. A parentRef without a port attaches the rules to the virtual
host of the Service in the table of every HTTP port it has; with a port, in
that port's table. A backendRef port is a Service port, resolved through
the port map to
<backend fqdn>:<pod port>. - A port nobody declared is not routed. HTTP is refused with a
421that lists the service's ports. Raw TCP has no chain and falls to the passthrough, where the answer is the cluster's: a port the Service does not have (decision 10).
The addressing rule¶
Stated once. Everything else in this document that mentions a dialled port or
a Host header follows from this section.
Why the dialled port decides (measured)¶
The first version of this proposal kept one HCM chain for all captured HTTP and
selected the port from Host ("Host wins; a port-less Host is port 80").
The owner asked whether the filter-chain match could do it instead: the
dialled port and the sniffed protocol choose the chain, as the prior art does
(from general knowledge of Istio's sidecar; not verified against its source).
It can, and the single-chain layout turns out to be broken, today, on main.
Measured on the pinned proxy (Envoy 1.40.0-dev, proxy commit 3119155a),
Linux 6.8, one network namespace, a transparent TCP listener on 18001 with the
shipped capture listener's filters and timeouts (original_dst,
http_inspector, tls_inspector; listener_filters_timeout 1 s, continue on
timeout; proxy/capture.go:196-215,274-280, read) and TCP diverted to it with
the shipped rule shape (mark on output, tproxy to :18001 on prerouting).
Service A declares 9000 and 3306 as raw TCP; service B serves HTTP; address C
is not in the mesh. The method is in the appendix.
| # | Layout | Connection | Selected | First byte |
|---|---|---|---|---|
| L1 | today: one HCM chain with no destination port, plus A's /32 + 9000 chain |
HTTP to B:8080 | the HCM | 1 ms |
| L1 | HTTP to B:9000 | the passthrough, not the HCM | 2 ms | |
| L1 | raw bytes to A:9000; HTTP bytes to A:9000 | A's TCP chain, both | 2 ms | |
| L2 | per port: an HCM chain for 8080 and for 9000 (destination port + sniffed HTTP), A's TCP chains, the port-less HCM chain, the passthrough | HTTP to B:8080, Host port-less or :8080 |
table 8080, B's virtual host | 1 ms |
| L2 | HTTP to B:9000, Host port-less |
table 9000, B's virtual host | 1 ms | |
| L2 | HTTP to B:8080, Host: b:9000 |
table 8080, no virtual host: the table's 421 |
1 ms | |
| L2 | raw bytes to A:9000; HTTP bytes to A:9000 | A's TCP chain, both | 2 ms | |
| L2 | non-HTTP bytes to B:9000 | the passthrough | 2 ms | |
| L2 | HTTP to B:7777 (no chain names 7777) | the port-less HCM chain | 1 ms | |
| L2 | h2c with prior knowledge to B:8080 | table 8080 (the proxy answers with HTTP/2 SETTINGS) | 1 ms | |
| L2 | a TLS ClientHello to B:8080 | the passthrough | ||
| L2 | GET mykey\r\n (no HTTP version) to B:8080 |
the passthrough: not detected as HTTP | 1 ms |
The second row is the finding. Envoy's filter-chain match is a tree walked one
criterion at a time (destination port, then address, then server name,
transport protocol, application protocol), and it does not backtrack: once
any chain names destination port 9000, a connection to port 9000 is matched
only among chains that name 9000. One raw-TCP declaration of a port number by
any service therefore takes that port number away from the port-less HCM chain
for every service on that listener, and the HTTP request leaves through the
passthrough, outside the mesh. TCP chains are built for every mesh Service
with an address, not only for the pod's dependencies
(cache/capture.go:329-336, read), so the effect is mesh-wide. This holds on
main today for any port number that some service declares as raw TCP (a
non-primary =tcp port, or a TCP primary), and it is reported as a finding
with this pull request. The repository uses filter_chain_match; the unified
filter_chain_matcher is not used anywhere (grep) and was not measured.
Keeping one HCM chain would therefore need a per-port HCM chain anyway for every port number that any service declares as raw TCP. Two rules for "which port" depending on what an unrelated service declared is worse than either rule alone, so the per-port layout is adopted for every HTTP port number.
Also measured: the route table itself cannot branch on the destination port
(a virtual-host matcher on DestinationPortInput is rejected at load:
"Route table can only match on request headers"), so the port has to be chosen
by the chain, with a route table per port.
Amendment: one HTTP chain, selected by a matching tree¶
Chart 3.0.0 shipped the per-port layout above. On the reference cluster it
measured as 1.6× the capture listeners' LDS size, an RDS apply 8–10× slower,
a slower successor start at a proxy hot restart and +34% proxy memory: a node
held pods × (H + 1) HCM configurations. It was replaced by one HTTP chain
per listener, with the same answers.
Both findings above were about the legacy filter_chain_match, and both have
an answer that does not need a chain per port:
- The listener's
filter_chain_matcher(the generic matching tree) backtracks. A branch that ends without a match hands the connection to theon_no_matchof the map it came from (source/common/matcher/exact_map_matcher.h, at the pinned proxy). So "the destination address is a service's, the port is not one of its raw-TCP ports and the server name is none of its TLSRoutes" falls back to the HTTP chain instead of the passthrough. Measured on the pinned proxy (agent/test/mtlspool/onehcm_probe_test.go): every case of the L1/L2 table below, and the same chains underfilter_chain_match(control 1) get the five trap cases wrong. - The route table cannot match
DestinationPortInput, but it can match filter state. The HTTP chain stores the connection's destination port (%DOWNSTREAM_LOCAL_PORT%, the restored original destination) in filter state underaether.dialled_port, connection-level and never shared upstream, and each route of the table matches it withRouteMatch.filter_state: a stream's filter state falls back to the connection's (source/stream_info/filter_state_impl.cc,getDataReadOnlyGeneric; the route reads it inRouteEntryImplBase,source/common/router/config_impl.cc), measured on the pinned proxy (mtlspool/onehcm_filterstate_route_test.go). A client cannot set it. A chain that does not set it gets the answer of a port no service declares: every mesh name is refused, none is delivered (control 2).
The agent still builds the per-port tables cap_http_<P> and
cap_http_other: they state the rule, port by port. What it publishes is one
table, cap_http, compiled from them (proxy.MergeCaptureRouteTables): one
virtual host per mesh name (and per short name), whose domains are its
spellings port-less and :*, with a plain, ordered route list and no
matcher:
- for each port D the name is served on, the name's routes for D (its route
rules, ExtensionRef and service-wide filters, imported rules, the HTTP/3
selection: everything the per-port table has), each also matching
filter_state aether.dialled_port == "D"and an:authorityregex on the Host's port suffix (none, or exactly:D;<name>:8080:8080is no such suffix); then a route forfilter_state == "D"alone: 421 port-mismatch; - last, every other port: 421 port-not-declared for a name the node knows.
The * virtual host branches on the dialled port the same way (each port's
on-demand spelling). A short name's caller-namespace matcher is flattened
too: each namespace's routes under a filter-state match on the caller's
namespace. Every list ends in a route every request takes, so no port's
routes can take another port's requests.
A service-wide filter's config stays a virtual-host config: a full
name's compiled virtual host belongs to one service, so it carries the
filter as the per-port virtual host (and the proxy port's) does, and the
proxy combines it with a rule's own config itself. The refusals in that
virtual host (port-mismatch, port-not-declared) disable it per route with a
route-level FilterConfig{disabled: true}: the route's answer to "is this
filter disabled" wins over the virtual host's
(RouteEntryImplBase::filterDisabled, source/common/router/config_impl.cc),
and a disabled filter is not created for the stream
(source/common/http/filter_chain_helper.cc); measured live for
header_mutation, rbac and ext_authz, with a red control
(mtlspool/onehcm_levels_live_test.go). A short name's virtual host is
shared by several namespaces' services, so there the two levels are folded
onto each route (extensionfilter.CombineLevels: header_mutation applies the
route's mutations before the virtual host's; ext_authz never inherits the
virtual host's service override), and a pair no single config can carry in
the proxy's order is refused 500 filter-levels rather than run differently:
header_mutation applies each level whole, route first, and within a level its
request mutations before its query mutations, each query mutation parsing and
re-rendering :path (header_mutation.cc), so a rule that mutates the query
under a service that mutates a request header or the query cannot be folded.
A full name is never folded (the compiler folds only a virtual host whose
answers disagree on the virtual-host config, which no builder produces).
The answer for every (name, dialled port, Host spelling) is the per-port
table's: status, x-aether-error, the route's target and the filters and
rules applied to it (a test compares the first route every request takes,
on every spelling of every name, every port of the mesh, one other and none,
for every path and header a route matches: cache/onehcm_equivalence_043_test.go),
and every table the agent's tests read is checked the same way. Two answers
changed on purpose with this amendment: a refusal's body names no dialled
port (the reason header is the contract), and on a port the name is not
served on the answer is port-not-declared whatever port the Host names (it
was port-mismatch when the Host named another port).
The proxy port kept its own chain and table (cap_http_18081, the loopback
table's virtual hosts) through chart 3.0.1. Chart 3.0.2 folds it in: see
"Folding the proxy port into cap_http" below. Port 18081 itself stays,
permanently, the HTTP proxy port; only its separate chain and table went.
Folding the proxy port into cap_http (chart 3.0.2)¶
A dial to 18081 is one more dialled-port branch of cap_http's table. The
table of that branch is the one cap_http_18081 was
(BuildProxyPortCaptureRouteConfiguration: the loopback table's virtual
hosts, their short names, and a floor of its own that names any port's
cluster on demand, refuses :18081 and :18082 by name, and passes
everything else through, or 404 in scoped capture), handed to the compiler
as perPort[18081]; the chain records the dialled port for 18081 as for any
other port, and the matching tree no longer has a branch for it (scoped
capture keeps its 18082 blackhole). The listener has one HTTP connection
manager and the node one capture route table fewer.
What the compiler had to learn. Until then every per-port table answered all
the names of one compiled virtual host alike for one suffix (<name>,
<name>:<P>, <name>:*), so a route told suffixes apart by a small
:authority regex and nothing else, and a group whose names disagreed was
refused. The proxy port's table did not (through chart 3.0.2): it knew a service by its mesh
name only, so for the same suffix the mesh name gets the service and its
cluster.local and namespace names get the floor. The compiler now
partitions a group's names, per dialled port, into classes a suffix-only
block answers exactly (nameClasses: a name joins a class when every suffix
they share has one answer, and a port only one of them spells is answered by
the other's :* answer where that one has a wildcard). One class is the case
of every per-port table, and its routes are what they were. With several,
each name of every class but the largest gets its own routes under an
:authority condition on the name, an exact match (<name>) or a prefix
(<name>:), case-insensitive, never a regex: a name can be longer than the
proxy's 100-instruction regex budget. Names hold no :, so no Host meets
two names' conditions, and every name's routes end in a route each of its
Hosts takes. The largest class follows with no name condition. A construct
the compiler does not model is still an error, and the table is then not
published (the fail-closed RefuseCaptureMeshNames).
The dialled-port blocks of a compiled virtual host are disjoint (each
matches one dialled port) and each ends in a route every request of its port
takes, so their order is free; only the other ports' block must be last. The
proxy port's block is placed after the declared ports' (mergePortOrder), so
a request to a service's own port is matched against fewer routes first.
The capture tables are memoized while their inputs' generations hold
(#1105, #1115). The proxy port's table is built from the cluster entries'
virtual hosts, which no such generation covers, so the sha256 of its
deterministic encoding is part of the key: one hash per build, which is what
the version memo paid for cap_http_18081 when it was published on its own
(a new object every build).
Gates, each seen red first: the compiled table against the proxy port's own
table for every class of Host on 18081 (a declared port, none, 18081,
18082, an undeclared port, upper case, the cluster.local and namespace
names, short names with and without the caller's namespace, a cold name, a
name that is not the mesh's), in redirect-all and scoped capture, with a
service-wide filter and rules (cache/onehcm_equivalence_043_test.go,
proxy/capturemerge_test.go, and every table the cache's tests read, the
walks included); the listener's shape (one connection manager, no
cap_http_18081 anywhere in the snapshot); the agent's listener and table
under envoy --mode validate; and the live probe through the pinned proxy
(mtlspool/onehcm_probe_test.go, X1-X9; X10 since chart 3.0.3).
The worked example (200 services, HTTP on 80, 8080 and 9090, 20 raw-TCP
ports beside an HTTP declared port, 50 pods), measured: 21 chains per pod
listener with the default (22 before), 1 connection manager per pod (2), 2
route tables per node (3); the capture listener 24 KiB per pod (26), and the
capture tables 2,765 KiB (2,772 for cap_http and cap_http_18081), with
401 virtual hosts where the two had 1,779.
TLS carrying a TLSRoute's server name to a port the service itself serves as HTTP (or to 18081, when it serves any HTTP) leaves through the passthrough, as it did with the per-port chains: the tree does not test the server name on those ports. That is the owner's ruling for this change, kept from 3.0.0.
A request with no path (a CONNECT without :path) is matched by a
CONNECT matcher route and by no other on the pinned proxy: with :path
absent, VirtualHostImpl::getRouteFromRoutes skips every route whose
supportsPathlessHeaders() is false, and only ConnectRouteEntryImpl's is
true. A regex or an empty prefix does not take it (seen with the pinned
binary). No route the agent builds is a CONNECT route, so no such request
takes any route of a compiled table, as of its own per-port table. Until
chart 3.0.3 the test evaluator (xdseval) let a regex or an empty prefix
take it; the "fall-through" chart 3.0.2 guarded against was that model's.
Three guards stay, for a CONNECT route should one be built
(takesAPathlessRequest); none adds anything to a table the agent builds
today:
- between name classes, the largest class is name-conditioned too (chart 3.0.2);
- within a class, the default port answer is conditioned on the Host's suffix
being none of the other answers' (chart 3.0.3): one inverted
:authoritymatcher per other answer, each the regex that answer's routes already carry. One regex over all of them exceeded the proxy's RE2 budget of 100 at about two dozen ports, and the proxy refused the whole table; - the other ports' block, which matches no dialled port, takes a request
without a
:pathby such a route only when the dialled port is none of the ports with a block of their own (chart 3.0.3). A filter-state match cannot be inverted, so this is one small regex per piece of the complement of those ports (complementPatterns). On a connection with no dialled port (no chain the agent builds) such a request gets no route: it fails closed.
envoy --mode validate accepts both chart-3.0.3 shapes at 32 ports
(envoy_validate/capture_pathless_budget_test.go).
Kubernetes names on the proxy port (chart 3.0.3)¶
Through chart 3.0.2 the proxy port answered a service by its mesh name only.
On the reference cluster, from a meshed pod,
Host: <svc>.<ns>.svc.cluster.local:8080 sent to <mesh name>:18081
answered 503 (the passthrough), where <svc>.<ns>.<mesh domain>:8080
answered 200. The owner's decision: the proxy port answers the Kubernetes
names exactly as the mesh name.
- Where. The loopback table's virtual hosts claim, for every domain
<mesh name><suffix>, the service's Kubernetes names with the same suffix (withKubernetesNames, applied once where the snapshot build assembles them).out_httpand the proxy port's block ofcap_httpare both built from those virtual hosts, so they cannot disagree. - Which names. The ones a dial honors (
addressNames, which the dialled ports' tables use as well):<svc>.<ns>.svc.cluster.local,<svc>.<ns>.svcand<svc>.<ns>, for a service whose Service the mesh generated or that a route names. The short name keeps its own virtual host (by the caller's namespace), as before. - Ports. A generated Service's ports are the mesh ports:
<k8s name>:<P>is portP's answer, and a name without a port is the mesh name's (port 80, or421 port-not-declared). A user's Service with a known port map: the port inHostis the Service's port, translated through the map, exactly as on a dial (the mesh name's domains already spell the Service's ports). - Unanswered. A mesh service behind a user's Service that no route names
(
meshOnly) keeps its Kubernetes names unanswered on the proxy port: they get the floor, as on a dial (owner's decision). So does a service the node holds no endpoint of and knows no ports of: its mesh name is parked for the cluster it spells, and a Kubernetes name spells none, so its Kubernetes names get the floor until the node knows the ports. - One class. In the proxy port's block of
cap_httpa service's mesh and Kubernetes names now get the same answer for every suffix. They fall into one name class (nameClasses) and need no name condition, socap_httpshrinks.
Measured with the unit-test harness: services with HTTP on 8080 and 9090, generated Services, redirect-all, one run each, chart 3.0.3 against the compiler of chart 3.0.3 without this change.
| services | cap_http |
out_http |
out_http domains |
cap_http compile |
|---|---|---|---|---|
| 60 | 1,027,794 → 953,014 bytes (−7.3%) | 518,985 → 535,665 bytes (+3.2%) | 481 → 1,201 | 128 → 145 ms |
| 2,000 | 34,366,034 → 31,860,474 bytes (−7.3%) | 17,390,785 → 17,985,465 bytes (+3.4%) | 16,001 → 40,001 | 2.49 → 2.46 s |
The compile times include the constant-time wildcard lookup of chart 3.0.3
(mergeTable.pick), which took the 2,000-service compile from 25.2 s to
2.49 s, byte-identical output.
Gates, each seen red first: on both tables, every Kubernetes name of every
fixture service with every port suffix (:8080, none, :18081, an undeclared
port, mixed case) answers as the mesh name (cache/proxyport_kubernetes_names_043_test.go);
<svc>.<ns>.svc.cluster.local:8080 reaches port 8080; a user's Service's
:8000 reaches pod port 8080 and its :8080 is refused; meshOnly names
get the floor; and every published table claims each domain once with no
claim dropped. The fold's equivalence oracle (the proxy port's block against
the proxy port's own table) stays green, with its mutation on the name
condition replaced by one on the port condition. Through the pinned proxy,
a dial to 18081 with the cluster.local name reaches the service's port, in
upper case too (mtlspool/onehcm_probe_test.go, X7 and X10; X7 expected the
passthrough through chart 3.0.2). envoy --mode validate accepts the agent's
tables; the per-port fixture keeps the name-conditioned shape in front of it.
The chains of a capture listener¶
| Chain | Selected when (filter_chain_matcher) | Does |
|---|---|---|
cap_tcp_<cluster>_<P>, one per (service, raw-TCP port) |
destination address is the service's, destination port P, and the server name is none of its TLSRoutes |
tcp_proxy to tcp:<fqdn>:<P> |
cap_tls_<cluster>_<i>, one per TLSRoute rule of a service |
destination address is the service's and the server name is one of the rule's, on any port the service does not serve as HTTP (and not 18081 when it serves any) | tcp_proxy to the rule's backends |
cap_http |
anything else that is cleartext (raw_buffer) sniffed http/1.0, http/1.1 or h2c, on any port, 18081 included |
records the dialled port; HCM with route table cap_http, which branches on it (18081's branch is the loopback table's virtual hosts) |
| default | nothing above | passthrough to the original destination |
The table above is the listener since chart 3.0.2. From the amendment
through chart 3.0.1 it had one more chain, cap_http_18081 (cleartext HTTP
on port 18081, an HCM with the loopback table's virtual hosts under that
name), and cap_http took every port but 18081. Until the amendment (chart
3.0.0) it was:
| Chain | Match | Does |
|---|---|---|
cap_http_<P>, one per distinct HTTP port number P in the mesh |
destination port P, raw_buffer, sniffed http/1.0, http/1.1 or h2c |
HCM with route table cap_http_<P> |
cap_tcp_<cluster>_<P>, one per (service, raw-TCP port) |
destination port P, the service's address /32 |
tcp_proxy to tcp:<fqdn>:<P> |
cap_http_other |
raw_buffer, sniffed HTTP, no destination port |
HCM for HTTP on a port number no service declares as HTTP: refuses mesh names, passes everything else through |
| default | none | passthrough to the original destination |
"Port number in the mesh" is read from the Services (generated ones carry
real ports from Phase 1; a user Service that owns a mesh name contributes its
own ports), which the agent already lists for every namespace
(capture/reconciler.go:145, read). It is not demand-scoped, so a first call
to a service the pod has never used still finds its port's chain.
Inside the tree for port P, an address with a /32 chain is raw TCP whatever
the bytes are, and every other address is offered to the HCM chain if the
bytes are HTTP and to the passthrough if they are not.
Names¶
A mesh service answers to two families of name, with or without :<port>:
- the mesh name
<svc>.<ns>.<mesh-domain>; - the Kubernetes names
<svc>.<ns>.svc.cluster.local,<svc>.<ns>.svc,<svc>.<ns>, and the bare<svc>when exactly one namespace owns it (today's rule,cache/capture.go:1144-1156, read).
The rule is the same for both. Today it is not: a Kubernetes name with a
non-primary port reaches the primary port (#1682, measured on kind), because
the known-target route accepts any :port and sends it to the default
cluster. Phase 1 fixes #1682, and under this layout it cannot recur: the
request to :9090 is in table 9090, which has no route to anything else.
The rule, per entry path¶
- (a) Captured. The pod dialled a Service address and port
D. Tablecap_http_<D>holds, for each service with HTTP onD, one virtual host whose domains are every name of the service, port-less and with:D. (Since the amendment above, that table is the branchDof the one published table,cap_http.) - (b) Loopback. The pod dialled
127.0.0.1:18081and wrote the service intoHost. No capture rule is involved (loopback is accepted before the marks), so this is identical with capture on and withcapture.aether.io/redirect-all: "false". There is no dialled service port, so the port inHostis the port. Mesh names only, as today. - (c) Edge to a mesh backend. The request's authority selects nothing in the
mesh: the virtual host comes from the Gateway listener's hostnames and the
port from the route's
backendRef.port, through the port map.
| Request | (a) captured, dialled D |
(b) loopback | Today |
|---|---|---|---|
Host: <name>:P, the Service declares P as HTTP |
P = D: port D. P ≠ D: 421, x-aether-error: port-mismatch |
port P |
mesh name: reaches P, without GAMMA rules unless a parentRef names P (inferred). Kubernetes name: the primary (#1682) |
port-less Host: <name> |
port D, if the Service declares D as HTTP |
port 80 if the Service declares it, else 421, x-aether-error: port-required |
the primary, on both paths |
the Service does not declare the port as HTTP (D on the captured path, P on loopback) and has no raw-TCP chain for it |
421, x-aether-error: port-not-declared |
421: port-not-declared, or port-not-http if P is raw TCP or UDP |
mesh name: 503 after the 2 s on-demand timeout. Kubernetes name: the primary, silently |
D is a raw-TCP port of the Service |
the TCP chain: the bytes go to the port, Host is never read |
not applicable | the same for a non-primary TCP port |
| non-HTTP bytes to a port the Service declares as HTTP | the passthrough | not applicable (the listener is HTTP) | the same |
| a name that is no mesh service | passthrough to the original destination | 404 |
the same |
Every refusal is an immediate direct_response, status 421 Misdirected
Request, with one of four x-aether-error values and a body listing the
service's ports and classes. A refusal, not a passthrough, because passing an
HTTP request for a known mesh service to the cluster's own routing fails open.
(The existing no-http-port answer for a service with no HTTP port at all
stays through Phase 2 and becomes one of these in Phase 3.)
What changed from the first version of this table, and why it is better:
- A port-less
Hostmeans the port that was dialled, as it does to every HTTP server. Dialling 8080 with a port-lessHostworks. Port 80 is not special on the captured path, andport-requireddoes not exist there. - A
Hostport that disagrees with the dialled port is refused, not obeyed. The first version letHostwin, which meant a request could be answered by a port the client never connected to. :schemeis irrelevant on the captured path: nothing is defaulted.- Loopback is the one place a default is needed, because nothing was dialled: port-less is port 80 when the Service declares it, else refused (decisions 5 and 9).
Conformance. The dialled port is the Service port. http://echo/
dials 80, lands in table 80, matches the virtual host of echo, and the
route's cluster comes from the Service port map (80 → targetPort 8080 →
<fqdn>:8080). Tables are keyed by Service port numbers, which for a
generated Service are the pod's.
443. The generated Service's https 443 port is a placeholder, like http
80 (generator.go:157-176, read), and goes in Phase 3. After that,
https://<svc>/ reaches an application only if a pod declares 443 as raw TCP:
an ordinary TCP port carrying the application's own TLS inside the mesh's. A
TLS connection to a port declared HTTP is not raw_buffer, matches no HCM
chain and is passed through (measured). 443 is never a default for anything.
UDP is a different listener with no filter chains. None of this applies to it (fact 3).
What sniffing is trusted for¶
Sniffing chooses between the HCM chain and the passthrough for a connection to a port number that some service declares as HTTP, on an address that has no raw-TCP chain for it. It is never the source of a port's class:
- A port declared raw TCP is
tcp_proxywhatever the bytes are (measured: HTTP bytes to A:9000 reach A's TCP port). - A port number that is HTTP for one service and raw TCP for another on the same listener is unambiguous: the address decides (measured, L2).
- h2c with prior knowledge, which is how cleartext gRPC arrives, is detected (measured). HTTP/1.1 is detected from its request line.
- A client that does TLS itself is not HTTP to the mesh: passthrough, or a
TLSRoutechain on the service's address if one exists. - Bytes that only resemble HTTP (a request line with no version) are not detected and are passed through (measured). Bytes that are a valid HTTP request line on a port declared HTTP are HTTP, by declaration.
- Non-HTTP bytes to a port declared HTTP are passed through, not refused. For a generated Service that ends at a Service port with no endpoints, which the cluster's routing rejects (inferred). For a user Service with a selector it reaches the pod without the mesh. That is true today and is not closed here: closing it needs an HCM chain per (service, port), which is the cost this layout avoids. Policy that must not be bypassed belongs on the destination's inbound side, which no client byte pattern can avoid.
The inspectors and server-first protocols (measured)¶
http_inspector and tls_inspector wait for client bytes before any chain is
chosen. A server-first protocol (MySQL, SMTP) sends none.
| Connection | Today's listener | Per-port layout | With the inspectors disabled for that destination port |
|---|---|---|---|
| server-first to A:3306, a declared raw-TCP port | first byte after 1.00 s | 1.00 s | 1 ms |
| server-first to C:5432, not in the mesh | 1.00 s | 1.00 s | 1.00 s (5432 was not disabled) |
| a 4-byte first write to A:9000 | 1.00 s | not re-measured | |
| a silent client on B:8080, an HTTP port | 1.00 s, then the passthrough |
The shipped listener disables nothing per port: it bounds the wait at 1 s and
continues (proxy/capture.go:200-215, read). So every server-first connection
through the mesh pays one second today, declared or not, and the per-port
layout neither adds to that nor removes it. A listener filter's
filter_disabled predicate does remove it, and it does see the recovered
destination port (measured, layout L3). The predicate matches a port, not an
address, so it can only be used for a port number that no service declares as
HTTP and no TLSRoute needs. This is an improvement that the per-port set
makes possible; it is listed in Phase 1 as optional and gated on its own leg.
Scale¶
S services with an address, H distinct HTTP port numbers in the mesh, T
(service, raw-TCP port) pairs, N pods on the node. Route tables are shared by
name across the node's pods (cap_http is one resource, proxy/capture.go:29,
read).
| Chains per pod listener | HCM configs per pod | Route tables per node | |
|---|---|---|---|
| Today | 1 + T' + default (T' counts three chains for a TCP-primary service) |
1 | 2 (cap_http, out_http) |
| One HCM chain (first version) | 1 + T + default |
1 | 2 |
| Per port (chart 3.0.0) | H + 1 + T + default |
H + 1 |
H + 3 (cap_http_<P> × H, cap_http_other, out_http, and cap_http_18081) |
| One HCM, matching tree (the amendment, through chart 3.0.1) | 2 + T + default |
2 | 3 (cap_http, cap_http_18081, out_http) |
The proxy port folded into cap_http (chart 3.0.2) |
1 + T + default |
1 | 2 (cap_http, out_http) |
Worked: 200 services, 3 HTTP port numbers (80, 8080, 9090), 20 raw-TCP pairs,
50 pods on a node. Chains per pod: 22 today (if none of the 20 is a primary),
22 with one chain, 25 per port. Per node: 1,100 against 1,250 (+14%).
HCM configs per node: 50 against 200. Route tables per node: 2 against
6, 5 after Phase 3. H grows with distinct port numbers, not with
services: 2,000 services on the same three numbers add nothing to the chain
count and lengthen the three tables.
What changes on RDS:
- Fan-out is narrower. A route change for one service rewrites the tables of that service's ports, not one table every HCM on the node holds.
- Each table is a resource with a lifetime. A chain that names a route
table the snapshot does not carry leaves the listener warming until the
initial-fetch timeout, and a table is torn down with the last chain that
names it (#817, #844). A port number appearing in or leaving the mesh must
add or remove its chain and its table in one snapshot. The ACK tracker
follows
Hmore resources. - The first-use path is not designed. Today's catch-all turns
:authorityinto a cluster name verbatim. With a port-lessHostin tablePthe cluster is<fqdn>:<P>, which no header spells. The intended answer is an explicit virtual host per mesh Service in each of its ports' tables (the Service list is not demand-scoped), leaving the on-demand filter to fetch the named cluster; whether the pinned proxy's on-demand filter does that for a route that names its cluster directly was not established. - GAMMA rules per port are what this proposal wants in any case (fact 1, row 10).
What stays¶
- Pod annotations do not change.
endpoint.aether.io/portsdeclares ports with per-port suffixes;endpoint.aether.io/protocolstays the default class for unsuffixed ports (a pod-level shorthand, not a service fact);endpoint.aether.io/portstays an accepted way to declare one port and names the liveness probe port (decision 6). No harness-contract entry for an annotation changes. - The registry API, the wire, the etcd key layout and the registrar snapshot
key.
ServiceEndpoint.portstays on the wire for older readers. - The mesh hop:
pod_ip:18008, SNI selects the port, EDS is a membership filter. The waypoint's SNI form. Mesh DNS. UDS delivery. - TCP capture by TPROXY to 18001. East-west QUIC on 18008.
The legacy spellings during Phases 1 and 2¶
They keep working where they work on main, and stop being broken where they
are broken, by one rule that names no service-level protocol:
The legacy target of a service is the port its pods declare in
endpoint.aether.io/port.:18081and the bare name reach it if that port is declared HTTP;:18082reaches it if that port is declared raw TCP (or UDP, for the UDP listener). Otherwise the spelling names nothing.
On the capture listener these are a dial to port 18081 (its own table,
cap_http_18081) and a dial to port 80 of a service that does not declare 80
(a legacy virtual host in table 80); on the loopback listener, a port-less
Host or one that names :18081.
- For a single-protocol service this is exactly today's behaviour.
- For a mixed service whose legacy target is HTTP it is today's behaviour too: legs M0 to M5 keep their meaning.
- For a mixed service whose legacy target is raw TCP (#1672),
:18082starts working, and:18081and the bare name are refused at once with421andx-aether-error: port-required, the body listing the service's ports, wheremainanswers 502. It is one of the refusals of the addressing rule, with the same body, so it does not describe a primary and does not have to be unlearned. - No any-port chain for a service that declares any HTTP port. On
mainthe portless/32chain of a TCP-primary mixed service outranks the HCM and takes the service's HTTP ports (measured in #1677 on the kubernetes backend).
The alternative, making the legacy spellings work for mixed services by a new rule (lowest port of the class, as 037 wrote and never built), adds semantics to something scheduled for deletion. Refusing is the least surprising: a client either gets what it got before or an answer that tells it what to dial.
Rulings after acceptance¶
Decisions the owner took while the work was being built. Where one differs from the text above or below, the ruling holds; the Phases section is rewritten with the last change of the release.
2026-10-12: a short name resolves in the caller's namespace. A Host with
no namespace (echo, echo:8080) means echo.<the caller's namespace>, as
the cluster's DNS search does. Never a service in another namespace, unique or
not; refused (421, x-aether-error: name-not-in-namespace; superseded the same day, see the next ruling) when the
caller's namespace has no mesh service of the name. This replaces "a bare name
that two namespaces share is refused" (decided 2026-10-11) and the older
"served where exactly one namespace owns it". Both paths; the per-port rules,
service-wide filters and HTTPRoute rules apply to the short name exactly as to
the full one.
- Mechanism. The route tables are shared by every pod of a node, so the
caller is read off the connection: each pod's HTTP chains store the pod's
namespace in filter state (
aether.source.namespace), and a short name's virtual host has amatcherwhose input is that key and whose arms are the routes the full name of each namespace has in the same table. A client cannot set filter state. Measured on the pinned proxy before it was built. - Cost. No new table. Per table, one virtual host per (short name, authority suffix), with one arm per namespace that has the name; one more network filter on each pod's HTTP chains.
- Conformance evidence (corrected). The mesh conformance suite at v1.6.3
sends short authorities:
mesh-basic.go(Host: "echo"),mesh-frontend.go("echo-v2"),httproute-weight.go("echo"), and others, always from a client in the Services' own namespace.echothere is not a mesh service of its own: it is a Service that an HTTPRoute is parented to, or, with no route, a name the mesh passes through. The suite does not depend on a short name reaching another namespace.
2026-10-12: a cold request is strict. A request is delivered only where it
would be once the node holds an endpoint of the service. http://svc is
svc:80: it asks for the cluster of port 80, and fails when the service does
not declare 80, whether or not the node already knows the service. The bare
cluster name (<fqdn>, the internal name of the cluster of the port the pods
declare) is not reachable from an authority a client typed. <name>:18081 has
no rule of its own: 18081 is a port no service declares, and gets the general
port-not-declared.
- Mechanism. The agent reads a service's real ports from its generated Service, which it already watches, and routes and refuses by them while it holds no endpoint. On-demand routes name a cluster only from an authority with a port.
- The proxy's port. No cluster is named
<fqdn>:18081(the alias of chart 2.6 is removed), and an authority that names 18081 or 18082 is refused at once on the tables whose on-demand route takes the port from the authority, for a name the node knows nothing of as well: it is never parked, so no service that enters the node's scope meanwhile can answer it. - Reserved. 18081 and 18082 cannot be declared as an application port, in
any protocol. The controller's pod webhook denies such a pod; for one that
runs anyway the registrar does not write the port onto the Service and the
agent does not serve it, each with a WARN that names the service and the
port. One definition (
mesh.ReservedPort), three enforcers.
2026-10-12: a short name that is not a mesh service in the caller's
namespace is not a mesh name for that caller. It replaces the refusal
(421 name-not-in-namespace) in the ruling above. A name without a namespace
means <name>.<caller's namespace>; when that namespace has no MESH service of
the name, the request is treated exactly like any other destination that is
not the mesh's, whether or not another namespace has a mesh service called the
same: on a captured dial, the passthrough to the original destination; on the
proxy port, where nothing was dialled, what any such name gets there (404).
A connection that carries no caller namespace is still refused
(caller-namespace-unknown): that is a fault state, not a name that is not the
mesh's. The passthrough goes to the address that was dialled and so cannot
reach another namespace's mesh service.
- Mechanism. The short name's virtual host asks first whether the caller's namespace is on the connection (refused when it is not), then picks that namespace's arm, or, when it has none, a copy of the routes of the table's own catch-all. The short names are built by each table's builder for that reason: the loopback table and the proxy port's capture table have the same virtual hosts and different catch-alls.
After the third review (2026-10-12). What the rulings above came to mean where they met another feature:
- Filter levels. A short name's arm has one place for a filter's per-route
config, the route; the service's own virtual host has two, and the allow-
listed filters do not combine them alike.
extensionfilter.CombineLevelssays, per filter, what the two amount to, and a test over the allow-list keeps a filter from being added without saying it. - A user's Service. The port in an authority is the Service's port on the proxy port as on a dial; a pod port's number is no port of the name. Cold, the name is routed and refused by the Service's port map.
- Imported rules apply on every known HTTP port while the port the pods declare is not known. Interim: the projection should carry the parent port.
- One limit: a service known by name only whose route has a backendRef (the mixed-version window with a chart 2.6 registrar). See the runbook.
2026-10-12: port_protocols is required. A registry row that does not
name every port it lists, each with a protocol, is not served: no endpoint, no
port, no class, on any path. The fallback this replaces ("a row without the
field, and a port its map does not name, take the protocol of the listing the
row was read from") had three readers that had to agree (the agent's port
derivation, its HTTP membership test, the registrar's Service generator) and
existed for rows written before proposal 037; no writer since chart 0.93.2
produces one.
- Where it is enforced. One definition (
endpointmeta.RowProblem). A write is refused at the registrar'sRegisterEndpoint, whatever the backend, and by the etcd backend and the registrar client themselves. A row already in etcd is filtered at the registrar when it lists the store (chosen over listing it and leaving every reader to refuse it: the snapshot the agents watch and the Service generator are both fed from that one read). An agent drops one it is handed anyway, in the one place every refresh reads the registry through. Each of the three says so once per row and counts it by reason. - The kubernetes backend stores nothing and derives each row from the pod's annotations with the parser the agent registers with, which names every port; a test walks the annotation shapes and holds that.
- Other clusters. A peer's row without the field, in a shared or mirrored etcd, is not served by a cluster of this release; the peer is unaffected.
2026-10-12: a retained HTTP virtual host is kept against a removal and replaced at once against a replacement. The grace of #1635 (the cluster and virtual host of a service whose HTTP rows left the registry stay, empty, for 90 s) is for a loss with nothing saying what the service is now. With positive evidence of the service's shape that no longer declares the port as HTTP, the entry and its virtual host go in the same refresh and the port is answered by what the evidence says, same number or another. No HTTP request is delivered to an endpoint whose row does not declare that port as HTTP, grace or not.
- The evidence, in order: the rows the refresh lists of the service under any protocol (they are typed, by the ruling above); with no row, the real ports of its generated Service when it declares any. A Service whose last endpoint left declares none and is no evidence.
- What this changed in #1635's own tests. The cases with no evidence (every listing of the service gone) are as they were. "The HTTP listing goes and the raw-TCP or UDP one stays" is a replacement by this ruling's definition: the rows that stay do not declare the HTTP port. Kept for the grace, the HTTP cluster resolved its endpoints through the bare service name, which the live floor publishes, so requests reached the remaining pods on a port they do not serve as HTTP; those tests pinned exactly that as "the port alias and the QUIC twin carry the live endpoints". They now expect the entry gone and keep their assertions on the one load assignment per name.
- The state that cannot be told apart. A registry that returns no HTTP row for a service while returning its rows under another protocol is read as the truth. Were it a fault of the registry, the port is refused until the rows are back. A listing that errors abandons the refresh and changes nothing.
- The Service as evidence can be older than the entry. It is derived from the registry by the registrar and read by the agent on another path; an agent that has seen a newer registry state than the Service reflects can replace an entry on the Service's older word, until the Service catches up. Not measured; the window is the registrar's reconcile lag.
2026-10-12: a mesh service fronted by a user-owned Service is not a captured mesh name. Left as it is. The registrar generates a Service only for a name no other Service owns; a dial to a user's Service is a dial to an address the node does not capture as the mesh's, and leaves through the passthrough. The mesh name and the proxy port reach such a service through the mesh, by the user Service's port map. A service-wide HTTP filter that targets the Service changes none of it: the filter is in force on the mesh name and the proxy port, and the Service's Kubernetes names, the one without a namespace included, still pass through unfiltered. That half needed code: the short name had become a refused mesh name as soon as the service had a filter (found by the fourth review), and a dialled-port table now derives no short name from a name it claims by its mesh name alone. An HTTPRoute that names the Service does make its Kubernetes names the mesh's. The runbook states the limit.
2026-10-12: 18081 is the proxy port, and the authority names the target.
18081 is the mesh's HTTP proxy port, permanently, and a port of no service.
A client that connects to it (the loopback listener, http_proxy, or a dial
to <name>:18081) is asked, by the Host header or the absolute request
line, for a service AND a port: Host: <name>:<port> reaches that declared
port, a Host without a port is port 80, and Host: <name>:18081 asks for a
port no service has and is refused (421 port-not-declared). No cluster is
named for it, no pod may declare it (nor 18082), and "the port the pods
declare" is not something an address can mean.
A reference is strict and visible (decided with the proposal, restated
2026-10-12 because older text below says otherwise). A backendRef whose
port is not an HTTP port of its Service is invalid: its share of the rule is
answered 500 and the route's status says ResolvedRefs=False
(BackendNotFound, UnsupportedProtocol). A parentRef to such a port
attaches the route to nothing and the status says Accepted=False
(NoMatchingParent). The table and the status are built from the same facts:
whenever the data plane answers 500 for a reference the status says False,
and where no node can know (a Service of the user's with no registered pod)
none answers 500 and the status says "Not checked". For a user's Service the
facts are what the pods behind it declare, and the status writer does not
read the registry for them: it is handed the class of each pod port by its
own node's snapshot cache, the value that node's tables were built from (see
the ruling of 2026-10-13 below). "A named targetPort is reported as a WARN
only" and "any other wrong port falls back to the declared port", under
Phase 1 as built, are superseded by this. A named targetPort that does
not resolve was the last exception; it has a ruling of its own, 2026-10-13.
2026-10-13: one truth per port, for every table and for the status; and one status from many writers. A node derives the class of a port of a service once, by one function, from the registry rows it holds (the class most pods declare where they disagree; a row whose port map names a port it does not list is refused). Every route table of the node is built from that value: the loopback table, the proxy-port table, the table of each dialled port. A virtual host is marked as built from a given state of the rules and of the ports only by the code that built it, entry by entry: a refresh that carries an entry over (a retained one), or that forgets a service's ports after building, does not vouch for it, and the next snapshot rebuilds it. The route's status is computed from the same value, handed out by the cache, and is judged again when the value changes, by an event, not on a timer; a status write that fails is retried.
A route has one status and every node's agent writes it. Each condition is a
function of facts, computed anew on every reconcile; nothing a status said
before is carried forward, with one exception. What the pods behind a port of
a user's Service serve is known only to a node that holds the service. That
verdict, per (Service, Service port, pod port) and per generation of the
route, is recorded in the route's own status (a condition of type
gateway.aether.io/PodPortsJudged on aether's parent entry), which is the
one place every agent reads. A node that holds the pods writes its own
verdict; a node that does not computes its conditions from the recorded one,
a True like a False. So a node that knows beats one that does not, a False
goes when its cause does, and nodes that hold the same rows (or none) reach a
status that none of them writes again. A service whose Service says nothing
of its ports (no Service object yet, or a generated one with no real port) is
judged the same way, by the ports its pods declare.
Writers that do NOT compute the same status cannot settle, and are bounded instead: two nodes that hold different rows of one service (a registry update in flight), and an agent of the chart before during the roll. An agent that finds its own entry overwritten with something else writes it back once at once and then no sooner than 5 s, 10 s, 20 s ... up to 5 minutes after its previous write (per route and per node: at most 5 writes in the first minute, at most one every 5 minutes from about ten minutes on; N agents, N times that), and the last word is that of the agent whose turn comes last, which is an agent of this chart once the other stops. The wait is only for writing the same status again over a writer that overwrote it: a verdict that changed is written at once, always. One thing is not claimed: for a generated Service that carries ports the status follows the Service, which trails the rows by one registrar write.
2026-10-13: a filter does not turn a working call into a refusal on a
captured dial; what such a dial does is decided by the service's own pods.
A mesh service whose Kubernetes names a Service of the user's owns, and that
no route names, is in a node's tables by its mesh authority alone, because
something (a service-wide filter) applies to it. On the table of a captured
dial an authority that names it is never refused. The request is a mesh
delivery, with the service's filters, only when the port number that was
dialled is one the service's own pods declare as HTTP and Host names that
port or none. Every other captured dial leaves to the address that was
dialled, as it did before the filter existed: any other port number, the ports
of the owning Service included, and a Host that names another port than the
one dialled. Nothing another service declares changes which of the two it is.
It is never fetched on demand without the filter. Two things are outside the
sentence in the heading and are stated, not hidden: while the node holds no
row of the service, a port-less mesh Host on a port number that has a chain
is parked on demand with the filter, where it left to the dialled address
before the filter; and the proxy port and the loopback listener are not a
captured dial to another address, and refuse there as for any service.
2026-10-13: a service's filters and rules are in force on every path that delivers to it. The worst class of defect this work produced, more than once, is a request delivered to a mesh service by a path that does not run what applies to the service: its service-wide HTTP filters (authorization among them), the rules of its routes. It happened through the on-demand catch-all (which carries neither), through a virtual host kept from an older state, and through a table built from other facts than its neighbour. The rule it is held to: for every route table of a snapshot and every authority that names a service, the answer is a delivery WITH everything that applies to the service, a refusal, or a hand-off that reaches no mesh cluster; never a delivery without. A change that adds a way to reach a service adds it to the tests that walk every table for that property.
No status-code rewrites (2026-10-12). The mesh does not turn one HTTP
status into another to signal between proxies or to a client: a refusal is a
421 (or the 500 the Gateway API specifies for an invalid reference) with
an x-aether-error header that names the reason, and an upstream's own status
is passed on as it is. Where two proxies need to tell each other something,
it is a header.
2026-10-12: the release is chart 3.0.0. The removals are made without a deprecation release, which the compatibility window allows only across a major version. Charts 2.7.0 and 2.7.1 carry numbers the work used while it was built; they were published as intermediates when its first pull requests were merged and are not releases.
2026-10-13: an unresolved named targetPort is refused. A Service port
whose targetPort is a name is resolved through the Service's EndpointSlices.
While the name does not resolve (no slice names the port yet, the slices
resolve it to different numbers, or they cannot be read) the port names
nothing: a backendRef to it is an invalid reference, 500 for its share on
every table and ResolvedRefs=False with the Service, the Service port and
the name; a parentRef to it is Accepted=False and its rule is attached to
no port. There is no fallback to the port the pods declare, which is what
such a reference reached until now and what the strict-reference ruling
forbids everywhere else. The status and the tables come from the same fact,
and a node that has not read the slices refuses too: it does not guess. The
agent judges again on the EndpointSlice's event (it watches, metadata only,
the slices of the Services whose names it resolves), so the refusal ends when
the slice appears; slices that disagree stay refused and the status says so;
a failed read refuses and is retried, with a back-off capped at 30 s. The
names are resolved for every user Service the node's tables can reach a mesh
service through (one a route names, one a filter targets, one that fronts a
mesh service the node holds), not only for a routed one: "refused until it
resolves" must not mean "refused for ever" for a Service nobody routes. This was the
one behaviour left "awaiting an owner ruling", and that heading is gone with
it.
The release¶
One release, chart 3.0.0, built as a stack of pull requests. The first
twelve were merged to main ahead of the last two; every merge to main
publishes images and a chart, so charts 2.7.0 and 2.7.1 exist and hold a part
of the addressing rule each. They are intermediates, never deployed, and not
releases.
What shipped, against the plan below:
| Planned | Shipped in 3.0.0 |
|---|---|
| Phase 1: per-port truth on both backends; the dialled port decides; the four refusals | yes |
| Phase 2: a deprecation release with a counter per legacy spelling | not built. The spellings are removed outright; the runbook says how to find their clients on 2.6.x from the access log and the any-port chain's counter |
Phase 2: the prober, the e2e suites, udsecho and the docs moved to real ports |
yes (prober chart 1.1.1) |
Phase 3: <name>:18081 and a name without a port no longer mean "the port the pods declare" |
yes. 18081 stays, permanently, as the HTTP proxy port: a client that connects to it is asked, by Host, for a service and a port. No cluster is named for it and no pod may declare it |
Phase 3: the any-port chain; aether.io/app-protocol and aether.io/port; the http/https placeholder ports |
removed |
Phase 3: :18082 for raw TCP, the Service's mesh and mesh-tcp ports, the floor cluster |
kept. A raw-TCP service is reached on its declared ports and on :18082 |
| Phase 3: scoped capture removed | kept |
Phase 2/3: UDP through EndpointSlices, the per-pod UDP listener removed |
not built; UDP is as in 2.6 |
Phase 2: per-port QUIC twins; a port on GammaBackend and on the config projection |
not built. An imported rule carries no port (the runbook says what that means while a node holds no endpoint) |
| Phase 4: the receiver-side forms kept for older sources removed | not done: nothing on the receiving side was removed |
Decided while it was built, each under Rulings after acceptance: a name
without a namespace is the caller's namespace's, and is not a mesh name where
that namespace has no mesh service of it; a cold request is strict; 18081 and
18082 are a port of no service; port_protocols is required; a retained HTTP
virtual host is replaced on evidence; a user-owned Service is left as it is.
- Harness contract: version 4 (the any-port chain's stat entry removed).
- Skew: 2.6 with 3.0 in one cluster is not a supported pair, and the upgrade is not hitless. What each mix does was measured on kind with 2.6.2 images on one side, both backends, and is in the runbook ("During the roll, and rolling back").
- Conformance, Gateway API v1.6.3, on the heads of the stack as each was finished: MESH-HTTP 7 passed, 0 failed; GATEWAY-HTTP 36 passed, 0 failed, and 10 passed, 0 failed (the two gateway runs), read from the job logs of the dispatched runs.
- Not run: the 8-hour soak the plan gated Phase 3 on.
Phases as planned (superseded by The release)¶
Kept as the record of what was accepted on 2026-10-11. The chart numbers are the plan's. None of 2.8.0 to 2.10.0 exists; 2.7.0 and 2.7.1 were published as intermediates when the first pull requests were merged and are not releases (see The release). Where this text and The release or a ruling differ, they hold.
Phase 1 (chart 2.7.0): per-port truth, no client-visible change¶
Fixes #1672 and the kubernetes-backend failure of the gated shape. After it, a call to a real port works for both mixed shapes on both backends.
Registry and registrar:
- kubernetes backend: a pod is listed under every protocol it serves a port
of.
RequireProtocolDisjointis replaced by the per-port contract and run against both backends. - The generated Service gains the real ports with per-port
appProtocol, beside the four legacy ports.applyconvergestargetPortandappProtocol. aether.io/app-protocolandaether.io/portare frozen: the registrar stamps what the 2.6 registrar would have stamped on the same backend (onetcd, the first class the service is listed under; onkubernetes, the class of the declared port). No 2.7 component reads them. They exist for 2.6 agents and are deleted in Phase 3.
Agent:
- Every reader in fact 1 takes a per-port answer from the endpoints it holds.
The capture reconciler stops reading the annotation;
PrimaryIsTCPgoes. - HTTP clusters and endpoint sets are built per (port, class): no h2 cluster
for a port declared raw TCP (
cache/cluster.go:671-686builds one today, inferred). - The capture listener gets an HCM chain and a route table per HTTP port
number (The addressing rule), each table holding a virtual host per
service with HTTP on that port, for both name families, carrying the GAMMA
rules.
out_httpgets a virtual host per HTTP port. The known-target safety net goes: its job is done by the per-port tables, which fixes #1682. The refusals for a named or dialled port (port-mismatch,port-not-http,port-not-declared). This closes the "real ports bypass the routes" gap and the passthrough of HTTP on a port number another service declares raw TCP. - Optional, with its own leg (A13):
filter_disabledon both inspectors for destination ports that no service declares as HTTP and noTLSRouteuses. - The Service port map: the agent reads the ports of the Service that owns each mesh name and resolves parentRef and backendRef ports through it.
- Destination: an explicit chain for every port, including the declared one
(
in_tcp_<pod>_<p>for a raw-TCP port instead of an HCM chain). The no-SNI chains stay. Sources keep sending what they send today. - Liveness: the probe is typed by the probe port's own class.
- Edge: per-port TCP clusters are published.
- The legacy rule above, in one function used by one reader.
What of #1677 this phase keeps, and what it must not ship:
| #1677 | Phase 1 |
|---|---|
| The measurements: both backends, both shapes, the skew table | Keep, as this proposal's evidence and as the Phase 1 gate's baseline |
e2e/multiprotocol.sh legs X1 to X3 in the gate, X6, MP_REGISTRY_BACKEND=kubernetes |
Keep |
| The kubernetes listing change | Keep |
| Per-port HTTP clusters only for ports declared HTTP | Keep |
| The capture floor decided from the endpoints, not the label | Keep |
| No any-port chain for a service that also serves HTTP | Keep |
| The liveness probe fix | Keep |
| The tests for the above, and the two corrected cache fixtures | Keep |
ServicePrimaryOf with the majority rule, as the service's primary and protocol |
Drop as an exported, service-level answer. Its order-independent vote survives in two private places: the per-port class conflict (fact 4) and the legacy target when pods disagree (decision 4) |
421 with x-aether-error: primary-not-http |
Drop. port-required instead |
| Labelling the whole Service with the primary's class, and the WARN about disagreeing primaries | Drop. The annotation is frozen at its 2.6 value. This also avoids the one skew cell #1677 measured as a regression (new registrar, old agent, etcd: the HTTP secondary stopped) |
Leg X4 (asserts the primary-not-http refusal) and X5 (asserts the Service is labelled tcp) |
Rewrite: X4 asserts port-required; X5 asserts the real ports and their appProtocol |
The edits to workload-requirements.md, the runbook and proposal 037 that define the spellings by the primary |
Drop; Phase 2 rewrites those pages |
- Gate (can fail):
multiprotocolM0 to M7 and X1 to X6 on both backends in CI (today:etcdonly; M6 and X6 read registry keys and run onetcdalone); MESH-HTTP 7/7 and GATEWAY-HTTP 46/46; the addressing legs below; the skew table re-measured on kind for both backends and both roll orders with zero cells worse than 2.6. M7 changes meaning: it asserts the real ports as well as the legacy four.
Addressing legs (Phase 1). In e2e/multiprotocol.sh, which already has
the client, the per-port echo and three of the five workloads. Each echo
answers with a string naming its own port, so "reached the wrong port" is a
failure and not a 200.
| Workload | Declares | In the suite today |
|---|---|---|
mp-httponly |
8080 HTTP | yes |
mp-multi |
8080, 9090 HTTP | new |
mp-eighty |
80, 8080, 9090 HTTP | new (its container binds 80) |
mp-mixed |
8080 HTTP, 9000 raw TCP | yes |
mp-tcpfirst |
9000 raw TCP (declared port), 8080 HTTP | yes, ungated today |
mp-nine |
9000 HTTP | new: the port number mp-mixed declares raw TCP |
Two clients: the existing captured one, and mp-client-nocapture, a managed
pod with capture.aether.io/redirect-all: "false", which dials only
127.0.0.1:18081.
| Leg | Dial and Host |
Path | Workloads | Must answer | On main |
|---|---|---|---|---|---|
| A1 | <mesh name>:P, Host the same, each HTTP port P |
captured | all six | port P's echo |
passes, except mp-nine (A10) |
| A2 | <svc>.<ns>.svc.cluster.local:P and <svc>.<ns>:P, each HTTP port P |
captured | mp-multi, mp-eighty, mp-mixed |
port P's echo |
fails: the primary answers (#1682) |
| A3 | 127.0.0.1:18081, Host: <mesh name>:P, each HTTP port P |
loopback, capture off | all six | port P's echo |
expected to pass; guards decision 5 |
| A4 | an HTTPRoute with a port-less parentRef adds a response header; request <mesh name>:9090 |
captured and loopback | mp-multi |
the header is present | fails (inferred): the rule is not on the port's virtual host |
| A5 | dial <mesh name>:8080, Host: <mesh name>:9090 |
captured | mp-multi |
421, x-aether-error: port-mismatch |
answers from 9090 (inferred) |
| A6 | Host: <mesh name>:9000, a raw-TCP port |
loopback | mp-mixed, mp-tcpfirst |
421, x-aether-error: port-not-http, under 1 s |
not measured |
| A7 | dial and Host <name>:7777, both name families; and Host: <mesh name>:7777 on loopback |
both | mp-multi |
421, x-aether-error: port-not-declared, under 1 s |
fails: 503 after 2 s (mesh name), the primary's echo (Kubernetes name) |
| A8 | raw TCP to <mesh name>:7777 |
captured | mp-mixed |
no application answers | passes; what the client sees is the cluster's (decision 10) and is not asserted |
| A9 | dial <name>:P with a port-less Host, each HTTP port P other than 80 |
captured | mp-multi, mp-eighty, mp-mixed |
port P's echo |
fails for every P but the primary: the primary answers |
| A10 | <mesh name>:9000, HTTP |
captured | mp-nine |
mp-nine's echo |
fails (measured in the rig): mp-mixed's raw-TCP 9000 sends it to the passthrough |
| A11 | HTTP bytes to mp-mixed:9000; non-HTTP bytes to mp-nine:9000 |
captured | the first reaches mp-mixed's TCP echo; the second reaches no application |
||
| A12 | dial port 80 or :18081 with any Host; port-less Host on loopback |
both | all six | unchanged in this phase: the legacy target, and for mp-tcpfirst 421, x-aether-error: port-required |
mp-tcpfirst: fails (502) |
| A13 (optional, with the inspector change) | a server-first exchange on a declared raw-TCP port whose number no service declares as HTTP | captured | a new server-first echo | first byte in under 200 ms | fails: 1 s |
A9 is a client-visible change in this phase: a port-less Host on a dialled
real port other than 80 reaches that port, where main sends it to the
primary. It is the same defect as #1682 seen through Host, and the answer on
main is the wrong application, so it is fixed and not deprecated.
A12 is where this phase deliberately does not yet apply the rule. The legacy
spellings are whole tables: cap_http_18081, and in table 80 a virtual host
per service that does not declare 80, both routing to the legacy target. For
mp-eighty, port 80 dialled is port 8080's echo now and port 80's later
(decision 11).
Runner time (estimate, not measured): the multiprotocol job took 2m24s with
both shapes in #1677, of which verify was 29 s for 14 legs. Three more
Deployments, one more client and about 60 more requests should add about a
minute to one backend. Running the suite on both backends doubles the job,
which is the larger cost: about 7 minutes of nightly runner time in place of
2.5.
- Skew: a 2.6 agent behind a 2.7 registrar sees the same annotation and
extra Service ports it does not read; on
kubernetesit also sees pods in more listings, which #1677 measured as "nothing gets worse" in that order. A 2.7 agent behind a 2.6 registrar needs nothing from the Service but its address. Inferred from #1677's table for the parts that differ from it; the gate measures it. - Operator sees: more ports on generated Services;
kubectl get svcshows what a service serves; more filter chains and route tables per proxy (Scale). Two behaviours change for clients, both from a wrong answer to a right one or a refusal: a port-lessHoston a dialled real port reaches that port (leg A9), and aHostport that contradicts the dialled port is refused (leg A5). - Harness contract: version 3, unchanged (new facts only).
- Rollback: to 2.6.x. The extra Service ports are removed by the 2.6 registrar's own convergence.
Phase 2 (chart 2.8.0): deprecation, counted¶
- Deprecated, removed in chart 2.9.0:
:18081and:18082on a mesh name; the bare name where the Service has no port 80; a portlessHoston the loopback listener; the any-port chain;agent.captureRedirectAllDefaultand scoped capture;aether.io/app-protocolandaether.io/port. Announced in the runbook's "Chart 2.8.0" section and where each is defined, with the removing version, as the window requires. - Counted. Each legacy spelling is served by its own named route or chain:
legacy_mesh_port,legacy_portless,cap_tcp_legacy18082_<cluster>, the existingcap_tcp_anyport_<cluster>, and the UDP listener's own prefix. Per destination and spelling this is a native proxy counter (a per-routestat_prefix; the chain'stcp_proxyprefix). Per source namespace and service it needs theaether_statsfilter to carry the route's name as a tag, which is our filter but C++ that was not read for this proposal; the floor, if that turns out to be more than a tag, is theroute_nameandfilter_chain_namefields the access logs already carry (proxy/accesslog.go:145,265-269). The agent logs each (source, destination, spelling) once. - Moved to real ports: the prober (mesh_dns targets with real ports, a
Hostwith a port on the loopback listener, the default for a bare target no longer:18081), every e2e suite in fact 1,charts/udsecho, the docs and website pages, and the contract's addressing sentence. - Twins and config: QUIC twins become per-port (new names; the old names go
dormant, never forgotten, as #1036 requires).
GammaBackendand the projection gain an additiveport; exporters write it beside the legacy cluster name. - UDP (decision 1): first a measurement on kind of a generated Service
with a UDP port and registrar-written
EndpointSlices, on both backends (delivery, the reply's source, drain on a terminating pod). If it holds, the registrar writes the slices;:18082/udpkeeps working and is counted. If it does not, 18082 stays for UDP and the finding comes back to the owner. - Gate (can fail): every suite green on real ports only; the Phase 1
addressing legs unchanged in their answers (nothing about the rule changes in
this phase); in
multiprotocol, leg D1: each legacy spelling dialled once (a dial to:18081, a dial to port 80, a port-lessHoston loopback,:18082, an undeclared port of a pure-TCP service) and its counter asserted to have moved (a zero is evidence only if the counter can be seen to count); leg D2: a dial to port 80 ofmp-eightymoves the counter of decision 11 and not the plain one; inl4-routes, the UDP leg dials the real port on both backends; MESH-HTTP 7/7. D1 and D2 add a few seconds. - Skew: none new on the wire. A 2.7 importer ignores the new
portfield. - Operator sees: WARN lines naming clients on a legacy spelling; a dashboard row per spelling; the deprecation in the runbook.
- Harness contract: version 4. The meaning of how a service is addressed changes (a bump by the README's last row); the counter is a new entry.
- Rollback: to 2.7.x; nothing was removed.
Phase 3 (chart 2.9.0): removal¶
Entry condition: the legacy counters at zero on the reference cluster over the whole of 2.8.x, with the witness leg showing they count. If not zero, the counter names the client and the removal waits a minor, as 037 did.
Deleted:
- The
cap_http_18081chain and table: done in chart 3.0.2, folded intocap_httpas a dialled-port branch ("Folding the proxy port intocap_http"). This item never meant port 18081 itself: it stays, permanently, the HTTP proxy port (the ruling of 2026-10-12), and so does the scoped mark rule that diverts it. - The legacy virtual hosts in table 80, the
:18082chains and the alias clusters; the 421 body's spelling of:18082;ProxyL4OutboundPort; the scoped mark rule for 18082. - The any-port chain and its contract entry (037 Phase 4).
- The default cluster
<fqdn>, the floor clustertcp:<fqdn>, the bare EDS name's special owner, the default virtual host,aliasAuthorityPorts. - The portless authority as "the primary": it is port 80 or it is refused.
- The Service's
mesh,mesh-tcp,httpandhttpsplaceholder ports; the clusterset Service'smeshport. aether.io/app-protocol,aether.io/port,PrimaryIsTCP,CaptureTCPServiceas a class carrier, the generator's first-wins loop, every "a service is registered under exactly one protocol" comment.- The legacy-target function and both private uses of the vote that served it.
- Scoped capture: the chart value, the CNI flag's effect (the flag is accepted
and ignored for this minor, because a 2.8 chart still passes it), the scoped
listener shape and its
blackholecluster. - UDP (decision 1), if the Phase 2 measurement held: the per-pod UDP listener, the
udp:clusters,UDPStatKey, the UDP mark rule. -
Sources switch to SNI = port for every TCP port; destinations have had the explicit chains since 2.7.
-
Gate (can fail): conformance unchanged; an 8-hour soak on real ports; and the addressing legs below, where the answer changes in this phase.
- Skew: a 2.8 agent beside a 2.9 registrar loses the legacy Service ports and the annotation; it stopped reading both in 2.7. A 2.8 source beside a 2.9 destination still sends the no-SNI forms, which the destination keeps for this minor.
- Operator sees: a client still on a legacy spelling gets
421with the ports to dial, or a refused connection. The runbook's "Chart 2.9.0" section says what a cluster that skipped 2.8 sees: the same, with no counter having warned it. - Harness contract: version 5 (entries removed).
- Rollback: to 2.8.x restores every spelling; clients moved in 2.8 keep working on both.
Addressing legs (Phase 3). In e2e/multiprotocol.sh.
| Leg | Dial and Host |
Path | Workloads | Must answer |
|---|---|---|---|---|
| R1 | dial port 80 (http://<name>/), both name families; port-less Host on loopback |
captured and loopback | mp-eighty |
port 80's echo (it was 8080's through 2.8) |
| R2 | dial port 80, both name families | captured | the other five | 421, x-aether-error: port-not-declared, under 1 s, the body listing the ports |
| R3 | port-less Host |
loopback | the other five | 421, x-aether-error: port-required, under 1 s |
| R4 | dial <name>:18081; Host: <mesh name>:18081 on loopback |
captured and loopback | mp-httponly |
421, x-aether-error: port-not-declared |
| R5 | raw TCP to <mesh name>:18082 and to an undeclared port |
captured | mp-tcpfirst |
no application answers (A8's assertion; decision 10 says what else may be asserted) |
| R6 | A1 to A11 | unchanged |
A12 is retired by R1 to R4. No new workload; a few seconds of runner time.
Phase 4 (chart 2.10.0): receiver-side tolerances¶
The destination's no-SNI h2 chain and its default raw-TCP chain, kept through
2.9 for 2.8 sources, and the CNI flag. ServiceEndpoint.port stays on the wire
as the liveness probe port.
Risks¶
- Every existing client breaks at Phase 3. There is no service whose
callers are unaffected: the documented way to call a mesh service is
:18081(getting-started.md:69-72). The mitigation is the counter and one whole minor; a cluster that skips 2.8 gets no warning. - The port 80 convenience goes.
http://<svc>.<ns>.<domain>/works only for a service with a port 80. With the default application port at 8080 that is almost none. The two ways back are to listen on 80, or to own the name with a Service that maps 80 to the pod's port, which only covers thecluster.localname today (fact 4, Collisions). - UDP. The mesh stops seeing application UDP: no per-service UDP stats,
no L4 log line, and
UDPRouteis served by registrar-written slices for one backend only. The option has not run in any cluster. Until it has, port 18082 stays for UDP, which is a standing exception to the decision. - Multi-cluster peers. A peer more than one chart minor behind exports routes with no port; at Phase 3 those backends cannot be resolved and are dropped with a counted WARN. Clusters must cross 2.8 together or within the window.
- Measured only in a rig, or only on kind. The UDP table is one network namespace on one kernel with one user ID: no CNI, no cluster service routing, no second node. #1677's tables are single-node kind; its agent-first order was not run on the kubernetes backend. The reference cluster has run none of this.
- The addressing rule (Design, The addressing rule). The dialled
port decides and
Hostnames the service. A client that sends aHostport different from the port it dialled is refused (port-mismatch), where today it is routed byHost; a proxy inside the pod that forwards requests with their originalHostto a different port is the case to watch for. On the loopback listener a port-lessHostis port 80 or refused. The cost isH+ 1 HCM chains per pod andH+ 1 capture route tables per node in place of one each, each table a resource whose chain and table must appear and disappear together (the #817 class), and a first-use path that is not designed yet. Legs A5, A9, A10, R1 to R3 hold the rule. - The Service port map is a new reader. A user Service with a named
targetPort, or with a selector that is not the ServiceAccount's pods, resolves to nothing or to the wrong set. It is reported, not guessed. - Capture listener churn. More per-port chains mean more reasons to update a per-pod listener, which 037 (Risk 4) says drains it. The HTTP chains change only when a port number enters or leaves the mesh, which is rarer than a service changing; the trigger must stay edge-triggered on the derived sets. Whether the pinned proxy applies a change of filter chains alone without draining the unchanged ones was not measured.
- Two transports on one port number cannot be declared (fact 4). DNS is the obvious case, and under (d) it is the first thing a user will try.
Decisions¶
Decided by the owner on 2026-10-11, each as recommended. The alternative that was not chosen is kept in one clause.
- Application UDP is no longer proxied. The generated Service's UDP ports
get
EndpointSlices written by the registrar, and the cluster's own service routing carries them. Port 18082 stays for UDP until that ships.UDPRouteis kept for the single-backend shape it supports today and withdrawn otherwise. Not chosen: keeping 18082 for UDP for good, and conntrack port translation to reserved listener ports. This option is not measured, so its phase starts with a measurement (Phase 2). - The port 80 break is accepted.
http://<svc>/works only for a service with a port 80; a user who wants 80 listens on it or writes a Service that maps it. Not chosen: a way for the generated Service to map 80 to a declared port. capture.aether.io/redirect-all: "false"is kept and means "outbound not captured" after Phase 3. Not chosen: removing the annotation and rejecting the value.- When a service's pods disagree about the legacy target during Phases 1
and 2, the order-independent vote from #1677 decides, private to the legacy
path and deleted in Phase 3. Not chosen: today's
endpoints[0]. - The loopback listener
127.0.0.1:18081is kept. It addresses the proxy, not a service; the CNI readiness probe and the prober's liveness tier use it. Nothing is dialled there, so the port inHostis the port:Host: <mesh name>:<port>with a port the Service declares as HTTP is matched by that port's virtual host; a port-lessHostmeans port 80 and is served only when the Service declares 80, else421 port-required. It works the same for a pod withcapture.aether.io/redirect-all: "false"(decision 3), which is that pod's only way into the mesh (legs A3, A6, R1, R3). Not chosen: removing the listener, which would be a separate proposal. endpoint.aether.io/portkeeps its name. With no primary it is one more declared port and the liveness probe port, and the documentation says so. Not chosen: a newhealth-portannotation and a contract bump.- TCP and UDP on one port number is out of scope here. It needs an additive registry field and is its own change, due before Phase 3 because of decision 1. Not chosen: solving it inside this proposal.
- Three chart minors: Phase 1 in 2.7.0, Phase 2 in 2.8.0, Phase 3 in 2.9.0, and the receiver-side tolerances in 2.10.0. Not chosen: Phases 1 and 2 sharing 2.7.x with removal in 2.8.0; Phase 1 fixes a bug and does not wait for the prober and the harness to move.
- A port-less
Hoston the loopback listener is port 80 whatever:schemesays;:schemeis never consulted. Not chosen: 443 for:scheme: https, which would route on a header the client controls. The question does not exist on the captured path, where nothing is defaulted. - Raw TCP to a port the Service does not declare gets the cluster's answer once the placeholder ports are gone: it reaches the passthrough and then a ClusterIP port that does not exist, which 037 describes as CNI-dependent and typically a hang. Phase 3 asserts only that no application answers. Not chosen: a per-service refusal chain, which under the per-port layout would catch only port numbers no chain names and so be partial without an HCM chain per (service, port); it is built if a hang is reported. HTTP, the common case, is refused by name either way.
- A service that declares port 80 while its legacy target is another
port (
mp-eighty) keeps the legacy meaning of a dial to port 80 through 2.8. Those requests are counted under their own name (legacy_portless_shadows_80), logged once per source, and the service is named in the 2.9 runbook section, because the change at 2.9 is from one working answer to another and would otherwise be silent. Not chosen: port 80 from 2.7 when declared, which changes behaviour with no deprecation window. - The capture listener has an HCM chain and a route table per HTTP port
number; the dialled port decides. It is the only layout in which a port
number one service declares as raw TCP still carries HTTP for another
(measured), and it makes a port-less
Hostmean what it means to any HTTP server. It costsH+ 1 HCM chains per pod and as many route tables per node, with the resource-lifetime and first-use questions under Scale. Not chosen: one chain routed onHost, which would also have needed a per-port chain for every port number any service declares as raw TCP.
Phase 1 as built¶
Added with the first pull request of the stack (then numbered chart 2.7.0; released in 3.0.0). What was built is the phase above. This section records where a measurement or the code contradicted this document, what was decided in each case and by whom it still has to be confirmed, and what the phase did not do. The text above is otherwise as accepted.
Measured¶
- The legs.
e2e/multiprotocol.shon kind, both registry backends, the new suite against images built frommainat7a9cca0aand then against the branch. Onmain: A2, A4, A5, A6, A7, A9, A10, A12 (through X4), X4, X5 and M7 fail on both backends; onetcdalso X1 and X2 (#1672); onkubernetesalso M4, X3, A1, A3 and A11 (the mixed shapes have no cluster for their second class). A1 onetcdfails formp-ninealone, which is A10. On the branch every leg passes on both backends (M6 and X6 read registry keys and are skipped onkubernetes). - HTTP on a port number another service declares raw TCP leaves the mesh.
Listed under Not established as measured in the rig only. Leg A10 measured
it on kind on
main:http://mp-nine...:9000/gets no reply whilemp-mixeddeclares9000=tcp. - The first-use path (Scale, "not designed"; Not established, second bullet). It needs no new mechanism. Every mesh Service with an address is in the node's dependency set since 037, so its clusters are in the snapshot and each table names them directly; the on-demand route of a table is kept for a mesh authority this node holds no endpoint of yet, and it accepts only an authority that spells the table's own port (the two legacy tables also the port-less name). The pinned proxy's on-demand filter does fetch a cluster a route names directly: the QUIC twins have relied on that since #1020.
- Conformance on the branch: MESH-HTTP 7 passed, 0 failed; GATEWAY-HTTP 36 + 10 passed, 0 failed (Gateway API v1.6.3, the nightly workflow dispatched on the branch).
Where this document was wrong, or silent¶
- Scale: the chain count is one higher, and a raw-TCP-only service costs
more. The table's
H+ 1 +T+ default leaves out the chain of the legacy tablecap_http_18081, which the same row counts as a route table. Measured for the worked example (200 services, HTTP on 80, 8080 and 9090, 20 raw-TCP ports beside an HTTP declared port, 50 pods): 26 chains per pod listener with the default (25 stated), 1,300 per node (1,250), 5 HTTP connection managers per pod and 250 per node (4 and 200), 6 route tables (6). The capture listener is 24 KiB per pod and the five capture tables 1.3 MiB per node. If the 20 raw-TCP ports are instead the declared port of 20 services with no HTTP port, a listener has 146 chains: each such service has its own port,:18082, the any-port chain and one copy of the any-port chain per HTTP port number (item 2). By this document's own count that shape is 62 onmain(not measured there). - The any-port chain needs a copy on every HTTP port number. The match
does not backtrack, so a connection to a port number that any chain names is
never offered to a port-less chain. With an HTTP chain per port number, a
pure-TCP service dialled on, say, 8080 would have stopped reaching its
declared port the day another service declared 8080 as HTTP. The copies
keep the legacy spelling working as it does on
main, share the any-port stat prefix so the counter that gates its removal sees every port, and go with it in Phase 3. Owner's call: dropping the copies instead would saveH+ 1 chains per raw-TCP-only service per pod and turn those connections into a passthrough (raw bytes) or421 no-http-port(HTTP). TLSRoutechains get the same treatment, on every named port except the ports the same service declares as HTTP (a per-address copy there would take the service's own cleartext HTTP away from the HTTP chain).- A1 against A12 and decision 11, for a service that declares port 80
beside another declared port (
mp-eighty). A12 says a dial to port 80 "with anyHost" reaches the legacy target; A1 says<name>:80with the sameHostreaches port 80's echo, and passes onmain. Built: a port-lessHoston a dial to port 80 is the legacy target (decision 11), andHost: <name>:80spelled out is port 80, as onmain. - A dial to
:18081with aHostthat names another port is refused (port-mismatch). A12's "anyHost" would route it to the legacy target; onmainit was routed byHost. No documented spelling does this. (Superseded by the ruling of 2026-10-12, "18081 is the proxy port, and the authority names the target": such a dial reaches the portHostnames, as on the loopback listener.) - A bare name that two namespaces share is no longer answered. The known-target route matched a bare name for every namespace that had it and took the first, so half the callers got another namespace's service. A virtual-host domain cannot be claimed twice, and a table is shared by the node's pods, so the bare name is a name only when one namespace owns it (the rule Names states); otherwise the request is passed through and the cluster refuses it. This is a client-visible change the phase's summary did not list.
- A
backendRefport that is not an HTTP port of the backend routes to nothing, where it reached the backend's declared port. 18081, and 80 for a generated Service that does not declare it, keep the legacy meaning. The port is carried from the route to the agent beside the shared projection, not in it: the proto field is Phase 2's. - The frozen annotation on
kubernetes. "The class of the declared port" is not what a 2.6 registrar stamped for a pod whose pod-level protocol is contradicted by a suffix on the declared port itself (protocol: http,port: 8080,ports: "8080=tcp"): 2.6 listed by the pod-level value. The row does not carry the pod-level annotation, so the registrar cannot reproduce it; it stamps the class of the declared port, as written here. - Pods of one service that disagree are decided once for the whole service, over every listing, and every builder reads that answer (the legacy target; the class of a port for the chains and the tables). Both endpoint sets of a port still exist, as fact 4 says.
Not done in Phase 1¶
- A13 and the
filter_disabledinspector change (optional). - A named
targetPortis reported as a WARN only, not on the route's status. (Superseded: see Rulings after acceptance, "A reference is strict and visible".) - The Service port map is read for the Services a route names (as its
parent or as a backend). A user Service that owns a mesh service's name and
has no route is passed through, as on
main. - Edge: the per-port raw-TCP clusters are published; an edge
TCPRoutebackend still resolves to the floor cluster (row 19's "the port's cluster, always" is not built). - The skew table was not re-measured on kind. What is stated for a roll is read from the code and from #1677's table.
Out of scope¶
- TLS termination for
https://<svc>/(037 option (b)); a TLS port is a raw-TCP port the application declares. - Protocol sniffing as a fallback for an undeclared port.
- Inferring ports from a user Service's
appProtocolinstead of the pod. - mTLS or identity for UDP.
- Dropping the protocol segment from the registry key.
- SPIRE-off raw TCP (#877).
HTTPRouteParentRefPortconformance.
Not established¶
- Listener and filter-chain counts on a real cluster, for any option.
- Whether the pinned proxy's on-demand filter fetches a cluster that a route names directly (the per-port first-use path depends on it).
- The filter-chain measurements on a cluster, with a real CNI, more than one
worker, or the unified
filter_chain_matcher. The passthrough of HTTP on a port number another service declares raw TCP was measured in the rig with the shipped chain shapes, not onmainin a cluster. - What a client sees for non-HTTP bytes to a declared HTTP port of a generated Service (inferred: the cluster rejects a Service port with no endpoints).
- Whether
aether_statscan tag a request with its route's name without a filter change (the C++ was not read). - Whether anything can program a pod's capture rules after CNI ADD (option (c) depends on it).
- The
SO_REUSEPORTcase of the bind conflict under two user IDs. - Option (d) end to end, on any cluster, on either backend.
- Which upstream GATEWAY-HTTP fixtures use which Service ports (they are not in this repository); only the MESH overlay was read.
- Whether the clusterset Service is pruned by the mesh-Service generator's
label selector (it carries
aether.io/mesh-service=trueand is never in the generator's desired set; inferred fromimport_generator.go:285-288andgenerator.go:139-152, not exercised).
Appendix: the UDP measurement¶
One network namespace (unshare -Urn), the pinned proxy with --concurrency 1,
one static listener per case with transparent: true and a udp_proxy whose
matcher routes to cluster A on a match and to B on on_no_match. A dummy
interface holds 192.0.2.1/24 and the default route; 10.96.0.10 is not a
local address.
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
# divert (T cases): mark on output, deliver locally on prerouting
nft add chain ip t out '{ type route hook output priority mangle; }'
nft add chain ip t pre '{ type filter hook prerouting priority mangle; }'
nft add rule ip t out ip daddr 10.96.0.10 udp dport 5353 meta mark set 1
nft add rule ip t pre ip daddr 10.96.0.10 udp dport 5353 tproxy to :<listener port> meta mark set 1 accept
# port translation (N cases): the address is kept, conntrack owns the port
nft add chain ip t nat '{ type nat hook output priority -100; }'
nft add rule ip t nat ip daddr 10.96.0.10 udp dport 5353 dnat to 10.96.0.10:<listener port>
nft add rule ip t pre udp dport <listener port> meta mark set 1
Each case sends one datagram from a connected socket and one from an
unconnected socket and records what came back and from where; the listener's
downstream_sess_total confirmed in every case that the datagram arrived. T1
uses an explicit tproxy port rewrite where the shipped CNI rule uses a bare
accept to a listener on the dialled port; the kernel delivers to the same
socket either way, and T1c is the shipped shape. The bind cases (C1, C2, S) use
no rule: a plain socket binds the port before or after the proxy starts. The
route-table check in The addressing rule is two listeners sharing one inline route table
whose virtual host carries a matcher on DestinationPortInput; the proxy
refuses the configuration at load.
Appendix: the filter-chain measurement¶
The same network namespace, with TCP to 10.96.0.0/16 marked on output and
iif lo meta mark 1 meta l4proto tcp tproxy to :18001 accept on prerouting. One
static listener on 0.0.0.0:18001, transparent: true, use_original_dst:
false, listener filters original_dst, http_inspector, tls_inspector,
listener_filters_timeout: 1s, continue_on_listener_filters_timeout: true,
and a default_filter_chain. HCM chains answer with a direct_response that
names their route table and virtual host; tcp_proxy chains and the default
chain lead to three local servers that tag what they receive, two of which
send a greeting on accept (the server-first application and the stand-in for
the original destination). Each case records which server accepted, the HTTP
status and body if any, and the time from connect to the first byte.
- L1:
{raw_buffer, HTTP}→ HCM;{9000, A/32}and{3306, A/32}→tcp_proxy. - L2: L1 plus
{8080, raw_buffer, HTTP}and{9000, raw_buffer, HTTP}→ an HCM each, with a virtual host forb,b:<port>and a*virtual host that answers421. - L3: L2 with
filter_disabled: {destination_port_range: [3306, 3307)}onhttp_inspectorandtls_inspector.
The TLS case is a real ClientHello from a TLS client; the h2c case is the client preface and an empty SETTINGS frame.