aether-proxy¶

A custom Envoy build for the Aether data plane, structured after envoyproxy/examples/filter-cc — the upstream downstream-build template published alongside Envoy's WORKSPACE removal (envoy#42890 / envoy#47217).

This is a separate Bazel module from the root aether repo. The root //.bazelignore lists proxy, so bazel build //... / Gazelle in the repo root never descend here. It pins its own Bazel version (.bazelversion = 8.7.0), its own module graph (MODULE.bazel), and its own registries (.bazelrc). The two workspaces share only the git repo. See docs/proposals/010.

Build¶

All commands run from inside proxy/ (so the right Bazel version and module graph are used):

cd proxy

# Build the custom Envoy binary (multi-hour C++ build; use a warm cache / CI).
bazel build //:envoy

# Build + load the custom aether-proxy image into the local Docker daemon.
# --config=release bakes the optimized Envoy (plain builds are fastbuild/dev).
# The Makefile `load-proxy-image` target does this for you.
bazel build --config=release //:image
bazel run --config=release //:load   # quay.io/aethermesh/proxy:latest

# Image smoke test (container-structure-test).
bazel test //:image_test

You almost certainly cannot build this locally. A cold Envoy build fetches tens of GB of module archives (BoringSSL, V8, ICU, gRPC, the LLVM 22.1.8 distributions, …) and needs hundreds of GB of output base even with the compile offloaded to RBE — analysis still fetches everything. What is cheap locally, and worth running before pushing, is module resolution:
# No unresolved modules; envoy/envoy_api at the pinned version.
bazel mod graph --depth=1

# The single assertion that override_repo took: this MUST print our
# local_repository at bazel/build_config, not Envoy's default_envoy_build_config.
bazel mod show_repo --base_module=envoy @envoy_build_config

# Selected versions match the registry pin, with no MVS surprises. By module
# name: quiche is not a direct dependency any more, so `@quiche` does not resolve.
bazel mod explain quiche protobuf abseil-cpp
Everything from bazel build --nobuild //:envoy onward is validated by .github/workflows/proxy.yml, the PR check, which builds both arches on BuildBuddy RBE. The release path is a separate workflow, .github/workflows/proxy-release.yml: on a push to main touching proxy/** it builds and pushes each arch by digest, assembles the multi-arch manifest tagged with the commit SHA, and opens the chart pin PR that moves charts/aether/values.yaml's proxy.image onto it (#703/#727). Never push a proxy image or edit that pin by hand.

MODULE.bazel.lock is committed as a dev convenience. CI runs with the default --lockfile_mode=update, not error: the envoy_toolchains_extension (arch_alias → ctx.os.arch) and toolchains_llvm's llvm extension are host-arch dependent, and the two CI legs drive from different arches.

The image is distroless/cc base + the custom //:envoy binary at /usr/local/bin/envoy. Plain bazel build //:image produces a fastbuild (unoptimized) binary — always pass --config=release (CI and the Makefile targets do) to bake the production binary.

The released binary is not fully stripped, on purpose. --strip=always removes DWARF but keeps .symtab (~22 MB, ~944k symbols), which is what lets Pyroscope put names on the proxy fleet's native frames — the profiler itself symbolizes nothing native (aether #651). //integration:symtab_test guards it, and //integration:build_id_test guards the content-derived GNU build-ID the symbol upload is keyed by (#653).

aether_stats is a compiled-in C++ extension (proposal 012), built into //:envoy via AETHER_EXTENSIONS in BUILD.bazel. It records source→destination request metrics from StreamInfo at stream completion. The earlier Rust dynamic-module approach was dropped — no Rust toolchain or dynamic-module wiring is needed.

Layout¶

Path	Purpose
`MODULE.bazel`	module graph; pins `envoy` / `envoy_api` (see "Envoy version bumps")
`.bazelrc`	build config; pins the envoy bazel-registry commit; clang/RBE/release configs
`.bazelversion`	`8.7.0` (independent of the root repo)
`bazel/build_config/`	the compiled-in Envoy extension set, as a tiny local module (`envoy_build_config`)
`bazel/platforms/BUILD.bazel`	BuildBuddy RBE exec platforms (amd64 / arm64), derived from Envoy's
`bazel/get_workspace_status`	`--workspace_status_command` stamping script
`bazel/image_metadata.sh`	emits the image's OCI provenance labels (aether commit + the Envoy pin)
`BUILD.bazel`	custom `envoy_cc_binary` + `oci_image`/`oci_push`/`oci_load` + `image_test`
`integration/`	`build_id_test`, `symtab_test`, container-structure-test config
`source/extensions/filters/http/aether_stats/`	native C++ `aether_stats` filter (compiled into `//:envoy`)

Customizing¶

Compiled extensions: add your envoy_cc_library config target to AETHER_EXTENSIONS in BUILD.bazel. It is a dep of the binary, not an entry in bazel/build_config/extensions_build_config.bzl — that dict is only for extensions that live in @envoy.
Dropping upstream extensions: add a substring to _DROPPED in bazel/build_config/extensions_build_config.bzl. It filters Envoy's own default dict, so an Envoy bump picks up new upstream extensions for free. Today we drop wasm and dynamic_module.
Source patches: the exception, not the mechanism — @envoy is a registry module, so a patch needs a single_version_override(patches = [...]) in MODULE.bazel (the bazel/patches/ directory). Prefer upstreaming; a carried patch is an upstream PR that has not reached a pin yet, and it goes away with the next pin bump that contains it. A carried patch's own Envoy tests run from //bazel/patches:carried_patch_tests, since //... does not reach @envoy tests.

Carried today¶

Applied in this order (single_version_override in MODULE.bazel, which also says what each one changes in detail). scripts/bump-envoy-pin.sh reports, for a new pin, which of them the new Envoy already contains.

Patch (`bazel/patches/envoy-…`)	What it fixes	Upstream
`47740-quic-hostname-check-explicit-san`	The QUIC client defers its hostname check to explicit SAN matchers (aether#957); behind `envoy.reloadable_features.quic_hostname_check_deferred_to_explicit_san_match`.	envoyproxy/envoy#47740, open
`aether1050-hotrestart-nonblocking-forward`	The hot-restart main-thread deadlock (aether#1050): parent UDP/QUIC forwarding to the child no longer blocks the parent's main thread, and the child keeps servicing forwarded packets while it waits, with a bound, for the parent's replies. Counts what it forwards in `server.hot_restart_udp_forwarding_{datagrams,retries,dropped}`. A signal interrupting a send or a wait on the hot-restart domain socket (EINTR, which the socket timeouts make non-restartable) is retried instead of aborting the child on a `RELEASE_ASSERT`.	envoyproxy/envoy#47834, open (the patch is that pull request's diff)
`aether1054-http3-drain-idle-connections`	Idle HTTP/3 connections drain themselves inside the drain window (aether#1054), behind `envoy.reloadable_features.http3_drain_idle_connections`, beside upstream's own proactive drain (envoyproxy/envoy#47991), which arms nothing at the chart's 10s drain time.	not proposed
`aether1054c-paused-udp-listener-no-read`	A paused hot-restart child UDP listener no longer reads the parent's socket when the QUIC listener injects a read to process forwarded handshakes (aether#1054).	envoyproxy/envoy#47868, merged on Envoy main after v1.40.0 (a later pin drops it)
`aether1074-quic-persistent-info-lifetime`	HTTP/3 pools keep the per-cluster persistent QUIC info alive after a CDS removal, instead of their draining connections reading its freed clock (aether#1074).	envoyproxy/envoy#47893, open
`aether1126-forwarded-udp-worker-index`	A hot-restart child with fewer workers than its parent no longer indexes past its worker table when the parent forwards a UDP/QUIC datagram tagged with one of its extra workers: the index is mapped onto `index % concurrency` (aether#1126).	not reported

No longer carried, because the pinned release contains them: envoyproxy/envoy#47743 (UDP hot-restart forwarding keyed by listener address + network namespace, aether#967) and envoyproxy/envoy#47776 (execInNetworkNamespace restores the calling thread's netns via /proc/thread-self/ns/net, aether#1022; the carried patch also fell back to /proc/self/task/<tid>/ns/net on kernels older than 3.17, upstream does not), both dropped at the 189f198 snapshot (aether#980); and envoyproxy/envoy#48206 (a draining hot-restart parent answers packets for its own recently closed QUIC connections from its time-wait list instead of forwarding them to the child, aether#1054; behind the same envoy.reloadable_features.quic_hot_restart_time_wait_before_forward), dropped at the 1.40.0.envoy release, which contains it as 69ff216f86.

Dropped without an upstream merge: upstream Envoy PR 48319, which kept the hot-restart child's inherited UDP listeners paused until the parent had exited (behind envoy.reloadable_features.hot_restart_wait_for_parent_exit). It had replaced the first half of the earlier aether1054 patch, and was closed upstream after an A/B soak showed no measurable benefit. The child now completes its drains, unpausing those listeners, at Terminate, as unpatched Envoy does. An 8 h soak of exactly this configuration (the 48319 guard off) on the reference cluster showed 0 misses near handoffs and 0 QUIC stateless resets. The aether1054 patch stays the HTTP/3 drain only; its old hot-restart half is not restored.

Which Envoy is this? (`envoy_server_version`, and the image labels)¶

What `envoy_server_version` actually reports¶

It reports the aether commit the proxy image was built from. It says nothing about Envoy. Do not try to match it against the pin — it will never agree, and the disagreement is not a bug (aether #837).

You do not have to take that on inference. The pinned-Envoy target — today //agent/test/envoy_validate:envoy_bin, via //bazel/proxy_pin; #841 factors it out, so bazel query for it if the label has moved — is the Envoy binary extracted from the published aether-proxy image at the digest charts/aether/values.yaml pins. You can ask it directly, locally, with no cluster:

$ bazel build //agent/test/envoy_validate:envoy_bin
$ "$(bazel info output_base)/$(bazel cquery --output=files //agent/test/envoy_validate:envoy_bin)" --version
envoy  version: cff8beb87eb685a44e00cf70c1a534695039da1b/1.40.0-dev/Clean/RELEASE/BoringSSL

cff8beb87eb685a44e00cf70c1a534695039da1b is an aether commit — it is that image's own tag: in values.yaml. The Envoy part of the string is just 1.40.0-dev, with no Envoy revision in it anywhere. So for that image the gauge is 0xcff8be; a proxy reporting something else is simply an older image.

The chain that produces it, end to end:

.bazelrc sets --workspace_status_command="bash bazel/get_workspace_status", and that script runs git rev-parse HEAD. proxy/ is a nested workspace inside the aether git repository, so BUILD_SCM_REVISION is the aether commit. Envoy is a bzlmod dependency here; nothing ever asks its git for a revision.
envoy_cc_binary defaults to stamp = 1 (Envoy's bazel/envoy_binary.bzl), which is cc_binary's "always stamp, even under --nostamp". So the //source/common/version:version_linkstamp linkstamp compiles that real sha in even though no build here passes --stamp. (Verified: with a plain cc_binary, --nostamp yields the redacted BUILD_SCM_REVISION of "0"; with stamp = 1 it yields the workspace-status value.)
VersionInfo::revision() returns it, and Envoy's server.cc does atoull(revision().substr(0, 6), 16) to set the server.version gauge.

So the gauge is the first six hex digits of the aether commit. The reading that opened #837 decodes cleanly that way:

envoy_server_version = 3121738 = 0x2FA24A -> aether commit 2fa24a8
  "test: drop duplicate integration targets ... (#772) (#788)"

That is a genuinely useful fact — it tells you which aether tree cut the proxy — but it is only 24 bits of it, and it is not the Envoy revision. 189f19, the first six digits of the Envoy commit in a dev-snapshot pin such as 1.40.0-dev.20261009.189f198.envoy, is what you would be looking for and it is nowhere in the process.

Where the Envoy revision actually is: the image labels¶

//:image_metadata (see bazel/image_metadata.sh) parses the pin out of the two files that define it and puts it on the image as both OCI labels (image config) and annotations (manifest):

key	value
`org.opencontainers.image.source`	`https://github.com/bpalermo/aether` — overrides the `GoogleContainerTools/distroless` value inherited from the base, which used to be the image's only annotation
`org.opencontainers.image.revision`	the aether commit; the same sha the gauge reports the first six digits of
`dev.aethermesh.envoy.module-version`	e.g. `1.40.0.envoy` (a release) or `1.40.0-dev.20261009.189f198.envoy` (a `main` snapshot)
`dev.aethermesh.envoy.revision`	e.g. `189f198` — the upstream Envoy commit; only on a `-dev` snapshot pin, whose version carries it. A release pin (`1.40.0.envoy`) has no such label: the version names the release tag (`v1.40.0`) instead
`dev.aethermesh.envoy.bazel-registry`	the `envoyproxy/bazel-registry` commit; the other half of the pin (see "Envoy version bumps")

To read them off a published image, without pulling it:

crane config quay.io/aethermesh/proxy@sha256:… | jq .config.Labels
crane manifest quay.io/aethermesh/proxy@sha256:… | jq .annotations

None of this costs reproducibility: the labels are fixed strings, created stays unset (1970-01-01T00:00:00Z) and the history entries stay bazel bu…. The metadata genrule is tagged no-cache on purpose — BUILD_SCM_REVISION is a volatile workspace-status key, which Bazel deliberately does not invalidate on, so a cacheable action would re-stamp a stale commit.

Envoy version bumps¶

The module version and the registry commit are ONE pin and must move together. The envoy bazel-registry (envoyproxy/bazel-registry) publishes main dev snapshots and, since its first tagged release (v0.0.1, 2026-10-13), Envoy releases. Until then it kept exactly one version per module and deleted the previous version directory on every bump; it now keeps published versions side by side (at tag v0.0.3, modules/envoy holds both 1.40.0-dev.20261009.189f198.envoy and 1.40.0.envoy) and its CI enforces that a published module is immutable. Pin the commit anyway: the .envoy-suffixed closure a version resolves against is only fixed by the commit, and an unpinned (or mismatched) registry commit drifts with every sibling bump.

Worse, registry HEAD is frequently not self-consistent: sibling modules get bumped before modules/envoy/<version>/MODULE.bazel is updated to match, so at HEAD the envoy module can reference .envoy-suffixed versions that no longer exist. Envoy's own .bazelrc pins a commit that predates the publication of its own snapshot, so it cannot be copied either.

Check modules/envoy/metadata.json first. It lists every envoy version the registry currently publishes, and one API call answers the whole question:

gh api repos/envoyproxy/bazel-registry/contents/modules/envoy/metadata.json \
  --jq '.content' | base64 -d | jq .versions

If the only version listed is the one already pinned, there is nothing to bump to and the loop below will confirm it the slow way. The real gate on a bump is a new envoy snapshot, not a newer registry commit.

Otherwise the bump recipe is a resolvability loop — find the newest registry commit at which the whole .envoy-suffixed closure resolves:

git clone https://github.com/envoyproxy/bazel-registry.git /tmp/br

# For each recent commit, newest first:
#   1. envoy_ver=$(git -C /tmp/br ls-tree -d --name-only <commit> modules/envoy/)
#   2. BFS `bazel_dep(name=..., version=...)` from
#      modules/envoy/$envoy_ver/MODULE.bazel and modules/envoy_api/...
#   3. every dep whose version ends in `.envoy` must have a
#      modules/<name>/<version>/MODULE.bazel in that tree
# Take the newest commit with zero misses.

Two traps when checking a candidate commit:

--registry accumulates; it does not override. Adding --registry=…/<candidate> on the command line appends a registry behind the two .bazelrc already pins, so the pinned commit still answers every lookup and the probe passes no matter how broken the candidate is. To test a candidate you must edit the --registry= line in .bazelrc, and pass --lockfile_mode=off so MODULE.bazel.lock does not answer from cache.
A missing version directory is only fatal if nothing requests a higher one. Bazel selects the MVS maximum, so foo@1.3.0.envoy resolves fine if some other module in the closure asks for foo@1.3.3.envoy and that directory exists. The step-3 rule above is the conservative approximation; if it rejects a commit you otherwise want, re-check against the selected version rather than the requested one.

If the loop finds no commit newer than the current pin, that is the normal steady state between snapshots, not a failed bump: the registry bumps siblings ahead of modules/envoy/<version>, so after a snapshot publishes it was (while the registry kept one version per module) usually resolvable for only a handful of commits. For a release, prefer the commit that added it (the release tag's commit) over a later HEAD. Do not "fix" the misses with single_version_override — forcing a sibling past what the envoy module was published against buys an unvalidated combination that only a multi-hour CI build can disprove. Wait for the next snapshot.

Then, in one commit:

MODULE.bazel: set envoy and envoy_api to that snapshot version.
.bazelrc: set --registry=https://raw.githubusercontent.com/envoyproxy/bazel-registry/<that commit>.
Update every other .envoy-suffixed bazel_dep in MODULE.bazel (proxy-wasm-cpp-host, rules_rust, toolchains_llvm, protobuf) to the versions that commit carries — they must match what the envoy module requests or MVS will fail. Move the plain BCR ones the envoy module also requests (rules_python, abseil-cpp, rules_cc, …) to the version it asks for too, so MODULE.bazel states what is selected.
Re-diff .bazelrc against the two upstream sources named at the top of that file (the filter-cc template and Envoy's own .bazelrc at the new pin).
Run the local bazel mod checks above, then push and let CI build both arches.

When a stable release publishes a 1.NN.0.envoy module, move to it. The pin is 1.40.0.envoy at registry tag v0.0.3 (a1264b53, the commit that added it); it replaced the 1.40.0-dev.20261009.189f198.envoy snapshot. A release pin needs the matching go-control-plane/envoy bindings (v1.NN.0) on main first: scripts/check-envoy-api-parity.sh accepts bindings one minor behind only for a -dev pin.

Nothing in the root workspace needs re-pinning alongside it any more. The root MODULE.bazel used to carry @envoy_binary_linux_*, a stock Envoy release asset that //agent/test/envoy_validate ran envoy --mode validate with, and that pin had to move in step. #709 replaced it: //bazel/proxy_pin now lifts /usr/local/bin/envoy out of the aether-proxy image at the digest charts/aether/values.yaml pins, so the validate gate follows the chart pin — and therefore this module — with no second version to keep in sync.

The aether_stats C++ extension builds against this same Envoy tree, so there is no separate SDK version to keep in sync.

Status¶

Custom Envoy + aether_stats C++ extension building on CI (proposals 010 / 012), on bzlmod against the Envoy 1.40.0.envoy release (main dev snapshots from #697 until the 1.40.0 release). The Envoy compile runs on BuildBuddy RBE — amd64 driven from an x64 runner, arm64 driven from a native ubuntu-24.04-arm runner (@clang_platform and @llvm_toolchain derive their exec constraints from the driver host's arch, so an x64 driver cannot drive the arm64 pool).