Helm charts¶

Bazel-built Helm charts for Aether, using rules_helm.

Chart	Path	Deploys
`crds`	`charts/crds`	Aether CustomResourceDefinitions (`MeshConfig`, `HTTPFilter`, `EdgeConfig`, `EndpointPolicy`). Install first; standalone so CRDs can be upgraded independently.
`aether`	`charts/aether`	The whole system: agent DaemonSet (xDS + CNI install) + per-node Envoy proxy + the `mesh-dns` DaemonSet, registrar Deployment, the controller (validating webhooks for `MeshConfig`/`HTTPFilter`/`EdgeConfig`/`EndpointPolicy`/`HTTPRoute`, a pod-mutating webhook, and the `MeshConfig`→ConfigMap reconciler), and the optional north-south edge. Owns the `aether-system` namespace and RBAC.
`prober`	`charts/prober`	External mesh-availability prober (proposal 013).
`udsecho`	`charts/udsecho`	UDS validation workloads (proposal 034): both socket-delivery paths (annotation + `EndpointPolicy`) under continuous mesh traffic.

Agent, proxy, mesh-dns, registrar and controller deploy together as one system from the single aether chart — system config (OTEL, SPIRE, mesh domain) is set once at the top level of its values.yaml and inherited by every component. The proxy data plane can override its own observability at runtime via the MeshConfig CR. See docs/proposals/015_mesh-config.md.

Resource names are derived from the release name, so install with helm install aether … to get aether-agent, aether-registrar, aether-controller, etc.

Images (mirroring)¶

Each image is configured as repository + digest (in-repo images, digest-pinned for immutability). The external proxy image carries repository + tag + digest — the aether.image helper prefers the digest, so the tag is documentation of the publishing commit, not the reference that is pulled. To mirror to a private registry, override repository alone, e.g.:

agent:
  image:
    repository: my-registry.example.com/aether/agent
registrar:
  image:
    repository: my-registry.example.com/aether/registrar

For in-repo images the repository/digest are written as {@//path/to:image_push.repository} / .digest placeholders that rules_helm substitutes at package time, so packaged charts are pinned to a concrete digest.

Versioning & stamping¶

Field	Value	Notes
Chart `version`	e.g. `0.8.0-{GIT_COMMIT}` (crds), `0.3.0-{GIT_COMMIT}` (prober)	SemVer pre-release; commit becomes the OCI tag (dash-separated — `+build` metadata would be rewritten to `_` by helm). The `aether` chart is published twice: under its plain `Chart.yaml` version (`<version>`, what Flux and the deploy procedure consume) and — via `:aether_commit`, whose `Chart.yaml` is derived from the same file — under `<version>-{GIT_COMMIT}`, so every commit's chart stays addressable after the mutable bare tag has moved on (#692). Bump the chart's `version:` on any change to its templates/values (enforced in CI). These versions move on nearly every release: read the live value from each chart's `Chart.yaml`, or from the version the publish workflow prints.
Chart `appVersion`	`{STABLE_GIT_VERSION}`	`git describe` value: the commit the chart was packaged from. It is not what a binary reports about itself. A component's `--version` and its OTel `service.version` are that binary's GNU build ID, which changes only when the binary does (#1378, #1429); the runbook's "Which build is this binary" maps one to the other.
Image refs	`repo@sha256:…`	Pinned to the exact built digest (strongest form).

The {...} placeholders in Chart.yaml are filled from the workspace status (bazel/workspace_status.sh) only on --stamp builds:

bazel build --stamp //charts/aether   # embed git version
bazel run   --stamp //charts/aether:aether.chart_push   # publish a stamped chart

Without --stamp the braces are stripped (e.g. 0.4.0-GIT_COMMIT) so the chart still lints/templates — but release builds should pass --stamp. Image digests are stamped independently of this flag (they always reflect the built image).

Build & test¶

# Package a chart (.tgz under bazel-bin/charts/<name>/)
bazel build //charts/crds //charts/aether

# Lint + `helm template` smoke tests
bazel test //charts/...

Writing a template test¶

A chart's tests live in its BUILD.bazel. Which rule to use:

The test is about	Rule
a template that renders one document, in a chart whose render holds no Secret	rules_helm's `helm_template_test` (`template_patterns`, Go regular expressions)
one document of a template that renders several (RBAC, the webhook configurations, a ServiceAccount next to its Deployment)	`helm_template_match_test` with `document_patterns = {"<Kind>/<name>": [...]}`
a block wherever it renders	`helm_template_match_test` with `patterns`
something that must not render (an off switch)	`helm_template_absent_test`
a value the chart must reject	`helm_template_fail_test`
a derived line that must follow a value (a config checksum)	`helm_template_value_changes_test`
templates that must not change with a value (a file the pod watches, so no roll)	`helm_template_value_ignored_test`
pod templates that must not change with a file the chart is packaged with (`.Files.Get` of something a build step derived), which the chart must also render without	`helm_pod_template_file_test`
a mesh name the chart writes on its own pods, which code reads (the mesh-managed label, a mesh annotation, the CSI driver)	`helm_mesh_names_test`

All but the first are in //bazel/helm:defs.bzl, which documents each.

charts/aether/BUILD.bazel loads the rules it uses from charts/aether/test_defs.bzl, not from there. The aether chart requires clusterName and has no default for it, so every render has to name a cluster; those wrappers pass --set clusterName=test-cluster ahead of a target's own opts (which win), and a new test needs nothing for it. A test of the requirement itself renders without the value and loads the rule straight from //bazel/helm:defs.bzl (:aether_cluster_name_required_test).

The last row is for what a pattern cannot say: a pattern holds a literal, and a literal in a BUILD file agrees with a literal in a template while the constant the agent reads has moved on. charts/prober and charts/udsecho write the mesh-managed label, the mesh annotations and the csi.aether.io driver on their own pods; :prober_mesh_names_test and :udsecho_mesh_names_test list what each workload carries by symbol, and the checker (//bazel/helm/meshnames) compares the render with common/constants and common/udspath (#1589). It is not a go_test: one with a chart in its data cannot be analysed under --config=race. Like the pattern rules, it never prints a render.

Two facts about rules_helm's helm_template_test decide the first row:

It keeps one document per template file, the one helm prints last, and helm prints a release's objects sorted by kind, not in template order. A pattern for any other document of that file fails; it cannot pass by accident, but it cannot be written either (#1371).
Its runner prints the whole render into the test log, whether the test passes or fails. With default values the aether chart generates the webhook's private key at render time, so every such test in charts/aether/BUILD.bazel passes --set controller.webhook.spire=true, which renders no Secret; :aether_upstream_template_tests_render_no_secret_test fails a target that leaves it out (#1382). A test that needs the default webhook path uses one of the other rules.

helm_template_match_test patterns are POSIX extended regular expressions (bash <span class="aether-private-note">(private note: =~)</span>) plus \n, \s and \S; no \d, (?s) or lazy quantifiers. When one fails it prints the pattern, the document it searched, and a few lines from where the pattern stopped matching, from a masked copy of the render. It never prints the render.

The mask fails closed. A document is printed as it is only when it is plainly something else than a Secret: a single top-level kind: <Word> line (optionally quoted, optionally with a trailing comment) whose word is not Secret and does not end in List (a typed list such as SecretList holds items with no kind line of their own), in a document that is not flow style or JSON, has no explicit (?) or merge (<<) key at any depth, and has no nested kind that is Secret, a typed list, or written with an anchor, a tag or an alias. In a document that might be a Secret, only these lines are ever printed, and only as simple key: scalar lines: the # Source: comment, apiVersion:, kind:, type:, metadata: and the name: and namespace: directly under it. Everything else is <masked>. A ConfigMap written the usual way therefore stays readable in a failure excerpt, and a failure in a Secret shows which Secret and where, never what is in it.

document_patterns can name only a document written in block style (a kind: line at column 0, metadata: and its name:), which is how every template in these charts is written. A document written as a flow mapping or JSON is not indexed: the test fails saying so and lists the documents it found; match such a document with patterns.

Compatibility window¶

The unit of compatibility is one minor version of the aether chart. That is what "one release" means wherever these docs say it.

Skew that is supported. The chart pins every image by digest, so a release is one chart with its own images; skew exists while a roll is under way (the agent, the node proxy, mesh-dns, uds-csi, the registrar and the controller roll separately, node by node) and after a rollback.

Combination	Supported
Components of one chart minor, any patch versions, side by side	yes
Components of two adjacent minors side by side (an upgrade or a rollback in progress, e.g. a 2.5.x agent with a 2.6.x proxy, or a 2.6.x registrar with 2.5.x agents)	yes, for the length of the roll
An upgrade that skips a minor (2.4.x straight to 2.6.x)	no: go through the minor in between and let it finish rolling
A chart with image digests overridden to those of another minor, as a steady state	no
A major version step	only as its own runbook section says ("Upgrading to chart 2.0.0", "Chart 3.0.0"). A major version may remove something without a deprecation in a published chart; that section then lists each removal as BREAKING and says what a mixed cluster does during the roll, measured, or states exactly which mix was not measured

Deprecation. Something deprecated in a chart of minor N (a flag, a value, a metric or one of its labels, a file or wire format, a tolerance for an older peer) is still there in every chart of minor N and may be removed in N+1.0: deprecated in 2.5.x, removable in 2.6.0, never within 2.5.x. It is announced in two places:

the runbook (docs/runbook.md, "Which workloads a chart upgrade rolls"), in the "Chart X.Y.Z" section of the chart that deprecates it: what is deprecated, what to use instead, and the chart minor that removes it. The section of the chart that removes it says what an operator who skipped the window sees;
where the thing is defined: the flag's --help text, the metric's description, the comment on the value in values.yaml, or the comment beside the template or the code.

Read the sections between the chart you run and the chart you are moving to before every upgrade. "Chart 2.5.7" lists what was deprecated when this rule was written down, with the chart that may remove each.

Install¶

CRDs first, then the system:

bazel run //charts/crds:crds.install
bazel run //charts/aether:aether.install
# Counterparts: .upgrade, .uninstall

From the published OCI registry, quay.io/aethermesh (use the semver +/- form for --version; helm maps it to the dash-separated tag). Each chart lives in its own chart-<name> repository.

Do not copy a version out of this file — the charts are republished on nearly every merge. Take <crds-version> / <aether-version> from the corresponding Chart.yaml at the commit you are installing (or from the versions the publish workflow prints for that run):

helm install aether-crds oci://quay.io/aethermesh/chart-crds \
  --version <crds-version>-<git-commit>
# The namespace, labelled for Pod Security admission (the agent, proxy, mesh-dns
# and uds-csi pods need hostNetwork, hostPath volumes and NET_ADMIN).
kubectl create namespace aether-system
kubectl label namespace aether-system \
  pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/audit=privileged \
  pod-security.kubernetes.io/warn=privileged
# Use the commit-pinned tag; the bare `--version <aether-version>` also
# resolves, but that tag is mutable: see below.
helm install aether oci://quay.io/aethermesh/chart-aether \
  --version <aether-version>-<git-commit> -n aether-system --create-namespace \
  --set clusterName=my-cluster

clusterName is required and has no default since chart 2.6.0: without it the render fails and names the value, and nothing is installed or upgraded. It is this cluster's name in the mesh (--cluster-name of the agent, the registrar and the edge control plane, and a part of the registry's keys): choose it once and keep it. Upgrading a release that never set it is in docs/runbook.md, "Chart 2.6.0".

Pin the commit tag or the digest, never the bare tag with a digest (#1588). chart-aether:<aether-version> is pushed again by every commit on main that still carries that Chart.yaml version, which is most commits: its digest changes while the version does not. Only <aether-version>-<git-commit> is written once. A reference that pins a digest under the bare tag (chart-aether:<aether-version>@sha256:…, or a lock entry of tag plus digest) breaks when the next commit publishes; Helm reports chart reference digest mismatch. Whatever pins a released chart (a values lock, a GitOps source, a mirror) names <aether-version>-<git-commit>, or the digest alone (oci://quay.io/aethermesh/chart-aether@sha256:…). The other three charts have no bare tag: their version carries the commit. docs/runbook.md, "Pinning a released chart", has the table.

The aether chart does not create its namespace (namespace.create=false, the default since 2.4.21): Helm or you do, and you label it. See "Who creates the namespace" below.

Moving from ghcr.io (chart 1.0.0, proposal 040). Until the 1.x charts, everything was published to GitHub Container Registry under a different path (charts/<name> for charts, aether-proxy for the proxy image). Every chart took a major version bump with the move because the default image repositories changed: re-point helm at the oci://quay.io/aethermesh/chart-<name> coordinates above, and if you override an image repository: by prefix (a mirror), mirror from quay.io/aethermesh/<component> now — override each image's repository individually. Nothing publishes to or verifies against ghcr.io any more (proposal 040 phase 4); releases from before the move were not copied to quay.io.

Multiple instances & labels¶

Resource names are release-prefixed (<release>-agent, …) and cluster-scoped resources (ClusterRole/ClusterRoleBinding) additionally include the namespace. Customize naming with nameOverride / fullnameOverride, and target a namespace with helm install <release> -n <ns>.

One aether release per cluster (#1540). Not every name follows the release: the node proxy's DaemonSet and ServiceAccount are aether-proxy and its ConfigMap aether-proxy-config whatever the release is called, and the csi.aether.io CSIDriver is cluster-scoped. More to the point, the agent, the proxy, mesh-dns and uds-csi own node-level state (see the note below), so two releases on the same nodes cannot both work. Since 2.4.27 the chart refuses a second release at render time when it can see the first one's aether-proxy DaemonSet (same namespace) or CSIDriver; it cannot see one in another namespace when the first runs with udsCsi.enabled=false, or anything in a render without a cluster. docs/configuration.md, "One release per cluster", has the list of names and why the proxy's is a constant.

Who creates the namespace (#1403). Helm stores the release record in the release namespace before it creates anything a chart renders, so a chart cannot create the namespace its own release is stored in.

`namespace.create`	Release stored in	Who creates the namespace	What you do
`false` (default)	the same namespace (`-n aether-system`)	Helm (`--create-namespace`) or you	Label it for Pod Security admission before installing (the three `kubectl label` values above); on a cluster that enforces nothing, the helm command alone is enough.
`true`	a different namespace (`-n <other> --set namespace.name=aether-system`)	the chart, with the three privileged labels and `helm.sh/resource-policy: keep`	Nothing.
`true`	the same namespace	nobody: the render fails, naming the two rows above	Pick one of them.

A namespace the release already owns (it carries Helm's two ownership annotations for this release: every install made while namespace.create defaulted to true) is rendered whatever namespace.create says, so upgrading across 2.4.21 with no values cannot make Helm delete it. A release of an older chart installed with namespace.create=false is asked once for kubectl annotate namespace <ns> helm.sh/resource-policy=keep on its first upgrade (runbook, "Chart 2.4.21"). Ownership is read from the cluster at render time (lookup); a render without a cluster (helm template) does not see it and shows what a first install gets.

helm uninstall no longer deletes a namespace the chart rendered (keep): delete it yourself. The prober chart follows the same rules since 1.0.6 (#1405); its namespace.create was already false by default.

The agent is effectively singleton-per-node by design: it owns host paths (/run/aether, /opt/cni/bin, /etc/cni/net.d, …) and the CNI plugin, so only one agent release should target a given set of nodes.

Every object the chart renders (not the pods its workloads create: see below) carries the recommended app.kubernetes.io/* labels (name, instance, version, component, part-of: aether, managed-by) plus helm.sh/chart. Workload selectors use the immutable subset (name + instance + component).

Pods carry only the labels that do not change between releases: the selector labels, part-of and managed-by. helm.sh/chart and app.kubernetes.io/version are on the DaemonSet / Deployment object, not on its pod template (aether chart

= 2.4.15, #1363), so a chart release rolls only the workloads whose pod template changed (its spec, or its metadata: the config checksum annotations are there to do exactly that). The image digests are part of the pod template, and today every in-repo image gets a new digest with every commit, so a deploy of a new commit still rolls everything: see the runbook. To see which release a pod belongs to, read its owner (kubectl get ds,deploy -A -l app.kubernetes.io/part-of=aether -L helm.sh/chart,app.kubernetes.io/version) or helm list -A. See docs/runbook.md, "Which workloads a chart upgrade rolls". The prober chart follows the same rule since 1.0.5 (#1372), and its authz canary's objects carry app.kubernetes.io/component: authz-canary instead of the prober's component (#1373).

Publish to Quay (OCI)¶

Charts and images both publish to the aethermesh organisation on quay.io — the one setting in bazel/registry/registry.bzl (proposal 040); every coordinate below is derived from it:

Artifact	Reference
crds chart	`quay.io/aethermesh/chart-crds`
aether chart	`quay.io/aethermesh/chart-aether`
prober chart	`quay.io/aethermesh/chart-prober`
udsecho chart	`quay.io/aethermesh/chart-udsecho`
agent image	`quay.io/aethermesh/agent`
mesh-dns image	`quay.io/aethermesh/mesh-dns`
proxy-supervisor image	`quay.io/aethermesh/proxy-supervisor`
registrar image	`quay.io/aethermesh/registrar`
controller image	`quay.io/aethermesh/controller`
cni-install image	`quay.io/aethermesh/cni-install`
prober image	`quay.io/aethermesh/prober`
udsecho image	`quay.io/aethermesh/udsecho`
proxy image	`quay.io/aethermesh/proxy` (built by the `//proxy` workspace, published by `.github/workflows/proxy-release.yml`; the chart's pin moves there with the first proxy release after the cut-over)

Images and charts are pushed separately, images first. A chart is NOT pushed with helm push: helm appends the chart's name: to its base, so it can only write <base>/aether, and Quay has no nested repositories to hold charts/aether (and a bare prober / udsecho would collide with those images). chart_push (//bazel/helm:defs.bzl) writes the identical artifact — the packaged .tgz as the helm chart-content layer, Chart.yaml as the helm config — with the pinned oras (//bazel/oras) to the repository it names, so helm pull oci://quay.io/aethermesh/chart-aether --version <v> reads it like any helm-pushed chart.

docker login quay.io        # image_push and oras both read the Docker config

# The images a chart references (by digest, plus the latest / dev-<sha> tags):
bazel run --stamp //charts/aether:aether.push_images

# The chart, under its packaged version:
bazel run --stamp //charts/aether:aether.chart_push

# The same chart again under `<version>-<git-commit>` (chart only — the images
# are already up as digests). CI does both (#692).
bazel run --stamp //charts/aether:aether_commit.chart_push

In CI, .github/workflows/publish.yaml does exactly this, logged in with the Quay robot account (secrets QUAY_USERNAME / QUAY_TOKEN of the release environment, which only runs on main). Never push by hand: the release workflow is the only publisher. To target a different registry, change bazel/registry/registry.bzl — nothing else spells the registry out.