
# nats-operator v1 design

Three Kubernetes controllers that between them deploy NATS clusters, own their auth plane, and manage and balance JetStream. Vocabulary is `CONTEXT.md`'s; "operator" unqualified is not used. Every nats-server citation is against v2.15.0 unless stated.

## 1. Scope

In scope: NATS clusters and superclusters spanning Kubernetes clusters, leaf nodes, the auth plane (NATS operator, system account, accounts, users), JetStream resources (streams, consumers, key-value buckets, object stores), balancing, evacuation, rollout, scale-down and server replacement.

Out of scope:

- Spreading client connections after a rollout.
- Delivering creds past the Kubernetes cluster boundary; External Secrets, SOPS or similar carry them.
- Node pools, storage classes, DNS zones; GitOps around the controllers provides them.
- General self-healing beyond what rollout, scale-down and balancing need.
- Moving a stream to a NATS cluster outside its supercluster.
- Migrating any existing deployment onto these controllers.
- A KMS integration for signing keys.
- Rotating identity keys: a new identity is a new NATS operator or account, which is a migration.

## 2. Architecture

### 2.1 Controllers and API groups

Three separately deployable controllers, each owning one API group, plus one shared group every controller reads. All groups are `v1beta1`, every kind is namespaced, and every kind name is prefixed `Nats`. An API group marks one controller's boundary for installation, RBAC and versioning, so a group with one kind is not a smell (ADR 0001).

| Group | Owner | Kinds |
|---|---|---|
| `nats-operator.io` | shared | `NatsReferenceGrant`, `NatsOperatorTrust`, `NatsAccountTrust`, `NatsConnection` |
| `cluster.nats-operator.io` | cluster controller | `NatsCluster` |
| `auth.nats-operator.io` | auth controller | `NatsOperator`, `NatsSystemAccount`, `NatsAccount`, `NatsUser` |
| `jetstream.nats-operator.io` | JetStream controller | `NatsStream`, `NatsConsumer`, `NatsKeyValue`, `NatsObjectStore`, `NatsBalancer`, `NatsSystemBalancer`, `NatsClusterEvacuation` |

### 2.2 What each controller reads

No controller reads another controller's group. Every kind two controllers share lives in `nats.` (ADR 0002).

- Cluster controller: `cluster.*`, and `NatsOperatorTrust`, `NatsAccountTrust`, `NatsConnection`, `NatsReferenceGrant`.
- Auth controller: `auth.*` and `nats.*`; it writes the status of reference-form trust objects (section 5.1).
- JetStream controller: `jetstream.*`, and `NatsConnection`, `NatsReferenceGrant`.

The JetStream and auth controllers act on NATS clusters the cluster controller did not deploy as readily as on managed ones.

### 2.3 What each kind does

| Role | Kinds | What exists because of it |
|---|---|---|
| Creates | `NatsCluster` | one StatefulSet and ConfigMap per server, Services, PDB, self-signed route certificates |
| Creates | `NatsOperator`, `NatsSystemAccount`, `NatsAccount`, `NatsUser` | keys (unless supplied), signed JWTs pushed to resolvers, creds Secrets |
| Creates | `NatsStream`, `NatsConsumer`, `NatsKeyValue`, `NatsObjectStore` | objects on a NATS server |
| Acts | `NatsSystemBalancer`, `NatsBalancer`, `NatsClusterEvacuation` | nothing lasting; leader and placement moves |
| Data | `NatsConnection`, `NatsReferenceGrant`, `NatsOperatorTrust`, `NatsAccountTrust` | nothing; read by the controllers |

There is no supercluster kind: each `NatsCluster` carries its gateway list, and the supercluster is what those lists form (ADR 0006).

## 3. Resource API

The API is specified by the user stories under `docs/content/docs/stories/`, whose manifests are the source of truth; this section states only the rules the manifests do not show on their own.

| Story | Covers |
|---|---|
| 01-quickstart | `NatsCluster`, `NatsConnection`, `NatsStream`; rollout status |
| 02-auth-plane | `NatsOperator`, `NatsSystemAccount`, `NatsAccount`, `NatsUser`, `NatsOperatorTrust`; bring-your-own-key users |
| 03-unmanaged | JetStream against a NATS cluster nobody here deployed; adoption; `NatsConsumer`, `NatsKeyValue`, `NatsObjectStore` |
| 04-team-self-service | namespaces and `NatsReferenceGrant` |
| 05-account-wiring | exports and imports |
| 06-supercluster | gateways, literal `NatsOperatorTrust`, remote controllers' users |
| 07-balancing | `NatsSystemBalancer`, `NatsBalancer`, pools, the stepdown export preset |
| 08-stream-transfer | moving a stream between NATS clusters of a supercluster |
| 09-acceptance | a three-cluster production supercluster |
| 10-leafnodes | leaves, with and without the hub's auth plane; `NatsAccountTrust` |
| 11-evacuation | `NatsClusterEvacuation`; the `NatsCluster` deletion guard |
| 12-metrics | the controllers' metrics over verified TLS, and their egress |
| 13-join-supercluster | joining a supercluster nobody here deployed; `auth.accountTrustRefs` |
| 14-nsc-adoption | adopting the keys of a NATS operator and accounts made with `nsc`; what the first re-sign replaces |

Rules across kinds:

- A Secret is always read through `secretKeyRef: {name, key}`. `key` defaults to `user.creds` for credentials and `ca.crt` for a CA.
- A `NatsUser` writes creds, optionally, in the shape a `NatsConnection` reads: `credentials.secretKeyRef`.
- A reference names its kind wherever more than one kind can answer it.
- Mutually exclusive fields (`publicKey` and `credentials`, `accountRef` and `publicKey`, `stream` and `streamRef`, `keys.identity` and `jwt`, `preset` and `permissions`) are refused together at apply by CEL.
- A one-shot imperative is an annotation the controller clears once it has acted (`force-step`, `restart-all`, `replace-server`); desired state is always a typed field. `force-delete` is read only while its `NatsCluster` is being deleted, and never cleared.
- TLS is optional on every listener (client, routes, gateways, leafnodes). Route TLS is on by default, self-signed when no certificate is named, and can be turned off.

## 4. Cluster controller

### 4.1 Workload

Each server is its own StatefulSet with `replicas: 1`, its own rendered ConfigMap, and an explicit `server_name` (ADR 0004). The controller also owns the client and headless Services, a PodDisruptionBudget with `maxUnavailable: 1` across the NATS cluster's pods, and the external gateway and leafnode Services rendered from the templates in spec. It writes no object of those names that exists and it does not control; the `NatsCluster` then reads `Ready=False, reason: ReconcileFailed`, naming every such object. The servers' pods prefer distinct nodes unless `podTemplate` sets `affinity`. Pod and PVC templates pass through; `GOMEMLIMIT` and the JetStream memory store derive from `resources.limits.memory` and the file store from the PVC size, each overridable under `jetstream.limits`. A `prometheus-nats-exporter` sidecar, its image pinned by digest and its resources small unless `exporter.resources` sets them, is on by default; `exporter.enabled: false` turns it off. The `nats` container's startup probe is `/healthz?js-meta-only=true`, which passes once the server is current with a meta leader, its readiness probe `/healthz?js-server-only=true` and its liveness probe `/healthz?js-enabled-only=true`; the full `/healthz` is not probed, since it looks up every account the meta group has a stream for, up to 1.9 seconds each where no resolver answers (`monitor.go:4072`, `const.go:250`), as in a supercluster whose other members hold accounts this NATS cluster does not preload. A probe or lifecycle hook `podTemplate` sets on the `nats` container merges over the rendered one field by field: a handler of the rendered kind that omits `port` keeps the rendered port, and a handler of another kind replaces the rendered handler. `tls` puts the client listener under TLS, its certificate from `secretRef` or from `certManager` issued for the client Service's names; clients then dial `tls://`, as `status.endpoints.client` reads, and need the CA that issued it, and the controller's own system connection trusts the Secret's `ca.crt`, or the system roots without one. Unset, clients connect in the clear. The monitoring port, which has no authentication, is on the headless Service and never the client Service; it stays reachable at each pod's address, so the controller renders a NetworkPolicy over the pods that admits it only from the cluster controller's own namespace. The same policy admits the exporter's metrics port, which republishes what the monitoring port serves, from that namespace and the peers `exporter.from` names; the route port, which has no authentication of its own, only from the NATS cluster's own pods; and every other port the controller renders from anywhere. The exporter reads it over the pod's loopback, which no NetworkPolicy governs. `monitor.networkPolicy: false` renders none; a port `podTemplate` adds is not admitted by it. A self-signed route certificate is valid for a year and reissued under the same CA key once a third of that remains, the key kept in the Secret `<name>-routes-ca`, which no pod mounts. Pods meet the restricted Pod Security Standard and mount no ServiceAccount token, but `podTemplate` merges over that too, so whoever may write a `NatsCluster` runs pods with any privilege its namespace admits.

`spec.version` names the nats-server version the controller renders for; `image` sets the repository and a digest, never the tag. 2.15.0 is the minimum: lower is refused at apply. Any patch change is allowed, and so is one minor up or down; a move of two or more minors, either way, is refused at apply. The hazard is the Raft log, not the file store: a newer minor writes log entries an older one may not replay (2.15's `deleteRangeOp` panics 2.12), and upstream reads a new entry one minor before it writes it. Nothing constrains versions across a supercluster. A config key the version does not know fails the server's config load rather than warning (`opts.go:1920`).

### 4.2 Config changes

A rendered-config change reloads a server where every changed field is in the reload allow-list for its version, and restarts it otherwise. A reload succeeds or fails whole: one restart-only field rejects the batch. The allow-list mirrors `diffOptions` per supported version. Restart-only fields include gateway remotes, route and gateway listen addresses, JetStream `store_dir`, lame-duck settings, `system_account` (`reload.go:1969-1973`), the trusted NATS operators (`reload.go:1619`), an account preloaded from a `NatsAccountTrust`, and any resolver change: a reload accepts one but installs a resolver it never starts (`reload.go:2236`), so a claims update still lands in the old directory while lookups read the new one. Routes, TLS certificate and key, `server_tags`, authorization, `max_payload`, and raised JetStream limits reload. So does the preloaded system account JWT, which a revocation or a `jetstream-stepdown` export re-signs: a reload keeps the preload it started with (`reload.go:1614`, the preloads being unexported), the re-signed JWT reaches running servers as a claims update, and the preload is read again only at the next start. Leafnode remotes can be added and removed by reload (`reload.go:931-990`).

The controller requests a reload with `$SYS.REQ.SERVER.<id>.RELOAD` over its system user, confirms it by `/varz` `config_digest` and `config_load_time`, and falls back to a restart on any error.

### 4.3 Trust, resolver, gateways, leaf nodes

Every `NatsCluster` with an auth plane names its trust roots with `auth.trustRef` to a `NatsOperatorTrust` (section 5.1), and its own system credentials with `auth.systemCredentials`. It renders `operator`, `system_account`, and a full resolver with the system account's JWT preloaded, so a server boots before its gateways are up. `auth.accountTrustRefs` names `NatsAccountTrust`s whose account JWTs every server preloads beside it, on a leaf or not, so its accounts work where nothing pushes them. A trust there carrying no JWT is refused, reason `AccountTrustInvalid`, or `AccountTrustNotReady` while the auth controller has not written one; one account named twice, or by the list and a leaf remote, is preloaded once, and two different JWTs for one account are refused. `auth.resolver` is `Full` or `Cache`, `Full` by default except on a leaf that preloads no account; `Memory` is not offered because accounts arrive over the network. A preload is stored in the resolver directory only where that holds nothing newer for the account (`saveIfNewer`, `accounts.go:4689`), and a fetch never replaces it. A `NatsCluster` that is not a leaf runs `Full` on whatever directory it has, the JetStream volume or an `emptyDir`, and is never refused for it: on an `emptyDir` a restarted server serves the preloaded copy until a `Full` peer's sync or a claims push replaces it, and on the volume an account whose reference is removed stays in the directory and is served after the restart that removal causes.

Gateways: `gateway.remotes` lists every member of the supercluster (a NATS cluster's own entry is skipped), `gateway.discovery` is `Explicit` (every remote, `reject_unknown` on) or `Gossip` (seeds, `reject_unknown` off; gossip discovery then completes the mesh). The same list is in every member, kept alike by GitOps. Any change restarts servers: gateway config is not reloadable (`reload.go:1765-1790`). Members without `jetstream.domain` share one JetStream meta group, and two meta groups meeting over a gateway do not merge: the higher term wins, and a side whose log is shorter installs the winner's snapshot, deleting every stream the snapshot does not list (`jetstream_cluster.go:2413-2418`). A member that starts with an empty store joins cleanly, so a `NatsCluster` with JetStream, no domain and a remote other than itself is held where its servers may hold a meta log of their own, with `Progressing=False, reason: OwnMetaGroup`: it creates no server whose data volume claim is older than the `NatsCluster`, as one left by an earlier `NatsCluster` of the same name is, and it renders no change to the remotes, the domain or JetStream of running servers whose meta group is led by one of them and lists no peer outside the NATS cluster, as `/jsz` or `JSZ` reports it. The way out is deleting the servers' claims, or setting a domain. Such a change to servers the controller cannot observe, or whose leader it cannot read, waits with `Progressing=False, reason: ObservationFailed`. A meta group with no leader is not judged its own, nor is a store JetStream does not run on, and a NATS cluster that grows into a supercluster from one that holds streams is refused too, since a new member and one with a log look alike from its side. Enabling JetStream restarts a server: 2.15 enables it on reload (`reload.go:1809`), but no stream can then be placed on the server for an account it already held. `gateway.service` renders the external Service from its `type`, `annotations`, `loadBalancerSourceRanges` and `loadBalancerClass`; the last two are refused unless `type` is LoadBalancer, and the class when it changes while `type` stays LoadBalancer, a change Kubernetes refuses on the Service (CEL-enforced). An update keeps a class the template leaves unset, such as a webhook defaults. The Service publishes servers that are not Ready (`publishNotReadyAddresses`), on create and on every update: a JetStream server is Ready once it reaches a meta leader, and the members reach a server joining the supercluster's meta group only through this Service. The client Service and the leafnode Service publish Ready servers only. The source ranges are what narrows who reaches a gateway without TLS, where the network is the only admission. `gateway.advertise` is the address the server advertises. Gateway TLS, from `secretRef` or `certManager`, verifies peers both ways against the Secret's `ca.crt`, and a Secret without one holds the servers with `Progressing=True, reason: GatewayCertificateNotReady`. A gateway without `tls` is refused, `Ready` and `Progressing` False with reason `GatewayWithoutTLS` and nothing rendered, unless the cluster controller runs with `--allow-gateway-without-tls`. nats-server demands a certificate from every inbound gateway (`opts.go:3310`), and without a CA file verifies it against the system roots, so a certificate from any public issuer would join the supercluster, which has no other authentication; a certificate from ACME, which issues no `ca.crt`, is therefore not accepted.

Leaf nodes, hub side: `leafnodes` with `tls`, `service` (the same template as `gateway.service`, source ranges and class included) and `advertise`, refused without `auth` (CEL-enforced), since a listener without an auth plane admits any leaf into the global account. Leaf side: `leafRemotes[]`, each naming a `NatsConnection` (URL, CA, credentials) and optionally `localAccountTrustRef` or `localSystemAccount: true`; no `localAccount` is offered, since a leaf without an auth plane has only the global account. The remotes' creds and CAs go into a Secret projected into the config volume beside `nats.conf`, so kubelet refreshes both together and a remote added or removed reloads, where `auth.systemCredentials` gives the controller a system user to request the reload with; without one every config change restarts. A remote whose `NatsConnection` no grant admits is rendered out of the config and the Secret, the other remotes kept, and `LeafnodesConnected` reads False with the refusal naming it. A leaf running JetStream must set `jetstream.domain` (CEL-enforced). A leaf that trusts the hub's NATS operator needs a remote bound to its system account, authenticated as a user of the hub's system account; without that remote it cannot fetch an account it has not cached (`auth.go:1020`, `leafnode.go:721`). Its resolver is `Cache` when it preloads no account. A leaf preloading an account, through a remote's `localAccountTrustRef` or `auth.accountTrustRefs`, runs `Full` on `jetstream.volumeClaimTemplate` and is refused without one unless it sets `Cache`: a fetch never replaces a preload, only `Full`'s sync with the hub or a claims push does, and a persistent directory keeps what was synced across a restart. The preload is a copy, valid only within the account's `jwtTTL` unless that account sets `jwtTTL: 0`. `LeafnodesConnected` and `status.leafRemotes` count the servers holding each remote, from `LEAFZ` over the system user or `/leafz`; remotes binding the same local account are told apart by count alone, since `LEAFZ` names no remote.

### 4.4 Rollout

One server at a time: a step writes the server's ConfigMap and its pod template, which names the revision, and its StatefulSet restarts the pod. Order: highest ordinal first, the meta leader's server last. The controller moves no leader before a restart: the `preStop` enters lame-duck mode (`nats-server --signal ldm=<pid file>`), which steps down every Raft leader on the server and makes each of its nodes an observer before any client is told (`server.go:4472`, `raft.go:913-936`). The rendered `lame_duck_duration` of two minutes ends inside the pod's `terminationGracePeriodSeconds` of 300.

The gate to the next server: every server's pod is Ready, every server on the target revision reports it, with JetStream every server that answers is a member of the meta group, and the NATS cluster is Settled. A meta leader in another NATS cluster of the supercluster is asked for its peers, and the meta group's members are this NATS cluster's servers among them; when that leader does not answer, the meta group is judged from its followers (`FromFollowers`), and every server that answers counts as a member. The NATS cluster is observed over the system user while every server answers there, and otherwise from each server's `/varz`, `/gatewayz` and `/jsz`, as while servers still run a config without `auth` or with another system account. A closed gate reads `Progressing=True, reason: RollingRestart`, and after ten minutes `GateBlocked`, naming the groups that are not current, while `status.rollout.gate.since` says since when; it holds indefinitely, and nothing proceeds on a timeout. `spec.rollout.paused` stops a rollout before its next step; the annotation `cluster.nats-operator.io/force-step: "<server>"` restarts that waiting server now, through a closed gate and through `paused`, and is cleared once read. The same gate holds the next voluntary step after any involuntary disruption.

A rollout that adds or removes `auth` is not taken one server at a time, since no intermediate state of it is one NATS cluster. Every server pins a route to its own system account (`opts.go:6056-6072`), `$SYS` without `auth` and the system account's public key with it, and a server that cannot look up the account a route is pinned to closes it (`route.go:2142-2149`): a server under a NATS operator fetches only account keys (`accounts.go:4167`), and one without has no resolver to fetch with. So a server on one side of the change routes to none on the other, in either direction. The controller holds every server on the config it runs, with `Progressing=False, reason: AuthChangeBlocked` naming the two ways out and `status.rollout.pending` the servers it holds. The annotation `cluster.nats-operator.io/restart-all` restarts every one of them at once, through a closed gate and through `paused`, accepting the outage, and is cleared once read, whether or not a change waited for it; the rollout then waits on the gate as after any step, and ends once every server is Ready and the NATS cluster Settled under the new config. Recreating the `NatsCluster` is the other way out, and loses the JetStream data on any claim deleted with it.

### 4.5 Scale-down and server replacement

Scale-down and replacement are steps of the rollout (section 4.4), under its gate, one server at a time: servers beyond `replicas` first, highest ordinal first, then restarts, then replacements, the meta leader's server last. A removal once begun is carried through, `paused` or not.

Scale-down removes the servers beyond `replicas`: `$JS.API.SERVER.EVACUATE` one, wait for it to hold no Raft group and for Settled apart from itself, step it down if it leads the meta group, `$JS.API.SERVER.REMOVE` it, then delete its StatefulSet, its PVC only when the PVC carries the server's labels, and its ConfigMap only when the `NatsCluster` controls it; the `NatsCluster` then reads `Ready=False, reason: ReconcileFailed`, naming each one left. A stream whose replica count exceeds the new size stops the shrink with `Progressing=False, reason: ScaleDownBlocked`; nothing is forced. Without JetStream a server is deleted with no evacuation; with JetStream and no `auth.systemCredentials` nothing can be evacuated, and scale-down and replacement are blocked.

A change to `jetstream.volumeClaimTemplate` (class, size, zone) replaces servers: evacuate, remove, delete that server's StatefulSet and, under the same rule, its PVC, and recreate it under the same `server_name` once the old PVC is gone. The annotation `cluster.nats-operator.io/replace-server: "<server>"` replaces one server on demand. A server's peer ID is a hash of its name (`server.go:4079-4081`), so it rejoins as the same peer, but only after the five-minute removal tombstone lapses (`raft.go:316`, readmission `raft.go:4121-4169`, measured at 5m1s). The fresh pod starts at once and answers outside the meta group; the rollout gate holds on it until the meta leader lists it again. `Progressing` reads `ScalingDown` or `ReplacingServer` while a step runs.

### 4.6 Deletion guard

Deleting a `NatsCluster` with JetStream that still holds stream groups waits, with `Deleting=True, reason: JetStreamDataRemains` naming them, found through the controller's own system JSZ; a NATS cluster that cannot be observed waits too, with `reason: ObservationFailed`. The annotation `cluster.nats-operator.io/force-delete` overrides both. `NatsClusterEvacuation` (section 6.6) is how the data leaves first.

### 4.7 Status

`Ready`, `Settled`, `Progressing`, and where they apply `GatewaysConnected` and `LeafnodesConnected`; `endpoints` (client, monitor, gateway), the config revision and how it was applied, the rollout block (updated, current, pending servers, the gate and since when), and one entry per server.

## 5. Auth controller

The auth controller runs in the home cluster only, and stops at JWT claims: it configures nothing about JetStream beyond the limits signed into account JWTs. A `NatsAccount`'s `limits.jetstream` holds those for the account as a whole or, under `tiers`, by tier, an enum of `R1` to `R5`: nats-server looks a stream's tier up as `R` and its replica count, at most five, and reads no other name; never both (CEL-enforced), since an account JWT carrying both is invalid.

### 5.1 Keys and trust

Keys (ADR 0005). `NatsOperator`, `NatsSystemAccount` and `NatsAccount` take optional `keys: {identity: {secretKeyRef}, signing: [{name, secretKeyRef, retiring}]}`. Set, the keys are adopted; an adopted account keeps its public key, so user JWTs already issued under it keep working without `NatsUser` resources, while its claims are re-signed from spec. A first signing for which the servers hold a JWT of the key is an adoption, and the two JWTs are compared before the new one is signed: a claim the held JWT carries that the signed one would not, whether no field expresses it or the spec omits it, refuses the signing, Ready False with reason `AdoptionDropsClaims` naming each by its JWT field path, until the spec declares it or `adoption.droppedClaims: Accept` (`Refuse` when omitted) accepts the loss; a claim set to another value is a change and passes; a JetStream tier under a name other than `R1` to `R5` refuses the signing outright, reason `TierInexpressible`, since no spec names it. A refused account records no JWT, so the servers keep theirs, and the servers are asked again on every reconcile until it is signed. A `NatsSystemAccount` is compared the same way by its NatsOperator, which carries the refusal on its own Ready condition and signs no system account JWT; the `NatsSystemAccount` mirrors the condition, and its own `adoption.droppedClaims` accepts the loss. Omitted, the controller generates keys into Secrets named `<name>-<operator|systemaccount|account>-<identity|signing-1>` and annotated `auth.nats-operator.io/generated-for: <operator|systemaccount|account>/<name>`, with no owner reference, so they outlive the object and an object of the same kind and name applied again takes the same keys; it reads a Secret under such a name only where that annotation names the object; any other is Ready False, reason `SecretConflict`.

Offline identities. `NatsOperator.spec.jwt` holds a NATS operator JWT signed elsewhere, exclusive with `keys.identity`; an account's `publicKey` is exclusive with `keys.identity`. Either requires at least one signing key. A `NatsAccount` no grant admits to its NatsOperator is refused before its key is recorded, Ready False, reason `ReferenceNotPermitted`. Deleting that grant once the account is signed deletes it from the servers through its NatsOperator's `status.deletedAccounts` and empties its `status.jwt` once that list records it. A `NatsUser` of an account not admitted is not signed, Ready False, reason `AccountNotAdmitted`. An account's public key is unique under its NatsOperator: a `NatsAccount` whose key is the identity of the `NatsSystemAccount` that NatsOperator references, as its status, `publicKey` or identity seed Secret gives it, or another `NatsAccount` under it records, is not signed, Ready False, reason `PublicKeyInUse`, naming the holder; where two accounts record one key, the older keeps it. nats-server takes any account JWT its NATS operator signed for the system account's key in place of the system account's own and cuts off the system account's users (pinned in-process by `TestResolvers_SystemAccountKeyPushed`), so this refusal is what keeps a namespace granted the NatsOperator out of the system account. The Kubernetes cluster then holds signing seeds only: one NATS operator signing key for account JWTs (the system account's included), and one per account for its users and its activation tokens, which an account signing key may sign (`activation_claims.go:34` in jwt/v2, accepted at `accounts.go:3094-3108`).

Custody. Seeds live only in Kubernetes Secrets, never in spec or status. Through `secretKeyRef` those Secrets may come from an external store, and that copy is the backup; losing the home namespace without one loses the ability to change any account. A generated identity seed is generated once: where its Secret is gone while the object's `status.publicKey` records the identity, the object reads Ready False, reason `SeedLost`, and no new identity replaces it, since one would orphan every JWT issued under the old.

Rotation. A signing key is added, the controller re-signs what it manages (users, or for the NATS operator, accounts) and rewrites creds Secrets, then the key marked `retiring: true` is removed, which invalidates everything still signed by it. Any change to the NATS operator JWT (signing keys, system account) is restart-only (`reload.go:1619`): the `NatsOperatorTrust` copies are updated first, then every server in the supercluster rolls. With an offline identity each such change also needs the JWT re-signed offline.

Scoped signing keys. A signing key of a `NatsAccount` or a `NatsSystemAccount` may carry a `scope`: a role, and the publish and subscribe permissions, connection types and limits for subscriptions and payload that the servers apply to every user the key signs, in place of the user JWT's own (`UserScope` in jwt/v2, applied at `auth.go:1031-1045` and `client.go:961-966`). A scoped key signs only the `NatsUser`s whose `role` is its role, the first such key not retiring, so a scoped key rotates as any other does; every other user and every activation token is signed by the first key that is neither retiring nor scoped. The rest of a scope's template (response permissions, source networks, times, locale, bearer token, data limit) and its description have no field. The schema refuses a `scope` on a `NatsOperator` signing key, whose JWT has no place for one.

Trust objects (ADR 0003). `NatsOperatorTrust` holds either `operatorRef` (same Kubernetes cluster; the auth controller writes the NATS operator and system account JWTs into its status) or the literal `operatorJWT` and `systemAccountJWT`, copied by GitOps wherever no auth controller runs. `NatsAccountTrust` likewise holds `accountRef` or a literal `publicKey` and optional `jwt`, which a `NatsCluster` preloads through `auth.accountTrustRefs` or a leaf remote. `NatsOperator.spec.systemAccountRef` selects the live system account among any number of `NatsSystemAccount`s; an unreferenced one is not signed.

### 5.2 Distribution

Account JWTs reach servers through the full resolver: `$SYS.REQ.CLAIMS.UPDATE` crosses gateways (every account runs interest-only over gateways since 2.9.0) and restarts nothing. The system account JWT is signed into the `NatsOperator` status and pushed as the `NatsSystemAccount`'s own, whose `status.distribution` and `JWTPushed` event report it; it is not pushed while that NatsOperator reads `RevocationsUnrecovered` True, and is signed again where a server holds a newer one. `status.distribution` reports how many servers hold the current JWT, built from the STATSZ roster and each server's `CLAIMS.LOOKUP`, since the server offers no aggregate. A full resolver stores a pushed JWT before anything checks who signed it (`accounts.go:4563-4573`) and refuses it only when it is applied (`server.go:2211`), at the next start if the account is not loaded. So the controller reads from each server's `VARZ` the NATS operator JWT it runs under (`monitor.go:1294`), sends nothing where no server lists the signing key, and counts no server current that does not list it; the account then reads `Distributed` False, reason `UntrustedSigner`, until every server lists the key. The `NatsOperator` names the key: `SigningKeyUntrusted` True, reason `UntrustedSigner`, gives its active signing key and how many servers do not list it, and is False, reason `SignerTrusted`, where none is known not to; it is Unknown, reason `Unreachable`, while no server can be asked, and is looked at again every five seconds while it is True or `Unreachable`, and whenever a server joins, leaves or reports other keys on `VARZ`. A server whose `VARZ` reports no NATS operator JWT, or that does not answer `VARZ`, is not known to trust any key, and `status.distribution` counts at most the servers known to trust the signer. One configured by `trusted_keys` reports none, since nats-server never puts those keys on `VARZ` (`opts.go:536`), and runs no resolver a push reaches, since a full or cache resolver will not start without a NATS operator JWT (`accounts.go:4443`); one does not answer while the auth controller's system user lacks the `$SYS.REQ.SERVER.PING.VARZ` permission, which the `auth-controller` preset of a user signed before 0.4.0 does not hold. Where no server is known not to list the signer and such a server is in the roster, the JWT is sent to every server, and the account reads `Distributed` False, reason `TrustUnknown`, naming how many servers do not say, and the `NatsOperator` `SigningKeyUntrusted` Unknown with the same reason. Reissuing the system user's creds clears the second case; nothing clears the first but running the server under a NATS operator JWT. Without `--system-connection` nothing is pushed and no deleted user is kicked, and every account and user says so with `Distributed` False, reason `NoSystemConnection`, and every `NatsOperator` with `SigningKeyUntrusted` Unknown for the same reason.

Deletes. The resolver's catch-up sync only adds, so an account deleted while a server was away survives there. Two measures close it: the controller re-sends `$SYS.REQ.CLAIMS.DELETE` for deleted accounts whenever a server or NATS cluster rejoins (seen in STATSZ), and every account JWT carries an expiry, `NatsAccount.spec.jwtTTL` (default 48h), re-signed at half its TTL. A home outage therefore costs nothing for half a TTL and expires accounts after a whole one. System account JWTs carry no expiry, because their literal copies in `NatsOperatorTrust` would otherwise go stale; for the same reason `jwtTTL: 0` signs an account JWT with no expiry, which an account preloaded through a literal `NatsAccountTrust` sets, since that copy otherwise stops working a TTL after it was taken. The controller pushes `CLAIMS.UPDATE` for every change and never an older JWT: a push carries no `iat` guard, and an older JWT pushed over a newer one stays.

### 5.3 Users

A `NatsUser` takes `permissions` or a `preset`: `cluster-controller`, `jetstream-controller` and the auth controller's own (claims updates, the `CONNZ` ping, `KICK`) for system users; `readonly`, `leafnode` for ordinary accounts. A controller preset grants exactly the subjects its controller requests, listed on the NATS permissions page, and subscribes only under its own inbox, `_INBOX.<preset>.>`, which its controller dials with, so no controller reads another's replies. With `role` in their place, and exclusive with them and with `connectionTypes`, the user is signed by the account's scoped signing key of that role with no permissions or limits of its own, which is the only user JWT nats-server accepts from a scoped key (`signingkeys.go:102` in jwt/v2, called at `auth.go:1035`); a role that no key not retiring has leaves the user unsigned, Ready False, reason `InvalidJWT`. It writes creds to `credentials.secretKeyRef` if given, or, with `publicKey` instead, receives only a signed JWT in status and no Secret. A user's public key is unique in its account: a `publicKey` another `NatsUser` of the account records is not signed, Ready False, reason `PublicKeyInUse`, naming the holder, and deleting that user revokes nothing; where two record one key, the older keeps it. A grant admitting `NatsUser`s to an account lets the grantee claim and revoke any user key of that account, keys issued outside the controller included, since no NATS record tells those apart. Controllers outside the home cluster run as system users declared at home, scoped by preset, never expiring, and carried across by External Secrets or SOPS.

Deletion: the user's public key is added to the account JWT's `revocations` and pushed; a finalizer holds the resource until `distribution` shows every server current; the controller then finds the user's live connections with a system `$SYS.REQ.SERVER.PING.CONNZ` (`events.go:1286`), which each server filters by account and user public key (`monitor.go:407-413`) and kicks each with `$SYS.REQ.SERVER.<id>.KICK` (`events.go:62`); the creds Secret goes last. Revocations are recorded in the `NatsAccount` or `NatsSystemAccount` `status.revocations`, each with the keys that may have issued a revoked JWT, and the account JWT is signed from that record and the JWT it replaces, so losing either keeps them. Losing both, with the whole status, the controller takes them back from the JWT the servers hold (`$SYS.REQ.ACCOUNT.<key>.CLAIMS.LOOKUP`, `accounts.go:4579`) before signing again, as it does for an account or a system account made elsewhere and adopted; a revocation taken from that JWT carries the signing keys the JWT lists and the account's identity key, which may have signed a user JWT outside the controller. A JWT made elsewhere may revoke every user issued at or before a time, under the key `*` in place of a user's (`accounts.go:3035-3045`); the controller writes no such revocation itself, and takes one from the servers' JWT, records it and signs it as any other, under the same rules. While any server of the STATSZ roster does not answer, since it may hold a newer JWT than those that do, an account whose `status.distribution` records it distributed is not signed, Ready False with reason `RecoveringRevocations`; any other, a new account among them, is signed with the condition `RevocationsUnrecovered` True and not pushed, `Distributed` False with reason `Unreachable`, since a push would replace a JWT made elsewhere on the servers that do answer; the servers are asked again every five seconds and on a roster change, and the JWT is pushed once every roster server answers. A server gone for good leaves the roster after three missed polls, so a new account waits on it no longer than that. A server outside the roster is not asked, and revocations only it holds are not recovered. A revocation is dropped once none of those keys is left on the account, or once the revoked JWT has expired, which for users is never unless a user expiry is added. The controller signs users with a signing key only, so the revocation of a deleted or denied user whose key it issued, or of a replaced key, carries signing keys alone, and rotating an account's signing key is what resets that part of its list. A user whose `status.publicKey` is the `publicKey` of its spec claimed a key the identity key may have signed outside the controller, so its revocation, in a `NatsAccount` or a `NatsSystemAccount`, carries the identity key as well. The identity key never rotates, so a revocation taken from the servers' JWT, for a `NatsAccount` or a `NatsSystemAccount`, or one of a claimed user's key, is never dropped. A replaced key does not record whether it was claimed, so a claimed key replaced by another `publicKey` or a creds Secret is revoked by signing keys alone, and a JWT the identity key signed for it connects again once they have all rotated out. The system account's JWT is signed by its NatsOperator's reconcile, which records the revocations it signed, each with its issuers, in `status.systemAccount.revocations`; the `NatsSystemAccount` takes its record from those and the JWT, and the next signing takes them from both records. A user's `accountRef` cannot change. A user whose key changes, by a new `publicKey` or a creds Secret that had to be written afresh, has the key it held revoked from that moment through the same record, and the server closes that key's connections when it takes the account JWT (`accounts.go:3994-4026`); `status.replacedKeys` holds each such key until the account JWT revokes it, and a deleted user's finalizer waits on those too. `publicKey` stays mutable, since a client rotating its own key is ordinary and a creds Secret can be rewritten under any rule on spec.

### 5.4 Exports and imports

An import names an export by name and its exporter one of three ways. By `accountRef` of kind `NatsAccount`, under the same NatsOperator: the subject, type and response type come from the exporting account, a private export lists its importers, and the controller mints each an activation token; a cross-namespace import needs both a `NatsReferenceGrant` in the exporter's namespace and, for a private export, the importer listed. By `accountRef` of kind `NatsSystemAccount`, the one its NatsOperator references: the export is one of the two monitoring exports below, and the controller writes the importer's public key into the subject at the position the export fixes, since nats-server serves such an import for no other subject (`accounts.go:2872-2873`). By `publicKey`, for an account no `NatsAccount` or `NatsSystemAccount` under the NatsOperator describes, which may be one adopted from elsewhere or under another NATS operator: the controller cannot read that account's exports, so the import carries `subject` and `type` itself, and for a private export `activation.secretKeyRef`, a Secret holding the token the exporter issued, as a signed JWT belongs in a Secret rather than in spec; the token is checked as nats-server will check it, issued by that account for this one, for that subject and type, and an import whose token does not fit is left out with the reason. The other direction is a private export's importer named by `publicKey` in place of `kind` and `name`, for an account no `NatsAccount` describes: the exporter's reconcile mints that key an activation token into `status.exports[].importers[]`, where the reader takes it to the importing account, since a token is a signed JWT that admits no account but the one it names and needs no Secret; the token held is kept while the export's subject and type stand and the key that signed it is the one activation tokens are signed with, and minted again otherwise, so a rotation hands the reader a new token; an importer key no token can be minted for, one that is not an account public key, is left out, Ready False with reason `ActivationsUnsigned` naming it, and the account is signed regardless. `share` and `allowTrace` are signed as given, `share` on a service import and `allowTrace` on a stream import only, refused by CEL where `type` is in spec and left out by the controller where it is not. An import whose exporting `NatsAccount` or `NatsSystemAccount` has no `status.publicKey` yet holds the importer back: it is not signed, Ready and `ReferencesResolved` False with reason `ExporterPending` naming the exporter, until the exporter records its key, which reconciles the importer again; two accounts importing from each other still converge, since an account records its own key before it waits. An import that does not resolve for any other reason, an exporter that does not exist or that no grant admits, a `NatsSystemAccount` the NatsOperator does not reference, or an activation Secret that is missing among them, is left out, and the account is signed without it, Ready False with reason `ImportsUnresolved` or `ReferenceNotPermitted`. The account is reconciled again when an activation Secret it names changes. The export preset `jetstream-stepdown` expands to service exports of `$JS.API.STREAM.LEADER.STEPDOWN.*` and `$JS.API.CONSUMER.LEADER.STEPDOWN.*.*`, and the controller adds the matching prefixed imports to the system account's JWT; that is what lets the system balancer move the account's leaders, in a supercluster as in a lone NATS cluster. Every system account JWT carries the two exports `nsc` gives a system account, the service `$SYS.REQ.ACCOUNT.*.*` (`account-monitoring-services`) and the stream `$SYS.ACCOUNT.*.>` (`account-monitoring-streams`), each with the importer's public key fixed at the wildcard, so an account that imports them, in a JWT signed outside the controller or through a `NatsAccount`, reaches the monitoring of itself alone and keeps doing so when the system account is adopted. No controller imports them, and the server grants each account `CONNZ` and `STATZ` of itself without them (`events.go:1595-1625`, `2385-2404`). A system account JWT has no other export, and no import but those of the preset.

## 6. JetStream controller

### 6.1 One mode

The JetStream controller reaches every NATS cluster, managed or not, through a `NatsConnection`: `servers`, `tls.ca`, `credentials`. The credentials decide the account, so every JetStream resource names only `connectionRef` (ADR 0002). The system account cannot create, update, delete or step down another account's streams: the whole `$JS.API.>` surface resolves the caller's account (`jetstream_api.go:1107`, `:955-969`), so a per-account user is required for everything but observation, placement moves and evacuation.

### 6.2 Kinds

`NatsStream`, `NatsConsumer`, `NatsKeyValue`, `NatsObjectStore`. Each kind's fields are the corresponding NATS config in camelCase, beside `connectionRef` and the lifecycle policies of section 6.3: `StreamConfig` and `ConsumerConfig` from nats-server, `KeyValueConfig` and `ObjectStoreConfig` from nats.go v1.54.0 (`jetstream/kv.go`, `jetstream/object.go`), whose `bucket` is `name` here. A user-supplied `metadata` map is merged with the ownership marker, which the controller owns. Mirrors and sources are stream and key-value fields. One consumer kind: push when `deliverSubject` is set, pull otherwise. A consumer names its stream by `stream` (server-side name, for a stream with no resource) or `streamRef` (it then waits for the stream and inherits its connection). `connectionRef`, and a consumer's `stream` and `streamRef`, cannot change. A server-side name defaults to `metadata.name`.

### 6.3 Lifecycle policies

Typed fields, beside `connectionRef`:

- `adoptionPolicy: Never | Adopt | AdoptOrCreate` (default `Never`), with ACK's semantics: `Adopt` requires the object to exist and writes its config into spec; `AdoptOrCreate` applies the spec and late-initializes omitted fields; `Never` goes Terminal on an object it does not own.
- `deletionPolicy: Retain | Delete`, default `Retain` for streams, key-value buckets and object stores, `Delete` for consumers.
- `terminalPolicy: Hold | Retry` (default `Hold`): whether a Terminal condition waits for an edit or is rechecked every resync.

A resource being deleted whose `NatsConnection` no longer exists, or is no longer admitted by a grant, is released without its deletion policy running, and its server object stays.

Ownership is a marker naming the resource's UID in the object's own `metadata` map (`stream.go:129`, `consumer.go:133`).

### 6.4 Drift and immutable fields

The controller re-reads on a resync period and corrects drift by reapplying spec, reporting `Synced=False, reason: DriftCorrected` once. Fields nats-server will not change are refused at apply by CEL transition rules: for streams `name`, `storage`, `retention` to or from `WorkQueue`, `mirror`, `allowMsgCounter`, `persistMode`, and the one-way `sealed`, `denyDelete`, `denyPurge`, `allowMsgTTL`, `allowMsgSchedules` (`stream.go:2542-2604`); for key-value buckets and object stores `storage`, since each is a stream; for consumers `deliverPolicy`, `ackPolicy`, `replayPolicy`, start sequence and time, heartbeats, flow control, `maxWaiting` (`consumer.go:2542-2594`). `NatsConsumer` alone offers `recreateOnImmutableChange`.

### 6.5 Balancing

Two layered balancers. `NatsSystemBalancer` (at most one per NATS cluster, system credentials) evens leaders and copies over every account: placement moves by `$JS.API.ACCOUNT.STREAM.MOVE.<account>.<stream>`, which accepts the system account for any account (`jetstream_api.go:3188`) and picks servers matching the stream's configured placement tags as well as the request's (`jetstream_api.go:3269-3280`), and leader moves through each account's `jetstream-stepdown` export; `status.capabilities` reports `leader: Partial` for accounts without it. `NatsBalancer` (per account, its own connection) evens within pools and yields to the system balancer, including any stream the system balancer has a move pending on.

A pool selects `NatsStream`, `NatsKeyValue` and `NatsObjectStore` resources by label; a stream matching several pools belongs to the first and status reports `Overlapping`; streams in no pool, and streams with no resource, form the default pool.

A balancer's scope and boundary is the NATS cluster its connection reaches: it judges only groups placed there, its Settled gate reads only that NATS cluster (a cluster-filtered `PING.JSZ`), balancers in different Kubernetes clusters do not coordinate, and none moves the meta group. A declared `placement.cluster` pins a stream; balancers never move one across NATS clusters, and never widen or override declared placement. Leader moves are on by default; placement moves are opt-in. One move per pass, none while unsettled; `interval` paces it. One balancer at a time moves on a NATS cluster: each takes the NATS cluster's move lease, kept in the memory of the leader-elected JetStream controller, before a move and holds it until a pass of its own finds the move done, while the others hold with `Holding` reason `MoveLeaseHeld`; a lease lapses a minute after its holder's last pass. During a rollout a balancer pauses per step through its own Settled gate.

### 6.6 Evacuation

`NatsClusterEvacuation` (system credentials, whole NATS cluster only; its `connectionRef`, `from` and `to` cannot change) moves every stream, key-value bucket and object store in every account off `from.cluster`, save those whose resource sets `placement.cluster`, to servers matching `to.serverTags`, one `ACCOUNT.STREAM.MOVE` per stream with those tags as ephemeral placement; peer selection falls back to any NATS cluster matching them (`jetstream_api.go:3280-3313`), and consumers follow (`jetstream_cluster.go:9859`). `SERVER.EVACUATE` and `STREAM.PEER.EVACUATE` cannot do this: they stay in the stream's NATS cluster (`jetstream_cluster.go:9723-9763`). It refuses to start if a source server carries the target tags. While a server of the source does not answer, the meta group has no leader, or its leader reports a server of the NATS system offline, it makes no move, counts none done and holds with reason `ServersDown`. A resource whose own spec sets `placement.cluster` is never moved: one naming the source is listed in `status.pinned`, and the evacuation never becomes Ready while any remains; the rest are left to their owners. The tags are not written to a moved stream's config, and a config that still names the source is pulled back by its owner's next placement edit while the source exists; the evacuation rewrites nothing and lists moved streams with no resource whose config names the source in `status.stalePlacement`. Deleting it mid-run cancels pending moves with `CANCEL_MOVE`. Balancers skip streams under evacuation.

## 7. Tenancy

Every kind is namespaced. A reference crosses namespaces only where a `NatsReferenceGrant` in the target namespace admits it (`from`: group, kind, namespace; `to`: group, kind, optional name), checked on every reconcile, so deleting a grant revokes what it admitted (ADR 0007). Withdrawing a grant that admits a `NatsCluster` to a `NatsOperatorTrust` or a `NatsAccountTrust` holds the `NatsCluster` at its last render instead, since both carry only public material. There is no admission webhook. Creds Secrets land in the user's namespace. `NatsSystemAccount` is a kind of its own so RBAC can grant it to the platform team alone. A grant admitting a `NatsCluster`'s leaf remote to a `NatsConnection` hands that connection's credentials to the `NatsCluster`'s namespace, where they are copied into the Secret `<name>-leaf-remotes`. Whichever `NatsAccount` records an account key first holds it, so a grant admitting `NatsAccount`s to a `NatsOperator` lets the grantee take any account key no `NatsAccount` records yet, an adopted account's before its own `NatsAccount` reconciles included. A preset scopes subjects, not NATS clusters: `$JS.API.SERVER.REMOVE` names its peer in the payload, not the subject, so a `cluster-controller` user in one Kubernetes cluster can remove a server from any NATS cluster of the supercluster.

## 8. Guarantees

The design promises about client behaviour during leader moves, placement moves, rollouts and evacuations exactly what NATS promises, and no more. A prototype at 200 msg/s on 2.15.0 saw no acknowledged message lost or delivered twice, at most one publish-ack timeout per move absorbed by a `Nats-Msg-Id` retry, and one `Leadership Changed` on pull consumers per leader change; that is evidence, not contract.

## 9. Observability

Every resource reports conditions. The controllers' own metrics carry skew, pending moves and held passes.


