Story 02 · The auth plane
Put a NATS cluster under a NATS operator
Contents
This guide shows you how to run a NATS cluster under a NATS operator that the auth controller owns, with its accounts and users declared as Kubernetes resources.
The manifests create the NATS operator demo, the accounts sys and orders, the users of both, the NATS cluster demo, and the stream ORDERS in the account orders.
Before you begin#
You need:
- A Kubernetes cluster with the cluster controller, the auth controller and the JetStream controller installed, and the chart value
auth.systemConnectionset tonats-system/auth-controller. Install shows how. - The namespace
nats-system, with noNatsClusternameddemoin it. If you followed the quickstart, do its clean-up first. - Nodes with 3 CPUs and 12Gi of memory that pods can still request, and a StorageClass named
standard. - The NATS CLI.
Create the NATS operator#
Apply the NatsOperator.
Its system account is sys.
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsOperator
metadata:
name: demo
namespace: nats-system
spec:
systemAccountRef:
name: sys
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsoperator.yamlThe manifest has no keys, so the auth controller generates the identity key and a signing key, and keeps each seed in a Secret.
To adopt seeds that you already have, set keys instead, as Adopt a NATS operator and accounts made with nsc does.
Keys and trust in the design describes both, and how to rotate a key.
Read the status:
kubectl -n nats-system get natsoperator demo -o yamlThe status in the output is similar to this:
status:
observedGeneration: 1
conditions:
- type: Ready
status: "True"
reason: Signed
publicKey: ODEMO...IDENTITY
signingKeys:
- ODEMO...SIGNING1
seedSecrets:
identity: demo-operator-identity
signing: [demo-operator-signing-1]
jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
systemAccount:
name: sys
publicKey: ADEMO...SYS
jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
seedSecrets lists the Secrets that contain the seeds.
Declare the accounts#
Apply the system account sys and the account orders.
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsSystemAccount
metadata:
name: sys
namespace: nats-system
spec:
operatorRef:
name: demo
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
name: orders
namespace: nats-system
spec:
operatorRef:
name: demo
jwtTTL: 48h
limits:
connections: 500
subscriptions: 10000
payload: 1Mi
jetstream:
memoryStorage: 1Gi
diskStorage: 50Gi
streams: 20
consumers: 200
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsaccounts.yamlThe limits of orders are signed into its JWT.
An account without limits.jetstream has no JetStream.
AccountLimits in the API reference lists every limit.
The JWT of orders expires after its jwtTTL of 48h, and the auth controller signs it again before then.
If the auth controller is down for longer than half of jwtTTL, the JWT can expire, and the servers then close the connections of the account.
Watch the gauge nats_operator.account.jwt_expiry, described in Telemetry.
Declare the users#
Apply the users.
orders-service is the client of this guide, orders-batch brings its own key, orders-jetstream is the user the JetStream controller acts as in orders, and the last three are the controllers’ own users in the system account.
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: orders-service
namespace: nats-system
spec:
accountRef:
kind: NatsAccount
name: orders
permissions:
publish:
allow: ["orders.>", "$JS.API.>"]
subscribe:
allow: ["orders.>", "_INBOX.>"]
credentials:
secretKeyRef:
name: orders-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: orders-batch
namespace: nats-system
spec:
accountRef:
kind: NatsAccount
name: orders
publicKey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4
permissions:
publish:
allow: ["orders.batch.>"]
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: orders-jetstream
namespace: nats-system
spec:
accountRef:
kind: NatsAccount
name: orders
permissions:
publish:
allow: ["$JS.API.>"]
subscribe:
allow: ["_INBOX.>"]
credentials:
secretKeyRef:
name: orders-jetstream-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: cluster-controller
namespace: nats-system
spec:
accountRef:
kind: NatsSystemAccount
name: sys
preset: cluster-controller
credentials:
secretKeyRef:
name: cluster-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: jetstream-controller
namespace: nats-system
spec:
accountRef:
kind: NatsSystemAccount
name: sys
preset: jetstream-controller
credentials:
secretKeyRef:
name: jetstream-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: auth-controller
namespace: nats-system
spec:
accountRef:
kind: NatsSystemAccount
name: sys
preset: auth-controller
credentials:
secretKeyRef:
name: auth-controller-creds
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsusers.yamlEach user with credentials gets its creds in that Secret, under the key user.creds.
A controller’s user sets a preset in place of permissions.
NATS permissions lists the subjects of each preset.
A user that sets publicKey gets no Secret.
Read the JWT that the auth controller signed for orders-batch:
kubectl -n nats-system get natsuser orders-batch -o yamlThe status in the output is similar to this:
status:
observedGeneration: 1
conditions:
- type: Ready
status: "True"
reason: Signed
publicKey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4
jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
Give status.jwt to the client, which already has the seed.
Connect the auth controller to the servers#
The auth controller pushes account JWTs to the servers through the NatsConnection set in auth.systemConnection.
Apply it, with the creds of the user auth-controller:
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
name: auth-controller
namespace: nats-system
spec:
servers: ["nats://demo.nats-system.svc:4222"]
credentials:
secretKeyRef:
name: auth-controller-creds
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsconnection-auth-controller.yamlUntil the connection reaches a server, the auth controller signs JWTs and no server receives them.
Deploy the NATS cluster under the NATS operator#
Apply the NatsOperatorTrust.
It points at the NatsOperator, and the auth controller writes the trust roots into its status.
apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
name: demo
namespace: nats-system
spec:
operatorRef:
name: demo
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsoperatortrust.yamlIn a Kubernetes cluster where no auth controller runs, the NatsOperatorTrust contains the JWTs themselves.
Build a supercluster across Kubernetes clusters shows that form.
Apply the NatsCluster.
auth.trustRef points at the trust roots, and auth.systemCredentials at the creds that the cluster controller connects with.
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: demo
namespace: nats-system
spec:
version: 2.15.0
replicas: 3
resources:
requests:
cpu: "1"
memory: 4Gi
limits:
memory: 4Gi
jetstream:
volumeClaimTemplate:
spec:
storageClassName: standard
resources:
requests:
storage: 20Gi
auth:
trustRef:
name: demo
systemCredentials:
secretKeyRef:
name: cluster-controller-creds
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natscluster.yamlCreate the stream in the account#
A NatsConnection decides the account of every resource that uses it, through its credentials.
Apply a connection with the creds of orders-jetstream, and the stream ORDERS on it:
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
name: demo-orders
namespace: nats-system
spec:
servers: ["nats://demo.nats-system.svc:4222"]
credentials:
secretKeyRef:
name: orders-jetstream-creds
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsconnection.yamlapiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
name: orders
namespace: nats-system
spec:
connectionRef:
name: demo-orders
adoptionPolicy: Never
deletionPolicy: Retain
terminalPolicy: Hold
name: ORDERS
subjects: ["orders.>"]
storage: File
replicas: 3
retention: Limits
maxAge: 72h
maxBytes: 5Gi
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsstream.yamlPublish as a user of the account#
Write the creds of
orders-serviceto a file:kubectl -n nats-system get secret orders-service-creds -o jsonpath='{.data.user\.creds}' | base64 -d > orders.credsForward the client port of the NATS cluster to your machine:
kubectl -n nats-system port-forward svc/demo 4222:4222 &Publish a message, and read the state of the stream:
nats -s nats://localhost:4222 --creds orders.creds pub orders.created '{"id": 1}' nats -s nats://localhost:4222 --creds orders.creds stream info ORDERS
Change the limits of an account#
Apply the account orders with its connection limit raised from 500 to 1000:
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
name: orders
namespace: nats-system
spec:
operatorRef:
name: demo
jwtTTL: 48h
limits:
connections: 1000
subscriptions: 10000
payload: 1Mi
jetstream:
memoryStorage: 1Gi
diskStorage: 50Gi
streams: 20
consumers: 200
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/02-natsaccount-orders.yamlThe auth controller pushes the new JWT to the servers, and no server restarts. Read the status of the account:
kubectl -n nats-system get natsaccount orders -o yamlThe status in the output is similar to this:
status:
observedGeneration: 2
conditions:
- type: Ready
status: "True"
reason: Distributed
- type: Distributed
status: "True"
reason: AllServersCurrent
message: 3 of 3 servers hold this JWT
publicKey: AORDERS...
jwtHash: 9c41e7
distribution:
servers: 3
current: 3
lastPushTime: "2026-09-25T16:20:03Z"
distribution counts the servers that hold the current JWT.
Distribution in the design describes how the auth controller finds that out.
Put a running NATS cluster under the NATS operator#
This guide deploys a new NATS cluster under the NATS operator.
To add auth to a NatsCluster that already runs without it, know that its servers cannot restart one at a time: a server with auth and a server without it cannot route to each other.
The cluster controller keeps every server on the config it runs, and the NatsCluster reads Progressing False with the reason AuthChangeBlocked.
Removing auth from a running NatsCluster is held the same way.
Take one of the two ways out:
- Set the annotation
cluster.nats-operator.io/restart-allon theNatsCluster, with any value, together with the change or after it. The cluster controller restarts every server at once on the new config, and removes the annotation. The NATS cluster serves no client until its servers are back. - Delete the
NatsCluster, and apply it again with the change. You lose the JetStream data on every claim that you delete with it.
Rollout in the design describes why the servers cannot route to each other.