Story 02 · The auth plane

Put a NATS cluster under a NATS operator

Contents

This guide shows you how to run a NATS cluster under a NATS operator that the auth controller owns, with its accounts and users declared as Kubernetes resources. The manifests create the NATS operator demo, the accounts sys and orders, the users of both, the NATS cluster demo, and the stream ORDERS in the account orders.

Before you begin#

You need:

  • A Kubernetes cluster with the cluster controller, the auth controller and the JetStream controller installed, and the chart value auth.systemConnection set to nats-system/auth-controller. Install shows how.
  • The namespace nats-system, with no NatsCluster named demo in it. If you followed the quickstart, do its clean-up first.
  • Nodes with 3 CPUs and 12Gi of memory that pods can still request, and a StorageClass named standard.
  • The NATS CLI.

Create the NATS operator#

Apply the NatsOperator. Its system account is sys.

01-natsoperator.yaml yaml ·
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsOperator
metadata:
  name: demo
  namespace: nats-system
spec:
  systemAccountRef:
    name: sys
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsoperator.yaml

The manifest has no keys, so the auth controller generates the identity key and a signing key, and keeps each seed in a Secret. To adopt seeds that you already have, set keys instead, as Adopt a NATS operator and accounts made with nsc does. Keys and trust in the design describes both, and how to rotate a key.

Read the status:

kubectl -n nats-system get natsoperator demo -o yaml

The status in the output is similar to this:

01-status-natsoperator.yaml yaml ·
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Signed
  publicKey: ODEMO...IDENTITY
  signingKeys:
    - ODEMO...SIGNING1
  seedSecrets:
    identity: demo-operator-identity
    signing: [demo-operator-signing-1]
  jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  systemAccount:
    name: sys
    publicKey: ADEMO...SYS
    jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...

seedSecrets lists the Secrets that contain the seeds.

Declare the accounts#

Apply the system account sys and the account orders.

01-natsaccounts.yaml yaml ·
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsSystemAccount
metadata:
  name: sys
  namespace: nats-system
spec:
  operatorRef:
    name: demo
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
  name: orders
  namespace: nats-system
spec:
  operatorRef:
    name: demo
  jwtTTL: 48h
  limits:
    connections: 500
    subscriptions: 10000
    payload: 1Mi
    jetstream:
      memoryStorage: 1Gi
      diskStorage: 50Gi
      streams: 20
      consumers: 200
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsaccounts.yaml

The limits of orders are signed into its JWT. An account without limits.jetstream has no JetStream. AccountLimits in the API reference lists every limit.

The JWT of orders expires after its jwtTTL of 48h, and the auth controller signs it again before then. If the auth controller is down for longer than half of jwtTTL, the JWT can expire, and the servers then close the connections of the account. Watch the gauge nats_operator.account.jwt_expiry, described in Telemetry.

Declare the users#

Apply the users. orders-service is the client of this guide, orders-batch brings its own key, orders-jetstream is the user the JetStream controller acts as in orders, and the last three are the controllers’ own users in the system account.

01-natsusers.yaml yaml ·
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: orders-service
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: orders
  permissions:
    publish:
      allow: ["orders.>", "$JS.API.>"]
    subscribe:
      allow: ["orders.>", "_INBOX.>"]
  credentials:
    secretKeyRef:
      name: orders-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: orders-batch
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: orders
  publicKey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4
  permissions:
    publish:
      allow: ["orders.batch.>"]
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: orders-jetstream
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: orders
  permissions:
    publish:
      allow: ["$JS.API.>"]
    subscribe:
      allow: ["_INBOX.>"]
  credentials:
    secretKeyRef:
      name: orders-jetstream-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: cluster-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: cluster-controller
  credentials:
    secretKeyRef:
      name: cluster-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: jetstream-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: jetstream-controller
  credentials:
    secretKeyRef:
      name: jetstream-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: auth-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: auth-controller
  credentials:
    secretKeyRef:
      name: auth-controller-creds
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsusers.yaml

Each user with credentials gets its creds in that Secret, under the key user.creds. A controller’s user sets a preset in place of permissions. NATS permissions lists the subjects of each preset.

A user that sets publicKey gets no Secret. Read the JWT that the auth controller signed for orders-batch:

kubectl -n nats-system get natsuser orders-batch -o yaml

The status in the output is similar to this:

01-status-natsuser-orders-batch.yaml yaml ·
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Signed
  publicKey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4
  jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...

Give status.jwt to the client, which already has the seed.

Connect the auth controller to the servers#

The auth controller pushes account JWTs to the servers through the NatsConnection set in auth.systemConnection. Apply it, with the creds of the user auth-controller:

01-natsconnection-auth-controller.yaml yaml ·
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: auth-controller
  namespace: nats-system
spec:
  servers: ["nats://demo.nats-system.svc:4222"]
  credentials:
    secretKeyRef:
      name: auth-controller-creds
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsconnection-auth-controller.yaml

Until the connection reaches a server, the auth controller signs JWTs and no server receives them.

Deploy the NATS cluster under the NATS operator#

Apply the NatsOperatorTrust. It points at the NatsOperator, and the auth controller writes the trust roots into its status.

01-natsoperatortrust.yaml yaml ·
apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
  name: demo
  namespace: nats-system
spec:
  operatorRef:
    name: demo
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsoperatortrust.yaml

In a Kubernetes cluster where no auth controller runs, the NatsOperatorTrust contains the JWTs themselves. Build a supercluster across Kubernetes clusters shows that form.

Apply the NatsCluster. auth.trustRef points at the trust roots, and auth.systemCredentials at the creds that the cluster controller connects with.

01-natscluster.yaml yaml ·
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: demo
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "1"
      memory: 4Gi
    limits:
      memory: 4Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: standard
        resources:
          requests:
            storage: 20Gi
  auth:
    trustRef:
      name: demo
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natscluster.yaml

Create the stream in the account#

A NatsConnection decides the account of every resource that uses it, through its credentials. Apply a connection with the creds of orders-jetstream, and the stream ORDERS on it:

01-natsconnection.yaml yaml ·
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: demo-orders
  namespace: nats-system
spec:
  servers: ["nats://demo.nats-system.svc:4222"]
  credentials:
    secretKeyRef:
      name: orders-jetstream-creds
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsconnection.yaml
01-natsstream.yaml yaml ·
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
  name: orders
  namespace: nats-system
spec:
  connectionRef:
    name: demo-orders
  adoptionPolicy: Never
  deletionPolicy: Retain
  terminalPolicy: Hold
  name: ORDERS
  subjects: ["orders.>"]
  storage: File
  replicas: 3
  retention: Limits
  maxAge: 72h
  maxBytes: 5Gi
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/01-natsstream.yaml

Publish as a user of the account#

  1. Write the creds of orders-service to a file:

    kubectl -n nats-system get secret orders-service-creds -o jsonpath='{.data.user\.creds}' | base64 -d > orders.creds
  2. Forward the client port of the NATS cluster to your machine:

    kubectl -n nats-system port-forward svc/demo 4222:4222 &
  3. Publish a message, and read the state of the stream:

    nats -s nats://localhost:4222 --creds orders.creds pub orders.created '{"id": 1}'
    nats -s nats://localhost:4222 --creds orders.creds stream info ORDERS

Change the limits of an account#

Apply the account orders with its connection limit raised from 500 to 1000:

02-natsaccount-orders.yaml yaml ·
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
  name: orders
  namespace: nats-system
spec:
  operatorRef:
    name: demo
  jwtTTL: 48h
  limits:
    connections: 1000
    subscriptions: 10000
    payload: 1Mi
    jetstream:
      memoryStorage: 1Gi
      diskStorage: 50Gi
      streams: 20
      consumers: 200
kubectl apply -f https://nats-operator.io/docs/stories/02-auth-plane/02-natsaccount-orders.yaml

The auth controller pushes the new JWT to the servers, and no server restarts. Read the status of the account:

kubectl -n nats-system get natsaccount orders -o yaml

The status in the output is similar to this:

02-status-natsaccount-orders.yaml yaml ·
status:
  observedGeneration: 2
  conditions:
    - type: Ready
      status: "True"
      reason: Distributed
    - type: Distributed
      status: "True"
      reason: AllServersCurrent
      message: 3 of 3 servers hold this JWT
  publicKey: AORDERS...
  jwtHash: 9c41e7
  distribution:
    servers: 3
    current: 3
    lastPushTime: "2026-09-25T16:20:03Z"

distribution counts the servers that hold the current JWT. Distribution in the design describes how the auth controller finds that out.

Put a running NATS cluster under the NATS operator#

This guide deploys a new NATS cluster under the NATS operator. To add auth to a NatsCluster that already runs without it, know that its servers cannot restart one at a time: a server with auth and a server without it cannot route to each other. The cluster controller keeps every server on the config it runs, and the NatsCluster reads Progressing False with the reason AuthChangeBlocked. Removing auth from a running NatsCluster is held the same way.

Take one of the two ways out:

  • Set the annotation cluster.nats-operator.io/restart-all on the NatsCluster, with any value, together with the change or after it. The cluster controller restarts every server at once on the new config, and removes the annotation. The NATS cluster serves no client until its servers are back.
  • Delete the NatsCluster, and apply it again with the change. You lose the JetStream data on every claim that you delete with it.

Rollout in the design describes why the servers cannot route to each other.

Esc
↓ ↑ Navigate Enter Select Esc Close