Story 07 · JetStream
Balance JetStream leaders and copies
Contents
This guide shows you how to keep the leaders and the copies of streams spread evenly over the servers of a NATS cluster.
The manifests create a NatsSystemBalancer that balances every account of the NATS cluster demo, and a NatsBalancer that balances the account payments in pools of its own streams.
Before you begin#
You need:
- The NATS cluster
demoand its NATS operator from Put a NATS cluster under a NATS operator, with the Secretjetstream-controller-creds. - The account
payments, and theNatsConnectiondemoin the namespacepayments, from Let a team declare its own users and streams.
Balance every account of the NATS cluster#
As the platform team, apply a NatsConnection with the creds of the system account user jetstream-controller, and a NatsSystemBalancer on it.
A NATS cluster has at most one NatsSystemBalancer.
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
name: demo-sys
namespace: nats-system
spec:
servers: ["nats://demo.nats-system.svc:4222"]
credentials:
secretKeyRef:
name: jetstream-controller-creds
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsSystemBalancer
metadata:
name: demo
namespace: nats-system
spec:
connectionRef:
name: demo-sys
moves:
placement: true
leader: true
interval: 1m
kubectl apply -f https://nats-operator.io/docs/stories/07-balancing/01-system.yamlmoves.placementlets the balancer move copies between servers. It is off unless you set it.moves.leaderlets the balancer move leaders, in the accounts that have thejetstream-stepdownexport.intervalis the time between passes. A pass makes at most one move.
A balancer keeps a stream inside its NATS cluster. To move a stream to another NATS cluster, see Move a stream to another NATS cluster. NatsSystemBalancerSpec in the API reference lists every field.
Let the system balancer move the leaders of an account#
The system balancer can make a placement move in any account.
It makes a leader move only in an account that exports the preset jetstream-stepdown.
Apply the account payments with that export added:
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
name: payments
namespace: nats-system
spec:
operatorRef:
name: demo
limits:
connections: 200
jetstream:
memoryStorage: 512Mi
diskStorage: 100Gi
streams: 10
consumers: 100
exports:
- preset: jetstream-stepdown
kubectl apply -f https://nats-operator.io/docs/stories/07-balancing/01-account-export.yamlThe manifest repeats the limits that the account already has. If you leave them out, the account loses them. jetstream-stepdown in the NATS permissions reference lists the subjects of the preset.
Check the system balancer#
Read its status:
kubectl -n nats-system get natssystembalancer demo -o yamlThe status in the output is similar to this:
status:
observedGeneration: 1
conditions:
- type: Ready
status: "True"
reason: Balancing
- type: Holding
status: "False"
reason: Settled
capabilities:
placement: true
leader: Partial
leaderReason: 1 of 2 accounts carry no jetstream-stepdown export; their leaders are not moved
servers:
- name: demo-0
leaders: 1
replicas: 3
- name: demo-1
leaders: 1
replicas: 3
- name: demo-2
leaders: 1
replicas: 3
skew:
leaders: 0
replicas: 0
lastMove:
kind: Leader
account: ABFAGFB4TX4F4GEPOPBDD4SUQR23BDIBE3NCLZFE3ZF3VP77JJHFHGQF
stream: PAYMENTS
from: demo-2
to: demo-0
time: "2026-09-26T21:00:10Z"
pending: []
capabilities.leaderis Partial, because the accountordershas nojetstream-stepdownexport. The balancer leaves the leaders oforderswhere they are.serverscounts the leaders and the copies on each server, andskewis how uneven they are. The NATS cluster has three servers, and each of its three streams has three copies, so every server has a copy of every stream and the balancer can move only leaders.lastMoveis absent until the balancer has made a move.
Balance the streams of an account in pools#
As the payments team, apply a NatsBalancer on the team’s own connection, and a stream with the label that the pool requests selects:
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsBalancer
metadata:
name: payments
namespace: payments
spec:
connectionRef:
name: demo
pools:
- name: requests
selector:
matchLabels:
traffic: requests
- name: responses
selector:
matchLabels:
traffic: responses
moves:
leader: true
placement: false
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
name: req-07
namespace: payments
labels:
traffic: requests
spec:
connectionRef:
name: demo
name: REQ_07
subjects: ["req.07.>"]
replicas: 3
kubectl apply -f https://nats-operator.io/docs/stories/07-balancing/01-account.yamlThe balancer evens the leaders in each pool separately.
A pool selects NatsStream, NatsKeyValue and NatsObjectStore resources in the namespace of the balancer by label, and the consumers of a stream belong to the pool of the stream.
- A stream that no pool selects, or that has no resource, is in the default pool.
- A stream that several pools select is in the first of them, and the condition
Overlappingis True. - If you declare no pools, the whole account is one pool.
NatsBalancerSpec in the API reference lists every field.
Check the account balancer#
Read its status:
kubectl -n payments get natsbalancer payments -o yamlThe status in the output is similar to this:
status:
observedGeneration: 1
conditions:
- type: Ready
status: "True"
reason: Balancing
- type: Holding
status: "False"
reason: Settled
- type: Overlapping
status: "False"
reason: PoolsDisjoint
pools:
- name: requests
streams: 1
leaderSkew: 1
- name: responses
streams: 0
- name: (default)
streams: 1
leaderSkew: 1
pools counts the streams of each pool, and leaderSkew is how uneven their leaders are.
REQ_07 is in the pool requests, and PAYMENTS has no pool label, so it is in the default pool.
A pool of one stream has a leaderSkew of 1.
The account balancer yields to the system balancer.
While the system balancer has a move pending on a stream of the account, Holding is True with the reason YieldingToSystemBalancer, and the account balancer moves nothing.
Neither balancer moves anything while the NATS cluster is not Settled.
Balancing in the design describes how the two balancers work together.