Story 07 · JetStream

Balance JetStream leaders and copies

Contents

This guide shows you how to keep the leaders and the copies of streams spread evenly over the servers of a NATS cluster. The manifests create a NatsSystemBalancer that balances every account of the NATS cluster demo, and a NatsBalancer that balances the account payments in pools of its own streams.

Before you begin#

You need:

Balance every account of the NATS cluster#

As the platform team, apply a NatsConnection with the creds of the system account user jetstream-controller, and a NatsSystemBalancer on it. A NATS cluster has at most one NatsSystemBalancer.

01-system.yaml yaml ·
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: demo-sys
  namespace: nats-system
spec:
  servers: ["nats://demo.nats-system.svc:4222"]
  credentials:
    secretKeyRef:
      name: jetstream-controller-creds
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsSystemBalancer
metadata:
  name: demo
  namespace: nats-system
spec:
  connectionRef:
    name: demo-sys
  moves:
    placement: true
    leader: true
  interval: 1m
kubectl apply -f https://nats-operator.io/docs/stories/07-balancing/01-system.yaml
  • moves.placement lets the balancer move copies between servers. It is off unless you set it.
  • moves.leader lets the balancer move leaders, in the accounts that have the jetstream-stepdown export.
  • interval is the time between passes. A pass makes at most one move.

A balancer keeps a stream inside its NATS cluster. To move a stream to another NATS cluster, see Move a stream to another NATS cluster. NatsSystemBalancerSpec in the API reference lists every field.

Let the system balancer move the leaders of an account#

The system balancer can make a placement move in any account. It makes a leader move only in an account that exports the preset jetstream-stepdown. Apply the account payments with that export added:

01-account-export.yaml yaml ·
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
  name: payments
  namespace: nats-system
spec:
  operatorRef:
    name: demo
  limits:
    connections: 200
    jetstream:
      memoryStorage: 512Mi
      diskStorage: 100Gi
      streams: 10
      consumers: 100
  exports:
    - preset: jetstream-stepdown
kubectl apply -f https://nats-operator.io/docs/stories/07-balancing/01-account-export.yaml

The manifest repeats the limits that the account already has. If you leave them out, the account loses them. jetstream-stepdown in the NATS permissions reference lists the subjects of the preset.

Check the system balancer#

Read its status:

kubectl -n nats-system get natssystembalancer demo -o yaml

The status in the output is similar to this:

01-status-natssystembalancer.yaml yaml ·
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Balancing
    - type: Holding
      status: "False"
      reason: Settled
  capabilities:
    placement: true
    leader: Partial
    leaderReason: 1 of 2 accounts carry no jetstream-stepdown export; their leaders are not moved
  servers:
    - name: demo-0
      leaders: 1
      replicas: 3
    - name: demo-1
      leaders: 1
      replicas: 3
    - name: demo-2
      leaders: 1
      replicas: 3
  skew:
    leaders: 0
    replicas: 0
  lastMove:
    kind: Leader
    account: ABFAGFB4TX4F4GEPOPBDD4SUQR23BDIBE3NCLZFE3ZF3VP77JJHFHGQF
    stream: PAYMENTS
    from: demo-2
    to: demo-0
    time: "2026-09-26T21:00:10Z"
  pending: []
  • capabilities.leader is Partial, because the account orders has no jetstream-stepdown export. The balancer leaves the leaders of orders where they are.
  • servers counts the leaders and the copies on each server, and skew is how uneven they are. The NATS cluster has three servers, and each of its three streams has three copies, so every server has a copy of every stream and the balancer can move only leaders.
  • lastMove is absent until the balancer has made a move.

Balance the streams of an account in pools#

As the payments team, apply a NatsBalancer on the team’s own connection, and a stream with the label that the pool requests selects:

01-account.yaml yaml ·
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsBalancer
metadata:
  name: payments
  namespace: payments
spec:
  connectionRef:
    name: demo
  pools:
    - name: requests
      selector:
        matchLabels:
          traffic: requests
    - name: responses
      selector:
        matchLabels:
          traffic: responses
  moves:
    leader: true
    placement: false
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
  name: req-07
  namespace: payments
  labels:
    traffic: requests
spec:
  connectionRef:
    name: demo
  name: REQ_07
  subjects: ["req.07.>"]
  replicas: 3
kubectl apply -f https://nats-operator.io/docs/stories/07-balancing/01-account.yaml

The balancer evens the leaders in each pool separately. A pool selects NatsStream, NatsKeyValue and NatsObjectStore resources in the namespace of the balancer by label, and the consumers of a stream belong to the pool of the stream.

  • A stream that no pool selects, or that has no resource, is in the default pool.
  • A stream that several pools select is in the first of them, and the condition Overlapping is True.
  • If you declare no pools, the whole account is one pool.

NatsBalancerSpec in the API reference lists every field.

Check the account balancer#

Read its status:

kubectl -n payments get natsbalancer payments -o yaml

The status in the output is similar to this:

01-status-natsbalancer.yaml yaml ·
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Balancing
    - type: Holding
      status: "False"
      reason: Settled
    - type: Overlapping
      status: "False"
      reason: PoolsDisjoint
  pools:
    - name: requests
      streams: 1
      leaderSkew: 1
    - name: responses
      streams: 0
    - name: (default)
      streams: 1
      leaderSkew: 1

pools counts the streams of each pool, and leaderSkew is how uneven their leaders are. REQ_07 is in the pool requests, and PAYMENTS has no pool label, so it is in the default pool. A pool of one stream has a leaderSkew of 1.

The account balancer yields to the system balancer. While the system balancer has a move pending on a stream of the account, Holding is True with the reason YieldingToSystemBalancer, and the account balancer moves nothing. Neither balancer moves anything while the NATS cluster is not Settled. Balancing in the design describes how the two balancers work together.

Esc
↓ ↑ Navigate Enter Select Esc Close