Story 11 · JetStream
Retire a NATS cluster
Contents
This guide shows you how to retire a NATS cluster of a supercluster without losing its streams.
You deploy a replacement beside it in the same supercluster, move the streams to the replacement with a NatsClusterEvacuation, and then delete the old NatsCluster.
The manifests retire the NATS cluster prod-east and replace it with prod-east-2.
Before you begin#
You need:
- The NATS cluster
prod-eastand its supercluster from Deploy a production supercluster of three NATS clusters, with theNatsConnectionprod-east-sys, which has creds of the system account. - The DNS name
nats.prod-east-2.acme.examplepointing at the gateway Service of the replacement, for example through external-dns.
Add the replacement to the gateway list#
Add prod-east-2 to gateway.remotes of prod-east, and apply it.
Add the same entry to every other member of the supercluster.
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: prod-east
namespace: nats-system
spec:
version: 2.15.0
replicas: 5
resources:
requests:
cpu: "6"
memory: 29Gi
limits:
memory: 29Gi
jetstream:
limits:
maxMemoryStore: 10Gi
volumeClaimTemplate:
spec:
storageClassName: gp3
resources:
requests:
storage: 500Gi
serverTags:
cluster: prod-east
region: us-east-2
auth:
trustRef:
name: acme
systemCredentials:
secretKeyRef:
name: cluster-controller-creds
gateway:
discovery: Explicit
remotes:
- name: prod-east
url: tls://nats.prod-east.acme.example:7222
- name: dev-east
url: tls://nats.dev-east.acme.example:7222
- name: prod-west
url: tls://nats.prod-west.acme.example:7222
- name: prod-east-2
url: tls://nats.prod-east-2.acme.example:7222
tls:
certManager:
issuerRef:
kind: Issuer
name: nats-gateway-ca
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
external-dns.alpha.kubernetes.io/hostname: nats.prod-east.acme.example
advertise: nats.prod-east.acme.example:7222
kubectl apply -f https://nats-operator.io/docs/stories/11-evacuation/01-natscluster-prod-east.yamlDeploy the replacement and start the evacuation#
Apply the NatsCluster prod-east-2 and the NatsClusterEvacuation.
prod-east-2 is the same as prod-east, except for its name, its gateway address and the server tag cluster:prod-east-2, which no other server has.
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: prod-east-2
namespace: nats-system
spec:
version: 2.15.0
replicas: 5
resources:
requests:
cpu: "6"
memory: 29Gi
limits:
memory: 29Gi
jetstream:
limits:
maxMemoryStore: 10Gi
volumeClaimTemplate:
spec:
storageClassName: gp3
resources:
requests:
storage: 500Gi
serverTags:
cluster: prod-east-2
region: us-east-2
auth:
trustRef:
name: acme
systemCredentials:
secretKeyRef:
name: cluster-controller-creds
gateway:
discovery: Explicit
remotes:
- name: prod-east
url: tls://nats.prod-east.acme.example:7222
- name: dev-east
url: tls://nats.dev-east.acme.example:7222
- name: prod-west
url: tls://nats.prod-west.acme.example:7222
- name: prod-east-2
url: tls://nats.prod-east-2.acme.example:7222
tls:
certManager:
issuerRef:
kind: Issuer
name: nats-gateway-ca
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
external-dns.alpha.kubernetes.io/hostname: nats.prod-east-2.acme.example
advertise: nats.prod-east-2.acme.example:7222
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsClusterEvacuation
metadata:
name: retire-prod-east
namespace: nats-system
spec:
connectionRef:
name: prod-east-sys
from:
cluster: prod-east
to:
serverTags: ["cluster:prod-east-2"]
kubectl apply -f https://nats-operator.io/docs/stories/11-evacuation/01-evacuation.yamlconnectionRefis a connection with creds of the system account.from.clusteris the NATS cluster to empty. An evacuation always empties a whole NATS cluster.to.serverTagsmust match only servers of the target NATS cluster. If a server offrom.clusterhas the same tags, the evacuation is refused before it moves anything.
The evacuation moves every stream, key-value bucket and object store of every account, with their consumers.
It does not move one whose resource sets placement.cluster.
Balancers skip a stream that the evacuation is moving.
NatsClusterEvacuationSpec in the API reference describes the fields.
Check the evacuation#
Read its status:
kubectl -n nats-system get natsclusterevacuation retire-prod-east -o yamlWhen the evacuation has moved everything that it can move, the status in the output is similar to this:
status:
observedGeneration: 1
conditions:
- type: Ready
status: "False"
reason: PinnedObjects
message: 2 resources pin placement.cluster prod-east
- type: Progressing
status: "False"
reason: NothingMovable
moved: 41
inFlight: 0
pinned:
- kind: NatsStream
namespace: orders
name: orders
- kind: NatsKeyValue
namespace: payments
name: sessions
stalePlacement:
- account: ADTWXTPEOI65FGIGO36XGZBDFOYDDPV6XZPRQWDIRXIYCAJD6TBP76JN
name: audit
moved counts the streams that the evacuation moved, and inFlight the moves that are still running.
pinned lists the resources whose own spec sets placement.cluster to prod-east.
The evacuation is not Ready while any of them remains.
Ask the owner of each to move it, as Move a stream to another NATS cluster shows.
stalePlacement lists the moved streams that no resource owns and whose config still has prod-east as its placement.
While prod-east exists, an update that changes the placement of such a stream moves it back there.
Evacuation in the design describes how the streams are moved.
Delete the retired NATS cluster#
When pinned is empty, delete the NatsCluster prod-east:
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: prod-east
namespace: nats-system
kubectl delete -f https://nats-operator.io/docs/stories/11-evacuation/02-delete-natscluster-prod-east.yamlThe deletion waits while the NATS cluster still has JetStream data.
If the deletion does not finish, read the status of prod-east:
kubectl -n nats-system get natscluster prod-east -o yamlThe status in the output is similar to this:
status:
conditions:
- type: Deleting
status: "True"
reason: JetStreamDataRemains
message: 2 stream groups still placed in prod-east (orders/ORDERS, payments/KV_sessions)
Deleting is True with the reason JetStreamDataRemains, and its message lists the stream groups that are still in prod-east.
Move them, and the deletion finishes.
To delete the NatsCluster together with the data that it still has, set the annotation cluster.nats-operator.io/force-delete on it.
Deletion guard in the design describes what the deletion waits for.