Story 11 · JetStream

Retire a NATS cluster

Contents

This guide shows you how to retire a NATS cluster of a supercluster without losing its streams. You deploy a replacement beside it in the same supercluster, move the streams to the replacement with a NatsClusterEvacuation, and then delete the old NatsCluster. The manifests retire the NATS cluster prod-east and replace it with prod-east-2.

Before you begin#

You need:

  • The NATS cluster prod-east and its supercluster from Deploy a production supercluster of three NATS clusters, with the NatsConnection prod-east-sys, which has creds of the system account.
  • The DNS name nats.prod-east-2.acme.example pointing at the gateway Service of the replacement, for example through external-dns.

Add the replacement to the gateway list#

Add prod-east-2 to gateway.remotes of prod-east, and apply it. Add the same entry to every other member of the supercluster.

01-natscluster-prod-east.yaml yaml ·
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 5
  resources:
    requests:
      cpu: "6"
      memory: 29Gi
    limits:
      memory: 29Gi
  jetstream:
    limits:
      maxMemoryStore: 10Gi
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 500Gi
  serverTags:
    cluster: prod-east
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
      - name: prod-east-2
        url: tls://nats.prod-east-2.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-east.acme.example
    advertise: nats.prod-east.acme.example:7222
kubectl apply -f https://nats-operator.io/docs/stories/11-evacuation/01-natscluster-prod-east.yaml

Deploy the replacement and start the evacuation#

Apply the NatsCluster prod-east-2 and the NatsClusterEvacuation. prod-east-2 is the same as prod-east, except for its name, its gateway address and the server tag cluster:prod-east-2, which no other server has.

01-evacuation.yaml yaml ·
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east-2
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 5
  resources:
    requests:
      cpu: "6"
      memory: 29Gi
    limits:
      memory: 29Gi
  jetstream:
    limits:
      maxMemoryStore: 10Gi
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 500Gi
  serverTags:
    cluster: prod-east-2
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
      - name: prod-east-2
        url: tls://nats.prod-east-2.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-east-2.acme.example
    advertise: nats.prod-east-2.acme.example:7222
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsClusterEvacuation
metadata:
  name: retire-prod-east
  namespace: nats-system
spec:
  connectionRef:
    name: prod-east-sys
  from:
    cluster: prod-east
  to:
    serverTags: ["cluster:prod-east-2"]
kubectl apply -f https://nats-operator.io/docs/stories/11-evacuation/01-evacuation.yaml
  • connectionRef is a connection with creds of the system account.
  • from.cluster is the NATS cluster to empty. An evacuation always empties a whole NATS cluster.
  • to.serverTags must match only servers of the target NATS cluster. If a server of from.cluster has the same tags, the evacuation is refused before it moves anything.

The evacuation moves every stream, key-value bucket and object store of every account, with their consumers. It does not move one whose resource sets placement.cluster. Balancers skip a stream that the evacuation is moving. NatsClusterEvacuationSpec in the API reference describes the fields.

Check the evacuation#

Read its status:

kubectl -n nats-system get natsclusterevacuation retire-prod-east -o yaml

When the evacuation has moved everything that it can move, the status in the output is similar to this:

01-status-natsclusterevacuation.yaml yaml ·
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "False"
      reason: PinnedObjects
      message: 2 resources pin placement.cluster prod-east
    - type: Progressing
      status: "False"
      reason: NothingMovable
  moved: 41
  inFlight: 0
  pinned:
    - kind: NatsStream
      namespace: orders
      name: orders
    - kind: NatsKeyValue
      namespace: payments
      name: sessions
  stalePlacement:
    - account: ADTWXTPEOI65FGIGO36XGZBDFOYDDPV6XZPRQWDIRXIYCAJD6TBP76JN
      name: audit

moved counts the streams that the evacuation moved, and inFlight the moves that are still running.

pinned lists the resources whose own spec sets placement.cluster to prod-east. The evacuation is not Ready while any of them remains. Ask the owner of each to move it, as Move a stream to another NATS cluster shows.

stalePlacement lists the moved streams that no resource owns and whose config still has prod-east as its placement. While prod-east exists, an update that changes the placement of such a stream moves it back there.

Evacuation in the design describes how the streams are moved.

Delete the retired NATS cluster#

When pinned is empty, delete the NatsCluster prod-east:

02-delete-natscluster-prod-east.yaml yaml ·
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east
  namespace: nats-system
kubectl delete -f https://nats-operator.io/docs/stories/11-evacuation/02-delete-natscluster-prod-east.yaml

The deletion waits while the NATS cluster still has JetStream data. If the deletion does not finish, read the status of prod-east:

kubectl -n nats-system get natscluster prod-east -o yaml

The status in the output is similar to this:

02-status-natscluster-prod-east.yaml yaml ·
status:
  conditions:
    - type: Deleting
      status: "True"
      reason: JetStreamDataRemains
      message: 2 stream groups still placed in prod-east (orders/ORDERS, payments/KV_sessions)

Deleting is True with the reason JetStreamDataRemains, and its message lists the stream groups that are still in prod-east. Move them, and the deletion finishes.

To delete the NatsCluster together with the data that it still has, set the annotation cluster.nats-operator.io/force-delete on it. Deletion guard in the design describes what the deletion waits for.

Esc
↓ ↑ Navigate Enter Select Esc Close