The platform team retires the NATS cluster prod-east: it rolls out prod-east-2 beside it in the same supercluster, moves every movable stream across, and only then deletes prod-east.

The evacuation #

The NATS cluster to retire:

01-natscluster-prod-east.yaml

# The NATS cluster being retired: story 9's prod-east, whose gateway list, like
# every member's, now names prod-east-2 as well.
apiVersion: cluster.nats.mikluko.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 5
  resources:
    requests:
      cpu: "6"
      memory: 29Gi
    limits:
      memory: 29Gi
  jetstream:
    # Overrides the memory store derived from limits.memory.
    limits:
      maxMemoryStore: 10Gi
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 500Gi
  serverTags:
    cluster: prod-east
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
      - name: prod-east-2
        url: tls://nats.prod-east-2.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-east.acme.example
    advertise: nats.prod-east.acme.example:7222

kubectl apply -f https://nats-operator.io/docs/stories/11-evacuation/01-natscluster-prod-east.yaml

Its replacement, with a tag of its own, and one system-level evacuation of the whole of prod-east.

01-evacuation.yaml

# The replacement NATS cluster, prod-east-2, a member beside prod-east with a
# placement tag no other server carries; otherwise as prod-east.
apiVersion: cluster.nats.mikluko.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east-2
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 5
  resources:
    requests:
      cpu: "6"
      memory: 29Gi
    limits:
      memory: 29Gi
  jetstream:
    # Overrides the memory store derived from limits.memory.
    limits:
      maxMemoryStore: 10Gi
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 500Gi
  serverTags:
    cluster: prod-east-2
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
      - name: prod-east-2
        url: tls://nats.prod-east-2.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-east-2.acme.example
    advertise: nats.prod-east-2.acme.example:7222
---
# Empties prod-east of every stream, key-value bucket and object store, in every
# account, with their consumers, save those whose resource sets
# placement.cluster. System-level and whole NATS cluster only. Each stream is
# moved by $JS.API.ACCOUNT.STREAM.MOVE with the target tags.
apiVersion: jetstream.nats.mikluko.io/v1beta1
kind: NatsClusterEvacuation
metadata:
  name: retire-prod-east
  namespace: nats-system
spec:
  connectionRef:
    name: prod-east-sys   # system credentials
  from:
    cluster: prod-east
  # Must match servers of the target NATS cluster only; refused before any move
  # if a server of `from` carries them too.
  to:
    serverTags: ["cluster:prod-east-2"]

kubectl apply -f https://nats-operator.io/docs/stories/11-evacuation/01-evacuation.yaml

Resources whose own spec pins prod-east are left alone and listed; the evacuation is not Ready until their owners move them. A moved stream that no resource owns keeps a config naming prod-east, and is listed under stalePlacement: while prod-east exists, an update that changes that stream’s placement moves it back there.

01-status-natsclusterevacuation.yaml

# Balancers skip streams an evacuation is moving.
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "False"
      reason: PinnedObjects
      message: 2 resources pin placement.cluster prod-east
    - type: Progressing
      status: "False"
      reason: NothingMovable
  moved: 41
  inFlight: 0
  pinned:
    - kind: NatsStream
      namespace: orders
      name: orders
    - kind: NatsKeyValue
      namespace: payments
      name: sessions
  stalePlacement:
    - account: ADTWXTPEOI65FGIGO36XGZBDFOYDDPV6XZPRQWDIRXIYCAJD6TBP76JN
      name: audit

Deleting prod-east #

Deletion waits while its NATS cluster still holds JetStream data.

02-delete-natscluster-prod-east.yaml

# kubectl delete natscluster prod-east
apiVersion: cluster.nats.mikluko.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east
  namespace: nats-system

kubectl delete -f https://nats-operator.io/docs/stories/11-evacuation/02-delete-natscluster-prod-east.yaml

02-status-natscluster-prod-east.yaml

# prod-east once deleted: deletion waits while its NATS cluster still holds
# JetStream data. The annotation cluster.nats.mikluko.io/force-delete
# overrides it.
status:
  conditions:
    - type: Deleting
      status: "True"
      reason: JetStreamDataRemains
      message: 2 stream groups still placed in prod-east (orders/ORDERS, payments/KV_sessions)