# Retire a NATS cluster


This guide shows you how to retire a NATS cluster of a supercluster without losing its streams.
You deploy a replacement beside it in the same supercluster, move the streams to the replacement with a `NatsClusterEvacuation`, and then delete the old `NatsCluster`.
The manifests retire the NATS cluster `prod-east` and replace it with `prod-east-2`.

## Before you begin

You need:

- The NATS cluster `prod-east` and its supercluster from [Deploy a production supercluster of three NATS clusters](https://nats-operator.io/docs/stories/09-acceptance/index.md), with the `NatsConnection` `prod-east-sys`, which has creds of the system account.
- The DNS name `nats.prod-east-2.acme.example` pointing at the gateway Service of the replacement, for example through external-dns.

## Add the replacement to the gateway list

Add `prod-east-2` to `gateway.remotes` of `prod-east`, and apply it.
Add the same entry to every other member of the supercluster.

`01-natscluster-prod-east.yaml`

```yaml
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 5
  resources:
    requests:
      cpu: "6"
      memory: 29Gi
    limits:
      memory: 29Gi
  jetstream:
    limits:
      maxMemoryStore: 10Gi
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 500Gi
  serverTags:
    cluster: prod-east
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
      - name: prod-east-2
        url: tls://nats.prod-east-2.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-east.acme.example
    advertise: nats.prod-east.acme.example:7222
```

```sh
kubectl apply -f https://nats-operator.io/docs/stories/11-evacuation/01-natscluster-prod-east.yaml
```


## Deploy the replacement and start the evacuation

Apply the `NatsCluster` `prod-east-2` and the `NatsClusterEvacuation`.
`prod-east-2` is the same as `prod-east`, except for its name, its gateway address and the server tag `cluster:prod-east-2`, which no other server has.

`01-evacuation.yaml`

```yaml
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east-2
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 5
  resources:
    requests:
      cpu: "6"
      memory: 29Gi
    limits:
      memory: 29Gi
  jetstream:
    limits:
      maxMemoryStore: 10Gi
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 500Gi
  serverTags:
    cluster: prod-east-2
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
      - name: prod-east-2
        url: tls://nats.prod-east-2.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-east-2.acme.example
    advertise: nats.prod-east-2.acme.example:7222
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsClusterEvacuation
metadata:
  name: retire-prod-east
  namespace: nats-system
spec:
  connectionRef:
    name: prod-east-sys
  from:
    cluster: prod-east
  to:
    serverTags: ["cluster:prod-east-2"]
```

```sh
kubectl apply -f https://nats-operator.io/docs/stories/11-evacuation/01-evacuation.yaml
```


- `connectionRef` is a connection with creds of the system account.
- `from.cluster` is the NATS cluster to empty. An evacuation always empties a whole NATS cluster.
- `to.serverTags` must match only servers of the target NATS cluster.
  If a server of `from.cluster` has the same tags, the evacuation is refused before it moves anything.

The evacuation moves every stream, key-value bucket and object store of every account, with their consumers.
It does not move one whose resource sets `placement.cluster`.
Balancers skip a stream that the evacuation is moving.
[NatsClusterEvacuationSpec](https://nats-operator.io/docs/reference/api/index.md#NatsClusterEvacuationSpec) in the API reference describes the fields.

## Check the evacuation

Read its status:

```sh
kubectl -n nats-system get natsclusterevacuation retire-prod-east -o yaml
```

When the evacuation has moved everything that it can move, the `status` in the output is similar to this:

`01-status-natsclusterevacuation.yaml`

```yaml
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "False"
      reason: PinnedObjects
      message: 2 resources pin placement.cluster prod-east
    - type: Progressing
      status: "False"
      reason: NothingMovable
  moved: 41
  inFlight: 0
  pinned:
    - kind: NatsStream
      namespace: orders
      name: orders
    - kind: NatsKeyValue
      namespace: payments
      name: sessions
  stalePlacement:
    - account: ADTWXTPEOI65FGIGO36XGZBDFOYDDPV6XZPRQWDIRXIYCAJD6TBP76JN
      name: audit
```


`moved` counts the streams that the evacuation moved, and `inFlight` the moves that are still running.

`pinned` lists the resources whose own spec sets `placement.cluster` to `prod-east`.
The evacuation is not Ready while any of them remains.
Ask the owner of each to move it, as [Move a stream to another NATS cluster](https://nats-operator.io/docs/stories/08-stream-transfer/index.md) shows.

`stalePlacement` lists the moved streams that no resource owns and whose config still has `prod-east` as its placement.
While `prod-east` exists, an update that changes the placement of such a stream moves it back there.

[Evacuation](https://nats-operator.io/docs/design/v1/index.md#66-evacuation) in the design describes how the streams are moved.

## Delete the retired NATS cluster

When `pinned` is empty, delete the `NatsCluster` `prod-east`:

`02-delete-natscluster-prod-east.yaml`

```yaml
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east
  namespace: nats-system
```

```sh
kubectl delete -f https://nats-operator.io/docs/stories/11-evacuation/02-delete-natscluster-prod-east.yaml
```


The deletion waits while the NATS cluster still has JetStream data.
If the deletion does not finish, read the status of `prod-east`:

```sh
kubectl -n nats-system get natscluster prod-east -o yaml
```

The `status` in the output is similar to this:

`02-status-natscluster-prod-east.yaml`

```yaml
status:
  conditions:
    - type: Deleting
      status: "True"
      reason: JetStreamDataRemains
      message: 2 stream groups still placed in prod-east (orders/ORDERS, payments/KV_sessions)
```


`Deleting` is True with the reason `JetStreamDataRemains`, and its message lists the stream groups that are still in `prod-east`.
Move them, and the deletion finishes.

To delete the `NatsCluster` together with the data that it still has, set the annotation `cluster.nats-operator.io/force-delete` on it.
[Deletion guard](https://nats-operator.io/docs/design/v1/index.md#46-deletion-guard) in the design describes what the deletion waits for.

