# Add a NATS cluster to a supercluster that you did not deploy


This guide shows you how to add a NATS cluster to a supercluster that these controllers do not run.
The servers of that supercluster were deployed by other means, someone else has the keys of its NATS operator, and nothing pushes account JWTs to its servers.
The manifests add the NATS cluster `west` to a supercluster whose one member is `central`, and create a stream on `west` in the account `orders` of the supercluster.

To build a supercluster of NATS clusters that you deploy yourself, see [Build a supercluster across Kubernetes clusters](https://nats-operator.io/docs/stories/06-supercluster/index.md).

## Before you begin

You need:

- A Kubernetes cluster with the cluster controller and the JetStream controller installed, and the namespaces `nats-system` and `orders`.
  [Install](https://nats-operator.io/docs/install/index.md) shows how to install the controllers.
- From whoever has the keys of the NATS operator: the NATS operator JWT, the system account JWT, and the public key and the JWT of each account that you use on `west`.
- The creds of a user of the system account, for the cluster controller, in the Secret `west-cluster-controller-creds` in `nats-system`.
- The creds of a user of the account `orders` in the Secret `orders-creds` in `orders`.
- The DNS name `nats-west.example.net` pointing at the gateway Service of `west`, for example through external-dns.
- The StorageClass `gp3`, or your own in its place in the manifest.

## Add the new member to the existing members

A `NatsCluster` does not change a NATS cluster that it did not deploy.
Ask whoever runs `central` to add `west` to its gateways before you deploy `west`.
In this guide, `central` sets `reject_unknown`, so it refuses a gateway that its own list does not have.

This is the part of the config of `central` that matters to `west`:

```text
gateway {
  name: central
  port: 7222
  reject_unknown: true
  gateways: [
    {name: west, urls: ["nats://nats-west.example.net:7222"]}
  ]
}
operator: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
system_account: ASYS...
resolver: MEMORY
resolver_preload: {
  ASYS...: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  AORDERS...: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  APAYMENTS...: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
}
```

`central` has its accounts as preloads under a `MEMORY` resolver.
Its gateway has no TLS and no `authorization`.

The servers of `west` reach the gateways of `central` through the Service behind `nats-central.example.net`.
If a server of `central` is Ready only once its JetStream reaches a meta leader, that Service must also publish servers that are not Ready, with `publishNotReadyAddresses: true`.
The meta group spans both NATS clusters, so `central` alone may not hold a majority of it, and a server of `central` that `west` cannot reach may never become Ready.

## Copy the trust roots

Copy the NATS operator JWT and the system account JWT into a `NatsOperatorTrust`, and apply it:

`01-natsoperatortrust.yaml`

```yaml
apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
  name: central
  namespace: nats-system
spec:
  operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
```

```sh
kubectl apply -f https://nats-operator.io/docs/stories/13-join-supercluster/01-natsoperatortrust.yaml
```


## Copy the accounts

Nothing pushes account JWTs to this supercluster, so `west` must preload them, as `central` does.
For each account, copy its public key and the JWT that `central` preloads into a `NatsAccountTrust`, and apply them:

`01-natsaccounttrusts.yaml`

```yaml
apiVersion: nats-operator.io/v1beta1
kind: NatsAccountTrust
metadata:
  name: orders
  namespace: nats-system
spec:
  publicKey: AORDERS...
  jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
---
apiVersion: nats-operator.io/v1beta1
kind: NatsAccountTrust
metadata:
  name: payments
  namespace: nats-system
spec:
  publicKey: APAYMENTS...
  jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
```

```sh
kubectl apply -f https://nats-operator.io/docs/stories/13-join-supercluster/01-natsaccounttrusts.yaml
```


A copy is good until its JWT expires.
A JWT that the auth controller signs expires after the `jwtTTL` of its `NatsAccount`, which is 48h by default, or never if `jwtTTL` is 0.
A JWT signed elsewhere expires when its signer decided.

## Allow a gateway without TLS

Do this step only if the existing members run their gateways without TLS, as `central` does.
The cluster controller refuses a `NatsCluster` whose gateway has no `tls`, unless it runs with `--allow-gateway-without-tls`.
To pass the flag, upgrade the release with this chart value:

```yaml
cluster:
  allowGatewayWithoutTLS: true
```

## Deploy the new member

The servers of `west` must start with no JetStream state.
They then join the JetStream meta group of the supercluster.
Servers that bring a meta group of their own do not merge with it, and either side can lose its streams.
Use new claims: if a `NatsCluster` named `west` ran in `nats-system` before, without these gateways or with a `jetstream.domain`, delete its data volume claims first:

```sh
kubectl -n nats-system delete pvc data-west-0-0 data-west-1-0 data-west-2-0
```

The cluster controller creates no server whose claim is older than the `NatsCluster`, and reads `Progressing` False with the reason `OwnMetaGroup` instead.
It also refuses to add gateway remotes to servers that already lead a meta group of their own.
[Trust, resolver, gateways, leaf nodes](https://nats-operator.io/docs/design/v1/index.md#43-trust-resolver-gateways-leaf-nodes) in the design describes what it checks.

Apply the `NatsCluster` `west`:

`01-west.yaml`

```yaml
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: west
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "1"
      memory: 4Gi
    limits:
      memory: 4Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 100Gi
  auth:
    trustRef:
      name: central
    systemCredentials:
      secretKeyRef:
        name: west-cluster-controller-creds
    accountTrustRefs:
      - name: orders
      - name: payments
  gateway:
    discovery: Explicit
    remotes:
      - name: central
        url: nats://nats-central.example.net:7222
      - name: west
        url: nats://nats-west.example.net:7222
    service:
      type: LoadBalancer
      annotations:
        external-dns.alpha.kubernetes.io/hostname: nats-west.example.net
    advertise: nats-west.example.net:7222
```

```sh
kubectl apply -f https://nats-operator.io/docs/stories/13-join-supercluster/01-west.yaml
```


- The name of the `NatsCluster` is its gateway name. It must be the name that the existing members have in their lists.
- `auth.systemCredentials` is the Secret with the creds for the cluster controller.
- `auth.accountTrustRefs` lists the accounts that every server preloads beside the system account.
  A change to the list, or to the JWT in a `NatsAccountTrust` on it, restarts the servers.
- `gateway.remotes` lists every member of the existing supercluster, and `west` itself.
- `gateway` has no `tls`, because the gateways of the existing members have none.
- `gateway.service` renders the Service `west-gateway`, which publishes the servers of `west` before they are Ready.
  A server of `west` is Ready once its JetStream reaches the meta leader, and the gateways of `central` reach it through this Service.

The servers run a `Full` resolver on the JetStream volume.
If you remove an account from `auth.accountTrustRefs`, the account stays in the directory of the resolver, and the servers still serve it after the restart.
[Trust, resolver, gateways, leaf nodes](https://nats-operator.io/docs/design/v1/index.md#43-trust-resolver-gateways-leaf-nodes) in the design describes what the resolver stores.

## Check the new member

Read the status of `west`:

```sh
kubectl -n nats-system get natscluster west -o yaml
```

The `status` in the output is similar to this:

`01-status-natscluster.yaml`

```yaml
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: AllServersReady
    - type: Settled
      status: "True"
      reason: AllGroupsCurrent
    - type: GatewaysConnected
      status: "True"
      reason: AllMembersReachable
      message: 1 of 1 remote members connected
  gateways:
    - name: central
      connected: true
      inbound: 3
      outbound: 3
  endpoints:
    client: nats://west.nats-system.svc:4222
    gateway: nats-west.example.net:7222
```


`GatewaysConnected` is True, and `gateways` has an entry for `central`.
A supercluster has one JetStream meta group, so `Settled` is True once the servers of `west` are among the peers of the meta leader.

To check that `west` joined the meta group of `central`, read `meta_cluster` from `/jsz` on a server of each, on the monitoring port 8222:

```sh
kubectl -n nats-system port-forward pod/west-0-0 8222:8222
curl -s localhost:8222/jsz | jq .meta_cluster
```

In one meta group, every server reports the same `leader` and the same `cluster_size`, which counts the servers of `west` and of `central`.
If the servers of `west` report a leader among themselves and a `cluster_size` that counts only them, `west` runs a meta group of its own beside that of `central`.
Stop `west` at once, as [Recover from a meta group of its own](#recover-from-a-meta-group-of-its-own) shows: while both groups run, either can delete the streams of the other.

## Create a stream on the new member

Apply a `NatsConnection` with the creds of the user of `orders`, and a stream on it:

`02-natsconnection.yaml`

```yaml
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: west
  namespace: orders
spec:
  servers: ["nats://west.nats-system.svc:4222"]
  credentials:
    secretKeyRef:
      name: orders-creds
```

```sh
kubectl apply -f https://nats-operator.io/docs/stories/13-join-supercluster/02-natsconnection.yaml
```


`02-natsstream.yaml`

```yaml
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
  name: orders
  namespace: orders
spec:
  connectionRef:
    name: west
  name: ORDERS
  subjects: ["orders.>"]
  storage: File
  replicas: 3
  placement:
    cluster: west
  maxAge: 72h
```

```sh
kubectl apply -f https://nats-operator.io/docs/stories/13-join-supercluster/02-natsstream.yaml
```


`west` authenticates the user against the JWT of `orders` that it preloads.
`placement.cluster` puts the stream on the servers of `west` and not on those of `central`, which are in the same meta group.

Read the status of the stream:

```sh
kubectl -n orders get natsstream orders -o yaml
```

The `status` in the output is similar to this:

`02-status-natsstream.yaml`

```yaml
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Synced
    - type: Synced
      status: "True"
      reason: MatchesSpec
  ownership:
    origin: Created
  server:
    leader: west-1
    replicas:
      - name: west-0
        current: true
      - name: west-2
        current: true
```


`server.leader` and `server.replicas` are servers of `west`.

## Recover from a meta group of its own

1. Stop `west`.
   Set the annotation `cluster.nats-operator.io/force-delete` on it, since the deletion otherwise waits for its streams, and delete it:

   ```sh
   kubectl -n nats-system annotate natscluster west cluster.nats-operator.io/force-delete=true
   kubectl -n nats-system delete natscluster west
   ```

2. On a server of `central`, check that its streams are still listed.
   A stream that is gone comes back only from a backup.
3. Delete the data volume claims of `west`:

   ```sh
   kubectl -n nats-system delete pvc data-west-0-0 data-west-1-0 data-west-2-0
   ```

4. Apply the `NatsCluster` `west` again.