# Build a supercluster across Kubernetes clusters


This guide shows you how to join NATS clusters in two Kubernetes clusters into one supercluster with gateways.
The manifests deploy the NATS cluster `east` in the Kubernetes cluster `east` and the NATS cluster `west` in the Kubernetes cluster `west`.
`east` is the home cluster: the auth controller runs there, and every account is declared there.

## Before you begin

You need:

- Two Kubernetes clusters, with kubeconfig contexts named `east` and `west`.
  Each manifest on this page is followed by the command that applies it with `--context`.
- In both, the cluster controller and the JetStream controller, cert-manager, the namespace `nats-system`, and LoadBalancer Services that the other Kubernetes cluster can reach.
  [Install](https://nats-operator.io/docs/install/index.md) shows how to install the controllers.
- In `east`, the auth controller, a `NatsOperator` with its system account `sys`, and the cluster controller's user with its creds in the Secret `cluster-controller-creds`.
  [Put a NATS cluster under a NATS operator](https://nats-operator.io/docs/stories/02-auth-plane/index.md) declares them.
- The DNS names `nats-east.example.net` and `nats-west.example.net` pointing at the gateway Services, for example through external-dns.
- The StorageClass `gp3` in both, or your own in its place in the manifests.

## Copy the trust roots to both Kubernetes clusters

Copy the NATS operator JWT and the system account JWT from the status of the `NatsOperator` in `east` into a `NatsOperatorTrust`, and apply the same one in both Kubernetes clusters.
Keep the two copies alike with GitOps, and copy the JWTs again whenever you rotate a key of the NATS operator.

`01-natsoperatortrust.yaml`

```yaml
apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
  name: acme
  namespace: nats-system
spec:
  operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
```

```sh
kubectl --context east apply -f https://nats-operator.io/docs/stories/06-supercluster/01-natsoperatortrust.yaml
kubectl --context west apply -f https://nats-operator.io/docs/stories/06-supercluster/01-natsoperatortrust.yaml
```


## Create the gateway CA

Gateways accept each other by certificate alone.
Give them a private CA: replicate its key pair into the Secret `nats-gateway-ca` in `nats-system` of both Kubernetes clusters, and create an Issuer for it in both:

```yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: nats-gateway-ca
  namespace: nats-system
spec:
  ca:
    secretName: nats-gateway-ca
```

Do not use a public issuer or an ACME issuer.
[Trust, resolver, gateways, leaf nodes](https://nats-operator.io/docs/design/v1/index.md#43-trust-resolver-gateways-leaf-nodes) in the design explains why neither works.

Do not use a ClusterIssuer either.
A ClusterIssuer gives a gateway certificate, and with it a place in the supercluster, to anyone who can write a `NatsCluster` in any namespace that the cluster controller watches.

## Deploy the NATS cluster in the home cluster

Apply the `NatsCluster` `east`.
Its name is its gateway name.
`gateway.remotes` lists every member of the supercluster, itself included.
Give every member the same list.

`01-east.yaml`

```yaml
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: east
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "2"
      memory: 8Gi
    limits:
      memory: 8Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 200Gi
  serverTags:
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: east
        url: tls://nats-east.example.net:7222
      - name: west
        url: tls://nats-west.example.net:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats-east.example.net
    advertise: nats-east.example.net:7222
```

```sh
kubectl --context east apply -f https://nats-operator.io/docs/stories/06-supercluster/01-east.yaml
```


- To render the remotes as seeds and let gossip find the rest of the members, set `gateway.discovery` to `Gossip` in place of `Explicit`.
- `gateway.service` is the template of the external Service, and `gateway.advertise` is the address that the servers advertise.
- A `NatsCluster` whose gateway has no `tls` is refused, unless the cluster controller runs with `--allow-gateway-without-tls`.

A change to the gateway restarts the servers.
[Gateway](https://nats-operator.io/docs/reference/api/index.md#Gateway) in the API reference lists every field.

## Declare the users of the controllers in the other Kubernetes cluster

No auth controller runs in `west`, so declare the users of its controllers in `east`:

`01-east-auth.yaml`

```yaml
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: west-cluster-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: cluster-controller
  credentials:
    secretKeyRef:
      name: west-cluster-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: west-jetstream-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: jetstream-controller
  credentials:
    secretKeyRef:
      name: west-jetstream-controller-creds
```

```sh
kubectl --context east apply -f https://nats-operator.io/docs/stories/06-supercluster/01-east-auth.yaml
```


The auth controller writes their creds Secrets in `east`.
Deliver the Secrets `west-cluster-controller-creds` and `west-jetstream-controller-creds` to `nats-system` in `west`, for example with External Secrets or SOPS.

## Deploy the NATS cluster in the other Kubernetes cluster

Apply the `NatsCluster` `west`.
It refers to the same `NatsOperatorTrust` and lists the same gateway remotes as `east`, and its controller connects with the creds from `east`.
`west` receives the JWT of every account through its resolver.

`01-west.yaml`

```yaml
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: west
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "1"
      memory: 4Gi
    limits:
      memory: 4Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 100Gi
  serverTags:
    region: us-west-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: west-cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: east
        url: tls://nats-east.example.net:7222
      - name: west
        url: tls://nats-west.example.net:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats-west.example.net
    advertise: nats-west.example.net:7222
```

```sh
kubectl --context west apply -f https://nats-operator.io/docs/stories/06-supercluster/01-west.yaml
```


## Check the gateways

Read the status of `west`:

```sh
kubectl --context west -n nats-system get natscluster west -o yaml
```

The `status` in the output is similar to this:

`01-status-natscluster-west.yaml`

```yaml
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: AllServersReady
    - type: Settled
      status: "True"
      reason: AllGroupsCurrent
    - type: GatewaysConnected
      status: "True"
      reason: AllMembersReachable
      message: 1 of 1 remote members connected
  gateways:
    - name: east
      connected: true
      inbound: 3
      outbound: 3
  endpoints:
    client: nats://west.nats-system.svc:4222
    gateway: nats-west.example.net:7222
```


`GatewaysConnected` is True once every remote member is connected.

