# Deploy a production supercluster of three NATS clusters


This guide shows you how to deploy a supercluster for production: three NATS clusters in three Kubernetes clusters, under one NATS operator, with JetStream balanced.
It combines the other guides, and each step links to the guide that has its options.

The manifests deploy these NATS clusters:

- `prod-east`, five servers, in the home cluster.
- `dev-east`, three servers, all in one zone.
- `prod-west`, three servers, in another region.

Four services use the supercluster, and each service has its own account.
The manifests have the accounts of the production environment.
The accounts of the development environment are the same under `-dev` names, and are left out.

## Before you begin

You need:

- Three Kubernetes clusters, with kubeconfig contexts named `prod-east`, `dev-east` and `prod-west`.
  Each manifest on this page is followed by the command that applies it with `--context`.
- In each, the cluster controller and the JetStream controller, cert-manager, the namespace `nats-system`, the StorageClass `gp3`, and LoadBalancer Services that the other Kubernetes clusters can reach.
  [Install](https://nats-operator.io/docs/install/index.md) shows how to install the controllers.
- In each, the Issuer `nats-gateway-ca` in `nats-system`, from a private CA.
  See [Create the gateway CA](https://nats-operator.io/docs/stories/06-supercluster/index.md#create-the-gateway-ca).
- In `prod-east`, the auth controller and the namespace `monitoring`.
- The DNS names `nats.prod-east.acme.example`, `nats.dev-east.acme.example` and `nats.prod-west.acme.example` pointing at the gateway Services, for example through external-dns.

## Declare the NATS operator and the accounts

Apply the NATS operator `acme`, its system account `sys` and the four accounts in the home cluster, `prod-east`:

`01-auth.yaml`

```yaml
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsOperator
metadata:
  name: acme
  namespace: nats-system
spec:
  systemAccountRef:
    name: sys
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsSystemAccount
metadata:
  name: sys
  namespace: nats-system
spec:
  operatorRef:
    name: acme
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
  name: checks-prod
  namespace: nats-system
spec:
  operatorRef:
    name: acme
  exports:
    - name: run
      type: Service
      subject: "checks.run.>"
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
  name: monitoring-prod
  namespace: nats-system
spec:
  operatorRef:
    name: acme
  limits:
    jetstream:
      memoryStorage: 36Gi
      diskStorage: 300Gi
  exports:
    - name: executions
      type: Stream
      subject: "monitoring.execution.>"
    - name: state-changes
      type: Stream
      subject: "monitoring.state.>"
    - name: execute
      type: Service
      subject: "monitoring.execute"
    - name: configure
      type: Service
      subject: "monitoring.configure"
  imports:
    - accountRef:
        kind: NatsAccount
        name: checks-prod
      export: run
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
  name: core-prod
  namespace: nats-system
spec:
  operatorRef:
    name: acme
  imports:
    - accountRef:
        kind: NatsAccount
        name: monitoring-prod
      export: state-changes
    - accountRef:
        kind: NatsAccount
        name: monitoring-prod
      export: configure
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
  name: collector-prod
  namespace: nats-system
spec:
  operatorRef:
    name: acme
  limits:
    jetstream:
      diskStorage: 50Gi
  imports:
    - accountRef:
        kind: NatsAccount
        name: monitoring-prod
      export: executions
```

```sh
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-auth.yaml
```


The accounts form a chain.
`checks-prod` exports a service to `monitoring-prod`, and `monitoring-prod` exports streams and services to `core-prod` and `collector-prod`.
[Share a stream and a service between accounts](https://nats-operator.io/docs/stories/05-account-wiring/index.md) shows how to declare exports and imports.

Declare the users of the home cluster's own controllers and the auth controller's connection as well.
[Put a NATS cluster under a NATS operator](https://nats-operator.io/docs/stories/02-auth-plane/index.md) shows them, with the Secrets `cluster-controller-creds` and `jetstream-controller-creds` that the manifests on this page refer to.

## Declare the users

Apply the users in `prod-east`:

`01-users.yaml`

```yaml
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: monitoring-prod-service
  namespace: monitoring
spec:
  accountRef:
    kind: NatsAccount
    namespace: nats-system
    name: monitoring-prod
  permissions:
    publish:
      allow: [">"]
    subscribe:
      allow: [">"]
  credentials:
    secretKeyRef:
      name: nats-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: monitoring-prod-readonly
  namespace: monitoring
spec:
  accountRef:
    kind: NatsAccount
    namespace: nats-system
    name: monitoring-prod
  preset: readonly
  credentials:
    secretKeyRef:
      name: nats-readonly-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: checks-prod-service
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: checks-prod
  permissions:
    publish:
      allow: [">"]
    subscribe:
      allow: [">"]
  credentials:
    secretKeyRef:
      name: checks-prod-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: checks-prod-readonly
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: checks-prod
  preset: readonly
  credentials:
    secretKeyRef:
      name: checks-prod-readonly-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: core-prod-service
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: core-prod
  permissions:
    publish:
      allow: [">"]
    subscribe:
      allow: [">"]
  credentials:
    secretKeyRef:
      name: core-prod-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: core-prod-readonly
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: core-prod
  preset: readonly
  credentials:
    secretKeyRef:
      name: core-prod-readonly-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: collector-prod-service
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: collector-prod
  permissions:
    publish:
      allow: [">"]
    subscribe:
      allow: [">"]
  credentials:
    secretKeyRef:
      name: collector-prod-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: collector-prod-readonly
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: collector-prod
  preset: readonly
  credentials:
    secretKeyRef:
      name: collector-prod-readonly-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: dev-east-cluster-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: cluster-controller
  credentials:
    secretKeyRef:
      name: dev-east-cluster-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: dev-east-jetstream-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: jetstream-controller
  credentials:
    secretKeyRef:
      name: dev-east-jetstream-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: prod-west-cluster-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: cluster-controller
  credentials:
    secretKeyRef:
      name: prod-west-cluster-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: prod-west-jetstream-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: jetstream-controller
  credentials:
    secretKeyRef:
      name: prod-west-jetstream-controller-creds
---
apiVersion: nats-operator.io/v1beta1
kind: NatsReferenceGrant
metadata:
  name: monitoring-users
  namespace: nats-system
spec:
  from:
    - group: auth.nats-operator.io
      kind: NatsUser
      namespace: monitoring
  to:
    - group: auth.nats-operator.io
      kind: NatsAccount
      name: monitoring-prod
    - group: auth.nats-operator.io
      kind: NatsAccount
      name: monitoring-dev
```

```sh
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-users.yaml
```


- Every account has a `service` user and a `readonly` user.
  A `readonly` user sets the preset `readonly`, and a user with a preset cannot set `permissions`.
  [readonly](https://nats-operator.io/docs/reference/nats-permissions/index.md#readonly) in the NATS permissions reference lists the subjects of the preset.
- The users of the monitoring team are in the namespace `monitoring`, and the `NatsReferenceGrant` lets them refer to the team's accounts in `nats-system`.
  See [Let a team declare its own users and streams](https://nats-operator.io/docs/stories/04-team-self-service/index.md).
- Each of the other two Kubernetes clusters has two users, one for its cluster controller and one for its JetStream controller.

The auth controller writes the creds Secrets in `prod-east`.
Deliver the Secrets of the controllers' users to `nats-system` in `dev-east` and `prod-west`, as [Build a supercluster across Kubernetes clusters](https://nats-operator.io/docs/stories/06-supercluster/index.md) shows.

## Copy the trust roots to every Kubernetes cluster

Copy the NATS operator JWT and the system account JWT from the status of the `NatsOperator` into a `NatsOperatorTrust`, and apply the same one in all three Kubernetes clusters.
Keep the copies alike with GitOps.

`01-natsoperatortrust.yaml`

```yaml
apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
  name: acme
  namespace: nats-system
spec:
  operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
```

```sh
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-natsoperatortrust.yaml
kubectl --context dev-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-natsoperatortrust.yaml
kubectl --context prod-west apply -f https://nats-operator.io/docs/stories/09-acceptance/01-natsoperatortrust.yaml
```


## Deploy the NATS clusters

Apply one `NatsCluster` in each Kubernetes cluster.
All three have the same `gateway.remotes`.

`01-prod-east.yaml`

```yaml
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 5
  resources:
    requests:
      cpu: "6"
      memory: 29Gi
    limits:
      memory: 29Gi
  jetstream:
    limits:
      maxMemoryStore: 10Gi
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 500Gi
  serverTags:
    cluster: prod-east
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-east.acme.example
    advertise: nats.prod-east.acme.example:7222
```

```sh
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-prod-east.yaml
```


`jetstream.limits.maxMemoryStore` sets the size of the memory store.
Without it, the cluster controller derives the size from `resources.limits.memory`.

`01-dev-east.yaml`

```yaml
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: dev-east
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "1500m"
      memory: 3Gi
    limits:
      memory: 3Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 100Gi
  serverTags:
    cluster: dev-east
    region: us-east-2
  podTemplate:
    spec:
      nodeSelector:
        topology.kubernetes.io/zone: us-east-2a
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: dev-east-cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.dev-east.acme.example
    advertise: nats.dev-east.acme.example:7222
```

```sh
kubectl --context dev-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-dev-east.yaml
```


`podTemplate` passes scheduling fields to the pods.
Here, a `nodeSelector` puts every server of `dev-east` in one zone.

`01-prod-west.yaml`

```yaml
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-west
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "1500m"
      memory: 3Gi
    limits:
      memory: 3Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 100Gi
  serverTags:
    cluster: west
    region: us-west-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: prod-west-cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-west.acme.example
    advertise: nats.prod-west.acme.example:7222
```

```sh
kubectl --context prod-west apply -f https://nats-operator.io/docs/stories/09-acceptance/01-prod-west.yaml
```


`serverTags` are yours to choose.
The placement tag of `prod-west` is `cluster:west`, which is not its name.

[NatsClusterSpec](https://nats-operator.io/docs/reference/api/index.md#NatsClusterSpec) in the API reference lists every field.

## Check the gateways

Read the status of `prod-west`:

```sh
kubectl --context prod-west -n nats-system get natscluster prod-west -o yaml
```

The `status` in the output is similar to this:

`01-status-natscluster-prod-west.yaml`

```yaml
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: AllServersReady
    - type: Settled
      status: "True"
      reason: AllGroupsCurrent
    - type: GatewaysConnected
      status: "True"
      reason: AllMembersReachable
      message: 2 of 2 remote members connected
  gateways:
    - name: dev-east
      connected: true
      inbound: 3
      outbound: 3
    - name: prod-east
      connected: true
      inbound: 5
      outbound: 3
  endpoints:
    client: nats://prod-west.nats-system.svc:4222
    gateway: nats.prod-west.acme.example:7222
```


`GatewaysConnected` is True once both of the other members are connected.
The status of every `NatsCluster` has an entry under `gateways` for each of the other members.

## Check that the accounts reached every server

Every account reaches every server of the supercluster through the resolver.
Read the status of an account:

```sh
kubectl --context prod-east -n nats-system get natsaccount monitoring-prod -o yaml
```

The `status` in the output is similar to this:

`01-status-natsaccount-monitoring-prod.yaml`

```yaml
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Distributed
    - type: Distributed
      status: "True"
      reason: AllServersCurrent
      message: 11 of 11 servers hold this JWT
  distribution:
    servers: 11
    current: 11
```


`distribution` counts the servers of all three NATS clusters.

## Adopt the streams and balance them

The application of the monitoring team created its streams `REQUESTS` and `RESPONSES` itself, before any `NatsStream` existed.
Apply the manifest in `prod-east`:

`02-jetstream.yaml`

```yaml
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: prod
  namespace: monitoring
spec:
  servers: ["nats://prod-east.nats-system.svc:4222"]
  credentials:
    secretKeyRef:
      name: nats-service-creds
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
  name: requests
  namespace: monitoring
  labels:
    traffic: requests
spec:
  connectionRef:
    name: prod
  adoptionPolicy: Adopt
  name: REQUESTS
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
  name: responses
  namespace: monitoring
  labels:
    traffic: responses
spec:
  connectionRef:
    name: prod
  adoptionPolicy: Adopt
  name: RESPONSES
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsBalancer
metadata:
  name: prod
  namespace: monitoring
spec:
  connectionRef:
    name: prod
  pools:
    - name: requests
      selector:
        matchLabels:
          traffic: requests
    - name: responses
      selector:
        matchLabels:
          traffic: responses
---
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: prod-east-sys
  namespace: nats-system
spec:
  servers: ["nats://prod-east.nats-system.svc:4222"]
  credentials:
    secretKeyRef:
      name: jetstream-controller-creds
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsSystemBalancer
metadata:
  name: prod-east
  namespace: nats-system
spec:
  connectionRef:
    name: prod-east-sys
  moves:
    placement: true
    leader: true
```

```sh
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/02-jetstream.yaml
```


- The two `NatsStream`s set `adoptionPolicy: Adopt`, so they adopt the streams and keep their config.
  See [Adopting streams](https://nats-operator.io/docs/stories/03-unmanaged/index.md#adopting-streams).
- The `NatsBalancer` balances the account of the monitoring team in two pools, which select the streams by label.
- The `NatsSystemBalancer` balances every account of `prod-east`.
  The platform team needs one for each NATS cluster, and the manifest has only the one for `prod-east`.

[Balance JetStream leaders and copies](https://nats-operator.io/docs/stories/07-balancing/index.md) describes both balancers.

Read the status of an adopted stream:

```sh
kubectl --context prod-east -n monitoring get natsstream requests -o yaml
```

The `status` in the output is similar to this:

`02-status-natsstream-requests.yaml`

```yaml
status:
  observedGeneration: 2
  conditions:
    - type: Ready
      status: "True"
      reason: Synced
    - type: Adopted
      status: "True"
      reason: FoundUnowned
      message: adopted existing stream REQUESTS; spec written from the server
    - type: Synced
      status: "True"
      reason: MatchesSpec
  ownership:
    origin: Adopted
  server:
    leader: prod-east-2
```


Read the status of the account balancer:

```sh
kubectl --context prod-east -n monitoring get natsbalancer prod -o yaml
```

The `status` in the output is similar to this:

`02-status-natsbalancer.yaml`

```yaml
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Balancing
    - type: Holding
      status: "False"
      reason: Settled
    - type: Overlapping
      status: "False"
      reason: PoolsDisjoint
  pools:
    - name: requests
      streams: 1
      leaderSkew: 1
    - name: responses
      streams: 1
      leaderSkew: 1
```


Each pool has one stream, so its `leaderSkew` is 1.
Every stream of the account has a resource in a pool, so the status has no default pool.

Read the status of the system balancer:

```sh
kubectl --context prod-east -n nats-system get natssystembalancer prod-east -o yaml
```

The `status` in the output is similar to this:

`02-status-natssystembalancer.yaml`

```yaml
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Balancing
    - type: Holding
      status: "False"
      reason: Settled
  capabilities:
    placement: true
    leader: None
    leaderReason: 1 of 1 accounts carry no jetstream-stepdown export; their leaders are not moved
```


`capabilities.leader` is None.
`monitoring-prod` is the one account with streams in `prod-east`, and it has no `jetstream-stepdown` export, so the system balancer moves no leader.
To let it, add the export as in [Let the system balancer move the leaders of an account](https://nats-operator.io/docs/stories/07-balancing/index.md#let-the-system-balancer-move-the-leaders-of-an-account).

