Add a NATS cluster to a supercluster that you did not deploy

This guide shows you how to add a NATS cluster to a supercluster that these controllers do not run. The servers of that supercluster were deployed by other means, someone else has the keys of its NATS operator, and nothing pushes account JWTs to its servers. The manifests add the NATS cluster west to a supercluster whose one member is central, and create a stream on west in the account orders of the supercluster.

To build a supercluster of NATS clusters that you deploy yourself, see Build a supercluster across Kubernetes clusters.

Before you begin#

You need:

• A Kubernetes cluster with the cluster controller and the JetStream controller installed, and the namespaces nats-system and orders. Install shows how to install the controllers.
• From whoever has the keys of the NATS operator: the NATS operator JWT, the system account JWT, and the public key and the JWT of each account that you use on west.
• The creds of a user of the system account, for the cluster controller, in the Secret west-cluster-controller-creds in nats-system.
• The creds of a user of the account orders in the Secret orders-creds in orders.
• The DNS name nats-west.example.net pointing at the gateway Service of west, for example through external-dns.
• The StorageClass gp3, or your own in its place in the manifest.

Add the new member to the existing members#

A NatsCluster does not change a NATS cluster that it did not deploy. Ask whoever runs central to add west to its gateways before you deploy west. In this guide, central sets reject_unknown, so it refuses a gateway that its own list does not have.

This is the part of the config of central that matters to west:

gateway {
  name: central
  port: 7222
  reject_unknown: true
  gateways: [
    {name: west, urls: ["nats://nats-west.example.net:7222"]}
  ]
}
operator: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
system_account: ASYS...
resolver: MEMORY
resolver_preload: {
  ASYS...: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  AORDERS...: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  APAYMENTS...: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
}

central has its accounts as preloads under a MEMORY resolver. Its gateway has no TLS and no authorization.

The servers of west reach the gateways of central through the Service behind nats-central.example.net. If a server of central is Ready only once its JetStream reaches a meta leader, that Service must also publish servers that are not Ready, with publishNotReadyAddresses: true. The meta group spans both NATS clusters, so central alone may not hold a majority of it, and a server of central that west cannot reach may never become Ready.

Copy the trust roots#

Copy the NATS operator JWT and the system account JWT into a NatsOperatorTrust, and apply it:

apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
  name: central
  namespace: nats-system
spec:
  operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...

kubectl apply -f https://nats-operator.io/docs/stories/13-join-supercluster/01-natsoperatortrust.yaml

Copy the accounts#

Nothing pushes account JWTs to this supercluster, so west must preload them, as central does. For each account, copy its public key and the JWT that central preloads into a NatsAccountTrust, and apply them:

apiVersion: nats-operator.io/v1beta1
kind: NatsAccountTrust
metadata:
  name: orders
  namespace: nats-system
spec:
  publicKey: AORDERS...
  jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
---
apiVersion: nats-operator.io/v1beta1
kind: NatsAccountTrust
metadata:
  name: payments
  namespace: nats-system
spec:
  publicKey: APAYMENTS...
  jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...

kubectl apply -f https://nats-operator.io/docs/stories/13-join-supercluster/01-natsaccounttrusts.yaml

A copy is good until its JWT expires. A JWT that the auth controller signs expires after the jwtTTL of its NatsAccount, which is 48h by default, or never if jwtTTL is 0. A JWT signed elsewhere expires when its signer decided.

Allow a gateway without TLS#

Do this step only if the existing members run their gateways without TLS, as central does. The cluster controller refuses a NatsCluster whose gateway has no tls, unless it runs with --allow-gateway-without-tls. To pass the flag, upgrade the release with this chart value:

cluster:
  allowGatewayWithoutTLS: true

Deploy the new member#

The servers of west must start with no JetStream state. They then join the JetStream meta group of the supercluster. Servers that bring a meta group of their own do not merge with it, and either side can lose its streams. Use new claims: if a NatsCluster named west ran in nats-system before, without these gateways or with a jetstream.domain, delete its data volume claims first:

kubectl -n nats-system delete pvc data-west-0-0 data-west-1-0 data-west-2-0

The cluster controller creates no server whose claim is older than the NatsCluster, and reads Progressing False with the reason OwnMetaGroup instead. It also refuses to add gateway remotes to servers that already lead a meta group of their own. Trust, resolver, gateways, leaf nodes in the design describes what it checks.

Apply the NatsCluster west:

apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: west
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "1"
      memory: 4Gi
    limits:
      memory: 4Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 100Gi
  auth:
    trustRef:
      name: central
    systemCredentials:
      secretKeyRef:
        name: west-cluster-controller-creds
    accountTrustRefs:
      - name: orders
      - name: payments
  gateway:
    discovery: Explicit
    remotes:
      - name: central
        url: nats://nats-central.example.net:7222
      - name: west
        url: nats://nats-west.example.net:7222
    service:
      type: LoadBalancer
      annotations:
        external-dns.alpha.kubernetes.io/hostname: nats-west.example.net
    advertise: nats-west.example.net:7222

kubectl apply -f https://nats-operator.io/docs/stories/13-join-supercluster/01-west.yaml

• The name of the NatsCluster is its gateway name. It must be the name that the existing members have in their lists.
• auth.systemCredentials is the Secret with the creds for the cluster controller.
• auth.accountTrustRefs lists the accounts that every server preloads beside the system account. A change to the list, or to the JWT in a NatsAccountTrust on it, restarts the servers.
• gateway.remotes lists every member of the existing supercluster, and west itself.
• gateway has no tls, because the gateways of the existing members have none.
• gateway.service renders the Service west-gateway, which publishes the servers of west before they are Ready. A server of west is Ready once its JetStream reaches the meta leader, and the gateways of central reach it through this Service.

The servers run a Full resolver on the JetStream volume. If you remove an account from auth.accountTrustRefs, the account stays in the directory of the resolver, and the servers still serve it after the restart. Trust, resolver, gateways, leaf nodes in the design describes what the resolver stores.

Check the new member#

Read the status of west:

kubectl -n nats-system get natscluster west -o yaml

The status in the output is similar to this:

status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: AllServersReady
    - type: Settled
      status: "True"
      reason: AllGroupsCurrent
    - type: GatewaysConnected
      status: "True"
      reason: AllMembersReachable
      message: 1 of 1 remote members connected
  gateways:
    - name: central
      connected: true
      inbound: 3
      outbound: 3
  endpoints:
    client: nats://west.nats-system.svc:4222
    gateway: nats-west.example.net:7222

GatewaysConnected is True, and gateways has an entry for central. A supercluster has one JetStream meta group, so Settled is True once the servers of west are among the peers of the meta leader.

To check that west joined the meta group of central, read meta_cluster from /jsz on a server of each, on the monitoring port 8222:

kubectl -n nats-system port-forward pod/west-0-0 8222:8222
curl -s localhost:8222/jsz | jq .meta_cluster

In one meta group, every server reports the same leader and the same cluster_size, which counts the servers of west and of central. If the servers of west report a leader among themselves and a cluster_size that counts only them, west runs a meta group of its own beside that of central. Stop west at once, as Recover from a meta group of its own shows: while both groups run, either can delete the streams of the other.

Create a stream on the new member#

Apply a NatsConnection with the creds of the user of orders, and a stream on it:

apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: west
  namespace: orders
spec:
  servers: ["nats://west.nats-system.svc:4222"]
  credentials:
    secretKeyRef:
      name: orders-creds

kubectl apply -f https://nats-operator.io/docs/stories/13-join-supercluster/02-natsconnection.yaml

apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
  name: orders
  namespace: orders
spec:
  connectionRef:
    name: west
  name: ORDERS
  subjects: ["orders.>"]
  storage: File
  replicas: 3
  placement:
    cluster: west
  maxAge: 72h

kubectl apply -f https://nats-operator.io/docs/stories/13-join-supercluster/02-natsstream.yaml

west authenticates the user against the JWT of orders that it preloads. placement.cluster puts the stream on the servers of west and not on those of central, which are in the same meta group.

Read the status of the stream:

kubectl -n orders get natsstream orders -o yaml

The status in the output is similar to this:

status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Synced
    - type: Synced
      status: "True"
      reason: MatchesSpec
  ownership:
    origin: Created
  server:
    leader: west-1
    replicas:
      - name: west-0
        current: true
      - name: west-2
        current: true

server.leader and server.replicas are servers of west.

Recover from a meta group of its own#

1. Stop west. Set the annotation cluster.nats-operator.io/force-delete on it, since the deletion otherwise waits for its streams, and delete it:

   kubectl -n nats-system annotate natscluster west cluster.nats-operator.io/force-delete=true
   kubectl -n nats-system delete natscluster west

2. On a server of central, check that its streams are still listed. A stream that is gone comes back only from a backup.

3. Delete the data volume claims of west:

   kubectl -n nats-system delete pvc data-west-0-0 data-west-1-0 data-west-2-0

4. Apply the NatsCluster west again.