Story 06 · Several NATS clusters

Build a supercluster across Kubernetes clusters

Contents

This guide shows you how to join NATS clusters in two Kubernetes clusters into one supercluster with gateways. The manifests deploy the NATS cluster east in the Kubernetes cluster east and the NATS cluster west in the Kubernetes cluster west. east is the home cluster: the auth controller runs there, and every account is declared there.

Before you begin#

You need:

  • Two Kubernetes clusters, with kubeconfig contexts named east and west. Each manifest on this page is followed by the command that applies it with --context.
  • In both, the cluster controller and the JetStream controller, cert-manager, the namespace nats-system, and LoadBalancer Services that the other Kubernetes cluster can reach. Install shows how to install the controllers.
  • In east, the auth controller, a NatsOperator with its system account sys, and the cluster controller’s user with its creds in the Secret cluster-controller-creds. Put a NATS cluster under a NATS operator declares them.
  • The DNS names nats-east.example.net and nats-west.example.net pointing at the gateway Services, for example through external-dns.
  • The StorageClass gp3 in both, or your own in its place in the manifests.

Copy the trust roots to both Kubernetes clusters#

Copy the NATS operator JWT and the system account JWT from the status of the NatsOperator in east into a NatsOperatorTrust, and apply the same one in both Kubernetes clusters. Keep the two copies alike with GitOps, and copy the JWTs again whenever you rotate a key of the NATS operator.

01-natsoperatortrust.yaml yaml ·
apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
  name: acme
  namespace: nats-system
spec:
  operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
kubectl --context east apply -f https://nats-operator.io/docs/stories/06-supercluster/01-natsoperatortrust.yaml
kubectl --context west apply -f https://nats-operator.io/docs/stories/06-supercluster/01-natsoperatortrust.yaml

Create the gateway CA#

Gateways accept each other by certificate alone. Give them a private CA: replicate its key pair into the Secret nats-gateway-ca in nats-system of both Kubernetes clusters, and create an Issuer for it in both:

yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: nats-gateway-ca
  namespace: nats-system
spec:
  ca:
    secretName: nats-gateway-ca

Do not use a public issuer or an ACME issuer. Trust, resolver, gateways, leaf nodes in the design explains why neither works.

Do not use a ClusterIssuer either. A ClusterIssuer gives a gateway certificate, and with it a place in the supercluster, to anyone who can write a NatsCluster in any namespace that the cluster controller watches.

Deploy the NATS cluster in the home cluster#

Apply the NatsCluster east. Its name is its gateway name. gateway.remotes lists every member of the supercluster, itself included. Give every member the same list.

01-east.yaml yaml ·
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: east
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "2"
      memory: 8Gi
    limits:
      memory: 8Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 200Gi
  serverTags:
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: east
        url: tls://nats-east.example.net:7222
      - name: west
        url: tls://nats-west.example.net:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats-east.example.net
    advertise: nats-east.example.net:7222
kubectl --context east apply -f https://nats-operator.io/docs/stories/06-supercluster/01-east.yaml
  • To render the remotes as seeds and let gossip find the rest of the members, set gateway.discovery to Gossip in place of Explicit.
  • gateway.service is the template of the external Service, and gateway.advertise is the address that the servers advertise.
  • A NatsCluster whose gateway has no tls is refused, unless the cluster controller runs with --allow-gateway-without-tls.

A change to the gateway restarts the servers. Gateway in the API reference lists every field.

Declare the users of the controllers in the other Kubernetes cluster#

No auth controller runs in west, so declare the users of its controllers in east:

01-east-auth.yaml yaml ·
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: west-cluster-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: cluster-controller
  credentials:
    secretKeyRef:
      name: west-cluster-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: west-jetstream-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: jetstream-controller
  credentials:
    secretKeyRef:
      name: west-jetstream-controller-creds
kubectl --context east apply -f https://nats-operator.io/docs/stories/06-supercluster/01-east-auth.yaml

The auth controller writes their creds Secrets in east. Deliver the Secrets west-cluster-controller-creds and west-jetstream-controller-creds to nats-system in west, for example with External Secrets or SOPS.

Deploy the NATS cluster in the other Kubernetes cluster#

Apply the NatsCluster west. It refers to the same NatsOperatorTrust and lists the same gateway remotes as east, and its controller connects with the creds from east. west receives the JWT of every account through its resolver.

01-west.yaml yaml ·
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: west
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "1"
      memory: 4Gi
    limits:
      memory: 4Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 100Gi
  serverTags:
    region: us-west-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: west-cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: east
        url: tls://nats-east.example.net:7222
      - name: west
        url: tls://nats-west.example.net:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats-west.example.net
    advertise: nats-west.example.net:7222
kubectl --context west apply -f https://nats-operator.io/docs/stories/06-supercluster/01-west.yaml

Check the gateways#

Read the status of west:

kubectl --context west -n nats-system get natscluster west -o yaml

The status in the output is similar to this:

01-status-natscluster-west.yaml yaml ·
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: AllServersReady
    - type: Settled
      status: "True"
      reason: AllGroupsCurrent
    - type: GatewaysConnected
      status: "True"
      reason: AllMembersReachable
      message: 1 of 1 remote members connected
  gateways:
    - name: east
      connected: true
      inbound: 3
      outbound: 3
  endpoints:
    client: nats://west.nats-system.svc:4222
    gateway: nats-west.example.net:7222

GatewaysConnected is True once every remote member is connected.

Esc
↓ ↑ Navigate Enter Select Esc Close