Two Kubernetes clusters, east and west, each with its own NATS cluster, joined by gateways into one supercluster with no hub. east is the home cluster: the auth controller runs there, holds the signing key, and every account is declared there.

Trust roots #

Identical in both Kubernetes clusters and replicated by GitOps: the trust roots every NATS cluster boots from.

01-natsoperatortrust.yaml

# Identical in every Kubernetes cluster of the supercluster, replicated by
# GitOps.
#
# Trust roots, public and signed, in the literal form: copied from the home
# Kubernetes cluster's NatsOperator status at bootstrap and on rotation. Every
# NATS cluster preloads the system account JWT from here. Each member names it
# in `auth.trustRef`, as a leaf does (story 10).
apiVersion: nats.mikluko.io/v1beta1
kind: NatsOperatorTrust
metadata:
  name: acme
  namespace: nats-system
spec:
  operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...

kubectl --context east apply -f https://nats-operator.io/docs/stories/06-supercluster/01-natsoperatortrust.yaml
kubectl --context west apply -f https://nats-operator.io/docs/stories/06-supercluster/01-natsoperatortrust.yaml

The gateways take their certificates from a private CA whose key pair every member holds in nats-system, replicated the same way: gateways authenticate each other by certificate alone, so a public issuer would admit any certificate it signs, and the cluster controller holds the servers until the certificate Secret carries ca.crt, which an ACME issuer never writes. The CA is an Issuer in the NATS cluster’s own namespace, not a ClusterIssuer: a ClusterIssuer named by a NatsCluster issues a gateway certificate to whoever may write a NatsCluster in any namespace the cluster controller watches, and with it a seat in the supercluster.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: nats-gateway-ca
  namespace: nats-system
spec:
  ca:
    secretName: nats-gateway-ca

The home cluster #

There is no supercluster resource: each NatsCluster lists the gateways it joins, the same list in every member. The external Service is rendered from a template.

01-east.yaml

# The home Kubernetes cluster: the auth controller runs here, and here alone.
apiVersion: cluster.nats.mikluko.io/v1beta1
kind: NatsCluster
metadata:
  name: east   # the gateway name, and the remote name the other members use
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "2"
      memory: 8Gi
    limits:
      memory: 8Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 200Gi
  serverTags:
    region: us-east-2
  auth:
    # The literal NatsOperatorTrust replicated to every member, even here in
    # the home Kubernetes cluster.
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    # Joining a supercluster is this list: every member's NatsCluster carries
    # the same one. Explicit renders each entry as a gateway remote with
    # reject_unknown on; Gossip renders them as seeds with reject_unknown off.
    # Either way a change restarts servers: gateway config is not reloadable.
    # A NATS cluster's own entry is allowed and skipped.
    discovery: Explicit
    remotes:
      - name: east
        url: tls://nats-east.example.net:7222
      - name: west
        url: tls://nats-west.example.net:7222
    # Without a `tls` block the NatsCluster is refused unless the cluster
    # controller runs with `--allow-gateway-without-tls`.
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    # The external Service is rendered from this template. The servers
    # advertise `advertise` below, the Service's hostname, which the other
    # members' remotes name too.
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats-east.example.net
    advertise: nats-east.example.net:7222

kubectl --context east apply -f https://nats-operator.io/docs/stories/06-supercluster/01-east.yaml

The remote Kubernetes cluster’s controllers run as system users declared here and carried across by External Secrets or SOPS.

01-east-auth.yaml

# Declared in the home cluster for the remote one: the west controllers'
# system users. Scoped by preset, never expiring, removed by
# revocation. Their Secrets are written here and carried to west.
apiVersion: auth.nats.mikluko.io/v1beta1
kind: NatsUser
metadata:
  name: west-cluster-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: cluster-controller
  credentials:
    secretKeyRef:
      name: west-cluster-controller-creds
---
apiVersion: auth.nats.mikluko.io/v1beta1
kind: NatsUser
metadata:
  name: west-jetstream-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: jetstream-controller
  credentials:
    secretKeyRef:
      name: west-jetstream-controller-creds

kubectl --context east apply -f https://nats-operator.io/docs/stories/06-supercluster/01-east-auth.yaml

The remote Kubernetes cluster #

The same shape, with no auth controller and no signing key. It receives every account’s JWT through the resolver.

01-west.yaml

# A remote Kubernetes cluster: no auth controller, no signing key. What it
# needs is the NatsOperatorTrust above, the same gateway remotes as every
# member, and the controllers' creds Secrets, which External Secrets or SOPS
# delivers from the home cluster.
apiVersion: cluster.nats.mikluko.io/v1beta1
kind: NatsCluster
metadata:
  name: west
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "1"
      memory: 4Gi
    limits:
      memory: 4Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 100Gi
  serverTags:
    region: us-west-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: west-cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: east
        url: tls://nats-east.example.net:7222
      - name: west
        url: tls://nats-west.example.net:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats-west.example.net
    advertise: nats-west.example.net:7222

kubectl --context west apply -f https://nats-operator.io/docs/stories/06-supercluster/01-west.yaml

01-status-natscluster-west.yaml

status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: AllServersReady
    - type: Settled
      status: "True"
      reason: AllGroupsCurrent
    - type: GatewaysConnected
      status: "True"
      reason: AllMembersReachable
      message: 1 of 1 remote members connected
  gateways:
    - name: east
      connected: true
      inbound: 3
      outbound: 3
  endpoints:
    client: nats://west.nats-system.svc:4222
    gateway: nats-west.example.net:7222