Story 06 · Several NATS clusters
Build a supercluster across Kubernetes clusters
Contents
This guide shows you how to join NATS clusters in two Kubernetes clusters into one supercluster with gateways.
The manifests deploy the NATS cluster east in the Kubernetes cluster east and the NATS cluster west in the Kubernetes cluster west.
east is the home cluster: the auth controller runs there, and every account is declared there.
Before you begin#
You need:
- Two Kubernetes clusters, with kubeconfig contexts named
eastandwest. Each manifest on this page is followed by the command that applies it with--context. - In both, the cluster controller and the JetStream controller, cert-manager, the namespace
nats-system, and LoadBalancer Services that the other Kubernetes cluster can reach. Install shows how to install the controllers. - In
east, the auth controller, aNatsOperatorwith its system accountsys, and the cluster controller’s user with its creds in the Secretcluster-controller-creds. Put a NATS cluster under a NATS operator declares them. - The DNS names
nats-east.example.netandnats-west.example.netpointing at the gateway Services, for example through external-dns. - The StorageClass
gp3in both, or your own in its place in the manifests.
Copy the trust roots to both Kubernetes clusters#
Copy the NATS operator JWT and the system account JWT from the status of the NatsOperator in east into a NatsOperatorTrust, and apply the same one in both Kubernetes clusters.
Keep the two copies alike with GitOps, and copy the JWTs again whenever you rotate a key of the NATS operator.
apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
name: acme
namespace: nats-system
spec:
operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
kubectl --context east apply -f https://nats-operator.io/docs/stories/06-supercluster/01-natsoperatortrust.yaml
kubectl --context west apply -f https://nats-operator.io/docs/stories/06-supercluster/01-natsoperatortrust.yamlCreate the gateway CA#
Gateways accept each other by certificate alone.
Give them a private CA: replicate its key pair into the Secret nats-gateway-ca in nats-system of both Kubernetes clusters, and create an Issuer for it in both:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: nats-gateway-ca
namespace: nats-system
spec:
ca:
secretName: nats-gateway-caDo not use a public issuer or an ACME issuer. Trust, resolver, gateways, leaf nodes in the design explains why neither works.
Do not use a ClusterIssuer either.
A ClusterIssuer gives a gateway certificate, and with it a place in the supercluster, to anyone who can write a NatsCluster in any namespace that the cluster controller watches.
Deploy the NATS cluster in the home cluster#
Apply the NatsCluster east.
Its name is its gateway name.
gateway.remotes lists every member of the supercluster, itself included.
Give every member the same list.
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: east
namespace: nats-system
spec:
version: 2.15.0
replicas: 3
resources:
requests:
cpu: "2"
memory: 8Gi
limits:
memory: 8Gi
jetstream:
volumeClaimTemplate:
spec:
storageClassName: gp3
resources:
requests:
storage: 200Gi
serverTags:
region: us-east-2
auth:
trustRef:
name: acme
systemCredentials:
secretKeyRef:
name: cluster-controller-creds
gateway:
discovery: Explicit
remotes:
- name: east
url: tls://nats-east.example.net:7222
- name: west
url: tls://nats-west.example.net:7222
tls:
certManager:
issuerRef:
kind: Issuer
name: nats-gateway-ca
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
external-dns.alpha.kubernetes.io/hostname: nats-east.example.net
advertise: nats-east.example.net:7222
kubectl --context east apply -f https://nats-operator.io/docs/stories/06-supercluster/01-east.yaml- To render the remotes as seeds and let gossip find the rest of the members, set
gateway.discoverytoGossipin place ofExplicit. gateway.serviceis the template of the external Service, andgateway.advertiseis the address that the servers advertise.- A
NatsClusterwhose gateway has notlsis refused, unless the cluster controller runs with--allow-gateway-without-tls.
A change to the gateway restarts the servers. Gateway in the API reference lists every field.
Declare the users of the controllers in the other Kubernetes cluster#
No auth controller runs in west, so declare the users of its controllers in east:
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: west-cluster-controller
namespace: nats-system
spec:
accountRef:
kind: NatsSystemAccount
name: sys
preset: cluster-controller
credentials:
secretKeyRef:
name: west-cluster-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: west-jetstream-controller
namespace: nats-system
spec:
accountRef:
kind: NatsSystemAccount
name: sys
preset: jetstream-controller
credentials:
secretKeyRef:
name: west-jetstream-controller-creds
kubectl --context east apply -f https://nats-operator.io/docs/stories/06-supercluster/01-east-auth.yamlThe auth controller writes their creds Secrets in east.
Deliver the Secrets west-cluster-controller-creds and west-jetstream-controller-creds to nats-system in west, for example with External Secrets or SOPS.
Deploy the NATS cluster in the other Kubernetes cluster#
Apply the NatsCluster west.
It refers to the same NatsOperatorTrust and lists the same gateway remotes as east, and its controller connects with the creds from east.
west receives the JWT of every account through its resolver.
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: west
namespace: nats-system
spec:
version: 2.15.0
replicas: 3
resources:
requests:
cpu: "1"
memory: 4Gi
limits:
memory: 4Gi
jetstream:
volumeClaimTemplate:
spec:
storageClassName: gp3
resources:
requests:
storage: 100Gi
serverTags:
region: us-west-2
auth:
trustRef:
name: acme
systemCredentials:
secretKeyRef:
name: west-cluster-controller-creds
gateway:
discovery: Explicit
remotes:
- name: east
url: tls://nats-east.example.net:7222
- name: west
url: tls://nats-west.example.net:7222
tls:
certManager:
issuerRef:
kind: Issuer
name: nats-gateway-ca
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
external-dns.alpha.kubernetes.io/hostname: nats-west.example.net
advertise: nats-west.example.net:7222
kubectl --context west apply -f https://nats-operator.io/docs/stories/06-supercluster/01-west.yamlCheck the gateways#
Read the status of west:
kubectl --context west -n nats-system get natscluster west -o yamlThe status in the output is similar to this:
status:
observedGeneration: 1
conditions:
- type: Ready
status: "True"
reason: AllServersReady
- type: Settled
status: "True"
reason: AllGroupsCurrent
- type: GatewaysConnected
status: "True"
reason: AllMembersReachable
message: 1 of 1 remote members connected
gateways:
- name: east
connected: true
inbound: 3
outbound: 3
endpoints:
client: nats://west.nats-system.svc:4222
gateway: nats-west.example.net:7222
GatewaysConnected is True once every remote member is connected.