Story 09 · Several NATS clusters
Deploy a production supercluster of three NATS clusters
Contents
This guide shows you how to deploy a supercluster for production: three NATS clusters in three Kubernetes clusters, under one NATS operator, with JetStream balanced. It combines the other guides, and each step links to the guide that has its options.
The manifests deploy these NATS clusters:
prod-east, five servers, in the home cluster.dev-east, three servers, all in one zone.prod-west, three servers, in another region.
Four services use the supercluster, and each service has its own account.
The manifests have the accounts of the production environment.
The accounts of the development environment are the same under -dev names, and are left out.
Before you begin#
You need:
- Three Kubernetes clusters, with kubeconfig contexts named
prod-east,dev-eastandprod-west. Each manifest on this page is followed by the command that applies it with--context. - In each, the cluster controller and the JetStream controller, cert-manager, the namespace
nats-system, the StorageClassgp3, and LoadBalancer Services that the other Kubernetes clusters can reach. Install shows how to install the controllers. - In each, the Issuer
nats-gateway-cainnats-system, from a private CA. See Create the gateway CA. - In
prod-east, the auth controller and the namespacemonitoring. - The DNS names
nats.prod-east.acme.example,nats.dev-east.acme.exampleandnats.prod-west.acme.examplepointing at the gateway Services, for example through external-dns.
Declare the NATS operator and the accounts#
Apply the NATS operator acme, its system account sys and the four accounts in the home cluster, prod-east:
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsOperator
metadata:
name: acme
namespace: nats-system
spec:
systemAccountRef:
name: sys
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsSystemAccount
metadata:
name: sys
namespace: nats-system
spec:
operatorRef:
name: acme
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
name: checks-prod
namespace: nats-system
spec:
operatorRef:
name: acme
exports:
- name: run
type: Service
subject: "checks.run.>"
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
name: monitoring-prod
namespace: nats-system
spec:
operatorRef:
name: acme
limits:
jetstream:
memoryStorage: 36Gi
diskStorage: 300Gi
exports:
- name: executions
type: Stream
subject: "monitoring.execution.>"
- name: state-changes
type: Stream
subject: "monitoring.state.>"
- name: execute
type: Service
subject: "monitoring.execute"
- name: configure
type: Service
subject: "monitoring.configure"
imports:
- accountRef:
kind: NatsAccount
name: checks-prod
export: run
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
name: core-prod
namespace: nats-system
spec:
operatorRef:
name: acme
imports:
- accountRef:
kind: NatsAccount
name: monitoring-prod
export: state-changes
- accountRef:
kind: NatsAccount
name: monitoring-prod
export: configure
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
name: collector-prod
namespace: nats-system
spec:
operatorRef:
name: acme
limits:
jetstream:
diskStorage: 50Gi
imports:
- accountRef:
kind: NatsAccount
name: monitoring-prod
export: executions
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-auth.yamlThe accounts form a chain.
checks-prod exports a service to monitoring-prod, and monitoring-prod exports streams and services to core-prod and collector-prod.
Share a stream and a service between accounts shows how to declare exports and imports.
Declare the users of the home cluster’s own controllers and the auth controller’s connection as well.
Put a NATS cluster under a NATS operator shows them, with the Secrets cluster-controller-creds and jetstream-controller-creds that the manifests on this page refer to.
Declare the users#
Apply the users in prod-east:
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: monitoring-prod-service
namespace: monitoring
spec:
accountRef:
kind: NatsAccount
namespace: nats-system
name: monitoring-prod
permissions:
publish:
allow: [">"]
subscribe:
allow: [">"]
credentials:
secretKeyRef:
name: nats-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: monitoring-prod-readonly
namespace: monitoring
spec:
accountRef:
kind: NatsAccount
namespace: nats-system
name: monitoring-prod
preset: readonly
credentials:
secretKeyRef:
name: nats-readonly-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: checks-prod-service
namespace: nats-system
spec:
accountRef:
kind: NatsAccount
name: checks-prod
permissions:
publish:
allow: [">"]
subscribe:
allow: [">"]
credentials:
secretKeyRef:
name: checks-prod-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: checks-prod-readonly
namespace: nats-system
spec:
accountRef:
kind: NatsAccount
name: checks-prod
preset: readonly
credentials:
secretKeyRef:
name: checks-prod-readonly-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: core-prod-service
namespace: nats-system
spec:
accountRef:
kind: NatsAccount
name: core-prod
permissions:
publish:
allow: [">"]
subscribe:
allow: [">"]
credentials:
secretKeyRef:
name: core-prod-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: core-prod-readonly
namespace: nats-system
spec:
accountRef:
kind: NatsAccount
name: core-prod
preset: readonly
credentials:
secretKeyRef:
name: core-prod-readonly-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: collector-prod-service
namespace: nats-system
spec:
accountRef:
kind: NatsAccount
name: collector-prod
permissions:
publish:
allow: [">"]
subscribe:
allow: [">"]
credentials:
secretKeyRef:
name: collector-prod-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: collector-prod-readonly
namespace: nats-system
spec:
accountRef:
kind: NatsAccount
name: collector-prod
preset: readonly
credentials:
secretKeyRef:
name: collector-prod-readonly-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: dev-east-cluster-controller
namespace: nats-system
spec:
accountRef:
kind: NatsSystemAccount
name: sys
preset: cluster-controller
credentials:
secretKeyRef:
name: dev-east-cluster-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: dev-east-jetstream-controller
namespace: nats-system
spec:
accountRef:
kind: NatsSystemAccount
name: sys
preset: jetstream-controller
credentials:
secretKeyRef:
name: dev-east-jetstream-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: prod-west-cluster-controller
namespace: nats-system
spec:
accountRef:
kind: NatsSystemAccount
name: sys
preset: cluster-controller
credentials:
secretKeyRef:
name: prod-west-cluster-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: prod-west-jetstream-controller
namespace: nats-system
spec:
accountRef:
kind: NatsSystemAccount
name: sys
preset: jetstream-controller
credentials:
secretKeyRef:
name: prod-west-jetstream-controller-creds
---
apiVersion: nats-operator.io/v1beta1
kind: NatsReferenceGrant
metadata:
name: monitoring-users
namespace: nats-system
spec:
from:
- group: auth.nats-operator.io
kind: NatsUser
namespace: monitoring
to:
- group: auth.nats-operator.io
kind: NatsAccount
name: monitoring-prod
- group: auth.nats-operator.io
kind: NatsAccount
name: monitoring-dev
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-users.yaml- Every account has a
serviceuser and areadonlyuser. Areadonlyuser sets the presetreadonly, and a user with a preset cannot setpermissions. readonly in the NATS permissions reference lists the subjects of the preset. - The users of the monitoring team are in the namespace
monitoring, and theNatsReferenceGrantlets them refer to the team’s accounts innats-system. See Let a team declare its own users and streams. - Each of the other two Kubernetes clusters has two users, one for its cluster controller and one for its JetStream controller.
The auth controller writes the creds Secrets in prod-east.
Deliver the Secrets of the controllers’ users to nats-system in dev-east and prod-west, as Build a supercluster across Kubernetes clusters shows.
Copy the trust roots to every Kubernetes cluster#
Copy the NATS operator JWT and the system account JWT from the status of the NatsOperator into a NatsOperatorTrust, and apply the same one in all three Kubernetes clusters.
Keep the copies alike with GitOps.
apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
name: acme
namespace: nats-system
spec:
operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-natsoperatortrust.yaml
kubectl --context dev-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-natsoperatortrust.yaml
kubectl --context prod-west apply -f https://nats-operator.io/docs/stories/09-acceptance/01-natsoperatortrust.yamlDeploy the NATS clusters#
Apply one NatsCluster in each Kubernetes cluster.
All three have the same gateway.remotes.
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: prod-east
namespace: nats-system
spec:
version: 2.15.0
replicas: 5
resources:
requests:
cpu: "6"
memory: 29Gi
limits:
memory: 29Gi
jetstream:
limits:
maxMemoryStore: 10Gi
volumeClaimTemplate:
spec:
storageClassName: gp3
resources:
requests:
storage: 500Gi
serverTags:
cluster: prod-east
region: us-east-2
auth:
trustRef:
name: acme
systemCredentials:
secretKeyRef:
name: cluster-controller-creds
gateway:
discovery: Explicit
remotes:
- name: prod-east
url: tls://nats.prod-east.acme.example:7222
- name: dev-east
url: tls://nats.dev-east.acme.example:7222
- name: prod-west
url: tls://nats.prod-west.acme.example:7222
tls:
certManager:
issuerRef:
kind: Issuer
name: nats-gateway-ca
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
external-dns.alpha.kubernetes.io/hostname: nats.prod-east.acme.example
advertise: nats.prod-east.acme.example:7222
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-prod-east.yamljetstream.limits.maxMemoryStore sets the size of the memory store.
Without it, the cluster controller derives the size from resources.limits.memory.
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: dev-east
namespace: nats-system
spec:
version: 2.15.0
replicas: 3
resources:
requests:
cpu: "1500m"
memory: 3Gi
limits:
memory: 3Gi
jetstream:
volumeClaimTemplate:
spec:
storageClassName: gp3
resources:
requests:
storage: 100Gi
serverTags:
cluster: dev-east
region: us-east-2
podTemplate:
spec:
nodeSelector:
topology.kubernetes.io/zone: us-east-2a
auth:
trustRef:
name: acme
systemCredentials:
secretKeyRef:
name: dev-east-cluster-controller-creds
gateway:
discovery: Explicit
remotes:
- name: prod-east
url: tls://nats.prod-east.acme.example:7222
- name: dev-east
url: tls://nats.dev-east.acme.example:7222
- name: prod-west
url: tls://nats.prod-west.acme.example:7222
tls:
certManager:
issuerRef:
kind: Issuer
name: nats-gateway-ca
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
external-dns.alpha.kubernetes.io/hostname: nats.dev-east.acme.example
advertise: nats.dev-east.acme.example:7222
kubectl --context dev-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-dev-east.yamlpodTemplate passes scheduling fields to the pods.
Here, a nodeSelector puts every server of dev-east in one zone.
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: prod-west
namespace: nats-system
spec:
version: 2.15.0
replicas: 3
resources:
requests:
cpu: "1500m"
memory: 3Gi
limits:
memory: 3Gi
jetstream:
volumeClaimTemplate:
spec:
storageClassName: gp3
resources:
requests:
storage: 100Gi
serverTags:
cluster: west
region: us-west-2
auth:
trustRef:
name: acme
systemCredentials:
secretKeyRef:
name: prod-west-cluster-controller-creds
gateway:
discovery: Explicit
remotes:
- name: prod-east
url: tls://nats.prod-east.acme.example:7222
- name: dev-east
url: tls://nats.dev-east.acme.example:7222
- name: prod-west
url: tls://nats.prod-west.acme.example:7222
tls:
certManager:
issuerRef:
kind: Issuer
name: nats-gateway-ca
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
external-dns.alpha.kubernetes.io/hostname: nats.prod-west.acme.example
advertise: nats.prod-west.acme.example:7222
kubectl --context prod-west apply -f https://nats-operator.io/docs/stories/09-acceptance/01-prod-west.yamlserverTags are yours to choose.
The placement tag of prod-west is cluster:west, which is not its name.
NatsClusterSpec in the API reference lists every field.
Check the gateways#
Read the status of prod-west:
kubectl --context prod-west -n nats-system get natscluster prod-west -o yamlThe status in the output is similar to this:
status:
observedGeneration: 1
conditions:
- type: Ready
status: "True"
reason: AllServersReady
- type: Settled
status: "True"
reason: AllGroupsCurrent
- type: GatewaysConnected
status: "True"
reason: AllMembersReachable
message: 2 of 2 remote members connected
gateways:
- name: dev-east
connected: true
inbound: 3
outbound: 3
- name: prod-east
connected: true
inbound: 5
outbound: 3
endpoints:
client: nats://prod-west.nats-system.svc:4222
gateway: nats.prod-west.acme.example:7222
GatewaysConnected is True once both of the other members are connected.
The status of every NatsCluster has an entry under gateways for each of the other members.
Check that the accounts reached every server#
Every account reaches every server of the supercluster through the resolver. Read the status of an account:
kubectl --context prod-east -n nats-system get natsaccount monitoring-prod -o yamlThe status in the output is similar to this:
status:
observedGeneration: 1
conditions:
- type: Ready
status: "True"
reason: Distributed
- type: Distributed
status: "True"
reason: AllServersCurrent
message: 11 of 11 servers hold this JWT
distribution:
servers: 11
current: 11
distribution counts the servers of all three NATS clusters.
Adopt the streams and balance them#
The application of the monitoring team created its streams REQUESTS and RESPONSES itself, before any NatsStream existed.
Apply the manifest in prod-east:
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
name: prod
namespace: monitoring
spec:
servers: ["nats://prod-east.nats-system.svc:4222"]
credentials:
secretKeyRef:
name: nats-service-creds
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
name: requests
namespace: monitoring
labels:
traffic: requests
spec:
connectionRef:
name: prod
adoptionPolicy: Adopt
name: REQUESTS
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
name: responses
namespace: monitoring
labels:
traffic: responses
spec:
connectionRef:
name: prod
adoptionPolicy: Adopt
name: RESPONSES
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsBalancer
metadata:
name: prod
namespace: monitoring
spec:
connectionRef:
name: prod
pools:
- name: requests
selector:
matchLabels:
traffic: requests
- name: responses
selector:
matchLabels:
traffic: responses
---
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
name: prod-east-sys
namespace: nats-system
spec:
servers: ["nats://prod-east.nats-system.svc:4222"]
credentials:
secretKeyRef:
name: jetstream-controller-creds
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsSystemBalancer
metadata:
name: prod-east
namespace: nats-system
spec:
connectionRef:
name: prod-east-sys
moves:
placement: true
leader: true
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/02-jetstream.yaml- The two
NatsStreams setadoptionPolicy: Adopt, so they adopt the streams and keep their config. See Adopting streams. - The
NatsBalancerbalances the account of the monitoring team in two pools, which select the streams by label. - The
NatsSystemBalancerbalances every account ofprod-east. The platform team needs one for each NATS cluster, and the manifest has only the one forprod-east.
Balance JetStream leaders and copies describes both balancers.
Read the status of an adopted stream:
kubectl --context prod-east -n monitoring get natsstream requests -o yamlThe status in the output is similar to this:
status:
observedGeneration: 2
conditions:
- type: Ready
status: "True"
reason: Synced
- type: Adopted
status: "True"
reason: FoundUnowned
message: adopted existing stream REQUESTS; spec written from the server
- type: Synced
status: "True"
reason: MatchesSpec
ownership:
origin: Adopted
server:
leader: prod-east-2
Read the status of the account balancer:
kubectl --context prod-east -n monitoring get natsbalancer prod -o yamlThe status in the output is similar to this:
status:
observedGeneration: 1
conditions:
- type: Ready
status: "True"
reason: Balancing
- type: Holding
status: "False"
reason: Settled
- type: Overlapping
status: "False"
reason: PoolsDisjoint
pools:
- name: requests
streams: 1
leaderSkew: 1
- name: responses
streams: 1
leaderSkew: 1
Each pool has one stream, so its leaderSkew is 1.
Every stream of the account has a resource in a pool, so the status has no default pool.
Read the status of the system balancer:
kubectl --context prod-east -n nats-system get natssystembalancer prod-east -o yamlThe status in the output is similar to this:
status:
observedGeneration: 1
conditions:
- type: Ready
status: "True"
reason: Balancing
- type: Holding
status: "False"
reason: Settled
capabilities:
placement: true
leader: None
leaderReason: 1 of 1 accounts carry no jetstream-stepdown export; their leaders are not moved
capabilities.leader is None.
monitoring-prod is the one account with streams in prod-east, and it has no jetstream-stepdown export, so the system balancer moves no leader.
To let it, add the export as in Let the system balancer move the leaders of an account.