Story 10 · Several NATS clusters
Join a NATS cluster to a hub as a leaf
Contents
This guide shows you how to join a NATS cluster at an edge site to a hub as a leaf.
The manifests join the NATS cluster edge-site-1, in a Kubernetes cluster of its own, to the hub prod-east.
Clients at the edge site publish telemetry that reaches the account telemetry of the hub, and a stream on the leaf accepts messages while the link to the hub is down.
A leaf is not a member of the supercluster of its hub. To add a member, see Add a NATS cluster to a supercluster that you did not deploy.
Before you begin#
You need:
- Two Kubernetes clusters, with kubeconfig contexts named
hubandedge. Each manifest on this page is followed by the command that applies it with--context. - In
hub, the NATS clusterprod-eastwith its NATS operatoracmeand system accountsys, from Deploy a production supercluster of three NATS clusters, and aNatsAccountnamedtelemetry. - In
hub, the ClusterIssuerletsencrypt, and the DNS nameleaf.prod-east.acme.examplepointing at the leafnode Service, for example through external-dns. - In
edge, the cluster controller and the JetStream controller, and the namespacenats-system. Install shows how to install the controllers.
Open a leafnode listener on the hub#
Apply prod-east with leafnodes added, and the users that the leaves connect as.
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: prod-east
namespace: nats-system
spec:
version: 2.15.0
replicas: 5
resources:
requests:
cpu: "6"
memory: 29Gi
limits:
memory: 29Gi
jetstream:
limits:
maxMemoryStore: 10Gi
volumeClaimTemplate:
spec:
storageClassName: gp3
resources:
requests:
storage: 500Gi
serverTags:
cluster: prod-east
region: us-east-2
auth:
trustRef:
name: acme
systemCredentials:
secretKeyRef:
name: cluster-controller-creds
gateway:
discovery: Explicit
remotes:
- name: prod-east
url: tls://nats.prod-east.acme.example:7222
- name: dev-east
url: tls://nats.dev-east.acme.example:7222
- name: prod-west
url: tls://nats.prod-west.acme.example:7222
tls:
certManager:
issuerRef:
kind: Issuer
name: nats-gateway-ca
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
external-dns.alpha.kubernetes.io/hostname: nats.prod-east.acme.example
advertise: nats.prod-east.acme.example:7222
leafnodes:
tls:
certManager:
issuerRef:
kind: ClusterIssuer
name: letsencrypt
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
external-dns.alpha.kubernetes.io/hostname: leaf.prod-east.acme.example
advertise: leaf.prod-east.acme.example:7422
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: edge-site-1
namespace: nats-system
spec:
accountRef:
kind: NatsAccount
name: telemetry
connectionTypes: [LEAFNODE]
permissions:
publish:
allow: ["telemetry.>"]
subscribe:
allow: ["telemetry.>", "_INBOX.>"]
credentials:
secretKeyRef:
name: edge-site-1-leaf-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
name: edge-site-2-system
namespace: nats-system
spec:
accountRef:
kind: NatsSystemAccount
name: sys
preset: leafnode
credentials:
secretKeyRef:
name: edge-site-2-system-leaf-creds
kubectl --context hub apply -f https://nats-operator.io/docs/stories/10-leafnodes/01-hub.yamlleafnodes.tlsis optional.edge-site-1is a user of the accounttelemetry. It setsconnectionTypesto LEAFNODE beside its permissions, so it can open only a leafnode connection.edge-site-2-systemis a user of the system account with the presetleafnode, which allows a leafnode connection on every subject of the account. Only a leaf that enforces the accounts of the hub needs it. See Enforce the accounts of the hub on a leaf.
Leafnodes in the API reference lists every field of the listener.
The auth controller writes the creds Secrets in hub.
Deliver the Secret edge-site-1-leaf-creds to nats-system in edge, for example with External Secrets.
Deploy the leaf#
A leaf is a NatsCluster with leafRemotes.
Each remote refers to a NatsConnection, which has the address of the hub and the creds to connect with.
Apply the connection, the NatsCluster edge-site-1, and a stream on it:
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
name: hub
namespace: nats-system
spec:
servers: ["tls://leaf.prod-east.acme.example:7422"]
credentials:
secretKeyRef:
name: edge-site-1-leaf-creds
---
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: edge-site-1
namespace: nats-system
spec:
version: 2.15.0
replicas: 3
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
memory: 1Gi
jetstream:
domain: edge-site-1
volumeClaimTemplate:
spec:
resources:
requests:
storage: 20Gi
leafRemotes:
- connectionRef:
name: hub
---
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
name: local
namespace: nats-system
spec:
servers: ["nats://edge-site-1.nats-system.svc:4222"]
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
name: telemetry-buffer
namespace: nats-system
spec:
connectionRef:
name: local
name: TELEMETRY_BUFFER
subjects: ["telemetry.>"]
replicas: 3
maxAge: 72h
kubectl --context edge apply -f https://nats-operator.io/docs/stories/10-leafnodes/01-edge.yaml- A
NatsClusterthat hasleafRemotesand JetStream must setjetstream.domain. edge-site-1has no auth plane, so it has one local account, the global account, and the remote binds that account. Every message that a local client publishes reaches the hub in the account of the remote’s creds.edge-site-1sets noauth.systemCredentials, so a change to its remotes restarts its servers.- The stream
TELEMETRY_BUFFERis on the leaf, through the connectionlocal, and accepts messages while the link is down. To drain it, give a stream in the accounttelemetryof the hub a source. This guide does not show that stream.
If the NatsConnection of a remote is in another namespace, that namespace needs a NatsReferenceGrant that admits the NatsCluster.
The grant gives the creds of the connection to the namespace of the leaf, where the cluster controller copies them into the Secret <name>-leaf-remotes.
LeafRemote in the API reference lists every field of a remote.
Check the leaf#
Read the status of edge-site-1:
kubectl --context edge -n nats-system get natscluster edge-site-1 -o yamlThe status in the output is similar to this:
status:
observedGeneration: 1
conditions:
- type: Ready
status: "True"
reason: AllServersReady
- type: Settled
status: "True"
reason: AllGroupsCurrent
- type: LeafnodesConnected
status: "True"
reason: AllRemotesConnected
message: 3 of 3 servers connected to 1 remote
leafRemotes:
- connectionNamespace: nats-system
connectionName: hub
connected: 3
account: ATELEMETRY...
endpoints:
client: nats://edge-site-1.nats-system.svc:4222
LeafnodesConnected is True once every server has a connection to every remote.
In leafRemotes, connected counts the servers that are connected, and account is the public key of the hub account that the creds belong to.
Enforce the accounts of the hub on a leaf#
To have the clients of a leaf authenticate against the accounts of the hub, give the leaf the trust roots of the hub.
The manifests deploy a second leaf, edge-site-2, in that form.
- Deliver the Secrets
edge-site-2-system-leaf-credsandedge-site-2-leaf-credstonats-systeminedge. The first has the creds of the useredge-site-2-system, and the second the creds of a user of the accounttelemetry. - Copy the NATS operator JWT and the system account JWT into the
NatsOperatorTrust, as for a member of the supercluster. - Copy the public key and the JWT of the account
telemetryinto theNatsAccountTrust.
Then apply the manifest:
apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
name: acme
namespace: nats-system
spec:
operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
---
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
name: hub-system
namespace: nats-system
spec:
servers: ["tls://leaf.prod-east.acme.example:7422"]
credentials:
secretKeyRef:
name: edge-site-2-system-leaf-creds
---
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
name: hub-telemetry
namespace: nats-system
spec:
servers: ["tls://leaf.prod-east.acme.example:7422"]
credentials:
secretKeyRef:
name: edge-site-2-leaf-creds
---
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: edge-site-2
namespace: nats-system
spec:
version: 2.15.0
replicas: 3
jetstream:
domain: edge-site-2
volumeClaimTemplate:
spec:
resources:
requests:
storage: 20Gi
auth:
trustRef:
name: acme
leafRemotes:
- connectionRef:
name: hub-system
localSystemAccount: true
- connectionRef:
name: hub-telemetry
localAccountTrustRef:
name: telemetry
---
apiVersion: nats-operator.io/v1beta1
kind: NatsAccountTrust
metadata:
name: telemetry
namespace: nats-system
spec:
publicKey: ATELEMETRY...
jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
kubectl --context edge apply -f https://nats-operator.io/docs/stories/10-leafnodes/01-edge-operator.yaml- The remote with
localSystemAccountconnects the system account of the leaf to the hub, as a user of the system account of the hub. Without that remote, the leaf cannot fetch an account that it has not cached. - The remote with
localAccountTrustRefbinds the accounttelemetry. The leaf preloads the JWT in theNatsAccountTrust, so the clients of the account can authenticate while the link is down. - The preloaded JWT is a copy.
It expires after the
jwtTTLof the account, 48h by default, unless theNatsAccountsetsjwtTTLto 0. - The manifest sets no
auth.resolver. A leaf that preloads an account runs the resolverFullon its JetStream volume, and a leaf that preloads none runsCache.
In the home cluster, a NatsAccountTrust can set accountRef in place of publicKey and jwt, and the auth controller writes the JWT.
You can set only one of accountRef and publicKey.
Trust, resolver, gateways, leaf nodes in the design describes the resolver of a leaf.