Story 10 · Several NATS clusters

Join a NATS cluster to a hub as a leaf

Contents

This guide shows you how to join a NATS cluster at an edge site to a hub as a leaf. The manifests join the NATS cluster edge-site-1, in a Kubernetes cluster of its own, to the hub prod-east. Clients at the edge site publish telemetry that reaches the account telemetry of the hub, and a stream on the leaf accepts messages while the link to the hub is down.

A leaf is not a member of the supercluster of its hub. To add a member, see Add a NATS cluster to a supercluster that you did not deploy.

Before you begin#

You need:

  • Two Kubernetes clusters, with kubeconfig contexts named hub and edge. Each manifest on this page is followed by the command that applies it with --context.
  • In hub, the NATS cluster prod-east with its NATS operator acme and system account sys, from Deploy a production supercluster of three NATS clusters, and a NatsAccount named telemetry.
  • In hub, the ClusterIssuer letsencrypt, and the DNS name leaf.prod-east.acme.example pointing at the leafnode Service, for example through external-dns.
  • In edge, the cluster controller and the JetStream controller, and the namespace nats-system. Install shows how to install the controllers.

Open a leafnode listener on the hub#

Apply prod-east with leafnodes added, and the users that the leaves connect as.

01-hub.yaml yaml ·
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 5
  resources:
    requests:
      cpu: "6"
      memory: 29Gi
    limits:
      memory: 29Gi
  jetstream:
    limits:
      maxMemoryStore: 10Gi
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 500Gi
  serverTags:
    cluster: prod-east
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-east.acme.example
    advertise: nats.prod-east.acme.example:7222
  leafnodes:
    tls:
      certManager:
        issuerRef:
          kind: ClusterIssuer
          name: letsencrypt
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: leaf.prod-east.acme.example
    advertise: leaf.prod-east.acme.example:7422
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: edge-site-1
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: telemetry
  connectionTypes: [LEAFNODE]
  permissions:
    publish:
      allow: ["telemetry.>"]
    subscribe:
      allow: ["telemetry.>", "_INBOX.>"]
  credentials:
    secretKeyRef:
      name: edge-site-1-leaf-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: edge-site-2-system
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: leafnode
  credentials:
    secretKeyRef:
      name: edge-site-2-system-leaf-creds
kubectl --context hub apply -f https://nats-operator.io/docs/stories/10-leafnodes/01-hub.yaml
  • leafnodes.tls is optional.
  • edge-site-1 is a user of the account telemetry. It sets connectionTypes to LEAFNODE beside its permissions, so it can open only a leafnode connection.
  • edge-site-2-system is a user of the system account with the preset leafnode, which allows a leafnode connection on every subject of the account. Only a leaf that enforces the accounts of the hub needs it. See Enforce the accounts of the hub on a leaf.

Leafnodes in the API reference lists every field of the listener.

The auth controller writes the creds Secrets in hub. Deliver the Secret edge-site-1-leaf-creds to nats-system in edge, for example with External Secrets.

Deploy the leaf#

A leaf is a NatsCluster with leafRemotes. Each remote refers to a NatsConnection, which has the address of the hub and the creds to connect with. Apply the connection, the NatsCluster edge-site-1, and a stream on it:

01-edge.yaml yaml ·
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: hub
  namespace: nats-system
spec:
  servers: ["tls://leaf.prod-east.acme.example:7422"]
  credentials:
    secretKeyRef:
      name: edge-site-1-leaf-creds
---
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: edge-site-1
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      memory: 1Gi
  jetstream:
    domain: edge-site-1
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: 20Gi
  leafRemotes:
    - connectionRef:
        name: hub
---
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: local
  namespace: nats-system
spec:
  servers: ["nats://edge-site-1.nats-system.svc:4222"]
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
  name: telemetry-buffer
  namespace: nats-system
spec:
  connectionRef:
    name: local
  name: TELEMETRY_BUFFER
  subjects: ["telemetry.>"]
  replicas: 3
  maxAge: 72h
kubectl --context edge apply -f https://nats-operator.io/docs/stories/10-leafnodes/01-edge.yaml
  • A NatsCluster that has leafRemotes and JetStream must set jetstream.domain.
  • edge-site-1 has no auth plane, so it has one local account, the global account, and the remote binds that account. Every message that a local client publishes reaches the hub in the account of the remote’s creds.
  • edge-site-1 sets no auth.systemCredentials, so a change to its remotes restarts its servers.
  • The stream TELEMETRY_BUFFER is on the leaf, through the connection local, and accepts messages while the link is down. To drain it, give a stream in the account telemetry of the hub a source. This guide does not show that stream.

If the NatsConnection of a remote is in another namespace, that namespace needs a NatsReferenceGrant that admits the NatsCluster. The grant gives the creds of the connection to the namespace of the leaf, where the cluster controller copies them into the Secret <name>-leaf-remotes.

LeafRemote in the API reference lists every field of a remote.

Check the leaf#

Read the status of edge-site-1:

kubectl --context edge -n nats-system get natscluster edge-site-1 -o yaml

The status in the output is similar to this:

01-status-natscluster-edge-site-1.yaml yaml ·
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: AllServersReady
    - type: Settled
      status: "True"
      reason: AllGroupsCurrent
    - type: LeafnodesConnected
      status: "True"
      reason: AllRemotesConnected
      message: 3 of 3 servers connected to 1 remote
  leafRemotes:
    - connectionNamespace: nats-system
      connectionName: hub
      connected: 3
      account: ATELEMETRY...
  endpoints:
    client: nats://edge-site-1.nats-system.svc:4222

LeafnodesConnected is True once every server has a connection to every remote. In leafRemotes, connected counts the servers that are connected, and account is the public key of the hub account that the creds belong to.

Enforce the accounts of the hub on a leaf#

To have the clients of a leaf authenticate against the accounts of the hub, give the leaf the trust roots of the hub. The manifests deploy a second leaf, edge-site-2, in that form.

  1. Deliver the Secrets edge-site-2-system-leaf-creds and edge-site-2-leaf-creds to nats-system in edge. The first has the creds of the user edge-site-2-system, and the second the creds of a user of the account telemetry.
  2. Copy the NATS operator JWT and the system account JWT into the NatsOperatorTrust, as for a member of the supercluster.
  3. Copy the public key and the JWT of the account telemetry into the NatsAccountTrust.

Then apply the manifest:

01-edge-operator.yaml yaml ·
apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
  name: acme
  namespace: nats-system
spec:
  operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
---
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: hub-system
  namespace: nats-system
spec:
  servers: ["tls://leaf.prod-east.acme.example:7422"]
  credentials:
    secretKeyRef:
      name: edge-site-2-system-leaf-creds
---
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: hub-telemetry
  namespace: nats-system
spec:
  servers: ["tls://leaf.prod-east.acme.example:7422"]
  credentials:
    secretKeyRef:
      name: edge-site-2-leaf-creds
---
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: edge-site-2
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  jetstream:
    domain: edge-site-2
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: 20Gi
  auth:
    trustRef:
      name: acme
  leafRemotes:
    - connectionRef:
        name: hub-system
      localSystemAccount: true
    - connectionRef:
        name: hub-telemetry
      localAccountTrustRef:
        name: telemetry
---
apiVersion: nats-operator.io/v1beta1
kind: NatsAccountTrust
metadata:
  name: telemetry
  namespace: nats-system
spec:
  publicKey: ATELEMETRY...
  jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
kubectl --context edge apply -f https://nats-operator.io/docs/stories/10-leafnodes/01-edge-operator.yaml
  • The remote with localSystemAccount connects the system account of the leaf to the hub, as a user of the system account of the hub. Without that remote, the leaf cannot fetch an account that it has not cached.
  • The remote with localAccountTrustRef binds the account telemetry. The leaf preloads the JWT in the NatsAccountTrust, so the clients of the account can authenticate while the link is down.
  • The preloaded JWT is a copy. It expires after the jwtTTL of the account, 48h by default, unless the NatsAccount sets jwtTTL to 0.
  • The manifest sets no auth.resolver. A leaf that preloads an account runs the resolver Full on its JetStream volume, and a leaf that preloads none runs Cache.

In the home cluster, a NatsAccountTrust can set accountRef in place of publicKey and jwt, and the auth controller writes the JWT. You can set only one of accountRef and publicKey. Trust, resolver, gateways, leaf nodes in the design describes the resolver of a leaf.

Esc
↓ ↑ Navigate Enter Select Esc Close