An edge site runs a small NATS cluster in its own Kubernetes cluster. It joins the production hub as a leaf: local clients publish telemetry that reaches the hub’s telemetry account, and a local stream keeps accepting while the link is down. The leaf is not a supercluster member and has no auth plane of its own.

The hub #

The hub is story 9’s prod-east, whose NatsCluster gains a leafnode listener, and the edge site gets a user in the account its traffic belongs to, usable only as a leaf.

01-hub.yaml

# In the hub's Kubernetes cluster: prod-east as story 9 declares it, with a
# leafnode listener under `leafnodes`.
apiVersion: cluster.nats.mikluko.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 5
  resources:
    requests:
      cpu: "6"
      memory: 29Gi
    limits:
      memory: 29Gi
  jetstream:
    # Overrides the memory store derived from limits.memory.
    limits:
      maxMemoryStore: 10Gi
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 500Gi
  serverTags:
    cluster: prod-east
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-east.acme.example
    advertise: nats.prod-east.acme.example:7222
  leafnodes:
    # Optional, as on every listener.
    tls:
      certManager:
        issuerRef:
          kind: ClusterIssuer
          name: letsencrypt
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: leaf.prod-east.acme.example
    advertise: leaf.prod-east.acme.example:7422
---
# What the edge site authenticates as: an ordinary user in the account its
# traffic lands in, usable only for a leafnode connection. Its Secret is
# carried to the edge by External Secrets, as the remote controllers' are.
apiVersion: auth.nats.mikluko.io/v1beta1
kind: NatsUser
metadata:
  name: edge-site-1
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: telemetry
  # A leaf user narrowed to some of the account's subjects names the connection
  # type beside its permissions; `preset: leafnode` alone is a leaf connection
  # with full account access.
  connectionTypes: [LEAFNODE]
  permissions:
    publish:
      allow: ["telemetry.>"]
    subscribe:
      allow: ["telemetry.>", "_INBOX.>"]
  credentials:
    secretKeyRef:
      name: edge-site-1-leaf-creds
---
# For a leaf under a NATS operator (01-edge-operator.yaml): the system-account
# leaf user its account lookups travel over.
apiVersion: auth.nats.mikluko.io/v1beta1
kind: NatsUser
metadata:
  name: edge-site-2-system
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: leafnode
  credentials:
    secretKeyRef:
      name: edge-site-2-system-leaf-creds

kubectl --context hub apply -f https://nats-operator.io/docs/stories/10-leafnodes/01-hub.yaml

The leaf #

A leaf is a NatsCluster that dials out through a NatsConnection, the same kind the JetStream controller uses. Its JetStream runs in a domain of its own. A NatsConnection in another namespace needs a NatsReferenceGrant there, and that grant hands the connection’s credentials to the leaf’s namespace, where the cluster controller copies them into the Secret <name>-leaf-remotes.

01-edge.yaml

# In the edge site's own Kubernetes cluster. No auth controller, no NATS
# operator: the leaf runs with no accounts, and everything its local clients
# publish reaches the hub in the account the remote's credentials sign into.
apiVersion: nats.mikluko.io/v1beta1
kind: NatsConnection
metadata:
  name: hub
  namespace: nats-system
spec:
  servers: ["tls://leaf.prod-east.acme.example:7422"]
  credentials:
    secretKeyRef:
      name: edge-site-1-leaf-creds
---
apiVersion: cluster.nats.mikluko.io/v1beta1
kind: NatsCluster
metadata:
  name: edge-site-1
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      memory: 1Gi
  jetstream:
    # Required with leafRemotes.
    domain: edge-site-1
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: 20Gi
  # A leaf is a NatsCluster that dials out. This one sets no
  # auth.systemCredentials, so any change to its remotes restarts its servers.
  leafRemotes:
    - connectionRef:
        name: hub
      # With no auth plane on the leaf there is one local account, the
      # global one, and the remote binds it.
---
# A local buffer that keeps accepting while the link is down, and a source
# on the hub side is how it drains; that part is ordinary JetStream config
# in the hub's telemetry account, not shown.
apiVersion: nats.mikluko.io/v1beta1
kind: NatsConnection
metadata:
  name: local
  namespace: nats-system
spec:
  servers: ["nats://edge-site-1.nats-system.svc:4222"]
---
apiVersion: jetstream.nats.mikluko.io/v1beta1
kind: NatsStream
metadata:
  name: telemetry-buffer
  namespace: nats-system
spec:
  connectionRef:
    name: local
  name: TELEMETRY_BUFFER
  subjects: ["telemetry.>"]
  replicas: 3
  maxAge: 72h

kubectl --context edge apply -f https://nats-operator.io/docs/stories/10-leafnodes/01-edge.yaml

01-status-natscluster-edge-site-1.yaml

status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: AllServersReady
    - type: Settled
      status: "True"
      reason: AllGroupsCurrent
    - type: LeafnodesConnected
      status: "True"
      reason: AllRemotesConnected
      message: 3 of 3 servers connected to 1 remote
  leafRemotes:
    - connectionNamespace: nats-system
      connectionName: hub
      connected: 3
      account: ATELEMETRY...   # the hub account the credentials sign into
  endpoints:
    client: nats://edge-site-1.nats-system.svc:4222

A leaf that enforces the hub’s accounts #

A second site trusts the hub’s NATS operator, so its clients authenticate against the hub’s accounts locally. It reads the same trust roots the supercluster members do, and resolves accounts over a second remote bound to the system account; without that remote it cannot fetch an account it has not cached.

01-edge-operator.yaml

# A second edge site that enforces the hub's accounts locally: it trusts the
# hub's NATS operator, and its clients authenticate against the same accounts
# as the hub's. Still not a supercluster member.
#
# The same NatsOperatorTrust the supercluster members read, copied here by
# GitOps.
apiVersion: nats.mikluko.io/v1beta1
kind: NatsOperatorTrust
metadata:
  name: acme
  namespace: nats-system
spec:
  operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
---
# The creds are a hub user in the system account.
apiVersion: nats.mikluko.io/v1beta1
kind: NatsConnection
metadata:
  name: hub-system
  namespace: nats-system
spec:
  servers: ["tls://leaf.prod-east.acme.example:7422"]
  credentials:
    secretKeyRef:
      name: edge-site-2-system-leaf-creds
---
apiVersion: nats.mikluko.io/v1beta1
kind: NatsConnection
metadata:
  name: hub-telemetry
  namespace: nats-system
spec:
  servers: ["tls://leaf.prod-east.acme.example:7422"]
  credentials:
    secretKeyRef:
      name: edge-site-2-leaf-creds
---
apiVersion: cluster.nats.mikluko.io/v1beta1
kind: NatsCluster
metadata:
  name: edge-site-2
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  jetstream:
    domain: edge-site-2
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: 20Gi
  auth:
    trustRef:
      name: acme
    # resolver is omitted: a leaf that preloads accounts runs Full, on the
    # JetStream volume; one that preloads nothing runs Cache. Memory is not
    # offered.
  leafRemotes:
    - connectionRef:
        name: hub-system
      # The leaf's system account, known from trustRef.
      localSystemAccount: true
    - connectionRef:
        name: hub-telemetry
      localAccountTrustRef:
        name: telemetry
---
# The account the telemetry remote binds to. In the home Kubernetes cluster
# this would be `accountRef: {name: telemetry}`, resolved from the live
# NatsAccount; here, with no auth controller, the literal form, copied by
# GitOps. `accountRef` and `publicKey` are exclusive.
apiVersion: nats.mikluko.io/v1beta1
kind: NatsAccountTrust
metadata:
  name: telemetry
  namespace: nats-system
spec:
  publicKey: ATELEMETRY...
  # Preloaded, so the account's clients authenticate while the leaf link is
  # down or before it first connects. It expires with the account's jwtTTL
  # (48h by default) unless the NatsAccount sets `jwtTTL: 0`.
  jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...

kubectl --context edge apply -f https://nats-operator.io/docs/stories/10-leafnodes/01-edge-operator.yaml