Story 09 · Several NATS clusters

Deploy a production supercluster of three NATS clusters

Contents

This guide shows you how to deploy a supercluster for production: three NATS clusters in three Kubernetes clusters, under one NATS operator, with JetStream balanced. It combines the other guides, and each step links to the guide that has its options.

The manifests deploy these NATS clusters:

  • prod-east, five servers, in the home cluster.
  • dev-east, three servers, all in one zone.
  • prod-west, three servers, in another region.

Four services use the supercluster, and each service has its own account. The manifests have the accounts of the production environment. The accounts of the development environment are the same under -dev names, and are left out.

Before you begin#

You need:

  • Three Kubernetes clusters, with kubeconfig contexts named prod-east, dev-east and prod-west. Each manifest on this page is followed by the command that applies it with --context.
  • In each, the cluster controller and the JetStream controller, cert-manager, the namespace nats-system, the StorageClass gp3, and LoadBalancer Services that the other Kubernetes clusters can reach. Install shows how to install the controllers.
  • In each, the Issuer nats-gateway-ca in nats-system, from a private CA. See Create the gateway CA.
  • In prod-east, the auth controller and the namespace monitoring.
  • The DNS names nats.prod-east.acme.example, nats.dev-east.acme.example and nats.prod-west.acme.example pointing at the gateway Services, for example through external-dns.

Declare the NATS operator and the accounts#

Apply the NATS operator acme, its system account sys and the four accounts in the home cluster, prod-east:

01-auth.yaml yaml ·
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsOperator
metadata:
  name: acme
  namespace: nats-system
spec:
  systemAccountRef:
    name: sys
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsSystemAccount
metadata:
  name: sys
  namespace: nats-system
spec:
  operatorRef:
    name: acme
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
  name: checks-prod
  namespace: nats-system
spec:
  operatorRef:
    name: acme
  exports:
    - name: run
      type: Service
      subject: "checks.run.>"
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
  name: monitoring-prod
  namespace: nats-system
spec:
  operatorRef:
    name: acme
  limits:
    jetstream:
      memoryStorage: 36Gi
      diskStorage: 300Gi
  exports:
    - name: executions
      type: Stream
      subject: "monitoring.execution.>"
    - name: state-changes
      type: Stream
      subject: "monitoring.state.>"
    - name: execute
      type: Service
      subject: "monitoring.execute"
    - name: configure
      type: Service
      subject: "monitoring.configure"
  imports:
    - accountRef:
        kind: NatsAccount
        name: checks-prod
      export: run
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
  name: core-prod
  namespace: nats-system
spec:
  operatorRef:
    name: acme
  imports:
    - accountRef:
        kind: NatsAccount
        name: monitoring-prod
      export: state-changes
    - accountRef:
        kind: NatsAccount
        name: monitoring-prod
      export: configure
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsAccount
metadata:
  name: collector-prod
  namespace: nats-system
spec:
  operatorRef:
    name: acme
  limits:
    jetstream:
      diskStorage: 50Gi
  imports:
    - accountRef:
        kind: NatsAccount
        name: monitoring-prod
      export: executions
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-auth.yaml

The accounts form a chain. checks-prod exports a service to monitoring-prod, and monitoring-prod exports streams and services to core-prod and collector-prod. Share a stream and a service between accounts shows how to declare exports and imports.

Declare the users of the home cluster’s own controllers and the auth controller’s connection as well. Put a NATS cluster under a NATS operator shows them, with the Secrets cluster-controller-creds and jetstream-controller-creds that the manifests on this page refer to.

Declare the users#

Apply the users in prod-east:

01-users.yaml yaml ·
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: monitoring-prod-service
  namespace: monitoring
spec:
  accountRef:
    kind: NatsAccount
    namespace: nats-system
    name: monitoring-prod
  permissions:
    publish:
      allow: [">"]
    subscribe:
      allow: [">"]
  credentials:
    secretKeyRef:
      name: nats-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: monitoring-prod-readonly
  namespace: monitoring
spec:
  accountRef:
    kind: NatsAccount
    namespace: nats-system
    name: monitoring-prod
  preset: readonly
  credentials:
    secretKeyRef:
      name: nats-readonly-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: checks-prod-service
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: checks-prod
  permissions:
    publish:
      allow: [">"]
    subscribe:
      allow: [">"]
  credentials:
    secretKeyRef:
      name: checks-prod-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: checks-prod-readonly
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: checks-prod
  preset: readonly
  credentials:
    secretKeyRef:
      name: checks-prod-readonly-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: core-prod-service
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: core-prod
  permissions:
    publish:
      allow: [">"]
    subscribe:
      allow: [">"]
  credentials:
    secretKeyRef:
      name: core-prod-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: core-prod-readonly
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: core-prod
  preset: readonly
  credentials:
    secretKeyRef:
      name: core-prod-readonly-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: collector-prod-service
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: collector-prod
  permissions:
    publish:
      allow: [">"]
    subscribe:
      allow: [">"]
  credentials:
    secretKeyRef:
      name: collector-prod-service-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: collector-prod-readonly
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: collector-prod
  preset: readonly
  credentials:
    secretKeyRef:
      name: collector-prod-readonly-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: dev-east-cluster-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: cluster-controller
  credentials:
    secretKeyRef:
      name: dev-east-cluster-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: dev-east-jetstream-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: jetstream-controller
  credentials:
    secretKeyRef:
      name: dev-east-jetstream-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: prod-west-cluster-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: cluster-controller
  credentials:
    secretKeyRef:
      name: prod-west-cluster-controller-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: prod-west-jetstream-controller
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: jetstream-controller
  credentials:
    secretKeyRef:
      name: prod-west-jetstream-controller-creds
---
apiVersion: nats-operator.io/v1beta1
kind: NatsReferenceGrant
metadata:
  name: monitoring-users
  namespace: nats-system
spec:
  from:
    - group: auth.nats-operator.io
      kind: NatsUser
      namespace: monitoring
  to:
    - group: auth.nats-operator.io
      kind: NatsAccount
      name: monitoring-prod
    - group: auth.nats-operator.io
      kind: NatsAccount
      name: monitoring-dev
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-users.yaml
  • Every account has a service user and a readonly user. A readonly user sets the preset readonly, and a user with a preset cannot set permissions. readonly in the NATS permissions reference lists the subjects of the preset.
  • The users of the monitoring team are in the namespace monitoring, and the NatsReferenceGrant lets them refer to the team’s accounts in nats-system. See Let a team declare its own users and streams.
  • Each of the other two Kubernetes clusters has two users, one for its cluster controller and one for its JetStream controller.

The auth controller writes the creds Secrets in prod-east. Deliver the Secrets of the controllers’ users to nats-system in dev-east and prod-west, as Build a supercluster across Kubernetes clusters shows.

Copy the trust roots to every Kubernetes cluster#

Copy the NATS operator JWT and the system account JWT from the status of the NatsOperator into a NatsOperatorTrust, and apply the same one in all three Kubernetes clusters. Keep the copies alike with GitOps.

01-natsoperatortrust.yaml yaml ·
apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
  name: acme
  namespace: nats-system
spec:
  operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-natsoperatortrust.yaml
kubectl --context dev-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-natsoperatortrust.yaml
kubectl --context prod-west apply -f https://nats-operator.io/docs/stories/09-acceptance/01-natsoperatortrust.yaml

Deploy the NATS clusters#

Apply one NatsCluster in each Kubernetes cluster. All three have the same gateway.remotes.

01-prod-east.yaml yaml ·
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 5
  resources:
    requests:
      cpu: "6"
      memory: 29Gi
    limits:
      memory: 29Gi
  jetstream:
    limits:
      maxMemoryStore: 10Gi
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 500Gi
  serverTags:
    cluster: prod-east
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-east.acme.example
    advertise: nats.prod-east.acme.example:7222
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-prod-east.yaml

jetstream.limits.maxMemoryStore sets the size of the memory store. Without it, the cluster controller derives the size from resources.limits.memory.

01-dev-east.yaml yaml ·
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: dev-east
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "1500m"
      memory: 3Gi
    limits:
      memory: 3Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 100Gi
  serverTags:
    cluster: dev-east
    region: us-east-2
  podTemplate:
    spec:
      nodeSelector:
        topology.kubernetes.io/zone: us-east-2a
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: dev-east-cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.dev-east.acme.example
    advertise: nats.dev-east.acme.example:7222
kubectl --context dev-east apply -f https://nats-operator.io/docs/stories/09-acceptance/01-dev-east.yaml

podTemplate passes scheduling fields to the pods. Here, a nodeSelector puts every server of dev-east in one zone.

01-prod-west.yaml yaml ·
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-west
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: "1500m"
      memory: 3Gi
    limits:
      memory: 3Gi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 100Gi
  serverTags:
    cluster: west
    region: us-west-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: prod-west-cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-west.acme.example
    advertise: nats.prod-west.acme.example:7222
kubectl --context prod-west apply -f https://nats-operator.io/docs/stories/09-acceptance/01-prod-west.yaml

serverTags are yours to choose. The placement tag of prod-west is cluster:west, which is not its name.

NatsClusterSpec in the API reference lists every field.

Check the gateways#

Read the status of prod-west:

kubectl --context prod-west -n nats-system get natscluster prod-west -o yaml

The status in the output is similar to this:

01-status-natscluster-prod-west.yaml yaml ·
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: AllServersReady
    - type: Settled
      status: "True"
      reason: AllGroupsCurrent
    - type: GatewaysConnected
      status: "True"
      reason: AllMembersReachable
      message: 2 of 2 remote members connected
  gateways:
    - name: dev-east
      connected: true
      inbound: 3
      outbound: 3
    - name: prod-east
      connected: true
      inbound: 5
      outbound: 3
  endpoints:
    client: nats://prod-west.nats-system.svc:4222
    gateway: nats.prod-west.acme.example:7222

GatewaysConnected is True once both of the other members are connected. The status of every NatsCluster has an entry under gateways for each of the other members.

Check that the accounts reached every server#

Every account reaches every server of the supercluster through the resolver. Read the status of an account:

kubectl --context prod-east -n nats-system get natsaccount monitoring-prod -o yaml

The status in the output is similar to this:

01-status-natsaccount-monitoring-prod.yaml yaml ·
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Distributed
    - type: Distributed
      status: "True"
      reason: AllServersCurrent
      message: 11 of 11 servers hold this JWT
  distribution:
    servers: 11
    current: 11

distribution counts the servers of all three NATS clusters.

Adopt the streams and balance them#

The application of the monitoring team created its streams REQUESTS and RESPONSES itself, before any NatsStream existed. Apply the manifest in prod-east:

02-jetstream.yaml yaml ·
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: prod
  namespace: monitoring
spec:
  servers: ["nats://prod-east.nats-system.svc:4222"]
  credentials:
    secretKeyRef:
      name: nats-service-creds
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
  name: requests
  namespace: monitoring
  labels:
    traffic: requests
spec:
  connectionRef:
    name: prod
  adoptionPolicy: Adopt
  name: REQUESTS
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
  name: responses
  namespace: monitoring
  labels:
    traffic: responses
spec:
  connectionRef:
    name: prod
  adoptionPolicy: Adopt
  name: RESPONSES
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsBalancer
metadata:
  name: prod
  namespace: monitoring
spec:
  connectionRef:
    name: prod
  pools:
    - name: requests
      selector:
        matchLabels:
          traffic: requests
    - name: responses
      selector:
        matchLabels:
          traffic: responses
---
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: prod-east-sys
  namespace: nats-system
spec:
  servers: ["nats://prod-east.nats-system.svc:4222"]
  credentials:
    secretKeyRef:
      name: jetstream-controller-creds
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsSystemBalancer
metadata:
  name: prod-east
  namespace: nats-system
spec:
  connectionRef:
    name: prod-east-sys
  moves:
    placement: true
    leader: true
kubectl --context prod-east apply -f https://nats-operator.io/docs/stories/09-acceptance/02-jetstream.yaml
  • The two NatsStreams set adoptionPolicy: Adopt, so they adopt the streams and keep their config. See Adopting streams.
  • The NatsBalancer balances the account of the monitoring team in two pools, which select the streams by label.
  • The NatsSystemBalancer balances every account of prod-east. The platform team needs one for each NATS cluster, and the manifest has only the one for prod-east.

Balance JetStream leaders and copies describes both balancers.

Read the status of an adopted stream:

kubectl --context prod-east -n monitoring get natsstream requests -o yaml

The status in the output is similar to this:

02-status-natsstream-requests.yaml yaml ·
status:
  observedGeneration: 2
  conditions:
    - type: Ready
      status: "True"
      reason: Synced
    - type: Adopted
      status: "True"
      reason: FoundUnowned
      message: adopted existing stream REQUESTS; spec written from the server
    - type: Synced
      status: "True"
      reason: MatchesSpec
  ownership:
    origin: Adopted
  server:
    leader: prod-east-2

Read the status of the account balancer:

kubectl --context prod-east -n monitoring get natsbalancer prod -o yaml

The status in the output is similar to this:

02-status-natsbalancer.yaml yaml ·
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Balancing
    - type: Holding
      status: "False"
      reason: Settled
    - type: Overlapping
      status: "False"
      reason: PoolsDisjoint
  pools:
    - name: requests
      streams: 1
      leaderSkew: 1
    - name: responses
      streams: 1
      leaderSkew: 1

Each pool has one stream, so its leaderSkew is 1. Every stream of the account has a resource in a pool, so the status has no default pool.

Read the status of the system balancer:

kubectl --context prod-east -n nats-system get natssystembalancer prod-east -o yaml

The status in the output is similar to this:

02-status-natssystembalancer.yaml yaml ·
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Balancing
    - type: Holding
      status: "False"
      reason: Settled
  capabilities:
    placement: true
    leader: None
    leaderReason: 1 of 1 accounts carry no jetstream-stepdown export; their leaders are not moved

capabilities.leader is None. monitoring-prod is the one account with streams in prod-east, and it has no jetstream-stepdown export, so the system balancer moves no leader. To let it, add the export as in Let the system balancer move the leaders of an account.

Esc
↓ ↑ Navigate Enter Select Esc Close