# Join a NATS cluster to a hub as a leaf


This guide shows you how to join a NATS cluster at an edge site to a hub as a leaf.
The manifests join the NATS cluster `edge-site-1`, in a Kubernetes cluster of its own, to the hub `prod-east`.
Clients at the edge site publish telemetry that reaches the account `telemetry` of the hub, and a stream on the leaf accepts messages while the link to the hub is down.

A leaf is not a member of the supercluster of its hub.
To add a member, see [Add a NATS cluster to a supercluster that you did not deploy](https://nats-operator.io/docs/stories/13-join-supercluster/index.md).

## Before you begin

You need:

- Two Kubernetes clusters, with kubeconfig contexts named `hub` and `edge`.
  Each manifest on this page is followed by the command that applies it with `--context`.
- In `hub`, the NATS cluster `prod-east` with its NATS operator `acme` and system account `sys`, from [Deploy a production supercluster of three NATS clusters](https://nats-operator.io/docs/stories/09-acceptance/index.md), and a `NatsAccount` named `telemetry`.
- In `hub`, the ClusterIssuer `letsencrypt`, and the DNS name `leaf.prod-east.acme.example` pointing at the leafnode Service, for example through external-dns.
- In `edge`, the cluster controller and the JetStream controller, and the namespace `nats-system`.
  [Install](https://nats-operator.io/docs/install/index.md) shows how to install the controllers.

## Open a leafnode listener on the hub

Apply `prod-east` with `leafnodes` added, and the users that the leaves connect as.

`01-hub.yaml`

```yaml
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: prod-east
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 5
  resources:
    requests:
      cpu: "6"
      memory: 29Gi
    limits:
      memory: 29Gi
  jetstream:
    limits:
      maxMemoryStore: 10Gi
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        resources:
          requests:
            storage: 500Gi
  serverTags:
    cluster: prod-east
    region: us-east-2
  auth:
    trustRef:
      name: acme
    systemCredentials:
      secretKeyRef:
        name: cluster-controller-creds
  gateway:
    discovery: Explicit
    remotes:
      - name: prod-east
        url: tls://nats.prod-east.acme.example:7222
      - name: dev-east
        url: tls://nats.dev-east.acme.example:7222
      - name: prod-west
        url: tls://nats.prod-west.acme.example:7222
    tls:
      certManager:
        issuerRef:
          kind: Issuer
          name: nats-gateway-ca
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: nats.prod-east.acme.example
    advertise: nats.prod-east.acme.example:7222
  leafnodes:
    tls:
      certManager:
        issuerRef:
          kind: ClusterIssuer
          name: letsencrypt
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        external-dns.alpha.kubernetes.io/hostname: leaf.prod-east.acme.example
    advertise: leaf.prod-east.acme.example:7422
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: edge-site-1
  namespace: nats-system
spec:
  accountRef:
    kind: NatsAccount
    name: telemetry
  connectionTypes: [LEAFNODE]
  permissions:
    publish:
      allow: ["telemetry.>"]
    subscribe:
      allow: ["telemetry.>", "_INBOX.>"]
  credentials:
    secretKeyRef:
      name: edge-site-1-leaf-creds
---
apiVersion: auth.nats-operator.io/v1beta1
kind: NatsUser
metadata:
  name: edge-site-2-system
  namespace: nats-system
spec:
  accountRef:
    kind: NatsSystemAccount
    name: sys
  preset: leafnode
  credentials:
    secretKeyRef:
      name: edge-site-2-system-leaf-creds
```

```sh
kubectl --context hub apply -f https://nats-operator.io/docs/stories/10-leafnodes/01-hub.yaml
```


- `leafnodes.tls` is optional.
- `edge-site-1` is a user of the account `telemetry`.
  It sets `connectionTypes` to LEAFNODE beside its permissions, so it can open only a leafnode connection.
- `edge-site-2-system` is a user of the system account with the preset `leafnode`, which allows a leafnode connection on every subject of the account.
  Only a leaf that enforces the accounts of the hub needs it. See [Enforce the accounts of the hub on a leaf](#enforce-the-accounts-of-the-hub-on-a-leaf).

[Leafnodes](https://nats-operator.io/docs/reference/api/index.md#Leafnodes) in the API reference lists every field of the listener.

The auth controller writes the creds Secrets in `hub`.
Deliver the Secret `edge-site-1-leaf-creds` to `nats-system` in `edge`, for example with External Secrets.

## Deploy the leaf

A leaf is a `NatsCluster` with `leafRemotes`.
Each remote refers to a `NatsConnection`, which has the address of the hub and the creds to connect with.
Apply the connection, the `NatsCluster` `edge-site-1`, and a stream on it:

`01-edge.yaml`

```yaml
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: hub
  namespace: nats-system
spec:
  servers: ["tls://leaf.prod-east.acme.example:7422"]
  credentials:
    secretKeyRef:
      name: edge-site-1-leaf-creds
---
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: edge-site-1
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      memory: 1Gi
  jetstream:
    domain: edge-site-1
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: 20Gi
  leafRemotes:
    - connectionRef:
        name: hub
---
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: local
  namespace: nats-system
spec:
  servers: ["nats://edge-site-1.nats-system.svc:4222"]
---
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
  name: telemetry-buffer
  namespace: nats-system
spec:
  connectionRef:
    name: local
  name: TELEMETRY_BUFFER
  subjects: ["telemetry.>"]
  replicas: 3
  maxAge: 72h
```

```sh
kubectl --context edge apply -f https://nats-operator.io/docs/stories/10-leafnodes/01-edge.yaml
```


- A `NatsCluster` that has `leafRemotes` and JetStream must set `jetstream.domain`.
- `edge-site-1` has no auth plane, so it has one local account, the global account, and the remote binds that account.
  Every message that a local client publishes reaches the hub in the account of the remote's creds.
- `edge-site-1` sets no `auth.systemCredentials`, so a change to its remotes restarts its servers.
- The stream `TELEMETRY_BUFFER` is on the leaf, through the connection `local`, and accepts messages while the link is down.
  To drain it, give a stream in the account `telemetry` of the hub a source. This guide does not show that stream.

If the `NatsConnection` of a remote is in another namespace, that namespace needs a `NatsReferenceGrant` that admits the `NatsCluster`.
The grant gives the creds of the connection to the namespace of the leaf, where the cluster controller copies them into the Secret `<name>-leaf-remotes`.

[LeafRemote](https://nats-operator.io/docs/reference/api/index.md#LeafRemote) in the API reference lists every field of a remote.

## Check the leaf

Read the status of `edge-site-1`:

```sh
kubectl --context edge -n nats-system get natscluster edge-site-1 -o yaml
```

The `status` in the output is similar to this:

`01-status-natscluster-edge-site-1.yaml`

```yaml
status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      reason: AllServersReady
    - type: Settled
      status: "True"
      reason: AllGroupsCurrent
    - type: LeafnodesConnected
      status: "True"
      reason: AllRemotesConnected
      message: 3 of 3 servers connected to 1 remote
  leafRemotes:
    - connectionNamespace: nats-system
      connectionName: hub
      connected: 3
      account: ATELEMETRY...
  endpoints:
    client: nats://edge-site-1.nats-system.svc:4222
```


`LeafnodesConnected` is True once every server has a connection to every remote.
In `leafRemotes`, `connected` counts the servers that are connected, and `account` is the public key of the hub account that the creds belong to.

## Enforce the accounts of the hub on a leaf

To have the clients of a leaf authenticate against the accounts of the hub, give the leaf the trust roots of the hub.
The manifests deploy a second leaf, `edge-site-2`, in that form.

1. Deliver the Secrets `edge-site-2-system-leaf-creds` and `edge-site-2-leaf-creds` to `nats-system` in `edge`.
   The first has the creds of the user `edge-site-2-system`, and the second the creds of a user of the account `telemetry`.
1. Copy the NATS operator JWT and the system account JWT into the `NatsOperatorTrust`, as for a member of the supercluster.
1. Copy the public key and the JWT of the account `telemetry` into the `NatsAccountTrust`.

Then apply the manifest:

`01-edge-operator.yaml`

```yaml
apiVersion: nats-operator.io/v1beta1
kind: NatsOperatorTrust
metadata:
  name: acme
  namespace: nats-system
spec:
  operatorJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
  systemAccountJWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
---
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: hub-system
  namespace: nats-system
spec:
  servers: ["tls://leaf.prod-east.acme.example:7422"]
  credentials:
    secretKeyRef:
      name: edge-site-2-system-leaf-creds
---
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: hub-telemetry
  namespace: nats-system
spec:
  servers: ["tls://leaf.prod-east.acme.example:7422"]
  credentials:
    secretKeyRef:
      name: edge-site-2-leaf-creds
---
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: edge-site-2
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  jetstream:
    domain: edge-site-2
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: 20Gi
  auth:
    trustRef:
      name: acme
  leafRemotes:
    - connectionRef:
        name: hub-system
      localSystemAccount: true
    - connectionRef:
        name: hub-telemetry
      localAccountTrustRef:
        name: telemetry
---
apiVersion: nats-operator.io/v1beta1
kind: NatsAccountTrust
metadata:
  name: telemetry
  namespace: nats-system
spec:
  publicKey: ATELEMETRY...
  jwt: eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ...
```

```sh
kubectl --context edge apply -f https://nats-operator.io/docs/stories/10-leafnodes/01-edge-operator.yaml
```


- The remote with `localSystemAccount` connects the system account of the leaf to the hub, as a user of the system account of the hub.
  Without that remote, the leaf cannot fetch an account that it has not cached.
- The remote with `localAccountTrustRef` binds the account `telemetry`.
  The leaf preloads the JWT in the `NatsAccountTrust`, so the clients of the account can authenticate while the link is down.
- The preloaded JWT is a copy.
  It expires after the `jwtTTL` of the account, 48h by default, unless the `NatsAccount` sets `jwtTTL` to 0.
- The manifest sets no `auth.resolver`.
  A leaf that preloads an account runs the resolver `Full` on its JetStream volume, and a leaf that preloads none runs `Cache`.

In the home cluster, a `NatsAccountTrust` can set `accountRef` in place of `publicKey` and `jwt`, and the auth controller writes the JWT.
You can set only one of `accountRef` and `publicKey`.
[Trust, resolver, gateways, leaf nodes](https://nats-operator.io/docs/design/v1/index.md#43-trust-resolver-gateways-leaf-nodes) in the design describes the resolver of a leaf.

