# Scrape the controllers' metrics over verified TLS


This guide shows you how to have Prometheus scrape the metrics of the controllers and verify the certificate of every scrape.
It also shows you how to limit the controllers to connections to the API server, DNS and the NATS clusters in `nats-system`.

## Before you begin

You need:

- The chart installed as the release `nats-operator` in the namespace `nats-operator`.
  [Install](https://nats-operator.io/docs/install/index.md) shows how.
- Prometheus, run by prometheus-operator in the namespace `monitoring` under the ServiceAccount `prometheus`.
- cert-manager, and a CA ClusterIssuer named `ca-issuer`.
- The namespaces `nats-system` and `nats-outside`, and a StorageClass named `standard`.

## Issue the certificate

Every controller serves its metrics with the certificate in one Secret in the namespace of the release.
The certificate must have the DNS name of the metrics Service of every controller that is enabled.
Apply a `Certificate` from `ca-issuer`, which also writes its CA to `ca.crt` in the Secret:

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nats-operator-metrics
  namespace: nats-operator
spec:
  secretName: nats-operator-metrics-tls
  issuerRef:
    kind: ClusterIssuer
    name: ca-issuer
  dnsNames:
    - nats-operator-cluster-controller-metrics.nats-operator.svc
    - nats-operator-auth-controller-metrics.nats-operator.svc
    - nats-operator-jetstream-controller-metrics.nats-operator.svc
```

## Set the chart values

Upgrade the release with these values:

```yaml
metrics:
  service:
    enabled: true
  tls:
    secretName: nats-operator-metrics-tls
  scraper:
    serviceAccount: monitoring/prometheus
  serviceMonitor:
    enabled: true
networkPolicy:
  enabled: true
  from:
    - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: monitoring}}
  egress:
    - to: [{ipBlock: {cidr: 10.96.0.1/32}}]
      ports: [{protocol: TCP, port: 443}]
    - to: [{ipBlock: {cidr: 172.18.0.2/32}}]
      ports: [{protocol: TCP, port: 6443}]
    - to:
        - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}
          podSelector: {matchLabels: {k8s-app: kube-dns}}
      ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
    - to:
        - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: nats-system}}
      ports: [{protocol: TCP, port: 4222}, {protocol: TCP, port: 8222}]
```

The first two egress rules are for the API server, and both addresses are examples.
Replace them with the addresses of your Kubernetes cluster:

1. For the first rule, read the address of the Service `kubernetes`:

   ```sh
   kubectl -n default get service kubernetes
   ```

1. For the second rule, read the addresses of its endpoints:

   ```sh
   kubectl -n default get endpointslices -l kubernetes.io/service-name=kubernetes
   ```

The last rule is for the NATS clusters in `nats-system`.
Port 4222 is the client port, which a `NatsConnection` has in `spec.servers`, and port 8222 is the monitoring port of the servers, which the cluster controller reads.

With these values, each ServiceMonitor scrapes with the token of the Prometheus pod, and `metrics.scraper.serviceAccount` allows that ServiceAccount to read `/metrics`.
The ServiceMonitor verifies the certificate against `ca.crt` in the Secret, under the DNS name of the Service, such as `nats-operator-cluster-controller-metrics.nats-operator.svc`.
[Metrics](https://nats-operator.io/docs/reference/chart/index.md#metrics) and [Network policy](https://nats-operator.io/docs/reference/chart/index.md#network-policy) in the chart reference describe the other values for metrics and the network policy.

## Check that the controllers work under the policy

Apply a NATS cluster in `nats-system`:

`01-natscluster.yaml`

```yaml
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: demo
  namespace: nats-system
spec:
  version: 2.15.0
  replicas: 3
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      memory: 256Mi
  jetstream:
    volumeClaimTemplate:
      spec:
        storageClassName: standard
        resources:
          requests:
            storage: 1Gi
```

```sh
kubectl apply -f https://nats-operator.io/docs/stories/12-metrics/01-natscluster.yaml
```


Read its status:

```sh
kubectl -n nats-system get natscluster demo -o yaml
```

The `status` in the output is similar to this:

`01-status-natscluster-demo.yaml`

```yaml
status:
  replicas: 3
  readyReplicas: 3
  conditions:
    - type: Ready
      status: "True"
      reason: AllServersReady
```


Apply a connection to the NATS cluster, and a stream on it:

`02-natsconnection.yaml`

```yaml
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: demo
  namespace: nats-system
spec:
  servers: ["nats://demo.nats-system.svc:4222"]
```

```sh
kubectl apply -f https://nats-operator.io/docs/stories/12-metrics/02-natsconnection.yaml
```


`02-natsstream.yaml`

```yaml
apiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
  name: events
  namespace: nats-system
spec:
  connectionRef:
    name: demo
  name: EVENTS
  subjects: ["events.>"]
  storage: File
  replicas: 3
```

```sh
kubectl apply -f https://nats-operator.io/docs/stories/12-metrics/02-natsstream.yaml
```


Read the status of the stream:

```sh
kubectl -n nats-system get natsstream events -o yaml
```

The `status` in the output is similar to this:

`02-status-natsstream.yaml`

```yaml
status:
  conditions:
    - type: Ready
      status: "True"
      reason: Synced
    - type: Synced
      status: "True"
      reason: MatchesSpec
```


## Check that the policy blocks other destinations

No egress rule allows the namespace `nats-outside`.
Apply a NATS cluster there:

`01-natscluster-outside.yaml`

```yaml
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
  name: outside
  namespace: nats-outside
spec:
  version: 2.15.0
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      memory: 128Mi
```

```sh
kubectl apply -f https://nats-operator.io/docs/stories/12-metrics/01-natscluster-outside.yaml
```


Read its status:

```sh
kubectl -n nats-outside get natscluster outside -o yaml
```

The `status` in the output is similar to this:

`01-status-natscluster-outside.yaml`

```yaml
status:
  replicas: 1
  readyReplicas: 1
  conditions:
    - type: Ready
      status: "True"
      reason: AllServersReady
```


The NATS cluster is Ready, because the cluster controller deploys it through the API server.

Apply a `NatsConnection` to it:

`02-natsconnection-outside.yaml`

```yaml
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
  name: outside
  namespace: nats-system
spec:
  servers: ["nats://outside.nats-outside.svc:4222"]
```

```sh
kubectl apply -f https://nats-operator.io/docs/stories/12-metrics/02-natsconnection-outside.yaml
```


Read its status:

```sh
kubectl -n nats-system get natsconnection outside -o yaml
```

The `status` in the output is similar to this:

`02-status-natsconnection-outside.yaml`

```yaml
status:
  conditions:
    - type: Ready
      status: "False"
      reason: ConnectFailed
```


`Ready` stays False with the reason `ConnectFailed`, because the policy drops the connection from the JetStream controller.

## Scrape by hand

To scrape the cluster controller as its ServiceMonitor does:

1. Write the CA to a file:

   ```sh
   kubectl -n nats-operator get secret nats-operator-metrics-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
   ```

1. Forward the port of the metrics Service to your machine:

   ```sh
   kubectl -n nats-operator port-forward svc/nats-operator-cluster-controller-metrics 8080 &
   ```

1. Scrape with a token of the ServiceAccount `prometheus`:

   ```sh
   curl --cacert ca.crt \
     --resolve nats-operator-cluster-controller-metrics.nats-operator.svc:8080:127.0.0.1 \
     -H "Authorization: Bearer $(kubectl -n monitoring create token prometheus)" \
     https://nats-operator-cluster-controller-metrics.nats-operator.svc:8080/metrics
   ```

