Story 12 · Metrics
Scrape the controllers' metrics over verified TLS
Contents
This guide shows you how to have Prometheus scrape the metrics of the controllers and verify the certificate of every scrape.
It also shows you how to limit the controllers to connections to the API server, DNS and the NATS clusters in nats-system.
Before you begin#
You need:
- The chart installed as the release
nats-operatorin the namespacenats-operator. Install shows how. - Prometheus, run by prometheus-operator in the namespace
monitoringunder the ServiceAccountprometheus. - cert-manager, and a CA ClusterIssuer named
ca-issuer. - The namespaces
nats-systemandnats-outside, and a StorageClass namedstandard.
Issue the certificate#
Every controller serves its metrics with the certificate in one Secret in the namespace of the release.
The certificate must have the DNS name of the metrics Service of every controller that is enabled.
Apply a Certificate from ca-issuer, which also writes its CA to ca.crt in the Secret:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: nats-operator-metrics
namespace: nats-operator
spec:
secretName: nats-operator-metrics-tls
issuerRef:
kind: ClusterIssuer
name: ca-issuer
dnsNames:
- nats-operator-cluster-controller-metrics.nats-operator.svc
- nats-operator-auth-controller-metrics.nats-operator.svc
- nats-operator-jetstream-controller-metrics.nats-operator.svcSet the chart values#
Upgrade the release with these values:
metrics:
service:
enabled: true
tls:
secretName: nats-operator-metrics-tls
scraper:
serviceAccount: monitoring/prometheus
serviceMonitor:
enabled: true
networkPolicy:
enabled: true
from:
- namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: monitoring}}
egress:
- to: [{ipBlock: {cidr: 10.96.0.1/32}}]
ports: [{protocol: TCP, port: 443}]
- to: [{ipBlock: {cidr: 172.18.0.2/32}}]
ports: [{protocol: TCP, port: 6443}]
- to:
- namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}
podSelector: {matchLabels: {k8s-app: kube-dns}}
ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
- to:
- namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: nats-system}}
ports: [{protocol: TCP, port: 4222}, {protocol: TCP, port: 8222}]The first two egress rules are for the API server, and both addresses are examples. Replace them with the addresses of your Kubernetes cluster:
For the first rule, read the address of the Service
kubernetes:kubectl -n default get service kubernetesFor the second rule, read the addresses of its endpoints:
kubectl -n default get endpointslices -l kubernetes.io/service-name=kubernetes
The last rule is for the NATS clusters in nats-system.
Port 4222 is the client port, which a NatsConnection has in spec.servers, and port 8222 is the monitoring port of the servers, which the cluster controller reads.
With these values, each ServiceMonitor scrapes with the token of the Prometheus pod, and metrics.scraper.serviceAccount allows that ServiceAccount to read /metrics.
The ServiceMonitor verifies the certificate against ca.crt in the Secret, under the DNS name of the Service, such as nats-operator-cluster-controller-metrics.nats-operator.svc.
Metrics and Network policy in the chart reference describe the other values for metrics and the network policy.
Check that the controllers work under the policy#
Apply a NATS cluster in nats-system:
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: demo
namespace: nats-system
spec:
version: 2.15.0
replicas: 3
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 256Mi
jetstream:
volumeClaimTemplate:
spec:
storageClassName: standard
resources:
requests:
storage: 1Gi
kubectl apply -f https://nats-operator.io/docs/stories/12-metrics/01-natscluster.yamlRead its status:
kubectl -n nats-system get natscluster demo -o yamlThe status in the output is similar to this:
status:
replicas: 3
readyReplicas: 3
conditions:
- type: Ready
status: "True"
reason: AllServersReady
Apply a connection to the NATS cluster, and a stream on it:
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
name: demo
namespace: nats-system
spec:
servers: ["nats://demo.nats-system.svc:4222"]
kubectl apply -f https://nats-operator.io/docs/stories/12-metrics/02-natsconnection.yamlapiVersion: jetstream.nats-operator.io/v1beta1
kind: NatsStream
metadata:
name: events
namespace: nats-system
spec:
connectionRef:
name: demo
name: EVENTS
subjects: ["events.>"]
storage: File
replicas: 3
kubectl apply -f https://nats-operator.io/docs/stories/12-metrics/02-natsstream.yamlRead the status of the stream:
kubectl -n nats-system get natsstream events -o yamlThe status in the output is similar to this:
status:
conditions:
- type: Ready
status: "True"
reason: Synced
- type: Synced
status: "True"
reason: MatchesSpec
Check that the policy blocks other destinations#
No egress rule allows the namespace nats-outside.
Apply a NATS cluster there:
apiVersion: cluster.nats-operator.io/v1beta1
kind: NatsCluster
metadata:
name: outside
namespace: nats-outside
spec:
version: 2.15.0
replicas: 1
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
memory: 128Mi
kubectl apply -f https://nats-operator.io/docs/stories/12-metrics/01-natscluster-outside.yamlRead its status:
kubectl -n nats-outside get natscluster outside -o yamlThe status in the output is similar to this:
status:
replicas: 1
readyReplicas: 1
conditions:
- type: Ready
status: "True"
reason: AllServersReady
The NATS cluster is Ready, because the cluster controller deploys it through the API server.
Apply a NatsConnection to it:
apiVersion: nats-operator.io/v1beta1
kind: NatsConnection
metadata:
name: outside
namespace: nats-system
spec:
servers: ["nats://outside.nats-outside.svc:4222"]
kubectl apply -f https://nats-operator.io/docs/stories/12-metrics/02-natsconnection-outside.yamlRead its status:
kubectl -n nats-system get natsconnection outside -o yamlThe status in the output is similar to this:
status:
conditions:
- type: Ready
status: "False"
reason: ConnectFailed
Ready stays False with the reason ConnectFailed, because the policy drops the connection from the JetStream controller.
Scrape by hand#
To scrape the cluster controller as its ServiceMonitor does:
Write the CA to a file:
kubectl -n nats-operator get secret nats-operator-metrics-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crtForward the port of the metrics Service to your machine:
kubectl -n nats-operator port-forward svc/nats-operator-cluster-controller-metrics 8080 &Scrape with a token of the ServiceAccount
prometheus:curl --cacert ca.crt \ --resolve nats-operator-cluster-controller-metrics.nats-operator.svc:8080:127.0.0.1 \ -H "Authorization: Bearer $(kubectl -n monitoring create token prometheus)" \ https://nats-operator-cluster-controller-metrics.nats-operator.svc:8080/metrics