# Telemetry

> The OpenTelemetry metrics and traces each controller exports, the environment that configures them, and the Kubernetes events the controllers record.


<!-- Generated by `just telemetry-docs` from hack/telemetrydocs and internal/telemetry. -->

Each controller exports metrics and traces through the OpenTelemetry Go SDK.
Only the environment configures them; no controller has a flag for them.
A signal is exported only when one of `OTEL_<SIGNAL>_EXPORTER`, `OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT` or `OTEL_EXPORTER_OTLP_ENDPOINT` is set, where `<SIGNAL>` is `METRICS` or `TRACES`.
With none of them set, or with `OTEL_SDK_DISABLED=true`, the controller exports nothing and does not collect the instruments below.
Once a signal is on, the controller sends it as OTLP over `http/protobuf` to `https://localhost:4318`, unless the variables below set otherwise, and logs a failed export.
controller-runtime's own Prometheus metrics are served separately, on `--metrics-bind-address`.

## Environment

Table: The environment variables that configure the OpenTelemetry SDK in each controller.

| Variable | Default | Effect |
|---|---|---|
| `OTEL_SDK_DISABLED` | `false` | `true`, in any letter case, turns off every export, whatever else is set. No exporter is built and no Prometheus listener is opened. |
| `OTEL_SERVICE_NAME` | the controller's name, such as `cluster-controller` | `service.name` of every metric and span. |
| `OTEL_RESOURCE_ATTRIBUTES` | unset | Further resource attributes, as `key=value` pairs separated by commas. |
| `OTEL_METRICS_EXPORTER` | `otlp` once metrics are on | `otlp`, `prometheus`, `console` or `none`. Any value but `none` turns metrics on. `none` turns metrics off, whatever else is set. |
| `OTEL_TRACES_EXPORTER` | `otlp` once traces are on | `otlp`, `console` or `none`. Any value but `none` turns traces on. `none` turns traces off, whatever else is set. |
| `OTEL_EXPORTER_OTLP_PROTOCOL`<br>`OTEL_EXPORTER_OTLP_METRICS_PROTOCOL`<br>`OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` | `http/protobuf` | `http/protobuf` or `grpc`. |
| `OTEL_EXPORTER_OTLP_ENDPOINT`<br>`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`<br>`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | `https://localhost:4318` over `http/protobuf`, `https://localhost:4317` over `grpc` | Where OTLP is sent. Setting it turns on the signals it applies to. An `http://` endpoint sends without TLS. |
| `OTEL_EXPORTER_OTLP_INSECURE`<br>`OTEL_EXPORTER_OTLP_METRICS_INSECURE`<br>`OTEL_EXPORTER_OTLP_TRACES_INSECURE` | `false` | `true` sends OTLP without TLS. |
| `OTEL_EXPORTER_OTLP_CERTIFICATE`<br>`OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE`<br>`OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE` | unset | A PEM file of the CA certificates that the collector's certificate is verified against. |
| `OTEL_EXPORTER_OTLP_HEADERS`<br>`OTEL_EXPORTER_OTLP_METRICS_HEADERS`<br>`OTEL_EXPORTER_OTLP_TRACES_HEADERS` | unset | Headers sent with each export, as `key=value` pairs separated by commas. |
| `OTEL_EXPORTER_OTLP_TIMEOUT`<br>`OTEL_EXPORTER_OTLP_METRICS_TIMEOUT`<br>`OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` | unset | Milliseconds an export may take. |
| `OTEL_EXPORTER_OTLP_COMPRESSION`<br>`OTEL_EXPORTER_OTLP_METRICS_COMPRESSION`<br>`OTEL_EXPORTER_OTLP_TRACES_COMPRESSION` | none | `gzip` compresses each export. |
| `OTEL_EXPORTER_PROMETHEUS_HOST`<br>`OTEL_EXPORTER_PROMETHEUS_PORT` | `localhost`, `9464` | Where `OTEL_METRICS_EXPORTER=prometheus` serves `/metrics` for scraping. The chart value `metrics.prometheus.enabled` sets `OTEL_METRICS_EXPORTER=prometheus` and `OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0`, and puts port `9464` on the metrics Service and ServiceMonitor. |
| `OTEL_METRIC_EXPORT_INTERVAL` | unset | Milliseconds between two exports of the `otlp` and `console` metrics exporters. Each export reads the resources' status. The `prometheus` exporter reads the status at each scrape instead. |
| `OTEL_METRIC_EXPORT_TIMEOUT` | unset | Milliseconds a metric export may take. |
| `OTEL_TRACES_SAMPLER`<br>`OTEL_TRACES_SAMPLER_ARG` | `parentbased_always_on` | Which reconcile spans are kept. |

A variable with `METRICS` or `TRACES` in its name applies to that signal alone and overrides the one without.

## The OpenTelemetry Operator

The chart sets two of the variables above, `OTEL_METRICS_EXPORTER` and `OTEL_EXPORTER_PROMETHEUS_HOST`, and only when `metrics.prometheus.enabled` is set.
The [OpenTelemetry Operator](https://opentelemetry.io/docs/platforms/kubernetes/operator/) sets them from an `Instrumentation` in the release namespace, in every pod annotated `instrumentation.opentelemetry.io/inject-sdk: "true"`.
The chart's `podAnnotations` puts that annotation on the controllers' pods.

An `Instrumentation`:

```yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: nats-operator
  namespace: nats-operator
spec:
  exporter:
    endpoint: http://otel-collector.observability.svc:4318
```

The chart values that annotate the controllers' pods for it:

```yaml
podAnnotations:
  instrumentation.opentelemetry.io/inject-sdk: "true"
cluster:
  podAnnotations:
    resource.opentelemetry.io/service.name: cluster-controller
auth:
  podAnnotations:
    resource.opentelemetry.io/service.name: auth-controller
jetstream:
  podAnnotations:
    resource.opentelemetry.io/service.name: jetstream-controller
```

`inject-sdk` injects environment variables only ([SDK environment variables only](https://github.com/open-telemetry/opentelemetry-operator/blob/v0.159.0/docs/auto-instrumentation/languages/sdk-only.md)):

- `OTEL_EXPORTER_OTLP_ENDPOINT`, from `spec.exporter.endpoint`. It turns metrics and traces on.
- `OTEL_SERVICE_NAME`.
- `OTEL_RESOURCE_ATTRIBUTES`, with the pod's Kubernetes attributes.
- `OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG`, from `spec.sampler`.
- Every entry of `spec.env`. Any other variable above goes there, such as `OTEL_EXPORTER_OTLP_PROTOCOL: grpc` for a collector's gRPC port.

The controllers send `http/protobuf` unless `OTEL_EXPORTER_OTLP_PROTOCOL` sets another protocol, so `spec.exporter.endpoint` must be the collector's OTLP/HTTP port.

`OTEL_SERVICE_NAME` is the Deployment's name, `<release>-<controller>`, unless the pod annotation `resource.opentelemetry.io/service.name` sets another, as the chart values above do ([Configure resource attributes](https://github.com/open-telemetry/opentelemetry-operator/blob/v0.159.0/docs/auto-instrumentation/resource-attributes.md)).
The three controllers' pods share their `app.kubernetes.io/name` and `app.kubernetes.io/instance` labels, so `spec.defaults.useLabelsForResourceAttributes` alone gives all three the same `service.name`.

A collector can run as a sidecar in each controller's pod ([Sidecar injection](https://github.com/open-telemetry/opentelemetry-operator/blob/v0.159.0/docs/collector/sidecar-injection.md)), with these three settings:

- an `OpenTelemetryCollector` (`opentelemetry.io/v1beta1`) with `mode: sidecar` in the release namespace, whose `otlp` receiver takes HTTP on port 4318
- `sidecar.opentelemetry.io/inject: "true"` in `podAnnotations`
- `http://localhost:4318` as the `Instrumentation`'s `spec.exporter.endpoint`

The OpenTelemetry Operator reads both annotations from the pod.
On the Deployment, where the chart's `annotations` puts them, the OpenTelemetry Operator does not see them.
The fields and annotations in this section are those of OpenTelemetry Operator v0.159.0 ([`Instrumentation` API reference](https://github.com/open-telemetry/opentelemetry-operator/blob/v0.159.0/docs/api/instrumentations.md)).

## Metrics

Each gauge reads the resources' status at each export.
Every point has the attributes `kind`, `namespace` and `name` of the resource it describes.
The Prometheus column gives each instrument's name as `OTEL_METRICS_EXPORTER=prometheus` serves it on port 9464, which `metrics.prometheus.enabled` sets.

Table: The instruments the controllers export.

| Instrument | Prometheus | Type | Unit | Controllers | Attributes | Reads | Value |
|---|---|---|---|---|---|---|---|
| `nats_operator.condition` | `nats_operator_condition` | gauge |  | cluster-controller, auth-controller, jetstream-controller | `kind`, `namespace`, `name`, `type`, `reason` | status.conditions of every kind the controller reconciles | 1 while the condition is True, 0 while it is False or Unknown. |
| `nats_operator.account.jwt_expiry` | `nats_operator_account_jwt_expiry_seconds` | gauge | `s` | auth-controller | `kind`, `namespace`, `name` | NatsAccount status.jwt | When the NatsAccount's current JWT expires, in seconds since the Unix epoch; a JWT that never expires has no point. |
| `nats_operator.rollout.pending_servers` | `nats_operator_rollout_pending_servers` | gauge | `{server}` | cluster-controller | `kind`, `namespace`, `name` | NatsCluster status.rollout.pending | Servers still to update in the NatsCluster's rollout. |
| `nats_operator.rollout.gate` | `nats_operator_rollout_gate` | gauge |  | cluster-controller | `kind`, `namespace`, `name`, `waiting_for` | NatsCluster status.rollout.gate.waitingFor | 1 while the rollout's gate is closed, by what it waits for. |
| `nats_operator.balancer.leader_skew` | `nats_operator_balancer_leader_skew` | gauge | `{leader}` | jetstream-controller | `kind`, `namespace`, `name`, `pool` | NatsBalancer status.pools[].leaderSkew, NatsSystemBalancer status.skew.leaders | The most leaders one server carries less the fewest another does: per pool for a NatsBalancer, over the NATS cluster for a NatsSystemBalancer. |
| `nats_operator.balancer.pending_moves` | `nats_operator_balancer_pending_moves` | gauge | `{move}` | jetstream-controller | `kind`, `namespace`, `name`, `move_kind` | NatsSystemBalancer status.pending | Moves the NatsSystemBalancer requested that are not yet complete, by kind of move. |
| `nats_operator.balancer.held_passes` | `nats_operator_balancer_held_passes_total` | counter | `{pass}` | jetstream-controller | `kind`, `namespace`, `name`, `reason` | NatsBalancer and NatsSystemBalancer status.conditions[Holding], after each pass | Balancer passes that ended with Holding True, by its reason. |
| `nats_operator.evacuation.remaining` | `nats_operator_evacuation_remaining` | gauge | `{stream}` | jetstream-controller | `kind`, `namespace`, `name` | NatsClusterEvacuation status.remaining | Streams still to leave the evacuation's source NATS cluster. |
| `nats_operator.evacuation.stale_placements` | `nats_operator_evacuation_stale_placements` | gauge | `{stream}` | jetstream-controller | `kind`, `namespace`, `name` | NatsClusterEvacuation status.stalePlacement | Moved streams no resource owns whose config still names the source NATS cluster. |

### Account JWT expiry

While the auth controller runs, it re-signs each account JWT at half its `jwtTTL`.
No account JWT then expires sooner than half the shortest `jwtTTL` from now, which is 24h under the default `jwtTTL` of 48h.

Scraped through the chart's ServiceMonitor, this expression fires when an account JWT expires within 23h:

```promql
min(nats_operator_account_jwt_expiry_seconds) - time() < 23 * 3600
```

The threshold of 23h assumes the default `jwtTTL` of 48h.
Under a shorter `jwtTTL`, the threshold must be below half of it.

The auth controller exports the gauge itself, and the gauge goes stale when its scrape fails.
While the auth controller is down, the expression above returns nothing.
This expression fires then, from the ServiceMonitor's endpoint `otel-metrics`:

```promql
absent(up{job=~".+-auth-controller-metrics", endpoint="otel-metrics"} == 1)
```

## Traces

Each reconcile runs in a span named `Reconcile <kind>`, such as `Reconcile NatsCluster`, with the attributes `kind`, `namespace` and `name`.
A reconcile that returns an error records the error on its span and sets the span's status to `Error`.

## Events

The controllers record these events through the `events.k8s.io` API.
An event's `regarding` is a resource of a kind in the Regarding column.

Table: The Kubernetes events the controllers record.

| Reason | Type | Controller | Regarding | Recorded when |
|---|---|---|---|---|
| `MoveStarted` | Normal | jetstream-controller | `NatsBalancer`, `NatsSystemBalancer`, `NatsClusterEvacuation` | A balancer moves a stream's leader or placement, or an evacuation requests a stream's move. |
| `MoveDone` | Normal | jetstream-controller | `NatsSystemBalancer`, `NatsClusterEvacuation` | A move the system balancer or the evacuation requested is seen complete. |
| `MoveCancelled` | Normal | jetstream-controller | `NatsClusterEvacuation` | Deleting an evacuation cancels a move still in flight. |
| `MoveRefused` | Warning | jetstream-controller | `NatsClusterEvacuation` | The server refuses a move the evacuation requests. |
| `EvacuationRefused` | Warning | jetstream-controller | `NatsClusterEvacuation` | The evacuation refuses to start because a server of its source NATS cluster carries the target tags. |
| `RolloutStep` | Normal | cluster-controller | `NatsCluster` | A rollout restarts a server, or starts removing or replacing one. |
| `GateBlocked` | Warning | cluster-controller | `NatsCluster` | A rollout's gate has been closed long enough for Progressing to read GateBlocked. |
| `ReconcileFailed` | Warning | cluster-controller | `NatsCluster` | A reconcile fails on anything but a write conflict; Progressing reads ReconcileFailed with the same message. |
| `JWTPushed` | Normal | auth-controller | `NatsSystemAccount`, `NatsAccount` | A newly signed account or system account JWT is pushed to the servers. |
| `JWTHeld` | Warning | auth-controller | `NatsOperator`, `NatsAccount` | An account JWT is not signed because its revocations cannot be recovered from the servers. |
| `UserKicked` | Normal | auth-controller | `NatsUser` | A deleted user's live connections are closed. |

