Reference
Telemetry
Each controller exports metrics and traces through the OpenTelemetry Go SDK.
Only the environment configures them; no controller has a flag for them.
A signal is exported only when one of OTEL_<SIGNAL>_EXPORTER, OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT is set, where <SIGNAL> is METRICS or TRACES.
With none of them set, or with OTEL_SDK_DISABLED=true, the controller exports nothing and does not collect the instruments below.
Once a signal is on, the controller sends it as OTLP over http/protobuf to https://localhost:4318, unless the variables below set otherwise, and logs a failed export.
controller-runtime’s own Prometheus metrics are served separately, on --metrics-bind-address.
Environment#
Table: The environment variables that configure the OpenTelemetry SDK in each controller.
| Variable | Default | Effect |
|---|---|---|
OTEL_SDK_DISABLED | false | true, in any letter case, turns off every export, whatever else is set. No exporter is built and no Prometheus listener is opened. |
OTEL_SERVICE_NAME | the controller’s name, such as cluster-controller | service.name of every metric and span. |
OTEL_RESOURCE_ATTRIBUTES | unset | Further resource attributes, as key=value pairs separated by commas. |
OTEL_METRICS_EXPORTER | otlp once metrics are on | otlp, prometheus, console or none. Any value but none turns metrics on. none turns metrics off, whatever else is set. |
OTEL_TRACES_EXPORTER | otlp once traces are on | otlp, console or none. Any value but none turns traces on. none turns traces off, whatever else is set. |
OTEL_EXPORTER_OTLP_PROTOCOLOTEL_EXPORTER_OTLP_METRICS_PROTOCOLOTEL_EXPORTER_OTLP_TRACES_PROTOCOL | http/protobuf | http/protobuf or grpc. |
OTEL_EXPORTER_OTLP_ENDPOINTOTEL_EXPORTER_OTLP_METRICS_ENDPOINTOTEL_EXPORTER_OTLP_TRACES_ENDPOINT | https://localhost:4318 over http/protobuf, https://localhost:4317 over grpc | Where OTLP is sent. Setting it turns on the signals it applies to. An http:// endpoint sends without TLS. |
OTEL_EXPORTER_OTLP_INSECUREOTEL_EXPORTER_OTLP_METRICS_INSECUREOTEL_EXPORTER_OTLP_TRACES_INSECURE | false | true sends OTLP without TLS. |
OTEL_EXPORTER_OTLP_CERTIFICATEOTEL_EXPORTER_OTLP_METRICS_CERTIFICATEOTEL_EXPORTER_OTLP_TRACES_CERTIFICATE | unset | A PEM file of the CA certificates that the collector’s certificate is verified against. |
OTEL_EXPORTER_OTLP_HEADERSOTEL_EXPORTER_OTLP_METRICS_HEADERSOTEL_EXPORTER_OTLP_TRACES_HEADERS | unset | Headers sent with each export, as key=value pairs separated by commas. |
OTEL_EXPORTER_OTLP_TIMEOUTOTEL_EXPORTER_OTLP_METRICS_TIMEOUTOTEL_EXPORTER_OTLP_TRACES_TIMEOUT | unset | Milliseconds an export may take. |
OTEL_EXPORTER_OTLP_COMPRESSIONOTEL_EXPORTER_OTLP_METRICS_COMPRESSIONOTEL_EXPORTER_OTLP_TRACES_COMPRESSION | none | gzip compresses each export. |
OTEL_EXPORTER_PROMETHEUS_HOSTOTEL_EXPORTER_PROMETHEUS_PORT | localhost, 9464 | Where OTEL_METRICS_EXPORTER=prometheus serves /metrics for scraping. The chart value metrics.prometheus.enabled sets OTEL_METRICS_EXPORTER=prometheus and OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0, and puts port 9464 on the metrics Service and ServiceMonitor. |
OTEL_METRIC_EXPORT_INTERVAL | unset | Milliseconds between two exports of the otlp and console metrics exporters. Each export reads the resources’ status. The prometheus exporter reads the status at each scrape instead. |
OTEL_METRIC_EXPORT_TIMEOUT | unset | Milliseconds a metric export may take. |
OTEL_TRACES_SAMPLEROTEL_TRACES_SAMPLER_ARG | parentbased_always_on | Which reconcile spans are kept. |
A variable with METRICS or TRACES in its name applies to that signal alone and overrides the one without.
The OpenTelemetry Operator#
The chart sets two of the variables above, OTEL_METRICS_EXPORTER and OTEL_EXPORTER_PROMETHEUS_HOST, and only when metrics.prometheus.enabled is set.
The OpenTelemetry Operator sets them from an Instrumentation in the release namespace, in every pod annotated instrumentation.opentelemetry.io/inject-sdk: "true".
The chart’s podAnnotations puts that annotation on the controllers’ pods.
An Instrumentation:
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: nats-operator
namespace: nats-operator
spec:
exporter:
endpoint: http://otel-collector.observability.svc:4318The chart values that annotate the controllers’ pods for it:
podAnnotations:
instrumentation.opentelemetry.io/inject-sdk: "true"
cluster:
podAnnotations:
resource.opentelemetry.io/service.name: cluster-controller
auth:
podAnnotations:
resource.opentelemetry.io/service.name: auth-controller
jetstream:
podAnnotations:
resource.opentelemetry.io/service.name: jetstream-controllerinject-sdk injects environment variables only (SDK environment variables only):
OTEL_EXPORTER_OTLP_ENDPOINT, fromspec.exporter.endpoint. It turns metrics and traces on.OTEL_SERVICE_NAME.OTEL_RESOURCE_ATTRIBUTES, with the pod’s Kubernetes attributes.OTEL_TRACES_SAMPLERandOTEL_TRACES_SAMPLER_ARG, fromspec.sampler.- Every entry of
spec.env. Any other variable above goes there, such asOTEL_EXPORTER_OTLP_PROTOCOL: grpcfor a collector’s gRPC port.
The controllers send http/protobuf unless OTEL_EXPORTER_OTLP_PROTOCOL sets another protocol, so spec.exporter.endpoint must be the collector’s OTLP/HTTP port.
OTEL_SERVICE_NAME is the Deployment’s name, <release>-<controller>, unless the pod annotation resource.opentelemetry.io/service.name sets another, as the chart values above do (Configure resource attributes).
The three controllers’ pods share their app.kubernetes.io/name and app.kubernetes.io/instance labels, so spec.defaults.useLabelsForResourceAttributes alone gives all three the same service.name.
A collector can run as a sidecar in each controller’s pod (Sidecar injection), with these three settings:
- an
OpenTelemetryCollector(opentelemetry.io/v1beta1) withmode: sidecarin the release namespace, whoseotlpreceiver takes HTTP on port 4318 sidecar.opentelemetry.io/inject: "true"inpodAnnotationshttp://localhost:4318as theInstrumentation’sspec.exporter.endpoint
The OpenTelemetry Operator reads both annotations from the pod.
On the Deployment, where the chart’s annotations puts them, the OpenTelemetry Operator does not see them.
The fields and annotations in this section are those of OpenTelemetry Operator v0.159.0 (Instrumentation API reference).
Metrics#
Each gauge reads the resources’ status at each export.
Every point has the attributes kind, namespace and name of the resource it describes.
The Prometheus column gives each instrument’s name as OTEL_METRICS_EXPORTER=prometheus serves it on port 9464, which metrics.prometheus.enabled sets.
Table: The instruments the controllers export.
| Instrument | Prometheus | Type | Unit | Controllers | Attributes | Reads | Value |
|---|---|---|---|---|---|---|---|
nats_operator.condition | nats_operator_condition | gauge | cluster-controller, auth-controller, jetstream-controller | kind, namespace, name, type, reason | status.conditions of every kind the controller reconciles | 1 while the condition is True, 0 while it is False or Unknown. | |
nats_operator.account.jwt_expiry | nats_operator_account_jwt_expiry_seconds | gauge | s | auth-controller | kind, namespace, name | NatsAccount status.jwt | When the NatsAccount’s current JWT expires, in seconds since the Unix epoch; a JWT that never expires has no point. |
nats_operator.rollout.pending_servers | nats_operator_rollout_pending_servers | gauge | {server} | cluster-controller | kind, namespace, name | NatsCluster status.rollout.pending | Servers still to update in the NatsCluster’s rollout. |
nats_operator.rollout.gate | nats_operator_rollout_gate | gauge | cluster-controller | kind, namespace, name, waiting_for | NatsCluster status.rollout.gate.waitingFor | 1 while the rollout’s gate is closed, by what it waits for. | |
nats_operator.balancer.leader_skew | nats_operator_balancer_leader_skew | gauge | {leader} | jetstream-controller | kind, namespace, name, pool | NatsBalancer status.pools[].leaderSkew, NatsSystemBalancer status.skew.leaders | The most leaders one server carries less the fewest another does: per pool for a NatsBalancer, over the NATS cluster for a NatsSystemBalancer. |
nats_operator.balancer.pending_moves | nats_operator_balancer_pending_moves | gauge | {move} | jetstream-controller | kind, namespace, name, move_kind | NatsSystemBalancer status.pending | Moves the NatsSystemBalancer requested that are not yet complete, by kind of move. |
nats_operator.balancer.held_passes | nats_operator_balancer_held_passes_total | counter | {pass} | jetstream-controller | kind, namespace, name, reason | NatsBalancer and NatsSystemBalancer status.conditions[Holding], after each pass | Balancer passes that ended with Holding True, by its reason. |
nats_operator.evacuation.remaining | nats_operator_evacuation_remaining | gauge | {stream} | jetstream-controller | kind, namespace, name | NatsClusterEvacuation status.remaining | Streams still to leave the evacuation’s source NATS cluster. |
nats_operator.evacuation.stale_placements | nats_operator_evacuation_stale_placements | gauge | {stream} | jetstream-controller | kind, namespace, name | NatsClusterEvacuation status.stalePlacement | Moved streams no resource owns whose config still names the source NATS cluster. |
Account JWT expiry#
While the auth controller runs, it re-signs each account JWT at half its jwtTTL.
No account JWT then expires sooner than half the shortest jwtTTL from now, which is 24h under the default jwtTTL of 48h.
Scraped through the chart’s ServiceMonitor, this expression fires when an account JWT expires within 23h:
min(nats_operator_account_jwt_expiry_seconds) - time() < 23 * 3600The threshold of 23h assumes the default jwtTTL of 48h.
Under a shorter jwtTTL, the threshold must be below half of it.
The auth controller exports the gauge itself, and the gauge goes stale when its scrape fails.
While the auth controller is down, the expression above returns nothing.
This expression fires then, from the ServiceMonitor’s endpoint otel-metrics:
absent(up{job=~".+-auth-controller-metrics", endpoint="otel-metrics"} == 1)Traces#
Each reconcile runs in a span named Reconcile <kind>, such as Reconcile NatsCluster, with the attributes kind, namespace and name.
A reconcile that returns an error records the error on its span and sets the span’s status to Error.
Events#
The controllers record these events through the events.k8s.io API.
An event’s regarding is a resource of a kind in the Regarding column.
Table: The Kubernetes events the controllers record.
| Reason | Type | Controller | Regarding | Recorded when |
|---|---|---|---|---|
MoveStarted | Normal | jetstream-controller | NatsBalancer, NatsSystemBalancer, NatsClusterEvacuation | A balancer moves a stream’s leader or placement, or an evacuation requests a stream’s move. |
MoveDone | Normal | jetstream-controller | NatsSystemBalancer, NatsClusterEvacuation | A move the system balancer or the evacuation requested is seen complete. |
MoveCancelled | Normal | jetstream-controller | NatsClusterEvacuation | Deleting an evacuation cancels a move still in flight. |
MoveRefused | Warning | jetstream-controller | NatsClusterEvacuation | The server refuses a move the evacuation requests. |
EvacuationRefused | Warning | jetstream-controller | NatsClusterEvacuation | The evacuation refuses to start because a server of its source NATS cluster carries the target tags. |
RolloutStep | Normal | cluster-controller | NatsCluster | A rollout restarts a server, or starts removing or replacing one. |
GateBlocked | Warning | cluster-controller | NatsCluster | A rollout’s gate has been closed long enough for Progressing to read GateBlocked. |
ReconcileFailed | Warning | cluster-controller | NatsCluster | A reconcile fails on anything but a write conflict; Progressing reads ReconcileFailed with the same message. |
JWTPushed | Normal | auth-controller | NatsSystemAccount, NatsAccount | A newly signed account or system account JWT is pushed to the servers. |
JWTHeld | Warning | auth-controller | NatsOperator, NatsAccount | An account JWT is not signed because its revocations cannot be recovered from the servers. |
UserKicked | Normal | auth-controller | NatsUser | A deleted user’s live connections are closed. |