Reference

Telemetry

Contents

Each controller exports metrics and traces through the OpenTelemetry Go SDK. Only the environment configures them; no controller has a flag for them. A signal is exported only when one of OTEL_<SIGNAL>_EXPORTER, OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT is set, where <SIGNAL> is METRICS or TRACES. With none of them set, or with OTEL_SDK_DISABLED=true, the controller exports nothing and does not collect the instruments below. Once a signal is on, the controller sends it as OTLP over http/protobuf to https://localhost:4318, unless the variables below set otherwise, and logs a failed export. controller-runtime’s own Prometheus metrics are served separately, on --metrics-bind-address.

Environment#

Table: The environment variables that configure the OpenTelemetry SDK in each controller.

VariableDefaultEffect
OTEL_SDK_DISABLEDfalsetrue, in any letter case, turns off every export, whatever else is set. No exporter is built and no Prometheus listener is opened.
OTEL_SERVICE_NAMEthe controller’s name, such as cluster-controllerservice.name of every metric and span.
OTEL_RESOURCE_ATTRIBUTESunsetFurther resource attributes, as key=value pairs separated by commas.
OTEL_METRICS_EXPORTERotlp once metrics are onotlp, prometheus, console or none. Any value but none turns metrics on. none turns metrics off, whatever else is set.
OTEL_TRACES_EXPORTERotlp once traces are onotlp, console or none. Any value but none turns traces on. none turns traces off, whatever else is set.
OTEL_EXPORTER_OTLP_PROTOCOL
OTEL_EXPORTER_OTLP_METRICS_PROTOCOL
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
http/protobufhttp/protobuf or grpc.
OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
https://localhost:4318 over http/protobuf, https://localhost:4317 over grpcWhere OTLP is sent. Setting it turns on the signals it applies to. An http:// endpoint sends without TLS.
OTEL_EXPORTER_OTLP_INSECURE
OTEL_EXPORTER_OTLP_METRICS_INSECURE
OTEL_EXPORTER_OTLP_TRACES_INSECURE
falsetrue sends OTLP without TLS.
OTEL_EXPORTER_OTLP_CERTIFICATE
OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE
OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE
unsetA PEM file of the CA certificates that the collector’s certificate is verified against.
OTEL_EXPORTER_OTLP_HEADERS
OTEL_EXPORTER_OTLP_METRICS_HEADERS
OTEL_EXPORTER_OTLP_TRACES_HEADERS
unsetHeaders sent with each export, as key=value pairs separated by commas.
OTEL_EXPORTER_OTLP_TIMEOUT
OTEL_EXPORTER_OTLP_METRICS_TIMEOUT
OTEL_EXPORTER_OTLP_TRACES_TIMEOUT
unsetMilliseconds an export may take.
OTEL_EXPORTER_OTLP_COMPRESSION
OTEL_EXPORTER_OTLP_METRICS_COMPRESSION
OTEL_EXPORTER_OTLP_TRACES_COMPRESSION
nonegzip compresses each export.
OTEL_EXPORTER_PROMETHEUS_HOST
OTEL_EXPORTER_PROMETHEUS_PORT
localhost, 9464Where OTEL_METRICS_EXPORTER=prometheus serves /metrics for scraping. The chart value metrics.prometheus.enabled sets OTEL_METRICS_EXPORTER=prometheus and OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0, and puts port 9464 on the metrics Service and ServiceMonitor.
OTEL_METRIC_EXPORT_INTERVALunsetMilliseconds between two exports of the otlp and console metrics exporters. Each export reads the resources’ status. The prometheus exporter reads the status at each scrape instead.
OTEL_METRIC_EXPORT_TIMEOUTunsetMilliseconds a metric export may take.
OTEL_TRACES_SAMPLER
OTEL_TRACES_SAMPLER_ARG
parentbased_always_onWhich reconcile spans are kept.

A variable with METRICS or TRACES in its name applies to that signal alone and overrides the one without.

The OpenTelemetry Operator#

The chart sets two of the variables above, OTEL_METRICS_EXPORTER and OTEL_EXPORTER_PROMETHEUS_HOST, and only when metrics.prometheus.enabled is set. The OpenTelemetry Operator sets them from an Instrumentation in the release namespace, in every pod annotated instrumentation.opentelemetry.io/inject-sdk: "true". The chart’s podAnnotations puts that annotation on the controllers’ pods.

An Instrumentation:

yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: nats-operator
  namespace: nats-operator
spec:
  exporter:
    endpoint: http://otel-collector.observability.svc:4318

The chart values that annotate the controllers’ pods for it:

yaml
podAnnotations:
  instrumentation.opentelemetry.io/inject-sdk: "true"
cluster:
  podAnnotations:
    resource.opentelemetry.io/service.name: cluster-controller
auth:
  podAnnotations:
    resource.opentelemetry.io/service.name: auth-controller
jetstream:
  podAnnotations:
    resource.opentelemetry.io/service.name: jetstream-controller

inject-sdk injects environment variables only (SDK environment variables only):

  • OTEL_EXPORTER_OTLP_ENDPOINT, from spec.exporter.endpoint. It turns metrics and traces on.
  • OTEL_SERVICE_NAME.
  • OTEL_RESOURCE_ATTRIBUTES, with the pod’s Kubernetes attributes.
  • OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG, from spec.sampler.
  • Every entry of spec.env. Any other variable above goes there, such as OTEL_EXPORTER_OTLP_PROTOCOL: grpc for a collector’s gRPC port.

The controllers send http/protobuf unless OTEL_EXPORTER_OTLP_PROTOCOL sets another protocol, so spec.exporter.endpoint must be the collector’s OTLP/HTTP port.

OTEL_SERVICE_NAME is the Deployment’s name, <release>-<controller>, unless the pod annotation resource.opentelemetry.io/service.name sets another, as the chart values above do (Configure resource attributes). The three controllers’ pods share their app.kubernetes.io/name and app.kubernetes.io/instance labels, so spec.defaults.useLabelsForResourceAttributes alone gives all three the same service.name.

A collector can run as a sidecar in each controller’s pod (Sidecar injection), with these three settings:

  • an OpenTelemetryCollector (opentelemetry.io/v1beta1) with mode: sidecar in the release namespace, whose otlp receiver takes HTTP on port 4318
  • sidecar.opentelemetry.io/inject: "true" in podAnnotations
  • http://localhost:4318 as the Instrumentation’s spec.exporter.endpoint

The OpenTelemetry Operator reads both annotations from the pod. On the Deployment, where the chart’s annotations puts them, the OpenTelemetry Operator does not see them. The fields and annotations in this section are those of OpenTelemetry Operator v0.159.0 (Instrumentation API reference).

Metrics#

Each gauge reads the resources’ status at each export. Every point has the attributes kind, namespace and name of the resource it describes. The Prometheus column gives each instrument’s name as OTEL_METRICS_EXPORTER=prometheus serves it on port 9464, which metrics.prometheus.enabled sets.

Table: The instruments the controllers export.

InstrumentPrometheusTypeUnitControllersAttributesReadsValue
nats_operator.conditionnats_operator_conditiongaugecluster-controller, auth-controller, jetstream-controllerkind, namespace, name, type, reasonstatus.conditions of every kind the controller reconciles1 while the condition is True, 0 while it is False or Unknown.
nats_operator.account.jwt_expirynats_operator_account_jwt_expiry_secondsgaugesauth-controllerkind, namespace, nameNatsAccount status.jwtWhen the NatsAccount’s current JWT expires, in seconds since the Unix epoch; a JWT that never expires has no point.
nats_operator.rollout.pending_serversnats_operator_rollout_pending_serversgauge{server}cluster-controllerkind, namespace, nameNatsCluster status.rollout.pendingServers still to update in the NatsCluster’s rollout.
nats_operator.rollout.gatenats_operator_rollout_gategaugecluster-controllerkind, namespace, name, waiting_forNatsCluster status.rollout.gate.waitingFor1 while the rollout’s gate is closed, by what it waits for.
nats_operator.balancer.leader_skewnats_operator_balancer_leader_skewgauge{leader}jetstream-controllerkind, namespace, name, poolNatsBalancer status.pools[].leaderSkew, NatsSystemBalancer status.skew.leadersThe most leaders one server carries less the fewest another does: per pool for a NatsBalancer, over the NATS cluster for a NatsSystemBalancer.
nats_operator.balancer.pending_movesnats_operator_balancer_pending_movesgauge{move}jetstream-controllerkind, namespace, name, move_kindNatsSystemBalancer status.pendingMoves the NatsSystemBalancer requested that are not yet complete, by kind of move.
nats_operator.balancer.held_passesnats_operator_balancer_held_passes_totalcounter{pass}jetstream-controllerkind, namespace, name, reasonNatsBalancer and NatsSystemBalancer status.conditions[Holding], after each passBalancer passes that ended with Holding True, by its reason.
nats_operator.evacuation.remainingnats_operator_evacuation_remaininggauge{stream}jetstream-controllerkind, namespace, nameNatsClusterEvacuation status.remainingStreams still to leave the evacuation’s source NATS cluster.
nats_operator.evacuation.stale_placementsnats_operator_evacuation_stale_placementsgauge{stream}jetstream-controllerkind, namespace, nameNatsClusterEvacuation status.stalePlacementMoved streams no resource owns whose config still names the source NATS cluster.

Account JWT expiry#

While the auth controller runs, it re-signs each account JWT at half its jwtTTL. No account JWT then expires sooner than half the shortest jwtTTL from now, which is 24h under the default jwtTTL of 48h.

Scraped through the chart’s ServiceMonitor, this expression fires when an account JWT expires within 23h:

promql
min(nats_operator_account_jwt_expiry_seconds) - time() < 23 * 3600

The threshold of 23h assumes the default jwtTTL of 48h. Under a shorter jwtTTL, the threshold must be below half of it.

The auth controller exports the gauge itself, and the gauge goes stale when its scrape fails. While the auth controller is down, the expression above returns nothing. This expression fires then, from the ServiceMonitor’s endpoint otel-metrics:

promql
absent(up{job=~".+-auth-controller-metrics", endpoint="otel-metrics"} == 1)

Traces#

Each reconcile runs in a span named Reconcile <kind>, such as Reconcile NatsCluster, with the attributes kind, namespace and name. A reconcile that returns an error records the error on its span and sets the span’s status to Error.

Events#

The controllers record these events through the events.k8s.io API. An event’s regarding is a resource of a kind in the Regarding column.

Table: The Kubernetes events the controllers record.

ReasonTypeControllerRegardingRecorded when
MoveStartedNormaljetstream-controllerNatsBalancer, NatsSystemBalancer, NatsClusterEvacuationA balancer moves a stream’s leader or placement, or an evacuation requests a stream’s move.
MoveDoneNormaljetstream-controllerNatsSystemBalancer, NatsClusterEvacuationA move the system balancer or the evacuation requested is seen complete.
MoveCancelledNormaljetstream-controllerNatsClusterEvacuationDeleting an evacuation cancels a move still in flight.
MoveRefusedWarningjetstream-controllerNatsClusterEvacuationThe server refuses a move the evacuation requests.
EvacuationRefusedWarningjetstream-controllerNatsClusterEvacuationThe evacuation refuses to start because a server of its source NATS cluster carries the target tags.
RolloutStepNormalcluster-controllerNatsClusterA rollout restarts a server, or starts removing or replacing one.
GateBlockedWarningcluster-controllerNatsClusterA rollout’s gate has been closed long enough for Progressing to read GateBlocked.
ReconcileFailedWarningcluster-controllerNatsClusterA reconcile fails on anything but a write conflict; Progressing reads ReconcileFailed with the same message.
JWTPushedNormalauth-controllerNatsSystemAccount, NatsAccountA newly signed account or system account JWT is pushed to the servers.
JWTHeldWarningauth-controllerNatsOperator, NatsAccountAn account JWT is not signed because its revocations cannot be recovered from the servers.
UserKickedNormalauth-controllerNatsUserA deleted user’s live connections are closed.
Esc
↓ ↑ Navigate Enter Select Esc Close