Telemetry

Each controller exports metrics and traces through the OpenTelemetry Go SDK, configured by the environment alone; it has no flag of its own for them. A signal is exported only once one of OTEL_<SIGNAL>_EXPORTER, OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT is set, <SIGNAL> being METRICS or TRACES; with none of them set, or with OTEL_SDK_DISABLED=true, the controller exports nothing and the instruments below are not collected. Once a signal is on, it is sent as OTLP over http/protobuf to https://localhost:4318 unless the variables below say otherwise, and a failed export is logged. controller-runtime’s own Prometheus metrics are served apart, on --metrics-bind-address.

Environment #

A variable with METRICS or TRACES in its name applies to that signal alone and overrides the one without.

The OpenTelemetry Operator #

Of the variables above, the chart sets only OTEL_METRICS_EXPORTER and OTEL_EXPORTER_PROMETHEUS_HOST, and only under metrics.prometheus.enabled. The OpenTelemetry Operator sets them from an Instrumentation in the release namespace:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: nats-operator
  namespace: nats-operator
spec:
  exporter:
    endpoint: http://otel-collector.observability.svc:4318

into every pod annotated instrumentation.opentelemetry.io/inject-sdk: "true", which the chart’s podAnnotations carries:

podAnnotations:
  instrumentation.opentelemetry.io/inject-sdk: "true"
cluster:
  podAnnotations:
    resource.opentelemetry.io/service.name: cluster-controller
auth:
  podAnnotations:
    resource.opentelemetry.io/service.name: auth-controller
jetstream:
  podAnnotations:
    resource.opentelemetry.io/service.name: jetstream-controller

inject-sdk injects environment only (SDK environment variables only): OTEL_EXPORTER_OTLP_ENDPOINT from spec.exporter.endpoint, which turns metrics and traces on; OTEL_SERVICE_NAME; OTEL_RESOURCE_ATTRIBUTES with the pod’s Kubernetes attributes; OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG from spec.sampler; and every entry of spec.env, where any other variable above goes, such as OTEL_EXPORTER_OTLP_PROTOCOL: grpc for a collector’s gRPC port. The controllers speak http/protobuf unless told otherwise, so the endpoint is the collector’s OTLP/HTTP port.

OTEL_SERVICE_NAME is the Deployment’s name, <release>-<controller>, unless the pod annotation resource.opentelemetry.io/service.name names another, as above (Configure resource attributes). The three controllers’ pods share their app.kubernetes.io/name and app.kubernetes.io/instance labels, so spec.defaults.useLabelsForResourceAttributes alone gives all three one service.name.

To send through a collector in each controller’s pod instead, create an OpenTelemetryCollector (opentelemetry.io/v1beta1) with mode: sidecar in the release namespace, whose otlp receiver takes HTTP on port 4318; add sidecar.opentelemetry.io/inject: "true" to podAnnotations; and set the Instrumentation’s spec.exporter.endpoint to http://localhost:4318 (Sidecar injection).

Both annotations are read off the pod: on the Deployment, under the chart’s annotations, the OpenTelemetry Operator does not see them. The fields and annotations here are those of OpenTelemetry Operator v0.159.0 (Instrumentation API reference).

Metrics #

The gauges are read off the resources’ status at each export. Every point carries kind, namespace and name of the resource it describes. The Prometheus column is each instrument’s name as OTEL_METRICS_EXPORTER=prometheus serves it on port 9464, which metrics.prometheus.enabled sets.

Account JWT expiry #

The auth controller re-signs each account JWT at half its jwtTTL, so while it runs, no account JWT expires sooner than half the shortest jwtTTL from now: 24h under the default 48h. Scraped through the chart’s ServiceMonitor, this fires once one expires within 23h:

min(nats_operator_account_jwt_expiry_seconds) - time() < 23 * 3600

The threshold assumes the default 48h jwtTTL; a shorter jwtTTL needs one below half of it.

The gauge is exported by the auth controller itself and goes stale once its scrape fails, so that expression returns nothing while the auth controller is down. This fires then, from the ServiceMonitor’s endpoint otel-metrics:

absent(up{job=~".+-auth-controller-metrics", endpoint="otel-metrics"} == 1)

Traces #

Each reconcile runs in a span named Reconcile <kind>, such as Reconcile NatsCluster, carrying kind, namespace and name. A reconcile that returns an error marks its span failed with it.

Events #

The controllers record these through the events.k8s.io API, regarding the resource named.