Metrics and observability

telark is designed to be observable from outside. Every service emits Prometheus metrics on a /metrics endpoint, writes structured JSON logs to stdout, and exposes liveness / readiness probes (see Operations → Health checks).

Endpoints

Service	Metrics endpoint	Notes
`discovery-service`	`/metrics` on the configured HTTP port	Cycle, leader, force-sync, plan controller, rollback controller.
`exporter-service`	`/metrics`	CRD writes, notification dispatch, snapshot writes, PVC consumption.
`auth-service`	`/metrics`	Session create / refresh / cleanup, passkey enrolment, OIDC exchange.

Every endpoint is unauthenticated by design — scrape from Prometheus pods running inside the same cluster, gated by NetworkPolicy.

What each service exports

The exact metric names follow telark_<service>_<unit> for counters / histograms and telark_<service>_<state> for gauges. The most useful ones, by service:

discovery-service

Metric	Type	Meaning
`telark_discovery_cycle_duration_seconds`	histogram	End-to-end cycle latency (leader-only).
`telark_discovery_cycle_changes_total`	counter	Per-class change counter. Label: `class`.
`telark_discovery_leader_is_elected`	gauge	1 on the leader, 0 on followers.
`telark_discovery_forcesync_inflight`	gauge	Force-sync markers picked up but not yet finished.
`telark_discovery_redis_errors_total`	counter	Redis call failures (any type).
`telark_discovery_plan_tick_duration_seconds`	histogram	Controller tick latency for protection plans.
`telark_discovery_plan_health_total`	counter	Health resolutions. Label: `result` (`healthy/drifted/degraded/unknown`).
`telark_discovery_rollback_inflight`	gauge	Rollbacks currently being applied.
`telark_discovery_rollback_result_total`	counter	Final rollback statuses. Label: `result` (`success/failed`).

exporter-service

Metric	Type	Meaning
`telark_exporter_crd_writes_total`	counter	Resource writes. Labels: `kind`, `op`.
`telark_exporter_notification_emitted_total`	counter	Notifications successfully dispatched. Label: `type`.
`telark_exporter_snapshot_writes_total`	counter	Snapshots persisted to the PVC.
`telark_exporter_snapshot_prune_total`	counter	Snapshots removed by retention pruning.
`telark_exporter_pvc_bytes_used`	gauge	Aggregate PVC consumption for snapshots.
`telark_exporter_pvc_snapshot_count`	gauge	Total snapshots on the PVC.

auth-service

Metric	Type	Meaning
`telark_auth_session_active`	gauge	Live (non-expired) sessions.
`telark_auth_session_created_total`	counter	Sessions opened. Label: `method` (`passkey/oidc`).
`telark_auth_session_expired_total`	counter	Sessions reaped by the cleanup controller.
`telark_auth_passkey_registered_total`	counter	Passkey enrolments.
`telark_auth_oidc_exchange_total`	counter	OIDC code-for-token exchanges. Label: `provider`, `result`.
`telark_auth_permission_resolve_duration_seconds`	histogram	Latency of `/auth/authorisation/permissions`.

Logs

All three services log JSON to stdout. Common fields:

Field	Notes
`ts`	RFC3339 UTC timestamp.
`level`	`debug` / `info` / `warn` / `error`.
`msg`	Short human label.
`service`	One of `discovery`, `exporter`, `auth`.
`component`	Sub-package (e.g. `rollback`, `plan-controller`, `passkey`).
`request_id`	Per-request correlation ID where applicable.
`user_id`	When the request was authenticated.
`resource_id`	When the operation targeted a telark resource.
`err`	Error string (only on `error` and `warn`).

Log shape is stable enough to grep on. A typical operator query:

kubectl logs deploy/telark-discovery -c discovery |
  jq 'select(.component=="rollback" and .level=="error")'

Scraping

The chart can emit a ServiceMonitor (Prometheus Operator) that scrapes every telark service's /metrics. It is off by default, because it needs the Prometheus Operator CRDs already in the cluster:

--set monitoring.serviceMonitor.enabled=true \
--set monitoring.serviceMonitor.labels.release=<prometheus-release>

The labels value is the part people miss: Prometheus only picks up a ServiceMonitor whose labels match its serviceMonitorSelector — usually release: <your-release>. Omit it and the monitor is silently ignored.

The monitor selects every telark service by the app.kubernetes.io/part-of: telark label on the http port. See Guides → Installing for the full key list.

telark ships no prebuilt dashboards today. Build your own against the metric names above.

Alerts

telark ships no opinionated alert rules — alerting is your team's prerogative. A reasonable starter set:

telark_discovery_redis_errors_total rate > 0 over 5 minutes.
telark_discovery_leader_is_elected sum across replicas ≠ 1.
telark_discovery_rollback_result_total{result="failed"} rate

0.
telark_exporter_pvc_bytes_used / pvc_capacity > 80%.

Tracing

OpenTelemetry tracing is not wired today. The structured logs carry request_id so you can correlate across services manually if needed.

Metrics and observability

On this page