Metrics and observability
Prometheus metrics, structured logs, and the dashboards telark emits.
telark is designed to be observable from outside. Every service
emits Prometheus metrics on a /metrics endpoint, writes
structured JSON logs to stdout, and exposes liveness / readiness
probes (see Operations → Health checks).
Endpoints
| Service | Metrics endpoint | Notes |
|---|---|---|
discovery-service | /metrics on the configured HTTP port | Cycle, leader, force-sync, plan controller, rollback controller. |
exporter-service | /metrics | CRD writes, notification dispatch, snapshot writes, PVC consumption. |
auth-service | /metrics | Session create / refresh / cleanup, passkey enrolment, OIDC exchange. |
Every endpoint is unauthenticated by design — scrape from Prometheus pods running inside the same cluster, gated by NetworkPolicy.
What each service exports
The exact metric names follow telark_<service>_<unit> for
counters / histograms and telark_<service>_<state> for gauges.
The most useful ones, by service:
discovery-service
| Metric | Type | Meaning |
|---|---|---|
telark_discovery_cycle_duration_seconds | histogram | End-to-end cycle latency (leader-only). |
telark_discovery_cycle_changes_total | counter | Per-class change counter. Label: class. |
telark_discovery_leader_is_elected | gauge | 1 on the leader, 0 on followers. |
telark_discovery_forcesync_inflight | gauge | Force-sync markers picked up but not yet finished. |
telark_discovery_redis_errors_total | counter | Redis call failures (any type). |
telark_discovery_plan_tick_duration_seconds | histogram | Controller tick latency for protection plans. |
telark_discovery_plan_health_total | counter | Health resolutions. Label: result (healthy/drifted/degraded/unknown). |
telark_discovery_rollback_inflight | gauge | Rollbacks currently being applied. |
telark_discovery_rollback_result_total | counter | Final rollback statuses. Label: result (success/failed). |
exporter-service
| Metric | Type | Meaning |
|---|---|---|
telark_exporter_crd_writes_total | counter | Resource writes. Labels: kind, op. |
telark_exporter_notification_emitted_total | counter | Notifications successfully dispatched. Label: type. |
telark_exporter_snapshot_writes_total | counter | Snapshots persisted to the PVC. |
telark_exporter_snapshot_prune_total | counter | Snapshots removed by retention pruning. |
telark_exporter_pvc_bytes_used | gauge | Aggregate PVC consumption for snapshots. |
telark_exporter_pvc_snapshot_count | gauge | Total snapshots on the PVC. |
auth-service
| Metric | Type | Meaning |
|---|---|---|
telark_auth_session_active | gauge | Live (non-expired) sessions. |
telark_auth_session_created_total | counter | Sessions opened. Label: method (passkey/oidc). |
telark_auth_session_expired_total | counter | Sessions reaped by the cleanup controller. |
telark_auth_passkey_registered_total | counter | Passkey enrolments. |
telark_auth_oidc_exchange_total | counter | OIDC code-for-token exchanges. Label: provider, result. |
telark_auth_permission_resolve_duration_seconds | histogram | Latency of /auth/authorisation/permissions. |
Logs
All three services log JSON to stdout. Common fields:
| Field | Notes |
|---|---|
ts | RFC3339 UTC timestamp. |
level | debug / info / warn / error. |
msg | Short human label. |
service | One of discovery, exporter, auth. |
component | Sub-package (e.g. rollback, plan-controller, passkey). |
request_id | Per-request correlation ID where applicable. |
user_id | When the request was authenticated. |
resource_id | When the operation targeted a telark resource. |
err | Error string (only on error and warn). |
Log shape is stable enough to grep on. A typical operator query:
kubectl logs deploy/telark-discovery -c discovery |
jq 'select(.component=="rollback" and .level=="error")'Dashboards
The release-manager Helm chart ships an optional Grafana set covering the most common views:
- Discovery overview — cycle latency, change rate, leader state, force-sync lag.
- Plan health — per-plan health distribution, drift rate, controller tick duration.
- Rollback — inflight rollbacks, success vs failure rate, per-rollback timing.
- Exporter storage — PVC consumption, snapshot count, prune rate.
- Auth — session create / expire, OIDC outcomes, permission resolve latency.
The dashboards are JSON ConfigMaps; install with the chart's
dashboards.enabled=true flag (see Helm values reference once
the public install path lands).
Alerts
telark ships no opinionated alert rules — alerting is your team's prerogative. A reasonable starter set:
telark_discovery_redis_errors_totalrate > 0 over 5 minutes.telark_discovery_leader_is_electedsum across replicas ≠ 1.telark_discovery_rollback_result_total{result="failed"}rate0.
telark_exporter_pvc_bytes_used / pvc_capacity> 80%.
Tracing
OpenTelemetry tracing is not wired today. The structured logs
carry request_id so you can correlate across services manually
if needed.