Troubleshooting

A catalogue of the failure modes that show up most often, the signal that confirms each one, and the path back to a working state.

If you do not see your problem here, capture the symptom plus the relevant kubectl logs and kubectl get outputs and open an issue (once the GitHub repo is published).

Plan stuck in `scheduled`

Symptom: A plan with timeMode=time_range has passed its startAt but the dashboard still shows phase scheduled.

Confirm:

Check the discovery leader is healthy: telark_discovery_leader_is_elected should be 1 on exactly one replica.
kubectl logs deploy/telark-discovery | grep plan-controller should show ticks at the configured interval (PROTECTION_PLAN_TICK_INTERVAL_SEC, default 31s).

Fix: Wait one tick. If the phase still does not advance, restart the discovery leader (kubectl delete pod -l app=telark-discovery -l role=leader or kill the pod with discovery_leader_is_elected=1). A new leader picks up the controller within the lock TTL.

Plan health `drifted` immediately after activation

Symptom: Phase goes to active, then health flips to drifted within one or two ticks.

Confirm:

Inspect the plan's detail page → Health panel. The healthDetail[] rows show which policy is missing, not ready, or has the wrong failureAction.
kubectl get policies.kyverno.io -A -l telark.io/plan=<plan-id> lists every policy telark deployed for the plan. Compare the set against the plan's renderedPolicies.

Fix: The most common causes:

Policy engine not installed / unhealthy. Reinstall the engine (Kyverno) per the Getting started → System requirements.
Manual edit of a deployed policy. Run Reactivate on the plan — the applier re-renders and re-deploys.
Webhook failure-policy mismatch. Check the chart's kyverno.features.forceFailurePolicyIgnore.enabled is set on the release-manager values.

Rollback stuck in `pending`

Symptom: A rollback was triggered, the entry shows up in the Manage Rollbacks panel, status stays pending indefinitely.

Confirm:

kubectl logs deploy/telark-discovery | grep rollback. The rollback controller's worker should log "picked up pending entry" within a few seconds of the trigger.
telark_discovery_rollback_inflight should be 0 for stuck rollbacks — the controller did not pick them up.

Fix: Restart the discovery leader. If the rollback was written but the controller did not enqueue, the informer cache may be stale. The new leader's informer re-syncs from scratch.

Rollback stuck in `in-progress`

Symptom: A rollback is in-progress for longer than RollbackProcessTimeout.

Confirm:

kubectl logs deploy/telark-discovery | grep rollback shows the apply phase. Most stuck rollbacks are blocked on a single resource — the log will name it.
kubectl describe <kind> <name> -n <ns> on the named resource often surfaces the conflict (admission webhook rejection, immutable field, missing namespace).

Fix: The controller's stale-detection (failStaleInProgress) will fail the rollback at the timeout. If you do not want to wait, restart the discovery leader — the failover triggers re-evaluation. The rollback's error field will explain what broke.

Snapshot PVC is full

Symptom: Snapshots stop being written; alerts on telark_exporter_pvc_bytes_used.

Confirm:

kubectl exec deploy/telark-exporter -- df -h /snapshots (path matches SNAPSHOT_PVC_PATH).
The exporter logs an error on snapshot write when the PVC is out of space.

Fix:

Short-term: lower spec.snapshots.maxPerApp on the GlobalConfig. The next snapshot write triggers a prune.
Medium-term: expand the PVC. Most CSI drivers support in-place expansion; bump the PVC storage request, the driver resizes the volume, the exporter sees the new capacity on the next mount.
Long-term: exclude noisy namespaces via spec.excludedNamespaces so they stop generating snapshots.

Symptom: Sign in with Google redirects to Google, authentication succeeds, the redirect comes back to telark, and then the user sees an error.

Confirm: kubectl logs deploy/telark-auth | grep oidc. Common error strings:

redirect_uri_mismatch — the configured OIDC_GOOGLE_REDIRECT_URL does not match the Authorised Redirect URI in Google Cloud.
email not verified — the user's Google account email_verified claim is false.
state nonce mismatch — the user's cookies are stale or the redirect bounced through a domain that stripped the cookie.

Fix: See Guides → Configuring Google OIDC.

Passkey enrolment fails with "WebAuthn setup failed"

Symptom: The dashboard prompts the user for a passkey, the browser shows a generic WebAuthn error.

Confirm: Check the auth-service env:

WEBAUTHN_RP_ID must equal the effective domain the dashboard is served from (e.g. telark.example.com).
WEBAUTHN_RP_ORIGINS must include the full origin (https://telark.example.com).

A localhost dashboard works with RP_ID=localhost, but production needs the public domain.

Fix: Update the auth-service env, roll the deployment, retry enrolment.

Discovery never picks up new applications

Symptom: Workloads are running in the cluster but the Applications page is empty.

Confirm:

telark_discovery_cycle_duration_seconds should be reporting on the leader. If no samples, the cycle is not running.
kubectl logs deploy/telark-discovery | grep informer should show informer sync.
Check spec.excludedNamespaces on GlobalConfig — if every namespace is excluded, discovery skips them all.

Fix: Verify the discovery ServiceAccount has cluster-wide read on the tracked workload kinds. The Helm chart sets the ClusterRole automatically; out-of-chart installs need to mirror it.

Notifications stop showing up

Symptom: The bell badge in the dashboard stays at zero despite rollbacks completing.

Confirm: kubectl logs deploy/telark-exporter | grep notif should show emit lines per event. If absent, the emit calls never reached the exporter — most often because Redis is unreachable.

Fix: Verify Redis is healthy and the exporter's EXPORTER_REDIS_ADDR resolves. Restart the exporter pod after fixing the Redis path.

Troubleshooting

On this page