Snapshots & rollback

Snapshots and rollbacks are part of the operational floor that the protection-plan story sits on top of. They are not the headline feature, but they are how telark turns a classified change into something you can undo.

The flow lives in three places:

• Discovery service — builds snapshot entries on the change cycle, runs the rollback controller that applies manifests.
• Exporter service — owns snapshot persistence (file-system backed) and the Application CRD that holds rollback entries.
• Dashboard UI — features/resources/applications/ surfaces manage-snapshots and manage-rollbacks panels.

Snapshots

What gets snapshotted

A snapshot captures the manifests of an application's tracked resources at the moment a change cycle completes. Discovery groups the items by namespace, applies a stable kind order, and writes one snapshot record per (application, namespace, generation) tuple.

The set of kinds telark tracks for snapshot purposes maps to the admission targets the protection-plan applier also recognises:

• ServiceAccount
• Secret
• ConfigMap
• Service
• NetworkPolicy
• Deployment
• StatefulSet
• DaemonSet
• Job
• CronJob

Plus the operational kinds carried in ApplicationSnapshot records when present in the application's resource inventory (PVCs, Ingresses, HPAs, VPAs).

Snapshot record shape

Every entry on an Application CRD's snapshots array carries:

Snapshots within a single generation can span multiple namespaces when the application crosses namespace boundaries; the rollback controller treats them as one logical set keyed by generation.

Where snapshots live

The exporter service owns the persistence layer. Snapshots are written to a PVC mounted into the exporter pod and addressed by {scope, namespace, generation, id}. The scope query parameter distinguishes application-scoped snapshots (scope=apps) from other scopes the exporter supports.

Reading a snapshot

Summary read (storage telemetry, no manifest body):

GET /snapshots/{id}/get?scope=apps&namespace={ns}&generation={n}

Returns size, consumed bytes, PVC totals, severity, taken-at, path. The dashboard uses this to render the storage rows under an application's Manage Snapshots panel.

Manifest read (the actual YAML/JSON blob):

GET /snapshots/{id}/manifest?scope=apps&namespace={ns}&generation={n}
Accept: application/yaml

Content-negotiated. application/yaml returns the manifest as YAML; anything else (or omitting Accept) returns JSON. The body is the exact set of objects telark captured, in the kind order it will re-apply them.

Aggregate storage telemetry across the cluster:

GET /snapshots/infos

Returns PVC utilisation, snapshot count, and bytes consumed across the snapshot store. Used by the Settings Snapshot storage section and the per-app Storage bar.

Retention

Retention is bounded — five versions per scope by default, pruned automatically as new snapshots land. The default is configurable via the GlobalConfig CRD (see the Reference section once the values reference lands).

Rollback

What a rollback is

A rollback restores an application to a captured generation. The in-cluster rollback controller reads the target snapshot's manifest set and re-applies the manifests in dependency order. There is no direct write path from the API to the cluster — every rollback runs through the controller, which makes status observable end-to-end.

Triggering a rollback

The UI posts to the discovery service:

POST /applications/{name}/rollbacks/trigger
Content-Type: application/json

{
  "snapshotGeneration": 87,
  "triggeredBy": "user-id"
}

Discovery appends a RollbackEntry to the application's CRD with status pending and the target generation. It does not apply anything synchronously — the controller picks the entry up on its next reconcile.

Listing rollbacks

GET /applications/{name}/rollbacks/get
GET /applications/{name}/rollbacks/{rollbackId}/get

The list endpoint returns every RollbackEntry recorded for the application. The single-entry endpoint is used by the dashboard to poll status until the controller finishes.

Rollback entry shape

The controller

The rollback controller (discovery-service/handlers/rollback/) runs a dynamic informer on erpi.telark/v1alpha1/applicationsasresources — telark's application CRD. Every add/update enqueues the object key into a typed rate-limited workqueue. A single worker drains the queue.

On reconcile, the controller:

1. Fetches the application CRD and extracts the spec.
2. Fails any in-progress rollback that has gone stale beyond RollbackProcessTimeout.
3. Picks the next pending rollback entry.
4. Patches its status to in-progress.
5. Loads the manifest set for the target generation from the exporter via the snapshot client.
6. Sorts the manifests by kind order:
   1. ServiceAccount
   2. Secret
   3. ConfigMap
   4. Service
   5. NetworkPolicy
   6. Deployment
   7. StatefulSet
   8. DaemonSet
   9. Job
   10. CronJob
7. Validates the sorted set against the cluster's REST mapper to surface unknown kinds before applying.
8. Applies each manifest using kcoreapply.Apply with retry-aware error handling — transient back-pressure (context.Canceled, client rate-limiter waits) re-enqueues the entry; permanent errors fail the rollback.
9. On success: appends a change-log entry of class rollback, patches status to success with completedAt, dispatches a user notification.
10. On failure: patches status to failed with the error message, dispatches a failure notification.

Notifications

Both success and failure dispatches go through the NotificationClient to the in-app notification stream (Redis pub/sub backed) with the rollback.completed notification type. Metadata carries the application name, application ID, rollback target ID, and final status so the dashboard can deep-link the toast straight to the application detail page.

Status semantics

Failed rollbacks do not attempt to roll forward. The cluster is left in whatever partial state the failure produced. This is intentional — the controller would rather a human inspect the result than silently reverse a half-applied change.

Idempotency

Re-applying the same snapshot is safe. telark uses server-side apply with a stable field manager so re-runs converge on the same manifest set; any object that was already at the target state becomes a no-op.

UI surface

In the dashboard, snapshots and rollbacks live under the Application detail page:

• Manage Snapshots panel — table of every snapshot for the current application, with size, taken-at, severity, and a manifest slide-out that reads /snapshots/{id}/manifest on demand. The Snapshot Compare view diffs two generations side-by-side.
• Manage Rollbacks panel — list of recorded rollback entries with status badges; clicking a pending or in-progress entry opens the live progress view.
• Storage bars — both per-application and aggregate. The aggregate bar reads /snapshots/infos; per-app rows read /snapshots/{id}/get with the application's snapshot generations.

What this is and is not

• Is — the audit trail of cluster state at every classified change, with a deterministic restore primitive sitting on top.
• Is not — a backup product. telark snapshots cover the manifest layer only; volume contents, secrets that resolve via external KMS, and cluster-level state outside the tracked kinds are out of scope. Use Velero or volume-snapshot tooling for the data plane.

Related

• Concepts → Snapshots and retention
• Guides → Performing your first rollback
• Reference → API reference