Multi-replica discovery setup
Run two or more discovery replicas safely behind Redis-backed leader election.
telark's discovery service is designed for horizontal scale. Two or more replicas share the work of running informers, classifying changes, and processing force-sync requests — coordinated through Redis.
This guide covers the operational shape and the signals to watch.
What gets coordinated
Discovery runs three things that must not run twice on the same input:
- The change cycle — building the application record from informer state and writing it back through the exporter.
- The snapshot write — persisting the manifest to the PVC.
- The protection-plan controller tick — transitioning plans and reconciling health.
These run only on the elected leader.
It also runs two things that scale linearly:
- The force-sync consumer group — every replica is a consumer of the Redis stream and picks up force-sync markers independently.
- Read traffic — the HTTP API is stateless and answers from the exporter via the rest-pkg client.
Leader election
Discovery uses Redis as the lock store. On startup each replica
calls coordination/leader.Elect with a lock key and a TTL. The
winner refreshes the lock periodically; followers retry on a
backoff.
If the leader pod is killed (drain, OOM, node restart), the lock TTL expires and a follower wins the next election within one TTL. Cycle work resumes on the new leader; in-flight HTTP requests on the old leader's pod fail back to the load balancer to retry.
Force-sync routing
Force-sync uses a Redis stream + consumer group. When the
dashboard's Force sync button posts to
/applications/{name}/sync, the discovery API writes a marker
into the stream. The next available replica in the consumer
group picks it up, runs a one-off discovery cycle for that
application, and acknowledges the message.
This deliberately bypasses the leader so a busy leader cannot block ad-hoc resyncs.
Deployment shape
Run discovery with replicas: 2 (or more). Each replica needs:
- Read-write access to the same Redis instance (used for both leader election and stream consumption).
- HTTP access to the exporter (
EXPORTER_BASE_URL). - Cluster-wide read permission on the workload kinds telark tracks (Deployment, StatefulSet, ConfigMap, Secret, …).
- Cluster-wide write permission on the
kyverno.io/v1/Policyresource (the protection-plan applier needs this).
The Helm chart wires the ServiceAccount + ClusterRole bindings automatically. If you are running outside the chart, mirror those RBAC bindings.
Sizing
A pragmatic baseline:
- 1 replica — fine for clusters under a few hundred workloads. Single point of failure; no HA.
- 2 replicas — the recommended floor for production. Survives a node drain.
- 3+ replicas — useful for very large clusters or for parallel force-sync throughput. The marginal benefit drops off past 3.
CPU is rarely the bottleneck. The constraint is the informer cache footprint (memory) and Redis stream throughput.
Signals to watch
If you scrape the discovery service's Prometheus metrics:
discovery_leader_is_elected— 1 on the leader, 0 on followers. Sum across replicas should always be exactly 1.discovery_cycle_duration_seconds— leader-only. A sudden jump is the canary for cluster scale outgrowing one replica.discovery_forcesync_consumer_lag— per-replica. Growing lag means consumer group bandwidth is saturated.discovery_redis_errors_total— should stay flat. Spikes indicate Redis health or network issues.
Tuning the controller tick
When running multi-replica, the protection-plan controller
tick still runs on the leader only. Lowering
PROTECTION_PLAN_TICK_INTERVAL_SEC makes plan phases react
faster (e.g. endAt → terminated) but increases the load on the
leader. Default of 31 seconds is reasonable; do not lower below 5
seconds without watching the leader CPU.
What multi-replica does not do
- Does not enable multi-cluster. Discovery is still single-cluster, single-tenant. Two replicas share work within one cluster.
- Does not replicate the snapshot PVC. The exporter PVC is one volume; if you need durability beyond the PVC, see Operations → Backups.
- Does not give you zero-downtime upgrades by itself. Helm rolling updates work as expected; the new replica races to win the leader lock when the old one drains.