Watch Mechanism

Upon establishing a watch, the server typically sends a full snapshot of the current state, followed by a stream of incremental updates. This ensures the client's local view is eventually consistent with the source of truth.

• Snapshot + Delta: The initial snapshot bootstrap is critical for recovery after a disconnection.
• Resumability: Watches often support resource versions or sequence IDs, allowing a client to reconnect and request all changes from a specific point, preventing missed updates.

Clients can scope their watch to a subset of resources, reducing noise and bandwidth. Filters are applied on the server side before events are pushed.

• Namespace/Scope: Watch only services within a specific namespace, cluster, or domain.
• Label Selectors: Subscribe only to services matching key-value label pairs (e.g., env=production, version=v2).
• Capability-Based: Filter for services advertising specific interfaces or functional capabilities.

Integrates with the registry's lease mechanism. A watch on a service entry is intrinsically tied to that entry's lease. If the agent fails to renew its lease (heartbeat), the watch fires a REMOVE event.

• Automatic Cleanup: Clients are notified of stale or crashed agents without manual intervention.
• TTL Integration: The watch event stream reflects the time-bound nature of service registrations in dynamic environments.

A robust watch mechanism provides guarantees about the order and uniqueness of delivered events to prevent client state corruption.

• Ordering: Events for a single resource are delivered in the order they occurred. Cross-resource ordering may be looser.
• At-Least-Once Delivery: The system guarantees no event is silently dropped, though duplicates may occur during retries.
• Idempotent Handling: Client logic should be designed to handle potential duplicate events gracefully.

A lease mechanism is a time-bound grant of registration in a service registry that must be periodically renewed by an agent, usually via a heartbeat. It provides a deterministic way to clean up failed agents without manual intervention.

• Creates ephemeral nodes in the registry.
• The watch mechanism notifies clients when a lease expires.
• Fundamental to systems like Kubernetes and Consul for maintaining cluster state.

A health check is a periodic probe sent to an agent to verify its operational status and availability for receiving requests. While heartbeats prove liveness, health checks validate functional readiness, and failures can propagate via the watch mechanism.

• Can be active (HTTP GET, TCP ping) or passive (monitoring request success rates).
• Results update the agent's status in the registry.
• Clients watching the registry can avoid routing to unhealthy instances.

A service mesh is a dedicated infrastructure layer for handling service-to-service communication, providing service discovery, load balancing, and security. It typically implements a watch mechanism internally to keep its data plane (proxies) updated with the latest service topology.

• Envoy Proxy uses a gRPC stream (a form of watch) to receive dynamic configuration from a control plane.
• Istio and Linkerd abstract service discovery and health monitoring away from application code.
• The mesh manages watch subscriptions and updates for all services.

Client-side discovery is a pattern where the service consumer (client) is responsible for querying a service registry and load balancing requests among available instances. A watch mechanism is crucial here to keep the client's local cache of service endpoints fresh and avoid stale routing.

• The client embeds the discovery logic and maintains a dynamic endpoint list.
• Reduces network hops compared to server-side discovery.
• Requires robust client libraries to handle watch connections and reconnection logic.