Service Discovery Health

The foundational metric indicating whether the service registry itself (e.g., Consul server, etcd cluster, Eureka server) is online and reachable. This is a binary health state: if the registry is down, service discovery fails entirely.

• High Availability (HA) Clusters: Production registries typically run in a clustered mode using consensus protocols like Raft or Paxos to tolerate node failures.
• Quorum Health: A healthy cluster maintains a quorum (a majority of nodes) to accept writes and provide consistent reads. Loss of quorum renders the registry read-only or unavailable.

The correctness and freshness of the data within the registry. This involves verifying that registered services accurately reflect their current network location and metadata.

• Heartbeat Mechanisms: Services or their sidecar agents (like a Consul agent) send periodic heartbeats (TTL-based checks) to confirm they are alive. Missing heartbeats trigger deregistration.
• Anti-Entropy: Processes that reconcile state between different registry nodes or between a service's actual state and its registered state to prevent stale entries.
• Metadata Consistency: Ensuring tags, health status, and version labels attached to service instances are current and accurate for routing decisions.

The system's ability to execute and aggregate health checks defined for each service instance. The registry doesn't just track existence; it assesses operational readiness.

• Check Types: Includes script-based checks (running a local script), HTTP GET requests to a service's /health endpoint, TCP connection attempts, and gRPC health check protocol.
• Status Aggregation: The registry aggregates individual check results into an overall service instance status (e.g., passing, warning, critical).
• Check Deregistration: A critical health check failure can be configured to automatically deregister the unhealthy instance from the registry, preventing traffic from being routed to it.

The registry's resilience to network issues, such as partitions that split the cluster. A healthy service discovery system must handle partitions gracefully to avoid split-brain scenarios.

• Consensus Protocol Health: Monitoring the status of the underlying consensus algorithm (e.g., Raft leader election, term changes, log replication).
• Gossip Protocol Health: In systems like Consul, a gossip protocol manages cluster membership and broadcasts. Health includes monitoring gossip pool convergence and message rates.
• Partition Behavior: Understanding if the registry operates in a availability-favoring or consistency-favoring mode (CAP theorem) during a partition is crucial for health interpretation.

The health of the interfaces through which clients discover services. The primary interfaces are a DNS server and an HTTP API.

• DNS Server Health: The registry often embeds or interfaces with a DNS server. Health involves DNS query latency, success rates, and correctness of SRV, A, or AAAA record responses.
• HTTP API Health: The REST or gRPC API used for programmatic service discovery must be responsive. Metrics include API endpoint latency, error rates (5xx), and connection limits.
• Watch/Stream Health: For clients using long-lived watches (e.g., Consul's blocking queries, etcd's watch), the health of these streaming connections is vital for real-time updates.

The operational state of the integration between the service registry and higher-level orchestration and networking layers. A breakdown here isolates discovery from deployment and routing.

• Orchestrator Integration: Health of controllers that sync pod/IP data from Kubernetes, Nomad, or other orchestrators into the native registry.
• Service Mesh Integration: For meshes like Istio or Linkerd, the health of the component (e.g., Istio's Pilot) that translates registry data into Envoy xDS configuration.
• Load Balancer Integration: Health of the process that populates dynamic load balancer pools (e.g., HAProxy, NGINX) with healthy instances from the registry.