Service Mesh Health

The control plane (e.g., Istiod, Linkerd's destination service) is the brain of the mesh, managing configuration and service discovery. Its failure causes cascading issues.

• Symptoms: Inability to update routing rules, new pods not receiving traffic, stale service discovery data.
• Root Causes: Resource exhaustion (CPU/memory), network partitions isolating control plane pods, storage backend (like etcd) failures for persisted configuration.
• Impact: The data plane may continue routing based on last-known-good state, but the system cannot adapt to changes, leading to traffic blackholes for new services.

The sidecar proxy (e.g., Envoy, Linkerd2-proxy) intercepts all application traffic. Proxy failures directly disrupt communication.

• Common Failure Modes:
  • Crash Loops: The proxy container repeatedly crashes, often due to invalid configuration pushed from the control plane or memory leaks.
  • High Latency: Increased tail latency (p95, p99) due to proxy processing, often from excessive telemetry collection, complex filter chains, or TLS overhead.
  • Out of Memory (OOM) Kills: The proxy is terminated by the kernel for exceeding memory limits, common under high connection or request concurrency.
• Detection: Use liveness probes on the proxy container and monitor for envoy_http_downstream_rq_active gauge spikes.

Service meshes use mutual TLS for secure communication, requiring automatic certificate issuance and rotation. Failures here break all service-to-service traffic.

• The Issue: Certificates have short lifespans (often 24 hours). If the automatic rotation mechanism fails, proxies will reject connections with expired peer certificates.
• Causes:
  • The mesh's internal certificate authority (CA) is unavailable.
  • Network policies block the proxy from the CA service.
  • Clock skew between nodes causes premature validation failure.
• Result: A silent, system-wide outage where services appear healthy but cannot communicate. Monitoring for certificate expiration timestamps is critical.

Mesh configuration (VirtualServices, DestinationRules) is declaratively applied. Failed or partial pushes create inconsistency.

• Failed Push: A new, invalid configuration is rejected by the control plane, leaving the old configuration active. This is a safe failure mode.
• Partial/Staggered Push: A valid configuration is accepted but only propagates to a subset of proxies. This creates split-brain routing where different pods follow different rules, leading to inconsistent application behavior and potential data corruption.
• Configuration Drift: The actual state of proxy configurations diverges from the intended state declared in the Kubernetes custom resources. This requires declarative state verification tooling to detect.

The mesh itself consumes resources. Poorly configured resource limits or traffic patterns can overwhelm it.

• Proxy Resource Limits: Under-provisioned CPU or memory limits cause throttling and OOM kills during traffic spikes.
• Thundering Herd on Startup: When a deployment scales up, hundreds of new proxies simultaneously request configuration and certificates from the control plane, potentially overwhelming it. Startup probes and pod disruption budgets can help stagger initialization.
• Connection Pool Saturation: Proxies maintain pools of upstream connections. A slow or failing upstream service can exhaust all connections in a pool, causing circuit breaker trips and failing healthy requests.

The mesh generates vast telemetry (metrics, logs, traces). The pipeline that processes this data can become a bottleneck.

• Symptoms: High proxy latency correlated with high telemetry generation, gaps in monitoring dashboards, elevated memory use in proxies.
• Causes:
  • Overly verbose access logging.
  • High cardinality metrics labels (e.g., full URLs, user IDs).
  • The telemetry collector (e.g., Prometheus, OpenTelemetry Collector) is unable to keep up with scrape volume.
• Impact: The health-check mechanism (observability) itself degrades the health of the system. Requires tuning sampling rates and label dimensionality.

A dedicated URL (e.g., /health or /status) exposed by a service that returns a standardized HTTP status code and JSON payload indicating its operational health. This is the primary mechanism load balancers, service meshes, and orchestrators use to determine if a service instance should receive traffic.

• Standard Response Codes: 200 OK for healthy, 503 Service Unavailable for unhealthy.
• Payload Structure: Often includes version, timestamp, and status of critical dependencies (database, cache).
• Implementation: A lightweight, internal check that avoids complex business logic to ensure fast response times.

A Kubernetes health check that determines if a container is running. If a liveness probe fails, the kubelet kills the container, and the pod's restart policy takes effect.

• Purpose: Catches deadlocks or states where the process is running but unable to make progress.
• Common Checks: HTTP GET request to a server endpoint, TCP socket connection, or execution of a command inside the container.
• Configuration: Defined by periodSeconds, timeoutSeconds, and failureThreshold in the pod spec.

A Kubernetes health check that determines if a container is ready to accept network traffic. A pod with a failing readiness probe is removed from the Service's load balancer pool.

• Purpose: Ensures a container is fully initialized (e.g., caches loaded, database connections established) before receiving traffic.
• Key Difference from Liveness: Readiness failures do not restart the container; they simply pause traffic until the probe passes.
• Critical for Rollouts: Prevents traffic from being sent to new pods before they are fully ready.

A design pattern that prevents an application from repeatedly trying to execute an operation that is likely to fail. It acts as a proxy for operations, monitoring for failures and "opening" the circuit to fail fast.

• Three States: Closed (normal operation), Open (failing fast, no requests passed), Half-Open (allowing a test request to see if the fault is resolved).
• Service Mesh Integration: Implemented at the proxy level (e.g., Envoy in Istio) to protect services from cascading failures.
• Configuration: Based on failure thresholds, timeout durations, and a reset period.

A deployment strategy where a new version of a service is released to a small, controlled subset of users or traffic. Its health and performance metrics are compared against the stable baseline version before a decision is made to proceed with a full rollout or rollback.

• Service Mesh Role: The mesh's traffic routing rules (e.g., Istio VirtualService) are used to split traffic between the stable (95%) and canary (5%) versions.
• Automated Gates: Analysis of key metrics like error rate, latency, and throughput can automatically promote or reject the canary.
• Progressive Delivery: Part of a broader practice that includes blue-green deployments and feature flagging.

A scripted, automated test that simulates a user's or system's path through an application or API to proactively monitor the health, performance, and correctness of critical business workflows from an external perspective.

• Proactive Monitoring: Runs on a schedule (e.g., every 1 minute) from multiple geographic locations.
• Beyond Ping: Validates multi-step business logic, such as "add item to cart, apply promo code, begin checkout."
• Integration with Observability: Failures and performance degradations from synthetic transactions trigger alerts and feed into SLO/SLI calculations, providing early warning before real users are impacted.