Service Discovery

Agents in a fleet must automatically register their presence and capabilities when they come online and deregister when they go offline or fail. This is critical for maintaining an accurate real-time inventory of available resources. In a warehouse, an AMR would register its unique ID, current battery level, payload capacity, and equipped tools (e.g., a lift mechanism). A health monitoring system uses periodic heartbeats to detect unresponsive agents and trigger automatic deregistration, ensuring the load balancer does not assign tasks to unavailable units.

Beyond simple availability, effective service discovery tracks rich metadata about each agent. This allows the orchestration layer to make intelligent assignment decisions. Key metadata includes:

• Capabilities: Can it lift pallets? Does it have a temperature-controlled compartment?
• Status: Current battery charge, operational health (e.g., sensor errors).
• Location: Last known position within the facility map.
• Current Load: Is it already carrying an item? This metadata is essential for algorithms like Weighted Least Connections or capability-based routing within the Dynamic Task Allocation system.

Service discovery implementations follow two primary patterns, each with trade-offs for fleet orchestration.

• Centralized Registry (e.g., a Service Catalog): A single, authoritative system (like Apache ZooKeeper, etcd, or a custom fleet manager) holds the state of all agents. Agents register with this central point. This simplifies state management but creates a single point of failure.
• Decentralized/Gossip-Based: Agents propagate their state to peers using gossip protocols (like SWIM). There is no central coordinator, improving resilience. This is analogous to Anycast routing in networks, where the 'nearest' or most appropriate agent is discovered through peer communication. Hybrid approaches are common in large-scale systems.

Service discovery is the data source for load balancing algorithms. The registry's list of healthy, capable agents is the input pool for algorithms like:

• Least Connections: Assigns the next task to the agent with the fewest active assignments.
• Weighted Round Robin: Distributes tasks based on agent capacity (weight), derived from its metadata.
• Zone-Aware: Prioritizes agents already in the same warehouse zone as the task to minimize travel. Without real-time discovery, load balancing would rely on static configurations, crippling adaptability in a dynamic Heterogeneous Fleet.

Several established protocols facilitate service discovery, each with different operational models.

• DNS-Based Discovery: Agents register SRV or A records. Simple but often lacks real-time updates and rich metadata. Similar to DNS Load Balancing for web services.
• HTTP/REST APIs: Agents POST their state to a registry API. Common in custom orchestration middleware.
• Custom Binary Protocols: Used for high-frequency, low-latency updates in real-time systems, often over UDP.
• Multicast DNS (mDNS)/DNS-SD: Protocols like Bonjour allow agents on the same local network to advertise and discover services without a central server, useful for peer-to-peer fleet communication.

A discovery system must distinguish between a registered agent that is healthy and one that has failed silently. This is achieved through health checks.

• Passive Checks: The registry monitors heartbeat messages. Missing heartbeats trigger a failure state.
• Active Checks: The registry or a sidecar agent proactively pings the agent's API endpoint or checks a system metric.
• Composite Health: Combines multiple signals (network reachability, software process status, hardware sensor readings). Unhealthy agents are marked as such or deregistered, preventing task assignment to a faulty AMR—a direct contributor to Fleet Health Monitoring and system reliability.

In the client-side discovery pattern, the client is responsible for determining the network locations of available service instances and load balancing requests across them. The client queries a service registry, a dedicated database of available service instances, to obtain a current list. It then uses a load-balancing algorithm (e.g., round-robin) to select an instance.

• Primary Components: Service Registry (e.g., etcd, Apache ZooKeeper), Client Logic.
• Advantages: Decouples service discovery logic from infrastructure; allows for intelligent, application-aware client-side load balancing.
• Disadvantages: Couples the client to the service registry; requires discovery logic to be implemented in each client language/framework.

The server-side discovery pattern uses an intermediary component, typically a load balancer or router, to handle discovery. The client makes a request to a well-known endpoint (the load balancer), which queries the service registry and forwards the request to an available service instance.

• Primary Components: Service Registry, Load Balancer/Router (e.g., AWS ALB, NGINX, Kubernetes Service).
• Advantages: Simplifies the client, which requires no discovery logic; allows for centralized management of traffic policies, SSL termination, and health checks.
• Disadvantages: The load balancer becomes a potential single point of failure and a performance bottleneck that must be managed.

The service registry is the central database for service discovery. It maintains a real-time list of service instances, each with its network location (IP address and port) and health status. Services register themselves upon startup and deregister upon shutdown, often sending periodic heartbeats to indicate liveness.

• Key Functions: Service Registration, Health Monitoring, Query API.
• Examples: etcd, Consul, Apache ZooKeeper, Netflix Eureka.
• Design Consideration: The registry itself must be highly available, typically deployed as a clustered, distributed system using consensus protocols like Raft for consistency.

DNS-based service discovery uses the Domain Name System as a simple service registry. Services are registered with DNS records (SRV, A, AAAA). Clients perform a DNS lookup to resolve a service name into a list of IP addresses.

• Protocols: Standard DNS, mDNS (Multicast DNS) for zero-configuration local networks.
• Use Case: Foundational for Kubernetes Services, where a headless Service or a standard Service creates DNS records that pods can query.
• Limitations: DNS caches can introduce delays in propagating instance changes; lacks rich health checking and metadata compared to dedicated registries.

A critical operational component where the service registry or load balancer continuously verifies the health of registered instances. Unhealthy instances are automatically removed from the pool of discoverable endpoints.

• Mechanisms:
  • Heartbeat: The service instance periodically sends a "I'm alive" signal to the registry.
  • Active Probe: The registry or load balancer actively probes the instance's health endpoint (e.g., HTTP /health).
• Outcomes: Failed health checks trigger deregistration, preventing traffic from being routed to failed instances. This is a core mechanism for enabling failover and high availability.

In a service mesh architecture (e.g., Istio, Linkerd), service discovery is abstracted into the data plane. A lightweight proxy (sidecar) is deployed alongside each service instance. These proxies communicate with a centralized control plane, which maintains the service registry and distributes routing rules.

• How it Works: The sidecar proxy automatically handles service registration, discovery, and intelligent load balancing (e.g., least requests, latency-based). The application code is completely decoupled from the discovery logic.
• Benefits: Provides a uniform layer for observability, secure mTLS communication, and traffic management (canary deployments, circuit breaking) across all services, regardless of implementation language.

Health checking is the mechanism by which a service discovery system periodically verifies the operational status of registered service instances, automatically removing unhealthy nodes from the pool.

• Purpose: Prevents traffic from being routed to failed or unresponsive agents.
• Methods: Can be active (HTTP GET, TCP ping) or passive (monitoring for failed client requests).
• Critical for Fleet Ops: A robot with a dead battery or a manual vehicle flagged for maintenance must be marked 'unhealthy' and excluded from task allocation until repaired.

In client-side discovery, the service client is responsible for querying the service registry, obtaining a list of available instances, and using a load-balancing algorithm to select one.

• Architecture: The client contains the logic for service lookup and selection.
• Example Pattern: Used by Netflix Ribbon with Eureka.
• Fleet Analogy: A central task dispatcher (the client) queries the registry for all 'Pick' capable robots, then uses a least connections algorithm to assign the next task.

In server-side discovery, the client makes a request to a dedicated load balancer or router. This intermediary component queries the service registry on the client's behalf and forwards the request to a healthy instance.

• Architecture: Discovery logic is offloaded from the client to a central router.
• Example: An API Gateway or a platform-specific load balancer (e.g., Kubernetes Service).
• Fleet Use Case: A warehouse management system sends a 'move to location' command to a fixed endpoint; the orchestration middleware (the router) discovers an available AMR and routes the command to it.

DNS-based discovery uses the Domain Name System to map a service name to a list of IP addresses. Clients perform a DNS lookup to get the current set of instance addresses.

• Mechanism: SRV records or A/AAAA records are updated dynamically as instances come online.
• Characteristics: Simple, universal, but often has caching delays (TTL) that reduce agility.
• In Distributed Systems: Often used for coarse-grained discovery (finding a data center region) before finer-grained methods (like a service registry) are used within that region.