Distributed Lock

A unique identifier attached to lock acquisition and release commands to prevent duplicate execution during network retries. Critical when lock operations are sent over unreliable transport.

• If a LOCK(resource_A, key=abc123) request times out, the orchestrator can safely retry with the same key
• The lock service recognizes the key and returns the original result without creating a second lock
• Prevents the classic double-lock or double-unlock race condition
• Typically implemented as a UUID generated by the client before the initial request

A periodic signal that an agent sends to the lock service to prove it is still alive while holding a distributed lock. Without heartbeats, a crashed agent would hold a lock indefinitely, deadlocking the fleet.

• The lock is granted with a time-to-live (TTL), e.g., 30 seconds
• The agent must refresh the lock before the TTL expires by sending a heartbeat
• If the orchestrator detects a missed heartbeat, it automatically releases the lock
• Enables liveness—the system can recover from agent failures without manual intervention

A resilience pattern that protects the fleet orchestrator from cascading failures when the distributed lock service becomes unresponsive or degraded.

• Closed state: Requests flow normally to the lock service
• Open state: After a threshold of failures, the circuit breaker trips and immediately fails lock requests without waiting for timeouts
• Half-open state: After a cooldown period, a limited number of test requests probe if the service has recovered
• Prevents resource exhaustion in the orchestrator while the lock backend (e.g., Redis, etcd) recovers

A distributed transaction pattern that coordinates a multi-step operation across agents where each step acquires and releases its own distributed lock. If any step fails, compensating transactions undo the preceding steps.

• Example: An AGV picks up a pallet (acquires pallet lock), then moves to a drop-off zone (acquires zone lock)
• If the drop-off zone is occupied and the AGV cannot proceed, the saga executes a compensating action: release the zone lock and return the pallet
• Each local transaction is paired with a semantic undo operation
• Maintains eventual consistency without long-lived distributed locks that block the entire fleet

An architectural pattern where every lock acquisition, renewal, and release is stored as an immutable event in an append-only log. This provides a complete, auditable history of all critical section access.

• The current lock state is derived by replaying the event stream: LOCK_ACQUIRED → LOCK_RENEWED → LOCK_RELEASED
• Enables time-travel debugging: replay events up to a specific timestamp to reconstruct fleet state during an incident
• The event log serves as a durable audit trail for safety investigations
• Often combined with CQRS (Command Query Responsibility Segregation) for read-optimized lock state queries