Real-Time Constraints

The system must guarantee a worst-case execution time (WCET) for critical tasks. This is not about average speed but about absolute predictability.

• Hard vs. Soft Deadlines: A hard deadline (e.g., collision avoidance) cannot be missed without catastrophic failure. A soft deadline (e.g., frame drop in AR) degrades quality but doesn't cause failure.
• On-Device Imperative: For spatial computing, deterministic latency is why processing happens locally. A cloud round-trip of 100+ ms is unacceptable for maintaining a stable AR overlay or robot pose.
• Example: A Visual Inertial Odometry (VIO) pipeline must process camera and IMU frames within a strict 16ms window (for 60 Hz) to prevent perceptual lag and tracking loss.

The system must sustain a consistent frame rate or processing rate under all expected operating conditions. Jitter (variance in latency) is as damaging as high latency.

• Bounded Jitter: Output must be delivered at regular intervals. For 30 FPS video reconstruction, each frame must be processed every ~33ms, not 20ms one time and 45ms the next.
• Resource Reservation: This often requires pre-allocating CPU/GPU/NPU cycles and memory bandwidth to the real-time task, isolating it from other system processes.
• Example: A Neural Radiance Field (NeRF) renderer for interactive view synthesis must maintain a minimum of 30 FPS. Techniques like voxel hashing and model quantization are used to achieve this predictable throughput on edge hardware.

Real-time systems must manage multiple concurrent tasks (sensor reading, processing, output) and be able to preempt (interrupt) lower-priority tasks to service higher-priority deadlines.

• Priority-Based Scheduling: A task with a closer deadline gets immediate access to compute resources. The pose estimation thread will always preempt a background mapping thread.
• Critical Sections: Code that accesses shared resources (like a global pose graph) must be carefully managed with locks or lock-free programming to avoid priority inversion, where a low-priority task blocks a high-priority one.
• Example: In a SLAM system, the tracking thread (high priority, hard deadline) must be able to preempt the local mapping thread (lower priority, soft deadline) to ensure the device's immediate location is always known.

When operating in dynamic, unpredictable environments, the system must handle sensor failures, ambiguous data, and missed internal deadlines without a total crash.

• Fallback Modes: If a primary sensor (e.g., camera) fails, the system should switch to a degraded mode (e.g., inertial-only dead reckoning) and signal the loss of fidelity.
• Input Validation: Algorithms must be robust to outliers and noise. RANSAC is commonly used in feature matching and bundle adjustment to reject erroneous data points.
• Example: An AR system experiencing rapid motion blur may temporarily accept higher pose uncertainty and simplify its rendering, rather than freezing, until visual tracking can be re-established.

The system's internal state (e.g., a 3D map, object positions) must remain consistent across time, even as new, potentially noisy sensor data arrives. This prevents visual flickering in AR or jerky movements in robotics.

• State Estimation Filters: Techniques like the Kalman filter or its non-linear variants (Extended/Unscented Kalman Filters) are used to fuse new measurements with prior beliefs, smoothing out noise.
• Temporal Fusion: In on-device 3D reconstruction, new depth frames are integrated into a Truncated Signed Distance Field (TSDF) over time, averaging out noise to produce a stable surface.
• Example: A dynamic neural scene representation for a moving object must update its latent codes smoothly frame-to-frame to avoid the reconstructed geometry "jumping" or flickering.

The system must monitor its own resource usage (CPU, memory, power) and adapt its algorithms or fidelity to stay within real-time bounds and thermal/power envelopes.

• Dynamic Level of Detail (LOD): A reconstruction system might render nearby geometry in high detail while using a coarse mesh for distant areas to maintain frame rate.
• Algorithmic Switching: It may switch from a dense, accurate Iterative Closest Point (ICP) alignment to a faster, sparse feature-based method when computational load is high.
• Example: A mobile device may trigger thermal throttling. A well-designed real-time system will detect this, reduce the resolution of its neural rendering model via knowledge distillation, and maintain the critical frame rate at the cost of visual quality.

Self-driving cars must process sensor data (LiDAR, cameras, radar) and execute control commands within hard real-time deadlines to ensure safety. A missed deadline in object detection or path planning can result in a collision.

• Latency Requirement: Object detection and classification must typically complete within 50-100 milliseconds.
• Sensor Fusion: Data from multiple sensors must be aligned and processed in a deterministic timeframe to create a coherent world model.
• Control Loop: The entire perception-planning-actuation cycle must operate at a minimum of 10 Hz to react to dynamic road conditions.

AR applications on headsets or smartphones must render virtual objects locked to the real world with imperceptible lag. High latency causes registration errors, where virtual content appears to 'swim' or drift, breaking immersion and causing user discomfort.

• Motion-to-Photon Latency: The total delay from a user's head movement to the updated display must be less than 20 milliseconds to avoid simulator sickness.
• Simultaneous Localization and Mapping (SLAM): The device's pose must be estimated at the camera's frame rate (e.g., 60 Hz).
• Rendering Pipeline: 3D graphics must be rendered at a consistent 60-90 FPS to maintain a smooth, believable experience.

Robotic arms on assembly lines perform precise, high-speed tasks like welding, picking, and placing. They operate under deterministic real-time constraints where a delayed control signal can cause physical damage to the robot, the product, or nearby workers.

• Control Frequency: Joint servo controllers often operate at 1 kHz or higher for precise torque and position control.
• Vision-Guided Robotics: Processing images from guidance cameras must introduce minimal latency to adjust the arm's trajectory in-flight.
• Safety Systems: Safety-rated monitored stop functions must react within milliseconds to sensor inputs indicating a human intrusion.

Trading algorithms analyze market data and execute buy/sell orders in microseconds. In this domain, latency is directly equated to profit loss. Being even a millisecond slower than a competitor can mean missing a lucrative arbitrage opportunity.

• End-to-End Latency: The time from receiving a market data tick to sending an order is often measured in microseconds.
• Deterministic Execution: Trading logic must run on real-time operating systems (RTOS) to avoid non-deterministic garbage collection pauses or context switches.
• Co-location: Servers are physically placed next to exchange servers to minimize network transmission delay, a practice highlighting the extreme nature of the constraint.

Systems like robotic surgical assistants (e.g., da Vinci) and real-time medical imaging (e.g., MRI-guided surgery) impose life-critical real-time constraints. Haptic feedback latency or delayed visualization can severely impact a surgeon's ability to operate safely.

• Haptic Feedback: The force feedback loop must have a latency of < 1 millisecond to provide a realistic sense of touch.
• Image-Guided Surgery: Processing and displaying intraoperative ultrasound or MRI must have minimal lag relative to the surgeon's movements.
• Control Stability: Delays in the robotic control loop can lead to oscillations and instability, posing direct physical risk.

Autonomous drones must process inertial data and stereo vision to maintain stable flight and navigate dynamic environments. A failure to meet control loop deadlines results in a crash.

• Visual Inertial Odometry (VIO): Pose estimation must run at the camera frame rate (200+ Hz for some systems) to prevent state estimation drift.
• Obstacle Avoidance: Processing depth maps to detect and react to new obstacles must occur within a single control cycle.
• Motor Control: Electronic speed controllers (ESCs) adjust motor RPMs at very high frequencies (hundreds of Hz to kHz) to maintain attitude.

Deterministic latency is the guarantee that a system's maximum response time is bounded and predictable, a stricter requirement than simply 'low latency'. It is non-negotiable for safety-critical systems like autonomous vehicle perception or robotic control loops.

• Hard vs. Soft Real-Time: Hard real-time systems (e.g., airbag deployment) fail catastrophically if deadlines are missed. Soft real-time systems (e.g., AR frame rendering) tolerate occasional misses with degraded performance.
• Worst-Case Execution Time (WCET): Analysis to determine the absolute longest time a task can take, considering all possible code paths and hardware states.
• Impact on 3D Reconstruction: For a 60Hz AR experience, the entire pipeline—from sensor capture to pose estimation to mesh update—must complete in under ~16.6ms every frame to avoid judder.

Memory footprint is the total amount of system memory (RAM) consumed by an application, model, and data structures during execution. For on-device 3D reconstruction, managing this footprint is as critical as managing latency.

• Volumetric Representations: Dense TSDF voxel grids are memory-prohibitive. Sparse structures like voxel hashing or octrees are essential for large-scale mapping.
• Model Weights & Activations: A neural network for depth estimation or NeRF rendering must have its parameters and intermediate tensors fit within the device's limited RAM, often requiring model quantization.
• Trade-off: Higher map resolution or more complex neural representations directly increase memory consumption, forcing design compromises between fidelity and feasibility on edge hardware.

The power budget is the strict limit on electrical energy consumption imposed by a device's battery and thermal design. It is the ultimate constraint for mobile and embedded spatial computing systems.

• Compute vs. Battery Life: Continuous sensor streaming (camera, IMU), GPU/NPU inference for neural networks, and CPU processing for SLAM algorithms rapidly drain batteries. Algorithms must be optimized for operations per watt.
• Thermal Throttling: Sustained high compute loads generate heat. If a device exceeds its thermal envelope, the operating system will forcibly reduce processor clock speeds (thermal throttling), violating real-time latency guarantees.
• Design Imperative: Efficient algorithms like Visual Inertial Odometry (VIO) exist partly to provide robust pose estimation using lower-power sensors compared to heavy LiDAR processing.

Computational throughput measures the amount of processing a system can perform per unit of time (e.g., FLOPS - Floating Point Operations Per Second). It defines the upper bound of algorithmic complexity for a given frame time.

• Hardware Dictates Capability: The throughput of a mobile GPU or Neural Processing Unit (NPU) determines what size of neural network can run in real-time. TensorFlow Lite and similar frameworks maximize utilization of this hardware.
• Bottleneck Analysis: In a reconstruction pipeline, throughput must be balanced across stages: feature extraction (< 2ms), pose tracking (< 3ms), depth estimation (< 5ms), surface integration (< 5ms). The slowest stage sets the achievable frame rate.
• Fixed-Point vs. Floating-Point: Using integer arithmetic (INT8) on specialized hardware can provide much higher effective throughput for inference than floating-point (FP32) on a general-purpose CPU.

The accuracy vs. latency trade-off is the fundamental engineering compromise in real-time systems: higher precision typically requires more computation time, directly conflicting with strict latency bounds.

• Examples in 3D Reconstruction:
  • Bundle Adjustment: Provides highly accurate camera poses and 3D structure but is computationally expensive. It is often run as a slower background thread, separate from the real-time Visual Odometry frontend.
  • ICP Iterations: More iterations of the Iterative Closest Point algorithm yield better point cloud alignment but increase latency.
  • Neural Rendering Quality: A NeRF model with more layers or a higher-resolution output buffer produces a sharper image but takes longer to render.
• Adaptive Algorithms: Advanced systems dynamically adjust parameters (e.g., number of features tracked, map resolution) based on available compute headroom to balance this trade-off in real-time.

Jitter is the variation in latency between successive operations or frames. For a system targeting a consistent 30Hz update, jitter is the deviation from the ideal 33.3ms frame time. Low latency is meaningless without low jitter for a smooth user experience.

• Causes: Garbage collection pauses, non-deterministic scheduling by the OS, variable input data complexity (e.g., a featureless wall vs. a textured scene), and memory access contention.
• Impact on AR/VR: High jitter in pose prediction or rendering causes visible stuttering, judder, and can induce simulator sickness. The perceptual impact is often worse than a consistently higher latency.
• Mitigation: Techniques include priority-based thread scheduling, memory pool pre-allocation to avoid runtime allocation, and pipelining stages of the reconstruction process to smooth out per-frame compute workloads.