Exteroceptive Sensing

Sensor fusion is the process of combining data from disparate exteroceptive sensors to form a more complete, accurate, and reliable model of the environment than any single sensor could provide. Common architectures include:

• Early Fusion: Raw data from different modalities is combined at the input level to a neural network.
• Late Fusion: Each modality is processed independently, and high-level features or decisions are merged.
• Mid-Fusion: Features are extracted from each modality and then aligned and combined at an intermediate representation level. This is critical for overcoming individual sensor limitations (e.g., camera darkness, LiDAR rain noise) and is the backbone of robust 3D scene understanding and real-time robotic perception.

Proprioceptive sensing is a robot's ability to sense its own internal state—such as joint angles, motor currents, link torques, and actuator positions—without external references. It provides the kinesthetic awareness necessary for closed-loop motor control. While exteroceptive sensing answers "What is around me?", proprioception answers "Where are my own limbs?" and "What forces am I exerting?".

• Key sensors: Encoders, resolvers, torque sensors, inertial measurement units (IMUs), and motor current sensors.
• Primary function: Enables precise joint-level control, gravity compensation, and internal force regulation.
• Integration with exteroception: A robust manipulation system fuses proprioceptive and exteroceptive data to create a complete state estimate, essential for tasks like impedance control or recovering from occluded vision.

Tactile servoing is a closed-loop control method that uses real-time tactile sensor feedback to guide robotic manipulation. Instead of relying solely on vision (exteroception), it uses direct contact sensations—like pressure distribution, shear forces, and micro-vibrations—to servo the end-effector. This is critical for contact-rich tasks where vision may be obstructed.

• Core principle: The controller minimizes an error function defined in tactile feature space (e.g., desired vs. actual contact pattern).
• Common applications: Inserting a peg into a hole, following a contour, maintaining stable grip force, and in-hand manipulation.
• Sensor types: Often employs high-resolution tactile arrays like GelSight or BioTac sensors that provide rich spatial contact data.

Visual servoing is a robot control technique that uses feedback from a vision sensor to directly control the motion of the robot's end-effector toward a desired pose relative to a target. It is a prime example of tightly coupling exteroceptive sensing with action.

• Image-Based Visual Servoing (IBVS): Controls robot motion to minimize error directly in the image feature space (e.g., pixel coordinates of target points).
• Position-Based Visual Servoing (PBVS): Uses a reconstructed 3D pose of the target from camera images to compute error in Cartesian space.
• Advantage: Provides robustness to calibration errors and modeling inaccuracies by closing the loop on perception.
• Use case: Precisely aligning a gripper with a moving object on a conveyor belt.

6D pose estimation is the computer vision task of determining the three-dimensional position (X, Y, Z) and three-dimensional orientation (roll, pitch, yaw) of an object relative to a camera. It is a critical preprocessing step that transforms raw exteroceptive data (pixels) into a actionable geometric state for manipulation planners.

• Input: Typically a single RGB image or an RGB-D (depth) image.
• Output: A 3D translation vector and a 3D rotation matrix (or quaternion).
• Methods: Range from classical point-pair feature matching to modern deep learning approaches like PoseCNN or DenseFusion.
• Manipulation role: Provides the target pose for grasp planning, bin picking, and assembly tasks. Accuracy directly impacts the success of pick-and-place operations.

Contact-implicit trajectory optimization is an advanced planning method that optimizes robot motions without pre-specifying contact sequences. The solver simultaneously discovers when, where, and how contacts should occur between the robot and the environment. This is essential for planning dexterous, non-prehensile, or dynamic manipulations where contact is fundamental.

• Core challenge: Contacts introduce discontinuous dynamics (making/breaking) and combinatorial complexity.
• Solution approach: Formulates contact forces as decision variables subject to complementarity constraints (e.g., the robot can only push, not pull, on a surface).
• Relation to exteroception: The optimized plan assumes a model of the environment (object geometry, friction), which is derived from exteroceptive sensing. It answers "How should I move, including making and breaking contact, to achieve my goal?"

A world model is a learned or engineered compact representation of an environment that enables prediction and planning. It is the internal "mental simulation" an agent uses to reason about the outcomes of its actions. For dexterous manipulation, this model must integrate exteroceptive observations into a useful state.

• Purpose: To compress high-dimensional sensor data (e.g., pixels from a camera) into a lower-dimensional latent state that captures task-relevant information.
• Types: Can be explicit (e.g., a list of object poses and properties) or implicit (a neural network latent vector).
• Function: Enables model-based reinforcement learning and planning algorithms to predict future states without interacting with the real world. It bridges the gap between raw exteroceptive sensing and long-horizon decision-making.