Convolutional Neural Network (CNN)

The convolutional layer is the fundamental building block of a CNN. It applies a set of learnable filters (or kernels) to the input. Each filter slides across the input's spatial dimensions, computing a dot product at each position to produce a 2D activation map. This process preserves spatial relationships and enables the network to detect local patterns like edges, textures, and shapes. Key parameters include:

• Kernel Size: The spatial dimensions of the filter (e.g., 3x3, 5x5).
• Stride: The step size with which the filter moves across the input.
• Padding: Adding pixels (often zeros) around the input border to control the output size.
• Number of Filters: Determines the depth of the output volume, with each filter learning to detect a distinct feature.

A pooling layer performs a downsampling operation to progressively reduce the spatial dimensions (height and width) of the feature maps. This serves several critical functions:

• Dimensionality Reduction: Decreases computational load for subsequent layers.
• Translation Invariance: Makes the network less sensitive to the exact position of features.
• Feature Hierarchy: Helps build a spatial pyramid of features, where lower layers detect simple patterns and higher layers detect complex objects.

Common types include:

• Max Pooling: Selects the maximum value from each region of the feature map. This is the most common approach, as it retains the most salient feature.
• Average Pooling: Computes the average value from each region, providing a smoother downsampling.

Activation functions introduce non-linearity into the network, allowing it to learn complex, non-linear mappings. The Rectified Linear Unit (ReLU) is the dominant activation function in modern CNNs for hidden layers.

ReLU is defined as f(x) = max(0, x). Its advantages include:

• Sparse Activation: It outputs zero for all negative inputs, creating sparse representations.
• Mitigates Vanishing Gradient: Unlike saturating functions like sigmoid or tanh, ReLU does not saturate for positive inputs, allowing gradients to flow more freely during backpropagation.
• Computational Efficiency: Involves a simple thresholding operation.

Variants like Leaky ReLU or Parametric ReLU (PReLU) address the 'dying ReLU' problem by allowing a small, non-zero gradient for negative inputs.

Fully Connected (FC) layers, also known as dense layers, are typically placed at the end of a CNN. They take the high-level, spatially-flattened feature maps from the final convolutional/pooling layers and perform global reasoning to produce the final output (e.g., class scores for image classification).

Key Characteristics:

• Global Connectivity: Every neuron in an FC layer is connected to every neuron in the previous layer.
• Classification/Regression: These layers map the learned, distributed features to the desired output space.
• Parameter Intensive: FC layers often contain the majority of a CNN's parameters, making them a primary target for model compression techniques like pruning. In modern architectures, global average pooling is sometimes used to replace large FC layers, reducing parameters and overfitting risk.

Skip connections are a critical architectural innovation that enable the training of very deep networks (e.g., ResNet with over 100 layers). They address the vanishing/exploding gradient problem by creating a shortcut path for the gradient to flow directly through the network.

In a Residual Block, the input to a set of layers is added to the output of those layers: Output = F(x) + x. The function F(x) learns the residual mapping (the change or difference needed), rather than the complete transformation.

Benefits:

• Eases Optimization: The identity shortcut allows gradients to propagate directly, making it easier to train extremely deep networks.
• Mitigates Degradation: Enables networks to maintain or improve performance as depth increases, countering the degradation observed in plain very-deep nets. This concept has been extended in architectures like DenseNet, where each layer receives feature maps from all preceding layers.

Batch Normalization (BatchNorm) is a technique that normalizes the activations of a layer across a mini-batch of training data. It stabilizes and accelerates the training of deep networks.

The operation for a feature x in a mini-batch is: x_hat = (x - μ) / √(σ² + ε), followed by a scale and shift: y = γ * x_hat + β, where γ and β are learnable parameters.

Primary Effects:

• Reduces Internal Covariate Shift: By maintaining consistent mean and variance of layer inputs, it allows for higher learning rates.
• Acts as a Regularizer: The noise introduced by mini-batch statistics has a slight regularization effect, sometimes reducing the need for Dropout.
• Faster Convergence: Networks with BatchNorm typically require fewer training epochs to converge. It is commonly inserted after convolutional/FC layers and before the activation function (e.g., Conv -> BatchNorm -> ReLU).

A Vision Transformer (ViT) is a neural network architecture that applies the transformer model—originally designed for natural language processing—directly to sequences of image patches for tasks like image classification and object detection. Unlike CNNs, which rely on inductive biases for spatial locality, ViTs use self-attention mechanisms to model global dependencies across an entire image from the first layer. While CNNs remain dominant for many dense prediction tasks in robotics due to their efficiency with spatial data, ViTs are increasingly used in vision-language models and for learning highly generalizable representations from massive datasets.

• Key Difference from CNN: Uses global self-attention instead of local convolutional filters.
• Common Use: Foundational backbone in large multimodal models (e.g., for Embodied Vision-Language Models).
• Trade-off: Often requires more data and compute for training than comparable CNNs.

U-Net is a specialized convolutional neural network architecture characterized by a symmetric encoder-decoder structure with skip connections. Originally designed for biomedical image segmentation, its design is now ubiquitous for dense prediction tasks in computer vision, many of which are critical for robotics.

• Encoder: A downsampling path (typically a CNN) that extracts hierarchical features.
• Decoder: An upsampling path that reconstructs spatial resolution for pixel-wise predictions.
• Skip Connections: Direct connections between corresponding encoder and decoder layers that preserve fine-grained spatial information lost during downsampling.

In egocentric perception, U-Net variants are essential for:

• Semantic and Instance Segmentation of a robot's surroundings.
• Monocular Depth Estimation from a single camera.
• 3D Scene Reconstruction tasks.

You Only Look Once (YOLO) is a family of real-time, single-stage object detection algorithms. It frames detection as a unified regression problem, predicting bounding boxes and class probabilities directly from full images in one forward pass of a CNN. This makes it significantly faster than traditional two-stage detectors (like R-CNN). For embodied systems requiring low-latency perception, YOLO variants are a cornerstone technology.

• Architecture: A single CNN simultaneously predicts multiple bounding boxes and class probabilities.
• Speed vs. Accuracy: Optimized for real-time inference, often at a slight trade-off in precision compared to slower models.
• Robotics Application: Critical for real-time object detection from a robot's onboard camera for navigation, manipulation, and human-robot interaction.

Self-supervised learning is a machine learning paradigm where a model learns useful visual representations from unlabeled data. It does this by defining a pretext task that generates automatic labels from the data's structure itself, such as predicting image rotations, solving jigsaw puzzles, or using contrastive learning on different views of an image. This is particularly valuable in robotics, where collecting vast amounts of manually labeled egocentric video is impractical.

• Pretext Task: An auxiliary task solved only to learn good feature representations for a downstream task (e.g., object detection).
• Contrastive Learning: A popular self-supervised method that teaches a model to identify which image pairs are related (positive pairs) and which are not (negative pairs).
• Impact on CNNs: Enables training of powerful CNN backbones (or ViTs) on massive uncurated video datasets from robots, leading to more robust and generalizable perception models.

Domain adaptation is a subfield of transfer learning focused on developing algorithms that perform well on a target data distribution (the target domain) when trained primarily on data from a different but related distribution (the source domain). This is a fundamental challenge in deploying CNNs for robotics, as models trained on curated datasets (e.g., ImageNet) often fail in the messy, changing real world a robot encounters.

• Source Domain: The data distribution used for initial training (e.g., synthetic data from simulation, labeled driving datasets).
• Target Domain: The real-world operational environment (e.g., a specific warehouse floor, a hospital corridor).
• The Reality Gap: The discrepancy between source and target domains. Techniques include:
  • Feature Alignment: Making CNN features from both domains statistically similar.
  • Sim2Real Transfer: Using domain randomization during simulation training to prepare CNNs for reality.
  • Test-Time Adaptation: Continuously adapting a pre-trained CNN using the robot's live sensor stream.

Optical flow is the pattern of apparent motion of image objects, surfaces, and edges between consecutive video frames, caused by the relative movement between an observer (camera) and a scene. Estimating optical flow is a core low-level vision task that provides critical motion cues for robotic navigation and perception. Modern optical flow estimation is predominantly solved using deep convolutional neural networks.

• Output: A dense vector field where each pixel has a 2D displacement vector showing its movement from one frame to the next.
• CNN-Based Methods: Models like FlowNet, PWC-Net, and RAFT use specialized CNN architectures to compute flow more accurately and robustly than classical methods.
• Robotics Applications:
  • Visual Odometry (VO) and Visual SLAM: Estimating robot ego-motion.
  • Obstacle Avoidance: Detecting independent moving objects.
  • Activity Recognition: Understanding agent movements from a first-person view.