Visual Commonsense Reasoning

VCR models must integrate implicit world knowledge not directly depicted in the image. This includes understanding physical laws (e.g., gravity, object permanence), social norms (e.g., typical human behavior in a setting), and functional affordances (e.g., a chair is for sitting).

• Example: Given an image of a person holding an umbrella under a clear sky, a VCR model might infer the person expects rain, requiring knowledge of umbrellas' purpose and weather forecasting.
• This is distinct from Visual Question Answering (VQA), which often focuses on explicit visual facts (e.g., 'What color is the umbrella?').

A hallmark of VCR is the ability to reason about cause and effect and consider counterfactual scenarios. This involves predicting likely causes of a scene or imagining plausible alternative outcomes.

• Causal: 'Why is the person running?' → 'To catch the departing bus.'
• Counterfactual: 'What would happen if the glass were pushed off the table?' → 'It would shatter on the floor.'
• This requires building an internal mental model of the scene that supports simulation of events, moving beyond static pattern recognition.

VCR often requires inferring past events or predicting future states from a single static image, constructing a narrative timeline.

• Pre-event: 'What likely happened just before this photo?' (e.g., a cake was cut).
• Post-event: 'What will happen next?' (e.g., the ball will fall into the cup).
• This capability is foundational for tasks like Embodied Question Answering (EQA) and action prediction, where agents must understand dynamics from visual snapshots.

Questions require parsing complex compositions of objects, attributes, and spatial or semantic relationships. This tests compositional generalization—the ability to understand novel combinations of known concepts.

• Example: 'Is the person to the left of the bicycle likely the owner?' requires understanding spatial relations ('left of'), object ownership norms, and human intent.
• Models often rely on intermediate structured representations like scene graphs to explicitly model objects (nodes) and their relationships (edges) for this reasoning.

Advanced VCR involves theory of mind—attributing mental states like goals, beliefs, and emotions to agents in the scene. This is critical for understanding social interactions.

• Goal-oriented: 'What is the woman trying to achieve by reading the map?' → 'She is likely lost and trying to navigate.'
• Emotional State: 'How is the child feeling?' → 'Frustrated, because the toy is stuck.'
• This moves reasoning from the physical domain (what is) to the psychological domain (why and how it feels).

VCR is rigorously evaluated by specialized datasets designed to probe specific reasoning gaps. Performance is measured by accuracy on Q->A (answer selection) and QA->R (rationale selection) tasks.

• VCR Dataset: The seminal benchmark containing 290k multiple-choice questions where models must select an answer and then a justifying rationale.
• VisualCOMET: Focuses on inferring past and future events relative to an image.
• SWAG-Visual: Tests grounded situational plausibility.
• High performance requires overcoming language priors (guessing from question text alone) and demonstrating genuine visual grounding.

This task requires inferring the goals, desires, or future actions of agents in a scene based on visual context and social norms.

• Example: An image shows a person holding an umbrella while looking at dark clouds. The question "Why is the person holding the umbrella?" tests understanding of preventative action based on weather prediction.
• Core Challenge: The model must link the visual cue (dark clouds) to the commonsense knowledge that dark clouds often precede rain, and that umbrellas are used for protection from rain, to infer the person's intent to stay dry.

Here, the model must predict the physical or social outcome of an event depicted in an image, relying on an understanding of cause-and-effect relationships.

• Example: An image shows a ball rolling off a table. The question "What will happen next?" requires knowledge of gravity and object permanence to predict the ball will fall to the floor and likely bounce or roll.
• Core Challenge: The system must simulate a short-term physical future state based on universal laws, not just describe the static scene.

This involves deducing what must have happened before the captured moment or the hidden properties of objects and scenes.

• Example: An image shows a person with wet hair and a towel. The question "What probably happened before this photo?" tests knowledge that wet hair often results from activities like showering or swimming.
• Core Challenge: Reasoning backwards from effects to likely causes, often involving occluded events or objects (e.g., inferring the existence of a shower outside the frame).

Tasks assess the model's ability to interpret social interactions, relationships, and the emotional states of people in an image.

• Example: An image shows two people facing each other with one offering a gift. The question "How is the person receiving the gift likely feeling?" requires understanding social rituals and typical emotional responses (e.g., happiness, surprise).
• Core Challenge: Interpreting subtle cues like body language, facial expressions (even if blurred), and contextual objects to attribute mental states.

This requires understanding the intrinsic properties of materials and objects, and what actions they enable (affordances).

• Example: An image shows a glass cup falling toward a tile floor. The question "Is the cup likely to break?" requires knowledge that glass is brittle, tiles are hard, and impact between them often causes shattering.
• Core Challenge: Moving beyond object recognition to reasoning about material composition, fragility, weight, and the consequences of physical interactions.

These questions test implicit knowledge about how objects are typically arranged in space and used for their intended functions.

• Example: An image shows a bookshelf with books placed horizontally, stacked on top of each other. The question "Is this an efficient way to store these books?" requires knowing that books are typically stored vertically on shelves to easily read spines and maximize space.
• Core Challenge: Leveraging normative knowledge about object use and organization to evaluate the rationality or atypicality of a scene's configuration.

Visual entailment is a multimodal reasoning task that determines if a given textual hypothesis can be logically inferred (entailed) from the visual information present in an image. It frames reasoning as a three-way classification problem: entailment, contradiction, or neutral.

• Task Structure: Given an image and a text hypothesis, the model must judge the truth of the statement based solely on visual evidence.
• Example: An image shows a person holding an umbrella under clear skies. The hypothesis "It is raining" would be classified as a contradiction.
• Relation to VCR: It is a formal, logic-based subset of VCR, focusing on deductive inference rather than open-ended question answering.

Scene Graph Generation is the task of parsing an image into a structured graph representation where nodes are objects and edges are their pairwise relationships or attributes. This creates a symbolic, machine-readable summary of the scene's contents and interactions.

• Graph Structure: Nodes represent object entities (e.g., person, dog, frisbee). Edges represent predicates (e.g., person -> walking -> dog, dog -> chasing -> frisbee).
• Foundation for Reasoning: The generated scene graph serves as an explicit knowledge base for downstream reasoning tasks, including VCR. A VCR model might infer that "the dog is tired" by analyzing the graph for activities like chasing over time.
• Limitation: Scene graphs capture explicit, visually present relationships but not the implicit commonsense knowledge (e.g., frisbee is for play, play can cause tiredness) required for full VCR.

Neuro-symbolic reasoning is an AI paradigm that combines the pattern recognition strength of neural networks (the "neuro" component) with the explicit, logical rules and knowledge representation of symbolic AI systems (the "symbolic" component).

• Hybrid Architecture: A neural network (e.g., a vision transformer) processes raw pixels to extract entities and features. A symbolic reasoner (e.g., a logic programming engine) applies rules of physics, causality, or social norms to draw conclusions.
• Application to VCR: This approach is particularly promising for VCR, as it can integrate hard-coded commonsense knowledge bases (e.g., ConceptNet, Cyc) with flexible visual perception. For example, a neuro-symbolic system might use a neural net to identify a wet street and a symbolic rule (if wet_street then possibly recent_rain) to answer "Why is the street shiny?"
• Advantage: Offers improved interpretability and robustness by separating learned perception from deterministic reasoning.

Compositional generalization is the ability of a model to understand and combine known concepts (e.g., objects, attributes, relations) in novel ways to interpret or generate new, unseen compositions. It tests a model's capacity for systematic reasoning beyond memorized patterns.

• Core Challenge: A model might correctly understand "red cube" and "blue sphere" but fail on the novel combination "blue cube" if it hasn't seen that specific pairing during training.
• Critical for VCR: VCR questions often involve novel compositions of familiar elements (e.g., "Why might the person be putting a book under the table leg?" combines book, table leg, and the support relation under for the novel purpose of stabilization). A model lacking compositional generalization will fail on these queries.
• Evaluation: Benchmarks like CLEVR or GQA often include splits specifically designed to test for this capability.

Visual dialog is a multimodal task where an AI agent holds a multi-turn, conversational dialogue about an image, answering a sequence of questions that may depend on the entire dialog history.

• Dynamic Context: Unlike single-turn VQA, each question in a visual dialog must be interpreted in the context of previous questions and answers (e.g., Q1: "What color is the car?" A1: "Red." Q2: "Is it convertible?" requires linking "it" back to the red car).
• Implicit Reasoning: Dialog often necessitates VCR-style inference. A question like "Would the driver need sunglasses?" requires reasoning about the time of day (inferred from shadows), the car's roof type (from Q2), and commonsense about sun glare.
• Dataset: The VisDial dataset is a prominent benchmark for this task, where models must rank a set of candidate answers for each dialog turn.

Embodied Question Answering is a task where an AI agent must actively navigate within a simulated 3D environment (e.g., a house) to gather visual information necessary to answer a natural language question.

• Active Perception: The agent cannot see the entire scene at once. It must execute a sequence of actions (e.g., move forward, turn left, look up) to explore the environment and find relevant visual evidence.
• Commonsense Navigation: Questions often require VCR for both the final answer and the navigation policy. "What is in the bedroom on the second floor?" requires commonsense knowledge of typical house layouts to find the stairs and the bedroom.
• Bridge to VCR: EQA extends VCR from passive image analysis to active, sequential perception in a spatially extended world, making implicit knowledge about object affordances (e.g., doors can be opened) and room purposes critical.