Scene Graph Generation

Objects are the primary entities in a scene, represented as nodes in the graph. Each object node is typically defined by:

• A bounding box or segmentation mask localizing it in the image.
• A class label (e.g., 'person', 'dog', 'car') from a predefined or open-vocabulary set.
• Optionally, a set of attributes (e.g., 'red', 'large', 'wooden') that describe the object's properties. Object detection and classification models form the foundation for populating these nodes.

Relationships are the directed or undirected edges connecting object nodes, representing their pairwise interactions or spatial configurations. They capture the semantic glue of the scene.

• Predicate: The type of relationship (e.g., 'riding', 'next to', 'holding', 'larger than').
• Subject-Object Pair: The edge connects a subject node to an object node (e.g., <person, riding, bicycle>).
• Relationship detection is often framed as a visual relationship detection task, requiring models to reason beyond individual objects.

Attributes are descriptive properties assigned to object nodes, providing fine-grained detail beyond the base class label. They are key-value pairs attached to nodes.

• Visual Attributes: Describe appearance (e.g., color: 'yellow', material: 'metal', state: 'broken').
• Spatial/Positional Attributes: Describe location or orientation (e.g., position: 'foreground', pose: 'sitting').
• Functional/Emotional Attributes: Describe utility or inferred state (e.g., action: 'running', emotion: 'happy'). Attribute prediction is closely related to visual grounding of adjectives and properties.

The spatial hierarchy organizes objects based on containment and relative positioning, forming a tree-like structure within the graph.

• Part-Whole Relationships: Edges denote that one object is a component of another (e.g., 'wheel' is part of 'car'). This is crucial for detailed 3D scene understanding.
• Relative Layout: Implicitly encodes depth order and proximity (e.g., an object 'on' a table is a child node of the table node).
• This hierarchy supports efficient occlusion reasoning and amodal segmentation by modeling which objects are in front of or contain others.

Global scene context refers to high-level, scene-wide properties that provide a semantic backdrop for all objects and relationships.

• Scene Type/Category: A label describing the overall setting (e.g., 'kitchen', 'street', 'park').
• Global Attributes: Properties applying to the entire image (e.g., 'indoors', 'sunny', 'cluttered').
• This context acts as a prior, guiding the interpretation of ambiguous objects or relationships and is essential for visual commonsense reasoning. It is often represented as a special root node or a global feature vector.

The graph connectivity defines the rules and semantics of how nodes and edges interact, turning a collection of components into a coherent knowledge structure.

• Graph Schema/Ontology: A predefined set of valid object classes, relationship predicates, and attribute types. This enforces consistency and enables neuro-symbolic reasoning.
• Multi-Relational Nature: A single object pair can have multiple relationships (e.g., 'person near car' and 'person looking at car').
• The graph's structure enables complex queries, such as finding all objects with a specific attribute that are involved in a particular relationship, forming the basis for visual question answering and scene graph-based retrieval.

The core computer vision task that directly feeds Scene Graph Generation. It involves detecting pairs of objects in an image and classifying the predicate (e.g., 'riding', 'next to', 'holding') that describes their interaction. While Scene Graph Generation produces a holistic graph, Visual Relationship Detection typically outputs a set of localized subject-predicate-object triplets.

• Key Distinction: Often considered a sub-task or intermediate output for building a full scene graph.
• Example: From an image, detect: <person-riding-bicycle> and <bicycle-next to-tree>.

A unified image segmentation task that provides the foundational pixel-level understanding necessary for accurate scene graph nodes. It requires classifying every pixel with a semantic label (stuff) and assigning a unique instance ID to each countable object (things).

• Role in SGG: Provides precise object masks and categories, which are crucial for generating accurate object nodes (<person#1>, <person#2>) and their spatial attributes in the scene graph.
• Contrast with Instance Segmentation: Panoptic segmentation includes both 'things' (countable objects) and 'stuff' (amorphous regions like sky, grass), offering a more complete scene parse.

Also known as phrase grounding, this is the inverse-localization task. Given a free-form natural language description (e.g., 'the tall man wearing a blue hat'), the model must localize the referred object or region in the image. It tests fine-grained visual-linguistic alignment.

• Relation to SGG: A robust scene graph serves as a structured, queryable representation that could be used to efficiently resolve referring expressions by traversing object and relationship nodes.
• Application: Critical for human-robot interaction and interactive image editing.

A multimodal reasoning task where a model answers a natural language question based on an image's content. Complex VQA often requires relational and compositional reasoning—precisely the information encoded in a scene graph.

• Scene Graphs as Intermediate Representation: Many advanced VQA models explicitly construct or leverage a scene graph to reason about object relationships before generating an answer (e.g., 'What is the person to the left of the bicycle doing?').
• Benchmarks: Datasets like GQA are designed to require scene graph-level understanding.

A foundational end-to-end object detection architecture that has influenced modern Scene Graph Generation models. DETR uses a transformer encoder-decoder to directly predict a set of object bounding boxes and classes, eliminating hand-crafted components like anchor boxes.

• Impact on SGG: Inspired transformer-based SGG models that treat objects and relationships as sets of queries to be decoded in parallel, improving relational reasoning.
• Key Innovation: Uses a bipartite matching loss to assign predictions to ground truth objects, which is adaptable for matching predicted relationship triplets.

An AI paradigm that combines neural networks (for perception, like SGG) with symbolic systems (for logical inference). A generated scene graph acts as the symbolic representation bridging these two worlds.

• Role of Scene Graphs: The graph's nodes (objects) and edges (relationships) form a symbolic knowledge base extracted from pixels. This structured data can then be queried or reasoned over using logical rules or a knowledge graph engine.
• Application: Enables complex query answering ('Find all scenes where a person is interacting with a dog') and consistency checking in visual domains.