Turn-Taking

A Turn-Transition Relevance Place (TRP) is a point in an interaction where a turn change is syntactically, semantically, or pragmatically possible. Robots identify TRPs using multimodal cues:

• Prosodic cues: Falling intonation or pauses in speech.
• Linguistic cues: Grammatical clause boundaries or completion of semantic units.
• Visual cues: Gaze shifts away from a partner or cessation of a manual action.
• Contextual cues: Completion of a predefined subtask in a collaborative workflow. Detection algorithms fuse these signals to predict the optimal moment for the robot to take or yield the turn.

Backchanneling refers to brief, non-interruptive vocalizations or gestures (e.g., "mm-hmm," head nods) that signal active listening and encourage the current speaker to continue. In HRI, robots employ backchannels to:

• Regulate flow: Provide feedback without seizing the turn.
• Build rapport: Demonstrate engagement and understanding.
• Signal comprehension: Use context-appropriate continuers after key information points. Systems typically trigger backchannels based on pauses, gaze from the human, or the completion of semantic chunks, using models trained on human conversational data.

Gaze is a primary turn-yielding signal. Robots use gaze estimation to infer human attention and coordinate turn exchanges:

• Turn-Yielding: A human looking at the robot during a TRP signals an invitation for the robot to take the turn.
• Turn-Holding: A human looking away while speaking often signals they are not yet finished.
• Mutual gaze: Establishes the engagement necessary for a smooth handoff. Robots control their own gaze (e.g., of a robotic head or eyes) to signal intent: looking at the human when yielding, looking away when processing or holding the floor.

This mechanism governs how the next speaker is selected. In HRI, allocation can be:

• Current-Speaker Selects Next: The robot, while speaking, can explicitly select the human via gaze, deictic gesture (pointing), or verbal address ("What do you think?").
• Self-Selection: The robot must decide when to proactively take the turn, often using action anticipation to interject at a non-disruptive moment.
• Sequence Completion: In task-oriented settings, turns are allocated by the workflow itself (e.g., the robot completes a screw placement, creating a TRP for the human to perform the next assembly step).

Overlap (simultaneous speech/action) is common in human interaction and requires sophisticated management in HRI. Mechanisms include:

• Discrimination: Classifying overlap as cooperative (backchannel) versus competitive (interruption).
• Resolution Policy: The robot may stop speaking (if interrupted), increase volume/pitch (to hold the floor), or employ a gesture hold signal.
• Repair Sequences: After an interruption, the system must track dialogue state to resume or clarify, often using a short prompt ("As I was saying..."). These policies prevent breakdowns and mimic the fluid negotiation of human conversation.

A Turn-Constructional Unit (TCU) is the fundamental segment of interaction that constitutes a turn (a sentence, a gesture, an action). In embodied HRI, TCUs are multimodal:

• Linguistic TCU: A spoken phrase.
• Gestural TCU: A pointing or iconic gesture.
• Action TCU: A physical movement like handing over a tool. Robots must segment continuous sensor input into these units, understand their completion points (TRPs), and generate multimodal TCUs that are interpretable as complete turns by the human partner. This requires tight integration of natural language processing, computer vision, and motion planning.

A human provides a high-level command (e.g., "Hand me the wrench"), and the robot must signal its turn to request clarification or confirm understanding before acting. This prevents errors from ambiguous instructions.

• Turn-Yielding Cue: Human pauses after speaking.
• Turn-Taking Cue: Robot uses a backchannel signal (e.g., a head nod or a light blink) to acknowledge, then asks, "Do you mean the adjustable wrench on the bench?"
• Example: In assembly tasks, this explicit verbal exchange ensures tool retrieval accuracy before the physical handover turn begins.

During a co-assembly task, the human and robot must coordinate the physical exchange of components. Smooth turn-taking prevents collisions and maintains workflow rhythm.

• Turn-Yielding Cue: The human finishes tightening a bolt and moves their hand away from the workspace, signaling readiness for the robot's turn.
• Turn-Taking Cue: The robot's vision system detects the cleared space and the human's averted gaze, then moves in to place the next component.
• The handover itself is a micro-turn-taking event, governed by force sensing and gaze coordination to signal 'release' and 'accept'.

In therapeutic scenarios, such as with children with autism spectrum disorder (ASD), robots use structured turn-taking to teach social interaction norms.

• The robot initiates a game (e.g., a building block game) and uses clear prosodic cues (vocal inflection) and gestural cues (pointing) to indicate the child's turn.
• It then yields by becoming still and directing its gaze toward the child, waiting for a response.
• This predictable, rule-based exchange helps users learn the patterns of social dialogue, with the robot providing consistent, patient feedback.

Advanced HRI systems fuse multiple cues to detect turn boundaries robustly, mimicking human conversational fluency.

• Cue Fusion: The system combines:
  • Linguistic: End-of-utterance detection and falling intonation.
  • Visual: Head pose, eye gaze, and gesture completion.
  • Proxemic: Changes in interpersonal distance.
• Example: A receptionist robot detects a visitor has finished speaking (audio), leans back slightly (pose), and looks expectantly (gaze). The robot integrates these signals to determine it is now its turn to respond with directions.

In robot-assisted surgery, turn-taking manifests as dynamic control authority shifts between the surgeon and the robot's autonomous capabilities.

• The surgeon performs a delicate dissection (human's turn).
• The robot, via intent recognition, anticipates the need for steady retraction. It signals (via a visual overlay) it can take over this stabilizing subtask.
• The surgeon yields control with a micro-joystick release or voice command ("Stabilize").
• The robot executes the retraction (robot's turn), freeing the surgeon to focus on the next step. This creates a tightly coupled action-response loop.

When a robot fails at a task, turn-taking protocols manage the repair sequence to re-establish collaboration.

1. Robot Failure: The robot drops a part (break in its turn).
2. Yielding: The robot signals failure through a distinct audio tone and orients its sensors away from the workspace, yielding the floor.
3. Human Takeover: The human intervenes, retrieves the part, and places it in a designated 'recovery' location.
4. Yielding Back: The human steps back and gives a verbal "Okay" or a clear gesture, signaling the robot to resume its turn. This structured repair prevents confusion and ensures task continuity after errors.

The computational process by which a robot infers a human's immediate goals or planned actions. This is a critical prerequisite for initiating a turn.

• Inputs: Observed behavior, contextual cues, interaction history.
• Methods: Often uses sequence models (e.g., LSTMs, Transformers) on fused sensor data.
• Example: A robot observing a human reaching for a tool infers the intent to use it, signaling the robot's turn to hand over the tool or move out of the way.

The task of predicting a future action or sequence from partially observed data. This enables a robot to prepare its response before a human's turn is fully complete, reducing latency in the exchange.

• Key Distinction: Goes beyond recognizing the current action to forecasting the next one.
• Application: Used in turn-taking to pre-compute the robot's next move (e.g., beginning to reach for an object the human is about to finish using).
• Models: Often based on spatiotemporal graph neural networks or vision transformers.

A control paradigm where authority is dynamically allocated between a human and an autonomous controller. Turn-taking is a high-level protocol that orchestrates these shifts in control.

• Mechanism: Uses blending or ** arbitration** algorithms to combine human inputs with autonomous plans.
• Turn-Taking Role: Determines when control shifts (the "turn") based on confidence, task phase, or explicit cues.
• Example: In assisted driving, the system takes a turn during lane-keeping, then cedes control back to the human for complex maneuvers.

The study of interpersonal space and social-spatial norms. In HRI, it governs the physical distances and orientations that signal turn-taking readiness or completion.

• Zones: Intimate, personal, social, and public distances adapted for robots.
• Turn-Taking Signal: A robot moving into a human's personal space may signal a request to take a turn in a collaborative task.
• Violations: Incorrect proxemic behavior can disrupt turn-taking fluency, making the robot seem intrusive or unresponsive.

The process of determining a person's point of regard by analyzing eye and head pose. Gaze is a fundamental turn-yielding and turn-requesting cue in human conversation and HRI.

• Turn-Yielding: A human looking at the robot after speaking often signals the end of their turn.
• Turn-Requesting: A robot directing its gaze (via cameras or animated eyes) at a human can signal a desire to initiate its turn.
• Technical Approach: Uses convolutional neural networks on eye region images, often combined with head pose from a 3D sensor.

Robots that provide aid through social interaction rather than physical contact. Turn-taking is a foundational skill for SAR, enabling therapeutic dialogue, tutoring, and coaching.

• Applications: Autism therapy, cognitive rehabilitation, eldercare companionship.
• Turn-Taking Focus: Requires nuanced understanding of verbal and non-verbal cues to maintain engagement and pace interactions appropriately.
• Challenge: Must adapt turn-taking rules to the user's cognitive and emotional state, which may differ from neurotypical norms.