This work introduces a novel multi-modal orchestration framework for semantic audio-driven humanoid control, enabling robots to autonomously select and execute appropriate motion skills in real time.
Abstract
Recent advances in humanoid robotics and reinforcement learning have enabled the acquisition of highly expressive whole-body motion policies. However, most robotic performances remain based on pre-scripted sequences or externally triggered behaviors, limiting autonomy and responsiveness to dynamic environments. In this work, we introduce a novel multi-modal orchestration framework for semantic audio-driven humanoid control, enabling robots to autonomously select and execute appropriate motion skills in real time. The system processes continuous audio streams and routes them into music or speech branches. Music input is handled via audio fingerprinting and semantic embeddings to retrieve track identity and temporal alignment, allowing dynamic mapping between musical segments and motion policies. Speech input is grounded into a discrete library of imitation-learned skills, enabling direct human-robot interaction. Both modalities share a unified interface that schedules skill execution over a reinforcement learning control pipeline. We validate the approach in simulation and on a Unitree G1 humanoid, showing robust sim-to-real transfer and consistent audio-conditioned policy selection. Supplementary materials are available at the following site: https://lab-rococo-sapienza.github.io/semantic-WBC/
The quest for intuitive and natural human-robot interaction (HRI) remains a significant challenge in robotics. Traditional methods often rely on rigid, pre-programmed commands that limit the robot's expressiveness and adaptability. This paper introduces a novel framework that leverages the reasoning capabilities of Large Language Models (LLMs) to synthesize complex robotic actions from a rich tapestry of multimodal human inputs: natural speech, hand gestures, and music/sound beats. Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection. These processed inputs are contextualized using prompt templates and fed into a LLM. The LLM, informed by a predefined robot action space, reasons over the combined inputs to generate a coherent sequence of actions. This sequence is dispatched to an action queue for execution on a quadruped robot over ROS. The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music. This work represents a step towards creating robots that can interact with humans in a more fluid, creative, and context-aware manner.
Snehasis Banerjee, R. Dasgupta· arXiv.org· 0 citations
Learning from demonstration (LfD) provides a developmental framework through which robots can develop motor skills by observing and imitating human dynamics, reducing reliance on explicit programming to teach a skill to a robot. The resulting human-like robot motion is recognised as a key factor in building trust and enabling natural collaboration in human-robot interaction. This paper presents a framework for learning human-like robot motion from demonstration, including data collection, probabilistic trajectory learning, and perceptual user evaluation. A dataset of 3,142 handwriting demonstrations was collected from 22 participants across all 52 Latin alphabet character-case combinations via a touchscreen teleoperation interface, capturing planar position, contact force, and timing. Building on the widely used Gaussian Mixture Model and Gaussian Mixture Regression approach for learning from demonstration, the framework is extended in this work by incorporating force and normalised time dimensions to enable richer representation of human dynamics, and adapting it to handle non-continuous, multi-segment trajectories, enabling generalisation across demonstrations. A user study with 21 participants evaluated the perceived human-likeness of the generated trajectories using a continuous scale anchored between robotic and human-like motion, normalised to 0-100 where 50 represents the neutral midpoint. The generated trajectories achieved an overall human-likeness score of 71.50 (SD=22.56), indicating that the majority of trajectories were perceived as more human-like. Participants identified geometric positioning and trajectory sequence as the most influential perceptual factors, and reported positive attitudes toward human-like robot behaviour. The datasets are released as open-source, providing a reproducible benchmark for developing and evaluating human-like robot motion methods.
Alperen Kenan, Paul A. Bremner, Manuel Giuliani· 1 citation
Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics. While recent offline motion generation approaches offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings. Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows. In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints. ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning. We propose a two-stage autoregressive transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints. By training on a large-scale motion capture dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses, ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations on the HumanML3D benchmark and the large-scale, high-fidelity Bones Rigplay dataset demonstrate ARDY's high motion quality and constraint adherence, validating the efficacy of our key architectural decisions. Finally, we demonstrate the method's practical versatility through an interactive demo featuring dynamic text control, diverse keyframe pose constraints, path following, and interactive locomotion control via mouse and keyboard. Supplementary video results, code, and model releases can be found at https://research.nvidia.com/labs/sil/projects/ardy/.
Kaifeng Zhao, Mathis Petrovich, Haotian Zhang et al.· ACM Transactions on Graphics· 1 citation
Reinforcement learning can produce robust humanoid controllers, but each new task is typically trained as a separate policy with its own reward design and training process. Motion imitation provides an alternative source of motor competence by training policies to track retargeted human motions, yet the resulting controllers remain reference trackers and are not directly usable as task policies. We propose a three-stage pipeline that turns motion-imitation skills into a reusable hybrid motion prior (HMP) for humanoid locomotion. First, an expert policy is trained to imitate retargeted human motion-capture clips. Second, the expert is distilled into a frozen architecture composed of a proprioceptive encoder, a residual vector-quantized (RVQ) codebook, and an action decoder. Third, task-level policies are trained to solve locomotion tasks by selecting discrete codebook entries while the HMP remains frozen. We evaluate the method on velocity tracking, point-goal navigation, and fall-recovery velocity tracking in simulation, and deploy the velocity-tracking policy on a real Unitree G1 robot. The distillation process preserves the tracking behavior of the expert, while the resulting HMP can be reused without retraining as the action interface for different downstream locomotion policies. The learned HMP reveals an interpretable codebook structure in which the number of active RVQ stages modulates the available gait patterns. We further show that training the codebook with the rotation trick improves latent organization and reduces downstream falls compared with a standard straight-through estimator.
The appeal of quadruped robots lies not only in their ability to mimic the diverse and agile locomotion of animals but also in their potential to integrate with human artistic expression to achieve complex multi-skill movements. Continuous multi-skill motion in quadruped robots requires the realization of diverse, continuous, and long-horizon behaviors, involving a broader state space and more complex motion generation and transitions. This presents significant challenges, including sparse rewards, incomplete data, long-horizon motion planning, and the design of fine-grained motion transitions. In this work, we categorize quadruped robot skills into three types: rhythmic motions, expressive motions, and high-dynamic motions and generate reference trajectories for each category using central pattern generators, animation design, and motion capture, respectively. We then design an asymmetric neural network architecture and employ an imitation–reinforcement learning algorithm to train policies for generating these three types of motions. By composing multiple motion skill trajectories, we avoid long-horizon motion planning; by leveraging reinforcement learning, we enable smooth and continuous skill transitions; and by introducing a two-stage reference state initialization curriculum, the robot is able to switch from arbitrary states to the target motion skill. Moreover, during training, the policy imitates only key characteristics of the reference motions rather than strictly tracking fixed trajectories, making it more robust. Finally, we achieve robust motion skill generation and seamless transitions on a quadruped robot equipped with a 6-DoF manipulator, validating the effectiveness and feasibility of the proposed multi-skill generation and transition method.
Chong Pi, Senwei Huang, Wei Li et al.· Robotica (Cambridge. Print)· 0 citations
Music performance enables robots to show their emotions to humans through musical expression. Traditional systems face difficulties when they need to adapt to changing musical expressions and performance environments. The purpose of this research is to develop a dynamic control strategy for a violin-playing robotic system using Reinforcement learning (RL) to improve expressive performance. The proposed approach adapts key bowing dynamics, including bowing speed, bow pressure, bow direction, and timing, based on musical cues. An Entropy-regularized radial basis function with deep Q-network (ER-RBF-DQNet) model is introduced as the core framework. This hybrid RL architecture enhances nonlinear feature mapping and adaptive decision-making. It also improves exploration for more expressive robotic violin performance. A dataset of symbolic musical scores labeled with pitch, duration, and target sound pressure served as input. Normalization techniques were applied to scale musical features, and noise filtering was used to remove inconsistencies in dynamic annotations. Mel-frequency cepstral coefficients (MFCCs), along with tempo, pitch contour, and dynamic range, were extracted as expressive control parameters. The proposed method combines RBFN, DQN, and ER to map musical input features, approximate optimal Q-values, and encourage exploration during training. This allows the robot to generate control signals that align with the musical score, enhancing human-robot interaction through music. Python was implemented, and the evaluation, the accuracy (98.6%) indicates the proportion of robotic control operations that accurately matched the annotated ground-truth musical score. The proposed RL-based dynamic control system successfully transforms score elements into performance actions which enable robots to deliver expressive music performances.