Robotic Contextual Awareness for Human-Robot Collaboration and Environmental Understanding

Adaptive Semantic Path Planning for Safe Human Robot Interaction (HRI)

Advancements in the field of Artificial Intelligence (AI) has enhanced the Human Robot Interaction (HRI) / Collaboration to another level beyond conventional into dynamic day to day conditions and environments. As robotics transformation evolves and integrated across various fields, the parallel need for adaptation to unpredictable environment is the demand of the hour. According to the latest research on human robot collaboration, the findings state that: 1) spatial comfort of this interaction is not fixed and predominantly depends on the chaotic nature of the environmental conditions, and 2) robot’s course of approach as well as proximity around humans explicitly affects the perceived safety and triggers psychological uncertainty in humans. The proposed system presented in this paper tackles this challenge by an adaptive semantic path planning for safe and anxiety less human robot collaboration. The proposed system executes following operations: 1) maps unstructured natural language to geometric baseline threshold, 2) evaluating cluster index value to analyze the chaotic conditions of the environmental surroundings and based on this index, A* with Manhattan algorithm calculates shortest destination route and 3) to comply with behavioral safety and anxiety less human robot interaction, the robot route points are regularized using cubic spline transformer to ensure it is avoid of orthogonal, jerky route and replace it by continuous, curvature trajectory in order to address latest research thereby signing safe movements near humans. The simulation results indicate that the proposed framework successfully converts natural human commands into flexible trajectory that effectively satisfies safe HRI. The performance metrics evaluated validates that the proposed system undertakes predictable, secure route with low computational power and latency in contrast to heavy computational architectures that bring forth behavioral uncertainty in HRI.

Knowledge-augmented embodied exploration for robotic grasping in constrained environments

It has always been expected that robots can actively manipulate complex environments to fulfill human requirements. This process typically necessitates that the robot be equipped with the ability for embodied exploration and manipulation. To achieve this goal, in this paper, we propose to incorporate multi-source knowledge to enhance the ability of robotic embodied exploration and manipulation. Specifically, to eliminate the inherent biases in decision-making of large language models (LLMs), we introduce a multi-source knowledge fusion module to generate more reasonable exploration sequences. Notably, grasping detection plays a critical role in the process of robot manipulation. To achieve a better balance between the accuracy and efficiency of the grasping detection network, we design a two-branch feature fusion module with residual blocks to improve network performance. Conditioned on the aforementioned innovations, the robot is capable of actively exploring and manipulating in constrained environments to meet human requirements. Extensive experiments are conducted in both simulation and real-world environments. The results demonstrate the effectiveness and efficiency of our proposed framework.

Learning Physical Interaction: A Survey of Tactile- and Force-aware Robot Learning

Physically grounded robot intelligence requires robots to perceive, reason about, and regulate their interactions with the physical world. This capability is particularly critical in contact-sensitive manipulation, where successful task execution depends not only on visual perception and motion generation, but also on force regulation and adaptive control. In this context, recent robot learning methods have made substantial progress by integrating force, tactile, vision, language, and proprioceptive sensing into learned manipulation policies. In parallel, many systems adopt multi-phase architectures that combine high-level policies, action-refinement modules, and low-level controllers to bridge semantic task understanding with reactive physical execution. Despite these advances, existing surveys have not explicitly reviewed force- and tactile-aware robot learning from a unified perspective that jointly captures multimodal sensing and multi-phase system design. This survey addresses this gap by proposing TF-ART, a Tactile/Force-Aware Robot learning Taxonomy for multimodal and multi-phase frameworks, which maps individual methods into a unified hierarchical structure. The framework characterizes how recent works organize observation modalities, encode and fuse heterogeneous sensory inputs, generate and refine actions across multiple phases, and connect learned policies to reactive robot-end control. Building on this methodological view, we further examine the task settings and infrastructure requirements of physical interaction, thereby integrating both algorithmic and practical perspectives on force- and tactile-aware robot learning.

A Glimpse into Long-term Physical Coexistence with Intelligent Robots

Long-term physical coexistence with intelligent robots requires more than capable robot policies. A persistent robotic assistant must support diverse user-facing interfaces, maintain long-horizon memory of people and preferences, coordinate across robot embodiments, and translate human intent into safe physical execution. We introduce PHILIA, a multi-robot agent built around a robot gateway abstraction. PHILIA retains the rich interaction and tool ecosystem of OpenClaw while exposing robot-local runtimes, onboard perception, navigation, speaker, and robot policies through a unified capability interface. This design decouples low-frequency, high-semantic agent reasoning from high-frequency, low-level robot execution, enabling plug-and-play integration of user interfaces, robot embodiments, and policy backends. As a result, the user experience becomes compositional: advances in user interfaces, robot embodiments, robot policies, navigation, or interaction algorithms can improve the overall experience without redesigning the system. We validate the architecture on Astribot S1 robots while designing the robot gateway contract to support future heterogeneous robot platforms through a shared capability interface for observation, task execution, navigation, speech playback, status monitoring, and task cancellation. We present representative use cases in which agent memory and scene understanding are grounded in robot actions. These span interactive household scenarios, ranging from simple organization to challenging long-horizon and dexterous service tasks, such as packing a backpack and lifting a garbage bag. We highlight the human-robot interaction flow, where contextual understanding of user intent and preferences, together with human-in-the-loop confirmation or adjustment during execution, is essential for effective assistance.

Embodied sensorimotor integration for whole-arm tactile sensing and adaptive robotic manipulation

Autonomous and collaborative robotic systems in 5G smart factories

(English) Autonomous and collaborative mobile manipulators are expected to play a central role in future Smart Factories by enabling flexible, human-centered production. Achieving this vision requires robotic systems that can safely react to dynamic human behavior while efficiently executing complex manipulation tasks. These requirements impose high computational demands on perception, planning, and control, which can exceed the capabilities of mobile robotic platforms. Distributed control architectures leveraging edge computing and wireless communication offer a promising solution by enabling computational offloading. However, their effectiveness depends on tightly coupled and interdisciplinary factors spanning communication, computer vision, and control. Beyond computational offloading, distributed control systems also enable distributed perception by integrating external visual sensors, extending the robot’s perceptual field beyond onboard sensing limitations. This thesis investigates how distributed control architectures for autonomous and collaborative mobile manipulators can be designed to safely and efficiently exploit wireless communication and edge computing in industrial environments. Two complementary architectures are developed and evaluated, differing fundamentally in perception placement, control strategy, and the role of communication latency. The first architecture, an Edge-Enabled system, targets safety-critical collaborative scenarios using onboard RGB-D sensing with optional edge offloading. A complete distributed perception-control loop is implemented over private 5G networks, integrating wireless communication, edge-based image processing, and closed-loop Cartesian velocity control. Extensive simulated and real experiments analyze how sensing rate, image resolution and compression, computation latency, communication technology, and quality-of-service settings jointly affect end-to-end reaction time. The results show that edge computing reduces latency only under specific conditions and that robust safety behavior requires explicit mechanisms to handle jitter and perception failures. The second architecture, an Edge-Dependent system, addresses deliberative manipulation under fixed distributed constraints, where perception and planning are performed externally. A learning-based motion planning framework is introduced that generates smooth, near-optimal, collision-free 3D trajectories with low online computation time using external visual data. Simulation and real-robot experiments demonstrate generalization across obstacle configurations and competitive performance compared to established planning baselines. Together, these contributions clarify when wireless edge computing enhances robotic performance and when it introduces fundamental constraints, providing practical system designs and conceptual insights for distributed robotic control in next-generation industrial environments. (Català) S'espera que els manipuladors mòbils autònoms i col·laboratius tinguin un paper central en les fàbriques intel·ligents del futur, en permetre una producció flexible i centrada en l'ésser humà. Per assolir aquesta visió calen sistemes robòtics que puguin reaccionar de manera segura al comportament humà dinàmic, alhora que executen de forma eficient tasques de manipulació complexes. Aquests requisits imposen altes demandes computacionals a la percepció, la planificació i el control, les quals poden superar les capacitats de les plataformes robòtiques mòbils. Les arquitectures de control distribuït que aprofiten el processament en el límit de la xarxa i la comunicació sense fils ofereixen una solució prometedora en permetre la delegació de càrrega computacional. No obstant això, la seva eficàcia depèn de factors estretament vinculats i interdisciplinaris que abasten la comunicació, la visió per computador i el control. Més enllà de la delegació de càlcul, els sistemes de control distribuït també permeten la percepció distribuïda mitjançant la integració de sensors visuals externs, ampliant el camp perceptiu del robot més enllà de les limitacions dels sensors a bord. Aquesta tesi investiga com es poden dissenyar arquitectures de control distribuït per a manipuladors mòbils autònoms i col·laboratius per aprofitar de manera segura i eficient la comunicació sense fils i el processament en el límit (edge computing) en entorns industrials. Es desenvolupen i avaluen dues arquitectures complementàries que difereixen fonamentalment en la ubicació de la percepció, l'estratègia de control i el paper de la latència de la comunicació. La primera arquitectura, un sistema habilitat per a la vora (Edge-Enabled), està orientada a escenaris col·laboratius de seguretat crítica que utilitzen sensors RGB-D a bord amb una càrrega lateral opcional a la vora. S'implementa un bucle complet de percepció i control distribuït sobre xarxes 5G privades, que integra comunicació sense fils, processament d'imatges basat en la vora i control de velocitat cartesiana en bucle tancat. Amplis experiments simulats i reals analitzen com la velocitat de captació, la resolució i la compressió d'imatges, la latència de càlcul, la tecnologia de comunicació i la configuració de la qualitat de servei afecten conjuntament el temps de reacció de principi a fi. Els resultats mostren que la computació en el límit només redueix la latència en condicions específiques i que un comportament de seguretat robust requereix mecanismes explícits per gestionar la variància i els errors de percepció. La segona arquitectura, un sistema dependent de l'edge (Edge-Dependent), aborda la manipulació deliberativa sota restriccions distribuïdes fixes, on la percepció i la planificació es duen a terme externament. Es presenta un marc de planificació de moviments basat en l'aprenentatge que genera trajectòries 3D suaus, gairebé òptimes i lliures de col·lisions amb un temps de computació en línia baix, utilitzant dades visuals externes. Les simulacions i els experiments amb robots reals demostren la generalització a través de configuracions d'obstacles i un rendiment competitiu en comparació amb línies de base de planificació establertes. En conjunt, aquestes contribucions aclareixen quan la computació edge sense fils millora el rendiment robòtic i quan introdueix limitacions fonamentals, proporcionant dissenys de sistema pràctics i perspectives conceptuals per al control robòtic distribuït en entorns industrials de nova generació. (Español) Se prevé que los manipuladores móviles autónomos y colaborativos desempeñen un papel fundamental en las futuras fábricas inteligentes, al permitir una producción flexible y centrada en las personas. Para hacer realidad esta visión se necesitan sistemas robóticos capaces de reaccionar de forma segura ante el comportamiento dinámico de las personas, al tiempo que ejecutan con eficiencia tareas de manipulación complejas. Estos requisitos imponen elevadas exigencias computacionales en materia de percepción, planificación y control, que pueden superar las capacidades de las plataformas robóticas móviles. Las arquitecturas de control distribuido que aprovechan la computación periférica y la comunicación inalámbrica ofrecen una solución prometedora al permitir la descarga computacional. Sin embargo, su eficacia depende de factores estrechamente relacionados e interdisciplinarios que abarcan la comunicación, la visión artificial y el control. Más allá de la descarga computacional, los sistemas de control distribuido también permiten la percepción distribuida mediante la integración de sensores visuales externos, ampliando el campo perceptivo del robot más allá de las limitaciones de los sensores integrados. Esta tesis investiga cómo se pueden diseñar arquitecturas de control distribuido para manipuladores móviles autónomos y colaborativos con el fin de aprovechar de forma segura y eficiente la comunicación inalámbrica y la computación periférica en entornos industriales. Se desarrollan y evalúan dos arquitecturas complementarias, que difieren fundamentalmente en la ubicación de la percepción, la estrategia de control y el papel de la latencia de la comunicación. La primera arquitectura, un sistema habilitado para el borde (Edge-Enabled), está orientada a escenarios colaborativos críticos para la seguridad que utilizan sensores RGB-D a bordo con descarga opcional al borde. Se implementa un bucle completo de percepción y control distribuido a través de redes 5G privadas, que integra comunicación inalámbrica, procesamiento de imágenes en el borde y control de velocidad cartesiano en bucle cerrado. Unos extensos experimentos simulados y reales analizan cómo la frecuencia de detección, la resolución y compresión de la imagen, la latencia de cálculo, la tecnología de comunicación y los ajustes de calidad de servicio afectan conjuntamente al tiempo de reacción de extremo a extremo. Los resultados muestran que la computación en el borde reduce la latencia solo en condiciones específicas y que un comportamiento de seguridad robusto requiere mecanismos explícitos para gestionar la fluctuación y los fallos de percepción. La segunda arquitectura, un sistema dependiente del borde (Edge-Dependent), aborda la manipulación deliberativa bajo restricciones distribuidas fijas, donde la percepción y la planificación se realizan externamente. Se introduce un marco de planificación de movimiento basado en el aprendizaje que genera trayectorias 3D fluidas, casi óptimas y sin colisiones con un tiempo de computación en línea reducido utilizando datos visuales externos. Los experimentos de simulación y con robots reales demuestran la generalización a través de configuraciones de obstáculos y un rendimiento competitivo en comparación con las líneas de base de planificación establecidas. En conjunto, estas aportaciones aclaran en qué casos la computación periférica inalámbrica mejora el rendimiento de los robots y en cuáles introduce limitaciones fundamentales, aportando di