Toward Cognitive Digital Twin System of Human-Robot Collaboration Manipulation

作者：Xin Li, Bin He, Zhipeng Wang, Yanmin Zhou, Gang Li, Xiang Li · 发表于：IEEE Transactions on Automation Science and Engineering · 年份：2024 · DOI：10.1109/tase.2024.3452149 · 被引用次数：16 · 研究领域：Digital Transformation in Industry、Engineering Education and Technology、Engineering Technology and Methodologies

Multielement decision-making is crucial for the robust deployment of human-robot collaboration (HRC) systems in flexible manufacturing environments with personalized tasks and dynamic scenes. Large Language Models (LLMs) have recently demonstrated remarkable reasoning capabilities in various robotic tasks, potentially offering this capability. However, the application of LLMs to actual HRC systems requires the timely and comprehensive capturing of real-scene information. In this study, we suggest incorporating real scene data into LLMs using digital twin (DT) technology and present a cognitive digital twin prototype system of HRC manipulation, known as HRC-CogiDT. Specifically, we initially construct a scene semantic graph encoding the geometric information of entities, spatial relations between entities, actions of humans and robots, and collaborative activities. Subsequently, we devise a prompt that merges scene semantics with prior knowledge of activities, linking the real scene with LLMs. To evaluate performance, we compile an HRC scene understanding dataset and set up a laboratory-level experimental platform. Empirical results indicate that HRC-CogiDT can swiftly perceive scene changes and make high-level decisions based on varying task requirements, such as task planning, anomaly detection, and schedule reasoning. This study provides promising insights for the future applications of LLMs in robotics.Note to Practitioners—Recently, LLMs have demonstrated significant succ...