A research team at Gwangju Institute of Science and Technology (GIST) has developed artificial intelligence technology that enables robots to perform precision tasks by learning the sense of force that humans feel when touching objects.
The team, led by Professor Lee Kyu-bin of GIST's Department of AI Convergence, who also serves as director of the Artificial Intelligence Research Institute, announced the breakthrough on Thursday.
The technology is distinguished by its ability to learn not only visual information but also the force and contact sensations felt at the fingertips when humans perform tasks. The research team developed a "hand force measurement device" and a "Frequency-aware Multi-modal Transformer (FMT) AI model" to achieve this capability.
In manufacturing, electronics, and automotive industries, robots perform precision manipulation tasks that replace human manual work, such as fitting gears, connecting plugs, and replacing batteries.
The hand force measurement device developed by the research team records human hand movements while simultaneously collecting work video captured by two cameras, force and torque measured by wrist-mounted sensors, and hand movement and position information.
The system measures video at 30 frames per second and force information at more than 200 times per second, enabling precise recording of not only visible scenes but also minute forces generated at the fingertips.
The device also incorporates "3D markers" that accurately track object and hand positions, along with a "device gravity compensation function" that eliminates forces caused by the device's own weight, ensuring only forces generated from actual contact are precisely measured.
This device differentiates itself from existing data collection methods by recording natural movements of humans directly manipulating objects with their hands, rather than relying on remote robot operation.
To address the problem of different recording speeds between video and force data, the research team also developed the FMT AI model. Since video is measured at 30 frames per second while force data is measured at more than 200 times per second, the time intervals between the two data types differ. The FMT analyzes data from these different sensor frequencies separately, then compares and integrates them for learning.
Through this approach, robots can simultaneously understand object positions and contact situations, performing more stable movements even in precision manipulation tasks involving frequent contact.
The research team verified the technology through actual robot experiments on six tasks: gear assembly, box flipping, battery insertion, internet cable plug connection, lid opening, and battery removal. After performing each task 20 times, the average success rate reached approximately 83 percent. This represents a significant improvement over existing methods using only RGB camera footage, which achieved success rates of approximately 20 percent.
"This research overcomes the limitations of existing robot learning methods that relied solely on camera footage and presents a new AI learning framework that can effectively utilize force sensory data," Professor Lee said. "We expect this will advance robot utilization to the next level in various fields requiring delicate force control, including parts assembly and connector fastening at manufacturing sites, as well as battery replacement and electronic device component assembly in home environments."
The research was conducted under the guidance of Professor Lee Kyu-bin of GIST's Department of AI Convergence, with participation from doctoral student Lee Geon-hyeop and others. The study was supported by the Robot Industry Technology Development Project funded by the Ministry of Trade, Industry and Energy and the National Research Foundation of Korea.