An Intelligent Robotic System Capable of Sensing and Describing Objects Based on Bimodal, Self‐Powered Flexible Sensors

Abstract This study presents an intelligent soft robotic system capable of perceiving, describing, and sorting objects based on their physical properties. This work introduces a bimodal self‐powered flexible sensor (BSFS) based on the triboelectric nanogenerator and giant magnetoelastic effect. The BSFS features a simplified structure comprising a magnetoelastic conductive film and a packaged liquid metal coil. The BSFS can precisely detect and distinguish touchless and tactile models, with a response time of 10 ms. By seamlessly integrating the BSFSs into the soft fingers, this study realizes an anthropomorphic soft robotic hand with remarkable multimodal perception capabilities. The touchless signals provide valuable insights into object shape and material composition, while the tactile signals offer precise information regarding surface roughness. Utilizing a convolutional neural network (CNN), this study integrates all sensing information, resulting in an intelligent soft robotic system that accurately describes objects based on their physical properties, including materials, surface roughness, and shapes, with an accuracy rate of up to 97%. This study may lay a robotic foundation for the hardware of the general artificial intelligence with capacities to interpret and interact with the physical world, which also serves as an interface between artificial intelligence and soft robots.