Bionic visual-audio photodetectors with in-sensor perception and preprocessing

Serving as the "eyes" and "ears" of the Internet of Things, optical and acoustic sensors are the fundamental components in hardware systems. Nowadays, mainstream hardware systems, often comprising numerous discrete sensors, conversion modules, and processing units, tend to result in complex architectures that are less efficient compared to human sensory pathways. Here, a visual-audio photodetector inspired by the human perception system is proposed to enable all-in-one visual and acoustic signal detection with computing capability. This device not only captures light but also optically records sound waves, thus achieving "watching" and "listening" within a single unit. The gate-tunable positive, negative, and zero photoresponses lead to highly programmable responsivities. This programmability enables the execution of diverse functions, including visual feature extraction, object classification, and sound wave manipulation. These results showcase the potential of expanding perception approaches in neuromorphic devices, opening up new possibilities to craft intelligent and compact hardware systems.