Broadband convolutional processing is critical to high-precision image recognition and is of use in remote sensing and environmental monitoring. Implementing in-sensor broadband convolutional processing using conventional complementary metal–oxide–semiconductor technology is, however, challenging because broadband sensing and convolutional processing require the use of the same physical processes. Here we show that a palladium diselenide/molybdenum ditelluride van der Waals heterostructure can provide simultaneous broadband image sensing and convolutional processing. The band alignment between type-II and type-III heterojunctions of the photovoltaic heterostructure is gate tunable, and the devices exhibit linear light-intensity dependence for both positive and negative photoconductivity, as well as linear gate dependence for the broadband photoresponse. Our in-sensor broadband convolutional processing improves recognition accuracy for multi-band images compared with conventional single-band-based convolutional neural networks.