An Encoding Framework With Brain Inner State for Natural Image Identification

Neural encoding and decoding, which aim to characterize the relationship between stimuli and brain activities, have emerged as an important area in cognitive neuroscience. Traditional encoding models, which focus on feature extraction and mapping, consider the brain as an input-output mapper without inner states. In this article, inspired by the fact that the human brain acts like a state machine, we proposed a novel encoding framework that combines information from both the external world and the inner state to predict brain activity. The framework comprises two parts: 1) forward encoding model that deals with visual stimuli and 2) inner state model that captures influence from intrinsic connections in the brain. The forward model can be any traditional encoding model, making the framework flexible. The inner state model is a linear model to utilize information in the prediction residuals of the forward model. The proposed encoding framework achieved much better performance on natural image identification than forward-only models, with a maximum identification accuracy of 100%. The identification accuracy decreased slightly with the data set size increasing, but remained relatively stable with different identification methods. The results confirm that the new encoding framework is effective and robust when used for brain decoding.