Maintenance and transformation of representational formats during working memory prioritization

Abstract Visual working memory depends on both material-specific brain areas in the ventral visual stream (VVS) that support the maintenance of stimulus representations and on regions in prefrontal cortex (PFC) that control these representations. Recent studies identified stimulus-specific working memory contents via representational similarity analysis (RSA) and analyzed their representational format using deep neural networks (DNNs) as models of the multi-layered hierarchy of information processing. How executive control prioritizes relevant working memory contents and whether this affects their representational formats remains an open question, however. Here, we addressed this issue using a multi-item working memory task involving a retro-cue that prompted participants to maintain one particular item. We exploited the excellent spatiotemporal resolution of intracranial EEG (iEEG) recordings in epilepsy patients and analyzed activity at electrodes in VVS (n=28 patients) and PFC (n=16 patients). During encoding, we identified category-specific information in both VVS and PFC. During maintenance, this information re-occurred in VVS but not in the PFC – suggesting a transformation of PFC representations from encoding to maintenance which putatively reflects the prioritization process. We thus applied RSA in combination with different DNN architectures to investigate the representational format of prioritized working memory contents. Representations during the maintenance period matched representations in deep layers of recurrent but not feedforward DNNs, in both VVS and PFC. While recurrent DNN representations matched PFC representations in the beta band following the retro-cue, they corresponded to VVS representations in a lower theta-alpha frequency range (3-14Hz) towards the end of the maintenance period. Findings could be replicated in recurrent DNNs with two different architectures and using two different training sets. Together, these results demonstrate that VWM relies on representational transformations in VVS and PFC that give rise to distinct coding schemes of prioritized contents.