Recent findings indicate that visual feedback derived from episodic memory can be traced down to the earliest stages of visual processing, whereas feedback stemming from schema-related memories only reach intermediate levels in the visual processing hierarchy. In this opinion piece, we examine these differences in light of the 'what' and 'where' streams of visual perception. We build upon this new framework to propose that the memory deficits observed in aphantasics might be better understood as a difference in high-level feedback processing along the 'what' stream, rather than an episodic memory impairment.