The Effect of Auditory Input on Multimodal Reading Comprehension: An Examination of Adult Readers’ Eye Movements

Abstract Comprehension of many types of texts involves constructing meaning from text and pictures. However, research examining how second language (L2) learners process text and pictures and the relationship with comprehension is scarce. Thus, while verbal input is often presented in written and auditory modes simultaneously (i.e., audio of text with simultaneous reading of it), we do not know how the auditory input affects L2 adult learners’ processing of text and pictures and its relation to comprehension. In the current study, L2 adult learners and native (L1) adults read and read while listening to an illustrated story while their eye movements were recorded. Immediately after reading, they completed a comprehension test. Results showed that the presence of auditory input allowed learners to spend more time looking at pictures and supported a better integration of text and pictures. No differences were observed between L2 and L1 readers’ allocation of attention to text and pictures. Both reading conditions led to similar levels of comprehension. Processing time on the text was positively related to comprehension for L2 readers, while it was associated to lower comprehension for L1 readers. Processing time on images was positively related to comprehension only for L1 readers.