Primates are distinguished from other mammals by their heavy reliance on the visual sense, which occurred as a result of natural selection continually favoring those individuals whose visual systems were more responsive to challenges in the natural world. Here we describe two independent but also interrelated visual systems, one cortical and the other subcortical, both of which have been modified and expanded in primates for different functions. Available evidence suggests that while the cortical visual system mainly functions to give primates the ability to assess and adjust to fluid social and ecological environments, the subcortical visual system appears to function as a rapid detector and first responder when time is of the essence, i.e., when survival requires very quick action. We focus here on the subcortical visual system with a review of behavioral and neurophysiological evidence that demonstrates its sensitivity to particular, often emotionally charged, ecological and social stimuli, i.e., snakes and fearful and aggressive facial expressions in conspecifics. We also review the literature on subcortical involvement during another, less emotional, situation that requires rapid detection and response - visually guided reaching and grasping during locomotion - to further emphasize our argument that the subcortical visual system evolved as a rapid detector/first responder, a function that remains in place today. Finally, we argue that investigating deficits in this subcortical system may provide greater understanding of Parkinson's disease and Autism Spectrum disorders.