Structural magnetic resonance imaging (sMRI) has been used to examine age-related neuroanatomical changes in the human brain. In the present work, a pre-trained deep learning model and an ensemble deep random vector functional link (edRVFL) classifier have been used to create a brain age classification framework from magnetic resonance imaging (MRI) scans. A total of 155 MRI scans of the brain are obtained from the open-access OpenNeuro database and categorized into three age groups (3–5 years old, 7–12 years old, and 18–40 years old). To visualize the age connection across different brain regions, all MRI scans are first segmented into Gray Matter (GM), White Matter (WM), and Cerebrospinal Fluid (CSF). The ResNet-50 network is used to extract features from MRI images, while the edRVFL network is used to classify the retrieved features. Classification accuracy for GM, WM, CSF, and whole brain images are 96.11%, 98.33%, 93.33%, and 94.00%, respectively, using the edRVFL classifier. Region-wise analysis has also been done using Pearson’s correlation coefficient (r), coefficient of determination (R2), and root mean square error (RMSE) to analyze the relationship between brain age and brain tissue volumes. According to the findings of the suggested deep model for brain age categorization, and region-wise analysis, alterations in WM volume are strongly linked to brain aging.