Plant-based dietary patterns have been associated with better cognitive function, whereas there has been limited research on the relationship between living conditions and cognitive function. This study aimed to examine these associations in older adults in China. This study utilized a ten-year cohort of older adults, including 1630 participants aged 65 and older from the Chinese Longitudinal Healthy Longevity Survey (CLHLS), all of whom had normal cognition at baseline. Living conditions were categorized into three levels: good, fair, and poor, based on three factors: housing stability, economic stability, and personal space. A simplified food frequency questionnaire was used to assess the overall plant-based diet index (PDI). Cognitive function was evaluated using the Chinese version of the Mini-Mental State Examination (MMSE). The latent class trajectory modeling (LCTM) was applied to classify participants into two PDI types: high-rapid-decreasing PDI and low-stable-decreasing PDI. Binomial logistic regression analysis was employed to explore the associations, with further stratified analyses conducted by PDI. Of the 1630 participants, 48.53% were male, with a median age of 73 years [IQR (Interquartile Range): 69-79]. Living in fair or poor conditions was associated with an 86.0% (OR = 1.860, 95% CI: 1.235-2.798) and 206.1% (OR = 3.061, 95% CI: 1.826-5.131) increased risk of developing cognitive impairment, respectively, compared to those in good living conditions (reference group). The associations between living conditions and cognitive impairment were more pronounced in the "Low-stable-decreasing PDI" group (OR for poor vs good = 3.774, 95% CI: 1.745-8.161) than in the "High-rapid-decreasing PDI" group (OR for poor vs good = 2.344, 95% CI: 1.071-5.130). A plant-based dietary pattern may help mitigate the negative effects of poor living conditions on cognitive function in older adults. Adhering to a plant-based dietary pattern may serve as an effective preventive strategy to safeguard cognitive health, especially among vulnerable populations exposed to adverse environmental factors, such as limited healthcare access or poor living conditions.