Objective: Our study aimed to explore whether physical condition might affect the association between genetic predisposition for Alzheimer's Disease (AD) and AD incidence. Methods: The sample of participants consisted of 561 community-dwelling adults over 64 years old, without baseline dementia (508 cognitively normal and 53 with mild cognitive impairment), deriving from the HELIAD, an ongoing longitudinal study with follow-up evaluations every 3 years. Physical condition was assessed at baseline through walking time (WT), while a Polygenic Risk Score for late onset AD (PRS-AD) was used to estimate genetic predisposition. The association between WT and PRS-AD with AD incidence was evaluated with Cox proportional hazard models adjusted for age, sex, education years, global cognition score and APOE ε-4 genotype. Then, the association between WT and AD incidence was investigated after stratifying participants by low and high PRS-AD. Finally, we examined the association between PRS-AD and AD incidence after stratifying participants by WT. Results: Both WT and PRS-AD were connected with increased AD incidence (p < 0.05), after adjustments. In stratified analyses, in the slow WT group participants with a greater genetic risk had a 2.5-fold higher risk of developing AD compared to participants with lower genetic risk (p = 0.047). No association was observed in the fast WT group or when participants were stratified based on PRS-AD. Conclusions: Genetic predisposition for AD is more closely related to AD incidence in the group of older adults with slow WT. Hence, physical condition might be a modifier in the relationship of genetic predisposition with AD incidence.