Previous studies about the relationship between nonalcoholic fatty liver disease (NAFLD) and diabetes are limited by inconsistent conclusions, mainly being cross-sectional and having a small sample size, no elderly people, or a lack of prediabetes. This study sought to examine the relationship between NAFLD and diabetes and prediabetes in a large cohort based on Chinese male elderly. This was a retrospective cohort study that was followed up for ∼5 years. This study was conducted in Beijing, China. Chinese male elderly (n = 18,507). Participants with diabetes/prediabetes at baseline were excluded. Ultrasound was used for diagnosis of NAFLD. Mean age of the 18,507 participants was 71.38 ± 14.15 years. The prevalence of NAFLD was 18.77% (3474/18,503), and participants with NAFLD had higher body mass index (BMI), blood pressure levels, blood lipid levels, and also higher alanine aminotransferase levels (P < 0.001). The total 5-year incidence was 2.448% for diabetes and 10.628% for prediabetes. Participants with NAFLD at baseline had a higher incidence of both diabetes and prediabetes. The adjusted relative risks (RRs) were 1.672 [95% confidence interval (CI), 1.361 to 2.052] and 1.336 (95% CI, 1.205 to 1.481). The RRs were closely related to BMI changes. The adjusted RRs for diabetes according to three BMI change groups (<−1.00 kg/m2, −1.00 to 1.00 kg/m2, >1.00 kg/m2) were 1.599 (95% CI, 1.054 to 2.426), 1.640 (95% CI, 1.241 to 2.167), and 1.918 (95% CI, 1.239 to 2.969), respectively. Similar results were obtained when prediabetes was used as the dependent variable. There was a strong and independent association between NAFLD and both diabetes and prediabetes among Chinese male elderly, and this relationship was closely related to BMI changes.