Environmental implications of food choice are the focus of increasingly extensive research, but less is known about the impacts of dietary patterns of different socio-economic groups of a country, and the trade-offs between nutritional quality and environmental impacts of diet within those groups. We evaluate the impacts of US household dietary patterns on greenhouse gas emissions, blue water footprint, land use and energy consumption across supply chains using an environmentally extended input–output analysis. We compare the nutritional quality of these dietary patterns using healthy eating index scores across individuals’ income and other socio-economic characteristics. Individuals with higher income or education levels are more likely to adopt healthier diets but are also responsible for larger environmental impacts of diet primarily due to a higher consumption of dairy and livestock products, seafood and items with lower energy density but higher nutrient density. Our optimization shows that a healthy diet with lower environmental impacts is achievable within current food budgets for almost 95% of people, and results in average decreases of 2% in food-related greenhouse gas emissions, 24% in land use and 4% in energy consumption, but a 28% increase in blue water consumption. However, such dietary patterns are unaffordable for 38% of Black and Hispanic individuals in the lowest income and education groups. Policies that affect income and food prices making nutritious food more affordable would be needed to achieve better nutrition and improved environmental outcomes simultaneously, particularly for more vulnerable socio-economic groups. Higher income is associated with healthier, but more environmentally detrimental, diets in the United States. Healthy diets with lower environmental impacts are achievable within current food budgets for the majority of the population, but are unaffordable for 38% of Black and Hispanic people in the lowest income and education groups.