How do humans build and navigate their complex social world? Standard theoretical frameworks often attribute this success to a foundational capacity to analyze other people’s appearance and behavior to make inferences about their unobservable mental states. Here we argue that this picture is incomplete. Human behavior leaves traces in our physical environment that reveal our presence, our goals, and even our beliefs and knowledge. A new body of research shows that, from early in life, humans easily detect these traces—sometimes spontaneously—and readily extract social information from the physical world. From the features and placement of inanimate objects, people make inferences about past events, and how people have shaped the physical world. This capacity develops early, and helps explain how people have such a rich understanding of others: By drawing not only on how others act, but also on the environments they’ve shaped. Overall, social cognition is crucial not only to our reasoning about people and actions, but also to our everyday reasoning about the inanimate world.