Holistic Reinforcement Learning: The Role of Structure and Attention

Recent advances have refined our understanding of reinforcement learning by emphasizing roles for attention and for structured knowledge in shaping ongoing learning. Bayesian cognitive models have made great strides towards describing how structured knowledge can be learned, but their computational complexity challenges neuroscientific implementation. Behavioral and neural evidence suggests that each class of algorithms describes unique aspects of human learning. We propose an integration of these computational approaches in which structured knowledge learned through approximate Bayesian inference acts as a source of top-down attention, which shapes the environmental representation over which reinforcement learning occurs. Compact representations of the environment allow humans to behave efficiently in a complex world. Reinforcement learning models capture many behavioral and neural effects but do not explain recent findings showing that structure in the environment influences learning. In parallel, Bayesian cognitive models predict how humans learn structured knowledge but do not have a clear neurobiological implementation. We propose an integration of these two model classes in which structured knowledge learned via approximate Bayesian inference acts as a source of selective attention. In turn, selective attention biases reinforcement learning towards relevant dimensions of the environment. An understanding of structure learning will help to resolve the fundamental challenge in decision science: explaining why people make the decisions they do. Compact representations of the environment allow humans to behave efficiently in a complex world. Reinforcement learning models capture many behavioral and neural effects but do not explain recent findings showing that structure in the environment influences learning. In parallel, Bayesian cognitive models predict how humans learn structured knowledge but do not have a clear neurobiological implementation. We propose an integration of these two model classes in which structured knowledge learned via approximate Bayesian inference acts as a source of selective attention. In turn, selective attention biases reinforcement learning towards relevant dimensions of the environment. An understanding of structure learning will help to resolve the fundamental challenge in decision science: explaining why people make the decisions they do. a response the participant makes (e.g., choosing an option, labeling a stimulus, predicting an outcome). a class of Bayesian cognitive models that group observations into sets of unobservable latent causes, or clusters. a subset of environmental features relevant to the agent’s goal (e.g., the feature red being more predictive of reward). a class of sampling methods for approximating arbitrary probability distributions in a sequential manner, by maintaining and updating a finite number of particles (hypotheses). a stimulus, potentially with multiple features. a class of Bayesian cognitive models that reason over structured concepts such as rules. a class of algorithms that learn an optimal behavioral policy, often through learning the values of different actions in different states. the process by which learners arrive at a representation of environmental states. consequence of an action (e.g., a reward or category label). the difference between the reward outcome and what was expected; used as a learning signal to update values of states and actions. the agent’s internal representation of the environmental state.