Learning and making decisions in changing environments with Variational Inference
(2018)
Files
Thesis.pdf
Open Access - Adobe PDF
- 17.38 MB
Details
- Authors
- Supervisors
- Duqué, Julie
- Abstract
- Statistics, Cognition and Learning -- either in machines or in animals -- have two common features: first, they are aimed at formulating prediction of unseen data. Second, they require a model of the observations to express these predictions. When data are acquired sequentially, the temporal structure of the acquisition can be taken into account to predict with a better accuracy the distribution of the variable of interest: for instance, an animal can predict the future from its past experience, and an algorithm can use past correlation of between successive observations to predict upcoming events. The present work will first focus on data-driven and informative algorithms aimed at predicting and explaining the behaviour of human subjects. We will be interested in linking together two fields of cognitive neuroscience: reinforcement learning, with a special focus on the study of inflexible behaviours, and decision making through the popular sequential sampling models. Both of these topics will be studied through the lens of the theories of the Bayesian brain, whom assume that the central nervous system performs some sort of Bayesian inference to predict and, sometimes, explain the observations that are provided to it. We will base the models presented in this thesis on three pillars: first, reinforcement learning will constitute the core psychological and neurophysiological model of learning how to behave. Next, we will frame the process of action selection in the popular framework of the Sequential Sampling family of decision making models. Finally, in Bayesian statistics, we will focus on Variational approximations to perform inference about the world (at the computational and algorithmic level) and to fit the models we propose to behavioural data. Our contribution to the field of cognitive neuroscience will be first methodological, as we will propose a novel Bayesian approach to fit known models of decision making. Next, we will introduce a new Bayesian conceptual framework around the concept of inflexibility. In the context of the popular Drift Diffusion Model, we will show that the trial-wise posterior distribution of latent variables of the Drift-diffusion model can be inferred simply by assuming that the data and generative process are correlated in a forward manner. We will provide the important result that, if the customary iid assumption is relaxed, the fit of the DDM parameters can be remarkably improved. Next, we will get interested in a crucial question for any agent interacting with the environment: in the presence of surprising events, when would one's model need to be changed? Inability to change the model may be a serious disadvantage in unsteady environment, but having a model that is too volatile may impair the agent ability to accurately predict future events, especially in the presence of noisy observations. This second part of our work will aim at treating the specific problem of modelling the emergence of inflexible behaviours in a Bayesian framework. The algorithm we will propose relies on the principle that the more data are stored in memory, the more precise is the inference the agent can achieve in a stable environment. We will propose a scheme of learning where this agent learns the stability of the environment in a hierarchical manner, adapting its forgetting rate in function of the surrounding volatility, and adapting this measure of the volatility in a similar manner. Finally, we will tie these two algorithms in a convenient and interpretable model of learning and decision-making.
- Affiliations
Citations
Moens, V. (2018). Learning and making decisions in changing environments with Variational Inference. https://hdl.handle.net/2078.5/123397