Circadian rhythms are endogenous 24-hour oscillations that are vital for maintaining our overall well-being. They are driven at a high level by a core circadian clock located in the brain, making their dynamics difficult to track. Various modeling approaches exist to predict the dynamics, but as the models are typically designed on population-level data, their performance is diminished on the individual level. This paper proposes a method for learning personalized latent state models, i.e., dynamical models that explicitly use latent state variables, that relate circadian input(s) to observable biometric signals. Our models combine an autoencoder with a recurrent neural network and use the pair to model the salient dynamics present in the data. We validate our method using experimental data, where the circadian input is light and the biometric data are actigraphy signals. We demonstrate that our method results in models with low-dimensional latent state that can accurately reconstruct and predict the observable biometric signals. Further, we show that the oscillation of the learned latent state agrees with the subjects' circadian clock oscillation as estimated with melatonin measurements.Clinical relevance - This proposes a technique for personal-ized modeling of the circadian system with potential applications in feedback control and individualized circadian studies.