KalmanAE: Deep Embedding Optimized Kalman Filter for Time Series Anomaly Detection

The Kalman filter performs well in system state estimation by inferring a joint probability distribution over time variables, which has numerous technological applications in time series analysis. However, the complex Kalman filter parameter settings prevent it from optimally estimating the system state, and this suboptimal estimation makes it difficult to effectively distinguish between normal and anomalous system states in anomaly detection. In this paper, we propose a deep embedding-optimized Kalman filter for unsupervised time series anomaly detection, where the system state of a normal time series can be fitted by the embedding-optimized Kalman filter in an unsupervised manner and anomalies can be detected from data points that deviate from the normal system state. Specifically, we use an autoencoder-enhanced Kalman filter to capture the normal pattern of the time series, where the original time series signal is first fed into the Kalman filter, then the autoencoder encodes the filtered signal and reconstructs the original signal, and the learned embedding from encoder is used to sequentially optimize the Kalman filter. Finally, the optimized filter captures the normal pattern of the time series and the reconstruction error from the filtered signal can be measured to detect anomalies. The validity of the method is verified on real-world time series datasets.