Spatio‐temporal smoothing and EM estimation for massive remote‐sensing data sets

The use of satellite measurements in climate studies promises many new scientific insights if those data can be efficiently exploited. Due to sparseness of daily data sets, there is a need to fill spatial gaps and to borrow strength from adjacent days. Nonetheless, these satellites are typically capable of conducting on the order of 100,000 retrievals per day, which makes it impossible to apply traditional spatio‐temporal statistical methods, even in supercomputing environments. To overcome these challenges, we make use of a spatio‐temporal mixed‐effects model. For each massive daily data set, dimension reduction is achieved by essentially modelling the underlying process as a linear combination of spatial basis functions on the globe. The application of a dynamical autoregressive model in time, over the reduced space, allows rapid sequential computation of optimal smoothing predictions via the Kalman smoother; this is known as Fixed Rank Smoothing (FRS). The dimension‐reduced mixed‐effects model contains a number of unknown parameters, including covariance and propagator matrices, which describe the spatial and temporal dependence structure in the reduced‐dimensional process. We take an empirical‐Bayes approach to inference, which involves estimating the parameters and substituting them into the optimal predictors. Method‐of‐moments (MM) parameter estimation (currently used in FRS) is typically inefficient compared to maximum likelihood (ML) estimation and can result in large sampling variability. Here, we develop ML estimation via an expectation‐maximization (EM) algorithm, which offers stable computation of valid estimators and makes efficient use of spatial and temporal dependence in the data. The two parameter‐estimation approaches, MM and ML, are compared in a simulation study. We also apply our methodology to global satellite CO 2 measurements: We optimally smooth the sparse daily CO 2 maps obtained by the Atmospheric InfraRed Sounder (AIRS) instrument on the Aqua satellite; then, using FRS with EM‐estimated parameters, a complete sequence of the daily global CO 2 fields can be obtained, together with their associated prediction uncertainties.