Accurately estimating the state-of-charge (SOC) and state-of-health (SOH) of lithium batteries used in electric vehicles is critical but challenging. Machine learning advances aid battery health monitoring, but optimizing model performance often requires adjusting hyperparameters which can lead to local optimization issues. Gaussian process regression (GPR), one of the commonly used methods, typically uses the conjugate gradient method to search for the optimal hyperparameters in lithium-ion battery state estimation, which often results in local optimization. In this paper, the improved firefly algorithm (IFA) is proposed to improve the predictive performance of the GPR model from the internal predictive process perspective. To be specific, the four swarm intelligence algorithms are compared for hyperparameter optimization and finally a novel IFA-GPR model is proposed. Compared with the traditional conjugate gradient method, the proposed model improves the accuracy by 6.75 % and 3.12 % in two current conditions for SOC estimation, and by 91.64 % and 78.12 % in two schemes for SOH prediction, respectively. Moreover, compared with other existing algorithms, the statistical results again verify the high precision and adaptability of the proposed method in battery diagnosis. Our method provides an enhanced and versatile intelligent approach to accurately monitoring battery states using machine learning.