Drive-by road pavement monitoring, using smartphone sensing, has faced longstanding challenges in adoption due to low accuracy and limited applicability. This stems from significant uncertainties during data collection in real-world scenarios, making it prohibitively difficult in applying conventional machine learning models to the detection of road pavement anomalies. This paper presents a two-stage machine learning approach that extracts potential anomalies from the dataset and classifies them into four typical road feature categories. Unlike time-series data analysis, this approach transforms time-series into geospatial series, allowing the analysis to be time-independent thereby capable of detecting road anomalies regardless of driving speeds. Additionally, a framework for a road pavement health monitoring system is proposed to collect data, integrate the machine learning engine, and visualise road anomalies. The developed system was tested on two shuttle buses with normal smartphones, which achieved 87% overall accuracy compared against manual inspection.