A collaborative perception method of human-urban environment based on machine learning and its application to the case area

Human perception refers to people's psychological feelings about a place. Understanding residents' perception and their activities is important for promoting people-oriented urban construction. Recently, with the development of machine learning, researchers used this technology to study human perception from the open-source street view images. However, the perception measurement is limited, caused by the inadequate feature extraction. Besides, human perceptions and their activities are separately studied, which hinders the process of revealing the relationship between human and environment. Hence, a human perception model was firstly proposed, where a Transformer network was introduced to extract more discriminative semantic features and visual elements were integrated to enhance the feature representations. Experiments showed that the average deviation of perceptual scores was controlled within 1.6 points, and its performance was improved by around 1%–18% in mean square error (MSE), root mean square error (RMSE) and mean absolute error (MAE) compared with the existing best results. Secondly, the collaborative study of environmental perception and residents' activities was carried out in the case area. Specifically, the perceptual measures of environment were implemented based on the street view video data. Meanwhile, the activities of residents were recognized by SlowFast network and quantified by a new informatics-based diversity indicator (Active Index). This study finally obtained their spatial distribution map, and showed that the perceptual dimensions lively, boring, safe, and depressing are correlated with information quantity of activities. The paper provides a novel method to understand better the urban environment and the distribution of residents' activities to facilitate urban planning.