ValueCSV: Evaluating Core Socialist Values Understanding in Large Language Models

The rapid development of large language models (LLMs) has made it crucial to align their values with those of humans. However, current efforts mainly focus on Western values in English contexts, such as Schwartz's values, yet neglect the diverse cultural or social values. In this paper, we aim to explore LLMs' alignment with Chinese values, the Core Socialist Values (CSV) which is a representative set of values in China. We introduce a novel framework called ValueCSV, which is capable of evaluating the extent to which existing LLMs align with CSV from bottom to top. Specifically, we first employ a human-LLM collaborative paradigm to collect and annotate a ValueCSV dataset with 5, 000 data. We then train a CSV evaluator to estimate value alignment based on the prompted generation of LLMs. We conduct extensive experiments to validate the annotation quality of the ValueCSV dataset, assess the performance of the value classifier, and analyze the CSV values maps across six LLMs. Our analysis reveals that these LLMs exhibit diverse value maps, with varying degrees of alignment across the 12 dimensions of CSV. Our framework is publicly available at https://github.com/ValueCSV .