A social recommendation system based on graph neural networks is a system that uses social relationships between users to generate personalized recommendations. To improve recommendation accuracy, it is usually necessary to collect a large amount of user behavior data, which may lead to user privacy breaches and data misuse. Existing privacy protection methods often sacrifice recommendation effectiveness or increase computational costs. Therefore, this article proposes a federated social recommendation system (FRSRec) based on Rényi differential privacy. By combining hash encryption technology with Rényi differential privacy, user data privacy can be protected while ensuring recommendation effectiveness and computational efficiency. This article conducted experimental evaluations on three real datasets. The experimental results show that the FRSRec model achieves optimal results in both MAE and RMSE, demonstrating that the FRSRec model can effectively achieve privacy protection while improving recommendation quality.