GrassJoin: Distributed Set Similarity Join Based on Graph Partitioning Model

Set Similarity Join (SSJ) plays a crucial role in a wide array of tasks, including plagiarism detection, data cleaning and near-duplicate detection in IOT information, as it effectively identifies similar pairs within two collections of sets. As the volume of data continues to soar, the necessity for distributed SSJ becomes apparent in order to manage large-scale datasets efficiently. Nonetheless, the extensively studied distributed SSJ solutions predominantly rely on a prefix-based framework. They often suffer from negative issues such as (1) duplicate verification (2) load imbalance. To address such issues, we propose a novel Graph-partitioning-based distributed set similarity Join (GrassJoin). Empirical evaluations conducted on four datasets substantiate the efficacy of our approach and demonstrate its substantial advantages over state-of-the-art solutions.