Recently, weakly supervised nuclei segmentation methods using only points are gaining attention, as they can ease the tedious labeling process. However, most methods often fail to separate adjacent nuclei and are particularly sensitive to point annotations that deviate from the center of nuclei, resulting in lower accuracy. In this study, we propose a novel weakly supervised method to effectively distinguish adjacent nuclei and maintain robustness regardless of point label deviation. We detect and segment nuclei by combining a binary segmentation module, an offset regression module, and a center detection module to determine foreground pixels, delineate boundaries and identify instances. In training, we first generate pseudo binary masks using geodesic distance-based Voronoi diagrams and k-means clustering. Next, segmentation predictions are used to repeatedly generate pseudo offset maps that indicate the most likely nuclei center. Finally, an Expectation Maximization (EM) based process iteratively refines initial point labels based on the offset map predictions to fine-tune our framework. Experimental results show that our model consistently outperforms state-of-the-art methods on public datasets regardless of the point annotation accuracy.