Alzheimer's disease (AD) is pathologically hallmarked by neurofibrillary tangles and senile plaques which are mainly composed of tau and β-amyloid (Aβ) respectively. Recent failures of medicine targeting Aβ have drawn more attention to tau protein and its fibrillization acting on neurodegeneration. The accumulation of tau is closely related to two amyloidogenic core of PHF6 and PHF6∗ fragments. Small molecules inhibiting the aggregation of tau or its core fragments are considered as potential inhibitors, while the underlying mechanism remains elusive. Here we carried out a series of microsecond all-atom molecular dynamics simulations to investigate the interaction of several promising inhibitors (e.g. melatonin, purpurin, etc.) with fibrillar oligomers of different tau fragments. Our results show that the binding behavior of different inhibitors is specific and sensitive, but still has characteristics in common. The influence of inhibitors on tau atomic structures, the binding dynamics and sites, and their key interactions in between are explored. The correlation between the binding of inhibitors and the structural changes in tau is discussed. In general, inhibitors targeting PHF6 have a significant disruptive effect on the stability of fibrillar oligomers. Our studies inspire new clues for the design of tau inhibitor and are helpful to AD diagnosis and therapies.