Progress in hardware and algorithms for artificial intelligence (AI) has ushered in large machine learning models and various applications impacting our everyday lives. However, today's AI, mainly artificial neural networks, still cannot compete with human brains because of two major issues: the high energy consumption of the hardware running AI models and the lack of ability to generalize knowledge and self-adapt to changes. Neuromorphic systems built upon emerging devices, for instance, memristors, provide a promising path to address these issues. Although innovative memristor devices and circuit designs have been proposed for neuromorphic computing and applied to different proof-of-concept applications, there is still a long way to go to build large-scale low-power memristor-based neuromorphic systems that can bridge the gap between AI and biological brains. This Perspective summarizes the progress and challenges from memristor devices to neuromorphic systems and proposes possible directions for neuromorphic system implementation based on memristive devices.