A Systematic Review of Echo State Networks From Design to Application

Recurrent Neural Network (RNN) has demonstrated its outstanding ability in sequence tasks and has achieved state-of-the-art in many applications, such as industrial and medical. Echo State Network (ESN) is a simple type of RNN and has emerged in the last decade as an alternative to gradient descent training-based RNN. ESN is practical, conceptually simple, and easy to implement with a strong theoretical ground. It can avoid non-converging and computationally expensive issues in gradient descent RNN methods. Since ESN was put forward in 2002, abundant existing works have promoted the progress of ESN, and the recently introduced deep ESN opened the way to uniting the merits of deep learning and reservoir computing. Besides, the combinations of ESNs with other machine learning models have also overperformed baselines in some applications. However, the apparent simplicity of ESNs can sometimes be deceptive. Successfully applying ESNs needs some experience. Thus, we reviewed over 300 related papers and provided a systematic overview for the first time. In this paper, we categorize the related methods into classical ESN, DeepESN, and combination. Then, we analyze them from the perspective of network designs and specific applications. Finally, we discuss the challenges and opportunities by proposing open problems and future work.