循环神经网络
计算机科学
深度学习
人工智能
构造(python库)
前馈神经网络
前馈
功能(生物学)
人工神经网络
工程类
进化生物学
生物
控制工程
程序设计语言
作者
Razvan Pascanu,Çağlar Gülçehre,Kyunghyun Cho,Yoshua Bengio
出处
期刊:International Conference on Learning Representations
日期:2014-01-01
被引量:156
摘要
Abstract: In this paper, we explore different ways to extend a recurrent neural network (RNN) to a \textit{deep} RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.
科研通智能强力驱动
Strongly Powered by AbleSci AI