STAM: A SpatioTemporal Attention Based Memory for Video Prediction

计算机科学 人工智能 代表(政治) 模式识别(心理学) 政治 政治学 法学
作者
Zheng Chang,Xinfeng Zhang,Shanshe Wang,Siwei Ma,Wen Gao
出处
期刊:IEEE Transactions on Multimedia [Institute of Electrical and Electronics Engineers]
卷期号:25: 2354-2367 被引量:12
标识
DOI:10.1109/tmm.2022.3146721
摘要

Video prediction has always been a very challenging problem in video representation learning due to the complexity in spatial structure and temporal variation. However, existing methods mainly predict videos by employing language-based memory structures from the traditional Long Short-Term Memories (LSTMs) or Gated Recurrent Units (GRUs), which may not be powerful enough to model the long-term dependencies in videos, consisting of much more complex spatiotemporal dynamics than sentences. In this paper, we propose a SpatioTemporal Attention based Memory (STAM), which can efficiently improve the long-term spatiotemporal memorizing capacity by incorporating the global spatiotemporal information in videos. In the temporal domain, the proposed STAM aims to observe temporal states from a wider temporal receptive field to capture accurate global motion information. In the spatial domain, the proposed STAM aims to jointly utilize both the high-level semantic spatial state and the low-level texture spatial states to model a more reliable global spatial representation for videos. In particular, the global spatiotemporal information is extracted with the help of an Efficient SpatioTemporal Attention Gate (ESTAG), which can adaptively apply different levels of attention scores to different spatiotemporal states according to their importance. Moreover, the proposed STAM are built with 3D convolutional layers due to their advantages in modeling spatiotemporal dynamics for videos. Experimental results show that the proposed STAM can achieve state-of-the-art performance on widely used datasets by leveraging the proposed spatiotemporal representations for videos.

科研通智能强力驱动
Strongly Powered by AbleSci AI
更新
PDF的下载单位、IP信息已删除 (2025-6-4)

科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
Tina发布了新的文献求助10
1秒前
深情安青应助TOBET采纳,获得10
1秒前
英俊的铭应助飘零枫叶采纳,获得10
2秒前
迷路向松发布了新的文献求助10
2秒前
2秒前
领导范儿应助北栀采纳,获得10
2秒前
2秒前
小蘑菇应助激昂的南烟采纳,获得10
2秒前
DiJia完成签到 ,获得积分10
2秒前
xzleee完成签到 ,获得积分10
2秒前
ATM完成签到,获得积分20
3秒前
求助123完成签到,获得积分10
3秒前
3秒前
3秒前
3秒前
3秒前
CodeCraft应助tdtk采纳,获得10
3秒前
风中的电脑完成签到,获得积分10
4秒前
王丽芳发布了新的文献求助10
4秒前
你66发布了新的文献求助10
4秒前
ATM发布了新的文献求助10
5秒前
小小怪完成签到,获得积分20
5秒前
科目三应助小汤采纳,获得10
5秒前
德爱完成签到,获得积分10
5秒前
可爱的函函应助mmRadio采纳,获得10
5秒前
科研通AI2S应助张小哥12采纳,获得10
6秒前
6秒前
6秒前
snitch完成签到,获得积分10
7秒前
7秒前
7秒前
tong完成签到,获得积分10
7秒前
黯然发布了新的文献求助10
7秒前
WWJ发布了新的文献求助10
7秒前
Luo发布了新的文献求助10
8秒前
琪七发布了新的文献求助10
8秒前
德爱发布了新的文献求助10
8秒前
9秒前
9秒前
贾千兰发布了新的文献求助10
9秒前
高分求助中
A new approach to the extrapolation of accelerated life test data 1000
Technical Brochure TB 814: LPIT applications in HV gas insulated switchgear 1000
ACSM’s Guidelines for Exercise Testing and Prescription, 12th edition 500
Picture Books with Same-sex Parented Families: Unintentional Censorship 500
Nucleophilic substitution in azasydnone-modified dinitroanisoles 500
不知道标题是什么 500
A Preliminary Study on Correlation Between Independent Components of Facial Thermal Images and Subjective Assessment of Chronic Stress 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 工程类 有机化学 生物化学 物理 内科学 纳米技术 计算机科学 化学工程 复合材料 遗传学 基因 物理化学 催化作用 冶金 细胞生物学 免疫学
热门帖子
关注 科研通微信公众号,转发送积分 3970632
求助须知:如何正确求助?哪些是违规求助? 3515261
关于积分的说明 11177794
捐赠科研通 3250448
什么是DOI,文献DOI怎么找? 1795314
邀请新用户注册赠送积分活动 875781
科研通“疑难数据库(出版商)”最低求助积分说明 805073