Edge Guided GANs With Multi-Scale Contrastive Learning for Semantic Image Synthesis

计算机科学 人工智能 情报检索 自然语言处理
作者
Hao Tang,Guolei Sun,Nicu Sebe,Luc Van Gool
出处
期刊:IEEE Transactions on Pattern Analysis and Machine Intelligence [Institute of Electrical and Electronics Engineers]
卷期号:45 (12): 14435-14452 被引量:1
标识
DOI:10.1109/tpami.2023.3298721
摘要

We propose a novel e dge guided g enerative a dversarial n etwork with c ontrastive learning (ECGAN) for the challenging semantic image synthesis task. Although considerable improvements have been achieved by the community in the recent period, the quality of synthesized images is far from satisfactory due to three largely unresolved challenges. 1) The semantic labels do not provide detailed structural information, making it challenging to synthesize local details and structures; 2) The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss and thus cannot fully preserve the original semantic information, leading to semantically inconsistent results (e.g., missing small objects); 3) Existing semantic image synthesis methods focus on modeling “local” semantic information from a single input semantic layout. However, they ignore “global” semantic information of multiple input semantic layouts, i.e., semantic cross-relations between pixels across different input layouts. To tackle 1), we propose to use the edge as an intermediate representation which is further adopted to guide image generation via a proposed attention guided edge transfer module. Edge information is produced by a convolutional generator and introduces detailed structure information. To tackle 2), we design an effective module to selectively highlight class-dependent feature maps according to the original semantic layout to preserve the semantic information. To tackle 3), inspired by current methods in contrastive learning, we propose a novel contrastive learning method, which aims to enforce pixel embeddings belonging to the same semantic class to generate more similar image content than those from different classes. We further propose a novel multi-scale contrastive learning method that aims to push same-class features from different scales closer together being able to capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple input semantic layouts from different scales. Experiments on three challenging datasets show that our methods achieve significantly better results than state-of-the-art approaches. The source code is available at https://github.com/Ha0Tang/ECGAN .
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
更新
大幅提高文件上传限制,最高150M (2024-4-1)

科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
1秒前
深情安青应助同城代打采纳,获得10
1秒前
heyunfan完成签到,获得积分20
3秒前
3秒前
开心的饼干应助豆子采纳,获得20
3秒前
mmyhn应助科研通管家采纳,获得20
5秒前
Ava应助科研通管家采纳,获得10
5秒前
6秒前
李健应助科研通管家采纳,获得10
6秒前
美好斓应助科研通管家采纳,获得100
6秒前
ding应助科研通管家采纳,获得10
6秒前
kuikichu发布了新的文献求助10
6秒前
NexusExplorer应助科研通管家采纳,获得10
6秒前
顾矜应助科研通管家采纳,获得10
6秒前
Owen应助科研通管家采纳,获得80
6秒前
Jasper应助科研通管家采纳,获得10
6秒前
所所应助科研通管家采纳,获得10
6秒前
genomed应助科研通管家采纳,获得10
7秒前
搜集达人应助科研通管家采纳,获得10
7秒前
所所应助科研通管家采纳,获得10
7秒前
汉堡包应助科研通管家采纳,获得10
7秒前
7秒前
sunrase完成签到,获得积分10
8秒前
乐乐应助chen采纳,获得10
10秒前
10秒前
Singularity应助舒服的摇伽采纳,获得10
10秒前
燕双鹰发布了新的文献求助10
11秒前
科研一一发布了新的文献求助10
12秒前
12秒前
kuikichu完成签到,获得积分10
13秒前
13秒前
14秒前
邪恶摇粒绒完成签到,获得积分20
14秒前
lkk完成签到,获得积分10
14秒前
孤独的冬云完成签到 ,获得积分20
15秒前
17秒前
ss13l完成签到,获得积分10
17秒前
同城代打发布了新的文献求助10
18秒前
anna完成签到,获得积分10
18秒前
YBY发布了新的文献求助10
18秒前
高分求助中
Sustainability in Tides Chemistry 2000
Pharmacogenomics: Applications to Patient Care, Third Edition 1000
Studien zur Ideengeschichte der Gesetzgebung 1000
TM 5-855-1(Fundamentals of protective design for conventional weapons) 1000
Ethnicities: Media, Health, and Coping 700
Genera Insectorum: Mantodea, Fam. Mantidæ, Subfam. Hymenopodinæ (Classic Reprint) 600
Development of a new synthetic process for the synthesis of (S)-methadone and (S)- and (R)-isomethadone as NMDA receptor antagonists for the treatment of depression 500
热门求助领域 (近24小时)
化学 医学 生物 材料科学 工程类 有机化学 生物化学 物理 内科学 纳米技术 计算机科学 化学工程 复合材料 基因 遗传学 催化作用 物理化学 免疫学 量子力学 细胞生物学
热门帖子
关注 科研通微信公众号,转发送积分 3088876
求助须知:如何正确求助?哪些是违规求助? 2741034
关于积分的说明 7562810
捐赠科研通 2391175
什么是DOI,文献DOI怎么找? 1268179
科研通“疑难数据库(出版商)”最低求助积分说明 614019
版权声明 598684