A Foundation Language-Image Model of the Retina (FLAIR): Encoding expert knowledge in text supervision

编码(内存) 计算机科学 基础(证据) 图像(数学) 流体衰减反转恢复 人工智能 自然语言处理 视网膜 计算机视觉 心理学 医学 神经科学 放射科 磁共振成像 考古 历史
作者
Julio Silva-Rodríguez,Hadi Chakor,Riadh Kobbi,José Dolz,Ismail Ben Ayed
出处
期刊:Medical Image Analysis [Elsevier BV]
卷期号:99: 103357-103357 被引量:9
标识
DOI:10.1016/j.media.2024.103357
摘要

Foundation vision-language models are currently transforming computer vision, and are on the rise in medical imaging fueled by their very promising generalization capabilities. However, the initial attempts to transfer this new paradigm to medical imaging have shown less impressive performances than those observed in other domains, due to the significant domain shift and the complex, expert domain knowledge inherent to medical-imaging tasks. Motivated by the need for domain-expert foundation models, we present FLAIR, a pre-trained vision-language model for universal retinal fundus image understanding. To this end, we compiled 38 open-access, mostly categorical fundus imaging datasets from various sources, with up to 101 different target conditions and 288,307 images. We integrate the expert's domain knowledge in the form of descriptive textual prompts, during both pre-training and zero-shot inference, enhancing the less-informative categorical supervision of the data. Such a textual expert's knowledge, which we compiled from the relevant clinical literature and community standards, describes the fine-grained features of the pathologies as well as the hierarchies and dependencies between them. We report comprehensive evaluations, which illustrate the benefit of integrating expert knowledge and the strong generalization capabilities of FLAIR under difficult scenarios with domain shifts or unseen categories. When adapted with a lightweight linear probe, FLAIR outperforms fully-trained, dataset-focused models, more so in the few-shot regimes. Interestingly, FLAIR outperforms by a wide margin larger-scale generalist image-language models and retina domain-specific self-supervised networks, which emphasizes the potential of embedding experts' domain knowledge and the limitations of generalist models in medical imaging. The pre-trained model is available at: https://github.com/jusiro/FLAIR.
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
刚刚
Alicer发布了新的文献求助20
1秒前
janejane发布了新的文献求助10
1秒前
1秒前
我是老大应助陶醉平松采纳,获得10
1秒前
王珊完成签到,获得积分10
2秒前
oxo关注了科研通微信公众号
2秒前
超级李包包完成签到,获得积分10
2秒前
酷酷发布了新的文献求助10
2秒前
2秒前
桐桐应助zhaoxiao采纳,获得10
3秒前
Akim应助xiaowang采纳,获得10
3秒前
不住完成签到,获得积分20
3秒前
zsc668完成签到 ,获得积分10
3秒前
3秒前
liverbool发布了新的文献求助10
4秒前
神说应助ffff采纳,获得10
4秒前
kx发布了新的文献求助10
4秒前
4秒前
4秒前
5秒前
5秒前
情怀应助一只耳采纳,获得10
5秒前
5秒前
无花果应助科研进化中采纳,获得10
6秒前
cnulee完成签到,获得积分10
6秒前
科研通AI5应助化羽归尘采纳,获得10
6秒前
沉默的小天鹅完成签到,获得积分10
6秒前
zhouting发布了新的文献求助10
6秒前
7秒前
研友_VZG7GZ应助暖阳采纳,获得10
7秒前
wanci应助德德采纳,获得10
8秒前
Wacky发布了新的文献求助10
8秒前
神勇猕猴桃完成签到,获得积分10
9秒前
所所应助天地侵略者采纳,获得10
9秒前
Ehgnix发布了新的文献求助10
9秒前
张丫丫发布了新的文献求助10
9秒前
9秒前
10秒前
小南子发布了新的文献求助10
11秒前
高分求助中
All the Birds of the World 4000
Production Logging: Theoretical and Interpretive Elements 3000
Les Mantodea de Guyane Insecta, Polyneoptera 2000
Machine Learning Methods in Geoscience 1000
Resilience of a Nation: A History of the Military in Rwanda 888
Musculoskeletal Pain - Market Insight, Epidemiology And Market Forecast - 2034 666
Crystal Nonlinear Optics: with SNLO examples (Second Edition) 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 工程类 有机化学 物理 生物化学 纳米技术 计算机科学 化学工程 内科学 复合材料 物理化学 电极 遗传学 量子力学 基因 冶金 催化作用
热门帖子
关注 科研通微信公众号,转发送积分 3735180
求助须知:如何正确求助?哪些是违规求助? 3279071
关于积分的说明 10012998
捐赠科研通 2995649
什么是DOI,文献DOI怎么找? 1643610
邀请新用户注册赠送积分活动 781338
科研通“疑难数据库(出版商)”最低求助积分说明 749369