Extracting the Full Story: A Multimodal Approach and Dataset to Crisis Summarization in Tweets

In our digitally connected world, the influx of microblog data poses a formidable challenge in extracting relevant information amid a continuous stream of updates. This challenge intensifies during crises, where the demand for timely and relevant information is crucial. Current summarization techniques often struggle with the intricacies of microblog data in such situations. To address this, our research explores crisis-related microblogs, recognizing the crucial role of multimedia content, such as images, in offering a comprehensive perspective. In response to these challenges, we introduce a multimodal extractive-abstractive summarization model. Leveraging a fusion of TF-IDF scoring and bigram filtering, coupled with the effectiveness of three distinct models—BIGBIRD, CLIP, and bootstrapping language-image pre-training (BLIP)—we aim to overcome the limitations of traditional extractive and text-only approaches. Our model is designed and evaluated on a newly curated Twitter dataset featuring 12 494 tweets and 3090 images across eight crisis events, each accompanied by gold-standard summaries. The experimental findings showcase the remarkable efficacy of our model, surpassing current benchmarks by a notable margin of 16% and 17%. This confirms our model's strength and its relevance in crisis scenarios with the crucial interplay of text and multimedia. Notably, our research contributes to multimodal, abstractive microblog summarization, addressing a key gap in the literature. It is also a valuable tool for swift information extraction in time-sensitive situations.