InteraRec: Interactive Recommendations Using Multimodal Large Language Models

Numerous recommendation algorithms leverage weblogs, employing strategies such as collaborative filtering, content-based filtering, and hybrid methods to provide personalized recommendations to users. Weblogs, comprised of records detailing user activities on any website, offer valuable insights into user preferences, behavior, and interests. Despite the wealth of information weblogs provide, extracting relevant features requires extensive feature engineering. The intricate nature of the data also poses a challenge for interpretation, especially for non-experts. Additionally, they often fall short of capturing visual details and contextual nuances that influence user choices. In the present study, we introduce a sophisticated and interactive recommendation framework denoted as InteraRec, which diverges from conventional approaches that exclusively depend on weblogs for recommendation generation. This framework provides recommendations by capturing high-frequency screenshots of web pages as users navigate through a website. Leveraging advanced multimodal large language models (MLLMs), we extract valuable insights into user preferences from these screenshots by generating a user profile summary. Subsequently, we employ the InteraRec framework to extract relevant information from the summary to generate optimal recommendations. Through extensive experiments, we demonstrate the remarkable effectiveness of our recommendation system in providing users with valuable and personalized offerings.