Natural conversations with a virtual being: How user experience with a current conversational AI model compares to expectations

Abstract The present work investigates the effect of natural conversations with virtual beings on user perceptions with a current conversational AI model (Meta's BlenderBot). To this aim, we designed a virtual being from a deep learning‐generated face and a conversational AI model acting as a virtual conversation partner in an online conferencing software and evaluated it in 11 perceptions of social attributes. Compared to prior expectations, participants perceived the virtual being as distinctly higher in warmth (engaging, empathic, and approachable) but lower in realism and credibility after 5 days of 10 min daily conversations (Study 1). Further, we explored the idea of simplifying the technical setup to reduce the technical entry barrier for such AI applications (Study 2). To this aim, we conducted several trials of fine‐tuning a small conversational model of 90 million parameters until its performance metrics improved. Testing this fine‐tuned model with users revealed that this model was not perceived differently from a large conversational model (1.4 billion parameters). In summary, our findings show that recent progress in conversational AI has added warmth‐related aspects to the user experience with virtual beings, and that fine‐tuning a conversational AI model can be effective to reduce technical complexity.