How well do collaboration quality estimation models generalize across authentic school contexts?

Abstract Multimodal learning analytics (MMLA) research has made significant progress in modelling collaboration quality for the purpose of understanding collaboration behaviour and building automated collaboration estimation models. Deploying these automated models in authentic classroom scenarios, however, remains a challenge. This paper presents findings from an evaluation of collaboration quality estimation models. We collected audio, video and log data from two different Estonian schools. These data were used in different combinations to build collaboration estimation models and then assessed across different subjects, different types of activities (collaborative‐writing, group‐discussion) and different schools. Our results suggest that the automated collaboration model can generalize to the context of different schools but with a 25% degradation in balanced accuracy (from 82% to 57%). Moreover, the results also indicate that multimodality brings more performance improvement in the case of group‐discussion‐based activities than collaborative‐writing‐based activities. Further, our results suggest that the video data could be an alternative for understanding collaboration in authentic settings where higher‐quality audio data cannot be collected due to contextual factors. The findings have implications for building automated collaboration estimation systems to assist teachers with monitoring their collaborative classrooms. Practitioners notes What is already known about this topic Multimodal learning analytics researchers have established several features as potential indicators for collaboration quality, e.g., speaking time or joint visual attention. The current state of the art has shown the feasibility of building automated collaboration quality models. Recent research has provided preliminary evidence of the generalizability of developed automated models across contexts different in terms of given task and subject. What does this paper add This paper offers collaboration indicators for different types of collaborative learning activities in authentic classroom settings. The paper includes a systematic investigation into collaboration quality automated model's generalizability across different tasks, types of tasks and schools. This paper also offers a comparison between different modalities' potential to estimate collaboration quality in authentic settings. Implications for practice The findings inform the development of automated collaboration monitoring systems for authentic classroom settings. This paper provides evidence on across‐school generalizability capabilities of collaboration quality estimation models.