Development and Validation of Emoji Response Scales for Assessing Patient-Reported Outcomes

PURPOSE Emoji are digital images or icons used to express an idea or emotion in electronic communication. The purpose of this study was to develop and evaluate the psychometric properties of two patient-reported scales that incorporate emoji. METHODS The Emoji Response Scale developed for this study has two parts: the Emoji-Ordinal and Emoji-Mood scales. A pilot study was designed to validate the ordinal nature of the Emoji-Ordinal Scale. Twenty patients were shown all possible pairs of five emoji and asked to select the most positive from each pair. The psychometric ordering was assessed using Coombs unfolding and Thurstone scaling. A separate pilot study was designed to determine which emoji to include in the Emoji-Mood Scale. Ten common feelings experienced by patients with cancer were chosen by the study team. Patients and providers were asked to select the one emoji that best represented each feeling from the selection. The most commonly selected emotions and representative emoji were chosen for the Emoji-Mood Scale. In a randomized study of 294 patients, Spearman correlations, Wilcoxon tests, and Bland-Altman analyses determined the construct validity of the scales compared with Linear Analog Scale Assessments (LASA) and Patient-Reported Outcomes Measurement Information System (PROMIS) scores. RESULTS Ninety-five percent of patients selected the same ordering among the ordinal emoji, and Thurstone scaling confirmed the ordinal nature of the response scale. The construct validity of the scales was high with correlations between the Emoji-Ordinal Scale and the LASA scale of 0.70 for emotional well-being, 0.72 for physical well-being, 0.74 for overall quality of life, and –0.81 for fatigue. Emoji-Mood Scale ratings were strongly related to PROMIS global mental, global physical, fatigue, anxiety, sleep disturbance, and social activity scales ( P < .0001). CONCLUSION This study provides evidence that scales incorporating emoji are valid for collecting patient-reported outcomes.