Can internet surveys represent the entire population? A practitioners’ analysis

A general concern with the representativeness of internet surveys is that they exclude the “offline” population that does not use the internet. We run a large-scale opinion survey with (1) onliners in internet-survey mode, (2) offliners in face-to-face mode, and (3) internet users in face-to-face mode. We find marked response differences between onliners and offliners in different modes (1 vs. 2). Response differences between onliners and offliners in the same face-to-face mode (2 vs. 3) disappear when controlling for background characteristics, indicating mode effects rather than unobserved population differences. Differences in background characteristics of onliners in the two modes (1 vs. 3) indicate that mode effects partly reflect sampling differences. In our setting, re-weighting online-survey observations appears a pragmatic solution when aiming at representativeness for the entire population.