Although media and communication scholars have suggested various analytical methods for measuring and comparing news audience polarization across countries, we lack a systematic assessment of the metrics produced by these techniques. Using survey data from the 2016 Reuters Institute Digital News Report on news use in 26 countries, we address this gap through a resampling simulation experiment. Our simulation revealed a strong impact of analytical choices, which invited disparate interpretations in terms of how polarized news audiences are, how strongly audience polarization structurally varies between news environments, and how news audience polarization is distributed cross-nationally. Alternative choices led to profound differences in the compatibility, consistency, and validity of the empirical news audience polarization estimates. We conclude from these results that a more precise methodological understanding of news audience polarization metrics informs our capability to draw meaningful inferences from empirical work.