Given a probability vector $x$ with its components sorted in non-increasing order, we consider the closed ball ${\mathcal{B}}^p_\epsilon(x)$ with $p \geq 1$ formed by the probability vectors whose $\ell^p$-norm distance to the center $x$ is less than or equal to a radius $\epsilon$. Here, we provide an order-theoretic characterization of these balls by using the majorization partial order. Unlike the case $p=1$ previously discussed in the literature, we find that the extremal probability vectors, in general, do not exist for the closed balls ${\mathcal{B}}^p_\epsilon(x)$ with $1