We prove spectral instability of peakons in the $b$-family of Camassa--Holm equations that includes the integrable cases of $b = 2$ and $b = 3$. We start with a linearized operator defined on functions in $H^1(\mathbb{R}) \cap W^{1,\infty}(\mathbb{R})$ and extend it to a linearized operator defined on weaker functions in $L^2(\mathbb{R})$. For $b \neq \frac{5}{2}$, the spectrum of the linearized operator in $L^2(\mathbb{R})$ is proved to cover a closed vertical strip of the complex plane. For $b = \frac{5}{2}$, the strip shrinks to the imaginary axis, but an additional pair of real eigenvalues exists due to projections to the peakon and its spatial translation. The spectral instability results agree with the linear instability results in the case of the Camassa--Holm equation for $b = 2$.