Let $(M,g)$ be a smooth, compact Riemannian manifold and $\{\phi_h\}$ an $L^2$-normalized sequence of Laplace eigenfunctions, $-h^2\Delta_g\phi_h=\phi_h$. Given a smooth submanifold $H \subset M$ of codimension $k\geq 1$, we find conditions on the pair $(\{\phi_h\},H)$ for which $$ \Big|\int_H\phi_hd\sigma_H\Big|=o(h^{\frac{1-k}{2}}),\qquad h\to 0^+. $$ One such condition is that the set of conormal directions to $H$ that are recurrent has measure $0$. In particular, we show that the upper bound holds for any $H$ if $(M,g)$ is surface with Anosov geodesic flow or a manifold of constant negative curvature. The results are obtained by characterizing the behavior of the defect measures of eigenfunctions with maximal averages.