We study the time evolution of the compaction of a noncohesive or cohesive granular material submitted to shaking through experiments and a stochastic model. Beyond well-known empirical expressions, we show that the characteristic time scales depend on the number of objects in the assembly. For a noncohesive granular material, the compaction time scale is governed by the number of individual grains in the system. In the case of a cohesive granular material, a two-scale model (individual particles and clusters) allows one to mimic the time evolution of the compaction of an actual cohesive powder driven by horizontal vibrations. In this case, the two time scales are associated with the numbers of clusters and grains, respectively.