Particle reinforced elastomer matrix composites always show a size-dependently mechanical behavior when the particle size shrinks down to micro- or even nano-scale. A systematic investigation on microscopic mechanisms of such a size effect is carried out in this paper based on tensile and tear experiments of silicone rubber elastomer filled with surface modified silica particles, in which the particle size ranges from tens of to several hundreds of nanometers. It is found that, when the particle content is fixed, the ultimate strength, fracture toughness and fracture tensile strain of the composite exhibit monotonic increase with the decrease of particle size. In the composite filled with monodispersed submicro-particles, the improved strength is due to the hindering effect of particles on the crack propagation and the strong interface bonding between particles and matrix, while the improved toughness is mainly resulted from the crack-pinning around particles. In the composite filled with nano-sized particles, both the filler-hindering effect and the strong interface still contribute to the strength of the composite, while not only the crack pinning but also the interface debonding around nanoparticle aggregates will toughen the composite. Furthermore, a hierarchical network structure consisting of differently-sized aggregates and bounded rubbers endows the composite with a better load bearing capacity than the one filled with separately distributed fillers. As a result, the composite filled with small nano-nanoparticles shows remarkably improved mechanical properties in comparison with the composite filled with submicro-particles. The present work should provide insights for optimally designing a flexible composite with both desirable strength and toughness.