Vitrimers---a recently invented new class of polymers---consist of covalent networks that can rearrange their topology via a bond shuffling mechanism, preserving the total number of network links. We introduce a patchy particle model whose dynamics directly mimic the bond exchange mechanism and reproduce the observed glass-forming ability. We calculate the free energy of this model in the limit of strong (chemical) bonds between the particles, both via the Wertheim thermodynamic perturbation theory and using computer simulations. The system exhibits an entropy-driven phase separation between a network phase and a dilute cluster gas, bringing new insight into the swelling behavior of vitrimers in solvents.