This Letter develops a framework for digital holography at optical wavelengths by merging phase-shifting interferometry with single-pixel optical imaging based on compressive sensing. The field diffracted by an input object is sampled by Hadamard patterns with a liquid crystal spatial light modulator. The concept of a single-pixel camera is then adapted to perform interferometric imaging of the sampled diffraction pattern by using a Mach–Zehnder interferometer. Phase-shifting techniques together with the application of a backward light propagation algorithm allow the complex amplitude of the object under scrutiny to be resolved. A proof-of-concept experiment evaluating the phase distribution of an ophthalmic lens with compressive phase-shifting holography is provided.