Abstract Sirtuins are NAD + ‐dependent deacetylases acting as sensors in metabolic pathways and stress response. In mammals there are seven isoforms. The mitochondrial sirtuin 5 is a weak deacetylase but a very efficient demalonylase and desuccinylase; however, its substrate acyl specificity has not been systematically analyzed. Herein, we investigated a carbamoyl phosphate synthetase 1 derived peptide substrate and modified the lysine side chain systematically to determine the acyl specificity of Sirt5. From that point we designed six potent peptide‐based inhibitors that interact with the NAD + binding pocket. To characterize the interaction details causing the different substrate and inhibition properties we report several X‐ray crystal structures of Sirt5 complexed with these peptides. Our results reveal the Sirt5 acyl selectivity and its molecular basis and enable the design of inhibitors for Sirt5.