Epigenetic enzyme-targeted therapy is a promising new development in the field of drug discovery. To date, histone deacetylases and DNA methyltransferases have been investigated as druggable epigenetic enzyme targets in cancer therapeutics. Histone methyltransferases and lysine demethylase inhibitors are the latest groups of epi-drugs being actively studied in clinical trials. KDM4s are JmjC domain-containing histone H3 lysine 9/36 demethylase enzymes, belonging to the 2-OG-dependent oxygenases, which are upregulated in multiple malignancies. In the recent years, these enzymes have captured much attention as a novel target in cancer therapy. Herein, we traverse the discovery path and current challenges in designing potent KDM4 inhibitors as potential anticancer agents. We discuss the considerable efforts and proposed future strategies to develop selective small molecule inhibitors of KDM4s, highlighting scaffold candidates and cyclic skeletons for which activity data, selectivity profiles and structure-activity relationships (SARs) have been studied.