Sexual dimorphism in growth has been frequently observed in fish species, which encourages the production of mono-sex population in aquaculture. Besides growth, feed intake and feed conversion efficiency (FCE) are two key traits for sustainable aquaculture. In this study, yellow catfish (Pelteobagrus fulvidraco), an important aquaculture fish with significantly sexual size dimorphism, was used as a model to investigate the associations among phenotypic traits, candidate SNPs and correlated gene expression. Sexual dimorphism in body length and body weight were observed as early as 68 days post hatching. One SNP in corticotropin releasing hormone receptor 1 (crhr1) has been shown to be significantly associated with body length and body weight but slightly differ between males and females. Compared to female fish, male fish had higher levels of feed intake and higher FCE, as well as higher expression of stat5b and agrp in liver and hypothalamus, and lower expression of crhr1, pomca and lepr in pituitary and hypothalamus. The high expression levels of stat5b and agrp as well as low level of crhr1 were positively associated with high FCE in both sexes, whereas the higher expressions of pomca and lepr were negatively with FCE only in female fish. Finally, we showed that the identified SNPs in stat5b and crhr1 were significantly associated with growth and FCE in yellow catfish but slightly differ between males and females. These results suggest that sexual dimorphism in phenotypic traits might be caused by sex difference in genetic architecture and gene regulation. Our study not only suggest that feed intake and feed efficiency are associated with the sexual size dimorphism, but also identify potential genes and genomic markers associated with these important traits that would benefit the genetic breeding in yellow catfish.