A radiomics-based classifier to distinguish phyllodes tumor and fibroadenoma on gray-scale breast ultrasonography was developed and validated. A total of 93 radiomics features were extracted from representative transverse plane ultrasound images of 182 fibroepithelial lesions initially diagnosed by core needle biopsy. High-throughput radiomics features were selected using the intra-class correlation coefficient between two radiologist readers and the Least Absolute Shrinkage and Selection Operator regression through 10-fold cross-validation. When applied to the validation set, the radiomics classifier for the differentiation of phyllodes tumors and benign/fibroadenomas achieved an area under the receiver operating characteristic curve of 0.765 (95% confidence interval [CI]: 0.597–0.888) with an accuracy of 0.703 (sensitivity: 0.857; specificity: 0.5). Our radiomics signature-based classifier may help predict phyllodes tumors among fibroepithelial lesions on breast ultrasonography.