A feature selection method for classification based on ensemble of penalized logistic models

Classification accuracy of any classifier can be enhanced by performing the classification on selected informative features. Features selection methods are generally approached for the purpose. Combining multiple models known as ensemble method emerged as prominent method for achieving classification accuracy. These techniques can be considered for variable selection for accuracy gain and interpretability of a classifier. we propose an ensemble of penalized logistic models (EPLM) for feature selection. EPLM employs Lasso, adaptive Lasso and elastic net for feature selection. EPLM is applied to high dimensional microarray data sets and simulated data sets. State-of-the-art classifiers, Lasso, Random Forest (RF), Support Vector Machine (SVM) and K- Nearest Neighbors (KNN) are employed for assessment of EPLM. Experimental comparisons reveal that significant improvement in classification accuracy is achieved for all the classifiers considered. The proposed method is also compared to the Ensemble of subset of KNN classifiers (ESKNN). In comparison to ESKNN EPLM achieves better performance accuracy for all the classifiers on simulated and microarray data sets. Moreover, it is also observed that EPLM has selected significantly smaller number of features as compared to the full feature set.