Electrocardiogram (ECG) is considered as an essential diagnostic tool to investigate life-threatening cardiac abnormalities, such as arrhythmia and congestive heart failure. It is observed that the atrial arrhythmias and congestive heart failure are closely related, wherein, one promotes the other and their co-existence can increase the mortality rate. Timely diagnosis of these diseases is essential to prevent sudden cardiac failure. In this work, we employ a two-fold approach to classify arrhythmia, congestive heart failure, and normal sinus rhythm using ECG fragments. First, we use a traditional hand-crafted feature based model which involves extraction of a number of linear and non-linear features from the ECG fragments. The linear features capture the time-varying and scale of variability information, whereas the non-linear features help to extract the hidden complexity and quantify the uncertainty of the non-stationary signal. Second, an automatic feature learning based approach is employed which uses a pre-trained deep learning network to automatically extract the relevant detailed information from the ECG time–frequency representations. Finally, we explored the combined effect of the two approaches to diagnose the arrhythmias and congestive heart failure patterns. Additionally, this study makes novel use of the subject-level ECG classification. This work on Physionet database shows that the proposed combined system gives an accuracy, sensitivity, specificity, and precision of 99.06%, 99.14%, 99.68%, and 99.32%, respectively, which are better than the state-of-the-art systems. For subject-specific experiment, a further improvement in these performance metrics is obtained using the voting procedure.