Deep neural networks enable real-time monitoring of in-vehicle driver, facilitating the timely prediction of distractions, fatigue, and potential hazards.This technology is now integral to intelligent transportation systems.Recent research has exposed unreliable cross-dataset end-to-end driver behavior recognition due to overfitting, often referred to as "shortcut learning", resulting from limited data samples.In this paper, we introduce the Score-Softmax classifier, which addresses this issue by enhancing inter-class independence and Intra-class uncertainty.Motivated by human rating patterns, we designed a two-dimensional supervisory matrix based on marginal Gaussian distributions to train the classifier.Gaussian distributions help amplify intra-class uncertainty while ensuring the Score-Softmax classifier learns accurate knowledge.Furthermore, leveraging the summation of independent Gaussian distributed random variables, we introduced a multi-channel information fusion method.This strategy effectively resolves the multi-information fusion challenge for the Score-Softmax classifier.Concurrently, we substantiate the necessity of transfer learning and multidataset combination.We conducted cross-dataset experiments using the SFD, AUCDD-V1, and 100-Driver datasets, demonstrating that Score-Softmax improves cross-dataset performance without modifying the model architecture.This provides a new approach for enhancing neural network generalization.Additionally, our information fusion approach outperforms traditional methods.