Quantifying human daily activities can provide relevant monitoring information about physical activities and fall risk, and wearable sensors are promising devices for activity monitoring in daily life scenarios. This paper designed a novel plantar pressure insole and inertial sensor system and presented classification algorithms for activity classification and fall detection. We designed each plantar pressure insole with eight thin uniaxial load cells placed in the key area of a foot. Twenty healthy young adults performed selected activities in the laboratory while wearing the plantar pressure shoes and six inertial measurement units (IMUs) on their feet, shanks, and thighs of both sides. We adopted the convolutional neural network (CNN), ensemble learning, and support vector machine (SVM) methods for activity classification, and the input data were inertial data, pressure data, and both data. We adopted CNN, RNN (recurrent neural network), LSTM (long short-term memory), and CNN-LSTM method for fall detection, and compared results before and after the Dempster-Shafer evidence theory. Results show that for activity classification, CNN with both inertial and plantar pressure data got the best accuracy of 97.1%. For fall detection, the accuracy of RNN, CNN, LSTM, and CNN-LSTM were 93.77%, 95.85%, 96.16%, and 97.76%, respectively. LSTM got comparable accuracy as CNN-LSTM but with much less latency. The presented wearable system and algorithms show good feasibility in activity classification and fall detection, which could serve as a foundation for physical activity monitoring and fall alert systems for elderly people.