A novel algorithm for collision-free navigation of a mobile robot on the campus road with pedestrian roaming is described in this paper. Proposed method uses the RGB-D depth sensor, utilizing optical flow estimation and object detection, to predict pedestrian locations. Given these environmental uncertainties, we present a Velocity Obstacle (VO) algorithm based on mobility rules to calculate the velocity has been presented. It is proposed to use the Markov Decision Process (MDP) for decision-making (to maneuver the robot whenever it approaches the target). The proposed algorithm is a hybrid combination of deep learning and model-based techniques and provides better results in terms of navigation time and collision avoidance success rate than conventional algorithms. The real-time performance of the proposed algorithm is highlighted using a real-world dynamic scenario for an up-to-date kid ride car that has been redesigned to work as an autonomous mobile robot.